2026-02-23T13:00:05.306 INFO:root:teuthology version: 1.2.4.dev9+gfa17720d0 2026-02-23T13:00:05.323 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com 2026-02-23T13:00:05.407 INFO:teuthology.run:Config: archive_path: /home/teuthworker/mnt/teuthology/shraddhaag-2026-02-23_12:27:41-crimson-rados-main-distro-debug-trial/66461 branch: main description: crimson-rados/thrash/{0-crimson_install 0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} clusters/{crimson-fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/cephadm objectstore/seastore/rbm$/{crimson_seastore_rbm} thrashers/{default} thrashosds-health workloads/radosbench-high-concurrency} email: null first_in_suite: false flavor: debug job_id: '66461' kernel: branch: distro kdb: true sha1: distro last_in_suite: false machine_type: trial name: shraddhaag-2026-02-23_12:27:41-crimson-rados-main-distro-debug-trial no_nested_subset: false os_type: centos os_version: 9.stream overrides: admin_socket: branch: main ceph: conf: client.0: debug ms: 1 debug objecter: 20 debug rados: 20 global: crimson_allow_pg_split: false enable experimental unrecoverable data corrupting features: crimson ms client mode: crc ms cluster mode: crc ms mon client mode: crc ms mon cluster mode: crc ms mon service mode: crc ms service mode: crc osd_pool_default_min_size: 2 osd_pool_default_size: 3 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 50 mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 osd pool default crimson: true paxos service trim min: 10 osd: crimson cpu num: 4 crimson osd obc lru size: 10 debug ms: 20 debug osd: 20 debug seastore: 20 debug seastore backref: 20 debug seastore cache: 20 debug seastore cleaner: 20 debug seastore device: 20 debug seastore epm: 20 debug seastore fixedkv tree: 20 debug seastore journal: 20 debug seastore lba: 20 debug seastore odata: 20 debug seastore omap: 20 debug seastore onode: 20 debug seastore t: 20 debug seastore tm: 20 osd blocked scrub grace period: 3600 osd debug reject backfill probability: 0.3 osd delete sleep: 1 osd max backfills: 3 osd max markdown count: 1000 osd objectstore: seastore osd scrub max interval: 120 osd scrub min interval: 60 osd shutdown pgref assert: true osd snap trim sleep: 2 osd target transaction size: 5 seastore cachepin size pershard: 64M seastore main device type: RANDOM_BLOCK_SSD seastore max concurrent transactions: 8 crimson_compat: true flavor: debug log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(POOL_APP_NOT_ENABLED\) - but it is still running - objects unfound and apparently lost - overall HEALTH_ - \(OSDMAP_FLAGS\) - \(OSD_ - \(PG_ - \(POOL_ - \(CACHE_POOL_ - \(SMALLER_PGP_NUM\) - \(OBJECT_ - SLOW_OPS - \(REQUEST_SLOW\) - \(TOO_FEW_PGS\) - slow request - timeout on replica - late reservation from - MON_DOWN - OSDMAP_FLAGS - OSD_DOWN - PG_DEGRADED - PG_AVAILABILITY - POOL_APP_NOT_ENABLED - mons down - mon down - out of quorum - noscrub - nodeep-scrub - Degraded data redundancy - is down - osds down - pg .*? is .*?degraded.*?, acting - pg .*? is stuck - pg degraded - PG_BACKFILL_FULL - Low space hindering backfill .*? backfill_toofull - OSD_ROOT_DOWN - pgs degraded - pgs undersized - is active.*backfill_toofull.* sha1: e371433fcd1390a0b82cfb18effc422c1f5d8c1d ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: debug sha1: e371433fcd1390a0b82cfb18effc422c1f5d8c1d thrashosds: min_in: 4 workunit: branch: wip-shraddhaag-enable-cephadm-tests env: CRIMSON_COMPAT: '1' sha1: a78040ddef3ba5a489697ca2fe95df59a7d8bd08 owner: scheduled_shraddhaag@soko04.front.sepia.ceph.com priority: 49 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.x - osd.0 - osd.1 - client.0 - node-exporter.a - - osd.2 - osd.3 - client.1 - prometheus.a - node-exporter.b seed: 7225 sha1: e371433fcd1390a0b82cfb18effc422c1f5d8c1d sleep_before_teardown: 0 suite: crimson-rados suite_branch: wip-shraddhaag-enable-cephadm-tests suite_path: /home/teuthworker/src/github.com_shraddhaag_ceph_a78040ddef3ba5a489697ca2fe95df59a7d8bd08/qa suite_relpath: qa suite_repo: https://github.com/shraddhaag/ceph.git suite_sha1: a78040ddef3ba5a489697ca2fe95df59a7d8bd08 targets: trial068.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGHEQ/ralpERbp6IXzjaCZbbWuE50Y3KnlRJypLlq53rar/9NyxS18p9hQ6Ijj55jhBD/oA16bjUTeQZcHk00Ww= trial185.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBQPF59gYdIIaAbtbGnYdqHKxlIK57NZrWDRY0riHhH7hiGOEee//r9qRIwN3EmoSr1T1Dc1wnL6S7/cX6nKkjE= tasks: - install: extra_packages: deb: - ceph-osd-crimson - ceph-osd-crimson-dbg rpm: - ceph-osd-crimson - ceph-osd-crimson-debuginfo - pexec: all: - sudo update-alternatives --set ceph-osd /usr/bin/ceph-osd-crimson - cephadm: conf: mgr: debug mgr: 20 debug ms: 1 debug osd: 10 osd_type: crimson - cephadm.shell: mon.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - thrashosds: ceph_objectstore_tool: false chance_inject_pause_short: 0 chance_reset_purged_snaps_last: 0 chance_thrash_cluster_full: 0 chance_thrash_pg_upmap: 0 chance_trim_stale_osdmaps: 0 dump_ops_enable: false min_in: 3 noscrub_toggle_delay: 0 reweight_osd: 0 sighup_delay: 0 thrash_primary_affinity: false timeout: 2400 - full_sequential: - radosbench: clients: - client.0 concurrency: 128 size: 8192 time: 90 - radosbench: clients: - client.0 concurrency: 128 size: 8192 time: 90 - radosbench: clients: - client.0 concurrency: 128 size: 8192 time: 90 - radosbench: clients: - client.0 concurrency: 128 size: 8192 time: 90 - radosbench: clients: - client.0 concurrency: 128 size: 8192 time: 90 - radosbench: clients: - client.0 concurrency: 128 size: 8192 time: 90 - radosbench: clients: - client.0 concurrency: 128 size: 8192 time: 90 - radosbench: clients: - client.0 concurrency: 128 size: 8192 time: 90 teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: fa17720d0088c3ac28e473468bfc79eeaff5cd38 timestamp: 2026-02-23_12:27:41 tube: trial user: shraddhaag verbose: false verify_ceph_hash: false worker_log: /home/teuthworker/mnt/teuthology/worker_logs/dispatcher.trial.3318653 2026-02-23T13:00:05.407 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_shraddhaag_ceph_a78040ddef3ba5a489697ca2fe95df59a7d8bd08/qa; will attempt to use it 2026-02-23T13:00:05.407 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_shraddhaag_ceph_a78040ddef3ba5a489697ca2fe95df59a7d8bd08/qa/tasks 2026-02-23T13:00:05.407 INFO:teuthology.run_tasks:Running task internal.save_config... 2026-02-23T13:00:05.409 INFO:teuthology.task.internal:Saving configuration 2026-02-23T13:00:05.417 INFO:teuthology.run_tasks:Running task internal.check_lock... 2026-02-23T13:00:05.429 INFO:teuthology.task.internal.check_lock:Checking locks... 2026-02-23T13:00:05.485 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'trial068.front.sepia.ceph.com', 'description': '/home/teuthworker/mnt/teuthology/shraddhaag-2026-02-23_12:27:41-crimson-rados-main-distro-debug-trial/66461', 'up': True, 'machine_type': 'trial', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2026-02-23 12:56:40.031604', 'locked_by': 'scheduled_shraddhaag@soko04.front.sepia.ceph.com', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGHEQ/ralpERbp6IXzjaCZbbWuE50Y3KnlRJypLlq53rar/9NyxS18p9hQ6Ijj55jhBD/oA16bjUTeQZcHk00Ww='} 2026-02-23T13:00:05.541 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'trial185.front.sepia.ceph.com', 'description': '/home/teuthworker/mnt/teuthology/shraddhaag-2026-02-23_12:27:41-crimson-rados-main-distro-debug-trial/66461', 'up': True, 'machine_type': 'trial', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2026-02-23 12:56:40.030813', 'locked_by': 'scheduled_shraddhaag@soko04.front.sepia.ceph.com', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBQPF59gYdIIaAbtbGnYdqHKxlIK57NZrWDRY0riHhH7hiGOEee//r9qRIwN3EmoSr1T1Dc1wnL6S7/cX6nKkjE='} 2026-02-23T13:00:05.541 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2026-02-23T13:00:05.553 INFO:teuthology.task.internal:roles: ubuntu@trial068.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'client.0', 'node-exporter.a'] 2026-02-23T13:00:05.553 INFO:teuthology.task.internal:roles: ubuntu@trial185.front.sepia.ceph.com - ['osd.2', 'osd.3', 'client.1', 'prometheus.a', 'node-exporter.b'] 2026-02-23T13:00:05.553 INFO:teuthology.run_tasks:Running task console_log... 2026-02-23T13:00:05.682 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7799eadfea20>, signals=[15]) 2026-02-23T13:00:05.683 INFO:teuthology.run_tasks:Running task internal.connect... 2026-02-23T13:00:05.694 INFO:teuthology.task.internal:Opening connections... 2026-02-23T13:00:05.694 DEBUG:teuthology.task.internal:connecting to ubuntu@trial068.front.sepia.ceph.com 2026-02-23T13:00:05.695 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial068.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2026-02-23T13:00:05.759 DEBUG:teuthology.task.internal:connecting to ubuntu@trial185.front.sepia.ceph.com 2026-02-23T13:00:05.759 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial185.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2026-02-23T13:00:05.822 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2026-02-23T13:00:05.833 DEBUG:teuthology.orchestra.run.trial068:> uname -m 2026-02-23T13:00:05.848 INFO:teuthology.orchestra.run.trial068.stdout:x86_64 2026-02-23T13:00:05.848 DEBUG:teuthology.orchestra.run.trial068:> cat /etc/os-release 2026-02-23T13:00:05.904 INFO:teuthology.orchestra.run.trial068.stdout:NAME="CentOS Stream" 2026-02-23T13:00:05.904 INFO:teuthology.orchestra.run.trial068.stdout:VERSION="9" 2026-02-23T13:00:05.904 INFO:teuthology.orchestra.run.trial068.stdout:ID="centos" 2026-02-23T13:00:05.904 INFO:teuthology.orchestra.run.trial068.stdout:ID_LIKE="rhel fedora" 2026-02-23T13:00:05.904 INFO:teuthology.orchestra.run.trial068.stdout:VERSION_ID="9" 2026-02-23T13:00:05.904 INFO:teuthology.orchestra.run.trial068.stdout:PLATFORM_ID="platform:el9" 2026-02-23T13:00:05.904 INFO:teuthology.orchestra.run.trial068.stdout:PRETTY_NAME="CentOS Stream 9" 2026-02-23T13:00:05.904 INFO:teuthology.orchestra.run.trial068.stdout:ANSI_COLOR="0;31" 2026-02-23T13:00:05.904 INFO:teuthology.orchestra.run.trial068.stdout:LOGO="fedora-logo-icon" 2026-02-23T13:00:05.904 INFO:teuthology.orchestra.run.trial068.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2026-02-23T13:00:05.904 INFO:teuthology.orchestra.run.trial068.stdout:HOME_URL="https://centos.org/" 2026-02-23T13:00:05.904 INFO:teuthology.orchestra.run.trial068.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2026-02-23T13:00:05.904 INFO:teuthology.orchestra.run.trial068.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2026-02-23T13:00:05.904 INFO:teuthology.orchestra.run.trial068.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2026-02-23T13:00:05.905 INFO:teuthology.lock.ops:Updating trial068.front.sepia.ceph.com on lock server 2026-02-23T13:00:06.098 DEBUG:teuthology.orchestra.run.trial185:> uname -m 2026-02-23T13:00:06.114 INFO:teuthology.orchestra.run.trial185.stdout:x86_64 2026-02-23T13:00:06.114 DEBUG:teuthology.orchestra.run.trial185:> cat /etc/os-release 2026-02-23T13:00:06.170 INFO:teuthology.orchestra.run.trial185.stdout:NAME="CentOS Stream" 2026-02-23T13:00:06.171 INFO:teuthology.orchestra.run.trial185.stdout:VERSION="9" 2026-02-23T13:00:06.171 INFO:teuthology.orchestra.run.trial185.stdout:ID="centos" 2026-02-23T13:00:06.171 INFO:teuthology.orchestra.run.trial185.stdout:ID_LIKE="rhel fedora" 2026-02-23T13:00:06.171 INFO:teuthology.orchestra.run.trial185.stdout:VERSION_ID="9" 2026-02-23T13:00:06.171 INFO:teuthology.orchestra.run.trial185.stdout:PLATFORM_ID="platform:el9" 2026-02-23T13:00:06.171 INFO:teuthology.orchestra.run.trial185.stdout:PRETTY_NAME="CentOS Stream 9" 2026-02-23T13:00:06.171 INFO:teuthology.orchestra.run.trial185.stdout:ANSI_COLOR="0;31" 2026-02-23T13:00:06.171 INFO:teuthology.orchestra.run.trial185.stdout:LOGO="fedora-logo-icon" 2026-02-23T13:00:06.171 INFO:teuthology.orchestra.run.trial185.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2026-02-23T13:00:06.172 INFO:teuthology.orchestra.run.trial185.stdout:HOME_URL="https://centos.org/" 2026-02-23T13:00:06.172 INFO:teuthology.orchestra.run.trial185.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2026-02-23T13:00:06.172 INFO:teuthology.orchestra.run.trial185.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2026-02-23T13:00:06.172 INFO:teuthology.orchestra.run.trial185.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2026-02-23T13:00:06.172 INFO:teuthology.lock.ops:Updating trial185.front.sepia.ceph.com on lock server 2026-02-23T13:00:06.278 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2026-02-23T13:00:06.290 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2026-02-23T13:00:06.302 INFO:teuthology.task.internal:Checking for old test directory... 2026-02-23T13:00:06.302 DEBUG:teuthology.orchestra.run.trial068:> test '!' -e /home/ubuntu/cephtest 2026-02-23T13:00:06.304 DEBUG:teuthology.orchestra.run.trial185:> test '!' -e /home/ubuntu/cephtest 2026-02-23T13:00:06.319 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2026-02-23T13:00:06.330 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2026-02-23T13:00:06.330 DEBUG:teuthology.orchestra.run.trial068:> test -z $(ls -A /var/lib/ceph) 2026-02-23T13:00:06.360 DEBUG:teuthology.orchestra.run.trial185:> test -z $(ls -A /var/lib/ceph) 2026-02-23T13:00:06.376 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2026-02-23T13:00:06.546 INFO:teuthology.run_tasks:Running task kernel... 2026-02-23T13:00:06.565 INFO:teuthology.task.kernel:normalize config orig: {'branch': 'distro', 'kdb': True, 'sha1': 'distro'} 2026-02-23T13:00:06.566 INFO:teuthology.task.kernel:config {'mon.a': {'branch': 'distro', 'kdb': True, 'sha1': 'distro'}, 'osd.2': {'branch': 'distro', 'kdb': True, 'sha1': 'distro'}}, timeout 300 2026-02-23T13:00:06.566 DEBUG:teuthology.orchestra.run.trial068:> test -f /run/.containerenv -o -f /.dockerenv 2026-02-23T13:00:06.566 DEBUG:teuthology.orchestra.run.trial185:> test -f /run/.containerenv -o -f /.dockerenv 2026-02-23T13:00:06.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-23T13:00:06.580 DEBUG:teuthology.orchestra.run.trial068:> uname -r 2026-02-23T13:00:06.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-23T13:00:06.580 DEBUG:teuthology.orchestra.run.trial185:> uname -r 2026-02-23T13:00:06.636 INFO:teuthology.orchestra.run.trial068.stdout:5.14.0-681.el9.x86_64 2026-02-23T13:00:06.636 INFO:teuthology.task.kernel:Running kernel on trial068: 5.14.0-681.el9.x86_64 2026-02-23T13:00:06.636 DEBUG:teuthology.orchestra.run.trial068:> sudo yum install -y kernel 2026-02-23T13:00:06.636 INFO:teuthology.orchestra.run.trial185.stdout:5.14.0-681.el9.x86_64 2026-02-23T13:00:06.637 INFO:teuthology.task.kernel:Running kernel on trial185: 5.14.0-681.el9.x86_64 2026-02-23T13:00:06.637 DEBUG:teuthology.orchestra.run.trial185:> sudo yum install -y kernel 2026-02-23T13:00:07.816 INFO:teuthology.orchestra.run.trial068.stdout:CentOS Stream 9 - BaseOS 11 MB/s | 8.9 MB 00:00 2026-02-23T13:00:12.187 INFO:teuthology.orchestra.run.trial185.stdout:CentOS Stream 9 - BaseOS 1.7 MB/s | 8.9 MB 00:05 2026-02-23T13:00:13.476 INFO:teuthology.orchestra.run.trial068.stdout:CentOS Stream 9 - AppStream 5.2 MB/s | 27 MB 00:05 2026-02-23T13:00:17.021 INFO:teuthology.orchestra.run.trial068.stdout:CentOS Stream 9 - CRB 6.6 MB/s | 7.8 MB 00:01 2026-02-23T13:00:18.239 INFO:teuthology.orchestra.run.trial068.stdout:CentOS Stream 9 - Extras packages 42 kB/s | 20 kB 00:00 2026-02-23T13:00:18.793 INFO:teuthology.orchestra.run.trial068.stdout:Extra Packages for Enterprise Linux 41 MB/s | 20 MB 00:00 2026-02-23T13:00:21.960 INFO:teuthology.orchestra.run.trial068.stdout:lab-extras 1.0 MB/s | 50 kB 00:00 2026-02-23T13:00:22.967 INFO:teuthology.orchestra.run.trial068.stdout:Package kernel-5.14.0-645.el9.x86_64 is already installed. 2026-02-23T13:00:22.967 INFO:teuthology.orchestra.run.trial068.stdout:Package kernel-5.14.0-681.el9.x86_64 is already installed. 2026-02-23T13:00:22.984 INFO:teuthology.orchestra.run.trial185.stdout:CentOS Stream 9 - AppStream 2.6 MB/s | 27 MB 00:10 2026-02-23T13:00:22.986 INFO:teuthology.orchestra.run.trial068.stdout:Dependencies resolved. 2026-02-23T13:00:22.987 INFO:teuthology.orchestra.run.trial068.stdout:Nothing to do. 2026-02-23T13:00:22.987 INFO:teuthology.orchestra.run.trial068.stdout:Complete! 2026-02-23T13:00:23.012 DEBUG:teuthology.orchestra.run.trial068:> echo no | sudo yum reinstall kernel || true 2026-02-23T13:00:23.351 INFO:teuthology.orchestra.run.trial068.stdout:Last metadata expiration check: 0:00:02 ago on Mon 23 Feb 2026 01:00:21 PM UTC. 2026-02-23T13:00:23.431 INFO:teuthology.orchestra.run.trial068.stdout:Dependencies resolved. 2026-02-23T13:00:23.432 INFO:teuthology.orchestra.run.trial068.stdout:================================================================================ 2026-02-23T13:00:23.432 INFO:teuthology.orchestra.run.trial068.stdout: Package Architecture Version Repository Size 2026-02-23T13:00:23.432 INFO:teuthology.orchestra.run.trial068.stdout:================================================================================ 2026-02-23T13:00:23.432 INFO:teuthology.orchestra.run.trial068.stdout:Reinstalling: 2026-02-23T13:00:23.432 INFO:teuthology.orchestra.run.trial068.stdout: kernel x86_64 5.14.0-681.el9 baseos 803 k 2026-02-23T13:00:23.432 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T13:00:23.432 INFO:teuthology.orchestra.run.trial068.stdout:Transaction Summary 2026-02-23T13:00:23.433 INFO:teuthology.orchestra.run.trial068.stdout:================================================================================ 2026-02-23T13:00:23.433 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T13:00:23.433 INFO:teuthology.orchestra.run.trial068.stdout:Total download size: 803 k 2026-02-23T13:00:23.433 INFO:teuthology.orchestra.run.trial068.stdout:Installed size: 0 2026-02-23T13:00:23.433 INFO:teuthology.orchestra.run.trial068.stderr:Operation aborted. 2026-02-23T13:00:23.452 INFO:teuthology.orchestra.run.trial068.stdout:Is this ok [y/N]: 2026-02-23T13:00:23.452 DEBUG:teuthology.orchestra.run.trial068:> sudo yum reinstall -y kernel || true 2026-02-23T13:00:23.752 INFO:teuthology.orchestra.run.trial068.stdout:Last metadata expiration check: 0:00:02 ago on Mon 23 Feb 2026 01:00:21 PM UTC. 2026-02-23T13:00:23.831 INFO:teuthology.orchestra.run.trial068.stdout:Dependencies resolved. 2026-02-23T13:00:23.832 INFO:teuthology.orchestra.run.trial068.stdout:================================================================================ 2026-02-23T13:00:23.832 INFO:teuthology.orchestra.run.trial068.stdout: Package Architecture Version Repository Size 2026-02-23T13:00:23.832 INFO:teuthology.orchestra.run.trial068.stdout:================================================================================ 2026-02-23T13:00:23.832 INFO:teuthology.orchestra.run.trial068.stdout:Reinstalling: 2026-02-23T13:00:23.832 INFO:teuthology.orchestra.run.trial068.stdout: kernel x86_64 5.14.0-681.el9 baseos 803 k 2026-02-23T13:00:23.832 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T13:00:23.832 INFO:teuthology.orchestra.run.trial068.stdout:Transaction Summary 2026-02-23T13:00:23.832 INFO:teuthology.orchestra.run.trial068.stdout:================================================================================ 2026-02-23T13:00:23.833 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T13:00:23.833 INFO:teuthology.orchestra.run.trial068.stdout:Total download size: 803 k 2026-02-23T13:00:23.833 INFO:teuthology.orchestra.run.trial068.stdout:Installed size: 0 2026-02-23T13:00:23.833 INFO:teuthology.orchestra.run.trial068.stdout:Downloading Packages: 2026-02-23T13:00:24.924 INFO:teuthology.orchestra.run.trial068.stdout:kernel-5.14.0-681.el9.x86_64.rpm 779 kB/s | 803 kB 00:01 2026-02-23T13:00:24.924 INFO:teuthology.orchestra.run.trial068.stdout:-------------------------------------------------------------------------------- 2026-02-23T13:00:24.924 INFO:teuthology.orchestra.run.trial068.stdout:Total 735 kB/s | 803 kB 00:01 2026-02-23T13:00:24.935 INFO:teuthology.orchestra.run.trial068.stdout:Running transaction check 2026-02-23T13:00:24.968 INFO:teuthology.orchestra.run.trial068.stdout:Transaction check succeeded. 2026-02-23T13:00:24.968 INFO:teuthology.orchestra.run.trial068.stdout:Running transaction test 2026-02-23T13:00:24.972 INFO:teuthology.orchestra.run.trial068.stdout:Transaction test succeeded. 2026-02-23T13:00:24.973 INFO:teuthology.orchestra.run.trial068.stdout:Running transaction 2026-02-23T13:00:24.996 INFO:teuthology.orchestra.run.trial068.stdout: Preparing : 1/1 2026-02-23T13:00:25.000 INFO:teuthology.orchestra.run.trial068.stdout: Reinstalling : kernel-5.14.0-681.el9.x86_64 1/2 2026-02-23T13:00:25.000 INFO:teuthology.orchestra.run.trial068.stdout: Cleanup : kernel-5.14.0-681.el9.x86_64 2/2 2026-02-23T13:00:25.113 INFO:teuthology.orchestra.run.trial068.stdout: Running scriptlet: kernel-5.14.0-681.el9.x86_64 2/2 2026-02-23T13:00:25.113 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : kernel-5.14.0-681.el9.x86_64 1/2 2026-02-23T13:00:25.164 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : kernel-5.14.0-681.el9.x86_64 2/2 2026-02-23T13:00:25.164 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T13:00:25.164 INFO:teuthology.orchestra.run.trial068.stdout:Reinstalled: 2026-02-23T13:00:25.164 INFO:teuthology.orchestra.run.trial068.stdout: kernel-5.14.0-681.el9.x86_64 2026-02-23T13:00:25.164 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T13:00:25.164 INFO:teuthology.orchestra.run.trial068.stdout:Complete! 2026-02-23T13:00:25.190 DEBUG:teuthology.orchestra.run.trial068:> rpm -q kernel | sort -rV | head -n 1 2026-02-23T13:00:25.215 INFO:teuthology.orchestra.run.trial068.stdout:kernel-5.14.0-681.el9.x86_64 2026-02-23T13:00:25.215 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-681.el9.x86_64 2026-02-23T13:00:25.216 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2026-02-23T13:00:25.216 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2026-02-23T13:00:25.216 DEBUG:teuthology.orchestra.run.trial068:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2026-02-23T13:00:25.279 INFO:teuthology.orchestra.run.trial068.stdout:ttyS1 2026-02-23T13:00:25.305 DEBUG:teuthology.parallel:result is None 2026-02-23T13:00:26.548 INFO:teuthology.orchestra.run.trial185.stdout:CentOS Stream 9 - CRB 6.4 MB/s | 7.8 MB 00:01 2026-02-23T13:00:28.080 INFO:teuthology.orchestra.run.trial185.stdout:CentOS Stream 9 - Extras packages 25 kB/s | 20 kB 00:00 2026-02-23T13:00:28.914 INFO:teuthology.orchestra.run.trial185.stdout:Extra Packages for Enterprise Linux 26 MB/s | 20 MB 00:00 2026-02-23T13:00:32.026 INFO:teuthology.orchestra.run.trial185.stdout:lab-extras 2.6 MB/s | 50 kB 00:00 2026-02-23T13:00:33.035 INFO:teuthology.orchestra.run.trial185.stdout:Package kernel-5.14.0-645.el9.x86_64 is already installed. 2026-02-23T13:00:33.035 INFO:teuthology.orchestra.run.trial185.stdout:Package kernel-5.14.0-681.el9.x86_64 is already installed. 2026-02-23T13:00:33.054 INFO:teuthology.orchestra.run.trial185.stdout:Dependencies resolved. 2026-02-23T13:00:33.055 INFO:teuthology.orchestra.run.trial185.stdout:Nothing to do. 2026-02-23T13:00:33.055 INFO:teuthology.orchestra.run.trial185.stdout:Complete! 2026-02-23T13:00:33.078 DEBUG:teuthology.orchestra.run.trial185:> echo no | sudo yum reinstall kernel || true 2026-02-23T13:00:33.380 INFO:teuthology.orchestra.run.trial185.stdout:Last metadata expiration check: 0:00:01 ago on Mon 23 Feb 2026 01:00:32 PM UTC. 2026-02-23T13:00:33.460 INFO:teuthology.orchestra.run.trial185.stdout:Dependencies resolved. 2026-02-23T13:00:33.461 INFO:teuthology.orchestra.run.trial185.stdout:================================================================================ 2026-02-23T13:00:33.461 INFO:teuthology.orchestra.run.trial185.stdout: Package Architecture Version Repository Size 2026-02-23T13:00:33.461 INFO:teuthology.orchestra.run.trial185.stdout:================================================================================ 2026-02-23T13:00:33.461 INFO:teuthology.orchestra.run.trial185.stdout:Reinstalling: 2026-02-23T13:00:33.461 INFO:teuthology.orchestra.run.trial185.stdout: kernel x86_64 5.14.0-681.el9 baseos 803 k 2026-02-23T13:00:33.461 INFO:teuthology.orchestra.run.trial185.stdout: 2026-02-23T13:00:33.461 INFO:teuthology.orchestra.run.trial185.stdout:Transaction Summary 2026-02-23T13:00:33.461 INFO:teuthology.orchestra.run.trial185.stdout:================================================================================ 2026-02-23T13:00:33.461 INFO:teuthology.orchestra.run.trial185.stdout: 2026-02-23T13:00:33.462 INFO:teuthology.orchestra.run.trial185.stdout:Total download size: 803 k 2026-02-23T13:00:33.462 INFO:teuthology.orchestra.run.trial185.stdout:Installed size: 0 2026-02-23T13:00:33.462 INFO:teuthology.orchestra.run.trial185.stderr:Operation aborted. 2026-02-23T13:00:33.481 INFO:teuthology.orchestra.run.trial185.stdout:Is this ok [y/N]: 2026-02-23T13:00:33.482 DEBUG:teuthology.orchestra.run.trial185:> sudo yum reinstall -y kernel || true 2026-02-23T13:00:33.781 INFO:teuthology.orchestra.run.trial185.stdout:Last metadata expiration check: 0:00:01 ago on Mon 23 Feb 2026 01:00:32 PM UTC. 2026-02-23T13:00:33.861 INFO:teuthology.orchestra.run.trial185.stdout:Dependencies resolved. 2026-02-23T13:00:33.862 INFO:teuthology.orchestra.run.trial185.stdout:================================================================================ 2026-02-23T13:00:33.862 INFO:teuthology.orchestra.run.trial185.stdout: Package Architecture Version Repository Size 2026-02-23T13:00:33.862 INFO:teuthology.orchestra.run.trial185.stdout:================================================================================ 2026-02-23T13:00:33.862 INFO:teuthology.orchestra.run.trial185.stdout:Reinstalling: 2026-02-23T13:00:33.862 INFO:teuthology.orchestra.run.trial185.stdout: kernel x86_64 5.14.0-681.el9 baseos 803 k 2026-02-23T13:00:33.862 INFO:teuthology.orchestra.run.trial185.stdout: 2026-02-23T13:00:33.862 INFO:teuthology.orchestra.run.trial185.stdout:Transaction Summary 2026-02-23T13:00:33.862 INFO:teuthology.orchestra.run.trial185.stdout:================================================================================ 2026-02-23T13:00:33.862 INFO:teuthology.orchestra.run.trial185.stdout: 2026-02-23T13:00:33.863 INFO:teuthology.orchestra.run.trial185.stdout:Total download size: 803 k 2026-02-23T13:00:33.863 INFO:teuthology.orchestra.run.trial185.stdout:Installed size: 0 2026-02-23T13:00:33.863 INFO:teuthology.orchestra.run.trial185.stdout:Downloading Packages: 2026-02-23T13:00:39.359 INFO:teuthology.orchestra.run.trial185.stdout:kernel-5.14.0-681.el9.x86_64.rpm 155 kB/s | 803 kB 00:05 2026-02-23T13:00:39.359 INFO:teuthology.orchestra.run.trial185.stdout:-------------------------------------------------------------------------------- 2026-02-23T13:00:39.360 INFO:teuthology.orchestra.run.trial185.stdout:Total 146 kB/s | 803 kB 00:05 2026-02-23T13:00:39.371 INFO:teuthology.orchestra.run.trial185.stdout:Running transaction check 2026-02-23T13:00:39.405 INFO:teuthology.orchestra.run.trial185.stdout:Transaction check succeeded. 2026-02-23T13:00:39.405 INFO:teuthology.orchestra.run.trial185.stdout:Running transaction test 2026-02-23T13:00:39.409 INFO:teuthology.orchestra.run.trial185.stdout:Transaction test succeeded. 2026-02-23T13:00:39.410 INFO:teuthology.orchestra.run.trial185.stdout:Running transaction 2026-02-23T13:00:39.433 INFO:teuthology.orchestra.run.trial185.stdout: Preparing : 1/1 2026-02-23T13:00:39.437 INFO:teuthology.orchestra.run.trial185.stdout: Reinstalling : kernel-5.14.0-681.el9.x86_64 1/2 2026-02-23T13:00:39.437 INFO:teuthology.orchestra.run.trial185.stdout: Cleanup : kernel-5.14.0-681.el9.x86_64 2/2 2026-02-23T13:00:39.545 INFO:teuthology.orchestra.run.trial185.stdout: Running scriptlet: kernel-5.14.0-681.el9.x86_64 2/2 2026-02-23T13:00:39.545 INFO:teuthology.orchestra.run.trial185.stdout: Verifying : kernel-5.14.0-681.el9.x86_64 1/2 2026-02-23T13:00:39.596 INFO:teuthology.orchestra.run.trial185.stdout: Verifying : kernel-5.14.0-681.el9.x86_64 2/2 2026-02-23T13:00:39.596 INFO:teuthology.orchestra.run.trial185.stdout: 2026-02-23T13:00:39.596 INFO:teuthology.orchestra.run.trial185.stdout:Reinstalled: 2026-02-23T13:00:39.596 INFO:teuthology.orchestra.run.trial185.stdout: kernel-5.14.0-681.el9.x86_64 2026-02-23T13:00:39.596 INFO:teuthology.orchestra.run.trial185.stdout: 2026-02-23T13:00:39.596 INFO:teuthology.orchestra.run.trial185.stdout:Complete! 2026-02-23T13:00:39.622 DEBUG:teuthology.orchestra.run.trial185:> rpm -q kernel | sort -rV | head -n 1 2026-02-23T13:00:39.646 INFO:teuthology.orchestra.run.trial185.stdout:kernel-5.14.0-681.el9.x86_64 2026-02-23T13:00:39.647 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-681.el9.x86_64 2026-02-23T13:00:39.647 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2026-02-23T13:00:39.647 INFO:teuthology.task.kernel:Enabling kdb on osd.2... 2026-02-23T13:00:39.648 DEBUG:teuthology.orchestra.run.trial185:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2026-02-23T13:00:39.713 INFO:teuthology.orchestra.run.trial185.stdout:ttyS1 2026-02-23T13:00:39.734 DEBUG:teuthology.parallel:result is None 2026-02-23T13:00:39.735 INFO:teuthology.run_tasks:Running task internal.base... 2026-02-23T13:00:39.742 INFO:teuthology.task.internal:Creating test directory... 2026-02-23T13:00:39.742 DEBUG:teuthology.orchestra.run.trial068:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2026-02-23T13:00:39.745 DEBUG:teuthology.orchestra.run.trial185:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2026-02-23T13:00:39.790 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2026-02-23T13:00:39.801 INFO:teuthology.run_tasks:Running task internal.archive... 2026-02-23T13:00:39.815 INFO:teuthology.task.internal:Creating archive directory... 2026-02-23T13:00:39.815 DEBUG:teuthology.orchestra.run.trial068:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2026-02-23T13:00:39.818 DEBUG:teuthology.orchestra.run.trial185:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2026-02-23T13:00:39.856 INFO:teuthology.run_tasks:Running task internal.coredump... 2026-02-23T13:00:39.867 INFO:teuthology.task.internal:Enabling coredump saving... 2026-02-23T13:00:39.868 DEBUG:teuthology.orchestra.run.trial068:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2026-02-23T13:00:39.875 DEBUG:teuthology.orchestra.run.trial185:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2026-02-23T13:00:39.898 INFO:teuthology.orchestra.run.trial068.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-23T13:00:39.907 INFO:teuthology.orchestra.run.trial068.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-23T13:00:39.912 INFO:teuthology.orchestra.run.trial185.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-23T13:00:39.920 INFO:teuthology.orchestra.run.trial185.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-23T13:00:39.922 INFO:teuthology.run_tasks:Running task internal.sudo... 2026-02-23T13:00:39.933 INFO:teuthology.task.internal:Configuring sudo... 2026-02-23T13:00:39.934 DEBUG:teuthology.orchestra.run.trial068:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2026-02-23T13:00:39.950 DEBUG:teuthology.orchestra.run.trial185:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2026-02-23T13:00:39.985 INFO:teuthology.run_tasks:Running task internal.syslog... 2026-02-23T13:00:40.001 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2026-02-23T13:00:40.001 DEBUG:teuthology.orchestra.run.trial068:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2026-02-23T13:00:40.015 DEBUG:teuthology.orchestra.run.trial185:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2026-02-23T13:00:40.043 DEBUG:teuthology.orchestra.run.trial068:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2026-02-23T13:00:40.187 DEBUG:teuthology.orchestra.run.trial068:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2026-02-23T13:00:40.211 DEBUG:teuthology.orchestra.run.trial068:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2026-02-23T13:00:40.356 DEBUG:teuthology.orchestra.run.trial068:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2026-02-23T13:00:40.380 DEBUG:teuthology.orchestra.run.trial068:> set -ex 2026-02-23T13:00:40.380 DEBUG:teuthology.orchestra.run.trial068:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2026-02-23T13:00:40.443 DEBUG:teuthology.orchestra.run.trial185:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2026-02-23T13:00:40.602 DEBUG:teuthology.orchestra.run.trial185:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2026-02-23T13:00:40.627 DEBUG:teuthology.orchestra.run.trial185:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2026-02-23T13:00:40.787 DEBUG:teuthology.orchestra.run.trial185:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2026-02-23T13:00:40.812 DEBUG:teuthology.orchestra.run.trial185:> set -ex 2026-02-23T13:00:40.812 DEBUG:teuthology.orchestra.run.trial185:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2026-02-23T13:00:40.875 DEBUG:teuthology.orchestra.run.trial068:> sudo service rsyslog restart 2026-02-23T13:00:40.877 DEBUG:teuthology.orchestra.run.trial185:> sudo service rsyslog restart 2026-02-23T13:00:40.904 INFO:teuthology.orchestra.run.trial068.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2026-02-23T13:00:40.943 INFO:teuthology.orchestra.run.trial185.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2026-02-23T13:00:41.387 INFO:teuthology.run_tasks:Running task internal.timer... 2026-02-23T13:00:41.399 INFO:teuthology.task.internal:Starting timer... 2026-02-23T13:00:41.399 INFO:teuthology.run_tasks:Running task pcp... 2026-02-23T13:00:41.420 INFO:teuthology.run_tasks:Running task selinux... 2026-02-23T13:00:41.438 DEBUG:teuthology.orchestra.run.trial068:> sudo service auditd rotate 2026-02-23T13:00:41.469 INFO:teuthology.orchestra.run.trial068.stdout:Rotating logs: 2026-02-23T13:00:41.471 DEBUG:teuthology.orchestra.run.trial185:> sudo service auditd rotate 2026-02-23T13:00:41.501 INFO:teuthology.orchestra.run.trial185.stdout:Rotating logs: 2026-02-23T13:00:41.502 DEBUG:teuthology.task.selinux:Getting current SELinux state 2026-02-23T13:00:41.502 DEBUG:teuthology.orchestra.run.trial068:> /usr/sbin/getenforce 2026-02-23T13:00:41.529 INFO:teuthology.orchestra.run.trial068.stdout:Permissive 2026-02-23T13:00:41.529 DEBUG:teuthology.orchestra.run.trial185:> /usr/sbin/getenforce 2026-02-23T13:00:41.559 INFO:teuthology.orchestra.run.trial185.stdout:Permissive 2026-02-23T13:00:41.559 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@trial068.front.sepia.ceph.com': 'permissive', 'ubuntu@trial185.front.sepia.ceph.com': 'permissive'} 2026-02-23T13:00:41.559 DEBUG:teuthology.orchestra.run.trial068:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2026-02-23T13:00:41.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-23T13:00:41.595 DEBUG:teuthology.orchestra.run.trial185:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2026-02-23T13:00:41.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-23T13:00:41.623 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2026-02-23T13:00:41.623 DEBUG:teuthology.orchestra.run.trial068:> sudo /usr/sbin/setenforce permissive 2026-02-23T13:00:41.658 DEBUG:teuthology.orchestra.run.trial185:> sudo /usr/sbin/setenforce permissive 2026-02-23T13:00:41.687 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2026-02-23T13:00:41.702 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2026-02-23T13:00:41.712 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2026-02-23T13:00:41.714 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit trial068.front.sepia.ceph.com,trial185.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2026-02-23T13:03:27.456 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@trial068.front.sepia.ceph.com'), Remote(name='ubuntu@trial185.front.sepia.ceph.com')] 2026-02-23T13:03:27.463 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@trial068.front.sepia.ceph.com' 2026-02-23T13:03:27.465 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial068.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2026-02-23T13:03:27.576 DEBUG:teuthology.orchestra.run.trial068:> true 2026-02-23T13:03:27.652 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@trial068.front.sepia.ceph.com' 2026-02-23T13:03:27.652 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@trial185.front.sepia.ceph.com' 2026-02-23T13:03:27.653 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial185.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2026-02-23T13:03:27.758 DEBUG:teuthology.orchestra.run.trial185:> true 2026-02-23T13:03:27.833 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@trial185.front.sepia.ceph.com' 2026-02-23T13:03:27.833 INFO:teuthology.run_tasks:Running task clock... 2026-02-23T13:03:27.847 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2026-02-23T13:03:27.847 INFO:teuthology.orchestra.run:Running command with timeout 360 2026-02-23T13:03:27.848 DEBUG:teuthology.orchestra.run.trial068:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2026-02-23T13:03:27.850 INFO:teuthology.orchestra.run:Running command with timeout 360 2026-02-23T13:03:27.850 DEBUG:teuthology.orchestra.run.trial185:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2026-02-23T13:03:27.876 INFO:teuthology.orchestra.run.trial068.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2026-02-23T13:03:27.888 INFO:teuthology.orchestra.run.trial068.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2026-02-23T13:03:27.899 INFO:teuthology.orchestra.run.trial185.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2026-02-23T13:03:27.907 INFO:teuthology.orchestra.run.trial068.stderr:sudo: ntpd: command not found 2026-02-23T13:03:27.911 INFO:teuthology.orchestra.run.trial185.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2026-02-23T13:03:27.916 INFO:teuthology.orchestra.run.trial068.stdout:506 Cannot talk to daemon 2026-02-23T13:03:27.928 INFO:teuthology.orchestra.run.trial068.stderr:Failed to start ntp.service: Unit ntp.service not found. 2026-02-23T13:03:27.932 INFO:teuthology.orchestra.run.trial185.stderr:sudo: ntpd: command not found 2026-02-23T13:03:27.939 INFO:teuthology.orchestra.run.trial068.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2026-02-23T13:03:27.942 INFO:teuthology.orchestra.run.trial185.stdout:506 Cannot talk to daemon 2026-02-23T13:03:27.953 INFO:teuthology.orchestra.run.trial185.stderr:Failed to start ntp.service: Unit ntp.service not found. 2026-02-23T13:03:27.966 INFO:teuthology.orchestra.run.trial185.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2026-02-23T13:03:27.983 INFO:teuthology.orchestra.run.trial068.stderr:bash: line 1: ntpq: command not found 2026-02-23T13:03:28.002 INFO:teuthology.orchestra.run.trial068.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2026-02-23T13:03:28.002 INFO:teuthology.orchestra.run.trial068.stdout:=============================================================================== 2026-02-23T13:03:28.002 INFO:teuthology.orchestra.run.trial068.stdout:^? ntp.maxhost.io 0 6 0 - +0ns[ +0ns] +/- 0ns 2026-02-23T13:03:28.002 INFO:teuthology.orchestra.run.trial068.stdout:^? 172-233-177-198.ip.linod> 0 6 0 - +0ns[ +0ns] +/- 0ns 2026-02-23T13:03:28.002 INFO:teuthology.orchestra.run.trial068.stdout:^? 172-104-209-204.ip.linod> 0 6 0 - +0ns[ +0ns] +/- 0ns 2026-02-23T13:03:28.002 INFO:teuthology.orchestra.run.trial068.stdout:^? pool-138-89-14-60.mad.ea> 0 6 0 - +0ns[ +0ns] +/- 0ns 2026-02-23T13:03:28.013 INFO:teuthology.orchestra.run.trial185.stderr:bash: line 1: ntpq: command not found 2026-02-23T13:03:28.027 INFO:teuthology.orchestra.run.trial185.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2026-02-23T13:03:28.027 INFO:teuthology.orchestra.run.trial185.stdout:=============================================================================== 2026-02-23T13:03:28.028 INFO:teuthology.orchestra.run.trial185.stdout:^? ntp1.glypnod.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2026-02-23T13:03:28.028 INFO:teuthology.orchestra.run.trial185.stdout:^? 149.28.200.179.vultruser> 0 6 0 - +0ns[ +0ns] +/- 0ns 2026-02-23T13:03:28.028 INFO:teuthology.orchestra.run.trial185.stdout:^? 72-14-182-49.ip.linodeus> 0 6 0 - +0ns[ +0ns] +/- 0ns 2026-02-23T13:03:28.028 INFO:teuthology.orchestra.run.trial185.stdout:^? ntp4.glypnod.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2026-02-23T13:03:28.029 INFO:teuthology.run_tasks:Running task install... 2026-02-23T13:03:28.042 DEBUG:teuthology.task.install:project ceph 2026-02-23T13:03:28.042 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'debug', 'sha1': 'e371433fcd1390a0b82cfb18effc422c1f5d8c1d'}} 2026-02-23T13:03:28.042 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['ceph-osd-crimson', 'ceph-osd-crimson-dbg'], 'rpm': ['ceph-osd-crimson', 'ceph-osd-crimson-debuginfo']}, 'flavor': 'debug', 'sha1': 'e371433fcd1390a0b82cfb18effc422c1f5d8c1d'} 2026-02-23T13:03:28.043 INFO:teuthology.task.install:Using flavor: debug 2026-02-23T13:03:28.051 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-osd', 'ceph-osd-classic', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-osd', 'ceph-osd-classic', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2026-02-23T13:03:28.051 INFO:teuthology.task.install:extra packages: {'deb': ['ceph-osd-crimson', 'ceph-osd-crimson-dbg'], 'rpm': ['ceph-osd-crimson', 'ceph-osd-crimson-debuginfo']} 2026-02-23T13:03:28.052 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['ceph-osd-crimson', 'ceph-osd-crimson-dbg'], 'rpm': ['ceph-osd-crimson', 'ceph-osd-crimson-debuginfo']}, 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'debug', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'e371433fcd1390a0b82cfb18effc422c1f5d8c1d', 'tag': None, 'wait_for_package': False} 2026-02-23T13:03:28.053 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=debug&distros=centos%2F9%2Fx86_64&sha1=e371433fcd1390a0b82cfb18effc422c1f5d8c1d 2026-02-23T13:03:28.055 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['ceph-osd-crimson', 'ceph-osd-crimson-dbg'], 'rpm': ['ceph-osd-crimson', 'ceph-osd-crimson-debuginfo']}, 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'debug', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'e371433fcd1390a0b82cfb18effc422c1f5d8c1d', 'tag': None, 'wait_for_package': False} 2026-02-23T13:03:28.055 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=debug&distros=centos%2F9%2Fx86_64&sha1=e371433fcd1390a0b82cfb18effc422c1f5d8c1d 2026-02-23T13:03:28.260 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-shraddhaag-enable-cephadm-tests/e371433fcd1390a0b82cfb18effc422c1f5d8c1d/centos/9/flavors/debug/ 2026-02-23T13:03:28.261 INFO:teuthology.task.install.rpm:Package version is 20.3.0-5439.ge371433f 2026-02-23T13:03:28.269 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-shraddhaag-enable-cephadm-tests/e371433fcd1390a0b82cfb18effc422c1f5d8c1d/centos/9/flavors/debug/ 2026-02-23T13:03:28.269 INFO:teuthology.task.install.rpm:Package version is 20.3.0-5439.ge371433f 2026-02-23T13:03:28.537 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-shraddhaag-enable-cephadm-tests/e371433fcd1390a0b82cfb18effc422c1f5d8c1d/centos/9/flavors/debug/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-shraddhaag-enable-cephadm-tests/e371433fcd1390a0b82cfb18effc422c1f5d8c1d/centos/9/flavors/debug/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-shraddhaag-enable-cephadm-tests/e371433fcd1390a0b82cfb18effc422c1f5d8c1d/centos/9/flavors/debug/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2026-02-23T13:03:28.537 DEBUG:teuthology.orchestra.run.trial068:> set -ex 2026-02-23T13:03:28.538 DEBUG:teuthology.orchestra.run.trial068:> sudo dd of=/etc/yum.repos.d/ceph.repo 2026-02-23T13:03:28.562 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-osd, ceph-osd-classic, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, ceph-osd-crimson, ceph-osd-crimson-debuginfo on remote rpm x86_64 2026-02-23T13:03:28.563 DEBUG:teuthology.orchestra.run.trial068:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/e371433fcd1390a0b82cfb18effc422c1f5d8c1d/;g' /etc/yum.repos.d/ceph.repo ; fi 2026-02-23T13:03:28.588 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-shraddhaag-enable-cephadm-tests/e371433fcd1390a0b82cfb18effc422c1f5d8c1d/centos/9/flavors/debug/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-shraddhaag-enable-cephadm-tests/e371433fcd1390a0b82cfb18effc422c1f5d8c1d/centos/9/flavors/debug/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-shraddhaag-enable-cephadm-tests/e371433fcd1390a0b82cfb18effc422c1f5d8c1d/centos/9/flavors/debug/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2026-02-23T13:03:28.588 DEBUG:teuthology.orchestra.run.trial185:> set -ex 2026-02-23T13:03:28.588 DEBUG:teuthology.orchestra.run.trial185:> sudo dd of=/etc/yum.repos.d/ceph.repo 2026-02-23T13:03:28.614 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-osd, ceph-osd-classic, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, ceph-osd-crimson, ceph-osd-crimson-debuginfo on remote rpm x86_64 2026-02-23T13:03:28.614 DEBUG:teuthology.orchestra.run.trial185:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/e371433fcd1390a0b82cfb18effc422c1f5d8c1d/;g' /etc/yum.repos.d/ceph.repo ; fi 2026-02-23T13:03:28.626 DEBUG:teuthology.orchestra.run.trial068:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2026-02-23T13:03:28.679 DEBUG:teuthology.orchestra.run.trial185:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2026-02-23T13:03:28.698 DEBUG:teuthology.orchestra.run.trial068:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2026-02-23T13:03:28.752 DEBUG:teuthology.orchestra.run.trial185:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2026-02-23T13:03:28.762 INFO:teuthology.orchestra.run.trial068.stdout:check_obsoletes = 1 2026-02-23T13:03:28.763 DEBUG:teuthology.orchestra.run.trial068:> sudo yum clean all 2026-02-23T13:03:28.817 INFO:teuthology.orchestra.run.trial185.stdout:check_obsoletes = 1 2026-02-23T13:03:28.818 DEBUG:teuthology.orchestra.run.trial185:> sudo yum clean all 2026-02-23T13:03:28.921 INFO:teuthology.orchestra.run.trial068.stdout:41 files removed 2026-02-23T13:03:28.935 DEBUG:teuthology.orchestra.run.trial068:> sudo yum -y install ceph-radosgw 2026-02-23T13:03:28.977 INFO:teuthology.orchestra.run.trial185.stdout:41 files removed 2026-02-23T13:03:28.992 DEBUG:teuthology.orchestra.run.trial185:> sudo yum -y install ceph-radosgw 2026-02-23T13:03:29.412 INFO:teuthology.orchestra.run.trial068.stdout:ceph packages for x86_64 276 kB/s | 92 kB 00:00 2026-02-23T13:03:29.470 INFO:teuthology.orchestra.run.trial185.stdout:ceph packages for x86_64 274 kB/s | 92 kB 00:00 2026-02-23T13:03:29.742 INFO:teuthology.orchestra.run.trial068.stdout:ceph noarch packages 78 kB/s | 24 kB 00:00 2026-02-23T13:03:29.805 INFO:teuthology.orchestra.run.trial185.stdout:ceph noarch packages 77 kB/s | 24 kB 00:00 2026-02-23T13:03:30.040 INFO:teuthology.orchestra.run.trial068.stdout:ceph source packages 7.5 kB/s | 2.1 kB 00:00 2026-02-23T13:03:30.102 INFO:teuthology.orchestra.run.trial185.stdout:ceph source packages 7.6 kB/s | 2.1 kB 00:00 2026-02-23T13:03:30.824 INFO:teuthology.orchestra.run.trial185.stdout:CentOS Stream 9 - BaseOS 13 MB/s | 8.9 MB 00:00 2026-02-23T13:03:34.168 INFO:teuthology.orchestra.run.trial185.stdout:CentOS Stream 9 - AppStream 9.5 MB/s | 27 MB 00:02 2026-02-23T13:03:38.732 INFO:teuthology.orchestra.run.trial068.stdout:CentOS Stream 9 - BaseOS 1.0 MB/s | 8.9 MB 00:08 2026-02-23T13:03:39.237 INFO:teuthology.orchestra.run.trial185.stdout:CentOS Stream 9 - CRB 2.9 MB/s | 7.8 MB 00:02 2026-02-23T13:03:41.005 INFO:teuthology.orchestra.run.trial185.stdout:CentOS Stream 9 - Extras packages 19 kB/s | 20 kB 00:01 2026-02-23T13:03:41.278 INFO:teuthology.orchestra.run.trial185.stdout:Extra Packages for Enterprise Linux 97 MB/s | 20 MB 00:00 2026-02-23T13:03:41.825 INFO:teuthology.orchestra.run.trial068.stdout:CentOS Stream 9 - AppStream 10 MB/s | 27 MB 00:02 2026-02-23T13:03:44.553 INFO:teuthology.orchestra.run.trial185.stdout:lab-extras 1.1 MB/s | 50 kB 00:00 2026-02-23T13:03:45.298 INFO:teuthology.orchestra.run.trial068.stdout:CentOS Stream 9 - CRB 7.7 MB/s | 7.8 MB 00:01 2026-02-23T13:03:45.751 INFO:teuthology.orchestra.run.trial185.stdout:Dependencies resolved. 2026-02-23T13:03:45.752 INFO:teuthology.orchestra.run.trial185.stdout:================================================================================ 2026-02-23T13:03:45.752 INFO:teuthology.orchestra.run.trial185.stdout: Package Arch Version Repository Size 2026-02-23T13:03:45.752 INFO:teuthology.orchestra.run.trial185.stdout:================================================================================ 2026-02-23T13:03:45.752 INFO:teuthology.orchestra.run.trial185.stdout:Installing: 2026-02-23T13:03:45.752 INFO:teuthology.orchestra.run.trial185.stdout: ceph-radosgw x86_64 2:20.3.0-5439.ge371433f.el9 ceph 11 M 2026-02-23T13:03:45.752 INFO:teuthology.orchestra.run.trial185.stdout:Upgrading: 2026-02-23T13:03:45.752 INFO:teuthology.orchestra.run.trial185.stdout: librados2 x86_64 2:20.3.0-5439.ge371433f.el9 ceph 3.3 M 2026-02-23T13:03:45.752 INFO:teuthology.orchestra.run.trial185.stdout: librbd1 x86_64 2:20.3.0-5439.ge371433f.el9 ceph 2.7 M 2026-02-23T13:03:45.752 INFO:teuthology.orchestra.run.trial185.stdout:Installing dependencies: 2026-02-23T13:03:45.752 INFO:teuthology.orchestra.run.trial185.stdout: boost-program-options x86_64 1.75.0-13.el9 appstream 104 k 2026-02-23T13:03:45.752 INFO:teuthology.orchestra.run.trial185.stdout: ceph-base x86_64 2:20.3.0-5439.ge371433f.el9 ceph 5.1 M 2026-02-23T13:03:45.752 INFO:teuthology.orchestra.run.trial185.stdout: ceph-common x86_64 2:20.3.0-5439.ge371433f.el9 ceph 21 M 2026-02-23T13:03:45.752 INFO:teuthology.orchestra.run.trial185.stdout: ceph-selinux x86_64 2:20.3.0-5439.ge371433f.el9 ceph 25 k 2026-02-23T13:03:45.752 INFO:teuthology.orchestra.run.trial185.stdout: gperftools-libs x86_64 2.9.1-3.el9 epel 308 k 2026-02-23T13:03:45.752 INFO:teuthology.orchestra.run.trial185.stdout: libarrow x86_64 9.0.0-15.el9 epel 4.4 M 2026-02-23T13:03:45.753 INFO:teuthology.orchestra.run.trial185.stdout: libarrow-doc noarch 9.0.0-15.el9 epel 25 k 2026-02-23T13:03:45.753 INFO:teuthology.orchestra.run.trial185.stdout: libcephfs2 x86_64 2:20.3.0-5439.ge371433f.el9 ceph 860 k 2026-02-23T13:03:45.753 INFO:teuthology.orchestra.run.trial185.stdout: libnbd x86_64 1.20.3-4.el9 appstream 164 k 2026-02-23T13:03:45.753 INFO:teuthology.orchestra.run.trial185.stdout: liboath x86_64 2.6.12-1.el9 epel 49 k 2026-02-23T13:03:45.753 INFO:teuthology.orchestra.run.trial185.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2026-02-23T13:03:45.753 INFO:teuthology.orchestra.run.trial185.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2026-02-23T13:03:45.753 INFO:teuthology.orchestra.run.trial185.stdout: libradosstriper1 x86_64 2:20.3.0-5439.ge371433f.el9 ceph 263 k 2026-02-23T13:03:45.753 INFO:teuthology.orchestra.run.trial185.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2026-02-23T13:03:45.753 INFO:teuthology.orchestra.run.trial185.stdout: librgw2 x86_64 2:20.3.0-5439.ge371433f.el9 ceph 5.6 M 2026-02-23T13:03:45.753 INFO:teuthology.orchestra.run.trial185.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2026-02-23T13:03:45.753 INFO:teuthology.orchestra.run.trial185.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2026-02-23T13:03:45.753 INFO:teuthology.orchestra.run.trial185.stdout: mailcap noarch 2.1.49-5.el9 baseos 33 k 2026-02-23T13:03:45.753 INFO:teuthology.orchestra.run.trial185.stdout: parquet-libs x86_64 9.0.0-15.el9 epel 838 k 2026-02-23T13:03:45.753 INFO:teuthology.orchestra.run.trial185.stdout: pciutils x86_64 3.7.0-7.el9 baseos 93 k 2026-02-23T13:03:45.753 INFO:teuthology.orchestra.run.trial185.stdout: python3-ceph-argparse x86_64 2:20.3.0-5439.ge371433f.el9 ceph 45 k 2026-02-23T13:03:45.753 INFO:teuthology.orchestra.run.trial185.stdout: python3-ceph-common x86_64 2:20.3.0-5439.ge371433f.el9 ceph 186 k 2026-02-23T13:03:45.753 INFO:teuthology.orchestra.run.trial185.stdout: python3-cephfs x86_64 2:20.3.0-5439.ge371433f.el9 ceph 204 k 2026-02-23T13:03:45.753 INFO:teuthology.orchestra.run.trial185.stdout: python3-rados x86_64 2:20.3.0-5439.ge371433f.el9 ceph 324 k 2026-02-23T13:03:45.753 INFO:teuthology.orchestra.run.trial185.stdout: python3-rbd x86_64 2:20.3.0-5439.ge371433f.el9 ceph 305 k 2026-02-23T13:03:45.753 INFO:teuthology.orchestra.run.trial185.stdout: python3-rgw x86_64 2:20.3.0-5439.ge371433f.el9 ceph 100 k 2026-02-23T13:03:45.753 INFO:teuthology.orchestra.run.trial185.stdout: qatlib x86_64 25.08.0-2.el9 appstream 240 k 2026-02-23T13:03:45.754 INFO:teuthology.orchestra.run.trial185.stdout: qatzip-libs x86_64 1.3.1-1.el9 appstream 66 k 2026-02-23T13:03:45.754 INFO:teuthology.orchestra.run.trial185.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2026-02-23T13:03:45.754 INFO:teuthology.orchestra.run.trial185.stdout: thrift x86_64 0.15.0-4.el9 epel 1.6 M 2026-02-23T13:03:45.754 INFO:teuthology.orchestra.run.trial185.stdout:Installing weak dependencies: 2026-02-23T13:03:45.754 INFO:teuthology.orchestra.run.trial185.stdout: qatlib-service x86_64 25.08.0-2.el9 appstream 37 k 2026-02-23T13:03:45.754 INFO:teuthology.orchestra.run.trial185.stdout: 2026-02-23T13:03:45.754 INFO:teuthology.orchestra.run.trial185.stdout:Transaction Summary 2026-02-23T13:03:45.754 INFO:teuthology.orchestra.run.trial185.stdout:================================================================================ 2026-02-23T13:03:45.754 INFO:teuthology.orchestra.run.trial185.stdout:Install 32 Packages 2026-02-23T13:03:45.754 INFO:teuthology.orchestra.run.trial185.stdout:Upgrade 2 Packages 2026-02-23T13:03:45.754 INFO:teuthology.orchestra.run.trial185.stdout: 2026-02-23T13:03:45.754 INFO:teuthology.orchestra.run.trial185.stdout:Total download size: 60 M 2026-02-23T13:03:45.754 INFO:teuthology.orchestra.run.trial185.stdout:Downloading Packages: 2026-02-23T13:03:46.200 INFO:teuthology.orchestra.run.trial068.stdout:CentOS Stream 9 - Extras packages 121 kB/s | 20 kB 00:00 2026-02-23T13:03:47.370 INFO:teuthology.orchestra.run.trial068.stdout:Extra Packages for Enterprise Linux 18 MB/s | 20 MB 00:01 2026-02-23T13:03:47.383 INFO:teuthology.orchestra.run.trial185.stdout:(1/34): ceph-base-20.3.0-5439.ge371433f.el9.x86 4.2 MB/s | 5.1 MB 00:01 2026-02-23T13:03:47.395 INFO:teuthology.orchestra.run.trial185.stdout:(2/34): ceph-selinux-20.3.0-5439.ge371433f.el9. 2.1 MB/s | 25 kB 00:00 2026-02-23T13:03:47.441 INFO:teuthology.orchestra.run.trial185.stdout:(3/34): libcephfs2-20.3.0-5439.ge371433f.el9.x8 18 MB/s | 860 kB 00:00 2026-02-23T13:03:47.457 INFO:teuthology.orchestra.run.trial185.stdout:(4/34): libradosstriper1-20.3.0-5439.ge371433f. 15 MB/s | 263 kB 00:00 2026-02-23T13:03:47.686 INFO:teuthology.orchestra.run.trial185.stdout:(5/34): ceph-radosgw-20.3.0-5439.ge371433f.el9. 7.4 MB/s | 11 MB 00:01 2026-02-23T13:03:47.706 INFO:teuthology.orchestra.run.trial185.stdout:(6/34): python3-ceph-argparse-20.3.0-5439.ge371 2.2 MB/s | 45 kB 00:00 2026-02-23T13:03:47.722 INFO:teuthology.orchestra.run.trial185.stdout:(7/34): librgw2-20.3.0-5439.ge371433f.el9.x86_6 21 MB/s | 5.6 MB 00:00 2026-02-23T13:03:47.727 INFO:teuthology.orchestra.run.trial185.stdout:(8/34): python3-ceph-common-20.3.0-5439.ge37143 8.9 MB/s | 186 kB 00:00 2026-02-23T13:03:47.739 INFO:teuthology.orchestra.run.trial185.stdout:(9/34): python3-cephfs-20.3.0-5439.ge371433f.el 12 MB/s | 204 kB 00:00 2026-02-23T13:03:47.746 INFO:teuthology.orchestra.run.trial185.stdout:(10/34): python3-rados-20.3.0-5439.ge371433f.el 16 MB/s | 324 kB 00:00 2026-02-23T13:03:47.761 INFO:teuthology.orchestra.run.trial185.stdout:(11/34): python3-rbd-20.3.0-5439.ge371433f.el9. 14 MB/s | 305 kB 00:00 2026-02-23T13:03:47.764 INFO:teuthology.orchestra.run.trial185.stdout:(12/34): python3-rgw-20.3.0-5439.ge371433f.el9. 5.7 MB/s | 100 kB 00:00 2026-02-23T13:03:47.948 INFO:teuthology.orchestra.run.trial185.stdout:(13/34): ceph-common-20.3.0-5439.ge371433f.el9. 12 MB/s | 21 MB 00:01 2026-02-23T13:03:48.013 INFO:teuthology.orchestra.run.trial185.stdout:(14/34): boost-program-options-1.75.0-13.el9.x8 1.6 MB/s | 104 kB 00:00 2026-02-23T13:03:48.039 INFO:teuthology.orchestra.run.trial185.stdout:(15/34): libnbd-1.20.3-4.el9.x86_64.rpm 6.3 MB/s | 164 kB 00:00 2026-02-23T13:03:48.062 INFO:teuthology.orchestra.run.trial185.stdout:(16/34): libpmemobj-1.12.1-1.el9.x86_64.rpm 6.8 MB/s | 160 kB 00:00 2026-02-23T13:03:48.085 INFO:teuthology.orchestra.run.trial185.stdout:(17/34): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.9 MB/s | 45 kB 00:00 2026-02-23T13:03:48.113 INFO:teuthology.orchestra.run.trial185.stdout:(18/34): librdkafka-1.6.1-102.el9.x86_64.rpm 23 MB/s | 662 kB 00:00 2026-02-23T13:03:48.138 INFO:teuthology.orchestra.run.trial185.stdout:(19/34): lttng-ust-2.12.0-6.el9.x86_64.rpm 12 MB/s | 292 kB 00:00 2026-02-23T13:03:48.163 INFO:teuthology.orchestra.run.trial185.stdout:(20/34): qatlib-25.08.0-2.el9.x86_64.rpm 9.2 MB/s | 240 kB 00:00 2026-02-23T13:03:48.186 INFO:teuthology.orchestra.run.trial185.stdout:(21/34): qatlib-service-25.08.0-2.el9.x86_64.rp 1.6 MB/s | 37 kB 00:00 2026-02-23T13:03:48.213 INFO:teuthology.orchestra.run.trial185.stdout:(22/34): qatzip-libs-1.3.1-1.el9.x86_64.rpm 2.4 MB/s | 66 kB 00:00 2026-02-23T13:03:48.346 INFO:teuthology.orchestra.run.trial185.stdout:(23/34): mailcap-2.1.49-5.el9.noarch.rpm 57 kB/s | 33 kB 00:00 2026-02-23T13:03:48.404 INFO:teuthology.orchestra.run.trial185.stdout:(24/34): gperftools-libs-2.9.1-3.el9.x86_64.rpm 1.6 MB/s | 308 kB 00:00 2026-02-23T13:03:48.490 INFO:teuthology.orchestra.run.trial185.stdout:(25/34): libarrow-doc-9.0.0-15.el9.noarch.rpm 287 kB/s | 25 kB 00:00 2026-02-23T13:03:48.503 INFO:teuthology.orchestra.run.trial185.stdout:(26/34): libarrow-9.0.0-15.el9.x86_64.rpm 28 MB/s | 4.4 MB 00:00 2026-02-23T13:03:48.515 INFO:teuthology.orchestra.run.trial185.stdout:(27/34): liboath-2.6.12-1.el9.x86_64.rpm 1.9 MB/s | 49 kB 00:00 2026-02-23T13:03:48.527 INFO:teuthology.orchestra.run.trial185.stdout:(28/34): libunwind-1.6.2-1.el9.x86_64.rpm 2.7 MB/s | 67 kB 00:00 2026-02-23T13:03:48.543 INFO:teuthology.orchestra.run.trial185.stdout:(29/34): parquet-libs-9.0.0-15.el9.x86_64.rpm 29 MB/s | 838 kB 00:00 2026-02-23T13:03:48.553 INFO:teuthology.orchestra.run.trial185.stdout:(30/34): re2-20211101-20.el9.x86_64.rpm 7.4 MB/s | 191 kB 00:00 2026-02-23T13:03:48.557 INFO:teuthology.orchestra.run.trial185.stdout:(31/34): pciutils-3.7.0-7.el9.x86_64.rpm 117 kB/s | 93 kB 00:00 2026-02-23T13:03:48.575 INFO:teuthology.orchestra.run.trial185.stdout:(32/34): thrift-0.15.0-4.el9.x86_64.rpm 49 MB/s | 1.6 MB 00:00 2026-02-23T13:03:48.640 INFO:teuthology.orchestra.run.trial185.stdout:(33/34): librbd1-20.3.0-5439.ge371433f.el9.x86_ 32 MB/s | 2.7 MB 00:00 2026-02-23T13:03:48.671 INFO:teuthology.orchestra.run.trial185.stdout:(34/34): librados2-20.3.0-5439.ge371433f.el9.x8 28 MB/s | 3.3 MB 00:00 2026-02-23T13:03:48.675 INFO:teuthology.orchestra.run.trial185.stdout:-------------------------------------------------------------------------------- 2026-02-23T13:03:48.675 INFO:teuthology.orchestra.run.trial185.stdout:Total 21 MB/s | 60 MB 00:02 2026-02-23T13:03:48.766 INFO:teuthology.orchestra.run.trial185.stdout:Running transaction check 2026-02-23T13:03:48.783 INFO:teuthology.orchestra.run.trial185.stdout:Transaction check succeeded. 2026-02-23T13:03:48.784 INFO:teuthology.orchestra.run.trial185.stdout:Running transaction test 2026-02-23T13:03:49.028 INFO:teuthology.orchestra.run.trial185.stdout:Transaction test succeeded. 2026-02-23T13:03:49.028 INFO:teuthology.orchestra.run.trial185.stdout:Running transaction 2026-02-23T13:03:49.306 INFO:teuthology.orchestra.run.trial185.stdout: Preparing : 1/1 2026-02-23T13:03:49.449 INFO:teuthology.orchestra.run.trial185.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/36 2026-02-23T13:03:49.455 INFO:teuthology.orchestra.run.trial185.stdout: Upgrading : librados2-2:20.3.0-5439.ge371433f.el9.x86_64 2/36 2026-02-23T13:03:49.486 INFO:teuthology.orchestra.run.trial185.stdout: Running scriptlet: librados2-2:20.3.0-5439.ge371433f.el9.x86_64 2/36 2026-02-23T13:03:49.523 INFO:teuthology.orchestra.run.trial185.stdout: Installing : python3-rados-2:20.3.0-5439.ge371433f.el9.x86_64 3/36 2026-02-23T13:03:49.524 INFO:teuthology.orchestra.run.trial185.stdout: Installing : libcephfs2-2:20.3.0-5439.ge371433f.el9.x86_64 4/36 2026-02-23T13:03:49.536 INFO:teuthology.orchestra.run.trial185.stdout: Running scriptlet: libcephfs2-2:20.3.0-5439.ge371433f.el9.x86_64 4/36 2026-02-23T13:03:49.543 INFO:teuthology.orchestra.run.trial185.stdout: Installing : liboath-2.6.12-1.el9.x86_64 5/36 2026-02-23T13:03:49.546 INFO:teuthology.orchestra.run.trial185.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 6/36 2026-02-23T13:03:49.549 INFO:teuthology.orchestra.run.trial185.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 7/36 2026-02-23T13:03:49.553 INFO:teuthology.orchestra.run.trial185.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 8/36 2026-02-23T13:03:49.564 INFO:teuthology.orchestra.run.trial185.stdout: Installing : python3-ceph-argparse-2:20.3.0-5439.ge371433f.el9. 9/36 2026-02-23T13:03:49.577 INFO:teuthology.orchestra.run.trial185.stdout: Installing : python3-cephfs-2:20.3.0-5439.ge371433f.el9.x86_64 10/36 2026-02-23T13:03:49.578 INFO:teuthology.orchestra.run.trial185.stdout: Installing : libradosstriper1-2:20.3.0-5439.ge371433f.el9.x86_6 11/36 2026-02-23T13:03:49.602 INFO:teuthology.orchestra.run.trial185.stdout: Running scriptlet: libradosstriper1-2:20.3.0-5439.ge371433f.el9.x86_6 11/36 2026-02-23T13:03:49.606 INFO:teuthology.orchestra.run.trial185.stdout: Installing : thrift-0.15.0-4.el9.x86_64 12/36 2026-02-23T13:03:49.639 INFO:teuthology.orchestra.run.trial185.stdout: Installing : re2-1:20211101-20.el9.x86_64 13/36 2026-02-23T13:03:49.642 INFO:teuthology.orchestra.run.trial185.stdout: Installing : libarrow-9.0.0-15.el9.x86_64 14/36 2026-02-23T13:03:49.648 INFO:teuthology.orchestra.run.trial185.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 15/36 2026-02-23T13:03:49.651 INFO:teuthology.orchestra.run.trial185.stdout: Installing : gperftools-libs-2.9.1-3.el9.x86_64 16/36 2026-02-23T13:03:49.656 INFO:teuthology.orchestra.run.trial185.stdout: Installing : libarrow-doc-9.0.0-15.el9.noarch 17/36 2026-02-23T13:03:49.772 INFO:teuthology.orchestra.run.trial185.stdout: Installing : libnbd-1.20.3-4.el9.x86_64 18/36 2026-02-23T13:03:49.773 INFO:teuthology.orchestra.run.trial185.stdout: Upgrading : librbd1-2:20.3.0-5439.ge371433f.el9.x86_64 19/36 2026-02-23T13:03:49.799 INFO:teuthology.orchestra.run.trial185.stdout: Running scriptlet: librbd1-2:20.3.0-5439.ge371433f.el9.x86_64 19/36 2026-02-23T13:03:49.804 INFO:teuthology.orchestra.run.trial185.stdout: Installing : python3-rbd-2:20.3.0-5439.ge371433f.el9.x86_64 20/36 2026-02-23T13:03:49.810 INFO:teuthology.orchestra.run.trial185.stdout: Installing : boost-program-options-1.75.0-13.el9.x86_64 21/36 2026-02-23T13:03:50.064 INFO:teuthology.orchestra.run.trial185.stdout: Installing : parquet-libs-9.0.0-15.el9.x86_64 22/36 2026-02-23T13:03:50.065 INFO:teuthology.orchestra.run.trial185.stdout: Installing : librgw2-2:20.3.0-5439.ge371433f.el9.x86_64 23/36 2026-02-23T13:03:50.081 INFO:teuthology.orchestra.run.trial185.stdout: Running scriptlet: librgw2-2:20.3.0-5439.ge371433f.el9.x86_64 23/36 2026-02-23T13:03:50.086 INFO:teuthology.orchestra.run.trial185.stdout: Installing : python3-rgw-2:20.3.0-5439.ge371433f.el9.x86_64 24/36 2026-02-23T13:03:50.090 INFO:teuthology.orchestra.run.trial185.stdout: Installing : pciutils-3.7.0-7.el9.x86_64 25/36 2026-02-23T13:03:50.092 INFO:teuthology.orchestra.run.trial185.stdout: Installing : qatlib-25.08.0-2.el9.x86_64 26/36 2026-02-23T13:03:50.093 INFO:teuthology.orchestra.run.trial185.stdout: Installing : qatlib-service-25.08.0-2.el9.x86_64 27/36 2026-02-23T13:03:50.112 INFO:teuthology.orchestra.run.trial185.stdout: Running scriptlet: qatlib-service-25.08.0-2.el9.x86_64 27/36 2026-02-23T13:03:50.115 INFO:teuthology.orchestra.run.trial185.stdout: Installing : qatzip-libs-1.3.1-1.el9.x86_64 28/36 2026-02-23T13:03:50.151 INFO:teuthology.orchestra.run.trial185.stdout: Installing : mailcap-2.1.49-5.el9.noarch 29/36 2026-02-23T13:03:50.153 INFO:teuthology.orchestra.run.trial185.stdout: Installing : python3-ceph-common-2:20.3.0-5439.ge371433f.el9.x8 30/36 2026-02-23T13:03:50.586 INFO:teuthology.orchestra.run.trial068.stdout:lab-extras 1.4 MB/s | 50 kB 00:00 2026-02-23T13:03:51.113 INFO:teuthology.orchestra.run.trial185.stdout: Running scriptlet: ceph-common-2:20.3.0-5439.ge371433f.el9.x86_64 31/36 2026-02-23T13:03:51.116 INFO:teuthology.orchestra.run.trial185.stdout: Installing : ceph-common-2:20.3.0-5439.ge371433f.el9.x86_64 31/36 2026-02-23T13:03:51.384 INFO:teuthology.orchestra.run.trial185.stdout: Running scriptlet: ceph-common-2:20.3.0-5439.ge371433f.el9.x86_64 31/36 2026-02-23T13:03:51.388 INFO:teuthology.orchestra.run.trial185.stdout: Installing : ceph-base-2:20.3.0-5439.ge371433f.el9.x86_64 32/36 2026-02-23T13:03:51.431 INFO:teuthology.orchestra.run.trial185.stdout: Running scriptlet: ceph-base-2:20.3.0-5439.ge371433f.el9.x86_64 32/36 2026-02-23T13:03:51.431 INFO:teuthology.orchestra.run.trial185.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2026-02-23T13:03:51.431 INFO:teuthology.orchestra.run.trial185.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2026-02-23T13:03:51.431 INFO:teuthology.orchestra.run.trial185.stdout: 2026-02-23T13:03:51.434 INFO:teuthology.orchestra.run.trial185.stdout: Installing : ceph-selinux-2:20.3.0-5439.ge371433f.el9.x86_64 33/36 2026-02-23T13:03:51.781 INFO:teuthology.orchestra.run.trial068.stdout:Dependencies resolved. 2026-02-23T13:03:51.782 INFO:teuthology.orchestra.run.trial068.stdout:================================================================================ 2026-02-23T13:03:51.782 INFO:teuthology.orchestra.run.trial068.stdout: Package Arch Version Repository Size 2026-02-23T13:03:51.782 INFO:teuthology.orchestra.run.trial068.stdout:================================================================================ 2026-02-23T13:03:51.782 INFO:teuthology.orchestra.run.trial068.stdout:Installing: 2026-02-23T13:03:51.782 INFO:teuthology.orchestra.run.trial068.stdout: ceph-radosgw x86_64 2:20.3.0-5439.ge371433f.el9 ceph 11 M 2026-02-23T13:03:51.783 INFO:teuthology.orchestra.run.trial068.stdout:Upgrading: 2026-02-23T13:03:51.783 INFO:teuthology.orchestra.run.trial068.stdout: librados2 x86_64 2:20.3.0-5439.ge371433f.el9 ceph 3.3 M 2026-02-23T13:03:51.783 INFO:teuthology.orchestra.run.trial068.stdout: librbd1 x86_64 2:20.3.0-5439.ge371433f.el9 ceph 2.7 M 2026-02-23T13:03:51.783 INFO:teuthology.orchestra.run.trial068.stdout:Installing dependencies: 2026-02-23T13:03:51.783 INFO:teuthology.orchestra.run.trial068.stdout: boost-program-options x86_64 1.75.0-13.el9 appstream 104 k 2026-02-23T13:03:51.783 INFO:teuthology.orchestra.run.trial068.stdout: ceph-base x86_64 2:20.3.0-5439.ge371433f.el9 ceph 5.1 M 2026-02-23T13:03:51.783 INFO:teuthology.orchestra.run.trial068.stdout: ceph-common x86_64 2:20.3.0-5439.ge371433f.el9 ceph 21 M 2026-02-23T13:03:51.783 INFO:teuthology.orchestra.run.trial068.stdout: ceph-selinux x86_64 2:20.3.0-5439.ge371433f.el9 ceph 25 k 2026-02-23T13:03:51.783 INFO:teuthology.orchestra.run.trial068.stdout: gperftools-libs x86_64 2.9.1-3.el9 epel 308 k 2026-02-23T13:03:51.783 INFO:teuthology.orchestra.run.trial068.stdout: libarrow x86_64 9.0.0-15.el9 epel 4.4 M 2026-02-23T13:03:51.783 INFO:teuthology.orchestra.run.trial068.stdout: libarrow-doc noarch 9.0.0-15.el9 epel 25 k 2026-02-23T13:03:51.784 INFO:teuthology.orchestra.run.trial068.stdout: libcephfs2 x86_64 2:20.3.0-5439.ge371433f.el9 ceph 860 k 2026-02-23T13:03:51.784 INFO:teuthology.orchestra.run.trial068.stdout: libnbd x86_64 1.20.3-4.el9 appstream 164 k 2026-02-23T13:03:51.784 INFO:teuthology.orchestra.run.trial068.stdout: liboath x86_64 2.6.12-1.el9 epel 49 k 2026-02-23T13:03:51.784 INFO:teuthology.orchestra.run.trial068.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2026-02-23T13:03:51.784 INFO:teuthology.orchestra.run.trial068.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2026-02-23T13:03:51.784 INFO:teuthology.orchestra.run.trial068.stdout: libradosstriper1 x86_64 2:20.3.0-5439.ge371433f.el9 ceph 263 k 2026-02-23T13:03:51.784 INFO:teuthology.orchestra.run.trial068.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2026-02-23T13:03:51.784 INFO:teuthology.orchestra.run.trial068.stdout: librgw2 x86_64 2:20.3.0-5439.ge371433f.el9 ceph 5.6 M 2026-02-23T13:03:51.784 INFO:teuthology.orchestra.run.trial068.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2026-02-23T13:03:51.784 INFO:teuthology.orchestra.run.trial068.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2026-02-23T13:03:51.784 INFO:teuthology.orchestra.run.trial068.stdout: mailcap noarch 2.1.49-5.el9 baseos 33 k 2026-02-23T13:03:51.784 INFO:teuthology.orchestra.run.trial068.stdout: parquet-libs x86_64 9.0.0-15.el9 epel 838 k 2026-02-23T13:03:51.784 INFO:teuthology.orchestra.run.trial068.stdout: pciutils x86_64 3.7.0-7.el9 baseos 93 k 2026-02-23T13:03:51.785 INFO:teuthology.orchestra.run.trial068.stdout: python3-ceph-argparse x86_64 2:20.3.0-5439.ge371433f.el9 ceph 45 k 2026-02-23T13:03:51.785 INFO:teuthology.orchestra.run.trial068.stdout: python3-ceph-common x86_64 2:20.3.0-5439.ge371433f.el9 ceph 186 k 2026-02-23T13:03:51.785 INFO:teuthology.orchestra.run.trial068.stdout: python3-cephfs x86_64 2:20.3.0-5439.ge371433f.el9 ceph 204 k 2026-02-23T13:03:51.785 INFO:teuthology.orchestra.run.trial068.stdout: python3-rados x86_64 2:20.3.0-5439.ge371433f.el9 ceph 324 k 2026-02-23T13:03:51.785 INFO:teuthology.orchestra.run.trial068.stdout: python3-rbd x86_64 2:20.3.0-5439.ge371433f.el9 ceph 305 k 2026-02-23T13:03:51.785 INFO:teuthology.orchestra.run.trial068.stdout: python3-rgw x86_64 2:20.3.0-5439.ge371433f.el9 ceph 100 k 2026-02-23T13:03:51.785 INFO:teuthology.orchestra.run.trial068.stdout: qatlib x86_64 25.08.0-2.el9 appstream 240 k 2026-02-23T13:03:51.785 INFO:teuthology.orchestra.run.trial068.stdout: qatzip-libs x86_64 1.3.1-1.el9 appstream 66 k 2026-02-23T13:03:51.785 INFO:teuthology.orchestra.run.trial068.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2026-02-23T13:03:51.785 INFO:teuthology.orchestra.run.trial068.stdout: thrift x86_64 0.15.0-4.el9 epel 1.6 M 2026-02-23T13:03:51.785 INFO:teuthology.orchestra.run.trial068.stdout:Installing weak dependencies: 2026-02-23T13:03:51.785 INFO:teuthology.orchestra.run.trial068.stdout: qatlib-service x86_64 25.08.0-2.el9 appstream 37 k 2026-02-23T13:03:51.786 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T13:03:51.786 INFO:teuthology.orchestra.run.trial068.stdout:Transaction Summary 2026-02-23T13:03:51.786 INFO:teuthology.orchestra.run.trial068.stdout:================================================================================ 2026-02-23T13:03:51.786 INFO:teuthology.orchestra.run.trial068.stdout:Install 32 Packages 2026-02-23T13:03:51.786 INFO:teuthology.orchestra.run.trial068.stdout:Upgrade 2 Packages 2026-02-23T13:03:51.786 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T13:03:51.787 INFO:teuthology.orchestra.run.trial068.stdout:Total download size: 60 M 2026-02-23T13:03:51.787 INFO:teuthology.orchestra.run.trial068.stdout:Downloading Packages: 2026-02-23T13:03:52.688 INFO:teuthology.orchestra.run.trial068.stdout:(1/34): ceph-base-20.3.0-5439.ge371433f.el9.x86 14 MB/s | 5.1 MB 00:00 2026-02-23T13:03:52.725 INFO:teuthology.orchestra.run.trial068.stdout:(2/34): ceph-selinux-20.3.0-5439.ge371433f.el9. 678 kB/s | 25 kB 00:00 2026-02-23T13:03:52.786 INFO:teuthology.orchestra.run.trial068.stdout:(3/34): libcephfs2-20.3.0-5439.ge371433f.el9.x8 14 MB/s | 860 kB 00:00 2026-02-23T13:03:52.832 INFO:teuthology.orchestra.run.trial068.stdout:(4/34): libradosstriper1-20.3.0-5439.ge371433f. 5.6 MB/s | 263 kB 00:00 2026-02-23T13:03:52.936 INFO:teuthology.orchestra.run.trial068.stdout:(5/34): ceph-radosgw-20.3.0-5439.ge371433f.el9. 18 MB/s | 11 MB 00:00 2026-02-23T13:03:52.984 INFO:teuthology.orchestra.run.trial068.stdout:(6/34): python3-ceph-argparse-20.3.0-5439.ge371 951 kB/s | 45 kB 00:00 2026-02-23T13:03:53.031 INFO:teuthology.orchestra.run.trial068.stdout:(7/34): python3-ceph-common-20.3.0-5439.ge37143 3.8 MB/s | 186 kB 00:00 2026-02-23T13:03:53.082 INFO:teuthology.orchestra.run.trial068.stdout:(8/34): python3-cephfs-20.3.0-5439.ge371433f.el 3.9 MB/s | 204 kB 00:00 2026-02-23T13:03:53.093 INFO:teuthology.orchestra.run.trial068.stdout:(9/34): librgw2-20.3.0-5439.ge371433f.el9.x86_6 22 MB/s | 5.6 MB 00:00 2026-02-23T13:03:53.117 INFO:teuthology.orchestra.run.trial068.stdout:(10/34): python3-rados-20.3.0-5439.ge371433f.el 9.1 MB/s | 324 kB 00:00 2026-02-23T13:03:53.132 INFO:teuthology.orchestra.run.trial068.stdout:(11/34): python3-rbd-20.3.0-5439.ge371433f.el9. 7.6 MB/s | 305 kB 00:00 2026-02-23T13:03:53.153 INFO:teuthology.orchestra.run.trial068.stdout:(12/34): python3-rgw-20.3.0-5439.ge371433f.el9. 2.7 MB/s | 100 kB 00:00 2026-02-23T13:03:53.214 INFO:teuthology.orchestra.run.trial068.stdout:(13/34): ceph-common-20.3.0-5439.ge371433f.el9. 24 MB/s | 21 MB 00:00 2026-02-23T13:03:53.268 INFO:teuthology.orchestra.run.trial068.stdout:(14/34): boost-program-options-1.75.0-13.el9.x8 1.9 MB/s | 104 kB 00:00 2026-02-23T13:03:53.287 INFO:teuthology.orchestra.run.trial068.stdout:(15/34): libnbd-1.20.3-4.el9.x86_64.rpm 8.5 MB/s | 164 kB 00:00 2026-02-23T13:03:53.299 INFO:teuthology.orchestra.run.trial068.stdout:(16/34): libpmemobj-1.12.1-1.el9.x86_64.rpm 12 MB/s | 160 kB 00:00 2026-02-23T13:03:53.308 INFO:teuthology.orchestra.run.trial068.stdout:(17/34): librabbitmq-0.11.0-7.el9.x86_64.rpm 4.8 MB/s | 45 kB 00:00 2026-02-23T13:03:53.331 INFO:teuthology.orchestra.run.trial068.stdout:(18/34): librdkafka-1.6.1-102.el9.x86_64.rpm 28 MB/s | 662 kB 00:00 2026-02-23T13:03:53.343 INFO:teuthology.orchestra.run.trial068.stdout:(19/34): lttng-ust-2.12.0-6.el9.x86_64.rpm 26 MB/s | 292 kB 00:00 2026-02-23T13:03:53.354 INFO:teuthology.orchestra.run.trial068.stdout:(20/34): qatlib-25.08.0-2.el9.x86_64.rpm 22 MB/s | 240 kB 00:00 2026-02-23T13:03:53.363 INFO:teuthology.orchestra.run.trial068.stdout:(21/34): qatlib-service-25.08.0-2.el9.x86_64.rp 4.0 MB/s | 37 kB 00:00 2026-02-23T13:03:53.372 INFO:teuthology.orchestra.run.trial068.stdout:(22/34): qatzip-libs-1.3.1-1.el9.x86_64.rpm 6.9 MB/s | 66 kB 00:00 2026-02-23T13:03:53.619 INFO:teuthology.orchestra.run.trial068.stdout:(23/34): mailcap-2.1.49-5.el9.noarch.rpm 68 kB/s | 33 kB 00:00 2026-02-23T13:03:53.771 INFO:teuthology.orchestra.run.trial068.stdout:(24/34): pciutils-3.7.0-7.el9.x86_64.rpm 151 kB/s | 93 kB 00:00 2026-02-23T13:03:53.776 INFO:teuthology.orchestra.run.trial068.stdout:(25/34): gperftools-libs-2.9.1-3.el9.x86_64.rpm 762 kB/s | 308 kB 00:00 2026-02-23T13:03:53.856 INFO:teuthology.orchestra.run.trial068.stdout:(26/34): liboath-2.6.12-1.el9.x86_64.rpm 613 kB/s | 49 kB 00:00 2026-02-23T13:03:53.946 INFO:teuthology.orchestra.run.trial068.stdout:(27/34): libunwind-1.6.2-1.el9.x86_64.rpm 748 kB/s | 67 kB 00:00 2026-02-23T13:03:53.982 INFO:teuthology.orchestra.run.trial068.stdout:(28/34): libarrow-doc-9.0.0-15.el9.noarch.rpm 117 kB/s | 25 kB 00:00 2026-02-23T13:03:54.081 INFO:teuthology.orchestra.run.trial068.stdout:(29/34): parquet-libs-9.0.0-15.el9.x86_64.rpm 6.0 MB/s | 838 kB 00:00 2026-02-23T13:03:54.261 INFO:teuthology.orchestra.run.trial068.stdout:(30/34): re2-20211101-20.el9.x86_64.rpm 686 kB/s | 191 kB 00:00 2026-02-23T13:03:54.280 INFO:teuthology.orchestra.run.trial068.stdout:(31/34): thrift-0.15.0-4.el9.x86_64.rpm 8.0 MB/s | 1.6 MB 00:00 2026-02-23T13:03:54.305 INFO:teuthology.orchestra.run.trial068.stdout:(32/34): libarrow-9.0.0-15.el9.x86_64.rpm 6.4 MB/s | 4.4 MB 00:00 2026-02-23T13:03:54.375 INFO:teuthology.orchestra.run.trial068.stdout:(33/34): librbd1-20.3.0-5439.ge371433f.el9.x86_ 28 MB/s | 2.7 MB 00:00 2026-02-23T13:03:54.414 INFO:teuthology.orchestra.run.trial068.stdout:(34/34): librados2-20.3.0-5439.ge371433f.el9.x8 22 MB/s | 3.3 MB 00:00 2026-02-23T13:03:54.417 INFO:teuthology.orchestra.run.trial068.stdout:-------------------------------------------------------------------------------- 2026-02-23T13:03:54.417 INFO:teuthology.orchestra.run.trial068.stdout:Total 23 MB/s | 60 MB 00:02 2026-02-23T13:03:54.506 INFO:teuthology.orchestra.run.trial068.stdout:Running transaction check 2026-02-23T13:03:54.523 INFO:teuthology.orchestra.run.trial068.stdout:Transaction check succeeded. 2026-02-23T13:03:54.523 INFO:teuthology.orchestra.run.trial068.stdout:Running transaction test 2026-02-23T13:03:54.767 INFO:teuthology.orchestra.run.trial068.stdout:Transaction test succeeded. 2026-02-23T13:03:54.767 INFO:teuthology.orchestra.run.trial068.stdout:Running transaction 2026-02-23T13:03:55.046 INFO:teuthology.orchestra.run.trial068.stdout: Preparing : 1/1 2026-02-23T13:03:55.188 INFO:teuthology.orchestra.run.trial068.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/36 2026-02-23T13:03:55.192 INFO:teuthology.orchestra.run.trial068.stdout: Upgrading : librados2-2:20.3.0-5439.ge371433f.el9.x86_64 2/36 2026-02-23T13:03:55.222 INFO:teuthology.orchestra.run.trial068.stdout: Running scriptlet: librados2-2:20.3.0-5439.ge371433f.el9.x86_64 2/36 2026-02-23T13:03:55.259 INFO:teuthology.orchestra.run.trial068.stdout: Installing : python3-rados-2:20.3.0-5439.ge371433f.el9.x86_64 3/36 2026-02-23T13:03:55.260 INFO:teuthology.orchestra.run.trial068.stdout: Installing : libcephfs2-2:20.3.0-5439.ge371433f.el9.x86_64 4/36 2026-02-23T13:03:55.272 INFO:teuthology.orchestra.run.trial068.stdout: Running scriptlet: libcephfs2-2:20.3.0-5439.ge371433f.el9.x86_64 4/36 2026-02-23T13:03:55.281 INFO:teuthology.orchestra.run.trial068.stdout: Installing : liboath-2.6.12-1.el9.x86_64 5/36 2026-02-23T13:03:55.284 INFO:teuthology.orchestra.run.trial068.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 6/36 2026-02-23T13:03:55.287 INFO:teuthology.orchestra.run.trial068.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 7/36 2026-02-23T13:03:55.291 INFO:teuthology.orchestra.run.trial068.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 8/36 2026-02-23T13:03:55.302 INFO:teuthology.orchestra.run.trial068.stdout: Installing : python3-ceph-argparse-2:20.3.0-5439.ge371433f.el9. 9/36 2026-02-23T13:03:55.315 INFO:teuthology.orchestra.run.trial068.stdout: Installing : python3-cephfs-2:20.3.0-5439.ge371433f.el9.x86_64 10/36 2026-02-23T13:03:55.316 INFO:teuthology.orchestra.run.trial068.stdout: Installing : libradosstriper1-2:20.3.0-5439.ge371433f.el9.x86_6 11/36 2026-02-23T13:03:55.339 INFO:teuthology.orchestra.run.trial068.stdout: Running scriptlet: libradosstriper1-2:20.3.0-5439.ge371433f.el9.x86_6 11/36 2026-02-23T13:03:55.343 INFO:teuthology.orchestra.run.trial068.stdout: Installing : thrift-0.15.0-4.el9.x86_64 12/36 2026-02-23T13:03:55.376 INFO:teuthology.orchestra.run.trial068.stdout: Installing : re2-1:20211101-20.el9.x86_64 13/36 2026-02-23T13:03:55.380 INFO:teuthology.orchestra.run.trial068.stdout: Installing : libarrow-9.0.0-15.el9.x86_64 14/36 2026-02-23T13:03:55.386 INFO:teuthology.orchestra.run.trial068.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 15/36 2026-02-23T13:03:55.388 INFO:teuthology.orchestra.run.trial068.stdout: Installing : gperftools-libs-2.9.1-3.el9.x86_64 16/36 2026-02-23T13:03:55.394 INFO:teuthology.orchestra.run.trial068.stdout: Installing : libarrow-doc-9.0.0-15.el9.noarch 17/36 2026-02-23T13:03:55.509 INFO:teuthology.orchestra.run.trial068.stdout: Installing : libnbd-1.20.3-4.el9.x86_64 18/36 2026-02-23T13:03:55.510 INFO:teuthology.orchestra.run.trial068.stdout: Upgrading : librbd1-2:20.3.0-5439.ge371433f.el9.x86_64 19/36 2026-02-23T13:03:55.538 INFO:teuthology.orchestra.run.trial068.stdout: Running scriptlet: librbd1-2:20.3.0-5439.ge371433f.el9.x86_64 19/36 2026-02-23T13:03:55.540 INFO:teuthology.orchestra.run.trial068.stdout: Installing : python3-rbd-2:20.3.0-5439.ge371433f.el9.x86_64 20/36 2026-02-23T13:03:55.547 INFO:teuthology.orchestra.run.trial068.stdout: Installing : boost-program-options-1.75.0-13.el9.x86_64 21/36 2026-02-23T13:03:55.795 INFO:teuthology.orchestra.run.trial068.stdout: Installing : parquet-libs-9.0.0-15.el9.x86_64 22/36 2026-02-23T13:03:55.796 INFO:teuthology.orchestra.run.trial068.stdout: Installing : librgw2-2:20.3.0-5439.ge371433f.el9.x86_64 23/36 2026-02-23T13:03:55.813 INFO:teuthology.orchestra.run.trial068.stdout: Running scriptlet: librgw2-2:20.3.0-5439.ge371433f.el9.x86_64 23/36 2026-02-23T13:03:55.818 INFO:teuthology.orchestra.run.trial068.stdout: Installing : python3-rgw-2:20.3.0-5439.ge371433f.el9.x86_64 24/36 2026-02-23T13:03:55.822 INFO:teuthology.orchestra.run.trial068.stdout: Installing : pciutils-3.7.0-7.el9.x86_64 25/36 2026-02-23T13:03:55.825 INFO:teuthology.orchestra.run.trial068.stdout: Installing : qatlib-25.08.0-2.el9.x86_64 26/36 2026-02-23T13:03:55.826 INFO:teuthology.orchestra.run.trial068.stdout: Installing : qatlib-service-25.08.0-2.el9.x86_64 27/36 2026-02-23T13:03:55.844 INFO:teuthology.orchestra.run.trial068.stdout: Running scriptlet: qatlib-service-25.08.0-2.el9.x86_64 27/36 2026-02-23T13:03:55.848 INFO:teuthology.orchestra.run.trial068.stdout: Installing : qatzip-libs-1.3.1-1.el9.x86_64 28/36 2026-02-23T13:03:55.883 INFO:teuthology.orchestra.run.trial068.stdout: Installing : mailcap-2.1.49-5.el9.noarch 29/36 2026-02-23T13:03:55.885 INFO:teuthology.orchestra.run.trial068.stdout: Installing : python3-ceph-common-2:20.3.0-5439.ge371433f.el9.x8 30/36 2026-02-23T13:03:55.941 INFO:teuthology.orchestra.run.trial185.stdout: Running scriptlet: ceph-selinux-2:20.3.0-5439.ge371433f.el9.x86_64 33/36 2026-02-23T13:03:55.941 INFO:teuthology.orchestra.run.trial185.stdout:skipping the directory /sys 2026-02-23T13:03:55.941 INFO:teuthology.orchestra.run.trial185.stdout:skipping the directory /proc 2026-02-23T13:03:55.941 INFO:teuthology.orchestra.run.trial185.stdout:skipping the directory /mnt 2026-02-23T13:03:55.941 INFO:teuthology.orchestra.run.trial185.stdout:skipping the directory /var/tmp 2026-02-23T13:03:55.941 INFO:teuthology.orchestra.run.trial185.stdout:skipping the directory /home 2026-02-23T13:03:55.941 INFO:teuthology.orchestra.run.trial185.stdout:skipping the directory /root 2026-02-23T13:03:55.941 INFO:teuthology.orchestra.run.trial185.stdout:skipping the directory /tmp 2026-02-23T13:03:55.941 INFO:teuthology.orchestra.run.trial185.stdout: 2026-02-23T13:03:56.440 INFO:teuthology.orchestra.run.trial185.stdout: Installing : ceph-radosgw-2:20.3.0-5439.ge371433f.el9.x86_64 34/36 2026-02-23T13:03:56.463 INFO:teuthology.orchestra.run.trial185.stdout: Running scriptlet: ceph-radosgw-2:20.3.0-5439.ge371433f.el9.x86_64 34/36 2026-02-23T13:03:56.463 INFO:teuthology.orchestra.run.trial185.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-23T13:03:56.463 INFO:teuthology.orchestra.run.trial185.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2026-02-23T13:03:56.463 INFO:teuthology.orchestra.run.trial185.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2026-02-23T13:03:56.463 INFO:teuthology.orchestra.run.trial185.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2026-02-23T13:03:56.463 INFO:teuthology.orchestra.run.trial185.stdout: 2026-02-23T13:03:56.465 INFO:teuthology.orchestra.run.trial185.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 35/36 2026-02-23T13:03:56.475 INFO:teuthology.orchestra.run.trial185.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 35/36 2026-02-23T13:03:56.475 INFO:teuthology.orchestra.run.trial185.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 36/36 2026-02-23T13:03:56.532 INFO:teuthology.orchestra.run.trial185.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 36/36 2026-02-23T13:03:56.532 INFO:teuthology.orchestra.run.trial185.stdout:Creating group 'qat' with GID 992. 2026-02-23T13:03:56.533 INFO:teuthology.orchestra.run.trial185.stdout: 2026-02-23T13:03:56.827 INFO:teuthology.orchestra.run.trial068.stdout: Running scriptlet: ceph-common-2:20.3.0-5439.ge371433f.el9.x86_64 31/36 2026-02-23T13:03:56.830 INFO:teuthology.orchestra.run.trial068.stdout: Installing : ceph-common-2:20.3.0-5439.ge371433f.el9.x86_64 31/36 2026-02-23T13:03:56.838 INFO:teuthology.orchestra.run.trial185.stdout: Verifying : ceph-base-2:20.3.0-5439.ge371433f.el9.x86_64 1/36 2026-02-23T13:03:56.838 INFO:teuthology.orchestra.run.trial185.stdout: Verifying : ceph-common-2:20.3.0-5439.ge371433f.el9.x86_64 2/36 2026-02-23T13:03:56.839 INFO:teuthology.orchestra.run.trial185.stdout: Verifying : ceph-radosgw-2:20.3.0-5439.ge371433f.el9.x86_64 3/36 2026-02-23T13:03:56.839 INFO:teuthology.orchestra.run.trial185.stdout: Verifying : ceph-selinux-2:20.3.0-5439.ge371433f.el9.x86_64 4/36 2026-02-23T13:03:56.839 INFO:teuthology.orchestra.run.trial185.stdout: Verifying : libcephfs2-2:20.3.0-5439.ge371433f.el9.x86_64 5/36 2026-02-23T13:03:56.839 INFO:teuthology.orchestra.run.trial185.stdout: Verifying : libradosstriper1-2:20.3.0-5439.ge371433f.el9.x86_6 6/36 2026-02-23T13:03:56.839 INFO:teuthology.orchestra.run.trial185.stdout: Verifying : librgw2-2:20.3.0-5439.ge371433f.el9.x86_64 7/36 2026-02-23T13:03:56.839 INFO:teuthology.orchestra.run.trial185.stdout: Verifying : python3-ceph-argparse-2:20.3.0-5439.ge371433f.el9. 8/36 2026-02-23T13:03:56.839 INFO:teuthology.orchestra.run.trial185.stdout: Verifying : python3-ceph-common-2:20.3.0-5439.ge371433f.el9.x8 9/36 2026-02-23T13:03:56.839 INFO:teuthology.orchestra.run.trial185.stdout: Verifying : python3-cephfs-2:20.3.0-5439.ge371433f.el9.x86_64 10/36 2026-02-23T13:03:56.839 INFO:teuthology.orchestra.run.trial185.stdout: Verifying : python3-rados-2:20.3.0-5439.ge371433f.el9.x86_64 11/36 2026-02-23T13:03:56.839 INFO:teuthology.orchestra.run.trial185.stdout: Verifying : python3-rbd-2:20.3.0-5439.ge371433f.el9.x86_64 12/36 2026-02-23T13:03:56.839 INFO:teuthology.orchestra.run.trial185.stdout: Verifying : python3-rgw-2:20.3.0-5439.ge371433f.el9.x86_64 13/36 2026-02-23T13:03:56.839 INFO:teuthology.orchestra.run.trial185.stdout: Verifying : mailcap-2.1.49-5.el9.noarch 14/36 2026-02-23T13:03:56.840 INFO:teuthology.orchestra.run.trial185.stdout: Verifying : pciutils-3.7.0-7.el9.x86_64 15/36 2026-02-23T13:03:56.840 INFO:teuthology.orchestra.run.trial185.stdout: Verifying : boost-program-options-1.75.0-13.el9.x86_64 16/36 2026-02-23T13:03:56.840 INFO:teuthology.orchestra.run.trial185.stdout: Verifying : libnbd-1.20.3-4.el9.x86_64 17/36 2026-02-23T13:03:56.840 INFO:teuthology.orchestra.run.trial185.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 18/36 2026-02-23T13:03:56.840 INFO:teuthology.orchestra.run.trial185.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 19/36 2026-02-23T13:03:56.840 INFO:teuthology.orchestra.run.trial185.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 20/36 2026-02-23T13:03:56.840 INFO:teuthology.orchestra.run.trial185.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 21/36 2026-02-23T13:03:56.840 INFO:teuthology.orchestra.run.trial185.stdout: Verifying : qatlib-25.08.0-2.el9.x86_64 22/36 2026-02-23T13:03:56.840 INFO:teuthology.orchestra.run.trial185.stdout: Verifying : qatlib-service-25.08.0-2.el9.x86_64 23/36 2026-02-23T13:03:56.840 INFO:teuthology.orchestra.run.trial185.stdout: Verifying : qatzip-libs-1.3.1-1.el9.x86_64 24/36 2026-02-23T13:03:56.840 INFO:teuthology.orchestra.run.trial185.stdout: Verifying : gperftools-libs-2.9.1-3.el9.x86_64 25/36 2026-02-23T13:03:56.840 INFO:teuthology.orchestra.run.trial185.stdout: Verifying : libarrow-9.0.0-15.el9.x86_64 26/36 2026-02-23T13:03:56.840 INFO:teuthology.orchestra.run.trial185.stdout: Verifying : libarrow-doc-9.0.0-15.el9.noarch 27/36 2026-02-23T13:03:56.841 INFO:teuthology.orchestra.run.trial185.stdout: Verifying : liboath-2.6.12-1.el9.x86_64 28/36 2026-02-23T13:03:56.841 INFO:teuthology.orchestra.run.trial185.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 29/36 2026-02-23T13:03:56.841 INFO:teuthology.orchestra.run.trial185.stdout: Verifying : parquet-libs-9.0.0-15.el9.x86_64 30/36 2026-02-23T13:03:56.841 INFO:teuthology.orchestra.run.trial185.stdout: Verifying : re2-1:20211101-20.el9.x86_64 31/36 2026-02-23T13:03:56.841 INFO:teuthology.orchestra.run.trial185.stdout: Verifying : thrift-0.15.0-4.el9.x86_64 32/36 2026-02-23T13:03:56.841 INFO:teuthology.orchestra.run.trial185.stdout: Verifying : librados2-2:20.3.0-5439.ge371433f.el9.x86_64 33/36 2026-02-23T13:03:56.841 INFO:teuthology.orchestra.run.trial185.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 34/36 2026-02-23T13:03:56.841 INFO:teuthology.orchestra.run.trial185.stdout: Verifying : librbd1-2:20.3.0-5439.ge371433f.el9.x86_64 35/36 2026-02-23T13:03:56.901 INFO:teuthology.orchestra.run.trial185.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 36/36 2026-02-23T13:03:56.901 INFO:teuthology.orchestra.run.trial185.stdout: 2026-02-23T13:03:56.901 INFO:teuthology.orchestra.run.trial185.stdout:Upgraded: 2026-02-23T13:03:56.901 INFO:teuthology.orchestra.run.trial185.stdout: librados2-2:20.3.0-5439.ge371433f.el9.x86_64 2026-02-23T13:03:56.902 INFO:teuthology.orchestra.run.trial185.stdout: librbd1-2:20.3.0-5439.ge371433f.el9.x86_64 2026-02-23T13:03:56.902 INFO:teuthology.orchestra.run.trial185.stdout:Installed: 2026-02-23T13:03:56.902 INFO:teuthology.orchestra.run.trial185.stdout: boost-program-options-1.75.0-13.el9.x86_64 2026-02-23T13:03:56.902 INFO:teuthology.orchestra.run.trial185.stdout: ceph-base-2:20.3.0-5439.ge371433f.el9.x86_64 2026-02-23T13:03:56.902 INFO:teuthology.orchestra.run.trial185.stdout: ceph-common-2:20.3.0-5439.ge371433f.el9.x86_64 2026-02-23T13:03:56.902 INFO:teuthology.orchestra.run.trial185.stdout: ceph-radosgw-2:20.3.0-5439.ge371433f.el9.x86_64 2026-02-23T13:03:56.902 INFO:teuthology.orchestra.run.trial185.stdout: ceph-selinux-2:20.3.0-5439.ge371433f.el9.x86_64 2026-02-23T13:03:56.902 INFO:teuthology.orchestra.run.trial185.stdout: gperftools-libs-2.9.1-3.el9.x86_64 2026-02-23T13:03:56.902 INFO:teuthology.orchestra.run.trial185.stdout: libarrow-9.0.0-15.el9.x86_64 2026-02-23T13:03:56.902 INFO:teuthology.orchestra.run.trial185.stdout: libarrow-doc-9.0.0-15.el9.noarch 2026-02-23T13:03:56.902 INFO:teuthology.orchestra.run.trial185.stdout: libcephfs2-2:20.3.0-5439.ge371433f.el9.x86_64 2026-02-23T13:03:56.903 INFO:teuthology.orchestra.run.trial185.stdout: libnbd-1.20.3-4.el9.x86_64 2026-02-23T13:03:56.903 INFO:teuthology.orchestra.run.trial185.stdout: liboath-2.6.12-1.el9.x86_64 2026-02-23T13:03:56.903 INFO:teuthology.orchestra.run.trial185.stdout: libpmemobj-1.12.1-1.el9.x86_64 2026-02-23T13:03:56.903 INFO:teuthology.orchestra.run.trial185.stdout: librabbitmq-0.11.0-7.el9.x86_64 2026-02-23T13:03:56.903 INFO:teuthology.orchestra.run.trial185.stdout: libradosstriper1-2:20.3.0-5439.ge371433f.el9.x86_64 2026-02-23T13:03:56.903 INFO:teuthology.orchestra.run.trial185.stdout: librdkafka-1.6.1-102.el9.x86_64 2026-02-23T13:03:56.903 INFO:teuthology.orchestra.run.trial185.stdout: librgw2-2:20.3.0-5439.ge371433f.el9.x86_64 2026-02-23T13:03:56.903 INFO:teuthology.orchestra.run.trial185.stdout: libunwind-1.6.2-1.el9.x86_64 2026-02-23T13:03:56.903 INFO:teuthology.orchestra.run.trial185.stdout: lttng-ust-2.12.0-6.el9.x86_64 2026-02-23T13:03:56.903 INFO:teuthology.orchestra.run.trial185.stdout: mailcap-2.1.49-5.el9.noarch 2026-02-23T13:03:56.903 INFO:teuthology.orchestra.run.trial185.stdout: parquet-libs-9.0.0-15.el9.x86_64 2026-02-23T13:03:56.904 INFO:teuthology.orchestra.run.trial185.stdout: pciutils-3.7.0-7.el9.x86_64 2026-02-23T13:03:56.904 INFO:teuthology.orchestra.run.trial185.stdout: python3-ceph-argparse-2:20.3.0-5439.ge371433f.el9.x86_64 2026-02-23T13:03:56.904 INFO:teuthology.orchestra.run.trial185.stdout: python3-ceph-common-2:20.3.0-5439.ge371433f.el9.x86_64 2026-02-23T13:03:56.904 INFO:teuthology.orchestra.run.trial185.stdout: python3-cephfs-2:20.3.0-5439.ge371433f.el9.x86_64 2026-02-23T13:03:56.904 INFO:teuthology.orchestra.run.trial185.stdout: python3-rados-2:20.3.0-5439.ge371433f.el9.x86_64 2026-02-23T13:03:56.904 INFO:teuthology.orchestra.run.trial185.stdout: python3-rbd-2:20.3.0-5439.ge371433f.el9.x86_64 2026-02-23T13:03:56.904 INFO:teuthology.orchestra.run.trial185.stdout: python3-rgw-2:20.3.0-5439.ge371433f.el9.x86_64 2026-02-23T13:03:56.904 INFO:teuthology.orchestra.run.trial185.stdout: qatlib-25.08.0-2.el9.x86_64 2026-02-23T13:03:56.904 INFO:teuthology.orchestra.run.trial185.stdout: qatlib-service-25.08.0-2.el9.x86_64 2026-02-23T13:03:56.904 INFO:teuthology.orchestra.run.trial185.stdout: qatzip-libs-1.3.1-1.el9.x86_64 2026-02-23T13:03:56.904 INFO:teuthology.orchestra.run.trial185.stdout: re2-1:20211101-20.el9.x86_64 2026-02-23T13:03:56.904 INFO:teuthology.orchestra.run.trial185.stdout: thrift-0.15.0-4.el9.x86_64 2026-02-23T13:03:56.905 INFO:teuthology.orchestra.run.trial185.stdout: 2026-02-23T13:03:56.905 INFO:teuthology.orchestra.run.trial185.stdout:Complete! 2026-02-23T13:03:56.942 DEBUG:teuthology.orchestra.run.trial185:> sudo yum -y install ceph-test 2026-02-23T13:03:57.090 INFO:teuthology.orchestra.run.trial068.stdout: Running scriptlet: ceph-common-2:20.3.0-5439.ge371433f.el9.x86_64 31/36 2026-02-23T13:03:57.094 INFO:teuthology.orchestra.run.trial068.stdout: Installing : ceph-base-2:20.3.0-5439.ge371433f.el9.x86_64 32/36 2026-02-23T13:03:57.133 INFO:teuthology.orchestra.run.trial068.stdout: Running scriptlet: ceph-base-2:20.3.0-5439.ge371433f.el9.x86_64 32/36 2026-02-23T13:03:57.133 INFO:teuthology.orchestra.run.trial068.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2026-02-23T13:03:57.134 INFO:teuthology.orchestra.run.trial068.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2026-02-23T13:03:57.134 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T13:03:57.137 INFO:teuthology.orchestra.run.trial068.stdout: Installing : ceph-selinux-2:20.3.0-5439.ge371433f.el9.x86_64 33/36 2026-02-23T13:03:57.256 INFO:teuthology.orchestra.run.trial185.stdout:Last metadata expiration check: 0:00:13 ago on Mon 23 Feb 2026 01:03:44 PM UTC. 2026-02-23T13:03:57.338 INFO:teuthology.orchestra.run.trial185.stdout:Dependencies resolved. 2026-02-23T13:03:57.339 INFO:teuthology.orchestra.run.trial185.stdout:================================================================================ 2026-02-23T13:03:57.339 INFO:teuthology.orchestra.run.trial185.stdout: Package Arch Version Repository Size 2026-02-23T13:03:57.339 INFO:teuthology.orchestra.run.trial185.stdout:================================================================================ 2026-02-23T13:03:57.339 INFO:teuthology.orchestra.run.trial185.stdout:Installing: 2026-02-23T13:03:57.339 INFO:teuthology.orchestra.run.trial185.stdout: ceph-test x86_64 2:20.3.0-5439.ge371433f.el9 ceph 126 M 2026-02-23T13:03:57.339 INFO:teuthology.orchestra.run.trial185.stdout:Installing dependencies: 2026-02-23T13:03:57.339 INFO:teuthology.orchestra.run.trial185.stdout: hwloc-libs x86_64 2.4.1-6.el9 baseos 2.1 M 2026-02-23T13:03:57.339 INFO:teuthology.orchestra.run.trial185.stdout: libasan8 x86_64 15.2.1-7.el9 appstream 525 k 2026-02-23T13:03:57.339 INFO:teuthology.orchestra.run.trial185.stdout: libatomic x86_64 11.5.0-14.el9 baseos 23 k 2026-02-23T13:03:57.339 INFO:teuthology.orchestra.run.trial185.stdout: libcephsqlite x86_64 2:20.3.0-5439.ge371433f.el9 ceph 153 k 2026-02-23T13:03:57.339 INFO:teuthology.orchestra.run.trial185.stdout: libubsan x86_64 11.5.0-14.el9 appstream 176 k 2026-02-23T13:03:57.339 INFO:teuthology.orchestra.run.trial185.stdout: libxslt x86_64 1.1.34-12.el9 appstream 233 k 2026-02-23T13:03:57.339 INFO:teuthology.orchestra.run.trial185.stdout: lksctp-tools x86_64 1.0.19-2.el9 baseos 94 k 2026-02-23T13:03:57.339 INFO:teuthology.orchestra.run.trial185.stdout: protobuf x86_64 3.14.0-17.el9 appstream 1.0 M 2026-02-23T13:03:57.339 INFO:teuthology.orchestra.run.trial185.stdout: socat x86_64 1.7.4.1-8.el9 appstream 303 k 2026-02-23T13:03:57.339 INFO:teuthology.orchestra.run.trial185.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2026-02-23T13:03:57.340 INFO:teuthology.orchestra.run.trial185.stdout: yaml-cpp x86_64 0.6.3-5.el9 epel 122 k 2026-02-23T13:03:57.340 INFO:teuthology.orchestra.run.trial185.stdout: 2026-02-23T13:03:57.340 INFO:teuthology.orchestra.run.trial185.stdout:Transaction Summary 2026-02-23T13:03:57.340 INFO:teuthology.orchestra.run.trial185.stdout:================================================================================ 2026-02-23T13:03:57.340 INFO:teuthology.orchestra.run.trial185.stdout:Install 12 Packages 2026-02-23T13:03:57.340 INFO:teuthology.orchestra.run.trial185.stdout: 2026-02-23T13:03:57.340 INFO:teuthology.orchestra.run.trial185.stdout:Total download size: 131 M 2026-02-23T13:03:57.340 INFO:teuthology.orchestra.run.trial185.stdout:Installed size: 977 M 2026-02-23T13:03:57.340 INFO:teuthology.orchestra.run.trial185.stdout:Downloading Packages: 2026-02-23T13:03:57.692 INFO:teuthology.orchestra.run.trial185.stdout:(1/12): libcephsqlite-20.3.0-5439.ge371433f.el9 1.6 MB/s | 153 kB 00:00 2026-02-23T13:03:58.094 INFO:teuthology.orchestra.run.trial185.stdout:(2/12): libatomic-11.5.0-14.el9.x86_64.rpm 57 kB/s | 23 kB 00:00 2026-02-23T13:03:58.335 INFO:teuthology.orchestra.run.trial185.stdout:(3/12): lksctp-tools-1.0.19-2.el9.x86_64.rpm 391 kB/s | 94 kB 00:00 2026-02-23T13:03:58.404 INFO:teuthology.orchestra.run.trial185.stdout:(4/12): libasan8-15.2.1-7.el9.x86_64.rpm 7.4 MB/s | 525 kB 00:00 2026-02-23T13:03:58.415 INFO:teuthology.orchestra.run.trial185.stdout:(5/12): libubsan-11.5.0-14.el9.x86_64.rpm 16 MB/s | 176 kB 00:00 2026-02-23T13:03:58.426 INFO:teuthology.orchestra.run.trial185.stdout:(6/12): libxslt-1.1.34-12.el9.x86_64.rpm 21 MB/s | 233 kB 00:00 2026-02-23T13:03:58.454 INFO:teuthology.orchestra.run.trial185.stdout:(7/12): protobuf-3.14.0-17.el9.x86_64.rpm 36 MB/s | 1.0 MB 00:00 2026-02-23T13:03:58.465 INFO:teuthology.orchestra.run.trial185.stdout:(8/12): socat-1.7.4.1-8.el9.x86_64.rpm 27 MB/s | 303 kB 00:00 2026-02-23T13:03:58.475 INFO:teuthology.orchestra.run.trial185.stdout:(9/12): xmlstarlet-1.6.1-20.el9.x86_64.rpm 6.8 MB/s | 64 kB 00:00 2026-02-23T13:03:58.503 INFO:teuthology.orchestra.run.trial185.stdout:(10/12): yaml-cpp-0.6.3-5.el9.x86_64.rpm 4.3 MB/s | 122 kB 00:00 2026-02-23T13:04:00.191 INFO:teuthology.orchestra.run.trial185.stdout:(11/12): ceph-test-20.3.0-5439.ge371433f.el9.x8 49 MB/s | 126 MB 00:02 2026-02-23T13:04:00.510 INFO:teuthology.orchestra.run.trial185.stdout:(12/12): hwloc-libs-2.4.1-6.el9.x86_64.rpm 737 kB/s | 2.1 MB 00:02 2026-02-23T13:04:00.514 INFO:teuthology.orchestra.run.trial185.stdout:-------------------------------------------------------------------------------- 2026-02-23T13:04:00.514 INFO:teuthology.orchestra.run.trial185.stdout:Total 41 MB/s | 131 MB 00:03 2026-02-23T13:04:00.597 INFO:teuthology.orchestra.run.trial185.stdout:Running transaction check 2026-02-23T13:04:00.609 INFO:teuthology.orchestra.run.trial185.stdout:Transaction check succeeded. 2026-02-23T13:04:00.609 INFO:teuthology.orchestra.run.trial185.stdout:Running transaction test 2026-02-23T13:04:01.034 INFO:teuthology.orchestra.run.trial185.stdout:Transaction test succeeded. 2026-02-23T13:04:01.034 INFO:teuthology.orchestra.run.trial185.stdout:Running transaction 2026-02-23T13:04:01.486 INFO:teuthology.orchestra.run.trial185.stdout: Preparing : 1/1 2026-02-23T13:04:01.495 INFO:teuthology.orchestra.run.trial185.stdout: Installing : yaml-cpp-0.6.3-5.el9.x86_64 1/12 2026-02-23T13:04:01.504 INFO:teuthology.orchestra.run.trial185.stdout: Installing : socat-1.7.4.1-8.el9.x86_64 2/12 2026-02-23T13:04:01.509 INFO:teuthology.orchestra.run.trial185.stdout: Installing : protobuf-3.14.0-17.el9.x86_64 3/12 2026-02-23T13:04:01.514 INFO:teuthology.orchestra.run.trial185.stdout: Installing : libxslt-1.1.34-12.el9.x86_64 4/12 2026-02-23T13:04:01.516 INFO:teuthology.orchestra.run.trial185.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 5/12 2026-02-23T13:04:01.520 INFO:teuthology.orchestra.run.trial185.stdout: Installing : libubsan-11.5.0-14.el9.x86_64 6/12 2026-02-23T13:04:01.521 INFO:teuthology.orchestra.run.trial185.stdout: Installing : libasan8-15.2.1-7.el9.x86_64 7/12 2026-02-23T13:04:01.539 INFO:teuthology.orchestra.run.trial185.stdout: Running scriptlet: libasan8-15.2.1-7.el9.x86_64 7/12 2026-02-23T13:04:01.541 INFO:teuthology.orchestra.run.trial185.stdout: Installing : lksctp-tools-1.0.19-2.el9.x86_64 8/12 2026-02-23T13:04:01.553 INFO:teuthology.orchestra.run.trial185.stdout: Installing : libatomic-11.5.0-14.el9.x86_64 9/12 2026-02-23T13:04:01.561 INFO:teuthology.orchestra.run.trial185.stdout: Installing : hwloc-libs-2.4.1-6.el9.x86_64 10/12 2026-02-23T13:04:01.562 INFO:teuthology.orchestra.run.trial185.stdout: Installing : libcephsqlite-2:20.3.0-5439.ge371433f.el9.x86_64 11/12 2026-02-23T13:04:01.684 INFO:teuthology.orchestra.run.trial068.stdout: Running scriptlet: ceph-selinux-2:20.3.0-5439.ge371433f.el9.x86_64 33/36 2026-02-23T13:04:01.684 INFO:teuthology.orchestra.run.trial068.stdout:skipping the directory /sys 2026-02-23T13:04:01.685 INFO:teuthology.orchestra.run.trial068.stdout:skipping the directory /proc 2026-02-23T13:04:01.685 INFO:teuthology.orchestra.run.trial068.stdout:skipping the directory /mnt 2026-02-23T13:04:01.685 INFO:teuthology.orchestra.run.trial068.stdout:skipping the directory /var/tmp 2026-02-23T13:04:01.685 INFO:teuthology.orchestra.run.trial068.stdout:skipping the directory /home 2026-02-23T13:04:01.685 INFO:teuthology.orchestra.run.trial068.stdout:skipping the directory /root 2026-02-23T13:04:01.685 INFO:teuthology.orchestra.run.trial068.stdout:skipping the directory /tmp 2026-02-23T13:04:01.685 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T13:04:02.179 INFO:teuthology.orchestra.run.trial068.stdout: Installing : ceph-radosgw-2:20.3.0-5439.ge371433f.el9.x86_64 34/36 2026-02-23T13:04:02.202 INFO:teuthology.orchestra.run.trial068.stdout: Running scriptlet: ceph-radosgw-2:20.3.0-5439.ge371433f.el9.x86_64 34/36 2026-02-23T13:04:02.202 INFO:teuthology.orchestra.run.trial068.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-23T13:04:02.202 INFO:teuthology.orchestra.run.trial068.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2026-02-23T13:04:02.202 INFO:teuthology.orchestra.run.trial068.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2026-02-23T13:04:02.202 INFO:teuthology.orchestra.run.trial068.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2026-02-23T13:04:02.202 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T13:04:02.204 INFO:teuthology.orchestra.run.trial068.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 35/36 2026-02-23T13:04:02.217 INFO:teuthology.orchestra.run.trial068.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 35/36 2026-02-23T13:04:02.218 INFO:teuthology.orchestra.run.trial068.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 36/36 2026-02-23T13:04:02.272 INFO:teuthology.orchestra.run.trial068.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 36/36 2026-02-23T13:04:02.272 INFO:teuthology.orchestra.run.trial068.stdout:Creating group 'qat' with GID 992. 2026-02-23T13:04:02.273 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T13:04:02.572 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : ceph-base-2:20.3.0-5439.ge371433f.el9.x86_64 1/36 2026-02-23T13:04:02.572 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : ceph-common-2:20.3.0-5439.ge371433f.el9.x86_64 2/36 2026-02-23T13:04:02.572 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : ceph-radosgw-2:20.3.0-5439.ge371433f.el9.x86_64 3/36 2026-02-23T13:04:02.572 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : ceph-selinux-2:20.3.0-5439.ge371433f.el9.x86_64 4/36 2026-02-23T13:04:02.572 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : libcephfs2-2:20.3.0-5439.ge371433f.el9.x86_64 5/36 2026-02-23T13:04:02.572 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : libradosstriper1-2:20.3.0-5439.ge371433f.el9.x86_6 6/36 2026-02-23T13:04:02.572 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : librgw2-2:20.3.0-5439.ge371433f.el9.x86_64 7/36 2026-02-23T13:04:02.573 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : python3-ceph-argparse-2:20.3.0-5439.ge371433f.el9. 8/36 2026-02-23T13:04:02.573 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : python3-ceph-common-2:20.3.0-5439.ge371433f.el9.x8 9/36 2026-02-23T13:04:02.573 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : python3-cephfs-2:20.3.0-5439.ge371433f.el9.x86_64 10/36 2026-02-23T13:04:02.573 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : python3-rados-2:20.3.0-5439.ge371433f.el9.x86_64 11/36 2026-02-23T13:04:02.573 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : python3-rbd-2:20.3.0-5439.ge371433f.el9.x86_64 12/36 2026-02-23T13:04:02.573 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : python3-rgw-2:20.3.0-5439.ge371433f.el9.x86_64 13/36 2026-02-23T13:04:02.573 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : mailcap-2.1.49-5.el9.noarch 14/36 2026-02-23T13:04:02.573 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : pciutils-3.7.0-7.el9.x86_64 15/36 2026-02-23T13:04:02.573 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : boost-program-options-1.75.0-13.el9.x86_64 16/36 2026-02-23T13:04:02.573 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : libnbd-1.20.3-4.el9.x86_64 17/36 2026-02-23T13:04:02.573 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 18/36 2026-02-23T13:04:02.573 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 19/36 2026-02-23T13:04:02.574 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 20/36 2026-02-23T13:04:02.574 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 21/36 2026-02-23T13:04:02.574 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : qatlib-25.08.0-2.el9.x86_64 22/36 2026-02-23T13:04:02.574 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : qatlib-service-25.08.0-2.el9.x86_64 23/36 2026-02-23T13:04:02.574 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : qatzip-libs-1.3.1-1.el9.x86_64 24/36 2026-02-23T13:04:02.574 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : gperftools-libs-2.9.1-3.el9.x86_64 25/36 2026-02-23T13:04:02.574 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : libarrow-9.0.0-15.el9.x86_64 26/36 2026-02-23T13:04:02.574 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : libarrow-doc-9.0.0-15.el9.noarch 27/36 2026-02-23T13:04:02.574 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : liboath-2.6.12-1.el9.x86_64 28/36 2026-02-23T13:04:02.574 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 29/36 2026-02-23T13:04:02.574 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : parquet-libs-9.0.0-15.el9.x86_64 30/36 2026-02-23T13:04:02.574 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : re2-1:20211101-20.el9.x86_64 31/36 2026-02-23T13:04:02.575 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : thrift-0.15.0-4.el9.x86_64 32/36 2026-02-23T13:04:02.575 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : librados2-2:20.3.0-5439.ge371433f.el9.x86_64 33/36 2026-02-23T13:04:02.575 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 34/36 2026-02-23T13:04:02.575 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : librbd1-2:20.3.0-5439.ge371433f.el9.x86_64 35/36 2026-02-23T13:04:02.633 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 36/36 2026-02-23T13:04:02.633 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T13:04:02.633 INFO:teuthology.orchestra.run.trial068.stdout:Upgraded: 2026-02-23T13:04:02.633 INFO:teuthology.orchestra.run.trial068.stdout: librados2-2:20.3.0-5439.ge371433f.el9.x86_64 2026-02-23T13:04:02.634 INFO:teuthology.orchestra.run.trial068.stdout: librbd1-2:20.3.0-5439.ge371433f.el9.x86_64 2026-02-23T13:04:02.634 INFO:teuthology.orchestra.run.trial068.stdout:Installed: 2026-02-23T13:04:02.634 INFO:teuthology.orchestra.run.trial068.stdout: boost-program-options-1.75.0-13.el9.x86_64 2026-02-23T13:04:02.634 INFO:teuthology.orchestra.run.trial068.stdout: ceph-base-2:20.3.0-5439.ge371433f.el9.x86_64 2026-02-23T13:04:02.634 INFO:teuthology.orchestra.run.trial068.stdout: ceph-common-2:20.3.0-5439.ge371433f.el9.x86_64 2026-02-23T13:04:02.634 INFO:teuthology.orchestra.run.trial068.stdout: ceph-radosgw-2:20.3.0-5439.ge371433f.el9.x86_64 2026-02-23T13:04:02.634 INFO:teuthology.orchestra.run.trial068.stdout: ceph-selinux-2:20.3.0-5439.ge371433f.el9.x86_64 2026-02-23T13:04:02.634 INFO:teuthology.orchestra.run.trial068.stdout: gperftools-libs-2.9.1-3.el9.x86_64 2026-02-23T13:04:02.634 INFO:teuthology.orchestra.run.trial068.stdout: libarrow-9.0.0-15.el9.x86_64 2026-02-23T13:04:02.634 INFO:teuthology.orchestra.run.trial068.stdout: libarrow-doc-9.0.0-15.el9.noarch 2026-02-23T13:04:02.634 INFO:teuthology.orchestra.run.trial068.stdout: libcephfs2-2:20.3.0-5439.ge371433f.el9.x86_64 2026-02-23T13:04:02.634 INFO:teuthology.orchestra.run.trial068.stdout: libnbd-1.20.3-4.el9.x86_64 2026-02-23T13:04:02.635 INFO:teuthology.orchestra.run.trial068.stdout: liboath-2.6.12-1.el9.x86_64 2026-02-23T13:04:02.635 INFO:teuthology.orchestra.run.trial068.stdout: libpmemobj-1.12.1-1.el9.x86_64 2026-02-23T13:04:02.635 INFO:teuthology.orchestra.run.trial068.stdout: librabbitmq-0.11.0-7.el9.x86_64 2026-02-23T13:04:02.635 INFO:teuthology.orchestra.run.trial068.stdout: libradosstriper1-2:20.3.0-5439.ge371433f.el9.x86_64 2026-02-23T13:04:02.635 INFO:teuthology.orchestra.run.trial068.stdout: librdkafka-1.6.1-102.el9.x86_64 2026-02-23T13:04:02.635 INFO:teuthology.orchestra.run.trial068.stdout: librgw2-2:20.3.0-5439.ge371433f.el9.x86_64 2026-02-23T13:04:02.635 INFO:teuthology.orchestra.run.trial068.stdout: libunwind-1.6.2-1.el9.x86_64 2026-02-23T13:04:02.635 INFO:teuthology.orchestra.run.trial068.stdout: lttng-ust-2.12.0-6.el9.x86_64 2026-02-23T13:04:02.635 INFO:teuthology.orchestra.run.trial068.stdout: mailcap-2.1.49-5.el9.noarch 2026-02-23T13:04:02.635 INFO:teuthology.orchestra.run.trial068.stdout: parquet-libs-9.0.0-15.el9.x86_64 2026-02-23T13:04:02.635 INFO:teuthology.orchestra.run.trial068.stdout: pciutils-3.7.0-7.el9.x86_64 2026-02-23T13:04:02.635 INFO:teuthology.orchestra.run.trial068.stdout: python3-ceph-argparse-2:20.3.0-5439.ge371433f.el9.x86_64 2026-02-23T13:04:02.636 INFO:teuthology.orchestra.run.trial068.stdout: python3-ceph-common-2:20.3.0-5439.ge371433f.el9.x86_64 2026-02-23T13:04:02.636 INFO:teuthology.orchestra.run.trial068.stdout: python3-cephfs-2:20.3.0-5439.ge371433f.el9.x86_64 2026-02-23T13:04:02.636 INFO:teuthology.orchestra.run.trial068.stdout: python3-rados-2:20.3.0-5439.ge371433f.el9.x86_64 2026-02-23T13:04:02.636 INFO:teuthology.orchestra.run.trial068.stdout: python3-rbd-2:20.3.0-5439.ge371433f.el9.x86_64 2026-02-23T13:04:02.636 INFO:teuthology.orchestra.run.trial068.stdout: python3-rgw-2:20.3.0-5439.ge371433f.el9.x86_64 2026-02-23T13:04:02.636 INFO:teuthology.orchestra.run.trial068.stdout: qatlib-25.08.0-2.el9.x86_64 2026-02-23T13:04:02.636 INFO:teuthology.orchestra.run.trial068.stdout: qatlib-service-25.08.0-2.el9.x86_64 2026-02-23T13:04:02.636 INFO:teuthology.orchestra.run.trial068.stdout: qatzip-libs-1.3.1-1.el9.x86_64 2026-02-23T13:04:02.636 INFO:teuthology.orchestra.run.trial068.stdout: re2-1:20211101-20.el9.x86_64 2026-02-23T13:04:02.636 INFO:teuthology.orchestra.run.trial068.stdout: thrift-0.15.0-4.el9.x86_64 2026-02-23T13:04:02.636 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T13:04:02.636 INFO:teuthology.orchestra.run.trial068.stdout:Complete! 2026-02-23T13:04:02.673 DEBUG:teuthology.orchestra.run.trial068:> sudo yum -y install ceph-test 2026-02-23T13:04:02.986 INFO:teuthology.orchestra.run.trial068.stdout:Last metadata expiration check: 0:00:12 ago on Mon 23 Feb 2026 01:03:50 PM UTC. 2026-02-23T13:04:03.068 INFO:teuthology.orchestra.run.trial068.stdout:Dependencies resolved. 2026-02-23T13:04:03.069 INFO:teuthology.orchestra.run.trial068.stdout:================================================================================ 2026-02-23T13:04:03.069 INFO:teuthology.orchestra.run.trial068.stdout: Package Arch Version Repository Size 2026-02-23T13:04:03.069 INFO:teuthology.orchestra.run.trial068.stdout:================================================================================ 2026-02-23T13:04:03.069 INFO:teuthology.orchestra.run.trial068.stdout:Installing: 2026-02-23T13:04:03.069 INFO:teuthology.orchestra.run.trial068.stdout: ceph-test x86_64 2:20.3.0-5439.ge371433f.el9 ceph 126 M 2026-02-23T13:04:03.070 INFO:teuthology.orchestra.run.trial068.stdout:Installing dependencies: 2026-02-23T13:04:03.070 INFO:teuthology.orchestra.run.trial068.stdout: hwloc-libs x86_64 2.4.1-6.el9 baseos 2.1 M 2026-02-23T13:04:03.070 INFO:teuthology.orchestra.run.trial068.stdout: libasan8 x86_64 15.2.1-7.el9 appstream 525 k 2026-02-23T13:04:03.070 INFO:teuthology.orchestra.run.trial068.stdout: libatomic x86_64 11.5.0-14.el9 baseos 23 k 2026-02-23T13:04:03.070 INFO:teuthology.orchestra.run.trial068.stdout: libcephsqlite x86_64 2:20.3.0-5439.ge371433f.el9 ceph 153 k 2026-02-23T13:04:03.070 INFO:teuthology.orchestra.run.trial068.stdout: libubsan x86_64 11.5.0-14.el9 appstream 176 k 2026-02-23T13:04:03.070 INFO:teuthology.orchestra.run.trial068.stdout: libxslt x86_64 1.1.34-12.el9 appstream 233 k 2026-02-23T13:04:03.070 INFO:teuthology.orchestra.run.trial068.stdout: lksctp-tools x86_64 1.0.19-2.el9 baseos 94 k 2026-02-23T13:04:03.070 INFO:teuthology.orchestra.run.trial068.stdout: protobuf x86_64 3.14.0-17.el9 appstream 1.0 M 2026-02-23T13:04:03.070 INFO:teuthology.orchestra.run.trial068.stdout: socat x86_64 1.7.4.1-8.el9 appstream 303 k 2026-02-23T13:04:03.070 INFO:teuthology.orchestra.run.trial068.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2026-02-23T13:04:03.070 INFO:teuthology.orchestra.run.trial068.stdout: yaml-cpp x86_64 0.6.3-5.el9 epel 122 k 2026-02-23T13:04:03.071 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T13:04:03.071 INFO:teuthology.orchestra.run.trial068.stdout:Transaction Summary 2026-02-23T13:04:03.071 INFO:teuthology.orchestra.run.trial068.stdout:================================================================================ 2026-02-23T13:04:03.071 INFO:teuthology.orchestra.run.trial068.stdout:Install 12 Packages 2026-02-23T13:04:03.071 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T13:04:03.072 INFO:teuthology.orchestra.run.trial068.stdout:Total download size: 131 M 2026-02-23T13:04:03.072 INFO:teuthology.orchestra.run.trial068.stdout:Installed size: 977 M 2026-02-23T13:04:03.072 INFO:teuthology.orchestra.run.trial068.stdout:Downloading Packages: 2026-02-23T13:04:03.666 INFO:teuthology.orchestra.run.trial068.stdout:(1/12): hwloc-libs-2.4.1-6.el9.x86_64.rpm 28 MB/s | 2.1 MB 00:00 2026-02-23T13:04:03.680 INFO:teuthology.orchestra.run.trial068.stdout:(2/12): libcephsqlite-20.3.0-5439.ge371433f.el9 1.7 MB/s | 153 kB 00:00 2026-02-23T13:04:03.684 INFO:teuthology.orchestra.run.trial068.stdout:(3/12): libatomic-11.5.0-14.el9.x86_64.rpm 1.2 MB/s | 23 kB 00:00 2026-02-23T13:04:03.701 INFO:teuthology.orchestra.run.trial068.stdout:(4/12): lksctp-tools-1.0.19-2.el9.x86_64.rpm 4.3 MB/s | 94 kB 00:00 2026-02-23T13:04:04.489 INFO:teuthology.orchestra.run.trial068.stdout:(5/12): libubsan-11.5.0-14.el9.x86_64.rpm 223 kB/s | 176 kB 00:00 2026-02-23T13:04:04.782 INFO:teuthology.orchestra.run.trial068.stdout:(6/12): libasan8-15.2.1-7.el9.x86_64.rpm 478 kB/s | 525 kB 00:01 2026-02-23T13:04:04.881 INFO:teuthology.orchestra.run.trial068.stdout:(7/12): libxslt-1.1.34-12.el9.x86_64.rpm 595 kB/s | 233 kB 00:00 2026-02-23T13:04:05.023 INFO:teuthology.orchestra.run.trial068.stdout:(8/12): protobuf-3.14.0-17.el9.x86_64.rpm 4.2 MB/s | 1.0 MB 00:00 2026-02-23T13:04:05.184 INFO:teuthology.orchestra.run.trial068.stdout:(9/12): xmlstarlet-1.6.1-20.el9.x86_64.rpm 397 kB/s | 64 kB 00:00 2026-02-23T13:04:05.349 INFO:teuthology.orchestra.run.trial068.stdout:(10/12): socat-1.7.4.1-8.el9.x86_64.rpm 647 kB/s | 303 kB 00:00 2026-02-23T13:04:05.437 INFO:teuthology.orchestra.run.trial068.stdout:(11/12): yaml-cpp-0.6.3-5.el9.x86_64.rpm 481 kB/s | 122 kB 00:00 2026-02-23T13:04:06.182 INFO:teuthology.orchestra.run.trial068.stdout:(12/12): ceph-test-20.3.0-5439.ge371433f.el9.x8 49 MB/s | 126 MB 00:02 2026-02-23T13:04:06.185 INFO:teuthology.orchestra.run.trial068.stdout:-------------------------------------------------------------------------------- 2026-02-23T13:04:06.186 INFO:teuthology.orchestra.run.trial068.stdout:Total 42 MB/s | 131 MB 00:03 2026-02-23T13:04:06.273 INFO:teuthology.orchestra.run.trial068.stdout:Running transaction check 2026-02-23T13:04:06.284 INFO:teuthology.orchestra.run.trial068.stdout:Transaction check succeeded. 2026-02-23T13:04:06.284 INFO:teuthology.orchestra.run.trial068.stdout:Running transaction test 2026-02-23T13:04:06.709 INFO:teuthology.orchestra.run.trial068.stdout:Transaction test succeeded. 2026-02-23T13:04:06.709 INFO:teuthology.orchestra.run.trial068.stdout:Running transaction 2026-02-23T13:04:07.161 INFO:teuthology.orchestra.run.trial068.stdout: Preparing : 1/1 2026-02-23T13:04:07.170 INFO:teuthology.orchestra.run.trial068.stdout: Installing : yaml-cpp-0.6.3-5.el9.x86_64 1/12 2026-02-23T13:04:07.179 INFO:teuthology.orchestra.run.trial068.stdout: Installing : socat-1.7.4.1-8.el9.x86_64 2/12 2026-02-23T13:04:07.184 INFO:teuthology.orchestra.run.trial068.stdout: Installing : protobuf-3.14.0-17.el9.x86_64 3/12 2026-02-23T13:04:07.188 INFO:teuthology.orchestra.run.trial068.stdout: Installing : libxslt-1.1.34-12.el9.x86_64 4/12 2026-02-23T13:04:07.191 INFO:teuthology.orchestra.run.trial068.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 5/12 2026-02-23T13:04:07.195 INFO:teuthology.orchestra.run.trial068.stdout: Installing : libubsan-11.5.0-14.el9.x86_64 6/12 2026-02-23T13:04:07.196 INFO:teuthology.orchestra.run.trial068.stdout: Installing : libasan8-15.2.1-7.el9.x86_64 7/12 2026-02-23T13:04:07.214 INFO:teuthology.orchestra.run.trial068.stdout: Running scriptlet: libasan8-15.2.1-7.el9.x86_64 7/12 2026-02-23T13:04:07.216 INFO:teuthology.orchestra.run.trial068.stdout: Installing : lksctp-tools-1.0.19-2.el9.x86_64 8/12 2026-02-23T13:04:07.227 INFO:teuthology.orchestra.run.trial068.stdout: Installing : libatomic-11.5.0-14.el9.x86_64 9/12 2026-02-23T13:04:07.235 INFO:teuthology.orchestra.run.trial068.stdout: Installing : hwloc-libs-2.4.1-6.el9.x86_64 10/12 2026-02-23T13:04:07.236 INFO:teuthology.orchestra.run.trial068.stdout: Installing : libcephsqlite-2:20.3.0-5439.ge371433f.el9.x86_64 11/12 2026-02-23T13:04:08.461 INFO:teuthology.orchestra.run.trial185.stdout: Running scriptlet: libcephsqlite-2:20.3.0-5439.ge371433f.el9.x86_64 11/12 2026-02-23T13:04:08.479 INFO:teuthology.orchestra.run.trial185.stdout: Installing : ceph-test-2:20.3.0-5439.ge371433f.el9.x86_64 12/12 2026-02-23T13:04:09.266 INFO:teuthology.orchestra.run.trial185.stdout: Running scriptlet: ceph-test-2:20.3.0-5439.ge371433f.el9.x86_64 12/12 2026-02-23T13:04:09.266 INFO:teuthology.orchestra.run.trial185.stdout: Verifying : ceph-test-2:20.3.0-5439.ge371433f.el9.x86_64 1/12 2026-02-23T13:04:09.266 INFO:teuthology.orchestra.run.trial185.stdout: Verifying : libcephsqlite-2:20.3.0-5439.ge371433f.el9.x86_64 2/12 2026-02-23T13:04:09.266 INFO:teuthology.orchestra.run.trial185.stdout: Verifying : hwloc-libs-2.4.1-6.el9.x86_64 3/12 2026-02-23T13:04:09.266 INFO:teuthology.orchestra.run.trial185.stdout: Verifying : libatomic-11.5.0-14.el9.x86_64 4/12 2026-02-23T13:04:09.266 INFO:teuthology.orchestra.run.trial185.stdout: Verifying : lksctp-tools-1.0.19-2.el9.x86_64 5/12 2026-02-23T13:04:09.266 INFO:teuthology.orchestra.run.trial185.stdout: Verifying : libasan8-15.2.1-7.el9.x86_64 6/12 2026-02-23T13:04:09.266 INFO:teuthology.orchestra.run.trial185.stdout: Verifying : libubsan-11.5.0-14.el9.x86_64 7/12 2026-02-23T13:04:09.266 INFO:teuthology.orchestra.run.trial185.stdout: Verifying : libxslt-1.1.34-12.el9.x86_64 8/12 2026-02-23T13:04:09.266 INFO:teuthology.orchestra.run.trial185.stdout: Verifying : protobuf-3.14.0-17.el9.x86_64 9/12 2026-02-23T13:04:09.266 INFO:teuthology.orchestra.run.trial185.stdout: Verifying : socat-1.7.4.1-8.el9.x86_64 10/12 2026-02-23T13:04:09.266 INFO:teuthology.orchestra.run.trial185.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 11/12 2026-02-23T13:04:09.321 INFO:teuthology.orchestra.run.trial185.stdout: Verifying : yaml-cpp-0.6.3-5.el9.x86_64 12/12 2026-02-23T13:04:09.321 INFO:teuthology.orchestra.run.trial185.stdout: 2026-02-23T13:04:09.321 INFO:teuthology.orchestra.run.trial185.stdout:Installed: 2026-02-23T13:04:09.321 INFO:teuthology.orchestra.run.trial185.stdout: ceph-test-2:20.3.0-5439.ge371433f.el9.x86_64 2026-02-23T13:04:09.321 INFO:teuthology.orchestra.run.trial185.stdout: hwloc-libs-2.4.1-6.el9.x86_64 2026-02-23T13:04:09.321 INFO:teuthology.orchestra.run.trial185.stdout: libasan8-15.2.1-7.el9.x86_64 2026-02-23T13:04:09.321 INFO:teuthology.orchestra.run.trial185.stdout: libatomic-11.5.0-14.el9.x86_64 2026-02-23T13:04:09.321 INFO:teuthology.orchestra.run.trial185.stdout: libcephsqlite-2:20.3.0-5439.ge371433f.el9.x86_64 2026-02-23T13:04:09.321 INFO:teuthology.orchestra.run.trial185.stdout: libubsan-11.5.0-14.el9.x86_64 2026-02-23T13:04:09.321 INFO:teuthology.orchestra.run.trial185.stdout: libxslt-1.1.34-12.el9.x86_64 2026-02-23T13:04:09.321 INFO:teuthology.orchestra.run.trial185.stdout: lksctp-tools-1.0.19-2.el9.x86_64 2026-02-23T13:04:09.322 INFO:teuthology.orchestra.run.trial185.stdout: protobuf-3.14.0-17.el9.x86_64 2026-02-23T13:04:09.322 INFO:teuthology.orchestra.run.trial185.stdout: socat-1.7.4.1-8.el9.x86_64 2026-02-23T13:04:09.322 INFO:teuthology.orchestra.run.trial185.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2026-02-23T13:04:09.322 INFO:teuthology.orchestra.run.trial185.stdout: yaml-cpp-0.6.3-5.el9.x86_64 2026-02-23T13:04:09.322 INFO:teuthology.orchestra.run.trial185.stdout: 2026-02-23T13:04:09.322 INFO:teuthology.orchestra.run.trial185.stdout:Complete! 2026-02-23T13:04:09.362 DEBUG:teuthology.orchestra.run.trial185:> sudo yum -y install ceph 2026-02-23T13:04:09.660 INFO:teuthology.orchestra.run.trial185.stdout:Last metadata expiration check: 0:00:25 ago on Mon 23 Feb 2026 01:03:44 PM UTC. 2026-02-23T13:04:09.743 INFO:teuthology.orchestra.run.trial185.stdout:Dependencies resolved. 2026-02-23T13:04:09.744 INFO:teuthology.orchestra.run.trial185.stdout:================================================================================ 2026-02-23T13:04:09.744 INFO:teuthology.orchestra.run.trial185.stdout: Package Arch Version Repository Size 2026-02-23T13:04:09.745 INFO:teuthology.orchestra.run.trial185.stdout:================================================================================ 2026-02-23T13:04:09.745 INFO:teuthology.orchestra.run.trial185.stdout:Installing: 2026-02-23T13:04:09.745 INFO:teuthology.orchestra.run.trial185.stdout: ceph x86_64 2:20.3.0-5439.ge371433f.el9 ceph 6.5 k 2026-02-23T13:04:09.745 INFO:teuthology.orchestra.run.trial185.stdout:Installing dependencies: 2026-02-23T13:04:09.745 INFO:teuthology.orchestra.run.trial185.stdout: ceph-mds x86_64 2:20.3.0-5439.ge371433f.el9 ceph 2.0 M 2026-02-23T13:04:09.745 INFO:teuthology.orchestra.run.trial185.stdout: ceph-mgr x86_64 2:20.3.0-5439.ge371433f.el9 ceph 937 k 2026-02-23T13:04:09.745 INFO:teuthology.orchestra.run.trial185.stdout: ceph-mgr-modules-core noarch 2:20.3.0-5439.ge371433f.el9 ceph-noarch 298 k 2026-02-23T13:04:09.745 INFO:teuthology.orchestra.run.trial185.stdout: ceph-mon x86_64 2:20.3.0-5439.ge371433f.el9 ceph 4.9 M 2026-02-23T13:04:09.745 INFO:teuthology.orchestra.run.trial185.stdout: ceph-osd x86_64 2:20.3.0-5439.ge371433f.el9 ceph 164 k 2026-02-23T13:04:09.745 INFO:teuthology.orchestra.run.trial185.stdout: ceph-osd-classic x86_64 2:20.3.0-5439.ge371433f.el9 ceph 15 M 2026-02-23T13:04:09.745 INFO:teuthology.orchestra.run.trial185.stdout: ceph-osd-crimson x86_64 2:20.3.0-5439.ge371433f.el9 ceph 87 M 2026-02-23T13:04:09.746 INFO:teuthology.orchestra.run.trial185.stdout: ledmon-libs x86_64 1.1.0-3.el9 baseos 40 k 2026-02-23T13:04:09.746 INFO:teuthology.orchestra.run.trial185.stdout: libconfig x86_64 1.7.2-9.el9 baseos 72 k 2026-02-23T13:04:09.746 INFO:teuthology.orchestra.run.trial185.stdout: libstoragemgmt x86_64 1.10.1-1.el9 appstream 246 k 2026-02-23T13:04:09.746 INFO:teuthology.orchestra.run.trial185.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2026-02-23T13:04:09.746 INFO:teuthology.orchestra.run.trial185.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2026-02-23T13:04:09.746 INFO:teuthology.orchestra.run.trial185.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2026-02-23T13:04:09.746 INFO:teuthology.orchestra.run.trial185.stdout: python3-autocommand noarch 2.2.2-8.el9 epel 29 k 2026-02-23T13:04:09.746 INFO:teuthology.orchestra.run.trial185.stdout: python3-backports-tarfile noarch 1.2.0-1.el9 epel 60 k 2026-02-23T13:04:09.746 INFO:teuthology.orchestra.run.trial185.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2026-02-23T13:04:09.746 INFO:teuthology.orchestra.run.trial185.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2026-02-23T13:04:09.746 INFO:teuthology.orchestra.run.trial185.stdout: python3-cheroot noarch 10.0.1-4.el9 epel 173 k 2026-02-23T13:04:09.746 INFO:teuthology.orchestra.run.trial185.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2026-02-23T13:04:09.747 INFO:teuthology.orchestra.run.trial185.stdout: python3-cryptography x86_64 36.0.1-5.el9 baseos 1.2 M 2026-02-23T13:04:09.747 INFO:teuthology.orchestra.run.trial185.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2026-02-23T13:04:09.747 INFO:teuthology.orchestra.run.trial185.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2026-02-23T13:04:09.747 INFO:teuthology.orchestra.run.trial185.stdout: python3-jaraco-collections 2026-02-23T13:04:09.747 INFO:teuthology.orchestra.run.trial185.stdout: noarch 3.0.0-8.el9 epel 23 k 2026-02-23T13:04:09.747 INFO:teuthology.orchestra.run.trial185.stdout: python3-jaraco-context noarch 6.0.1-3.el9 epel 20 k 2026-02-23T13:04:09.747 INFO:teuthology.orchestra.run.trial185.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2026-02-23T13:04:09.747 INFO:teuthology.orchestra.run.trial185.stdout: python3-jaraco-text noarch 4.0.0-2.el9 epel 26 k 2026-02-23T13:04:09.747 INFO:teuthology.orchestra.run.trial185.stdout: python3-libstoragemgmt x86_64 1.10.1-1.el9 appstream 177 k 2026-02-23T13:04:09.747 INFO:teuthology.orchestra.run.trial185.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2026-02-23T13:04:09.747 INFO:teuthology.orchestra.run.trial185.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2026-02-23T13:04:09.747 INFO:teuthology.orchestra.run.trial185.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2026-02-23T13:04:09.747 INFO:teuthology.orchestra.run.trial185.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2026-02-23T13:04:09.748 INFO:teuthology.orchestra.run.trial185.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2026-02-23T13:04:09.748 INFO:teuthology.orchestra.run.trial185.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2026-02-23T13:04:09.748 INFO:teuthology.orchestra.run.trial185.stdout: python3-pyparsing noarch 2.4.7-9.el9 baseos 150 k 2026-02-23T13:04:09.748 INFO:teuthology.orchestra.run.trial185.stdout: python3-requests noarch 2.25.1-10.el9 baseos 126 k 2026-02-23T13:04:09.748 INFO:teuthology.orchestra.run.trial185.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2026-02-23T13:04:09.748 INFO:teuthology.orchestra.run.trial185.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2026-02-23T13:04:09.748 INFO:teuthology.orchestra.run.trial185.stdout: python3-urllib3 noarch 1.26.5-7.el9 baseos 218 k 2026-02-23T13:04:09.748 INFO:teuthology.orchestra.run.trial185.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2026-02-23T13:04:09.748 INFO:teuthology.orchestra.run.trial185.stdout: unzip x86_64 6.0-59.el9 baseos 182 k 2026-02-23T13:04:09.748 INFO:teuthology.orchestra.run.trial185.stdout: zip x86_64 3.0-35.el9 baseos 266 k 2026-02-23T13:04:09.748 INFO:teuthology.orchestra.run.trial185.stdout: 2026-02-23T13:04:09.748 INFO:teuthology.orchestra.run.trial185.stdout:Transaction Summary 2026-02-23T13:04:09.749 INFO:teuthology.orchestra.run.trial185.stdout:================================================================================ 2026-02-23T13:04:09.749 INFO:teuthology.orchestra.run.trial185.stdout:Install 42 Packages 2026-02-23T13:04:09.749 INFO:teuthology.orchestra.run.trial185.stdout: 2026-02-23T13:04:09.750 INFO:teuthology.orchestra.run.trial185.stdout:Total download size: 114 M 2026-02-23T13:04:09.750 INFO:teuthology.orchestra.run.trial185.stdout:Installed size: 961 M 2026-02-23T13:04:09.750 INFO:teuthology.orchestra.run.trial185.stdout:Downloading Packages: 2026-02-23T13:04:10.085 INFO:teuthology.orchestra.run.trial185.stdout:(1/42): ceph-20.3.0-5439.ge371433f.el9.x86_64.r 127 kB/s | 6.5 kB 00:00 2026-02-23T13:04:10.170 INFO:teuthology.orchestra.run.trial185.stdout:(2/42): ceph-mgr-20.3.0-5439.ge371433f.el9.x86_ 6.7 MB/s | 937 kB 00:00 2026-02-23T13:04:10.193 INFO:teuthology.orchestra.run.trial185.stdout:(3/42): ceph-mds-20.3.0-5439.ge371433f.el9.x86_ 13 MB/s | 2.0 MB 00:00 2026-02-23T13:04:10.198 INFO:teuthology.orchestra.run.trial185.stdout:(4/42): ceph-osd-20.3.0-5439.ge371433f.el9.x86_ 5.7 MB/s | 164 kB 00:00 2026-02-23T13:04:10.330 INFO:teuthology.orchestra.run.trial185.stdout:(5/42): ceph-mon-20.3.0-5439.ge371433f.el9.x86_ 20 MB/s | 4.9 MB 00:00 2026-02-23T13:04:10.377 INFO:teuthology.orchestra.run.trial185.stdout:(6/42): ceph-mgr-modules-core-20.3.0-5439.ge371 6.1 MB/s | 298 kB 00:00 2026-02-23T13:04:10.481 INFO:teuthology.orchestra.run.trial185.stdout:(7/42): ledmon-libs-1.1.0-3.el9.x86_64.rpm 390 kB/s | 40 kB 00:00 2026-02-23T13:04:10.540 INFO:teuthology.orchestra.run.trial185.stdout:(8/42): libconfig-1.7.2-9.el9.x86_64.rpm 1.2 MB/s | 72 kB 00:00 2026-02-23T13:04:10.612 INFO:teuthology.orchestra.run.trial185.stdout:(9/42): python3-cffi-1.14.5-5.el9.x86_64.rpm 3.4 MB/s | 253 kB 00:00 2026-02-23T13:04:10.702 INFO:teuthology.orchestra.run.trial185.stdout:(10/42): ceph-osd-classic-20.3.0-5439.ge371433f 29 MB/s | 15 MB 00:00 2026-02-23T13:04:10.705 INFO:teuthology.orchestra.run.trial185.stdout:(11/42): python3-cryptography-36.0.1-5.el9.x86_ 13 MB/s | 1.2 MB 00:00 2026-02-23T13:04:10.733 INFO:teuthology.orchestra.run.trial185.stdout:(12/42): python3-pycparser-2.20-6.el9.noarch.rp 4.8 MB/s | 135 kB 00:00 2026-02-23T13:04:10.761 INFO:teuthology.orchestra.run.trial185.stdout:(13/42): python3-pyparsing-2.4.7-9.el9.noarch.r 5.3 MB/s | 150 kB 00:00 2026-02-23T13:04:10.789 INFO:teuthology.orchestra.run.trial185.stdout:(14/42): python3-requests-2.25.1-10.el9.noarch. 4.4 MB/s | 126 kB 00:00 2026-02-23T13:04:10.817 INFO:teuthology.orchestra.run.trial185.stdout:(15/42): python3-urllib3-1.26.5-7.el9.noarch.rp 7.6 MB/s | 218 kB 00:00 2026-02-23T13:04:10.827 INFO:teuthology.orchestra.run.trial185.stdout:(16/42): python3-ply-3.11-14.el9.noarch.rpm 850 kB/s | 106 kB 00:00 2026-02-23T13:04:10.845 INFO:teuthology.orchestra.run.trial185.stdout:(17/42): unzip-6.0-59.el9.x86_64.rpm 6.3 MB/s | 182 kB 00:00 2026-02-23T13:04:10.897 INFO:teuthology.orchestra.run.trial185.stdout:(18/42): zip-3.0-35.el9.x86_64.rpm 3.7 MB/s | 266 kB 00:00 2026-02-23T13:04:11.531 INFO:teuthology.orchestra.run.trial185.stdout:(19/42): lua-5.4.4-4.el9.x86_64.rpm 297 kB/s | 188 kB 00:00 2026-02-23T13:04:11.574 INFO:teuthology.orchestra.run.trial185.stdout:(20/42): libstoragemgmt-1.10.1-1.el9.x86_64.rpm 338 kB/s | 246 kB 00:00 2026-02-23T13:04:11.737 INFO:teuthology.orchestra.run.trial185.stdout:(21/42): python3-packaging-20.9-5.el9.noarch.rp 475 kB/s | 77 kB 00:00 2026-02-23T13:04:11.770 INFO:teuthology.orchestra.run.trial185.stdout:(22/42): python3-libstoragemgmt-1.10.1-1.el9.x8 739 kB/s | 177 kB 00:00 2026-02-23T13:04:11.813 INFO:teuthology.orchestra.run.trial185.stdout:(23/42): lua-devel-5.4.4-4.el9.x86_64.rpm 519 kB/s | 22 kB 00:00 2026-02-23T13:04:11.821 INFO:teuthology.orchestra.run.trial185.stdout:(24/42): python3-toml-0.10.2-6.el9.noarch.rpm 492 kB/s | 42 kB 00:00 2026-02-23T13:04:11.844 INFO:teuthology.orchestra.run.trial185.stdout:(25/42): python3-autocommand-2.2.2-8.el9.noarch 1.3 MB/s | 29 kB 00:00 2026-02-23T13:04:11.858 INFO:teuthology.orchestra.run.trial185.stdout:(26/42): luarocks-3.9.2-5.el9.noarch.rpm 3.3 MB/s | 151 kB 00:00 2026-02-23T13:04:11.859 INFO:teuthology.orchestra.run.trial185.stdout:(27/42): python3-backports-tarfile-1.2.0-1.el9. 4.1 MB/s | 60 kB 00:00 2026-02-23T13:04:11.866 INFO:teuthology.orchestra.run.trial185.stdout:(28/42): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 5.1 MB/s | 43 kB 00:00 2026-02-23T13:04:11.875 INFO:teuthology.orchestra.run.trial185.stdout:(29/42): python3-cheroot-10.0.1-4.el9.noarch.rp 11 MB/s | 173 kB 00:00 2026-02-23T13:04:11.883 INFO:teuthology.orchestra.run.trial185.stdout:(30/42): python3-cherrypy-18.6.1-2.el9.noarch.r 21 MB/s | 358 kB 00:00 2026-02-23T13:04:11.883 INFO:teuthology.orchestra.run.trial185.stdout:(31/42): python3-jaraco-8.2.1-3.el9.noarch.rpm 1.3 MB/s | 11 kB 00:00 2026-02-23T13:04:11.891 INFO:teuthology.orchestra.run.trial185.stdout:(32/42): python3-jaraco-classes-3.2.1-5.el9.noa 2.2 MB/s | 18 kB 00:00 2026-02-23T13:04:11.891 INFO:teuthology.orchestra.run.trial185.stdout:(33/42): python3-jaraco-collections-3.0.0-8.el9 2.9 MB/s | 23 kB 00:00 2026-02-23T13:04:11.898 INFO:teuthology.orchestra.run.trial185.stdout:(34/42): python3-jaraco-context-6.0.1-3.el9.noa 2.6 MB/s | 20 kB 00:00 2026-02-23T13:04:11.899 INFO:teuthology.orchestra.run.trial185.stdout:(35/42): python3-jaraco-functools-3.5.0-2.el9.n 2.5 MB/s | 19 kB 00:00 2026-02-23T13:04:11.906 INFO:teuthology.orchestra.run.trial185.stdout:(36/42): python3-jaraco-text-4.0.0-2.el9.noarch 3.4 MB/s | 26 kB 00:00 2026-02-23T13:04:11.907 INFO:teuthology.orchestra.run.trial185.stdout:(37/42): python3-more-itertools-8.12.0-2.el9.no 9.1 MB/s | 79 kB 00:00 2026-02-23T13:04:11.914 INFO:teuthology.orchestra.run.trial185.stdout:(38/42): python3-portend-3.1.0-2.el9.noarch.rpm 2.1 MB/s | 16 kB 00:00 2026-02-23T13:04:11.916 INFO:teuthology.orchestra.run.trial185.stdout:(39/42): python3-pyOpenSSL-21.0.0-1.el9.noarch. 10 MB/s | 90 kB 00:00 2026-02-23T13:04:11.922 INFO:teuthology.orchestra.run.trial185.stdout:(40/42): python3-tempora-5.0.0-2.el9.noarch.rpm 4.5 MB/s | 36 kB 00:00 2026-02-23T13:04:11.924 INFO:teuthology.orchestra.run.trial185.stdout:(41/42): python3-zc-lockfile-2.0-10.el9.noarch. 2.4 MB/s | 20 kB 00:00 2026-02-23T13:04:12.120 INFO:teuthology.orchestra.run.trial185.stdout:(42/42): ceph-osd-crimson-20.3.0-5439.ge371433f 45 MB/s | 87 MB 00:01 2026-02-23T13:04:12.123 INFO:teuthology.orchestra.run.trial185.stdout:-------------------------------------------------------------------------------- 2026-02-23T13:04:12.123 INFO:teuthology.orchestra.run.trial185.stdout:Total 48 MB/s | 114 MB 00:02 2026-02-23T13:04:12.261 INFO:teuthology.orchestra.run.trial185.stdout:Running transaction check 2026-02-23T13:04:12.281 INFO:teuthology.orchestra.run.trial185.stdout:Transaction check succeeded. 2026-02-23T13:04:12.281 INFO:teuthology.orchestra.run.trial185.stdout:Running transaction test 2026-02-23T13:04:12.672 INFO:teuthology.orchestra.run.trial185.stdout:Transaction test succeeded. 2026-02-23T13:04:12.673 INFO:teuthology.orchestra.run.trial185.stdout:Running transaction 2026-02-23T13:04:13.106 INFO:teuthology.orchestra.run.trial185.stdout: Preparing : 1/1 2026-02-23T13:04:13.108 INFO:teuthology.orchestra.run.trial185.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/42 2026-02-23T13:04:13.113 INFO:teuthology.orchestra.run.trial185.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/42 2026-02-23T13:04:13.118 INFO:teuthology.orchestra.run.trial185.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/42 2026-02-23T13:04:13.119 INFO:teuthology.orchestra.run.trial185.stdout: Installing : unzip-6.0-59.el9.x86_64 4/42 2026-02-23T13:04:13.125 INFO:teuthology.orchestra.run.trial185.stdout: Installing : ledmon-libs-1.1.0-3.el9.x86_64 5/42 2026-02-23T13:04:13.149 INFO:teuthology.orchestra.run.trial185.stdout: Installing : zip-3.0-35.el9.x86_64 6/42 2026-02-23T13:04:13.153 INFO:teuthology.orchestra.run.trial185.stdout: Installing : luarocks-3.9.2-5.el9.noarch 7/42 2026-02-23T13:04:13.157 INFO:teuthology.orchestra.run.trial185.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 8/42 2026-02-23T13:04:13.162 INFO:teuthology.orchestra.run.trial185.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 9/42 2026-02-23T13:04:13.168 INFO:teuthology.orchestra.run.trial185.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 10/42 2026-02-23T13:04:13.172 INFO:teuthology.orchestra.run.trial185.stdout: Installing : python3-backports-tarfile-1.2.0-1.el9.noarch 11/42 2026-02-23T13:04:13.178 INFO:teuthology.orchestra.run.trial185.stdout: Installing : python3-jaraco-context-6.0.1-3.el9.noarch 12/42 2026-02-23T13:04:13.183 INFO:teuthology.orchestra.run.trial185.stdout: Installing : python3-autocommand-2.2.2-8.el9.noarch 13/42 2026-02-23T13:04:13.186 INFO:teuthology.orchestra.run.trial185.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 14/42 2026-02-23T13:04:13.192 INFO:teuthology.orchestra.run.trial185.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 15/42 2026-02-23T13:04:13.196 INFO:teuthology.orchestra.run.trial185.stdout: Installing : python3-jaraco-text-4.0.0-2.el9.noarch 16/42 2026-02-23T13:04:13.202 INFO:teuthology.orchestra.run.trial185.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 17/42 2026-02-23T13:04:13.206 INFO:teuthology.orchestra.run.trial185.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 18/42 2026-02-23T13:04:13.228 INFO:teuthology.orchestra.run.trial185.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 19/42 2026-02-23T13:04:13.241 INFO:teuthology.orchestra.run.trial185.stdout: Installing : python3-urllib3-1.26.5-7.el9.noarch 20/42 2026-02-23T13:04:13.246 INFO:teuthology.orchestra.run.trial185.stdout: Installing : python3-requests-2.25.1-10.el9.noarch 21/42 2026-02-23T13:04:13.255 INFO:teuthology.orchestra.run.trial185.stdout: Installing : python3-pyparsing-2.4.7-9.el9.noarch 22/42 2026-02-23T13:04:13.262 INFO:teuthology.orchestra.run.trial185.stdout: Installing : python3-packaging-20.9-5.el9.noarch 23/42 2026-02-23T13:04:13.275 INFO:teuthology.orchestra.run.trial185.stdout: Installing : python3-ply-3.11-14.el9.noarch 24/42 2026-02-23T13:04:13.290 INFO:teuthology.orchestra.run.trial185.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 25/42 2026-02-23T13:04:13.366 INFO:teuthology.orchestra.run.trial185.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 26/42 2026-02-23T13:04:13.374 INFO:teuthology.orchestra.run.trial185.stdout: Installing : python3-cryptography-36.0.1-5.el9.x86_64 27/42 2026-02-23T13:04:13.397 INFO:teuthology.orchestra.run.trial185.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 28/42 2026-02-23T13:04:13.426 INFO:teuthology.orchestra.run.trial185.stdout: Installing : python3-cheroot-10.0.1-4.el9.noarch 29/42 2026-02-23T13:04:13.433 INFO:teuthology.orchestra.run.trial185.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 30/42 2026-02-23T13:04:13.485 INFO:teuthology.orchestra.run.trial185.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 31/42 2026-02-23T13:04:13.525 INFO:teuthology.orchestra.run.trial185.stdout: Installing : ceph-mgr-modules-core-2:20.3.0-5439.ge371433f.el9. 32/42 2026-02-23T13:04:13.526 INFO:teuthology.orchestra.run.trial185.stdout: Installing : ceph-mgr-2:20.3.0-5439.ge371433f.el9.x86_64 33/42 2026-02-23T13:04:13.549 INFO:teuthology.orchestra.run.trial185.stdout: Running scriptlet: ceph-mgr-2:20.3.0-5439.ge371433f.el9.x86_64 33/42 2026-02-23T13:04:13.550 INFO:teuthology.orchestra.run.trial185.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-23T13:04:13.550 INFO:teuthology.orchestra.run.trial185.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2026-02-23T13:04:13.550 INFO:teuthology.orchestra.run.trial185.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2026-02-23T13:04:13.550 INFO:teuthology.orchestra.run.trial185.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2026-02-23T13:04:13.550 INFO:teuthology.orchestra.run.trial185.stdout: 2026-02-23T13:04:13.554 INFO:teuthology.orchestra.run.trial185.stdout: Installing : libconfig-1.7.2-9.el9.x86_64 34/42 2026-02-23T13:04:13.572 INFO:teuthology.orchestra.run.trial185.stdout: Running scriptlet: libstoragemgmt-1.10.1-1.el9.x86_64 35/42 2026-02-23T13:04:13.572 INFO:teuthology.orchestra.run.trial185.stdout:Creating group 'libstoragemgmt' with GID 991. 2026-02-23T13:04:13.572 INFO:teuthology.orchestra.run.trial185.stdout:Creating user 'libstoragemgmt' (daemon account for libstoragemgmt) with UID 991 and GID 991. 2026-02-23T13:04:13.572 INFO:teuthology.orchestra.run.trial185.stdout: 2026-02-23T13:04:13.580 INFO:teuthology.orchestra.run.trial185.stdout: Installing : libstoragemgmt-1.10.1-1.el9.x86_64 35/42 2026-02-23T13:04:13.604 INFO:teuthology.orchestra.run.trial185.stdout: Running scriptlet: libstoragemgmt-1.10.1-1.el9.x86_64 35/42 2026-02-23T13:04:13.604 INFO:teuthology.orchestra.run.trial185.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/libstoragemgmt.service → /usr/lib/systemd/system/libstoragemgmt.service. 2026-02-23T13:04:13.604 INFO:teuthology.orchestra.run.trial185.stdout: 2026-02-23T13:04:14.136 INFO:teuthology.orchestra.run.trial068.stdout: Running scriptlet: libcephsqlite-2:20.3.0-5439.ge371433f.el9.x86_64 11/12 2026-02-23T13:04:14.155 INFO:teuthology.orchestra.run.trial068.stdout: Installing : ceph-test-2:20.3.0-5439.ge371433f.el9.x86_64 12/12 2026-02-23T13:04:14.204 INFO:teuthology.orchestra.run.trial185.stdout: Installing : python3-libstoragemgmt-1.10.1-1.el9.x86_64 36/42 2026-02-23T13:04:14.934 INFO:teuthology.orchestra.run.trial068.stdout: Running scriptlet: ceph-test-2:20.3.0-5439.ge371433f.el9.x86_64 12/12 2026-02-23T13:04:14.934 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : ceph-test-2:20.3.0-5439.ge371433f.el9.x86_64 1/12 2026-02-23T13:04:14.934 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : libcephsqlite-2:20.3.0-5439.ge371433f.el9.x86_64 2/12 2026-02-23T13:04:14.934 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : hwloc-libs-2.4.1-6.el9.x86_64 3/12 2026-02-23T13:04:14.934 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : libatomic-11.5.0-14.el9.x86_64 4/12 2026-02-23T13:04:14.934 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : lksctp-tools-1.0.19-2.el9.x86_64 5/12 2026-02-23T13:04:14.934 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : libasan8-15.2.1-7.el9.x86_64 6/12 2026-02-23T13:04:14.934 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : libubsan-11.5.0-14.el9.x86_64 7/12 2026-02-23T13:04:14.934 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : libxslt-1.1.34-12.el9.x86_64 8/12 2026-02-23T13:04:14.935 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : protobuf-3.14.0-17.el9.x86_64 9/12 2026-02-23T13:04:14.935 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : socat-1.7.4.1-8.el9.x86_64 10/12 2026-02-23T13:04:14.935 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 11/12 2026-02-23T13:04:14.988 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : yaml-cpp-0.6.3-5.el9.x86_64 12/12 2026-02-23T13:04:14.989 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T13:04:14.989 INFO:teuthology.orchestra.run.trial068.stdout:Installed: 2026-02-23T13:04:14.989 INFO:teuthology.orchestra.run.trial068.stdout: ceph-test-2:20.3.0-5439.ge371433f.el9.x86_64 2026-02-23T13:04:14.989 INFO:teuthology.orchestra.run.trial068.stdout: hwloc-libs-2.4.1-6.el9.x86_64 2026-02-23T13:04:14.989 INFO:teuthology.orchestra.run.trial068.stdout: libasan8-15.2.1-7.el9.x86_64 2026-02-23T13:04:14.989 INFO:teuthology.orchestra.run.trial068.stdout: libatomic-11.5.0-14.el9.x86_64 2026-02-23T13:04:14.989 INFO:teuthology.orchestra.run.trial068.stdout: libcephsqlite-2:20.3.0-5439.ge371433f.el9.x86_64 2026-02-23T13:04:14.989 INFO:teuthology.orchestra.run.trial068.stdout: libubsan-11.5.0-14.el9.x86_64 2026-02-23T13:04:14.989 INFO:teuthology.orchestra.run.trial068.stdout: libxslt-1.1.34-12.el9.x86_64 2026-02-23T13:04:14.990 INFO:teuthology.orchestra.run.trial068.stdout: lksctp-tools-1.0.19-2.el9.x86_64 2026-02-23T13:04:14.990 INFO:teuthology.orchestra.run.trial068.stdout: protobuf-3.14.0-17.el9.x86_64 2026-02-23T13:04:14.990 INFO:teuthology.orchestra.run.trial068.stdout: socat-1.7.4.1-8.el9.x86_64 2026-02-23T13:04:14.990 INFO:teuthology.orchestra.run.trial068.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2026-02-23T13:04:14.990 INFO:teuthology.orchestra.run.trial068.stdout: yaml-cpp-0.6.3-5.el9.x86_64 2026-02-23T13:04:14.990 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T13:04:14.990 INFO:teuthology.orchestra.run.trial068.stdout:Complete! 2026-02-23T13:04:15.029 DEBUG:teuthology.orchestra.run.trial068:> sudo yum -y install ceph 2026-02-23T13:04:15.330 INFO:teuthology.orchestra.run.trial068.stdout:Last metadata expiration check: 0:00:25 ago on Mon 23 Feb 2026 01:03:50 PM UTC. 2026-02-23T13:04:15.414 INFO:teuthology.orchestra.run.trial068.stdout:Dependencies resolved. 2026-02-23T13:04:15.415 INFO:teuthology.orchestra.run.trial068.stdout:================================================================================ 2026-02-23T13:04:15.416 INFO:teuthology.orchestra.run.trial068.stdout: Package Arch Version Repository Size 2026-02-23T13:04:15.416 INFO:teuthology.orchestra.run.trial068.stdout:================================================================================ 2026-02-23T13:04:15.416 INFO:teuthology.orchestra.run.trial068.stdout:Installing: 2026-02-23T13:04:15.416 INFO:teuthology.orchestra.run.trial068.stdout: ceph x86_64 2:20.3.0-5439.ge371433f.el9 ceph 6.5 k 2026-02-23T13:04:15.416 INFO:teuthology.orchestra.run.trial068.stdout:Installing dependencies: 2026-02-23T13:04:15.416 INFO:teuthology.orchestra.run.trial068.stdout: ceph-mds x86_64 2:20.3.0-5439.ge371433f.el9 ceph 2.0 M 2026-02-23T13:04:15.416 INFO:teuthology.orchestra.run.trial068.stdout: ceph-mgr x86_64 2:20.3.0-5439.ge371433f.el9 ceph 937 k 2026-02-23T13:04:15.416 INFO:teuthology.orchestra.run.trial068.stdout: ceph-mgr-modules-core noarch 2:20.3.0-5439.ge371433f.el9 ceph-noarch 298 k 2026-02-23T13:04:15.416 INFO:teuthology.orchestra.run.trial068.stdout: ceph-mon x86_64 2:20.3.0-5439.ge371433f.el9 ceph 4.9 M 2026-02-23T13:04:15.417 INFO:teuthology.orchestra.run.trial068.stdout: ceph-osd x86_64 2:20.3.0-5439.ge371433f.el9 ceph 164 k 2026-02-23T13:04:15.417 INFO:teuthology.orchestra.run.trial068.stdout: ceph-osd-classic x86_64 2:20.3.0-5439.ge371433f.el9 ceph 15 M 2026-02-23T13:04:15.417 INFO:teuthology.orchestra.run.trial068.stdout: ceph-osd-crimson x86_64 2:20.3.0-5439.ge371433f.el9 ceph 87 M 2026-02-23T13:04:15.417 INFO:teuthology.orchestra.run.trial068.stdout: ledmon-libs x86_64 1.1.0-3.el9 baseos 40 k 2026-02-23T13:04:15.417 INFO:teuthology.orchestra.run.trial068.stdout: libconfig x86_64 1.7.2-9.el9 baseos 72 k 2026-02-23T13:04:15.417 INFO:teuthology.orchestra.run.trial068.stdout: libstoragemgmt x86_64 1.10.1-1.el9 appstream 246 k 2026-02-23T13:04:15.417 INFO:teuthology.orchestra.run.trial068.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2026-02-23T13:04:15.417 INFO:teuthology.orchestra.run.trial068.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2026-02-23T13:04:15.417 INFO:teuthology.orchestra.run.trial068.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2026-02-23T13:04:15.418 INFO:teuthology.orchestra.run.trial068.stdout: python3-autocommand noarch 2.2.2-8.el9 epel 29 k 2026-02-23T13:04:15.418 INFO:teuthology.orchestra.run.trial068.stdout: python3-backports-tarfile noarch 1.2.0-1.el9 epel 60 k 2026-02-23T13:04:15.418 INFO:teuthology.orchestra.run.trial068.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2026-02-23T13:04:15.418 INFO:teuthology.orchestra.run.trial068.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2026-02-23T13:04:15.418 INFO:teuthology.orchestra.run.trial068.stdout: python3-cheroot noarch 10.0.1-4.el9 epel 173 k 2026-02-23T13:04:15.418 INFO:teuthology.orchestra.run.trial068.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2026-02-23T13:04:15.418 INFO:teuthology.orchestra.run.trial068.stdout: python3-cryptography x86_64 36.0.1-5.el9 baseos 1.2 M 2026-02-23T13:04:15.418 INFO:teuthology.orchestra.run.trial068.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2026-02-23T13:04:15.418 INFO:teuthology.orchestra.run.trial068.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2026-02-23T13:04:15.418 INFO:teuthology.orchestra.run.trial068.stdout: python3-jaraco-collections 2026-02-23T13:04:15.418 INFO:teuthology.orchestra.run.trial068.stdout: noarch 3.0.0-8.el9 epel 23 k 2026-02-23T13:04:15.419 INFO:teuthology.orchestra.run.trial068.stdout: python3-jaraco-context noarch 6.0.1-3.el9 epel 20 k 2026-02-23T13:04:15.419 INFO:teuthology.orchestra.run.trial068.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2026-02-23T13:04:15.419 INFO:teuthology.orchestra.run.trial068.stdout: python3-jaraco-text noarch 4.0.0-2.el9 epel 26 k 2026-02-23T13:04:15.419 INFO:teuthology.orchestra.run.trial068.stdout: python3-libstoragemgmt x86_64 1.10.1-1.el9 appstream 177 k 2026-02-23T13:04:15.419 INFO:teuthology.orchestra.run.trial068.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2026-02-23T13:04:15.419 INFO:teuthology.orchestra.run.trial068.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2026-02-23T13:04:15.419 INFO:teuthology.orchestra.run.trial068.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2026-02-23T13:04:15.419 INFO:teuthology.orchestra.run.trial068.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2026-02-23T13:04:15.419 INFO:teuthology.orchestra.run.trial068.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2026-02-23T13:04:15.419 INFO:teuthology.orchestra.run.trial068.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2026-02-23T13:04:15.419 INFO:teuthology.orchestra.run.trial068.stdout: python3-pyparsing noarch 2.4.7-9.el9 baseos 150 k 2026-02-23T13:04:15.420 INFO:teuthology.orchestra.run.trial068.stdout: python3-requests noarch 2.25.1-10.el9 baseos 126 k 2026-02-23T13:04:15.420 INFO:teuthology.orchestra.run.trial068.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2026-02-23T13:04:15.420 INFO:teuthology.orchestra.run.trial068.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2026-02-23T13:04:15.420 INFO:teuthology.orchestra.run.trial068.stdout: python3-urllib3 noarch 1.26.5-7.el9 baseos 218 k 2026-02-23T13:04:15.420 INFO:teuthology.orchestra.run.trial068.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2026-02-23T13:04:15.420 INFO:teuthology.orchestra.run.trial068.stdout: unzip x86_64 6.0-59.el9 baseos 182 k 2026-02-23T13:04:15.420 INFO:teuthology.orchestra.run.trial068.stdout: zip x86_64 3.0-35.el9 baseos 266 k 2026-02-23T13:04:15.420 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T13:04:15.420 INFO:teuthology.orchestra.run.trial068.stdout:Transaction Summary 2026-02-23T13:04:15.420 INFO:teuthology.orchestra.run.trial068.stdout:================================================================================ 2026-02-23T13:04:15.420 INFO:teuthology.orchestra.run.trial068.stdout:Install 42 Packages 2026-02-23T13:04:15.421 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T13:04:15.421 INFO:teuthology.orchestra.run.trial068.stdout:Total download size: 114 M 2026-02-23T13:04:15.422 INFO:teuthology.orchestra.run.trial068.stdout:Installed size: 961 M 2026-02-23T13:04:15.422 INFO:teuthology.orchestra.run.trial068.stdout:Downloading Packages: 2026-02-23T13:04:16.020 INFO:teuthology.orchestra.run.trial068.stdout:(1/42): ceph-20.3.0-5439.ge371433f.el9.x86_64.r 114 kB/s | 6.5 kB 00:00 2026-02-23T13:04:16.094 INFO:teuthology.orchestra.run.trial068.stdout:(2/42): ceph-mgr-20.3.0-5439.ge371433f.el9.x86_ 7.0 MB/s | 937 kB 00:00 2026-02-23T13:04:16.108 INFO:teuthology.orchestra.run.trial068.stdout:(3/42): ceph-mds-20.3.0-5439.ge371433f.el9.x86_ 14 MB/s | 2.0 MB 00:00 2026-02-23T13:04:16.121 INFO:teuthology.orchestra.run.trial068.stdout:(4/42): ceph-osd-20.3.0-5439.ge371433f.el9.x86_ 5.7 MB/s | 164 kB 00:00 2026-02-23T13:04:16.336 INFO:teuthology.orchestra.run.trial068.stdout:(5/42): ceph-mon-20.3.0-5439.ge371433f.el9.x86_ 15 MB/s | 4.9 MB 00:00 2026-02-23T13:04:16.390 INFO:teuthology.orchestra.run.trial068.stdout:(6/42): ceph-mgr-modules-core-20.3.0-5439.ge371 5.4 MB/s | 298 kB 00:00 2026-02-23T13:04:16.646 INFO:teuthology.orchestra.run.trial068.stdout:(7/42): ceph-osd-classic-20.3.0-5439.ge371433f. 27 MB/s | 15 MB 00:00 2026-02-23T13:04:16.803 INFO:teuthology.orchestra.run.trial068.stdout:(8/42): ledmon-libs-1.1.0-3.el9.x86_64.rpm 98 kB/s | 40 kB 00:00 2026-02-23T13:04:17.121 INFO:teuthology.orchestra.run.trial068.stdout:(9/42): python3-cffi-1.14.5-5.el9.x86_64.rpm 796 kB/s | 253 kB 00:00 2026-02-23T13:04:17.125 INFO:teuthology.orchestra.run.trial068.stdout:(10/42): libconfig-1.7.2-9.el9.x86_64.rpm 150 kB/s | 72 kB 00:00 2026-02-23T13:04:17.290 INFO:teuthology.orchestra.run.trial068.stdout:(11/42): python3-ply-3.11-14.el9.noarch.rpm 647 kB/s | 106 kB 00:00 2026-02-23T13:04:17.531 INFO:teuthology.orchestra.run.trial068.stdout:(12/42): python3-pycparser-2.20-6.el9.noarch.rp 561 kB/s | 135 kB 00:00 2026-02-23T13:04:17.771 INFO:teuthology.orchestra.run.trial068.stdout:(13/42): python3-pyparsing-2.4.7-9.el9.noarch.r 628 kB/s | 150 kB 00:00 2026-02-23T13:04:17.933 INFO:teuthology.orchestra.run.trial068.stdout:(14/42): python3-requests-2.25.1-10.el9.noarch. 779 kB/s | 126 kB 00:00 2026-02-23T13:04:18.065 INFO:teuthology.orchestra.run.trial068.stdout:(15/42): ceph-osd-crimson-20.3.0-5439.ge371433f 45 MB/s | 87 MB 00:01 2026-02-23T13:04:18.251 INFO:teuthology.orchestra.run.trial068.stdout:(16/42): python3-urllib3-1.26.5-7.el9.noarch.rp 685 kB/s | 218 kB 00:00 2026-02-23T13:04:18.589 INFO:teuthology.orchestra.run.trial068.stdout:(17/42): zip-3.0-35.el9.x86_64.rpm 787 kB/s | 266 kB 00:00 2026-02-23T13:04:18.649 INFO:teuthology.orchestra.run.trial068.stdout:(18/42): python3-cryptography-36.0.1-5.el9.x86_ 835 kB/s | 1.2 MB 00:01 2026-02-23T13:04:18.754 INFO:teuthology.orchestra.run.trial068.stdout:(19/42): libstoragemgmt-1.10.1-1.el9.x86_64.rpm 1.5 MB/s | 246 kB 00:00 2026-02-23T13:04:18.791 INFO:teuthology.orchestra.run.trial068.stdout:(20/42): python3-libstoragemgmt-1.10.1-1.el9.x8 4.7 MB/s | 177 kB 00:00 2026-02-23T13:04:18.795 INFO:teuthology.orchestra.run.trial068.stdout:(21/42): lua-5.4.4-4.el9.x86_64.rpm 1.3 MB/s | 188 kB 00:00 2026-02-23T13:04:18.821 INFO:teuthology.orchestra.run.trial068.stdout:(22/42): python3-packaging-20.9-5.el9.noarch.rp 2.5 MB/s | 77 kB 00:00 2026-02-23T13:04:18.823 INFO:teuthology.orchestra.run.trial068.stdout:(23/42): python3-toml-0.10.2-6.el9.noarch.rpm 1.4 MB/s | 42 kB 00:00 2026-02-23T13:04:18.860 INFO:teuthology.orchestra.run.trial068.stdout:(24/42): lua-devel-5.4.4-4.el9.x86_64.rpm 582 kB/s | 22 kB 00:00 2026-02-23T13:04:18.938 INFO:teuthology.orchestra.run.trial068.stdout:(25/42): luarocks-3.9.2-5.el9.noarch.rpm 1.3 MB/s | 151 kB 00:00 2026-02-23T13:04:18.940 INFO:teuthology.orchestra.run.trial068.stdout:(26/42): python3-autocommand-2.2.2-8.el9.noarch 365 kB/s | 29 kB 00:00 2026-02-23T13:04:18.960 INFO:teuthology.orchestra.run.trial068.stdout:(27/42): python3-backports-tarfile-1.2.0-1.el9. 2.6 MB/s | 60 kB 00:00 2026-02-23T13:04:18.970 INFO:teuthology.orchestra.run.trial068.stdout:(28/42): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 1.4 MB/s | 43 kB 00:00 2026-02-23T13:04:19.002 INFO:teuthology.orchestra.run.trial068.stdout:(29/42): python3-cheroot-10.0.1-4.el9.noarch.rp 4.1 MB/s | 173 kB 00:00 2026-02-23T13:04:19.007 INFO:teuthology.orchestra.run.trial068.stdout:(30/42): unzip-6.0-59.el9.x86_64.rpm 193 kB/s | 182 kB 00:00 2026-02-23T13:04:19.023 INFO:teuthology.orchestra.run.trial068.stdout:(31/42): python3-jaraco-8.2.1-3.el9.noarch.rpm 491 kB/s | 11 kB 00:00 2026-02-23T13:04:19.044 INFO:teuthology.orchestra.run.trial068.stdout:(32/42): python3-jaraco-collections-3.0.0-8.el9 1.1 MB/s | 23 kB 00:00 2026-02-23T13:04:19.053 INFO:teuthology.orchestra.run.trial068.stdout:(33/42): python3-cherrypy-18.6.1-2.el9.noarch.r 4.2 MB/s | 358 kB 00:00 2026-02-23T13:04:19.065 INFO:teuthology.orchestra.run.trial068.stdout:(34/42): python3-jaraco-context-6.0.1-3.el9.noa 957 kB/s | 20 kB 00:00 2026-02-23T13:04:19.083 INFO:teuthology.orchestra.run.trial068.stdout:(35/42): python3-jaraco-functools-3.5.0-2.el9.n 663 kB/s | 19 kB 00:00 2026-02-23T13:04:19.087 INFO:teuthology.orchestra.run.trial068.stdout:(36/42): python3-jaraco-classes-3.2.1-5.el9.noa 222 kB/s | 18 kB 00:00 2026-02-23T13:04:19.091 INFO:teuthology.orchestra.run.trial068.stdout:(37/42): python3-jaraco-text-4.0.0-2.el9.noarch 1.0 MB/s | 26 kB 00:00 2026-02-23T13:04:19.112 INFO:teuthology.orchestra.run.trial068.stdout:(38/42): python3-pyOpenSSL-21.0.0-1.el9.noarch. 4.1 MB/s | 90 kB 00:00 2026-02-23T13:04:19.114 INFO:teuthology.orchestra.run.trial068.stdout:(39/42): python3-more-itertools-8.12.0-2.el9.no 2.5 MB/s | 79 kB 00:00 2026-02-23T13:04:19.116 INFO:teuthology.orchestra.run.trial068.stdout:(40/42): python3-portend-3.1.0-2.el9.noarch.rpm 565 kB/s | 16 kB 00:00 2026-02-23T13:04:19.133 INFO:teuthology.orchestra.run.trial068.stdout:(41/42): python3-tempora-5.0.0-2.el9.noarch.rpm 1.7 MB/s | 36 kB 00:00 2026-02-23T13:04:19.143 INFO:teuthology.orchestra.run.trial068.stdout:(42/42): python3-zc-lockfile-2.0-10.el9.noarch. 675 kB/s | 20 kB 00:00 2026-02-23T13:04:19.148 INFO:teuthology.orchestra.run.trial068.stdout:-------------------------------------------------------------------------------- 2026-02-23T13:04:19.148 INFO:teuthology.orchestra.run.trial068.stdout:Total 31 MB/s | 114 MB 00:03 2026-02-23T13:04:19.296 INFO:teuthology.orchestra.run.trial068.stdout:Running transaction check 2026-02-23T13:04:19.315 INFO:teuthology.orchestra.run.trial068.stdout:Transaction check succeeded. 2026-02-23T13:04:19.316 INFO:teuthology.orchestra.run.trial068.stdout:Running transaction test 2026-02-23T13:04:19.546 INFO:teuthology.orchestra.run.trial185.stdout: Installing : ceph-osd-classic-2:20.3.0-5439.ge371433f.el9.x86_6 37/42 2026-02-23T13:04:19.556 INFO:teuthology.orchestra.run.trial185.stdout: Installing : ceph-osd-crimson-2:20.3.0-5439.ge371433f.el9.x86_6 38/42 2026-02-23T13:04:19.557 INFO:teuthology.orchestra.run.trial185.stdout: Installing : ceph-osd-2:20.3.0-5439.ge371433f.el9.x86_64 39/42 2026-02-23T13:04:19.582 INFO:teuthology.orchestra.run.trial185.stdout: Running scriptlet: ceph-osd-2:20.3.0-5439.ge371433f.el9.x86_64 39/42 2026-02-23T13:04:19.582 INFO:teuthology.orchestra.run.trial185.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-23T13:04:19.583 INFO:teuthology.orchestra.run.trial185.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2026-02-23T13:04:19.583 INFO:teuthology.orchestra.run.trial185.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2026-02-23T13:04:19.583 INFO:teuthology.orchestra.run.trial185.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2026-02-23T13:04:19.583 INFO:teuthology.orchestra.run.trial185.stdout: 2026-02-23T13:04:19.708 INFO:teuthology.orchestra.run.trial068.stdout:Transaction test succeeded. 2026-02-23T13:04:19.709 INFO:teuthology.orchestra.run.trial068.stdout:Running transaction 2026-02-23T13:04:19.788 INFO:teuthology.orchestra.run.trial185.stdout: Installing : ceph-mon-2:20.3.0-5439.ge371433f.el9.x86_64 40/42 2026-02-23T13:04:19.809 INFO:teuthology.orchestra.run.trial185.stdout: Running scriptlet: ceph-mon-2:20.3.0-5439.ge371433f.el9.x86_64 40/42 2026-02-23T13:04:19.809 INFO:teuthology.orchestra.run.trial185.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-23T13:04:19.809 INFO:teuthology.orchestra.run.trial185.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2026-02-23T13:04:19.809 INFO:teuthology.orchestra.run.trial185.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2026-02-23T13:04:19.809 INFO:teuthology.orchestra.run.trial185.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2026-02-23T13:04:19.809 INFO:teuthology.orchestra.run.trial185.stdout: 2026-02-23T13:04:19.892 INFO:teuthology.orchestra.run.trial185.stdout: Installing : ceph-mds-2:20.3.0-5439.ge371433f.el9.x86_64 41/42 2026-02-23T13:04:19.913 INFO:teuthology.orchestra.run.trial185.stdout: Running scriptlet: ceph-mds-2:20.3.0-5439.ge371433f.el9.x86_64 41/42 2026-02-23T13:04:19.913 INFO:teuthology.orchestra.run.trial185.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-23T13:04:19.913 INFO:teuthology.orchestra.run.trial185.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2026-02-23T13:04:19.913 INFO:teuthology.orchestra.run.trial185.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2026-02-23T13:04:19.913 INFO:teuthology.orchestra.run.trial185.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2026-02-23T13:04:19.913 INFO:teuthology.orchestra.run.trial185.stdout: 2026-02-23T13:04:19.914 INFO:teuthology.orchestra.run.trial185.stdout: Installing : ceph-2:20.3.0-5439.ge371433f.el9.x86_64 42/42 2026-02-23T13:04:19.921 INFO:teuthology.orchestra.run.trial185.stdout: Running scriptlet: ceph-osd-classic-2:20.3.0-5439.ge371433f.el9.x86_6 42/42 2026-02-23T13:04:19.944 INFO:teuthology.orchestra.run.trial185.stdout: Running scriptlet: ceph-osd-crimson-2:20.3.0-5439.ge371433f.el9.x86_6 42/42 2026-02-23T13:04:20.142 INFO:teuthology.orchestra.run.trial068.stdout: Preparing : 1/1 2026-02-23T13:04:20.144 INFO:teuthology.orchestra.run.trial068.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/42 2026-02-23T13:04:20.148 INFO:teuthology.orchestra.run.trial068.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/42 2026-02-23T13:04:20.153 INFO:teuthology.orchestra.run.trial068.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/42 2026-02-23T13:04:20.155 INFO:teuthology.orchestra.run.trial068.stdout: Installing : unzip-6.0-59.el9.x86_64 4/42 2026-02-23T13:04:20.161 INFO:teuthology.orchestra.run.trial068.stdout: Installing : ledmon-libs-1.1.0-3.el9.x86_64 5/42 2026-02-23T13:04:20.185 INFO:teuthology.orchestra.run.trial068.stdout: Installing : zip-3.0-35.el9.x86_64 6/42 2026-02-23T13:04:20.189 INFO:teuthology.orchestra.run.trial068.stdout: Installing : luarocks-3.9.2-5.el9.noarch 7/42 2026-02-23T13:04:20.194 INFO:teuthology.orchestra.run.trial068.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 8/42 2026-02-23T13:04:20.199 INFO:teuthology.orchestra.run.trial068.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 9/42 2026-02-23T13:04:20.205 INFO:teuthology.orchestra.run.trial068.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 10/42 2026-02-23T13:04:20.209 INFO:teuthology.orchestra.run.trial068.stdout: Installing : python3-backports-tarfile-1.2.0-1.el9.noarch 11/42 2026-02-23T13:04:20.215 INFO:teuthology.orchestra.run.trial068.stdout: Installing : python3-jaraco-context-6.0.1-3.el9.noarch 12/42 2026-02-23T13:04:20.220 INFO:teuthology.orchestra.run.trial068.stdout: Installing : python3-autocommand-2.2.2-8.el9.noarch 13/42 2026-02-23T13:04:20.223 INFO:teuthology.orchestra.run.trial068.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 14/42 2026-02-23T13:04:20.229 INFO:teuthology.orchestra.run.trial068.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 15/42 2026-02-23T13:04:20.232 INFO:teuthology.orchestra.run.trial068.stdout: Installing : python3-jaraco-text-4.0.0-2.el9.noarch 16/42 2026-02-23T13:04:20.239 INFO:teuthology.orchestra.run.trial068.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 17/42 2026-02-23T13:04:20.243 INFO:teuthology.orchestra.run.trial068.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 18/42 2026-02-23T13:04:20.265 INFO:teuthology.orchestra.run.trial068.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 19/42 2026-02-23T13:04:20.278 INFO:teuthology.orchestra.run.trial068.stdout: Installing : python3-urllib3-1.26.5-7.el9.noarch 20/42 2026-02-23T13:04:20.283 INFO:teuthology.orchestra.run.trial068.stdout: Installing : python3-requests-2.25.1-10.el9.noarch 21/42 2026-02-23T13:04:20.293 INFO:teuthology.orchestra.run.trial068.stdout: Installing : python3-pyparsing-2.4.7-9.el9.noarch 22/42 2026-02-23T13:04:20.299 INFO:teuthology.orchestra.run.trial068.stdout: Installing : python3-packaging-20.9-5.el9.noarch 23/42 2026-02-23T13:04:20.312 INFO:teuthology.orchestra.run.trial068.stdout: Installing : python3-ply-3.11-14.el9.noarch 24/42 2026-02-23T13:04:20.328 INFO:teuthology.orchestra.run.trial068.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 25/42 2026-02-23T13:04:20.403 INFO:teuthology.orchestra.run.trial068.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 26/42 2026-02-23T13:04:20.411 INFO:teuthology.orchestra.run.trial068.stdout: Installing : python3-cryptography-36.0.1-5.el9.x86_64 27/42 2026-02-23T13:04:20.434 INFO:teuthology.orchestra.run.trial068.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 28/42 2026-02-23T13:04:20.463 INFO:teuthology.orchestra.run.trial068.stdout: Installing : python3-cheroot-10.0.1-4.el9.noarch 29/42 2026-02-23T13:04:20.469 INFO:teuthology.orchestra.run.trial068.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 30/42 2026-02-23T13:04:20.521 INFO:teuthology.orchestra.run.trial068.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 31/42 2026-02-23T13:04:20.564 INFO:teuthology.orchestra.run.trial068.stdout: Installing : ceph-mgr-modules-core-2:20.3.0-5439.ge371433f.el9. 32/42 2026-02-23T13:04:20.565 INFO:teuthology.orchestra.run.trial068.stdout: Installing : ceph-mgr-2:20.3.0-5439.ge371433f.el9.x86_64 33/42 2026-02-23T13:04:20.588 INFO:teuthology.orchestra.run.trial068.stdout: Running scriptlet: ceph-mgr-2:20.3.0-5439.ge371433f.el9.x86_64 33/42 2026-02-23T13:04:20.588 INFO:teuthology.orchestra.run.trial068.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-23T13:04:20.589 INFO:teuthology.orchestra.run.trial068.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2026-02-23T13:04:20.589 INFO:teuthology.orchestra.run.trial068.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2026-02-23T13:04:20.589 INFO:teuthology.orchestra.run.trial068.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2026-02-23T13:04:20.589 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T13:04:20.593 INFO:teuthology.orchestra.run.trial068.stdout: Installing : libconfig-1.7.2-9.el9.x86_64 34/42 2026-02-23T13:04:20.610 INFO:teuthology.orchestra.run.trial068.stdout: Running scriptlet: libstoragemgmt-1.10.1-1.el9.x86_64 35/42 2026-02-23T13:04:20.610 INFO:teuthology.orchestra.run.trial068.stdout:Creating group 'libstoragemgmt' with GID 991. 2026-02-23T13:04:20.610 INFO:teuthology.orchestra.run.trial068.stdout:Creating user 'libstoragemgmt' (daemon account for libstoragemgmt) with UID 991 and GID 991. 2026-02-23T13:04:20.610 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T13:04:20.619 INFO:teuthology.orchestra.run.trial068.stdout: Installing : libstoragemgmt-1.10.1-1.el9.x86_64 35/42 2026-02-23T13:04:20.643 INFO:teuthology.orchestra.run.trial068.stdout: Running scriptlet: libstoragemgmt-1.10.1-1.el9.x86_64 35/42 2026-02-23T13:04:20.643 INFO:teuthology.orchestra.run.trial068.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/libstoragemgmt.service → /usr/lib/systemd/system/libstoragemgmt.service. 2026-02-23T13:04:20.644 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T13:04:20.862 INFO:teuthology.orchestra.run.trial185.stdout: Running scriptlet: ceph-2:20.3.0-5439.ge371433f.el9.x86_64 42/42 2026-02-23T13:04:20.862 INFO:teuthology.orchestra.run.trial185.stdout: Verifying : ceph-2:20.3.0-5439.ge371433f.el9.x86_64 1/42 2026-02-23T13:04:20.862 INFO:teuthology.orchestra.run.trial185.stdout: Verifying : ceph-mds-2:20.3.0-5439.ge371433f.el9.x86_64 2/42 2026-02-23T13:04:20.862 INFO:teuthology.orchestra.run.trial185.stdout: Verifying : ceph-mgr-2:20.3.0-5439.ge371433f.el9.x86_64 3/42 2026-02-23T13:04:20.862 INFO:teuthology.orchestra.run.trial185.stdout: Verifying : ceph-mon-2:20.3.0-5439.ge371433f.el9.x86_64 4/42 2026-02-23T13:04:20.862 INFO:teuthology.orchestra.run.trial185.stdout: Verifying : ceph-osd-2:20.3.0-5439.ge371433f.el9.x86_64 5/42 2026-02-23T13:04:20.862 INFO:teuthology.orchestra.run.trial185.stdout: Verifying : ceph-osd-classic-2:20.3.0-5439.ge371433f.el9.x86_6 6/42 2026-02-23T13:04:20.862 INFO:teuthology.orchestra.run.trial185.stdout: Verifying : ceph-osd-crimson-2:20.3.0-5439.ge371433f.el9.x86_6 7/42 2026-02-23T13:04:20.862 INFO:teuthology.orchestra.run.trial185.stdout: Verifying : ceph-mgr-modules-core-2:20.3.0-5439.ge371433f.el9. 8/42 2026-02-23T13:04:20.862 INFO:teuthology.orchestra.run.trial185.stdout: Verifying : ledmon-libs-1.1.0-3.el9.x86_64 9/42 2026-02-23T13:04:20.862 INFO:teuthology.orchestra.run.trial185.stdout: Verifying : libconfig-1.7.2-9.el9.x86_64 10/42 2026-02-23T13:04:20.862 INFO:teuthology.orchestra.run.trial185.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 11/42 2026-02-23T13:04:20.862 INFO:teuthology.orchestra.run.trial185.stdout: Verifying : python3-cryptography-36.0.1-5.el9.x86_64 12/42 2026-02-23T13:04:20.862 INFO:teuthology.orchestra.run.trial185.stdout: Verifying : python3-ply-3.11-14.el9.noarch 13/42 2026-02-23T13:04:20.862 INFO:teuthology.orchestra.run.trial185.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 14/42 2026-02-23T13:04:20.862 INFO:teuthology.orchestra.run.trial185.stdout: Verifying : python3-pyparsing-2.4.7-9.el9.noarch 15/42 2026-02-23T13:04:20.863 INFO:teuthology.orchestra.run.trial185.stdout: Verifying : python3-requests-2.25.1-10.el9.noarch 16/42 2026-02-23T13:04:20.863 INFO:teuthology.orchestra.run.trial185.stdout: Verifying : python3-urllib3-1.26.5-7.el9.noarch 17/42 2026-02-23T13:04:20.863 INFO:teuthology.orchestra.run.trial185.stdout: Verifying : unzip-6.0-59.el9.x86_64 18/42 2026-02-23T13:04:20.863 INFO:teuthology.orchestra.run.trial185.stdout: Verifying : zip-3.0-35.el9.x86_64 19/42 2026-02-23T13:04:20.863 INFO:teuthology.orchestra.run.trial185.stdout: Verifying : libstoragemgmt-1.10.1-1.el9.x86_64 20/42 2026-02-23T13:04:20.863 INFO:teuthology.orchestra.run.trial185.stdout: Verifying : lua-5.4.4-4.el9.x86_64 21/42 2026-02-23T13:04:20.863 INFO:teuthology.orchestra.run.trial185.stdout: Verifying : python3-libstoragemgmt-1.10.1-1.el9.x86_64 22/42 2026-02-23T13:04:20.863 INFO:teuthology.orchestra.run.trial185.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 23/42 2026-02-23T13:04:20.863 INFO:teuthology.orchestra.run.trial185.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 24/42 2026-02-23T13:04:20.863 INFO:teuthology.orchestra.run.trial185.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 25/42 2026-02-23T13:04:20.863 INFO:teuthology.orchestra.run.trial185.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 26/42 2026-02-23T13:04:20.863 INFO:teuthology.orchestra.run.trial185.stdout: Verifying : python3-autocommand-2.2.2-8.el9.noarch 27/42 2026-02-23T13:04:20.863 INFO:teuthology.orchestra.run.trial185.stdout: Verifying : python3-backports-tarfile-1.2.0-1.el9.noarch 28/42 2026-02-23T13:04:20.863 INFO:teuthology.orchestra.run.trial185.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 29/42 2026-02-23T13:04:20.863 INFO:teuthology.orchestra.run.trial185.stdout: Verifying : python3-cheroot-10.0.1-4.el9.noarch 30/42 2026-02-23T13:04:20.863 INFO:teuthology.orchestra.run.trial185.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 31/42 2026-02-23T13:04:20.863 INFO:teuthology.orchestra.run.trial185.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 32/42 2026-02-23T13:04:20.863 INFO:teuthology.orchestra.run.trial185.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 33/42 2026-02-23T13:04:20.863 INFO:teuthology.orchestra.run.trial185.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 34/42 2026-02-23T13:04:20.863 INFO:teuthology.orchestra.run.trial185.stdout: Verifying : python3-jaraco-context-6.0.1-3.el9.noarch 35/42 2026-02-23T13:04:20.863 INFO:teuthology.orchestra.run.trial185.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 36/42 2026-02-23T13:04:20.864 INFO:teuthology.orchestra.run.trial185.stdout: Verifying : python3-jaraco-text-4.0.0-2.el9.noarch 37/42 2026-02-23T13:04:20.864 INFO:teuthology.orchestra.run.trial185.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 38/42 2026-02-23T13:04:20.864 INFO:teuthology.orchestra.run.trial185.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 39/42 2026-02-23T13:04:20.864 INFO:teuthology.orchestra.run.trial185.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 40/42 2026-02-23T13:04:20.864 INFO:teuthology.orchestra.run.trial185.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 41/42 2026-02-23T13:04:20.922 INFO:teuthology.orchestra.run.trial185.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 42/42 2026-02-23T13:04:20.922 INFO:teuthology.orchestra.run.trial185.stdout: 2026-02-23T13:04:20.923 INFO:teuthology.orchestra.run.trial185.stdout:Installed: 2026-02-23T13:04:20.923 INFO:teuthology.orchestra.run.trial185.stdout: ceph-2:20.3.0-5439.ge371433f.el9.x86_64 2026-02-23T13:04:20.923 INFO:teuthology.orchestra.run.trial185.stdout: ceph-mds-2:20.3.0-5439.ge371433f.el9.x86_64 2026-02-23T13:04:20.923 INFO:teuthology.orchestra.run.trial185.stdout: ceph-mgr-2:20.3.0-5439.ge371433f.el9.x86_64 2026-02-23T13:04:20.923 INFO:teuthology.orchestra.run.trial185.stdout: ceph-mgr-modules-core-2:20.3.0-5439.ge371433f.el9.noarch 2026-02-23T13:04:20.923 INFO:teuthology.orchestra.run.trial185.stdout: ceph-mon-2:20.3.0-5439.ge371433f.el9.x86_64 2026-02-23T13:04:20.923 INFO:teuthology.orchestra.run.trial185.stdout: ceph-osd-2:20.3.0-5439.ge371433f.el9.x86_64 2026-02-23T13:04:20.923 INFO:teuthology.orchestra.run.trial185.stdout: ceph-osd-classic-2:20.3.0-5439.ge371433f.el9.x86_64 2026-02-23T13:04:20.923 INFO:teuthology.orchestra.run.trial185.stdout: ceph-osd-crimson-2:20.3.0-5439.ge371433f.el9.x86_64 2026-02-23T13:04:20.923 INFO:teuthology.orchestra.run.trial185.stdout: ledmon-libs-1.1.0-3.el9.x86_64 2026-02-23T13:04:20.924 INFO:teuthology.orchestra.run.trial185.stdout: libconfig-1.7.2-9.el9.x86_64 2026-02-23T13:04:20.924 INFO:teuthology.orchestra.run.trial185.stdout: libstoragemgmt-1.10.1-1.el9.x86_64 2026-02-23T13:04:20.924 INFO:teuthology.orchestra.run.trial185.stdout: lua-5.4.4-4.el9.x86_64 2026-02-23T13:04:20.924 INFO:teuthology.orchestra.run.trial185.stdout: lua-devel-5.4.4-4.el9.x86_64 2026-02-23T13:04:20.924 INFO:teuthology.orchestra.run.trial185.stdout: luarocks-3.9.2-5.el9.noarch 2026-02-23T13:04:20.924 INFO:teuthology.orchestra.run.trial185.stdout: python3-autocommand-2.2.2-8.el9.noarch 2026-02-23T13:04:20.924 INFO:teuthology.orchestra.run.trial185.stdout: python3-backports-tarfile-1.2.0-1.el9.noarch 2026-02-23T13:04:20.924 INFO:teuthology.orchestra.run.trial185.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2026-02-23T13:04:20.924 INFO:teuthology.orchestra.run.trial185.stdout: python3-cffi-1.14.5-5.el9.x86_64 2026-02-23T13:04:20.924 INFO:teuthology.orchestra.run.trial185.stdout: python3-cheroot-10.0.1-4.el9.noarch 2026-02-23T13:04:20.924 INFO:teuthology.orchestra.run.trial185.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2026-02-23T13:04:20.924 INFO:teuthology.orchestra.run.trial185.stdout: python3-cryptography-36.0.1-5.el9.x86_64 2026-02-23T13:04:20.925 INFO:teuthology.orchestra.run.trial185.stdout: python3-jaraco-8.2.1-3.el9.noarch 2026-02-23T13:04:20.925 INFO:teuthology.orchestra.run.trial185.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2026-02-23T13:04:20.925 INFO:teuthology.orchestra.run.trial185.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2026-02-23T13:04:20.925 INFO:teuthology.orchestra.run.trial185.stdout: python3-jaraco-context-6.0.1-3.el9.noarch 2026-02-23T13:04:20.925 INFO:teuthology.orchestra.run.trial185.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2026-02-23T13:04:20.925 INFO:teuthology.orchestra.run.trial185.stdout: python3-jaraco-text-4.0.0-2.el9.noarch 2026-02-23T13:04:20.925 INFO:teuthology.orchestra.run.trial185.stdout: python3-libstoragemgmt-1.10.1-1.el9.x86_64 2026-02-23T13:04:20.925 INFO:teuthology.orchestra.run.trial185.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2026-02-23T13:04:20.925 INFO:teuthology.orchestra.run.trial185.stdout: python3-packaging-20.9-5.el9.noarch 2026-02-23T13:04:20.925 INFO:teuthology.orchestra.run.trial185.stdout: python3-ply-3.11-14.el9.noarch 2026-02-23T13:04:20.925 INFO:teuthology.orchestra.run.trial185.stdout: python3-portend-3.1.0-2.el9.noarch 2026-02-23T13:04:20.925 INFO:teuthology.orchestra.run.trial185.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2026-02-23T13:04:20.926 INFO:teuthology.orchestra.run.trial185.stdout: python3-pycparser-2.20-6.el9.noarch 2026-02-23T13:04:20.926 INFO:teuthology.orchestra.run.trial185.stdout: python3-pyparsing-2.4.7-9.el9.noarch 2026-02-23T13:04:20.926 INFO:teuthology.orchestra.run.trial185.stdout: python3-requests-2.25.1-10.el9.noarch 2026-02-23T13:04:20.926 INFO:teuthology.orchestra.run.trial185.stdout: python3-tempora-5.0.0-2.el9.noarch 2026-02-23T13:04:20.926 INFO:teuthology.orchestra.run.trial185.stdout: python3-toml-0.10.2-6.el9.noarch 2026-02-23T13:04:20.926 INFO:teuthology.orchestra.run.trial185.stdout: python3-urllib3-1.26.5-7.el9.noarch 2026-02-23T13:04:20.926 INFO:teuthology.orchestra.run.trial185.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2026-02-23T13:04:20.926 INFO:teuthology.orchestra.run.trial185.stdout: unzip-6.0-59.el9.x86_64 2026-02-23T13:04:20.926 INFO:teuthology.orchestra.run.trial185.stdout: zip-3.0-35.el9.x86_64 2026-02-23T13:04:20.926 INFO:teuthology.orchestra.run.trial185.stdout: 2026-02-23T13:04:20.926 INFO:teuthology.orchestra.run.trial185.stdout:Complete! 2026-02-23T13:04:20.970 DEBUG:teuthology.orchestra.run.trial185:> sudo yum -y install ceph-base 2026-02-23T13:04:21.279 INFO:teuthology.orchestra.run.trial185.stdout:Last metadata expiration check: 0:00:37 ago on Mon 23 Feb 2026 01:03:44 PM UTC. 2026-02-23T13:04:21.284 INFO:teuthology.orchestra.run.trial068.stdout: Installing : python3-libstoragemgmt-1.10.1-1.el9.x86_64 36/42 2026-02-23T13:04:21.342 INFO:teuthology.orchestra.run.trial185.stdout:Package ceph-base-2:20.3.0-5439.ge371433f.el9.x86_64 is already installed. 2026-02-23T13:04:21.359 INFO:teuthology.orchestra.run.trial185.stdout:Dependencies resolved. 2026-02-23T13:04:21.360 INFO:teuthology.orchestra.run.trial185.stdout:Nothing to do. 2026-02-23T13:04:21.360 INFO:teuthology.orchestra.run.trial185.stdout:Complete! 2026-02-23T13:04:21.382 DEBUG:teuthology.orchestra.run.trial185:> sudo yum -y install cephadm 2026-02-23T13:04:21.686 INFO:teuthology.orchestra.run.trial185.stdout:Last metadata expiration check: 0:00:37 ago on Mon 23 Feb 2026 01:03:44 PM UTC. 2026-02-23T13:04:21.768 INFO:teuthology.orchestra.run.trial185.stdout:Dependencies resolved. 2026-02-23T13:04:21.769 INFO:teuthology.orchestra.run.trial185.stdout:================================================================================ 2026-02-23T13:04:21.769 INFO:teuthology.orchestra.run.trial185.stdout: Package Arch Version Repository Size 2026-02-23T13:04:21.769 INFO:teuthology.orchestra.run.trial185.stdout:================================================================================ 2026-02-23T13:04:21.769 INFO:teuthology.orchestra.run.trial185.stdout:Installing: 2026-02-23T13:04:21.769 INFO:teuthology.orchestra.run.trial185.stdout: cephadm noarch 2:20.3.0-5439.ge371433f.el9 ceph-noarch 1.0 M 2026-02-23T13:04:21.769 INFO:teuthology.orchestra.run.trial185.stdout: 2026-02-23T13:04:21.769 INFO:teuthology.orchestra.run.trial185.stdout:Transaction Summary 2026-02-23T13:04:21.769 INFO:teuthology.orchestra.run.trial185.stdout:================================================================================ 2026-02-23T13:04:21.769 INFO:teuthology.orchestra.run.trial185.stdout:Install 1 Package 2026-02-23T13:04:21.769 INFO:teuthology.orchestra.run.trial185.stdout: 2026-02-23T13:04:21.770 INFO:teuthology.orchestra.run.trial185.stdout:Total download size: 1.0 M 2026-02-23T13:04:21.770 INFO:teuthology.orchestra.run.trial185.stdout:Installed size: 1.0 M 2026-02-23T13:04:21.770 INFO:teuthology.orchestra.run.trial185.stdout:Downloading Packages: 2026-02-23T13:04:21.891 INFO:teuthology.orchestra.run.trial185.stdout:cephadm-20.3.0-5439.ge371433f.el9.noarch.rpm 8.4 MB/s | 1.0 MB 00:00 2026-02-23T13:04:21.891 INFO:teuthology.orchestra.run.trial185.stdout:-------------------------------------------------------------------------------- 2026-02-23T13:04:21.891 INFO:teuthology.orchestra.run.trial185.stdout:Total 8.3 MB/s | 1.0 MB 00:00 2026-02-23T13:04:21.891 INFO:teuthology.orchestra.run.trial185.stdout:Running transaction check 2026-02-23T13:04:21.893 INFO:teuthology.orchestra.run.trial185.stdout:Transaction check succeeded. 2026-02-23T13:04:21.894 INFO:teuthology.orchestra.run.trial185.stdout:Running transaction test 2026-02-23T13:04:21.897 INFO:teuthology.orchestra.run.trial185.stdout:Transaction test succeeded. 2026-02-23T13:04:21.897 INFO:teuthology.orchestra.run.trial185.stdout:Running transaction 2026-02-23T13:04:21.916 INFO:teuthology.orchestra.run.trial185.stdout: Preparing : 1/1 2026-02-23T13:04:21.976 INFO:teuthology.orchestra.run.trial185.stdout: Running scriptlet: cephadm-2:20.3.0-5439.ge371433f.el9.noarch 1/1 2026-02-23T13:04:21.990 INFO:teuthology.orchestra.run.trial185.stdout: Installing : cephadm-2:20.3.0-5439.ge371433f.el9.noarch 1/1 2026-02-23T13:04:22.056 INFO:teuthology.orchestra.run.trial185.stdout: Running scriptlet: cephadm-2:20.3.0-5439.ge371433f.el9.noarch 1/1 2026-02-23T13:04:22.110 INFO:teuthology.orchestra.run.trial185.stdout: Verifying : cephadm-2:20.3.0-5439.ge371433f.el9.noarch 1/1 2026-02-23T13:04:22.111 INFO:teuthology.orchestra.run.trial185.stdout: 2026-02-23T13:04:22.111 INFO:teuthology.orchestra.run.trial185.stdout:Installed: 2026-02-23T13:04:22.111 INFO:teuthology.orchestra.run.trial185.stdout: cephadm-2:20.3.0-5439.ge371433f.el9.noarch 2026-02-23T13:04:22.111 INFO:teuthology.orchestra.run.trial185.stdout: 2026-02-23T13:04:22.111 INFO:teuthology.orchestra.run.trial185.stdout:Complete! 2026-02-23T13:04:22.138 DEBUG:teuthology.orchestra.run.trial185:> sudo yum -y install ceph-immutable-object-cache 2026-02-23T13:04:22.450 INFO:teuthology.orchestra.run.trial185.stdout:Last metadata expiration check: 0:00:38 ago on Mon 23 Feb 2026 01:03:44 PM UTC. 2026-02-23T13:04:22.531 INFO:teuthology.orchestra.run.trial185.stdout:Dependencies resolved. 2026-02-23T13:04:22.532 INFO:teuthology.orchestra.run.trial185.stdout:================================================================================ 2026-02-23T13:04:22.533 INFO:teuthology.orchestra.run.trial185.stdout: Package Arch Version Repo Size 2026-02-23T13:04:22.533 INFO:teuthology.orchestra.run.trial185.stdout:================================================================================ 2026-02-23T13:04:22.533 INFO:teuthology.orchestra.run.trial185.stdout:Installing: 2026-02-23T13:04:22.533 INFO:teuthology.orchestra.run.trial185.stdout: ceph-immutable-object-cache x86_64 2:20.3.0-5439.ge371433f.el9 ceph 149 k 2026-02-23T13:04:22.533 INFO:teuthology.orchestra.run.trial185.stdout: 2026-02-23T13:04:22.533 INFO:teuthology.orchestra.run.trial185.stdout:Transaction Summary 2026-02-23T13:04:22.533 INFO:teuthology.orchestra.run.trial185.stdout:================================================================================ 2026-02-23T13:04:22.533 INFO:teuthology.orchestra.run.trial185.stdout:Install 1 Package 2026-02-23T13:04:22.533 INFO:teuthology.orchestra.run.trial185.stdout: 2026-02-23T13:04:22.533 INFO:teuthology.orchestra.run.trial185.stdout:Total download size: 149 k 2026-02-23T13:04:22.534 INFO:teuthology.orchestra.run.trial185.stdout:Installed size: 455 k 2026-02-23T13:04:22.534 INFO:teuthology.orchestra.run.trial185.stdout:Downloading Packages: 2026-02-23T13:04:22.619 INFO:teuthology.orchestra.run.trial185.stdout:ceph-immutable-object-cache-20.3.0-5439.ge37143 1.7 MB/s | 149 kB 00:00 2026-02-23T13:04:22.619 INFO:teuthology.orchestra.run.trial185.stdout:-------------------------------------------------------------------------------- 2026-02-23T13:04:22.619 INFO:teuthology.orchestra.run.trial185.stdout:Total 1.7 MB/s | 149 kB 00:00 2026-02-23T13:04:22.619 INFO:teuthology.orchestra.run.trial185.stdout:Running transaction check 2026-02-23T13:04:22.623 INFO:teuthology.orchestra.run.trial185.stdout:Transaction check succeeded. 2026-02-23T13:04:22.623 INFO:teuthology.orchestra.run.trial185.stdout:Running transaction test 2026-02-23T13:04:22.649 INFO:teuthology.orchestra.run.trial185.stdout:Transaction test succeeded. 2026-02-23T13:04:22.649 INFO:teuthology.orchestra.run.trial185.stdout:Running transaction 2026-02-23T13:04:22.698 INFO:teuthology.orchestra.run.trial185.stdout: Preparing : 1/1 2026-02-23T13:04:22.700 INFO:teuthology.orchestra.run.trial185.stdout: Installing : ceph-immutable-object-cache-2:20.3.0-5439.ge371433f. 1/1 2026-02-23T13:04:22.722 INFO:teuthology.orchestra.run.trial185.stdout: Running scriptlet: ceph-immutable-object-cache-2:20.3.0-5439.ge371433f. 1/1 2026-02-23T13:04:22.722 INFO:teuthology.orchestra.run.trial185.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-23T13:04:22.723 INFO:teuthology.orchestra.run.trial185.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2026-02-23T13:04:22.723 INFO:teuthology.orchestra.run.trial185.stdout: 2026-02-23T13:04:22.997 INFO:teuthology.orchestra.run.trial185.stdout: Verifying : ceph-immutable-object-cache-2:20.3.0-5439.ge371433f. 1/1 2026-02-23T13:04:22.997 INFO:teuthology.orchestra.run.trial185.stdout: 2026-02-23T13:04:22.997 INFO:teuthology.orchestra.run.trial185.stdout:Installed: 2026-02-23T13:04:22.997 INFO:teuthology.orchestra.run.trial185.stdout: ceph-immutable-object-cache-2:20.3.0-5439.ge371433f.el9.x86_64 2026-02-23T13:04:22.998 INFO:teuthology.orchestra.run.trial185.stdout: 2026-02-23T13:04:22.998 INFO:teuthology.orchestra.run.trial185.stdout:Complete! 2026-02-23T13:04:23.024 DEBUG:teuthology.orchestra.run.trial185:> sudo yum -y install ceph-mgr 2026-02-23T13:04:23.327 INFO:teuthology.orchestra.run.trial185.stdout:Last metadata expiration check: 0:00:39 ago on Mon 23 Feb 2026 01:03:44 PM UTC. 2026-02-23T13:04:23.391 INFO:teuthology.orchestra.run.trial185.stdout:Package ceph-mgr-2:20.3.0-5439.ge371433f.el9.x86_64 is already installed. 2026-02-23T13:04:23.408 INFO:teuthology.orchestra.run.trial185.stdout:Dependencies resolved. 2026-02-23T13:04:23.409 INFO:teuthology.orchestra.run.trial185.stdout:Nothing to do. 2026-02-23T13:04:23.409 INFO:teuthology.orchestra.run.trial185.stdout:Complete! 2026-02-23T13:04:23.430 DEBUG:teuthology.orchestra.run.trial185:> sudo yum -y install ceph-mgr-dashboard 2026-02-23T13:04:23.733 INFO:teuthology.orchestra.run.trial185.stdout:Last metadata expiration check: 0:00:39 ago on Mon 23 Feb 2026 01:03:44 PM UTC. 2026-02-23T13:04:23.815 INFO:teuthology.orchestra.run.trial185.stdout:Dependencies resolved. 2026-02-23T13:04:23.816 INFO:teuthology.orchestra.run.trial185.stdout:================================================================================ 2026-02-23T13:04:23.816 INFO:teuthology.orchestra.run.trial185.stdout: Package Arch Version Repository Size 2026-02-23T13:04:23.817 INFO:teuthology.orchestra.run.trial185.stdout:================================================================================ 2026-02-23T13:04:23.817 INFO:teuthology.orchestra.run.trial185.stdout:Installing: 2026-02-23T13:04:23.817 INFO:teuthology.orchestra.run.trial185.stdout: ceph-mgr-dashboard noarch 2:20.3.0-5439.ge371433f.el9 ceph-noarch 51 M 2026-02-23T13:04:23.817 INFO:teuthology.orchestra.run.trial185.stdout:Installing dependencies: 2026-02-23T13:04:23.817 INFO:teuthology.orchestra.run.trial185.stdout: abseil-cpp x86_64 20211102.0-4.el9 epel 551 k 2026-02-23T13:04:23.817 INFO:teuthology.orchestra.run.trial185.stdout: ceph-grafana-dashboards noarch 2:20.3.0-5439.ge371433f.el9 ceph-noarch 57 k 2026-02-23T13:04:23.817 INFO:teuthology.orchestra.run.trial185.stdout: ceph-prometheus-alerts noarch 2:20.3.0-5439.ge371433f.el9 ceph-noarch 17 k 2026-02-23T13:04:23.817 INFO:teuthology.orchestra.run.trial185.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2026-02-23T13:04:23.817 INFO:teuthology.orchestra.run.trial185.stdout: protobuf-compiler x86_64 3.14.0-17.el9 crb 862 k 2026-02-23T13:04:23.817 INFO:teuthology.orchestra.run.trial185.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2026-02-23T13:04:23.817 INFO:teuthology.orchestra.run.trial185.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2026-02-23T13:04:23.817 INFO:teuthology.orchestra.run.trial185.stdout: python3-jmespath noarch 1.0.1-1.el9 appstream 48 k 2026-02-23T13:04:23.818 INFO:teuthology.orchestra.run.trial185.stdout: python3-protobuf noarch 3.14.0-17.el9 appstream 267 k 2026-02-23T13:04:23.818 INFO:teuthology.orchestra.run.trial185.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2026-02-23T13:04:23.818 INFO:teuthology.orchestra.run.trial185.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2026-02-23T13:04:23.818 INFO:teuthology.orchestra.run.trial185.stdout: python3-xmltodict noarch 0.12.0-15.el9 epel 22 k 2026-02-23T13:04:23.818 INFO:teuthology.orchestra.run.trial185.stdout: 2026-02-23T13:04:23.818 INFO:teuthology.orchestra.run.trial185.stdout:Transaction Summary 2026-02-23T13:04:23.818 INFO:teuthology.orchestra.run.trial185.stdout:================================================================================ 2026-02-23T13:04:23.818 INFO:teuthology.orchestra.run.trial185.stdout:Install 13 Packages 2026-02-23T13:04:23.818 INFO:teuthology.orchestra.run.trial185.stdout: 2026-02-23T13:04:23.818 INFO:teuthology.orchestra.run.trial185.stdout:Total download size: 55 M 2026-02-23T13:04:23.818 INFO:teuthology.orchestra.run.trial185.stdout:Installed size: 262 M 2026-02-23T13:04:23.819 INFO:teuthology.orchestra.run.trial185.stdout:Downloading Packages: 2026-02-23T13:04:24.390 INFO:teuthology.orchestra.run.trial185.stdout:(1/13): ceph-prometheus-alerts-20.3.0-5439.ge37 251 kB/s | 17 kB 00:00 2026-02-23T13:04:24.394 INFO:teuthology.orchestra.run.trial185.stdout:(2/13): ceph-grafana-dashboards-20.3.0-5439.ge3 783 kB/s | 57 kB 00:00 2026-02-23T13:04:24.458 INFO:teuthology.orchestra.run.trial185.stdout:(3/13): python3-jmespath-1.0.1-1.el9.noarch.rpm 698 kB/s | 48 kB 00:00 2026-02-23T13:04:24.482 INFO:teuthology.orchestra.run.trial185.stdout:(4/13): python3-protobuf-3.14.0-17.el9.noarch.r 3.0 MB/s | 267 kB 00:00 2026-02-23T13:04:24.682 INFO:teuthology.orchestra.run.trial185.stdout:(5/13): abseil-cpp-20211102.0-4.el9.x86_64.rpm 2.7 MB/s | 551 kB 00:00 2026-02-23T13:04:24.711 INFO:teuthology.orchestra.run.trial185.stdout:(6/13): grpc-data-1.46.7-10.el9.noarch.rpm 678 kB/s | 19 kB 00:00 2026-02-23T13:04:24.827 INFO:teuthology.orchestra.run.trial185.stdout:(7/13): python3-grpcio-1.46.7-10.el9.x86_64.rpm 18 MB/s | 2.0 MB 00:00 2026-02-23T13:04:24.858 INFO:teuthology.orchestra.run.trial185.stdout:(8/13): python3-grpcio-tools-1.46.7-10.el9.x86_ 4.6 MB/s | 144 kB 00:00 2026-02-23T13:04:24.887 INFO:teuthology.orchestra.run.trial185.stdout:(9/13): python3-repoze-lru-0.7-16.el9.noarch.rp 1.0 MB/s | 31 kB 00:00 2026-02-23T13:04:24.919 INFO:teuthology.orchestra.run.trial185.stdout:(10/13): python3-routes-2.5.1-5.el9.noarch.rpm 5.8 MB/s | 188 kB 00:00 2026-02-23T13:04:24.948 INFO:teuthology.orchestra.run.trial185.stdout:(11/13): python3-xmltodict-0.12.0-15.el9.noarch 764 kB/s | 22 kB 00:00 2026-02-23T13:04:25.358 INFO:teuthology.orchestra.run.trial185.stdout:(12/13): ceph-mgr-dashboard-20.3.0-5439.ge37143 49 MB/s | 51 MB 00:01 2026-02-23T13:04:26.773 INFO:teuthology.orchestra.run.trial068.stdout: Installing : ceph-osd-classic-2:20.3.0-5439.ge371433f.el9.x86_6 37/42 2026-02-23T13:04:26.783 INFO:teuthology.orchestra.run.trial068.stdout: Installing : ceph-osd-crimson-2:20.3.0-5439.ge371433f.el9.x86_6 38/42 2026-02-23T13:04:26.784 INFO:teuthology.orchestra.run.trial068.stdout: Installing : ceph-osd-2:20.3.0-5439.ge371433f.el9.x86_64 39/42 2026-02-23T13:04:26.809 INFO:teuthology.orchestra.run.trial068.stdout: Running scriptlet: ceph-osd-2:20.3.0-5439.ge371433f.el9.x86_64 39/42 2026-02-23T13:04:26.810 INFO:teuthology.orchestra.run.trial068.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-23T13:04:26.810 INFO:teuthology.orchestra.run.trial068.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2026-02-23T13:04:26.810 INFO:teuthology.orchestra.run.trial068.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2026-02-23T13:04:26.810 INFO:teuthology.orchestra.run.trial068.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2026-02-23T13:04:26.810 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T13:04:27.025 INFO:teuthology.orchestra.run.trial068.stdout: Installing : ceph-mon-2:20.3.0-5439.ge371433f.el9.x86_64 40/42 2026-02-23T13:04:27.046 INFO:teuthology.orchestra.run.trial068.stdout: Running scriptlet: ceph-mon-2:20.3.0-5439.ge371433f.el9.x86_64 40/42 2026-02-23T13:04:27.046 INFO:teuthology.orchestra.run.trial068.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-23T13:04:27.047 INFO:teuthology.orchestra.run.trial068.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2026-02-23T13:04:27.047 INFO:teuthology.orchestra.run.trial068.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2026-02-23T13:04:27.047 INFO:teuthology.orchestra.run.trial068.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2026-02-23T13:04:27.047 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T13:04:27.070 INFO:teuthology.orchestra.run.trial185.stdout:(13/13): protobuf-compiler-3.14.0-17.el9.x86_64 330 kB/s | 862 kB 00:02 2026-02-23T13:04:27.075 INFO:teuthology.orchestra.run.trial185.stdout:-------------------------------------------------------------------------------- 2026-02-23T13:04:27.075 INFO:teuthology.orchestra.run.trial185.stdout:Total 17 MB/s | 55 MB 00:03 2026-02-23T13:04:27.101 INFO:teuthology.orchestra.run.trial185.stdout:Running transaction check 2026-02-23T13:04:27.112 INFO:teuthology.orchestra.run.trial185.stdout:Transaction check succeeded. 2026-02-23T13:04:27.112 INFO:teuthology.orchestra.run.trial185.stdout:Running transaction test 2026-02-23T13:04:27.134 INFO:teuthology.orchestra.run.trial068.stdout: Installing : ceph-mds-2:20.3.0-5439.ge371433f.el9.x86_64 41/42 2026-02-23T13:04:27.156 INFO:teuthology.orchestra.run.trial068.stdout: Running scriptlet: ceph-mds-2:20.3.0-5439.ge371433f.el9.x86_64 41/42 2026-02-23T13:04:27.156 INFO:teuthology.orchestra.run.trial068.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-23T13:04:27.156 INFO:teuthology.orchestra.run.trial068.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2026-02-23T13:04:27.156 INFO:teuthology.orchestra.run.trial068.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2026-02-23T13:04:27.156 INFO:teuthology.orchestra.run.trial068.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2026-02-23T13:04:27.156 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T13:04:27.157 INFO:teuthology.orchestra.run.trial068.stdout: Installing : ceph-2:20.3.0-5439.ge371433f.el9.x86_64 42/42 2026-02-23T13:04:27.164 INFO:teuthology.orchestra.run.trial068.stdout: Running scriptlet: ceph-osd-classic-2:20.3.0-5439.ge371433f.el9.x86_6 42/42 2026-02-23T13:04:27.187 INFO:teuthology.orchestra.run.trial068.stdout: Running scriptlet: ceph-osd-crimson-2:20.3.0-5439.ge371433f.el9.x86_6 42/42 2026-02-23T13:04:27.323 INFO:teuthology.orchestra.run.trial185.stdout:Transaction test succeeded. 2026-02-23T13:04:27.324 INFO:teuthology.orchestra.run.trial185.stdout:Running transaction 2026-02-23T13:04:27.551 INFO:teuthology.orchestra.run.trial185.stdout: Preparing : 1/1 2026-02-23T13:04:27.556 INFO:teuthology.orchestra.run.trial185.stdout: Installing : python3-xmltodict-0.12.0-15.el9.noarch 1/13 2026-02-23T13:04:27.568 INFO:teuthology.orchestra.run.trial185.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 2/13 2026-02-23T13:04:27.570 INFO:teuthology.orchestra.run.trial185.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 3/13 2026-02-23T13:04:27.597 INFO:teuthology.orchestra.run.trial185.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 4/13 2026-02-23T13:04:27.640 INFO:teuthology.orchestra.run.trial185.stdout: Installing : abseil-cpp-20211102.0-4.el9.x86_64 5/13 2026-02-23T13:04:27.648 INFO:teuthology.orchestra.run.trial185.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 6/13 2026-02-23T13:04:27.676 INFO:teuthology.orchestra.run.trial185.stdout: Installing : protobuf-compiler-3.14.0-17.el9.x86_64 7/13 2026-02-23T13:04:27.686 INFO:teuthology.orchestra.run.trial185.stdout: Installing : python3-protobuf-3.14.0-17.el9.noarch 8/13 2026-02-23T13:04:27.693 INFO:teuthology.orchestra.run.trial185.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/13 2026-02-23T13:04:27.696 INFO:teuthology.orchestra.run.trial185.stdout: Installing : python3-jmespath-1.0.1-1.el9.noarch 10/13 2026-02-23T13:04:27.705 INFO:teuthology.orchestra.run.trial185.stdout: Installing : ceph-prometheus-alerts-2:20.3.0-5439.ge371433f.el9 11/13 2026-02-23T13:04:28.161 INFO:teuthology.orchestra.run.trial068.stdout: Running scriptlet: ceph-2:20.3.0-5439.ge371433f.el9.x86_64 42/42 2026-02-23T13:04:28.161 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : ceph-2:20.3.0-5439.ge371433f.el9.x86_64 1/42 2026-02-23T13:04:28.161 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : ceph-mds-2:20.3.0-5439.ge371433f.el9.x86_64 2/42 2026-02-23T13:04:28.161 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : ceph-mgr-2:20.3.0-5439.ge371433f.el9.x86_64 3/42 2026-02-23T13:04:28.161 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : ceph-mon-2:20.3.0-5439.ge371433f.el9.x86_64 4/42 2026-02-23T13:04:28.161 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : ceph-osd-2:20.3.0-5439.ge371433f.el9.x86_64 5/42 2026-02-23T13:04:28.161 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : ceph-osd-classic-2:20.3.0-5439.ge371433f.el9.x86_6 6/42 2026-02-23T13:04:28.161 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : ceph-osd-crimson-2:20.3.0-5439.ge371433f.el9.x86_6 7/42 2026-02-23T13:04:28.162 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : ceph-mgr-modules-core-2:20.3.0-5439.ge371433f.el9. 8/42 2026-02-23T13:04:28.162 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : ledmon-libs-1.1.0-3.el9.x86_64 9/42 2026-02-23T13:04:28.162 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : libconfig-1.7.2-9.el9.x86_64 10/42 2026-02-23T13:04:28.162 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 11/42 2026-02-23T13:04:28.162 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : python3-cryptography-36.0.1-5.el9.x86_64 12/42 2026-02-23T13:04:28.162 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : python3-ply-3.11-14.el9.noarch 13/42 2026-02-23T13:04:28.162 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 14/42 2026-02-23T13:04:28.162 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : python3-pyparsing-2.4.7-9.el9.noarch 15/42 2026-02-23T13:04:28.162 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : python3-requests-2.25.1-10.el9.noarch 16/42 2026-02-23T13:04:28.162 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : python3-urllib3-1.26.5-7.el9.noarch 17/42 2026-02-23T13:04:28.162 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : unzip-6.0-59.el9.x86_64 18/42 2026-02-23T13:04:28.162 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : zip-3.0-35.el9.x86_64 19/42 2026-02-23T13:04:28.163 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : libstoragemgmt-1.10.1-1.el9.x86_64 20/42 2026-02-23T13:04:28.163 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : lua-5.4.4-4.el9.x86_64 21/42 2026-02-23T13:04:28.163 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : python3-libstoragemgmt-1.10.1-1.el9.x86_64 22/42 2026-02-23T13:04:28.163 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 23/42 2026-02-23T13:04:28.163 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 24/42 2026-02-23T13:04:28.163 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 25/42 2026-02-23T13:04:28.163 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 26/42 2026-02-23T13:04:28.163 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : python3-autocommand-2.2.2-8.el9.noarch 27/42 2026-02-23T13:04:28.163 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : python3-backports-tarfile-1.2.0-1.el9.noarch 28/42 2026-02-23T13:04:28.163 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 29/42 2026-02-23T13:04:28.163 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : python3-cheroot-10.0.1-4.el9.noarch 30/42 2026-02-23T13:04:28.163 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 31/42 2026-02-23T13:04:28.163 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 32/42 2026-02-23T13:04:28.164 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 33/42 2026-02-23T13:04:28.164 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 34/42 2026-02-23T13:04:28.164 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : python3-jaraco-context-6.0.1-3.el9.noarch 35/42 2026-02-23T13:04:28.164 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 36/42 2026-02-23T13:04:28.164 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : python3-jaraco-text-4.0.0-2.el9.noarch 37/42 2026-02-23T13:04:28.164 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 38/42 2026-02-23T13:04:28.164 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 39/42 2026-02-23T13:04:28.164 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 40/42 2026-02-23T13:04:28.164 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 41/42 2026-02-23T13:04:28.219 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 42/42 2026-02-23T13:04:28.219 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T13:04:28.219 INFO:teuthology.orchestra.run.trial068.stdout:Installed: 2026-02-23T13:04:28.219 INFO:teuthology.orchestra.run.trial068.stdout: ceph-2:20.3.0-5439.ge371433f.el9.x86_64 2026-02-23T13:04:28.219 INFO:teuthology.orchestra.run.trial068.stdout: ceph-mds-2:20.3.0-5439.ge371433f.el9.x86_64 2026-02-23T13:04:28.219 INFO:teuthology.orchestra.run.trial068.stdout: ceph-mgr-2:20.3.0-5439.ge371433f.el9.x86_64 2026-02-23T13:04:28.219 INFO:teuthology.orchestra.run.trial068.stdout: ceph-mgr-modules-core-2:20.3.0-5439.ge371433f.el9.noarch 2026-02-23T13:04:28.220 INFO:teuthology.orchestra.run.trial068.stdout: ceph-mon-2:20.3.0-5439.ge371433f.el9.x86_64 2026-02-23T13:04:28.220 INFO:teuthology.orchestra.run.trial068.stdout: ceph-osd-2:20.3.0-5439.ge371433f.el9.x86_64 2026-02-23T13:04:28.220 INFO:teuthology.orchestra.run.trial068.stdout: ceph-osd-classic-2:20.3.0-5439.ge371433f.el9.x86_64 2026-02-23T13:04:28.220 INFO:teuthology.orchestra.run.trial068.stdout: ceph-osd-crimson-2:20.3.0-5439.ge371433f.el9.x86_64 2026-02-23T13:04:28.220 INFO:teuthology.orchestra.run.trial068.stdout: ledmon-libs-1.1.0-3.el9.x86_64 2026-02-23T13:04:28.220 INFO:teuthology.orchestra.run.trial068.stdout: libconfig-1.7.2-9.el9.x86_64 2026-02-23T13:04:28.220 INFO:teuthology.orchestra.run.trial068.stdout: libstoragemgmt-1.10.1-1.el9.x86_64 2026-02-23T13:04:28.220 INFO:teuthology.orchestra.run.trial068.stdout: lua-5.4.4-4.el9.x86_64 2026-02-23T13:04:28.220 INFO:teuthology.orchestra.run.trial068.stdout: lua-devel-5.4.4-4.el9.x86_64 2026-02-23T13:04:28.220 INFO:teuthology.orchestra.run.trial068.stdout: luarocks-3.9.2-5.el9.noarch 2026-02-23T13:04:28.220 INFO:teuthology.orchestra.run.trial068.stdout: python3-autocommand-2.2.2-8.el9.noarch 2026-02-23T13:04:28.220 INFO:teuthology.orchestra.run.trial068.stdout: python3-backports-tarfile-1.2.0-1.el9.noarch 2026-02-23T13:04:28.220 INFO:teuthology.orchestra.run.trial068.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2026-02-23T13:04:28.221 INFO:teuthology.orchestra.run.trial068.stdout: python3-cffi-1.14.5-5.el9.x86_64 2026-02-23T13:04:28.221 INFO:teuthology.orchestra.run.trial068.stdout: python3-cheroot-10.0.1-4.el9.noarch 2026-02-23T13:04:28.221 INFO:teuthology.orchestra.run.trial068.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2026-02-23T13:04:28.221 INFO:teuthology.orchestra.run.trial068.stdout: python3-cryptography-36.0.1-5.el9.x86_64 2026-02-23T13:04:28.221 INFO:teuthology.orchestra.run.trial068.stdout: python3-jaraco-8.2.1-3.el9.noarch 2026-02-23T13:04:28.221 INFO:teuthology.orchestra.run.trial068.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2026-02-23T13:04:28.221 INFO:teuthology.orchestra.run.trial068.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2026-02-23T13:04:28.221 INFO:teuthology.orchestra.run.trial068.stdout: python3-jaraco-context-6.0.1-3.el9.noarch 2026-02-23T13:04:28.221 INFO:teuthology.orchestra.run.trial068.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2026-02-23T13:04:28.221 INFO:teuthology.orchestra.run.trial068.stdout: python3-jaraco-text-4.0.0-2.el9.noarch 2026-02-23T13:04:28.221 INFO:teuthology.orchestra.run.trial068.stdout: python3-libstoragemgmt-1.10.1-1.el9.x86_64 2026-02-23T13:04:28.221 INFO:teuthology.orchestra.run.trial068.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2026-02-23T13:04:28.222 INFO:teuthology.orchestra.run.trial068.stdout: python3-packaging-20.9-5.el9.noarch 2026-02-23T13:04:28.222 INFO:teuthology.orchestra.run.trial068.stdout: python3-ply-3.11-14.el9.noarch 2026-02-23T13:04:28.222 INFO:teuthology.orchestra.run.trial068.stdout: python3-portend-3.1.0-2.el9.noarch 2026-02-23T13:04:28.222 INFO:teuthology.orchestra.run.trial068.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2026-02-23T13:04:28.222 INFO:teuthology.orchestra.run.trial068.stdout: python3-pycparser-2.20-6.el9.noarch 2026-02-23T13:04:28.222 INFO:teuthology.orchestra.run.trial068.stdout: python3-pyparsing-2.4.7-9.el9.noarch 2026-02-23T13:04:28.222 INFO:teuthology.orchestra.run.trial068.stdout: python3-requests-2.25.1-10.el9.noarch 2026-02-23T13:04:28.222 INFO:teuthology.orchestra.run.trial068.stdout: python3-tempora-5.0.0-2.el9.noarch 2026-02-23T13:04:28.222 INFO:teuthology.orchestra.run.trial068.stdout: python3-toml-0.10.2-6.el9.noarch 2026-02-23T13:04:28.222 INFO:teuthology.orchestra.run.trial068.stdout: python3-urllib3-1.26.5-7.el9.noarch 2026-02-23T13:04:28.222 INFO:teuthology.orchestra.run.trial068.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2026-02-23T13:04:28.222 INFO:teuthology.orchestra.run.trial068.stdout: unzip-6.0-59.el9.x86_64 2026-02-23T13:04:28.223 INFO:teuthology.orchestra.run.trial068.stdout: zip-3.0-35.el9.x86_64 2026-02-23T13:04:28.223 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T13:04:28.223 INFO:teuthology.orchestra.run.trial068.stdout:Complete! 2026-02-23T13:04:28.263 DEBUG:teuthology.orchestra.run.trial068:> sudo yum -y install ceph-base 2026-02-23T13:04:28.572 INFO:teuthology.orchestra.run.trial068.stdout:Last metadata expiration check: 0:00:38 ago on Mon 23 Feb 2026 01:03:50 PM UTC. 2026-02-23T13:04:28.635 INFO:teuthology.orchestra.run.trial068.stdout:Package ceph-base-2:20.3.0-5439.ge371433f.el9.x86_64 is already installed. 2026-02-23T13:04:28.653 INFO:teuthology.orchestra.run.trial068.stdout:Dependencies resolved. 2026-02-23T13:04:28.653 INFO:teuthology.orchestra.run.trial068.stdout:Nothing to do. 2026-02-23T13:04:28.653 INFO:teuthology.orchestra.run.trial068.stdout:Complete! 2026-02-23T13:04:28.674 DEBUG:teuthology.orchestra.run.trial068:> sudo yum -y install cephadm 2026-02-23T13:04:28.982 INFO:teuthology.orchestra.run.trial068.stdout:Last metadata expiration check: 0:00:38 ago on Mon 23 Feb 2026 01:03:50 PM UTC. 2026-02-23T13:04:29.063 INFO:teuthology.orchestra.run.trial068.stdout:Dependencies resolved. 2026-02-23T13:04:29.064 INFO:teuthology.orchestra.run.trial068.stdout:================================================================================ 2026-02-23T13:04:29.064 INFO:teuthology.orchestra.run.trial068.stdout: Package Arch Version Repository Size 2026-02-23T13:04:29.064 INFO:teuthology.orchestra.run.trial068.stdout:================================================================================ 2026-02-23T13:04:29.064 INFO:teuthology.orchestra.run.trial068.stdout:Installing: 2026-02-23T13:04:29.064 INFO:teuthology.orchestra.run.trial068.stdout: cephadm noarch 2:20.3.0-5439.ge371433f.el9 ceph-noarch 1.0 M 2026-02-23T13:04:29.064 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T13:04:29.064 INFO:teuthology.orchestra.run.trial068.stdout:Transaction Summary 2026-02-23T13:04:29.064 INFO:teuthology.orchestra.run.trial068.stdout:================================================================================ 2026-02-23T13:04:29.064 INFO:teuthology.orchestra.run.trial068.stdout:Install 1 Package 2026-02-23T13:04:29.064 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T13:04:29.065 INFO:teuthology.orchestra.run.trial068.stdout:Total download size: 1.0 M 2026-02-23T13:04:29.065 INFO:teuthology.orchestra.run.trial068.stdout:Installed size: 1.0 M 2026-02-23T13:04:29.065 INFO:teuthology.orchestra.run.trial068.stdout:Downloading Packages: 2026-02-23T13:04:29.191 INFO:teuthology.orchestra.run.trial068.stdout:cephadm-20.3.0-5439.ge371433f.el9.noarch.rpm 8.0 MB/s | 1.0 MB 00:00 2026-02-23T13:04:29.191 INFO:teuthology.orchestra.run.trial068.stdout:-------------------------------------------------------------------------------- 2026-02-23T13:04:29.191 INFO:teuthology.orchestra.run.trial068.stdout:Total 8.0 MB/s | 1.0 MB 00:00 2026-02-23T13:04:29.191 INFO:teuthology.orchestra.run.trial068.stdout:Running transaction check 2026-02-23T13:04:29.194 INFO:teuthology.orchestra.run.trial068.stdout:Transaction check succeeded. 2026-02-23T13:04:29.194 INFO:teuthology.orchestra.run.trial068.stdout:Running transaction test 2026-02-23T13:04:29.198 INFO:teuthology.orchestra.run.trial068.stdout:Transaction test succeeded. 2026-02-23T13:04:29.198 INFO:teuthology.orchestra.run.trial068.stdout:Running transaction 2026-02-23T13:04:29.216 INFO:teuthology.orchestra.run.trial068.stdout: Preparing : 1/1 2026-02-23T13:04:29.277 INFO:teuthology.orchestra.run.trial068.stdout: Running scriptlet: cephadm-2:20.3.0-5439.ge371433f.el9.noarch 1/1 2026-02-23T13:04:29.291 INFO:teuthology.orchestra.run.trial068.stdout: Installing : cephadm-2:20.3.0-5439.ge371433f.el9.noarch 1/1 2026-02-23T13:04:29.364 INFO:teuthology.orchestra.run.trial068.stdout: Running scriptlet: cephadm-2:20.3.0-5439.ge371433f.el9.noarch 1/1 2026-02-23T13:04:29.419 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : cephadm-2:20.3.0-5439.ge371433f.el9.noarch 1/1 2026-02-23T13:04:29.419 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T13:04:29.419 INFO:teuthology.orchestra.run.trial068.stdout:Installed: 2026-02-23T13:04:29.419 INFO:teuthology.orchestra.run.trial068.stdout: cephadm-2:20.3.0-5439.ge371433f.el9.noarch 2026-02-23T13:04:29.419 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T13:04:29.419 INFO:teuthology.orchestra.run.trial068.stdout:Complete! 2026-02-23T13:04:29.447 DEBUG:teuthology.orchestra.run.trial068:> sudo yum -y install ceph-immutable-object-cache 2026-02-23T13:04:29.754 INFO:teuthology.orchestra.run.trial068.stdout:Last metadata expiration check: 0:00:39 ago on Mon 23 Feb 2026 01:03:50 PM UTC. 2026-02-23T13:04:29.836 INFO:teuthology.orchestra.run.trial068.stdout:Dependencies resolved. 2026-02-23T13:04:29.836 INFO:teuthology.orchestra.run.trial068.stdout:================================================================================ 2026-02-23T13:04:29.837 INFO:teuthology.orchestra.run.trial068.stdout: Package Arch Version Repo Size 2026-02-23T13:04:29.837 INFO:teuthology.orchestra.run.trial068.stdout:================================================================================ 2026-02-23T13:04:29.837 INFO:teuthology.orchestra.run.trial068.stdout:Installing: 2026-02-23T13:04:29.837 INFO:teuthology.orchestra.run.trial068.stdout: ceph-immutable-object-cache x86_64 2:20.3.0-5439.ge371433f.el9 ceph 149 k 2026-02-23T13:04:29.837 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T13:04:29.837 INFO:teuthology.orchestra.run.trial068.stdout:Transaction Summary 2026-02-23T13:04:29.837 INFO:teuthology.orchestra.run.trial068.stdout:================================================================================ 2026-02-23T13:04:29.837 INFO:teuthology.orchestra.run.trial068.stdout:Install 1 Package 2026-02-23T13:04:29.837 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T13:04:29.837 INFO:teuthology.orchestra.run.trial068.stdout:Total download size: 149 k 2026-02-23T13:04:29.837 INFO:teuthology.orchestra.run.trial068.stdout:Installed size: 455 k 2026-02-23T13:04:29.838 INFO:teuthology.orchestra.run.trial068.stdout:Downloading Packages: 2026-02-23T13:04:29.922 INFO:teuthology.orchestra.run.trial068.stdout:ceph-immutable-object-cache-20.3.0-5439.ge37143 1.7 MB/s | 149 kB 00:00 2026-02-23T13:04:29.923 INFO:teuthology.orchestra.run.trial068.stdout:-------------------------------------------------------------------------------- 2026-02-23T13:04:29.923 INFO:teuthology.orchestra.run.trial068.stdout:Total 1.7 MB/s | 149 kB 00:00 2026-02-23T13:04:29.923 INFO:teuthology.orchestra.run.trial068.stdout:Running transaction check 2026-02-23T13:04:29.927 INFO:teuthology.orchestra.run.trial068.stdout:Transaction check succeeded. 2026-02-23T13:04:29.927 INFO:teuthology.orchestra.run.trial068.stdout:Running transaction test 2026-02-23T13:04:29.955 INFO:teuthology.orchestra.run.trial068.stdout:Transaction test succeeded. 2026-02-23T13:04:29.955 INFO:teuthology.orchestra.run.trial068.stdout:Running transaction 2026-02-23T13:04:30.006 INFO:teuthology.orchestra.run.trial068.stdout: Preparing : 1/1 2026-02-23T13:04:30.007 INFO:teuthology.orchestra.run.trial068.stdout: Installing : ceph-immutable-object-cache-2:20.3.0-5439.ge371433f. 1/1 2026-02-23T13:04:30.030 INFO:teuthology.orchestra.run.trial068.stdout: Running scriptlet: ceph-immutable-object-cache-2:20.3.0-5439.ge371433f. 1/1 2026-02-23T13:04:30.030 INFO:teuthology.orchestra.run.trial068.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-23T13:04:30.030 INFO:teuthology.orchestra.run.trial068.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2026-02-23T13:04:30.030 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T13:04:30.307 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : ceph-immutable-object-cache-2:20.3.0-5439.ge371433f. 1/1 2026-02-23T13:04:30.308 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T13:04:30.308 INFO:teuthology.orchestra.run.trial068.stdout:Installed: 2026-02-23T13:04:30.308 INFO:teuthology.orchestra.run.trial068.stdout: ceph-immutable-object-cache-2:20.3.0-5439.ge371433f.el9.x86_64 2026-02-23T13:04:30.308 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T13:04:30.308 INFO:teuthology.orchestra.run.trial068.stdout:Complete! 2026-02-23T13:04:30.335 DEBUG:teuthology.orchestra.run.trial068:> sudo yum -y install ceph-mgr 2026-02-23T13:04:30.626 INFO:teuthology.orchestra.run.trial185.stdout: Installing : ceph-grafana-dashboards-2:20.3.0-5439.ge371433f.el 12/13 2026-02-23T13:04:30.633 INFO:teuthology.orchestra.run.trial185.stdout: Installing : ceph-mgr-dashboard-2:20.3.0-5439.ge371433f.el9.noa 13/13 2026-02-23T13:04:30.646 INFO:teuthology.orchestra.run.trial068.stdout:Last metadata expiration check: 0:00:40 ago on Mon 23 Feb 2026 01:03:50 PM UTC. 2026-02-23T13:04:30.710 INFO:teuthology.orchestra.run.trial068.stdout:Package ceph-mgr-2:20.3.0-5439.ge371433f.el9.x86_64 is already installed. 2026-02-23T13:04:30.728 INFO:teuthology.orchestra.run.trial068.stdout:Dependencies resolved. 2026-02-23T13:04:30.729 INFO:teuthology.orchestra.run.trial068.stdout:Nothing to do. 2026-02-23T13:04:30.729 INFO:teuthology.orchestra.run.trial068.stdout:Complete! 2026-02-23T13:04:30.750 DEBUG:teuthology.orchestra.run.trial068:> sudo yum -y install ceph-mgr-dashboard 2026-02-23T13:04:30.901 INFO:teuthology.orchestra.run.trial185.stdout: Running scriptlet: ceph-mgr-dashboard-2:20.3.0-5439.ge371433f.el9.noa 13/13 2026-02-23T13:04:30.901 INFO:teuthology.orchestra.run.trial185.stdout: Verifying : ceph-grafana-dashboards-2:20.3.0-5439.ge371433f.el 1/13 2026-02-23T13:04:30.902 INFO:teuthology.orchestra.run.trial185.stdout: Verifying : ceph-mgr-dashboard-2:20.3.0-5439.ge371433f.el9.noa 2/13 2026-02-23T13:04:30.902 INFO:teuthology.orchestra.run.trial185.stdout: Verifying : ceph-prometheus-alerts-2:20.3.0-5439.ge371433f.el9 3/13 2026-02-23T13:04:30.902 INFO:teuthology.orchestra.run.trial185.stdout: Verifying : python3-jmespath-1.0.1-1.el9.noarch 4/13 2026-02-23T13:04:30.902 INFO:teuthology.orchestra.run.trial185.stdout: Verifying : python3-protobuf-3.14.0-17.el9.noarch 5/13 2026-02-23T13:04:30.902 INFO:teuthology.orchestra.run.trial185.stdout: Verifying : protobuf-compiler-3.14.0-17.el9.x86_64 6/13 2026-02-23T13:04:30.902 INFO:teuthology.orchestra.run.trial185.stdout: Verifying : abseil-cpp-20211102.0-4.el9.x86_64 7/13 2026-02-23T13:04:30.902 INFO:teuthology.orchestra.run.trial185.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/13 2026-02-23T13:04:30.902 INFO:teuthology.orchestra.run.trial185.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/13 2026-02-23T13:04:30.902 INFO:teuthology.orchestra.run.trial185.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/13 2026-02-23T13:04:30.902 INFO:teuthology.orchestra.run.trial185.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/13 2026-02-23T13:04:30.902 INFO:teuthology.orchestra.run.trial185.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/13 2026-02-23T13:04:30.957 INFO:teuthology.orchestra.run.trial185.stdout: Verifying : python3-xmltodict-0.12.0-15.el9.noarch 13/13 2026-02-23T13:04:30.957 INFO:teuthology.orchestra.run.trial185.stdout: 2026-02-23T13:04:30.957 INFO:teuthology.orchestra.run.trial185.stdout:Installed: 2026-02-23T13:04:30.957 INFO:teuthology.orchestra.run.trial185.stdout: abseil-cpp-20211102.0-4.el9.x86_64 2026-02-23T13:04:30.957 INFO:teuthology.orchestra.run.trial185.stdout: ceph-grafana-dashboards-2:20.3.0-5439.ge371433f.el9.noarch 2026-02-23T13:04:30.957 INFO:teuthology.orchestra.run.trial185.stdout: ceph-mgr-dashboard-2:20.3.0-5439.ge371433f.el9.noarch 2026-02-23T13:04:30.957 INFO:teuthology.orchestra.run.trial185.stdout: ceph-prometheus-alerts-2:20.3.0-5439.ge371433f.el9.noarch 2026-02-23T13:04:30.957 INFO:teuthology.orchestra.run.trial185.stdout: grpc-data-1.46.7-10.el9.noarch 2026-02-23T13:04:30.957 INFO:teuthology.orchestra.run.trial185.stdout: protobuf-compiler-3.14.0-17.el9.x86_64 2026-02-23T13:04:30.957 INFO:teuthology.orchestra.run.trial185.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2026-02-23T13:04:30.957 INFO:teuthology.orchestra.run.trial185.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2026-02-23T13:04:30.957 INFO:teuthology.orchestra.run.trial185.stdout: python3-jmespath-1.0.1-1.el9.noarch 2026-02-23T13:04:30.957 INFO:teuthology.orchestra.run.trial185.stdout: python3-protobuf-3.14.0-17.el9.noarch 2026-02-23T13:04:30.957 INFO:teuthology.orchestra.run.trial185.stdout: python3-repoze-lru-0.7-16.el9.noarch 2026-02-23T13:04:30.957 INFO:teuthology.orchestra.run.trial185.stdout: python3-routes-2.5.1-5.el9.noarch 2026-02-23T13:04:30.957 INFO:teuthology.orchestra.run.trial185.stdout: python3-xmltodict-0.12.0-15.el9.noarch 2026-02-23T13:04:30.957 INFO:teuthology.orchestra.run.trial185.stdout: 2026-02-23T13:04:30.957 INFO:teuthology.orchestra.run.trial185.stdout:Complete! 2026-02-23T13:04:30.993 DEBUG:teuthology.orchestra.run.trial185:> sudo yum -y install ceph-mgr-diskprediction-local 2026-02-23T13:04:31.059 INFO:teuthology.orchestra.run.trial068.stdout:Last metadata expiration check: 0:00:41 ago on Mon 23 Feb 2026 01:03:50 PM UTC. 2026-02-23T13:04:31.142 INFO:teuthology.orchestra.run.trial068.stdout:Dependencies resolved. 2026-02-23T13:04:31.142 INFO:teuthology.orchestra.run.trial068.stdout:================================================================================ 2026-02-23T13:04:31.142 INFO:teuthology.orchestra.run.trial068.stdout: Package Arch Version Repository Size 2026-02-23T13:04:31.142 INFO:teuthology.orchestra.run.trial068.stdout:================================================================================ 2026-02-23T13:04:31.142 INFO:teuthology.orchestra.run.trial068.stdout:Installing: 2026-02-23T13:04:31.142 INFO:teuthology.orchestra.run.trial068.stdout: ceph-mgr-dashboard noarch 2:20.3.0-5439.ge371433f.el9 ceph-noarch 51 M 2026-02-23T13:04:31.143 INFO:teuthology.orchestra.run.trial068.stdout:Installing dependencies: 2026-02-23T13:04:31.143 INFO:teuthology.orchestra.run.trial068.stdout: abseil-cpp x86_64 20211102.0-4.el9 epel 551 k 2026-02-23T13:04:31.143 INFO:teuthology.orchestra.run.trial068.stdout: ceph-grafana-dashboards noarch 2:20.3.0-5439.ge371433f.el9 ceph-noarch 57 k 2026-02-23T13:04:31.143 INFO:teuthology.orchestra.run.trial068.stdout: ceph-prometheus-alerts noarch 2:20.3.0-5439.ge371433f.el9 ceph-noarch 17 k 2026-02-23T13:04:31.143 INFO:teuthology.orchestra.run.trial068.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2026-02-23T13:04:31.143 INFO:teuthology.orchestra.run.trial068.stdout: protobuf-compiler x86_64 3.14.0-17.el9 crb 862 k 2026-02-23T13:04:31.143 INFO:teuthology.orchestra.run.trial068.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2026-02-23T13:04:31.143 INFO:teuthology.orchestra.run.trial068.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2026-02-23T13:04:31.143 INFO:teuthology.orchestra.run.trial068.stdout: python3-jmespath noarch 1.0.1-1.el9 appstream 48 k 2026-02-23T13:04:31.143 INFO:teuthology.orchestra.run.trial068.stdout: python3-protobuf noarch 3.14.0-17.el9 appstream 267 k 2026-02-23T13:04:31.143 INFO:teuthology.orchestra.run.trial068.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2026-02-23T13:04:31.143 INFO:teuthology.orchestra.run.trial068.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2026-02-23T13:04:31.143 INFO:teuthology.orchestra.run.trial068.stdout: python3-xmltodict noarch 0.12.0-15.el9 epel 22 k 2026-02-23T13:04:31.143 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T13:04:31.143 INFO:teuthology.orchestra.run.trial068.stdout:Transaction Summary 2026-02-23T13:04:31.143 INFO:teuthology.orchestra.run.trial068.stdout:================================================================================ 2026-02-23T13:04:31.143 INFO:teuthology.orchestra.run.trial068.stdout:Install 13 Packages 2026-02-23T13:04:31.143 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T13:04:31.144 INFO:teuthology.orchestra.run.trial068.stdout:Total download size: 55 M 2026-02-23T13:04:31.144 INFO:teuthology.orchestra.run.trial068.stdout:Installed size: 262 M 2026-02-23T13:04:31.144 INFO:teuthology.orchestra.run.trial068.stdout:Downloading Packages: 2026-02-23T13:04:31.298 INFO:teuthology.orchestra.run.trial185.stdout:Last metadata expiration check: 0:00:47 ago on Mon 23 Feb 2026 01:03:44 PM UTC. 2026-02-23T13:04:31.381 INFO:teuthology.orchestra.run.trial185.stdout:Dependencies resolved. 2026-02-23T13:04:31.381 INFO:teuthology.orchestra.run.trial185.stdout:======================================================================================= 2026-02-23T13:04:31.381 INFO:teuthology.orchestra.run.trial185.stdout: Package Arch Version Repository Size 2026-02-23T13:04:31.381 INFO:teuthology.orchestra.run.trial185.stdout:======================================================================================= 2026-02-23T13:04:31.381 INFO:teuthology.orchestra.run.trial185.stdout:Installing: 2026-02-23T13:04:31.381 INFO:teuthology.orchestra.run.trial185.stdout: ceph-mgr-diskprediction-local noarch 2:20.3.0-5439.ge371433f.el9 ceph-noarch 7.4 M 2026-02-23T13:04:31.382 INFO:teuthology.orchestra.run.trial185.stdout:Installing dependencies: 2026-02-23T13:04:31.382 INFO:teuthology.orchestra.run.trial185.stdout: flexiblas x86_64 3.0.4-9.el9 appstream 30 k 2026-02-23T13:04:31.382 INFO:teuthology.orchestra.run.trial185.stdout: flexiblas-netlib x86_64 3.0.4-9.el9 appstream 3.0 M 2026-02-23T13:04:31.382 INFO:teuthology.orchestra.run.trial185.stdout: flexiblas-openblas-openmp x86_64 3.0.4-9.el9 appstream 15 k 2026-02-23T13:04:31.382 INFO:teuthology.orchestra.run.trial185.stdout: libgfortran x86_64 11.5.0-14.el9 baseos 794 k 2026-02-23T13:04:31.382 INFO:teuthology.orchestra.run.trial185.stdout: libquadmath x86_64 11.5.0-14.el9 baseos 184 k 2026-02-23T13:04:31.382 INFO:teuthology.orchestra.run.trial185.stdout: openblas x86_64 0.3.29-1.el9 appstream 42 k 2026-02-23T13:04:31.382 INFO:teuthology.orchestra.run.trial185.stdout: openblas-openmp x86_64 0.3.29-1.el9 appstream 5.3 M 2026-02-23T13:04:31.382 INFO:teuthology.orchestra.run.trial185.stdout: python3-devel x86_64 3.9.25-3.el9 appstream 244 k 2026-02-23T13:04:31.382 INFO:teuthology.orchestra.run.trial185.stdout: python3-numpy x86_64 1:1.23.5-2.el9 appstream 6.1 M 2026-02-23T13:04:31.382 INFO:teuthology.orchestra.run.trial185.stdout: python3-numpy-f2py x86_64 1:1.23.5-2.el9 appstream 442 k 2026-02-23T13:04:31.382 INFO:teuthology.orchestra.run.trial185.stdout: python3-scipy x86_64 1.9.3-2.el9 appstream 19 M 2026-02-23T13:04:31.382 INFO:teuthology.orchestra.run.trial185.stdout: 2026-02-23T13:04:31.382 INFO:teuthology.orchestra.run.trial185.stdout:Transaction Summary 2026-02-23T13:04:31.382 INFO:teuthology.orchestra.run.trial185.stdout:======================================================================================= 2026-02-23T13:04:31.382 INFO:teuthology.orchestra.run.trial185.stdout:Install 12 Packages 2026-02-23T13:04:31.382 INFO:teuthology.orchestra.run.trial185.stdout: 2026-02-23T13:04:31.383 INFO:teuthology.orchestra.run.trial185.stdout:Total download size: 43 M 2026-02-23T13:04:31.383 INFO:teuthology.orchestra.run.trial185.stdout:Installed size: 234 M 2026-02-23T13:04:31.383 INFO:teuthology.orchestra.run.trial185.stdout:Downloading Packages: 2026-02-23T13:04:31.647 INFO:teuthology.orchestra.run.trial068.stdout:(1/13): ceph-prometheus-alerts-20.3.0-5439.ge37 247 kB/s | 17 kB 00:00 2026-02-23T13:04:31.650 INFO:teuthology.orchestra.run.trial068.stdout:(2/13): ceph-grafana-dashboards-20.3.0-5439.ge3 785 kB/s | 57 kB 00:00 2026-02-23T13:04:31.741 INFO:teuthology.orchestra.run.trial068.stdout:(3/13): python3-jmespath-1.0.1-1.el9.noarch.rpm 509 kB/s | 48 kB 00:00 2026-02-23T13:04:31.768 INFO:teuthology.orchestra.run.trial185.stdout:(1/12): ceph-mgr-diskprediction-local-20.3.0-54 32 MB/s | 7.4 MB 00:00 2026-02-23T13:04:31.785 INFO:teuthology.orchestra.run.trial068.stdout:(4/13): python3-protobuf-3.14.0-17.el9.noarch.r 1.9 MB/s | 267 kB 00:00 2026-02-23T13:04:32.002 INFO:teuthology.orchestra.run.trial185.stdout:(2/12): libquadmath-11.5.0-14.el9.x86_64.rpm 393 kB/s | 184 kB 00:00 2026-02-23T13:04:32.159 INFO:teuthology.orchestra.run.trial185.stdout:(3/12): libgfortran-11.5.0-14.el9.x86_64.rpm 1.2 MB/s | 794 kB 00:00 2026-02-23T13:04:32.176 INFO:teuthology.orchestra.run.trial185.stdout:(4/12): flexiblas-3.0.4-9.el9.x86_64.rpm 72 kB/s | 30 kB 00:00 2026-02-23T13:04:32.228 INFO:teuthology.orchestra.run.trial068.stdout:(5/13): abseil-cpp-20211102.0-4.el9.x86_64.rpm 1.2 MB/s | 551 kB 00:00 2026-02-23T13:04:32.270 INFO:teuthology.orchestra.run.trial068.stdout:(6/13): grpc-data-1.46.7-10.el9.noarch.rpm 465 kB/s | 19 kB 00:00 2026-02-23T13:04:32.338 INFO:teuthology.orchestra.run.trial185.stdout:(5/12): openblas-0.3.29-1.el9.x86_64.rpm 261 kB/s | 42 kB 00:00 2026-02-23T13:04:32.404 INFO:teuthology.orchestra.run.trial068.stdout:(7/13): python3-grpcio-1.46.7-10.el9.x86_64.rpm 15 MB/s | 2.0 MB 00:00 2026-02-23T13:04:32.448 INFO:teuthology.orchestra.run.trial068.stdout:(8/13): python3-grpcio-tools-1.46.7-10.el9.x86_ 3.2 MB/s | 144 kB 00:00 2026-02-23T13:04:32.490 INFO:teuthology.orchestra.run.trial068.stdout:(9/13): python3-repoze-lru-0.7-16.el9.noarch.rp 729 kB/s | 31 kB 00:00 2026-02-23T13:04:32.536 INFO:teuthology.orchestra.run.trial068.stdout:(10/13): python3-routes-2.5.1-5.el9.noarch.rpm 4.1 MB/s | 188 kB 00:00 2026-02-23T13:04:32.558 INFO:teuthology.orchestra.run.trial185.stdout:(6/12): flexiblas-openblas-openmp-3.0.4-9.el9.x 37 kB/s | 15 kB 00:00 2026-02-23T13:04:32.578 INFO:teuthology.orchestra.run.trial068.stdout:(11/13): python3-xmltodict-0.12.0-15.el9.noarch 526 kB/s | 22 kB 00:00 2026-02-23T13:04:32.852 INFO:teuthology.orchestra.run.trial068.stdout:(12/13): ceph-mgr-dashboard-20.3.0-5439.ge37143 40 MB/s | 51 MB 00:01 2026-02-23T13:04:32.952 INFO:teuthology.orchestra.run.trial185.stdout:(7/12): python3-devel-3.9.25-3.el9.x86_64.rpm 621 kB/s | 244 kB 00:00 2026-02-23T13:04:33.319 INFO:teuthology.orchestra.run.trial068.stdout:(13/13): protobuf-compiler-3.14.0-17.el9.x86_64 546 kB/s | 862 kB 00:01 2026-02-23T13:04:33.324 INFO:teuthology.orchestra.run.trial068.stdout:-------------------------------------------------------------------------------- 2026-02-23T13:04:33.324 INFO:teuthology.orchestra.run.trial068.stdout:Total 25 MB/s | 55 MB 00:02 2026-02-23T13:04:33.351 INFO:teuthology.orchestra.run.trial068.stdout:Running transaction check 2026-02-23T13:04:33.362 INFO:teuthology.orchestra.run.trial068.stdout:Transaction check succeeded. 2026-02-23T13:04:33.362 INFO:teuthology.orchestra.run.trial068.stdout:Running transaction test 2026-02-23T13:04:33.575 INFO:teuthology.orchestra.run.trial068.stdout:Transaction test succeeded. 2026-02-23T13:04:33.576 INFO:teuthology.orchestra.run.trial068.stdout:Running transaction 2026-02-23T13:04:33.804 INFO:teuthology.orchestra.run.trial068.stdout: Preparing : 1/1 2026-02-23T13:04:33.809 INFO:teuthology.orchestra.run.trial068.stdout: Installing : python3-xmltodict-0.12.0-15.el9.noarch 1/13 2026-02-23T13:04:33.820 INFO:teuthology.orchestra.run.trial068.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 2/13 2026-02-23T13:04:33.822 INFO:teuthology.orchestra.run.trial068.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 3/13 2026-02-23T13:04:33.851 INFO:teuthology.orchestra.run.trial068.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 4/13 2026-02-23T13:04:33.893 INFO:teuthology.orchestra.run.trial068.stdout: Installing : abseil-cpp-20211102.0-4.el9.x86_64 5/13 2026-02-23T13:04:33.902 INFO:teuthology.orchestra.run.trial068.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 6/13 2026-02-23T13:04:33.929 INFO:teuthology.orchestra.run.trial068.stdout: Installing : protobuf-compiler-3.14.0-17.el9.x86_64 7/13 2026-02-23T13:04:33.941 INFO:teuthology.orchestra.run.trial068.stdout: Installing : python3-protobuf-3.14.0-17.el9.noarch 8/13 2026-02-23T13:04:33.948 INFO:teuthology.orchestra.run.trial068.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/13 2026-02-23T13:04:33.951 INFO:teuthology.orchestra.run.trial068.stdout: Installing : python3-jmespath-1.0.1-1.el9.noarch 10/13 2026-02-23T13:04:33.960 INFO:teuthology.orchestra.run.trial068.stdout: Installing : ceph-prometheus-alerts-2:20.3.0-5439.ge371433f.el9 11/13 2026-02-23T13:04:35.857 INFO:teuthology.orchestra.run.trial185.stdout:(8/12): flexiblas-netlib-3.0.4-9.el9.x86_64.rpm 794 kB/s | 3.0 MB 00:03 2026-02-23T13:04:36.333 INFO:teuthology.orchestra.run.trial185.stdout:(9/12): python3-numpy-f2py-1.23.5-2.el9.x86_64. 929 kB/s | 442 kB 00:00 2026-02-23T13:04:36.849 INFO:teuthology.orchestra.run.trial068.stdout: Installing : ceph-grafana-dashboards-2:20.3.0-5439.ge371433f.el 12/13 2026-02-23T13:04:36.855 INFO:teuthology.orchestra.run.trial068.stdout: Installing : ceph-mgr-dashboard-2:20.3.0-5439.ge371433f.el9.noa 13/13 2026-02-23T13:04:37.123 INFO:teuthology.orchestra.run.trial068.stdout: Running scriptlet: ceph-mgr-dashboard-2:20.3.0-5439.ge371433f.el9.noa 13/13 2026-02-23T13:04:37.124 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : ceph-grafana-dashboards-2:20.3.0-5439.ge371433f.el 1/13 2026-02-23T13:04:37.124 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : ceph-mgr-dashboard-2:20.3.0-5439.ge371433f.el9.noa 2/13 2026-02-23T13:04:37.124 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : ceph-prometheus-alerts-2:20.3.0-5439.ge371433f.el9 3/13 2026-02-23T13:04:37.124 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : python3-jmespath-1.0.1-1.el9.noarch 4/13 2026-02-23T13:04:37.124 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : python3-protobuf-3.14.0-17.el9.noarch 5/13 2026-02-23T13:04:37.124 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : protobuf-compiler-3.14.0-17.el9.x86_64 6/13 2026-02-23T13:04:37.125 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : abseil-cpp-20211102.0-4.el9.x86_64 7/13 2026-02-23T13:04:37.125 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/13 2026-02-23T13:04:37.125 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/13 2026-02-23T13:04:37.125 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/13 2026-02-23T13:04:37.125 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/13 2026-02-23T13:04:37.125 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/13 2026-02-23T13:04:37.180 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : python3-xmltodict-0.12.0-15.el9.noarch 13/13 2026-02-23T13:04:37.180 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T13:04:37.180 INFO:teuthology.orchestra.run.trial068.stdout:Installed: 2026-02-23T13:04:37.180 INFO:teuthology.orchestra.run.trial068.stdout: abseil-cpp-20211102.0-4.el9.x86_64 2026-02-23T13:04:37.181 INFO:teuthology.orchestra.run.trial068.stdout: ceph-grafana-dashboards-2:20.3.0-5439.ge371433f.el9.noarch 2026-02-23T13:04:37.181 INFO:teuthology.orchestra.run.trial068.stdout: ceph-mgr-dashboard-2:20.3.0-5439.ge371433f.el9.noarch 2026-02-23T13:04:37.181 INFO:teuthology.orchestra.run.trial068.stdout: ceph-prometheus-alerts-2:20.3.0-5439.ge371433f.el9.noarch 2026-02-23T13:04:37.181 INFO:teuthology.orchestra.run.trial068.stdout: grpc-data-1.46.7-10.el9.noarch 2026-02-23T13:04:37.181 INFO:teuthology.orchestra.run.trial068.stdout: protobuf-compiler-3.14.0-17.el9.x86_64 2026-02-23T13:04:37.181 INFO:teuthology.orchestra.run.trial068.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2026-02-23T13:04:37.181 INFO:teuthology.orchestra.run.trial068.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2026-02-23T13:04:37.181 INFO:teuthology.orchestra.run.trial068.stdout: python3-jmespath-1.0.1-1.el9.noarch 2026-02-23T13:04:37.181 INFO:teuthology.orchestra.run.trial068.stdout: python3-protobuf-3.14.0-17.el9.noarch 2026-02-23T13:04:37.181 INFO:teuthology.orchestra.run.trial068.stdout: python3-repoze-lru-0.7-16.el9.noarch 2026-02-23T13:04:37.181 INFO:teuthology.orchestra.run.trial068.stdout: python3-routes-2.5.1-5.el9.noarch 2026-02-23T13:04:37.182 INFO:teuthology.orchestra.run.trial068.stdout: python3-xmltodict-0.12.0-15.el9.noarch 2026-02-23T13:04:37.182 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T13:04:37.182 INFO:teuthology.orchestra.run.trial068.stdout:Complete! 2026-02-23T13:04:37.218 DEBUG:teuthology.orchestra.run.trial068:> sudo yum -y install ceph-mgr-diskprediction-local 2026-02-23T13:04:37.306 INFO:teuthology.orchestra.run.trial185.stdout:(10/12): openblas-openmp-0.3.29-1.el9.x86_64.rp 1.1 MB/s | 5.3 MB 00:04 2026-02-23T13:04:37.398 INFO:teuthology.orchestra.run.trial185.stdout:(11/12): python3-numpy-1.23.5-2.el9.x86_64.rpm 1.4 MB/s | 6.1 MB 00:04 2026-02-23T13:04:37.525 INFO:teuthology.orchestra.run.trial068.stdout:Last metadata expiration check: 0:00:47 ago on Mon 23 Feb 2026 01:03:50 PM UTC. 2026-02-23T13:04:37.554 INFO:teuthology.orchestra.run.trial185.stdout:(12/12): python3-scipy-1.9.3-2.el9.x86_64.rpm 16 MB/s | 19 MB 00:01 2026-02-23T13:04:37.556 INFO:teuthology.orchestra.run.trial185.stdout:-------------------------------------------------------------------------------- 2026-02-23T13:04:37.556 INFO:teuthology.orchestra.run.trial185.stdout:Total 6.9 MB/s | 43 MB 00:06 2026-02-23T13:04:37.608 INFO:teuthology.orchestra.run.trial068.stdout:Dependencies resolved. 2026-02-23T13:04:37.609 INFO:teuthology.orchestra.run.trial068.stdout:======================================================================================= 2026-02-23T13:04:37.609 INFO:teuthology.orchestra.run.trial068.stdout: Package Arch Version Repository Size 2026-02-23T13:04:37.609 INFO:teuthology.orchestra.run.trial068.stdout:======================================================================================= 2026-02-23T13:04:37.609 INFO:teuthology.orchestra.run.trial068.stdout:Installing: 2026-02-23T13:04:37.609 INFO:teuthology.orchestra.run.trial068.stdout: ceph-mgr-diskprediction-local noarch 2:20.3.0-5439.ge371433f.el9 ceph-noarch 7.4 M 2026-02-23T13:04:37.610 INFO:teuthology.orchestra.run.trial068.stdout:Installing dependencies: 2026-02-23T13:04:37.610 INFO:teuthology.orchestra.run.trial068.stdout: flexiblas x86_64 3.0.4-9.el9 appstream 30 k 2026-02-23T13:04:37.610 INFO:teuthology.orchestra.run.trial068.stdout: flexiblas-netlib x86_64 3.0.4-9.el9 appstream 3.0 M 2026-02-23T13:04:37.610 INFO:teuthology.orchestra.run.trial068.stdout: flexiblas-openblas-openmp x86_64 3.0.4-9.el9 appstream 15 k 2026-02-23T13:04:37.610 INFO:teuthology.orchestra.run.trial068.stdout: libgfortran x86_64 11.5.0-14.el9 baseos 794 k 2026-02-23T13:04:37.610 INFO:teuthology.orchestra.run.trial068.stdout: libquadmath x86_64 11.5.0-14.el9 baseos 184 k 2026-02-23T13:04:37.610 INFO:teuthology.orchestra.run.trial068.stdout: openblas x86_64 0.3.29-1.el9 appstream 42 k 2026-02-23T13:04:37.610 INFO:teuthology.orchestra.run.trial068.stdout: openblas-openmp x86_64 0.3.29-1.el9 appstream 5.3 M 2026-02-23T13:04:37.610 INFO:teuthology.orchestra.run.trial068.stdout: python3-devel x86_64 3.9.25-3.el9 appstream 244 k 2026-02-23T13:04:37.610 INFO:teuthology.orchestra.run.trial068.stdout: python3-numpy x86_64 1:1.23.5-2.el9 appstream 6.1 M 2026-02-23T13:04:37.610 INFO:teuthology.orchestra.run.trial068.stdout: python3-numpy-f2py x86_64 1:1.23.5-2.el9 appstream 442 k 2026-02-23T13:04:37.611 INFO:teuthology.orchestra.run.trial068.stdout: python3-scipy x86_64 1.9.3-2.el9 appstream 19 M 2026-02-23T13:04:37.611 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T13:04:37.611 INFO:teuthology.orchestra.run.trial068.stdout:Transaction Summary 2026-02-23T13:04:37.611 INFO:teuthology.orchestra.run.trial068.stdout:======================================================================================= 2026-02-23T13:04:37.611 INFO:teuthology.orchestra.run.trial068.stdout:Install 12 Packages 2026-02-23T13:04:37.611 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T13:04:37.611 INFO:teuthology.orchestra.run.trial068.stdout:Total download size: 43 M 2026-02-23T13:04:37.611 INFO:teuthology.orchestra.run.trial068.stdout:Installed size: 234 M 2026-02-23T13:04:37.611 INFO:teuthology.orchestra.run.trial068.stdout:Downloading Packages: 2026-02-23T13:04:37.715 INFO:teuthology.orchestra.run.trial185.stdout:Running transaction check 2026-02-23T13:04:37.729 INFO:teuthology.orchestra.run.trial185.stdout:Transaction check succeeded. 2026-02-23T13:04:37.729 INFO:teuthology.orchestra.run.trial185.stdout:Running transaction test 2026-02-23T13:04:37.929 INFO:teuthology.orchestra.run.trial185.stdout:Transaction test succeeded. 2026-02-23T13:04:37.929 INFO:teuthology.orchestra.run.trial185.stdout:Running transaction 2026-02-23T13:04:38.131 INFO:teuthology.orchestra.run.trial068.stdout:(1/12): libquadmath-11.5.0-14.el9.x86_64.rpm 1.4 MB/s | 184 kB 00:00 2026-02-23T13:04:38.144 INFO:teuthology.orchestra.run.trial185.stdout: Preparing : 1/1 2026-02-23T13:04:38.148 INFO:teuthology.orchestra.run.trial185.stdout: Installing : flexiblas-3.0.4-9.el9.x86_64 1/12 2026-02-23T13:04:38.154 INFO:teuthology.orchestra.run.trial185.stdout: Installing : libquadmath-11.5.0-14.el9.x86_64 2/12 2026-02-23T13:04:38.182 INFO:teuthology.orchestra.run.trial068.stdout:(2/12): libgfortran-11.5.0-14.el9.x86_64.rpm 4.3 MB/s | 794 kB 00:00 2026-02-23T13:04:38.187 INFO:teuthology.orchestra.run.trial185.stdout: Installing : libgfortran-11.5.0-14.el9.x86_64 3/12 2026-02-23T13:04:38.191 INFO:teuthology.orchestra.run.trial185.stdout: Installing : python3-devel-3.9.25-3.el9.x86_64 4/12 2026-02-23T13:04:38.250 INFO:teuthology.orchestra.run.trial185.stdout: Installing : openblas-0.3.29-1.el9.x86_64 5/12 2026-02-23T13:04:38.252 INFO:teuthology.orchestra.run.trial185.stdout: Installing : openblas-openmp-0.3.29-1.el9.x86_64 6/12 2026-02-23T13:04:38.273 INFO:teuthology.orchestra.run.trial185.stdout: Installing : flexiblas-openblas-openmp-3.0.4-9.el9.x86_64 7/12 2026-02-23T13:04:38.571 INFO:teuthology.orchestra.run.trial068.stdout:(3/12): flexiblas-3.0.4-9.el9.x86_64.rpm 67 kB/s | 30 kB 00:00 2026-02-23T13:04:38.592 INFO:teuthology.orchestra.run.trial185.stdout: Installing : flexiblas-netlib-3.0.4-9.el9.x86_64 8/12 2026-02-23T13:04:38.664 INFO:teuthology.orchestra.run.trial185.stdout: Installing : python3-numpy-1:1.23.5-2.el9.x86_64 9/12 2026-02-23T13:04:38.670 INFO:teuthology.orchestra.run.trial068.stdout:(4/12): flexiblas-openblas-openmp-3.0.4-9.el9.x 150 kB/s | 15 kB 00:00 2026-02-23T13:04:38.848 INFO:teuthology.orchestra.run.trial068.stdout:(5/12): openblas-0.3.29-1.el9.x86_64.rpm 237 kB/s | 42 kB 00:00 2026-02-23T13:04:39.322 INFO:teuthology.orchestra.run.trial185.stdout: Installing : python3-numpy-f2py-1:1.23.5-2.el9.x86_64 10/12 2026-02-23T13:04:39.675 INFO:teuthology.orchestra.run.trial068.stdout:(6/12): ceph-mgr-diskprediction-local-20.3.0-54 4.4 MB/s | 7.4 MB 00:01 2026-02-23T13:04:39.775 INFO:teuthology.orchestra.run.trial185.stdout: Installing : python3-scipy-1.9.3-2.el9.x86_64 11/12 2026-02-23T13:04:39.776 INFO:teuthology.orchestra.run.trial185.stdout: Installing : ceph-mgr-diskprediction-local-2:20.3.0-5439.ge3714 12/12 2026-02-23T13:04:40.052 INFO:teuthology.orchestra.run.trial185.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:20.3.0-5439.ge3714 12/12 2026-02-23T13:04:40.053 INFO:teuthology.orchestra.run.trial185.stdout: Verifying : ceph-mgr-diskprediction-local-2:20.3.0-5439.ge3714 1/12 2026-02-23T13:04:40.053 INFO:teuthology.orchestra.run.trial185.stdout: Verifying : libgfortran-11.5.0-14.el9.x86_64 2/12 2026-02-23T13:04:40.053 INFO:teuthology.orchestra.run.trial185.stdout: Verifying : libquadmath-11.5.0-14.el9.x86_64 3/12 2026-02-23T13:04:40.053 INFO:teuthology.orchestra.run.trial185.stdout: Verifying : flexiblas-3.0.4-9.el9.x86_64 4/12 2026-02-23T13:04:40.053 INFO:teuthology.orchestra.run.trial185.stdout: Verifying : flexiblas-netlib-3.0.4-9.el9.x86_64 5/12 2026-02-23T13:04:40.053 INFO:teuthology.orchestra.run.trial185.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-9.el9.x86_64 6/12 2026-02-23T13:04:40.054 INFO:teuthology.orchestra.run.trial185.stdout: Verifying : openblas-0.3.29-1.el9.x86_64 7/12 2026-02-23T13:04:40.054 INFO:teuthology.orchestra.run.trial185.stdout: Verifying : openblas-openmp-0.3.29-1.el9.x86_64 8/12 2026-02-23T13:04:40.054 INFO:teuthology.orchestra.run.trial185.stdout: Verifying : python3-devel-3.9.25-3.el9.x86_64 9/12 2026-02-23T13:04:40.054 INFO:teuthology.orchestra.run.trial185.stdout: Verifying : python3-numpy-1:1.23.5-2.el9.x86_64 10/12 2026-02-23T13:04:40.054 INFO:teuthology.orchestra.run.trial185.stdout: Verifying : python3-numpy-f2py-1:1.23.5-2.el9.x86_64 11/12 2026-02-23T13:04:40.109 INFO:teuthology.orchestra.run.trial185.stdout: Verifying : python3-scipy-1.9.3-2.el9.x86_64 12/12 2026-02-23T13:04:40.109 INFO:teuthology.orchestra.run.trial185.stdout: 2026-02-23T13:04:40.109 INFO:teuthology.orchestra.run.trial185.stdout:Installed: 2026-02-23T13:04:40.109 INFO:teuthology.orchestra.run.trial185.stdout: ceph-mgr-diskprediction-local-2:20.3.0-5439.ge371433f.el9.noarch 2026-02-23T13:04:40.110 INFO:teuthology.orchestra.run.trial185.stdout: flexiblas-3.0.4-9.el9.x86_64 2026-02-23T13:04:40.110 INFO:teuthology.orchestra.run.trial185.stdout: flexiblas-netlib-3.0.4-9.el9.x86_64 2026-02-23T13:04:40.110 INFO:teuthology.orchestra.run.trial185.stdout: flexiblas-openblas-openmp-3.0.4-9.el9.x86_64 2026-02-23T13:04:40.110 INFO:teuthology.orchestra.run.trial185.stdout: libgfortran-11.5.0-14.el9.x86_64 2026-02-23T13:04:40.110 INFO:teuthology.orchestra.run.trial185.stdout: libquadmath-11.5.0-14.el9.x86_64 2026-02-23T13:04:40.110 INFO:teuthology.orchestra.run.trial185.stdout: openblas-0.3.29-1.el9.x86_64 2026-02-23T13:04:40.110 INFO:teuthology.orchestra.run.trial185.stdout: openblas-openmp-0.3.29-1.el9.x86_64 2026-02-23T13:04:40.110 INFO:teuthology.orchestra.run.trial185.stdout: python3-devel-3.9.25-3.el9.x86_64 2026-02-23T13:04:40.110 INFO:teuthology.orchestra.run.trial185.stdout: python3-numpy-1:1.23.5-2.el9.x86_64 2026-02-23T13:04:40.110 INFO:teuthology.orchestra.run.trial185.stdout: python3-numpy-f2py-1:1.23.5-2.el9.x86_64 2026-02-23T13:04:40.110 INFO:teuthology.orchestra.run.trial185.stdout: python3-scipy-1.9.3-2.el9.x86_64 2026-02-23T13:04:40.111 INFO:teuthology.orchestra.run.trial185.stdout: 2026-02-23T13:04:40.111 INFO:teuthology.orchestra.run.trial185.stdout:Complete! 2026-02-23T13:04:40.146 DEBUG:teuthology.orchestra.run.trial185:> sudo yum -y install ceph-mgr-rook 2026-02-23T13:04:40.450 INFO:teuthology.orchestra.run.trial185.stdout:Last metadata expiration check: 0:00:56 ago on Mon 23 Feb 2026 01:03:44 PM UTC. 2026-02-23T13:04:40.534 INFO:teuthology.orchestra.run.trial185.stdout:Dependencies resolved. 2026-02-23T13:04:40.535 INFO:teuthology.orchestra.run.trial185.stdout:================================================================================ 2026-02-23T13:04:40.535 INFO:teuthology.orchestra.run.trial185.stdout: Package Arch Version Repository Size 2026-02-23T13:04:40.535 INFO:teuthology.orchestra.run.trial185.stdout:================================================================================ 2026-02-23T13:04:40.536 INFO:teuthology.orchestra.run.trial185.stdout:Installing: 2026-02-23T13:04:40.536 INFO:teuthology.orchestra.run.trial185.stdout: ceph-mgr-rook noarch 2:20.3.0-5439.ge371433f.el9 ceph-noarch 50 k 2026-02-23T13:04:40.536 INFO:teuthology.orchestra.run.trial185.stdout:Installing dependencies: 2026-02-23T13:04:40.536 INFO:teuthology.orchestra.run.trial185.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2026-02-23T13:04:40.536 INFO:teuthology.orchestra.run.trial185.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2026-02-23T13:04:40.536 INFO:teuthology.orchestra.run.trial185.stdout: python3-google-auth noarch 1:2.45.0-1.el9 epel 254 k 2026-02-23T13:04:40.536 INFO:teuthology.orchestra.run.trial185.stdout: python3-kubernetes noarch 1:26.1.0-3.el9 epel 1.0 M 2026-02-23T13:04:40.536 INFO:teuthology.orchestra.run.trial185.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2026-02-23T13:04:40.536 INFO:teuthology.orchestra.run.trial185.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2026-02-23T13:04:40.536 INFO:teuthology.orchestra.run.trial185.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2026-02-23T13:04:40.536 INFO:teuthology.orchestra.run.trial185.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2026-02-23T13:04:40.537 INFO:teuthology.orchestra.run.trial185.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2026-02-23T13:04:40.537 INFO:teuthology.orchestra.run.trial185.stdout: 2026-02-23T13:04:40.537 INFO:teuthology.orchestra.run.trial185.stdout:Transaction Summary 2026-02-23T13:04:40.537 INFO:teuthology.orchestra.run.trial185.stdout:================================================================================ 2026-02-23T13:04:40.537 INFO:teuthology.orchestra.run.trial185.stdout:Install 10 Packages 2026-02-23T13:04:40.537 INFO:teuthology.orchestra.run.trial185.stdout: 2026-02-23T13:04:40.537 INFO:teuthology.orchestra.run.trial185.stdout:Total download size: 2.0 M 2026-02-23T13:04:40.537 INFO:teuthology.orchestra.run.trial185.stdout:Installed size: 25 M 2026-02-23T13:04:40.537 INFO:teuthology.orchestra.run.trial185.stdout:Downloading Packages: 2026-02-23T13:04:40.946 INFO:teuthology.orchestra.run.trial068.stdout:(7/12): python3-devel-3.9.25-3.el9.x86_64.rpm 192 kB/s | 244 kB 00:01 2026-02-23T13:04:41.291 INFO:teuthology.orchestra.run.trial185.stdout:(1/10): ceph-mgr-rook-20.3.0-5439.ge371433f.el9 158 kB/s | 50 kB 00:00 2026-02-23T13:04:41.700 INFO:teuthology.orchestra.run.trial185.stdout:(2/10): python3-pyasn1-0.4.8-6.el9.noarch.rpm 218 kB/s | 159 kB 00:00 2026-02-23T13:04:41.752 INFO:teuthology.orchestra.run.trial185.stdout:(3/10): python3-pyasn1-modules-0.4.8-6.el9.noar 358 kB/s | 279 kB 00:00 2026-02-23T13:04:41.891 INFO:teuthology.orchestra.run.trial185.stdout:(4/10): python3-requests-oauthlib-1.3.0-12.el9. 89 kB/s | 54 kB 00:00 2026-02-23T13:04:41.955 INFO:teuthology.orchestra.run.trial185.stdout:(5/10): python3-certifi-2023.05.07-4.el9.noarch 70 kB/s | 14 kB 00:00 2026-02-23T13:04:41.979 INFO:teuthology.orchestra.run.trial185.stdout:(6/10): python3-cachetools-4.2.4-1.el9.noarch.r 115 kB/s | 32 kB 00:00 2026-02-23T13:04:42.107 INFO:teuthology.orchestra.run.trial185.stdout:(7/10): python3-rsa-4.9-2.el9.noarch.rpm 462 kB/s | 59 kB 00:00 2026-02-23T13:04:42.108 INFO:teuthology.orchestra.run.trial068.stdout:(8/12): flexiblas-netlib-3.0.4-9.el9.x86_64.rpm 780 kB/s | 3.0 MB 00:03 2026-02-23T13:04:42.202 INFO:teuthology.orchestra.run.trial185.stdout:(8/10): python3-websocket-client-1.2.3-2.el9.no 944 kB/s | 90 kB 00:00 2026-02-23T13:04:42.333 INFO:teuthology.orchestra.run.trial185.stdout:(9/10): python3-google-auth-2.45.0-1.el9.noarch 576 kB/s | 254 kB 00:00 2026-02-23T13:04:42.338 INFO:teuthology.orchestra.run.trial185.stdout:(10/10): python3-kubernetes-26.1.0-3.el9.noarch 2.7 MB/s | 1.0 MB 00:00 2026-02-23T13:04:42.340 INFO:teuthology.orchestra.run.trial185.stdout:-------------------------------------------------------------------------------- 2026-02-23T13:04:42.340 INFO:teuthology.orchestra.run.trial185.stdout:Total 1.1 MB/s | 2.0 MB 00:01 2026-02-23T13:04:42.363 INFO:teuthology.orchestra.run.trial185.stdout:Running transaction check 2026-02-23T13:04:42.372 INFO:teuthology.orchestra.run.trial185.stdout:Transaction check succeeded. 2026-02-23T13:04:42.372 INFO:teuthology.orchestra.run.trial185.stdout:Running transaction test 2026-02-23T13:04:42.413 INFO:teuthology.orchestra.run.trial185.stdout:Transaction test succeeded. 2026-02-23T13:04:42.413 INFO:teuthology.orchestra.run.trial185.stdout:Running transaction 2026-02-23T13:04:42.487 INFO:teuthology.orchestra.run.trial185.stdout: Preparing : 1/1 2026-02-23T13:04:42.538 INFO:teuthology.orchestra.run.trial185.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/10 2026-02-23T13:04:42.550 INFO:teuthology.orchestra.run.trial185.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/10 2026-02-23T13:04:42.565 INFO:teuthology.orchestra.run.trial185.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/10 2026-02-23T13:04:42.570 INFO:teuthology.orchestra.run.trial185.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/10 2026-02-23T13:04:42.577 INFO:teuthology.orchestra.run.trial185.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 5/10 2026-02-23T13:04:42.617 INFO:teuthology.orchestra.run.trial185.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/10 2026-02-23T13:04:42.629 INFO:teuthology.orchestra.run.trial185.stdout: Installing : python3-google-auth-1:2.45.0-1.el9.noarch 7/10 2026-02-23T13:04:42.687 INFO:teuthology.orchestra.run.trial068.stdout:(9/12): python3-numpy-f2py-1.23.5-2.el9.x86_64. 764 kB/s | 442 kB 00:00 2026-02-23T13:04:42.951 INFO:teuthology.orchestra.run.trial185.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 8/10 2026-02-23T13:04:42.965 INFO:teuthology.orchestra.run.trial185.stdout: Installing : python3-kubernetes-1:26.1.0-3.el9.noarch 9/10 2026-02-23T13:04:42.966 INFO:teuthology.orchestra.run.trial185.stdout: Installing : ceph-mgr-rook-2:20.3.0-5439.ge371433f.el9.noarch 10/10 2026-02-23T13:04:43.074 INFO:teuthology.orchestra.run.trial185.stdout: Running scriptlet: ceph-mgr-rook-2:20.3.0-5439.ge371433f.el9.noarch 10/10 2026-02-23T13:04:43.075 INFO:teuthology.orchestra.run.trial185.stdout: Verifying : ceph-mgr-rook-2:20.3.0-5439.ge371433f.el9.noarch 1/10 2026-02-23T13:04:43.075 INFO:teuthology.orchestra.run.trial185.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 2/10 2026-02-23T13:04:43.075 INFO:teuthology.orchestra.run.trial185.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 3/10 2026-02-23T13:04:43.075 INFO:teuthology.orchestra.run.trial185.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 4/10 2026-02-23T13:04:43.075 INFO:teuthology.orchestra.run.trial185.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 5/10 2026-02-23T13:04:43.075 INFO:teuthology.orchestra.run.trial185.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 6/10 2026-02-23T13:04:43.075 INFO:teuthology.orchestra.run.trial185.stdout: Verifying : python3-google-auth-1:2.45.0-1.el9.noarch 7/10 2026-02-23T13:04:43.075 INFO:teuthology.orchestra.run.trial185.stdout: Verifying : python3-kubernetes-1:26.1.0-3.el9.noarch 8/10 2026-02-23T13:04:43.075 INFO:teuthology.orchestra.run.trial185.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 9/10 2026-02-23T13:04:43.131 INFO:teuthology.orchestra.run.trial185.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 10/10 2026-02-23T13:04:43.131 INFO:teuthology.orchestra.run.trial185.stdout: 2026-02-23T13:04:43.131 INFO:teuthology.orchestra.run.trial185.stdout:Installed: 2026-02-23T13:04:43.132 INFO:teuthology.orchestra.run.trial185.stdout: ceph-mgr-rook-2:20.3.0-5439.ge371433f.el9.noarch 2026-02-23T13:04:43.132 INFO:teuthology.orchestra.run.trial185.stdout: python3-cachetools-4.2.4-1.el9.noarch 2026-02-23T13:04:43.132 INFO:teuthology.orchestra.run.trial185.stdout: python3-certifi-2023.05.07-4.el9.noarch 2026-02-23T13:04:43.132 INFO:teuthology.orchestra.run.trial185.stdout: python3-google-auth-1:2.45.0-1.el9.noarch 2026-02-23T13:04:43.132 INFO:teuthology.orchestra.run.trial185.stdout: python3-kubernetes-1:26.1.0-3.el9.noarch 2026-02-23T13:04:43.132 INFO:teuthology.orchestra.run.trial185.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2026-02-23T13:04:43.132 INFO:teuthology.orchestra.run.trial185.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2026-02-23T13:04:43.132 INFO:teuthology.orchestra.run.trial185.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2026-02-23T13:04:43.132 INFO:teuthology.orchestra.run.trial185.stdout: python3-rsa-4.9-2.el9.noarch 2026-02-23T13:04:43.132 INFO:teuthology.orchestra.run.trial185.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2026-02-23T13:04:43.132 INFO:teuthology.orchestra.run.trial185.stdout: 2026-02-23T13:04:43.133 INFO:teuthology.orchestra.run.trial185.stdout:Complete! 2026-02-23T13:04:43.162 DEBUG:teuthology.orchestra.run.trial185:> sudo yum -y install ceph-mgr-cephadm 2026-02-23T13:04:43.468 INFO:teuthology.orchestra.run.trial185.stdout:Last metadata expiration check: 0:00:59 ago on Mon 23 Feb 2026 01:03:44 PM UTC. 2026-02-23T13:04:43.551 INFO:teuthology.orchestra.run.trial185.stdout:Dependencies resolved. 2026-02-23T13:04:43.552 INFO:teuthology.orchestra.run.trial185.stdout:================================================================================ 2026-02-23T13:04:43.552 INFO:teuthology.orchestra.run.trial185.stdout: Package Arch Version Repository Size 2026-02-23T13:04:43.552 INFO:teuthology.orchestra.run.trial185.stdout:================================================================================ 2026-02-23T13:04:43.552 INFO:teuthology.orchestra.run.trial185.stdout:Installing: 2026-02-23T13:04:43.552 INFO:teuthology.orchestra.run.trial185.stdout: ceph-mgr-cephadm noarch 2:20.3.0-5439.ge371433f.el9 ceph-noarch 184 k 2026-02-23T13:04:43.552 INFO:teuthology.orchestra.run.trial185.stdout:Installing dependencies: 2026-02-23T13:04:43.552 INFO:teuthology.orchestra.run.trial185.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2026-02-23T13:04:43.552 INFO:teuthology.orchestra.run.trial185.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2026-02-23T13:04:43.553 INFO:teuthology.orchestra.run.trial185.stdout: python3-jinja2 noarch 2.11.3-8.el9 appstream 249 k 2026-02-23T13:04:43.553 INFO:teuthology.orchestra.run.trial185.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2026-02-23T13:04:43.553 INFO:teuthology.orchestra.run.trial185.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2026-02-23T13:04:43.553 INFO:teuthology.orchestra.run.trial185.stdout: python3-typing-extensions noarch 4.15.0-1.el9 epel 86 k 2026-02-23T13:04:43.553 INFO:teuthology.orchestra.run.trial185.stdout: 2026-02-23T13:04:43.553 INFO:teuthology.orchestra.run.trial185.stdout:Transaction Summary 2026-02-23T13:04:43.553 INFO:teuthology.orchestra.run.trial185.stdout:================================================================================ 2026-02-23T13:04:43.553 INFO:teuthology.orchestra.run.trial185.stdout:Install 7 Packages 2026-02-23T13:04:43.553 INFO:teuthology.orchestra.run.trial185.stdout: 2026-02-23T13:04:43.553 INFO:teuthology.orchestra.run.trial185.stdout:Total download size: 7.1 M 2026-02-23T13:04:43.553 INFO:teuthology.orchestra.run.trial185.stdout:Installed size: 34 M 2026-02-23T13:04:43.554 INFO:teuthology.orchestra.run.trial185.stdout:Downloading Packages: 2026-02-23T13:04:43.803 INFO:teuthology.orchestra.run.trial185.stdout:(1/7): ceph-mgr-cephadm-20.3.0-5439.ge371433f.e 2.1 MB/s | 184 kB 00:00 2026-02-23T13:04:43.871 INFO:teuthology.orchestra.run.trial185.stdout:(2/7): python3-jinja2-2.11.3-8.el9.noarch.rpm 1.6 MB/s | 249 kB 00:00 2026-02-23T13:04:43.873 INFO:teuthology.orchestra.run.trial185.stdout:(3/7): python3-markupsafe-1.1.1-12.el9.x86_64.r 499 kB/s | 35 kB 00:00 2026-02-23T13:04:43.933 INFO:teuthology.orchestra.run.trial185.stdout:(4/7): python3-natsort-7.1.1-5.el9.noarch.rpm 966 kB/s | 58 kB 00:00 2026-02-23T13:04:43.965 INFO:teuthology.orchestra.run.trial185.stdout:(5/7): python3-asyncssh-2.13.2-5.el9.noarch.rpm 5.7 MB/s | 548 kB 00:00 2026-02-23T13:04:43.966 INFO:teuthology.orchestra.run.trial185.stdout:(6/7): python3-typing-extensions-4.15.0-1.el9.n 2.5 MB/s | 86 kB 00:00 2026-02-23T13:04:43.994 INFO:teuthology.orchestra.run.trial185.stdout:(7/7): python3-babel-2.9.1-2.el9.noarch.rpm 22 MB/s | 6.0 MB 00:00 2026-02-23T13:04:43.996 INFO:teuthology.orchestra.run.trial185.stdout:-------------------------------------------------------------------------------- 2026-02-23T13:04:43.996 INFO:teuthology.orchestra.run.trial185.stdout:Total 16 MB/s | 7.1 MB 00:00 2026-02-23T13:04:44.034 INFO:teuthology.orchestra.run.trial185.stdout:Running transaction check 2026-02-23T13:04:44.040 INFO:teuthology.orchestra.run.trial185.stdout:Transaction check succeeded. 2026-02-23T13:04:44.040 INFO:teuthology.orchestra.run.trial185.stdout:Running transaction test 2026-02-23T13:04:44.112 INFO:teuthology.orchestra.run.trial185.stdout:Transaction test succeeded. 2026-02-23T13:04:44.112 INFO:teuthology.orchestra.run.trial185.stdout:Running transaction 2026-02-23T13:04:44.202 INFO:teuthology.orchestra.run.trial185.stdout: Preparing : 1/1 2026-02-23T13:04:44.251 INFO:teuthology.orchestra.run.trial185.stdout: Installing : python3-typing-extensions-4.15.0-1.el9.noarch 1/7 2026-02-23T13:04:44.261 INFO:teuthology.orchestra.run.trial185.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/7 2026-02-23T13:04:44.267 INFO:teuthology.orchestra.run.trial185.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/7 2026-02-23T13:04:44.487 INFO:teuthology.orchestra.run.trial185.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 4/7 2026-02-23T13:04:44.511 INFO:teuthology.orchestra.run.trial185.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 5/7 2026-02-23T13:04:44.535 INFO:teuthology.orchestra.run.trial185.stdout: Installing : python3-jinja2-2.11.3-8.el9.noarch 6/7 2026-02-23T13:04:44.537 INFO:teuthology.orchestra.run.trial185.stdout: Installing : ceph-mgr-cephadm-2:20.3.0-5439.ge371433f.el9.noarch 7/7 2026-02-23T13:04:44.646 INFO:teuthology.orchestra.run.trial185.stdout: Running scriptlet: ceph-mgr-cephadm-2:20.3.0-5439.ge371433f.el9.noarch 7/7 2026-02-23T13:04:44.646 INFO:teuthology.orchestra.run.trial185.stdout: Verifying : ceph-mgr-cephadm-2:20.3.0-5439.ge371433f.el9.noarch 1/7 2026-02-23T13:04:44.647 INFO:teuthology.orchestra.run.trial185.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/7 2026-02-23T13:04:44.647 INFO:teuthology.orchestra.run.trial185.stdout: Verifying : python3-jinja2-2.11.3-8.el9.noarch 3/7 2026-02-23T13:04:44.647 INFO:teuthology.orchestra.run.trial185.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 4/7 2026-02-23T13:04:44.647 INFO:teuthology.orchestra.run.trial185.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 5/7 2026-02-23T13:04:44.647 INFO:teuthology.orchestra.run.trial185.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 6/7 2026-02-23T13:04:44.703 INFO:teuthology.orchestra.run.trial185.stdout: Verifying : python3-typing-extensions-4.15.0-1.el9.noarch 7/7 2026-02-23T13:04:44.703 INFO:teuthology.orchestra.run.trial185.stdout: 2026-02-23T13:04:44.703 INFO:teuthology.orchestra.run.trial185.stdout:Installed: 2026-02-23T13:04:44.703 INFO:teuthology.orchestra.run.trial185.stdout: ceph-mgr-cephadm-2:20.3.0-5439.ge371433f.el9.noarch 2026-02-23T13:04:44.703 INFO:teuthology.orchestra.run.trial185.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2026-02-23T13:04:44.703 INFO:teuthology.orchestra.run.trial185.stdout: python3-babel-2.9.1-2.el9.noarch 2026-02-23T13:04:44.703 INFO:teuthology.orchestra.run.trial185.stdout: python3-jinja2-2.11.3-8.el9.noarch 2026-02-23T13:04:44.703 INFO:teuthology.orchestra.run.trial185.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2026-02-23T13:04:44.703 INFO:teuthology.orchestra.run.trial185.stdout: python3-natsort-7.1.1-5.el9.noarch 2026-02-23T13:04:44.703 INFO:teuthology.orchestra.run.trial185.stdout: python3-typing-extensions-4.15.0-1.el9.noarch 2026-02-23T13:04:44.703 INFO:teuthology.orchestra.run.trial185.stdout: 2026-02-23T13:04:44.703 INFO:teuthology.orchestra.run.trial185.stdout:Complete! 2026-02-23T13:04:44.734 DEBUG:teuthology.orchestra.run.trial185:> sudo yum -y install ceph-osd 2026-02-23T13:04:45.012 INFO:teuthology.orchestra.run.trial068.stdout:(10/12): openblas-openmp-0.3.29-1.el9.x86_64.rp 878 kB/s | 5.3 MB 00:06 2026-02-23T13:04:45.047 INFO:teuthology.orchestra.run.trial185.stdout:Last metadata expiration check: 0:01:01 ago on Mon 23 Feb 2026 01:03:44 PM UTC. 2026-02-23T13:04:45.112 INFO:teuthology.orchestra.run.trial185.stdout:Package ceph-osd-2:20.3.0-5439.ge371433f.el9.x86_64 is already installed. 2026-02-23T13:04:45.130 INFO:teuthology.orchestra.run.trial185.stdout:Dependencies resolved. 2026-02-23T13:04:45.131 INFO:teuthology.orchestra.run.trial185.stdout:Nothing to do. 2026-02-23T13:04:45.131 INFO:teuthology.orchestra.run.trial185.stdout:Complete! 2026-02-23T13:04:45.152 DEBUG:teuthology.orchestra.run.trial185:> sudo yum -y install ceph-osd-classic 2026-02-23T13:04:45.454 INFO:teuthology.orchestra.run.trial185.stdout:Last metadata expiration check: 0:01:01 ago on Mon 23 Feb 2026 01:03:44 PM UTC. 2026-02-23T13:04:45.520 INFO:teuthology.orchestra.run.trial185.stdout:Package ceph-osd-classic-2:20.3.0-5439.ge371433f.el9.x86_64 is already installed. 2026-02-23T13:04:45.538 INFO:teuthology.orchestra.run.trial185.stdout:Dependencies resolved. 2026-02-23T13:04:45.539 INFO:teuthology.orchestra.run.trial185.stdout:Nothing to do. 2026-02-23T13:04:45.539 INFO:teuthology.orchestra.run.trial185.stdout:Complete! 2026-02-23T13:04:45.560 DEBUG:teuthology.orchestra.run.trial185:> sudo yum -y install ceph-fuse 2026-02-23T13:04:45.870 INFO:teuthology.orchestra.run.trial185.stdout:Last metadata expiration check: 0:01:01 ago on Mon 23 Feb 2026 01:03:44 PM UTC. 2026-02-23T13:04:45.952 INFO:teuthology.orchestra.run.trial185.stdout:Dependencies resolved. 2026-02-23T13:04:45.952 INFO:teuthology.orchestra.run.trial185.stdout:================================================================================ 2026-02-23T13:04:45.952 INFO:teuthology.orchestra.run.trial185.stdout: Package Arch Version Repo Size 2026-02-23T13:04:45.952 INFO:teuthology.orchestra.run.trial185.stdout:================================================================================ 2026-02-23T13:04:45.952 INFO:teuthology.orchestra.run.trial185.stdout:Installing: 2026-02-23T13:04:45.952 INFO:teuthology.orchestra.run.trial185.stdout: ceph-fuse x86_64 2:20.3.0-5439.ge371433f.el9 ceph 914 k 2026-02-23T13:04:45.952 INFO:teuthology.orchestra.run.trial185.stdout: 2026-02-23T13:04:45.952 INFO:teuthology.orchestra.run.trial185.stdout:Transaction Summary 2026-02-23T13:04:45.952 INFO:teuthology.orchestra.run.trial185.stdout:================================================================================ 2026-02-23T13:04:45.952 INFO:teuthology.orchestra.run.trial185.stdout:Install 1 Package 2026-02-23T13:04:45.952 INFO:teuthology.orchestra.run.trial185.stdout: 2026-02-23T13:04:45.953 INFO:teuthology.orchestra.run.trial185.stdout:Total download size: 914 k 2026-02-23T13:04:45.953 INFO:teuthology.orchestra.run.trial185.stdout:Installed size: 2.9 M 2026-02-23T13:04:45.953 INFO:teuthology.orchestra.run.trial185.stdout:Downloading Packages: 2026-02-23T13:04:46.499 INFO:teuthology.orchestra.run.trial185.stdout:ceph-fuse-20.3.0-5439.ge371433f.el9.x86_64.rpm 1.6 MB/s | 914 kB 00:00 2026-02-23T13:04:46.499 INFO:teuthology.orchestra.run.trial185.stdout:-------------------------------------------------------------------------------- 2026-02-23T13:04:46.500 INFO:teuthology.orchestra.run.trial185.stdout:Total 1.6 MB/s | 914 kB 00:00 2026-02-23T13:04:46.500 INFO:teuthology.orchestra.run.trial185.stdout:Running transaction check 2026-02-23T13:04:46.504 INFO:teuthology.orchestra.run.trial185.stdout:Transaction check succeeded. 2026-02-23T13:04:46.504 INFO:teuthology.orchestra.run.trial185.stdout:Running transaction test 2026-02-23T13:04:46.537 INFO:teuthology.orchestra.run.trial185.stdout:Transaction test succeeded. 2026-02-23T13:04:46.537 INFO:teuthology.orchestra.run.trial185.stdout:Running transaction 2026-02-23T13:04:46.627 INFO:teuthology.orchestra.run.trial185.stdout: Preparing : 1/1 2026-02-23T13:04:46.644 INFO:teuthology.orchestra.run.trial185.stdout: Installing : ceph-fuse-2:20.3.0-5439.ge371433f.el9.x86_64 1/1 2026-02-23T13:04:46.895 INFO:teuthology.orchestra.run.trial185.stdout: Running scriptlet: ceph-fuse-2:20.3.0-5439.ge371433f.el9.x86_64 1/1 2026-02-23T13:04:46.952 INFO:teuthology.orchestra.run.trial185.stdout: Verifying : ceph-fuse-2:20.3.0-5439.ge371433f.el9.x86_64 1/1 2026-02-23T13:04:46.952 INFO:teuthology.orchestra.run.trial185.stdout: 2026-02-23T13:04:46.952 INFO:teuthology.orchestra.run.trial185.stdout:Installed: 2026-02-23T13:04:46.952 INFO:teuthology.orchestra.run.trial185.stdout: ceph-fuse-2:20.3.0-5439.ge371433f.el9.x86_64 2026-02-23T13:04:46.952 INFO:teuthology.orchestra.run.trial185.stdout: 2026-02-23T13:04:46.952 INFO:teuthology.orchestra.run.trial185.stdout:Complete! 2026-02-23T13:04:46.979 DEBUG:teuthology.orchestra.run.trial185:> sudo yum -y install ceph-volume 2026-02-23T13:04:47.288 INFO:teuthology.orchestra.run.trial185.stdout:Last metadata expiration check: 0:01:03 ago on Mon 23 Feb 2026 01:03:44 PM UTC. 2026-02-23T13:04:47.370 INFO:teuthology.orchestra.run.trial185.stdout:Dependencies resolved. 2026-02-23T13:04:47.372 INFO:teuthology.orchestra.run.trial185.stdout:================================================================================ 2026-02-23T13:04:47.372 INFO:teuthology.orchestra.run.trial185.stdout: Package Arch Version Repository Size 2026-02-23T13:04:47.372 INFO:teuthology.orchestra.run.trial185.stdout:================================================================================ 2026-02-23T13:04:47.372 INFO:teuthology.orchestra.run.trial185.stdout:Installing: 2026-02-23T13:04:47.372 INFO:teuthology.orchestra.run.trial185.stdout: ceph-volume noarch 2:20.3.0-5439.ge371433f.el9 ceph-noarch 301 k 2026-02-23T13:04:47.372 INFO:teuthology.orchestra.run.trial185.stdout:Installing dependencies: 2026-02-23T13:04:47.372 INFO:teuthology.orchestra.run.trial185.stdout: cryptsetup x86_64 2.8.1-3.el9 baseos 351 k 2026-02-23T13:04:47.373 INFO:teuthology.orchestra.run.trial185.stdout: 2026-02-23T13:04:47.373 INFO:teuthology.orchestra.run.trial185.stdout:Transaction Summary 2026-02-23T13:04:47.373 INFO:teuthology.orchestra.run.trial185.stdout:================================================================================ 2026-02-23T13:04:47.373 INFO:teuthology.orchestra.run.trial185.stdout:Install 2 Packages 2026-02-23T13:04:47.373 INFO:teuthology.orchestra.run.trial185.stdout: 2026-02-23T13:04:47.373 INFO:teuthology.orchestra.run.trial185.stdout:Total download size: 652 k 2026-02-23T13:04:47.373 INFO:teuthology.orchestra.run.trial185.stdout:Installed size: 2.2 M 2026-02-23T13:04:47.373 INFO:teuthology.orchestra.run.trial185.stdout:Downloading Packages: 2026-02-23T13:04:47.828 INFO:teuthology.orchestra.run.trial185.stdout:(1/2): ceph-volume-20.3.0-5439.ge371433f.el9.no 764 kB/s | 301 kB 00:00 2026-02-23T13:04:47.841 INFO:teuthology.orchestra.run.trial185.stdout:(2/2): cryptsetup-2.8.1-3.el9.x86_64.rpm 860 kB/s | 351 kB 00:00 2026-02-23T13:04:47.842 INFO:teuthology.orchestra.run.trial185.stdout:-------------------------------------------------------------------------------- 2026-02-23T13:04:47.842 INFO:teuthology.orchestra.run.trial185.stdout:Total 1.4 MB/s | 652 kB 00:00 2026-02-23T13:04:47.851 INFO:teuthology.orchestra.run.trial185.stdout:Running transaction check 2026-02-23T13:04:47.858 INFO:teuthology.orchestra.run.trial185.stdout:Transaction check succeeded. 2026-02-23T13:04:47.858 INFO:teuthology.orchestra.run.trial185.stdout:Running transaction test 2026-02-23T13:04:47.901 INFO:teuthology.orchestra.run.trial185.stdout:Transaction test succeeded. 2026-02-23T13:04:47.901 INFO:teuthology.orchestra.run.trial185.stdout:Running transaction 2026-02-23T13:04:47.981 INFO:teuthology.orchestra.run.trial185.stdout: Preparing : 1/1 2026-02-23T13:04:48.045 INFO:teuthology.orchestra.run.trial185.stdout: Installing : cryptsetup-2.8.1-3.el9.x86_64 1/2 2026-02-23T13:04:48.047 INFO:teuthology.orchestra.run.trial185.stdout: Installing : ceph-volume-2:20.3.0-5439.ge371433f.el9.noarch 2/2 2026-02-23T13:04:48.063 INFO:teuthology.orchestra.run.trial185.stdout: Running scriptlet: ceph-volume-2:20.3.0-5439.ge371433f.el9.noarch 2/2 2026-02-23T13:04:48.063 INFO:teuthology.orchestra.run.trial185.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-23T13:04:48.064 INFO:teuthology.orchestra.run.trial185.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2026-02-23T13:04:48.064 INFO:teuthology.orchestra.run.trial185.stdout: 2026-02-23T13:04:48.294 INFO:teuthology.orchestra.run.trial185.stdout: Verifying : ceph-volume-2:20.3.0-5439.ge371433f.el9.noarch 1/2 2026-02-23T13:04:48.351 INFO:teuthology.orchestra.run.trial185.stdout: Verifying : cryptsetup-2.8.1-3.el9.x86_64 2/2 2026-02-23T13:04:48.352 INFO:teuthology.orchestra.run.trial185.stdout: 2026-02-23T13:04:48.352 INFO:teuthology.orchestra.run.trial185.stdout:Installed: 2026-02-23T13:04:48.352 INFO:teuthology.orchestra.run.trial185.stdout: ceph-volume-2:20.3.0-5439.ge371433f.el9.noarch cryptsetup-2.8.1-3.el9.x86_64 2026-02-23T13:04:48.352 INFO:teuthology.orchestra.run.trial185.stdout: 2026-02-23T13:04:48.352 INFO:teuthology.orchestra.run.trial185.stdout:Complete! 2026-02-23T13:04:48.380 DEBUG:teuthology.orchestra.run.trial185:> sudo yum -y install librados-devel 2026-02-23T13:04:48.440 INFO:teuthology.orchestra.run.trial068.stdout:(11/12): python3-numpy-1.23.5-2.el9.x86_64.rpm 838 kB/s | 6.1 MB 00:07 2026-02-23T13:04:48.686 INFO:teuthology.orchestra.run.trial185.stdout:Last metadata expiration check: 0:01:04 ago on Mon 23 Feb 2026 01:03:44 PM UTC. 2026-02-23T13:04:48.770 INFO:teuthology.orchestra.run.trial185.stdout:Dependencies resolved. 2026-02-23T13:04:48.770 INFO:teuthology.orchestra.run.trial185.stdout:================================================================================ 2026-02-23T13:04:48.770 INFO:teuthology.orchestra.run.trial185.stdout: Package Arch Version Repo Size 2026-02-23T13:04:48.770 INFO:teuthology.orchestra.run.trial185.stdout:================================================================================ 2026-02-23T13:04:48.770 INFO:teuthology.orchestra.run.trial185.stdout:Installing: 2026-02-23T13:04:48.770 INFO:teuthology.orchestra.run.trial185.stdout: librados-devel x86_64 2:20.3.0-5439.ge371433f.el9 ceph 124 k 2026-02-23T13:04:48.770 INFO:teuthology.orchestra.run.trial185.stdout: 2026-02-23T13:04:48.770 INFO:teuthology.orchestra.run.trial185.stdout:Transaction Summary 2026-02-23T13:04:48.771 INFO:teuthology.orchestra.run.trial185.stdout:================================================================================ 2026-02-23T13:04:48.771 INFO:teuthology.orchestra.run.trial185.stdout:Install 1 Package 2026-02-23T13:04:48.771 INFO:teuthology.orchestra.run.trial185.stdout: 2026-02-23T13:04:48.771 INFO:teuthology.orchestra.run.trial185.stdout:Total download size: 124 k 2026-02-23T13:04:48.771 INFO:teuthology.orchestra.run.trial185.stdout:Installed size: 434 k 2026-02-23T13:04:48.771 INFO:teuthology.orchestra.run.trial185.stdout:Downloading Packages: 2026-02-23T13:04:49.145 INFO:teuthology.orchestra.run.trial185.stdout:librados-devel-20.3.0-5439.ge371433f.el9.x86_64 330 kB/s | 124 kB 00:00 2026-02-23T13:04:49.146 INFO:teuthology.orchestra.run.trial185.stdout:-------------------------------------------------------------------------------- 2026-02-23T13:04:49.146 INFO:teuthology.orchestra.run.trial185.stdout:Total 329 kB/s | 124 kB 00:00 2026-02-23T13:04:49.146 INFO:teuthology.orchestra.run.trial185.stdout:Running transaction check 2026-02-23T13:04:49.148 INFO:teuthology.orchestra.run.trial185.stdout:Transaction check succeeded. 2026-02-23T13:04:49.148 INFO:teuthology.orchestra.run.trial185.stdout:Running transaction test 2026-02-23T13:04:49.179 INFO:teuthology.orchestra.run.trial185.stdout:Transaction test succeeded. 2026-02-23T13:04:49.179 INFO:teuthology.orchestra.run.trial185.stdout:Running transaction 2026-02-23T13:04:49.232 INFO:teuthology.orchestra.run.trial185.stdout: Preparing : 1/1 2026-02-23T13:04:49.248 INFO:teuthology.orchestra.run.trial185.stdout: Installing : librados-devel-2:20.3.0-5439.ge371433f.el9.x86_64 1/1 2026-02-23T13:04:49.330 INFO:teuthology.orchestra.run.trial185.stdout: Running scriptlet: librados-devel-2:20.3.0-5439.ge371433f.el9.x86_64 1/1 2026-02-23T13:04:49.388 INFO:teuthology.orchestra.run.trial185.stdout: Verifying : librados-devel-2:20.3.0-5439.ge371433f.el9.x86_64 1/1 2026-02-23T13:04:49.389 INFO:teuthology.orchestra.run.trial185.stdout: 2026-02-23T13:04:49.389 INFO:teuthology.orchestra.run.trial185.stdout:Installed: 2026-02-23T13:04:49.389 INFO:teuthology.orchestra.run.trial185.stdout: librados-devel-2:20.3.0-5439.ge371433f.el9.x86_64 2026-02-23T13:04:49.389 INFO:teuthology.orchestra.run.trial185.stdout: 2026-02-23T13:04:49.389 INFO:teuthology.orchestra.run.trial185.stdout:Complete! 2026-02-23T13:04:49.416 DEBUG:teuthology.orchestra.run.trial185:> sudo yum -y install libcephfs2 2026-02-23T13:04:49.728 INFO:teuthology.orchestra.run.trial185.stdout:Last metadata expiration check: 0:01:05 ago on Mon 23 Feb 2026 01:03:44 PM UTC. 2026-02-23T13:04:49.793 INFO:teuthology.orchestra.run.trial185.stdout:Package libcephfs2-2:20.3.0-5439.ge371433f.el9.x86_64 is already installed. 2026-02-23T13:04:49.811 INFO:teuthology.orchestra.run.trial185.stdout:Dependencies resolved. 2026-02-23T13:04:49.812 INFO:teuthology.orchestra.run.trial185.stdout:Nothing to do. 2026-02-23T13:04:49.812 INFO:teuthology.orchestra.run.trial185.stdout:Complete! 2026-02-23T13:04:49.832 DEBUG:teuthology.orchestra.run.trial185:> sudo yum -y install libcephfs-devel 2026-02-23T13:04:50.137 INFO:teuthology.orchestra.run.trial185.stdout:Last metadata expiration check: 0:01:06 ago on Mon 23 Feb 2026 01:03:44 PM UTC. 2026-02-23T13:04:50.221 INFO:teuthology.orchestra.run.trial185.stdout:Dependencies resolved. 2026-02-23T13:04:50.221 INFO:teuthology.orchestra.run.trial185.stdout:================================================================================ 2026-02-23T13:04:50.221 INFO:teuthology.orchestra.run.trial185.stdout: Package Arch Version Repo Size 2026-02-23T13:04:50.221 INFO:teuthology.orchestra.run.trial185.stdout:================================================================================ 2026-02-23T13:04:50.221 INFO:teuthology.orchestra.run.trial185.stdout:Installing: 2026-02-23T13:04:50.222 INFO:teuthology.orchestra.run.trial185.stdout: libcephfs-devel x86_64 2:20.3.0-5439.ge371433f.el9 ceph 37 k 2026-02-23T13:04:50.223 INFO:teuthology.orchestra.run.trial185.stdout:Installing dependencies: 2026-02-23T13:04:50.223 INFO:teuthology.orchestra.run.trial185.stdout: libcephfs-proxy2 x86_64 2:20.3.0-5439.ge371433f.el9 ceph 26 k 2026-02-23T13:04:50.223 INFO:teuthology.orchestra.run.trial185.stdout: 2026-02-23T13:04:50.223 INFO:teuthology.orchestra.run.trial185.stdout:Transaction Summary 2026-02-23T13:04:50.223 INFO:teuthology.orchestra.run.trial185.stdout:================================================================================ 2026-02-23T13:04:50.223 INFO:teuthology.orchestra.run.trial185.stdout:Install 2 Packages 2026-02-23T13:04:50.223 INFO:teuthology.orchestra.run.trial185.stdout: 2026-02-23T13:04:50.223 INFO:teuthology.orchestra.run.trial185.stdout:Total download size: 64 k 2026-02-23T13:04:50.223 INFO:teuthology.orchestra.run.trial185.stdout:Installed size: 231 k 2026-02-23T13:04:50.223 INFO:teuthology.orchestra.run.trial185.stdout:Downloading Packages: 2026-02-23T13:04:50.490 INFO:teuthology.orchestra.run.trial185.stdout:(1/2): libcephfs-devel-20.3.0-5439.ge371433f.el 139 kB/s | 37 kB 00:00 2026-02-23T13:04:50.491 INFO:teuthology.orchestra.run.trial185.stdout:(2/2): libcephfs-proxy2-20.3.0-5439.ge371433f.e 98 kB/s | 26 kB 00:00 2026-02-23T13:04:50.491 INFO:teuthology.orchestra.run.trial185.stdout:-------------------------------------------------------------------------------- 2026-02-23T13:04:50.491 INFO:teuthology.orchestra.run.trial185.stdout:Total 236 kB/s | 64 kB 00:00 2026-02-23T13:04:50.492 INFO:teuthology.orchestra.run.trial185.stdout:Running transaction check 2026-02-23T13:04:50.494 INFO:teuthology.orchestra.run.trial185.stdout:Transaction check succeeded. 2026-02-23T13:04:50.494 INFO:teuthology.orchestra.run.trial185.stdout:Running transaction test 2026-02-23T13:04:50.526 INFO:teuthology.orchestra.run.trial185.stdout:Transaction test succeeded. 2026-02-23T13:04:50.526 INFO:teuthology.orchestra.run.trial185.stdout:Running transaction 2026-02-23T13:04:50.575 INFO:teuthology.orchestra.run.trial185.stdout: Preparing : 1/1 2026-02-23T13:04:50.576 INFO:teuthology.orchestra.run.trial185.stdout: Installing : libcephfs-proxy2-2:20.3.0-5439.ge371433f.el9.x86_64 1/2 2026-02-23T13:04:50.590 INFO:teuthology.orchestra.run.trial185.stdout: Running scriptlet: libcephfs-proxy2-2:20.3.0-5439.ge371433f.el9.x86_64 1/2 2026-02-23T13:04:50.606 INFO:teuthology.orchestra.run.trial185.stdout: Installing : libcephfs-devel-2:20.3.0-5439.ge371433f.el9.x86_64 2/2 2026-02-23T13:04:50.663 INFO:teuthology.orchestra.run.trial185.stdout: Running scriptlet: libcephfs-devel-2:20.3.0-5439.ge371433f.el9.x86_64 2/2 2026-02-23T13:04:50.663 INFO:teuthology.orchestra.run.trial185.stdout: Verifying : libcephfs-devel-2:20.3.0-5439.ge371433f.el9.x86_64 1/2 2026-02-23T13:04:50.720 INFO:teuthology.orchestra.run.trial185.stdout: Verifying : libcephfs-proxy2-2:20.3.0-5439.ge371433f.el9.x86_64 2/2 2026-02-23T13:04:50.720 INFO:teuthology.orchestra.run.trial185.stdout: 2026-02-23T13:04:50.721 INFO:teuthology.orchestra.run.trial185.stdout:Installed: 2026-02-23T13:04:50.721 INFO:teuthology.orchestra.run.trial185.stdout: libcephfs-devel-2:20.3.0-5439.ge371433f.el9.x86_64 2026-02-23T13:04:50.721 INFO:teuthology.orchestra.run.trial185.stdout: libcephfs-proxy2-2:20.3.0-5439.ge371433f.el9.x86_64 2026-02-23T13:04:50.721 INFO:teuthology.orchestra.run.trial185.stdout: 2026-02-23T13:04:50.721 INFO:teuthology.orchestra.run.trial185.stdout:Complete! 2026-02-23T13:04:50.748 DEBUG:teuthology.orchestra.run.trial185:> sudo yum -y install librados2 2026-02-23T13:04:51.060 INFO:teuthology.orchestra.run.trial185.stdout:Last metadata expiration check: 0:01:07 ago on Mon 23 Feb 2026 01:03:44 PM UTC. 2026-02-23T13:04:51.126 INFO:teuthology.orchestra.run.trial185.stdout:Package librados2-2:20.3.0-5439.ge371433f.el9.x86_64 is already installed. 2026-02-23T13:04:51.145 INFO:teuthology.orchestra.run.trial185.stdout:Dependencies resolved. 2026-02-23T13:04:51.145 INFO:teuthology.orchestra.run.trial185.stdout:Nothing to do. 2026-02-23T13:04:51.145 INFO:teuthology.orchestra.run.trial185.stdout:Complete! 2026-02-23T13:04:51.167 DEBUG:teuthology.orchestra.run.trial185:> sudo yum -y install librbd1 2026-02-23T13:04:51.482 INFO:teuthology.orchestra.run.trial185.stdout:Last metadata expiration check: 0:01:07 ago on Mon 23 Feb 2026 01:03:44 PM UTC. 2026-02-23T13:04:51.547 INFO:teuthology.orchestra.run.trial185.stdout:Package librbd1-2:20.3.0-5439.ge371433f.el9.x86_64 is already installed. 2026-02-23T13:04:51.565 INFO:teuthology.orchestra.run.trial185.stdout:Dependencies resolved. 2026-02-23T13:04:51.565 INFO:teuthology.orchestra.run.trial185.stdout:Nothing to do. 2026-02-23T13:04:51.566 INFO:teuthology.orchestra.run.trial185.stdout:Complete! 2026-02-23T13:04:51.587 DEBUG:teuthology.orchestra.run.trial185:> sudo yum -y install python3-rados 2026-02-23T13:04:51.897 INFO:teuthology.orchestra.run.trial185.stdout:Last metadata expiration check: 0:01:07 ago on Mon 23 Feb 2026 01:03:44 PM UTC. 2026-02-23T13:04:51.961 INFO:teuthology.orchestra.run.trial185.stdout:Package python3-rados-2:20.3.0-5439.ge371433f.el9.x86_64 is already installed. 2026-02-23T13:04:51.980 INFO:teuthology.orchestra.run.trial185.stdout:Dependencies resolved. 2026-02-23T13:04:51.981 INFO:teuthology.orchestra.run.trial185.stdout:Nothing to do. 2026-02-23T13:04:51.981 INFO:teuthology.orchestra.run.trial185.stdout:Complete! 2026-02-23T13:04:52.001 DEBUG:teuthology.orchestra.run.trial185:> sudo yum -y install python3-rgw 2026-02-23T13:04:52.310 INFO:teuthology.orchestra.run.trial185.stdout:Last metadata expiration check: 0:01:08 ago on Mon 23 Feb 2026 01:03:44 PM UTC. 2026-02-23T13:04:52.376 INFO:teuthology.orchestra.run.trial185.stdout:Package python3-rgw-2:20.3.0-5439.ge371433f.el9.x86_64 is already installed. 2026-02-23T13:04:52.394 INFO:teuthology.orchestra.run.trial185.stdout:Dependencies resolved. 2026-02-23T13:04:52.395 INFO:teuthology.orchestra.run.trial185.stdout:Nothing to do. 2026-02-23T13:04:52.395 INFO:teuthology.orchestra.run.trial185.stdout:Complete! 2026-02-23T13:04:52.416 DEBUG:teuthology.orchestra.run.trial185:> sudo yum -y install python3-cephfs 2026-02-23T13:04:52.719 INFO:teuthology.orchestra.run.trial185.stdout:Last metadata expiration check: 0:01:08 ago on Mon 23 Feb 2026 01:03:44 PM UTC. 2026-02-23T13:04:52.783 INFO:teuthology.orchestra.run.trial185.stdout:Package python3-cephfs-2:20.3.0-5439.ge371433f.el9.x86_64 is already installed. 2026-02-23T13:04:52.801 INFO:teuthology.orchestra.run.trial185.stdout:Dependencies resolved. 2026-02-23T13:04:52.802 INFO:teuthology.orchestra.run.trial185.stdout:Nothing to do. 2026-02-23T13:04:52.802 INFO:teuthology.orchestra.run.trial185.stdout:Complete! 2026-02-23T13:04:52.823 DEBUG:teuthology.orchestra.run.trial185:> sudo yum -y install python3-rbd 2026-02-23T13:04:53.136 INFO:teuthology.orchestra.run.trial185.stdout:Last metadata expiration check: 0:01:09 ago on Mon 23 Feb 2026 01:03:44 PM UTC. 2026-02-23T13:04:53.201 INFO:teuthology.orchestra.run.trial185.stdout:Package python3-rbd-2:20.3.0-5439.ge371433f.el9.x86_64 is already installed. 2026-02-23T13:04:53.219 INFO:teuthology.orchestra.run.trial185.stdout:Dependencies resolved. 2026-02-23T13:04:53.220 INFO:teuthology.orchestra.run.trial185.stdout:Nothing to do. 2026-02-23T13:04:53.220 INFO:teuthology.orchestra.run.trial185.stdout:Complete! 2026-02-23T13:04:53.242 DEBUG:teuthology.orchestra.run.trial185:> sudo yum -y install rbd-fuse 2026-02-23T13:04:53.548 INFO:teuthology.orchestra.run.trial185.stdout:Last metadata expiration check: 0:01:09 ago on Mon 23 Feb 2026 01:03:44 PM UTC. 2026-02-23T13:04:53.632 INFO:teuthology.orchestra.run.trial185.stdout:Dependencies resolved. 2026-02-23T13:04:53.632 INFO:teuthology.orchestra.run.trial185.stdout:================================================================================ 2026-02-23T13:04:53.632 INFO:teuthology.orchestra.run.trial185.stdout: Package Architecture Version Repository Size 2026-02-23T13:04:53.632 INFO:teuthology.orchestra.run.trial185.stdout:================================================================================ 2026-02-23T13:04:53.632 INFO:teuthology.orchestra.run.trial185.stdout:Installing: 2026-02-23T13:04:53.633 INFO:teuthology.orchestra.run.trial185.stdout: rbd-fuse x86_64 2:20.3.0-5439.ge371433f.el9 ceph 86 k 2026-02-23T13:04:53.633 INFO:teuthology.orchestra.run.trial185.stdout: 2026-02-23T13:04:53.633 INFO:teuthology.orchestra.run.trial185.stdout:Transaction Summary 2026-02-23T13:04:53.633 INFO:teuthology.orchestra.run.trial185.stdout:================================================================================ 2026-02-23T13:04:53.633 INFO:teuthology.orchestra.run.trial185.stdout:Install 1 Package 2026-02-23T13:04:53.633 INFO:teuthology.orchestra.run.trial185.stdout: 2026-02-23T13:04:53.633 INFO:teuthology.orchestra.run.trial185.stdout:Total download size: 86 k 2026-02-23T13:04:53.633 INFO:teuthology.orchestra.run.trial185.stdout:Installed size: 236 k 2026-02-23T13:04:53.633 INFO:teuthology.orchestra.run.trial185.stdout:Downloading Packages: 2026-02-23T13:04:53.707 INFO:teuthology.orchestra.run.trial185.stdout:rbd-fuse-20.3.0-5439.ge371433f.el9.x86_64.rpm 1.1 MB/s | 86 kB 00:00 2026-02-23T13:04:53.707 INFO:teuthology.orchestra.run.trial185.stdout:-------------------------------------------------------------------------------- 2026-02-23T13:04:53.707 INFO:teuthology.orchestra.run.trial185.stdout:Total 1.1 MB/s | 86 kB 00:00 2026-02-23T13:04:53.707 INFO:teuthology.orchestra.run.trial185.stdout:Running transaction check 2026-02-23T13:04:53.711 INFO:teuthology.orchestra.run.trial185.stdout:Transaction check succeeded. 2026-02-23T13:04:53.711 INFO:teuthology.orchestra.run.trial185.stdout:Running transaction test 2026-02-23T13:04:53.742 INFO:teuthology.orchestra.run.trial185.stdout:Transaction test succeeded. 2026-02-23T13:04:53.742 INFO:teuthology.orchestra.run.trial185.stdout:Running transaction 2026-02-23T13:04:53.793 INFO:teuthology.orchestra.run.trial185.stdout: Preparing : 1/1 2026-02-23T13:04:53.809 INFO:teuthology.orchestra.run.trial185.stdout: Installing : rbd-fuse-2:20.3.0-5439.ge371433f.el9.x86_64 1/1 2026-02-23T13:04:53.889 INFO:teuthology.orchestra.run.trial185.stdout: Running scriptlet: rbd-fuse-2:20.3.0-5439.ge371433f.el9.x86_64 1/1 2026-02-23T13:04:53.946 INFO:teuthology.orchestra.run.trial185.stdout: Verifying : rbd-fuse-2:20.3.0-5439.ge371433f.el9.x86_64 1/1 2026-02-23T13:04:53.946 INFO:teuthology.orchestra.run.trial185.stdout: 2026-02-23T13:04:53.946 INFO:teuthology.orchestra.run.trial185.stdout:Installed: 2026-02-23T13:04:53.947 INFO:teuthology.orchestra.run.trial185.stdout: rbd-fuse-2:20.3.0-5439.ge371433f.el9.x86_64 2026-02-23T13:04:53.947 INFO:teuthology.orchestra.run.trial185.stdout: 2026-02-23T13:04:53.947 INFO:teuthology.orchestra.run.trial185.stdout:Complete! 2026-02-23T13:04:53.975 DEBUG:teuthology.orchestra.run.trial185:> sudo yum -y install rbd-mirror 2026-02-23T13:04:54.289 INFO:teuthology.orchestra.run.trial185.stdout:Last metadata expiration check: 0:01:10 ago on Mon 23 Feb 2026 01:03:44 PM UTC. 2026-02-23T13:04:54.343 INFO:teuthology.orchestra.run.trial068.stdout:(12/12): python3-scipy-1.9.3-2.el9.x86_64.rpm 1.7 MB/s | 19 MB 00:11 2026-02-23T13:04:54.345 INFO:teuthology.orchestra.run.trial068.stdout:-------------------------------------------------------------------------------- 2026-02-23T13:04:54.346 INFO:teuthology.orchestra.run.trial068.stdout:Total 2.6 MB/s | 43 MB 00:16 2026-02-23T13:04:54.371 INFO:teuthology.orchestra.run.trial185.stdout:Dependencies resolved. 2026-02-23T13:04:54.371 INFO:teuthology.orchestra.run.trial185.stdout:================================================================================ 2026-02-23T13:04:54.372 INFO:teuthology.orchestra.run.trial185.stdout: Package Arch Version Repo Size 2026-02-23T13:04:54.372 INFO:teuthology.orchestra.run.trial185.stdout:================================================================================ 2026-02-23T13:04:54.372 INFO:teuthology.orchestra.run.trial185.stdout:Installing: 2026-02-23T13:04:54.372 INFO:teuthology.orchestra.run.trial185.stdout: rbd-mirror x86_64 2:20.3.0-5439.ge371433f.el9 ceph 2.6 M 2026-02-23T13:04:54.372 INFO:teuthology.orchestra.run.trial185.stdout: 2026-02-23T13:04:54.372 INFO:teuthology.orchestra.run.trial185.stdout:Transaction Summary 2026-02-23T13:04:54.372 INFO:teuthology.orchestra.run.trial185.stdout:================================================================================ 2026-02-23T13:04:54.372 INFO:teuthology.orchestra.run.trial185.stdout:Install 1 Package 2026-02-23T13:04:54.372 INFO:teuthology.orchestra.run.trial185.stdout: 2026-02-23T13:04:54.372 INFO:teuthology.orchestra.run.trial185.stdout:Total download size: 2.6 M 2026-02-23T13:04:54.373 INFO:teuthology.orchestra.run.trial185.stdout:Installed size: 11 M 2026-02-23T13:04:54.373 INFO:teuthology.orchestra.run.trial185.stdout:Downloading Packages: 2026-02-23T13:04:54.505 INFO:teuthology.orchestra.run.trial068.stdout:Running transaction check 2026-02-23T13:04:54.512 INFO:teuthology.orchestra.run.trial185.stdout:rbd-mirror-20.3.0-5439.ge371433f.el9.x86_64.rpm 18 MB/s | 2.6 MB 00:00 2026-02-23T13:04:54.513 INFO:teuthology.orchestra.run.trial185.stdout:-------------------------------------------------------------------------------- 2026-02-23T13:04:54.513 INFO:teuthology.orchestra.run.trial185.stdout:Total 18 MB/s | 2.6 MB 00:00 2026-02-23T13:04:54.513 INFO:teuthology.orchestra.run.trial185.stdout:Running transaction check 2026-02-23T13:04:54.518 INFO:teuthology.orchestra.run.trial185.stdout:Transaction check succeeded. 2026-02-23T13:04:54.518 INFO:teuthology.orchestra.run.trial185.stdout:Running transaction test 2026-02-23T13:04:54.518 INFO:teuthology.orchestra.run.trial068.stdout:Transaction check succeeded. 2026-02-23T13:04:54.518 INFO:teuthology.orchestra.run.trial068.stdout:Running transaction test 2026-02-23T13:04:54.556 INFO:teuthology.orchestra.run.trial185.stdout:Transaction test succeeded. 2026-02-23T13:04:54.556 INFO:teuthology.orchestra.run.trial185.stdout:Running transaction 2026-02-23T13:04:54.719 INFO:teuthology.orchestra.run.trial185.stdout: Preparing : 1/1 2026-02-23T13:04:54.720 INFO:teuthology.orchestra.run.trial185.stdout: Installing : rbd-mirror-2:20.3.0-5439.ge371433f.el9.x86_64 1/1 2026-02-23T13:04:54.721 INFO:teuthology.orchestra.run.trial068.stdout:Transaction test succeeded. 2026-02-23T13:04:54.721 INFO:teuthology.orchestra.run.trial068.stdout:Running transaction 2026-02-23T13:04:54.743 INFO:teuthology.orchestra.run.trial185.stdout: Running scriptlet: rbd-mirror-2:20.3.0-5439.ge371433f.el9.x86_64 1/1 2026-02-23T13:04:54.743 INFO:teuthology.orchestra.run.trial185.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-23T13:04:54.743 INFO:teuthology.orchestra.run.trial185.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2026-02-23T13:04:54.743 INFO:teuthology.orchestra.run.trial185.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2026-02-23T13:04:54.743 INFO:teuthology.orchestra.run.trial185.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2026-02-23T13:04:54.743 INFO:teuthology.orchestra.run.trial185.stdout: 2026-02-23T13:04:54.939 INFO:teuthology.orchestra.run.trial068.stdout: Preparing : 1/1 2026-02-23T13:04:54.942 INFO:teuthology.orchestra.run.trial068.stdout: Installing : flexiblas-3.0.4-9.el9.x86_64 1/12 2026-02-23T13:04:54.948 INFO:teuthology.orchestra.run.trial068.stdout: Installing : libquadmath-11.5.0-14.el9.x86_64 2/12 2026-02-23T13:04:54.982 INFO:teuthology.orchestra.run.trial068.stdout: Installing : libgfortran-11.5.0-14.el9.x86_64 3/12 2026-02-23T13:04:54.986 INFO:teuthology.orchestra.run.trial068.stdout: Installing : python3-devel-3.9.25-3.el9.x86_64 4/12 2026-02-23T13:04:55.046 INFO:teuthology.orchestra.run.trial068.stdout: Installing : openblas-0.3.29-1.el9.x86_64 5/12 2026-02-23T13:04:55.048 INFO:teuthology.orchestra.run.trial068.stdout: Installing : openblas-openmp-0.3.29-1.el9.x86_64 6/12 2026-02-23T13:04:55.070 INFO:teuthology.orchestra.run.trial068.stdout: Installing : flexiblas-openblas-openmp-3.0.4-9.el9.x86_64 7/12 2026-02-23T13:04:55.073 INFO:teuthology.orchestra.run.trial185.stdout: Verifying : rbd-mirror-2:20.3.0-5439.ge371433f.el9.x86_64 1/1 2026-02-23T13:04:55.073 INFO:teuthology.orchestra.run.trial185.stdout: 2026-02-23T13:04:55.073 INFO:teuthology.orchestra.run.trial185.stdout:Installed: 2026-02-23T13:04:55.073 INFO:teuthology.orchestra.run.trial185.stdout: rbd-mirror-2:20.3.0-5439.ge371433f.el9.x86_64 2026-02-23T13:04:55.073 INFO:teuthology.orchestra.run.trial185.stdout: 2026-02-23T13:04:55.073 INFO:teuthology.orchestra.run.trial185.stdout:Complete! 2026-02-23T13:04:55.101 DEBUG:teuthology.orchestra.run.trial185:> sudo yum -y install rbd-nbd 2026-02-23T13:04:55.388 INFO:teuthology.orchestra.run.trial068.stdout: Installing : flexiblas-netlib-3.0.4-9.el9.x86_64 8/12 2026-02-23T13:04:55.417 INFO:teuthology.orchestra.run.trial185.stdout:Last metadata expiration check: 0:01:11 ago on Mon 23 Feb 2026 01:03:44 PM UTC. 2026-02-23T13:04:55.460 INFO:teuthology.orchestra.run.trial068.stdout: Installing : python3-numpy-1:1.23.5-2.el9.x86_64 9/12 2026-02-23T13:04:55.500 INFO:teuthology.orchestra.run.trial185.stdout:Dependencies resolved. 2026-02-23T13:04:55.501 INFO:teuthology.orchestra.run.trial185.stdout:================================================================================ 2026-02-23T13:04:55.501 INFO:teuthology.orchestra.run.trial185.stdout: Package Architecture Version Repository Size 2026-02-23T13:04:55.501 INFO:teuthology.orchestra.run.trial185.stdout:================================================================================ 2026-02-23T13:04:55.501 INFO:teuthology.orchestra.run.trial185.stdout:Installing: 2026-02-23T13:04:55.501 INFO:teuthology.orchestra.run.trial185.stdout: rbd-nbd x86_64 2:20.3.0-5439.ge371433f.el9 ceph 166 k 2026-02-23T13:04:55.501 INFO:teuthology.orchestra.run.trial185.stdout: 2026-02-23T13:04:55.501 INFO:teuthology.orchestra.run.trial185.stdout:Transaction Summary 2026-02-23T13:04:55.501 INFO:teuthology.orchestra.run.trial185.stdout:================================================================================ 2026-02-23T13:04:55.501 INFO:teuthology.orchestra.run.trial185.stdout:Install 1 Package 2026-02-23T13:04:55.501 INFO:teuthology.orchestra.run.trial185.stdout: 2026-02-23T13:04:55.502 INFO:teuthology.orchestra.run.trial185.stdout:Total download size: 166 k 2026-02-23T13:04:55.502 INFO:teuthology.orchestra.run.trial185.stdout:Installed size: 496 k 2026-02-23T13:04:55.502 INFO:teuthology.orchestra.run.trial185.stdout:Downloading Packages: 2026-02-23T13:04:55.588 INFO:teuthology.orchestra.run.trial185.stdout:rbd-nbd-20.3.0-5439.ge371433f.el9.x86_64.rpm 1.9 MB/s | 166 kB 00:00 2026-02-23T13:04:55.588 INFO:teuthology.orchestra.run.trial185.stdout:-------------------------------------------------------------------------------- 2026-02-23T13:04:55.588 INFO:teuthology.orchestra.run.trial185.stdout:Total 1.9 MB/s | 166 kB 00:00 2026-02-23T13:04:55.588 INFO:teuthology.orchestra.run.trial185.stdout:Running transaction check 2026-02-23T13:04:55.593 INFO:teuthology.orchestra.run.trial185.stdout:Transaction check succeeded. 2026-02-23T13:04:55.593 INFO:teuthology.orchestra.run.trial185.stdout:Running transaction test 2026-02-23T13:04:55.624 INFO:teuthology.orchestra.run.trial185.stdout:Transaction test succeeded. 2026-02-23T13:04:55.624 INFO:teuthology.orchestra.run.trial185.stdout:Running transaction 2026-02-23T13:04:55.679 INFO:teuthology.orchestra.run.trial185.stdout: Preparing : 1/1 2026-02-23T13:04:55.695 INFO:teuthology.orchestra.run.trial185.stdout: Installing : rbd-nbd-2:20.3.0-5439.ge371433f.el9.x86_64 1/1 2026-02-23T13:04:55.772 INFO:teuthology.orchestra.run.trial185.stdout: Running scriptlet: rbd-nbd-2:20.3.0-5439.ge371433f.el9.x86_64 1/1 2026-02-23T13:04:55.830 INFO:teuthology.orchestra.run.trial185.stdout: Verifying : rbd-nbd-2:20.3.0-5439.ge371433f.el9.x86_64 1/1 2026-02-23T13:04:55.830 INFO:teuthology.orchestra.run.trial185.stdout: 2026-02-23T13:04:55.830 INFO:teuthology.orchestra.run.trial185.stdout:Installed: 2026-02-23T13:04:55.830 INFO:teuthology.orchestra.run.trial185.stdout: rbd-nbd-2:20.3.0-5439.ge371433f.el9.x86_64 2026-02-23T13:04:55.830 INFO:teuthology.orchestra.run.trial185.stdout: 2026-02-23T13:04:55.830 INFO:teuthology.orchestra.run.trial185.stdout:Complete! 2026-02-23T13:04:55.859 DEBUG:teuthology.orchestra.run.trial185:> sudo yum -y install ceph-osd-crimson 2026-02-23T13:04:56.121 INFO:teuthology.orchestra.run.trial068.stdout: Installing : python3-numpy-f2py-1:1.23.5-2.el9.x86_64 10/12 2026-02-23T13:04:56.171 INFO:teuthology.orchestra.run.trial185.stdout:Last metadata expiration check: 0:01:12 ago on Mon 23 Feb 2026 01:03:44 PM UTC. 2026-02-23T13:04:56.236 INFO:teuthology.orchestra.run.trial185.stdout:Package ceph-osd-crimson-2:20.3.0-5439.ge371433f.el9.x86_64 is already installed. 2026-02-23T13:04:56.254 INFO:teuthology.orchestra.run.trial185.stdout:Dependencies resolved. 2026-02-23T13:04:56.254 INFO:teuthology.orchestra.run.trial185.stdout:Nothing to do. 2026-02-23T13:04:56.254 INFO:teuthology.orchestra.run.trial185.stdout:Complete! 2026-02-23T13:04:56.276 DEBUG:teuthology.orchestra.run.trial185:> sudo yum -y install ceph-osd-crimson-debuginfo 2026-02-23T13:04:56.590 INFO:teuthology.orchestra.run.trial185.stdout:Last metadata expiration check: 0:01:12 ago on Mon 23 Feb 2026 01:03:44 PM UTC. 2026-02-23T13:04:56.597 INFO:teuthology.orchestra.run.trial068.stdout: Installing : python3-scipy-1.9.3-2.el9.x86_64 11/12 2026-02-23T13:04:56.598 INFO:teuthology.orchestra.run.trial068.stdout: Installing : ceph-mgr-diskprediction-local-2:20.3.0-5439.ge3714 12/12 2026-02-23T13:04:56.674 INFO:teuthology.orchestra.run.trial185.stdout:Dependencies resolved. 2026-02-23T13:04:56.675 INFO:teuthology.orchestra.run.trial185.stdout:================================================================================ 2026-02-23T13:04:56.675 INFO:teuthology.orchestra.run.trial185.stdout: Package Arch Version Repo Size 2026-02-23T13:04:56.675 INFO:teuthology.orchestra.run.trial185.stdout:================================================================================ 2026-02-23T13:04:56.675 INFO:teuthology.orchestra.run.trial185.stdout:Installing: 2026-02-23T13:04:56.675 INFO:teuthology.orchestra.run.trial185.stdout: ceph-osd-crimson-debuginfo x86_64 2:20.3.0-5439.ge371433f.el9 ceph 429 M 2026-02-23T13:04:56.675 INFO:teuthology.orchestra.run.trial185.stdout: 2026-02-23T13:04:56.675 INFO:teuthology.orchestra.run.trial185.stdout:Transaction Summary 2026-02-23T13:04:56.675 INFO:teuthology.orchestra.run.trial185.stdout:================================================================================ 2026-02-23T13:04:56.676 INFO:teuthology.orchestra.run.trial185.stdout:Install 1 Package 2026-02-23T13:04:56.676 INFO:teuthology.orchestra.run.trial185.stdout: 2026-02-23T13:04:56.676 INFO:teuthology.orchestra.run.trial185.stdout:Total download size: 429 M 2026-02-23T13:04:56.676 INFO:teuthology.orchestra.run.trial185.stdout:Installed size: 2.6 G 2026-02-23T13:04:56.676 INFO:teuthology.orchestra.run.trial185.stdout:Downloading Packages: 2026-02-23T13:04:56.872 INFO:teuthology.orchestra.run.trial068.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:20.3.0-5439.ge3714 12/12 2026-02-23T13:04:56.873 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : ceph-mgr-diskprediction-local-2:20.3.0-5439.ge3714 1/12 2026-02-23T13:04:56.873 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : libgfortran-11.5.0-14.el9.x86_64 2/12 2026-02-23T13:04:56.873 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : libquadmath-11.5.0-14.el9.x86_64 3/12 2026-02-23T13:04:56.873 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : flexiblas-3.0.4-9.el9.x86_64 4/12 2026-02-23T13:04:56.873 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : flexiblas-netlib-3.0.4-9.el9.x86_64 5/12 2026-02-23T13:04:56.873 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-9.el9.x86_64 6/12 2026-02-23T13:04:56.873 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : openblas-0.3.29-1.el9.x86_64 7/12 2026-02-23T13:04:56.873 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : openblas-openmp-0.3.29-1.el9.x86_64 8/12 2026-02-23T13:04:56.873 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : python3-devel-3.9.25-3.el9.x86_64 9/12 2026-02-23T13:04:56.873 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : python3-numpy-1:1.23.5-2.el9.x86_64 10/12 2026-02-23T13:04:56.873 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : python3-numpy-f2py-1:1.23.5-2.el9.x86_64 11/12 2026-02-23T13:04:56.931 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : python3-scipy-1.9.3-2.el9.x86_64 12/12 2026-02-23T13:04:56.931 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T13:04:56.931 INFO:teuthology.orchestra.run.trial068.stdout:Installed: 2026-02-23T13:04:56.931 INFO:teuthology.orchestra.run.trial068.stdout: ceph-mgr-diskprediction-local-2:20.3.0-5439.ge371433f.el9.noarch 2026-02-23T13:04:56.932 INFO:teuthology.orchestra.run.trial068.stdout: flexiblas-3.0.4-9.el9.x86_64 2026-02-23T13:04:56.932 INFO:teuthology.orchestra.run.trial068.stdout: flexiblas-netlib-3.0.4-9.el9.x86_64 2026-02-23T13:04:56.932 INFO:teuthology.orchestra.run.trial068.stdout: flexiblas-openblas-openmp-3.0.4-9.el9.x86_64 2026-02-23T13:04:56.932 INFO:teuthology.orchestra.run.trial068.stdout: libgfortran-11.5.0-14.el9.x86_64 2026-02-23T13:04:56.932 INFO:teuthology.orchestra.run.trial068.stdout: libquadmath-11.5.0-14.el9.x86_64 2026-02-23T13:04:56.932 INFO:teuthology.orchestra.run.trial068.stdout: openblas-0.3.29-1.el9.x86_64 2026-02-23T13:04:56.932 INFO:teuthology.orchestra.run.trial068.stdout: openblas-openmp-0.3.29-1.el9.x86_64 2026-02-23T13:04:56.932 INFO:teuthology.orchestra.run.trial068.stdout: python3-devel-3.9.25-3.el9.x86_64 2026-02-23T13:04:56.932 INFO:teuthology.orchestra.run.trial068.stdout: python3-numpy-1:1.23.5-2.el9.x86_64 2026-02-23T13:04:56.932 INFO:teuthology.orchestra.run.trial068.stdout: python3-numpy-f2py-1:1.23.5-2.el9.x86_64 2026-02-23T13:04:56.932 INFO:teuthology.orchestra.run.trial068.stdout: python3-scipy-1.9.3-2.el9.x86_64 2026-02-23T13:04:56.933 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T13:04:56.933 INFO:teuthology.orchestra.run.trial068.stdout:Complete! 2026-02-23T13:04:56.969 DEBUG:teuthology.orchestra.run.trial068:> sudo yum -y install ceph-mgr-rook 2026-02-23T13:04:57.286 INFO:teuthology.orchestra.run.trial068.stdout:Last metadata expiration check: 0:01:07 ago on Mon 23 Feb 2026 01:03:50 PM UTC. 2026-02-23T13:04:57.373 INFO:teuthology.orchestra.run.trial068.stdout:Dependencies resolved. 2026-02-23T13:04:57.374 INFO:teuthology.orchestra.run.trial068.stdout:================================================================================ 2026-02-23T13:04:57.374 INFO:teuthology.orchestra.run.trial068.stdout: Package Arch Version Repository Size 2026-02-23T13:04:57.374 INFO:teuthology.orchestra.run.trial068.stdout:================================================================================ 2026-02-23T13:04:57.375 INFO:teuthology.orchestra.run.trial068.stdout:Installing: 2026-02-23T13:04:57.375 INFO:teuthology.orchestra.run.trial068.stdout: ceph-mgr-rook noarch 2:20.3.0-5439.ge371433f.el9 ceph-noarch 50 k 2026-02-23T13:04:57.375 INFO:teuthology.orchestra.run.trial068.stdout:Installing dependencies: 2026-02-23T13:04:57.375 INFO:teuthology.orchestra.run.trial068.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2026-02-23T13:04:57.375 INFO:teuthology.orchestra.run.trial068.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2026-02-23T13:04:57.375 INFO:teuthology.orchestra.run.trial068.stdout: python3-google-auth noarch 1:2.45.0-1.el9 epel 254 k 2026-02-23T13:04:57.375 INFO:teuthology.orchestra.run.trial068.stdout: python3-kubernetes noarch 1:26.1.0-3.el9 epel 1.0 M 2026-02-23T13:04:57.375 INFO:teuthology.orchestra.run.trial068.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2026-02-23T13:04:57.376 INFO:teuthology.orchestra.run.trial068.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2026-02-23T13:04:57.376 INFO:teuthology.orchestra.run.trial068.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2026-02-23T13:04:57.376 INFO:teuthology.orchestra.run.trial068.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2026-02-23T13:04:57.376 INFO:teuthology.orchestra.run.trial068.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2026-02-23T13:04:57.376 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T13:04:57.376 INFO:teuthology.orchestra.run.trial068.stdout:Transaction Summary 2026-02-23T13:04:57.376 INFO:teuthology.orchestra.run.trial068.stdout:================================================================================ 2026-02-23T13:04:57.376 INFO:teuthology.orchestra.run.trial068.stdout:Install 10 Packages 2026-02-23T13:04:57.376 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T13:04:57.376 INFO:teuthology.orchestra.run.trial068.stdout:Total download size: 2.0 M 2026-02-23T13:04:57.376 INFO:teuthology.orchestra.run.trial068.stdout:Installed size: 25 M 2026-02-23T13:04:57.377 INFO:teuthology.orchestra.run.trial068.stdout:Downloading Packages: 2026-02-23T13:04:57.612 INFO:teuthology.orchestra.run.trial068.stdout:(1/10): python3-pyasn1-modules-0.4.8-6.el9.noar 3.5 MB/s | 279 kB 00:00 2026-02-23T13:04:57.613 INFO:teuthology.orchestra.run.trial068.stdout:(2/10): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.9 MB/s | 159 kB 00:00 2026-02-23T13:04:57.624 INFO:teuthology.orchestra.run.trial068.stdout:(3/10): python3-requests-oauthlib-1.3.0-12.el9. 4.6 MB/s | 54 kB 00:00 2026-02-23T13:04:57.676 INFO:teuthology.orchestra.run.trial068.stdout:(4/10): python3-cachetools-4.2.4-1.el9.noarch.r 512 kB/s | 32 kB 00:00 2026-02-23T13:04:57.685 INFO:teuthology.orchestra.run.trial068.stdout:(5/10): python3-certifi-2023.05.07-4.el9.noarch 231 kB/s | 14 kB 00:00 2026-02-23T13:04:57.710 INFO:teuthology.orchestra.run.trial068.stdout:(6/10): python3-google-auth-2.45.0-1.el9.noarch 7.5 MB/s | 254 kB 00:00 2026-02-23T13:04:57.728 INFO:teuthology.orchestra.run.trial068.stdout:(7/10): python3-rsa-4.9-2.el9.noarch.rpm 3.2 MB/s | 59 kB 00:00 2026-02-23T13:04:57.746 INFO:teuthology.orchestra.run.trial068.stdout:(8/10): python3-kubernetes-26.1.0-3.el9.noarch. 17 MB/s | 1.0 MB 00:00 2026-02-23T13:04:57.746 INFO:teuthology.orchestra.run.trial068.stdout:(9/10): python3-websocket-client-1.2.3-2.el9.no 4.8 MB/s | 90 kB 00:00 2026-02-23T13:04:57.827 INFO:teuthology.orchestra.run.trial068.stdout:(10/10): ceph-mgr-rook-20.3.0-5439.ge371433f.el 171 kB/s | 50 kB 00:00 2026-02-23T13:04:57.828 INFO:teuthology.orchestra.run.trial068.stdout:-------------------------------------------------------------------------------- 2026-02-23T13:04:57.828 INFO:teuthology.orchestra.run.trial068.stdout:Total 4.4 MB/s | 2.0 MB 00:00 2026-02-23T13:04:57.857 INFO:teuthology.orchestra.run.trial068.stdout:Running transaction check 2026-02-23T13:04:57.865 INFO:teuthology.orchestra.run.trial068.stdout:Transaction check succeeded. 2026-02-23T13:04:57.865 INFO:teuthology.orchestra.run.trial068.stdout:Running transaction test 2026-02-23T13:04:57.905 INFO:teuthology.orchestra.run.trial068.stdout:Transaction test succeeded. 2026-02-23T13:04:57.905 INFO:teuthology.orchestra.run.trial068.stdout:Running transaction 2026-02-23T13:04:57.980 INFO:teuthology.orchestra.run.trial068.stdout: Preparing : 1/1 2026-02-23T13:04:58.032 INFO:teuthology.orchestra.run.trial068.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/10 2026-02-23T13:04:58.045 INFO:teuthology.orchestra.run.trial068.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/10 2026-02-23T13:04:58.060 INFO:teuthology.orchestra.run.trial068.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/10 2026-02-23T13:04:58.064 INFO:teuthology.orchestra.run.trial068.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/10 2026-02-23T13:04:58.072 INFO:teuthology.orchestra.run.trial068.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 5/10 2026-02-23T13:04:58.112 INFO:teuthology.orchestra.run.trial068.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/10 2026-02-23T13:04:58.125 INFO:teuthology.orchestra.run.trial068.stdout: Installing : python3-google-auth-1:2.45.0-1.el9.noarch 7/10 2026-02-23T13:04:58.452 INFO:teuthology.orchestra.run.trial068.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 8/10 2026-02-23T13:04:58.467 INFO:teuthology.orchestra.run.trial068.stdout: Installing : python3-kubernetes-1:26.1.0-3.el9.noarch 9/10 2026-02-23T13:04:58.468 INFO:teuthology.orchestra.run.trial068.stdout: Installing : ceph-mgr-rook-2:20.3.0-5439.ge371433f.el9.noarch 10/10 2026-02-23T13:04:58.581 INFO:teuthology.orchestra.run.trial068.stdout: Running scriptlet: ceph-mgr-rook-2:20.3.0-5439.ge371433f.el9.noarch 10/10 2026-02-23T13:04:58.581 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : ceph-mgr-rook-2:20.3.0-5439.ge371433f.el9.noarch 1/10 2026-02-23T13:04:58.581 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 2/10 2026-02-23T13:04:58.581 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 3/10 2026-02-23T13:04:58.581 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 4/10 2026-02-23T13:04:58.581 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 5/10 2026-02-23T13:04:58.581 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 6/10 2026-02-23T13:04:58.581 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : python3-google-auth-1:2.45.0-1.el9.noarch 7/10 2026-02-23T13:04:58.581 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : python3-kubernetes-1:26.1.0-3.el9.noarch 8/10 2026-02-23T13:04:58.581 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 9/10 2026-02-23T13:04:58.638 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 10/10 2026-02-23T13:04:58.638 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T13:04:58.638 INFO:teuthology.orchestra.run.trial068.stdout:Installed: 2026-02-23T13:04:58.638 INFO:teuthology.orchestra.run.trial068.stdout: ceph-mgr-rook-2:20.3.0-5439.ge371433f.el9.noarch 2026-02-23T13:04:58.638 INFO:teuthology.orchestra.run.trial068.stdout: python3-cachetools-4.2.4-1.el9.noarch 2026-02-23T13:04:58.638 INFO:teuthology.orchestra.run.trial068.stdout: python3-certifi-2023.05.07-4.el9.noarch 2026-02-23T13:04:58.638 INFO:teuthology.orchestra.run.trial068.stdout: python3-google-auth-1:2.45.0-1.el9.noarch 2026-02-23T13:04:58.638 INFO:teuthology.orchestra.run.trial068.stdout: python3-kubernetes-1:26.1.0-3.el9.noarch 2026-02-23T13:04:58.639 INFO:teuthology.orchestra.run.trial068.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2026-02-23T13:04:58.639 INFO:teuthology.orchestra.run.trial068.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2026-02-23T13:04:58.639 INFO:teuthology.orchestra.run.trial068.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2026-02-23T13:04:58.639 INFO:teuthology.orchestra.run.trial068.stdout: python3-rsa-4.9-2.el9.noarch 2026-02-23T13:04:58.639 INFO:teuthology.orchestra.run.trial068.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2026-02-23T13:04:58.639 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T13:04:58.639 INFO:teuthology.orchestra.run.trial068.stdout:Complete! 2026-02-23T13:04:58.669 DEBUG:teuthology.orchestra.run.trial068:> sudo yum -y install ceph-mgr-cephadm 2026-02-23T13:04:58.973 INFO:teuthology.orchestra.run.trial068.stdout:Last metadata expiration check: 0:01:08 ago on Mon 23 Feb 2026 01:03:50 PM UTC. 2026-02-23T13:04:59.055 INFO:teuthology.orchestra.run.trial068.stdout:Dependencies resolved. 2026-02-23T13:04:59.056 INFO:teuthology.orchestra.run.trial068.stdout:================================================================================ 2026-02-23T13:04:59.057 INFO:teuthology.orchestra.run.trial068.stdout: Package Arch Version Repository Size 2026-02-23T13:04:59.057 INFO:teuthology.orchestra.run.trial068.stdout:================================================================================ 2026-02-23T13:04:59.057 INFO:teuthology.orchestra.run.trial068.stdout:Installing: 2026-02-23T13:04:59.057 INFO:teuthology.orchestra.run.trial068.stdout: ceph-mgr-cephadm noarch 2:20.3.0-5439.ge371433f.el9 ceph-noarch 184 k 2026-02-23T13:04:59.057 INFO:teuthology.orchestra.run.trial068.stdout:Installing dependencies: 2026-02-23T13:04:59.057 INFO:teuthology.orchestra.run.trial068.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2026-02-23T13:04:59.057 INFO:teuthology.orchestra.run.trial068.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2026-02-23T13:04:59.057 INFO:teuthology.orchestra.run.trial068.stdout: python3-jinja2 noarch 2.11.3-8.el9 appstream 249 k 2026-02-23T13:04:59.057 INFO:teuthology.orchestra.run.trial068.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2026-02-23T13:04:59.058 INFO:teuthology.orchestra.run.trial068.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2026-02-23T13:04:59.058 INFO:teuthology.orchestra.run.trial068.stdout: python3-typing-extensions noarch 4.15.0-1.el9 epel 86 k 2026-02-23T13:04:59.058 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T13:04:59.058 INFO:teuthology.orchestra.run.trial068.stdout:Transaction Summary 2026-02-23T13:04:59.058 INFO:teuthology.orchestra.run.trial068.stdout:================================================================================ 2026-02-23T13:04:59.058 INFO:teuthology.orchestra.run.trial068.stdout:Install 7 Packages 2026-02-23T13:04:59.058 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T13:04:59.058 INFO:teuthology.orchestra.run.trial068.stdout:Total download size: 7.1 M 2026-02-23T13:04:59.058 INFO:teuthology.orchestra.run.trial068.stdout:Installed size: 34 M 2026-02-23T13:04:59.058 INFO:teuthology.orchestra.run.trial068.stdout:Downloading Packages: 2026-02-23T13:04:59.264 INFO:teuthology.orchestra.run.trial068.stdout:(1/7): python3-jinja2-2.11.3-8.el9.noarch.rpm 7.5 MB/s | 249 kB 00:00 2026-02-23T13:04:59.270 INFO:teuthology.orchestra.run.trial068.stdout:(2/7): python3-markupsafe-1.1.1-12.el9.x86_64.r 6.3 MB/s | 35 kB 00:00 2026-02-23T13:04:59.286 INFO:teuthology.orchestra.run.trial068.stdout:(3/7): python3-babel-2.9.1-2.el9.noarch.rpm 111 MB/s | 6.0 MB 00:00 2026-02-23T13:04:59.575 INFO:teuthology.orchestra.run.trial068.stdout:(4/7): ceph-mgr-cephadm-20.3.0-5439.ge371433f.e 538 kB/s | 184 kB 00:00 2026-02-23T13:04:59.606 INFO:teuthology.orchestra.run.trial068.stdout:(5/7): python3-natsort-7.1.1-5.el9.noarch.rpm 182 kB/s | 58 kB 00:00 2026-02-23T13:04:59.858 INFO:teuthology.orchestra.run.trial068.stdout:(6/7): python3-asyncssh-2.13.2-5.el9.noarch.rpm 932 kB/s | 548 kB 00:00 2026-02-23T13:04:59.957 INFO:teuthology.orchestra.run.trial068.stdout:(7/7): python3-typing-extensions-4.15.0-1.el9.n 227 kB/s | 86 kB 00:00 2026-02-23T13:04:59.958 INFO:teuthology.orchestra.run.trial068.stdout:-------------------------------------------------------------------------------- 2026-02-23T13:04:59.958 INFO:teuthology.orchestra.run.trial068.stdout:Total 7.9 MB/s | 7.1 MB 00:00 2026-02-23T13:04:59.996 INFO:teuthology.orchestra.run.trial068.stdout:Running transaction check 2026-02-23T13:05:00.002 INFO:teuthology.orchestra.run.trial068.stdout:Transaction check succeeded. 2026-02-23T13:05:00.002 INFO:teuthology.orchestra.run.trial068.stdout:Running transaction test 2026-02-23T13:05:00.076 INFO:teuthology.orchestra.run.trial068.stdout:Transaction test succeeded. 2026-02-23T13:05:00.076 INFO:teuthology.orchestra.run.trial068.stdout:Running transaction 2026-02-23T13:05:00.166 INFO:teuthology.orchestra.run.trial068.stdout: Preparing : 1/1 2026-02-23T13:05:00.215 INFO:teuthology.orchestra.run.trial068.stdout: Installing : python3-typing-extensions-4.15.0-1.el9.noarch 1/7 2026-02-23T13:05:00.225 INFO:teuthology.orchestra.run.trial068.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/7 2026-02-23T13:05:00.231 INFO:teuthology.orchestra.run.trial068.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/7 2026-02-23T13:05:00.454 INFO:teuthology.orchestra.run.trial068.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 4/7 2026-02-23T13:05:00.477 INFO:teuthology.orchestra.run.trial068.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 5/7 2026-02-23T13:05:00.502 INFO:teuthology.orchestra.run.trial068.stdout: Installing : python3-jinja2-2.11.3-8.el9.noarch 6/7 2026-02-23T13:05:00.503 INFO:teuthology.orchestra.run.trial068.stdout: Installing : ceph-mgr-cephadm-2:20.3.0-5439.ge371433f.el9.noarch 7/7 2026-02-23T13:05:00.610 INFO:teuthology.orchestra.run.trial068.stdout: Running scriptlet: ceph-mgr-cephadm-2:20.3.0-5439.ge371433f.el9.noarch 7/7 2026-02-23T13:05:00.611 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : ceph-mgr-cephadm-2:20.3.0-5439.ge371433f.el9.noarch 1/7 2026-02-23T13:05:00.611 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/7 2026-02-23T13:05:00.611 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : python3-jinja2-2.11.3-8.el9.noarch 3/7 2026-02-23T13:05:00.611 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 4/7 2026-02-23T13:05:00.611 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 5/7 2026-02-23T13:05:00.611 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 6/7 2026-02-23T13:05:00.668 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : python3-typing-extensions-4.15.0-1.el9.noarch 7/7 2026-02-23T13:05:00.669 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T13:05:00.669 INFO:teuthology.orchestra.run.trial068.stdout:Installed: 2026-02-23T13:05:00.669 INFO:teuthology.orchestra.run.trial068.stdout: ceph-mgr-cephadm-2:20.3.0-5439.ge371433f.el9.noarch 2026-02-23T13:05:00.669 INFO:teuthology.orchestra.run.trial068.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2026-02-23T13:05:00.669 INFO:teuthology.orchestra.run.trial068.stdout: python3-babel-2.9.1-2.el9.noarch 2026-02-23T13:05:00.669 INFO:teuthology.orchestra.run.trial068.stdout: python3-jinja2-2.11.3-8.el9.noarch 2026-02-23T13:05:00.669 INFO:teuthology.orchestra.run.trial068.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2026-02-23T13:05:00.669 INFO:teuthology.orchestra.run.trial068.stdout: python3-natsort-7.1.1-5.el9.noarch 2026-02-23T13:05:00.669 INFO:teuthology.orchestra.run.trial068.stdout: python3-typing-extensions-4.15.0-1.el9.noarch 2026-02-23T13:05:00.669 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T13:05:00.669 INFO:teuthology.orchestra.run.trial068.stdout:Complete! 2026-02-23T13:05:00.699 DEBUG:teuthology.orchestra.run.trial068:> sudo yum -y install ceph-osd 2026-02-23T13:05:01.008 INFO:teuthology.orchestra.run.trial068.stdout:Last metadata expiration check: 0:01:11 ago on Mon 23 Feb 2026 01:03:50 PM UTC. 2026-02-23T13:05:01.073 INFO:teuthology.orchestra.run.trial068.stdout:Package ceph-osd-2:20.3.0-5439.ge371433f.el9.x86_64 is already installed. 2026-02-23T13:05:01.091 INFO:teuthology.orchestra.run.trial068.stdout:Dependencies resolved. 2026-02-23T13:05:01.092 INFO:teuthology.orchestra.run.trial068.stdout:Nothing to do. 2026-02-23T13:05:01.092 INFO:teuthology.orchestra.run.trial068.stdout:Complete! 2026-02-23T13:05:01.113 DEBUG:teuthology.orchestra.run.trial068:> sudo yum -y install ceph-osd-classic 2026-02-23T13:05:01.420 INFO:teuthology.orchestra.run.trial068.stdout:Last metadata expiration check: 0:01:11 ago on Mon 23 Feb 2026 01:03:50 PM UTC. 2026-02-23T13:05:01.486 INFO:teuthology.orchestra.run.trial068.stdout:Package ceph-osd-classic-2:20.3.0-5439.ge371433f.el9.x86_64 is already installed. 2026-02-23T13:05:01.504 INFO:teuthology.orchestra.run.trial068.stdout:Dependencies resolved. 2026-02-23T13:05:01.505 INFO:teuthology.orchestra.run.trial068.stdout:Nothing to do. 2026-02-23T13:05:01.505 INFO:teuthology.orchestra.run.trial068.stdout:Complete! 2026-02-23T13:05:01.526 DEBUG:teuthology.orchestra.run.trial068:> sudo yum -y install ceph-fuse 2026-02-23T13:05:01.837 INFO:teuthology.orchestra.run.trial068.stdout:Last metadata expiration check: 0:01:11 ago on Mon 23 Feb 2026 01:03:50 PM UTC. 2026-02-23T13:05:01.922 INFO:teuthology.orchestra.run.trial068.stdout:Dependencies resolved. 2026-02-23T13:05:01.922 INFO:teuthology.orchestra.run.trial068.stdout:================================================================================ 2026-02-23T13:05:01.922 INFO:teuthology.orchestra.run.trial068.stdout: Package Arch Version Repo Size 2026-02-23T13:05:01.922 INFO:teuthology.orchestra.run.trial068.stdout:================================================================================ 2026-02-23T13:05:01.922 INFO:teuthology.orchestra.run.trial068.stdout:Installing: 2026-02-23T13:05:01.923 INFO:teuthology.orchestra.run.trial068.stdout: ceph-fuse x86_64 2:20.3.0-5439.ge371433f.el9 ceph 914 k 2026-02-23T13:05:01.923 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T13:05:01.923 INFO:teuthology.orchestra.run.trial068.stdout:Transaction Summary 2026-02-23T13:05:01.923 INFO:teuthology.orchestra.run.trial068.stdout:================================================================================ 2026-02-23T13:05:01.923 INFO:teuthology.orchestra.run.trial068.stdout:Install 1 Package 2026-02-23T13:05:01.923 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T13:05:01.923 INFO:teuthology.orchestra.run.trial068.stdout:Total download size: 914 k 2026-02-23T13:05:01.923 INFO:teuthology.orchestra.run.trial068.stdout:Installed size: 2.9 M 2026-02-23T13:05:01.923 INFO:teuthology.orchestra.run.trial068.stdout:Downloading Packages: 2026-02-23T13:05:02.501 INFO:teuthology.orchestra.run.trial068.stdout:ceph-fuse-20.3.0-5439.ge371433f.el9.x86_64.rpm 1.5 MB/s | 914 kB 00:00 2026-02-23T13:05:02.502 INFO:teuthology.orchestra.run.trial068.stdout:-------------------------------------------------------------------------------- 2026-02-23T13:05:02.502 INFO:teuthology.orchestra.run.trial068.stdout:Total 1.5 MB/s | 914 kB 00:00 2026-02-23T13:05:02.502 INFO:teuthology.orchestra.run.trial068.stdout:Running transaction check 2026-02-23T13:05:02.506 INFO:teuthology.orchestra.run.trial068.stdout:Transaction check succeeded. 2026-02-23T13:05:02.506 INFO:teuthology.orchestra.run.trial068.stdout:Running transaction test 2026-02-23T13:05:02.542 INFO:teuthology.orchestra.run.trial068.stdout:Transaction test succeeded. 2026-02-23T13:05:02.542 INFO:teuthology.orchestra.run.trial068.stdout:Running transaction 2026-02-23T13:05:02.632 INFO:teuthology.orchestra.run.trial068.stdout: Preparing : 1/1 2026-02-23T13:05:02.649 INFO:teuthology.orchestra.run.trial068.stdout: Installing : ceph-fuse-2:20.3.0-5439.ge371433f.el9.x86_64 1/1 2026-02-23T13:05:02.860 INFO:teuthology.orchestra.run.trial068.stdout: Running scriptlet: ceph-fuse-2:20.3.0-5439.ge371433f.el9.x86_64 1/1 2026-02-23T13:05:02.917 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : ceph-fuse-2:20.3.0-5439.ge371433f.el9.x86_64 1/1 2026-02-23T13:05:02.918 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T13:05:02.918 INFO:teuthology.orchestra.run.trial068.stdout:Installed: 2026-02-23T13:05:02.918 INFO:teuthology.orchestra.run.trial068.stdout: ceph-fuse-2:20.3.0-5439.ge371433f.el9.x86_64 2026-02-23T13:05:02.918 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T13:05:02.918 INFO:teuthology.orchestra.run.trial068.stdout:Complete! 2026-02-23T13:05:02.946 DEBUG:teuthology.orchestra.run.trial068:> sudo yum -y install ceph-volume 2026-02-23T13:05:03.263 INFO:teuthology.orchestra.run.trial068.stdout:Last metadata expiration check: 0:01:13 ago on Mon 23 Feb 2026 01:03:50 PM UTC. 2026-02-23T13:05:03.347 INFO:teuthology.orchestra.run.trial068.stdout:Dependencies resolved. 2026-02-23T13:05:03.348 INFO:teuthology.orchestra.run.trial068.stdout:================================================================================ 2026-02-23T13:05:03.348 INFO:teuthology.orchestra.run.trial068.stdout: Package Arch Version Repository Size 2026-02-23T13:05:03.348 INFO:teuthology.orchestra.run.trial068.stdout:================================================================================ 2026-02-23T13:05:03.348 INFO:teuthology.orchestra.run.trial068.stdout:Installing: 2026-02-23T13:05:03.348 INFO:teuthology.orchestra.run.trial068.stdout: ceph-volume noarch 2:20.3.0-5439.ge371433f.el9 ceph-noarch 301 k 2026-02-23T13:05:03.348 INFO:teuthology.orchestra.run.trial068.stdout:Installing dependencies: 2026-02-23T13:05:03.348 INFO:teuthology.orchestra.run.trial068.stdout: cryptsetup x86_64 2.8.1-3.el9 baseos 351 k 2026-02-23T13:05:03.348 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T13:05:03.349 INFO:teuthology.orchestra.run.trial068.stdout:Transaction Summary 2026-02-23T13:05:03.349 INFO:teuthology.orchestra.run.trial068.stdout:================================================================================ 2026-02-23T13:05:03.349 INFO:teuthology.orchestra.run.trial068.stdout:Install 2 Packages 2026-02-23T13:05:03.349 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T13:05:03.349 INFO:teuthology.orchestra.run.trial068.stdout:Total download size: 652 k 2026-02-23T13:05:03.349 INFO:teuthology.orchestra.run.trial068.stdout:Installed size: 2.2 M 2026-02-23T13:05:03.349 INFO:teuthology.orchestra.run.trial068.stdout:Downloading Packages: 2026-02-23T13:05:03.472 INFO:teuthology.orchestra.run.trial068.stdout:(1/2): cryptsetup-2.8.1-3.el9.x86_64.rpm 11 MB/s | 351 kB 00:00 2026-02-23T13:05:03.823 INFO:teuthology.orchestra.run.trial068.stdout:(2/2): ceph-volume-20.3.0-5439.ge371433f.el9.no 785 kB/s | 301 kB 00:00 2026-02-23T13:05:03.824 INFO:teuthology.orchestra.run.trial068.stdout:-------------------------------------------------------------------------------- 2026-02-23T13:05:03.824 INFO:teuthology.orchestra.run.trial068.stdout:Total 1.3 MB/s | 652 kB 00:00 2026-02-23T13:05:03.835 INFO:teuthology.orchestra.run.trial068.stdout:Running transaction check 2026-02-23T13:05:03.841 INFO:teuthology.orchestra.run.trial068.stdout:Transaction check succeeded. 2026-02-23T13:05:03.841 INFO:teuthology.orchestra.run.trial068.stdout:Running transaction test 2026-02-23T13:05:03.885 INFO:teuthology.orchestra.run.trial068.stdout:Transaction test succeeded. 2026-02-23T13:05:03.885 INFO:teuthology.orchestra.run.trial068.stdout:Running transaction 2026-02-23T13:05:03.966 INFO:teuthology.orchestra.run.trial068.stdout: Preparing : 1/1 2026-02-23T13:05:04.030 INFO:teuthology.orchestra.run.trial068.stdout: Installing : cryptsetup-2.8.1-3.el9.x86_64 1/2 2026-02-23T13:05:04.033 INFO:teuthology.orchestra.run.trial068.stdout: Installing : ceph-volume-2:20.3.0-5439.ge371433f.el9.noarch 2/2 2026-02-23T13:05:04.048 INFO:teuthology.orchestra.run.trial068.stdout: Running scriptlet: ceph-volume-2:20.3.0-5439.ge371433f.el9.noarch 2/2 2026-02-23T13:05:04.048 INFO:teuthology.orchestra.run.trial068.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-23T13:05:04.048 INFO:teuthology.orchestra.run.trial068.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2026-02-23T13:05:04.048 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T13:05:04.278 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : ceph-volume-2:20.3.0-5439.ge371433f.el9.noarch 1/2 2026-02-23T13:05:04.339 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : cryptsetup-2.8.1-3.el9.x86_64 2/2 2026-02-23T13:05:04.339 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T13:05:04.339 INFO:teuthology.orchestra.run.trial068.stdout:Installed: 2026-02-23T13:05:04.340 INFO:teuthology.orchestra.run.trial068.stdout: ceph-volume-2:20.3.0-5439.ge371433f.el9.noarch cryptsetup-2.8.1-3.el9.x86_64 2026-02-23T13:05:04.340 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T13:05:04.340 INFO:teuthology.orchestra.run.trial068.stdout:Complete! 2026-02-23T13:05:04.367 DEBUG:teuthology.orchestra.run.trial068:> sudo yum -y install librados-devel 2026-02-23T13:05:04.678 INFO:teuthology.orchestra.run.trial068.stdout:Last metadata expiration check: 0:01:14 ago on Mon 23 Feb 2026 01:03:50 PM UTC. 2026-02-23T13:05:04.762 INFO:teuthology.orchestra.run.trial068.stdout:Dependencies resolved. 2026-02-23T13:05:04.762 INFO:teuthology.orchestra.run.trial068.stdout:================================================================================ 2026-02-23T13:05:04.763 INFO:teuthology.orchestra.run.trial068.stdout: Package Arch Version Repo Size 2026-02-23T13:05:04.763 INFO:teuthology.orchestra.run.trial068.stdout:================================================================================ 2026-02-23T13:05:04.763 INFO:teuthology.orchestra.run.trial068.stdout:Installing: 2026-02-23T13:05:04.763 INFO:teuthology.orchestra.run.trial068.stdout: librados-devel x86_64 2:20.3.0-5439.ge371433f.el9 ceph 124 k 2026-02-23T13:05:04.763 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T13:05:04.763 INFO:teuthology.orchestra.run.trial068.stdout:Transaction Summary 2026-02-23T13:05:04.763 INFO:teuthology.orchestra.run.trial068.stdout:================================================================================ 2026-02-23T13:05:04.763 INFO:teuthology.orchestra.run.trial068.stdout:Install 1 Package 2026-02-23T13:05:04.763 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T13:05:04.763 INFO:teuthology.orchestra.run.trial068.stdout:Total download size: 124 k 2026-02-23T13:05:04.764 INFO:teuthology.orchestra.run.trial068.stdout:Installed size: 434 k 2026-02-23T13:05:04.764 INFO:teuthology.orchestra.run.trial068.stdout:Downloading Packages: 2026-02-23T13:05:04.848 INFO:teuthology.orchestra.run.trial068.stdout:librados-devel-20.3.0-5439.ge371433f.el9.x86_64 1.4 MB/s | 124 kB 00:00 2026-02-23T13:05:04.849 INFO:teuthology.orchestra.run.trial068.stdout:-------------------------------------------------------------------------------- 2026-02-23T13:05:04.849 INFO:teuthology.orchestra.run.trial068.stdout:Total 1.4 MB/s | 124 kB 00:00 2026-02-23T13:05:04.849 INFO:teuthology.orchestra.run.trial068.stdout:Running transaction check 2026-02-23T13:05:04.852 INFO:teuthology.orchestra.run.trial068.stdout:Transaction check succeeded. 2026-02-23T13:05:04.852 INFO:teuthology.orchestra.run.trial068.stdout:Running transaction test 2026-02-23T13:05:04.876 INFO:teuthology.orchestra.run.trial185.stdout:ceph-osd-crimson-debuginfo-20.3.0-5439.ge371433 52 MB/s | 429 MB 00:08 2026-02-23T13:05:04.877 INFO:teuthology.orchestra.run.trial185.stdout:-------------------------------------------------------------------------------- 2026-02-23T13:05:04.878 INFO:teuthology.orchestra.run.trial185.stdout:Total 52 MB/s | 429 MB 00:08 2026-02-23T13:05:04.878 INFO:teuthology.orchestra.run.trial185.stdout:Running transaction check 2026-02-23T13:05:04.880 INFO:teuthology.orchestra.run.trial185.stdout:Transaction check succeeded. 2026-02-23T13:05:04.880 INFO:teuthology.orchestra.run.trial185.stdout:Running transaction test 2026-02-23T13:05:04.883 INFO:teuthology.orchestra.run.trial068.stdout:Transaction test succeeded. 2026-02-23T13:05:04.883 INFO:teuthology.orchestra.run.trial068.stdout:Running transaction 2026-02-23T13:05:04.939 INFO:teuthology.orchestra.run.trial068.stdout: Preparing : 1/1 2026-02-23T13:05:04.956 INFO:teuthology.orchestra.run.trial068.stdout: Installing : librados-devel-2:20.3.0-5439.ge371433f.el9.x86_64 1/1 2026-02-23T13:05:05.043 INFO:teuthology.orchestra.run.trial068.stdout: Running scriptlet: librados-devel-2:20.3.0-5439.ge371433f.el9.x86_64 1/1 2026-02-23T13:05:05.101 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : librados-devel-2:20.3.0-5439.ge371433f.el9.x86_64 1/1 2026-02-23T13:05:05.101 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T13:05:05.101 INFO:teuthology.orchestra.run.trial068.stdout:Installed: 2026-02-23T13:05:05.101 INFO:teuthology.orchestra.run.trial068.stdout: librados-devel-2:20.3.0-5439.ge371433f.el9.x86_64 2026-02-23T13:05:05.101 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T13:05:05.101 INFO:teuthology.orchestra.run.trial068.stdout:Complete! 2026-02-23T13:05:05.128 DEBUG:teuthology.orchestra.run.trial068:> sudo yum -y install libcephfs2 2026-02-23T13:05:05.438 INFO:teuthology.orchestra.run.trial068.stdout:Last metadata expiration check: 0:01:15 ago on Mon 23 Feb 2026 01:03:50 PM UTC. 2026-02-23T13:05:05.502 INFO:teuthology.orchestra.run.trial068.stdout:Package libcephfs2-2:20.3.0-5439.ge371433f.el9.x86_64 is already installed. 2026-02-23T13:05:05.520 INFO:teuthology.orchestra.run.trial068.stdout:Dependencies resolved. 2026-02-23T13:05:05.521 INFO:teuthology.orchestra.run.trial068.stdout:Nothing to do. 2026-02-23T13:05:05.521 INFO:teuthology.orchestra.run.trial068.stdout:Complete! 2026-02-23T13:05:05.541 DEBUG:teuthology.orchestra.run.trial068:> sudo yum -y install libcephfs-devel 2026-02-23T13:05:05.844 INFO:teuthology.orchestra.run.trial068.stdout:Last metadata expiration check: 0:01:15 ago on Mon 23 Feb 2026 01:03:50 PM UTC. 2026-02-23T13:05:05.927 INFO:teuthology.orchestra.run.trial068.stdout:Dependencies resolved. 2026-02-23T13:05:05.927 INFO:teuthology.orchestra.run.trial068.stdout:================================================================================ 2026-02-23T13:05:05.928 INFO:teuthology.orchestra.run.trial068.stdout: Package Arch Version Repo Size 2026-02-23T13:05:05.928 INFO:teuthology.orchestra.run.trial068.stdout:================================================================================ 2026-02-23T13:05:05.928 INFO:teuthology.orchestra.run.trial068.stdout:Installing: 2026-02-23T13:05:05.928 INFO:teuthology.orchestra.run.trial068.stdout: libcephfs-devel x86_64 2:20.3.0-5439.ge371433f.el9 ceph 37 k 2026-02-23T13:05:05.928 INFO:teuthology.orchestra.run.trial068.stdout:Installing dependencies: 2026-02-23T13:05:05.928 INFO:teuthology.orchestra.run.trial068.stdout: libcephfs-proxy2 x86_64 2:20.3.0-5439.ge371433f.el9 ceph 26 k 2026-02-23T13:05:05.928 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T13:05:05.928 INFO:teuthology.orchestra.run.trial068.stdout:Transaction Summary 2026-02-23T13:05:05.928 INFO:teuthology.orchestra.run.trial068.stdout:================================================================================ 2026-02-23T13:05:05.929 INFO:teuthology.orchestra.run.trial068.stdout:Install 2 Packages 2026-02-23T13:05:05.929 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T13:05:05.929 INFO:teuthology.orchestra.run.trial068.stdout:Total download size: 64 k 2026-02-23T13:05:05.929 INFO:teuthology.orchestra.run.trial068.stdout:Installed size: 231 k 2026-02-23T13:05:05.929 INFO:teuthology.orchestra.run.trial068.stdout:Downloading Packages: 2026-02-23T13:05:05.991 INFO:teuthology.orchestra.run.trial068.stdout:(1/2): libcephfs-proxy2-20.3.0-5439.ge371433f.e 414 kB/s | 26 kB 00:00 2026-02-23T13:05:05.994 INFO:teuthology.orchestra.run.trial068.stdout:(2/2): libcephfs-devel-20.3.0-5439.ge371433f.el 560 kB/s | 37 kB 00:00 2026-02-23T13:05:05.995 INFO:teuthology.orchestra.run.trial068.stdout:-------------------------------------------------------------------------------- 2026-02-23T13:05:05.995 INFO:teuthology.orchestra.run.trial068.stdout:Total 938 kB/s | 64 kB 00:00 2026-02-23T13:05:05.995 INFO:teuthology.orchestra.run.trial068.stdout:Running transaction check 2026-02-23T13:05:05.998 INFO:teuthology.orchestra.run.trial068.stdout:Transaction check succeeded. 2026-02-23T13:05:05.998 INFO:teuthology.orchestra.run.trial068.stdout:Running transaction test 2026-02-23T13:05:06.030 INFO:teuthology.orchestra.run.trial068.stdout:Transaction test succeeded. 2026-02-23T13:05:06.030 INFO:teuthology.orchestra.run.trial068.stdout:Running transaction 2026-02-23T13:05:06.080 INFO:teuthology.orchestra.run.trial068.stdout: Preparing : 1/1 2026-02-23T13:05:06.081 INFO:teuthology.orchestra.run.trial068.stdout: Installing : libcephfs-proxy2-2:20.3.0-5439.ge371433f.el9.x86_64 1/2 2026-02-23T13:05:06.096 INFO:teuthology.orchestra.run.trial068.stdout: Running scriptlet: libcephfs-proxy2-2:20.3.0-5439.ge371433f.el9.x86_64 1/2 2026-02-23T13:05:06.112 INFO:teuthology.orchestra.run.trial068.stdout: Installing : libcephfs-devel-2:20.3.0-5439.ge371433f.el9.x86_64 2/2 2026-02-23T13:05:06.155 INFO:teuthology.orchestra.run.trial185.stdout:Transaction test succeeded. 2026-02-23T13:05:06.155 INFO:teuthology.orchestra.run.trial185.stdout:Running transaction 2026-02-23T13:05:06.167 INFO:teuthology.orchestra.run.trial068.stdout: Running scriptlet: libcephfs-devel-2:20.3.0-5439.ge371433f.el9.x86_64 2/2 2026-02-23T13:05:06.167 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : libcephfs-devel-2:20.3.0-5439.ge371433f.el9.x86_64 1/2 2026-02-23T13:05:06.226 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : libcephfs-proxy2-2:20.3.0-5439.ge371433f.el9.x86_64 2/2 2026-02-23T13:05:06.226 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T13:05:06.226 INFO:teuthology.orchestra.run.trial068.stdout:Installed: 2026-02-23T13:05:06.226 INFO:teuthology.orchestra.run.trial068.stdout: libcephfs-devel-2:20.3.0-5439.ge371433f.el9.x86_64 2026-02-23T13:05:06.226 INFO:teuthology.orchestra.run.trial068.stdout: libcephfs-proxy2-2:20.3.0-5439.ge371433f.el9.x86_64 2026-02-23T13:05:06.226 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T13:05:06.226 INFO:teuthology.orchestra.run.trial068.stdout:Complete! 2026-02-23T13:05:06.253 DEBUG:teuthology.orchestra.run.trial068:> sudo yum -y install librados2 2026-02-23T13:05:06.570 INFO:teuthology.orchestra.run.trial068.stdout:Last metadata expiration check: 0:01:16 ago on Mon 23 Feb 2026 01:03:50 PM UTC. 2026-02-23T13:05:06.636 INFO:teuthology.orchestra.run.trial068.stdout:Package librados2-2:20.3.0-5439.ge371433f.el9.x86_64 is already installed. 2026-02-23T13:05:06.654 INFO:teuthology.orchestra.run.trial068.stdout:Dependencies resolved. 2026-02-23T13:05:06.655 INFO:teuthology.orchestra.run.trial068.stdout:Nothing to do. 2026-02-23T13:05:06.655 INFO:teuthology.orchestra.run.trial068.stdout:Complete! 2026-02-23T13:05:06.676 DEBUG:teuthology.orchestra.run.trial068:> sudo yum -y install librbd1 2026-02-23T13:05:06.978 INFO:teuthology.orchestra.run.trial068.stdout:Last metadata expiration check: 0:01:16 ago on Mon 23 Feb 2026 01:03:50 PM UTC. 2026-02-23T13:05:07.043 INFO:teuthology.orchestra.run.trial068.stdout:Package librbd1-2:20.3.0-5439.ge371433f.el9.x86_64 is already installed. 2026-02-23T13:05:07.061 INFO:teuthology.orchestra.run.trial068.stdout:Dependencies resolved. 2026-02-23T13:05:07.062 INFO:teuthology.orchestra.run.trial068.stdout:Nothing to do. 2026-02-23T13:05:07.062 INFO:teuthology.orchestra.run.trial068.stdout:Complete! 2026-02-23T13:05:07.083 DEBUG:teuthology.orchestra.run.trial068:> sudo yum -y install python3-rados 2026-02-23T13:05:07.397 INFO:teuthology.orchestra.run.trial068.stdout:Last metadata expiration check: 0:01:17 ago on Mon 23 Feb 2026 01:03:50 PM UTC. 2026-02-23T13:05:07.462 INFO:teuthology.orchestra.run.trial068.stdout:Package python3-rados-2:20.3.0-5439.ge371433f.el9.x86_64 is already installed. 2026-02-23T13:05:07.480 INFO:teuthology.orchestra.run.trial068.stdout:Dependencies resolved. 2026-02-23T13:05:07.481 INFO:teuthology.orchestra.run.trial068.stdout:Nothing to do. 2026-02-23T13:05:07.481 INFO:teuthology.orchestra.run.trial068.stdout:Complete! 2026-02-23T13:05:07.503 DEBUG:teuthology.orchestra.run.trial068:> sudo yum -y install python3-rgw 2026-02-23T13:05:07.810 INFO:teuthology.orchestra.run.trial068.stdout:Last metadata expiration check: 0:01:17 ago on Mon 23 Feb 2026 01:03:50 PM UTC. 2026-02-23T13:05:07.874 INFO:teuthology.orchestra.run.trial068.stdout:Package python3-rgw-2:20.3.0-5439.ge371433f.el9.x86_64 is already installed. 2026-02-23T13:05:07.893 INFO:teuthology.orchestra.run.trial068.stdout:Dependencies resolved. 2026-02-23T13:05:07.893 INFO:teuthology.orchestra.run.trial068.stdout:Nothing to do. 2026-02-23T13:05:07.893 INFO:teuthology.orchestra.run.trial068.stdout:Complete! 2026-02-23T13:05:07.915 DEBUG:teuthology.orchestra.run.trial068:> sudo yum -y install python3-cephfs 2026-02-23T13:05:08.220 INFO:teuthology.orchestra.run.trial068.stdout:Last metadata expiration check: 0:01:18 ago on Mon 23 Feb 2026 01:03:50 PM UTC. 2026-02-23T13:05:08.284 INFO:teuthology.orchestra.run.trial068.stdout:Package python3-cephfs-2:20.3.0-5439.ge371433f.el9.x86_64 is already installed. 2026-02-23T13:05:08.303 INFO:teuthology.orchestra.run.trial068.stdout:Dependencies resolved. 2026-02-23T13:05:08.303 INFO:teuthology.orchestra.run.trial068.stdout:Nothing to do. 2026-02-23T13:05:08.303 INFO:teuthology.orchestra.run.trial068.stdout:Complete! 2026-02-23T13:05:08.325 DEBUG:teuthology.orchestra.run.trial068:> sudo yum -y install python3-rbd 2026-02-23T13:05:08.633 INFO:teuthology.orchestra.run.trial068.stdout:Last metadata expiration check: 0:01:18 ago on Mon 23 Feb 2026 01:03:50 PM UTC. 2026-02-23T13:05:08.698 INFO:teuthology.orchestra.run.trial068.stdout:Package python3-rbd-2:20.3.0-5439.ge371433f.el9.x86_64 is already installed. 2026-02-23T13:05:08.716 INFO:teuthology.orchestra.run.trial068.stdout:Dependencies resolved. 2026-02-23T13:05:08.717 INFO:teuthology.orchestra.run.trial068.stdout:Nothing to do. 2026-02-23T13:05:08.717 INFO:teuthology.orchestra.run.trial068.stdout:Complete! 2026-02-23T13:05:08.738 DEBUG:teuthology.orchestra.run.trial068:> sudo yum -y install rbd-fuse 2026-02-23T13:05:09.048 INFO:teuthology.orchestra.run.trial068.stdout:Last metadata expiration check: 0:01:19 ago on Mon 23 Feb 2026 01:03:50 PM UTC. 2026-02-23T13:05:09.133 INFO:teuthology.orchestra.run.trial068.stdout:Dependencies resolved. 2026-02-23T13:05:09.133 INFO:teuthology.orchestra.run.trial068.stdout:================================================================================ 2026-02-23T13:05:09.133 INFO:teuthology.orchestra.run.trial068.stdout: Package Architecture Version Repository Size 2026-02-23T13:05:09.133 INFO:teuthology.orchestra.run.trial068.stdout:================================================================================ 2026-02-23T13:05:09.133 INFO:teuthology.orchestra.run.trial068.stdout:Installing: 2026-02-23T13:05:09.133 INFO:teuthology.orchestra.run.trial068.stdout: rbd-fuse x86_64 2:20.3.0-5439.ge371433f.el9 ceph 86 k 2026-02-23T13:05:09.133 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T13:05:09.134 INFO:teuthology.orchestra.run.trial068.stdout:Transaction Summary 2026-02-23T13:05:09.134 INFO:teuthology.orchestra.run.trial068.stdout:================================================================================ 2026-02-23T13:05:09.134 INFO:teuthology.orchestra.run.trial068.stdout:Install 1 Package 2026-02-23T13:05:09.134 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T13:05:09.134 INFO:teuthology.orchestra.run.trial068.stdout:Total download size: 86 k 2026-02-23T13:05:09.134 INFO:teuthology.orchestra.run.trial068.stdout:Installed size: 236 k 2026-02-23T13:05:09.134 INFO:teuthology.orchestra.run.trial068.stdout:Downloading Packages: 2026-02-23T13:05:09.208 INFO:teuthology.orchestra.run.trial068.stdout:rbd-fuse-20.3.0-5439.ge371433f.el9.x86_64.rpm 1.1 MB/s | 86 kB 00:00 2026-02-23T13:05:09.209 INFO:teuthology.orchestra.run.trial068.stdout:-------------------------------------------------------------------------------- 2026-02-23T13:05:09.209 INFO:teuthology.orchestra.run.trial068.stdout:Total 1.1 MB/s | 86 kB 00:00 2026-02-23T13:05:09.209 INFO:teuthology.orchestra.run.trial068.stdout:Running transaction check 2026-02-23T13:05:09.213 INFO:teuthology.orchestra.run.trial068.stdout:Transaction check succeeded. 2026-02-23T13:05:09.213 INFO:teuthology.orchestra.run.trial068.stdout:Running transaction test 2026-02-23T13:05:09.244 INFO:teuthology.orchestra.run.trial068.stdout:Transaction test succeeded. 2026-02-23T13:05:09.244 INFO:teuthology.orchestra.run.trial068.stdout:Running transaction 2026-02-23T13:05:09.295 INFO:teuthology.orchestra.run.trial068.stdout: Preparing : 1/1 2026-02-23T13:05:09.312 INFO:teuthology.orchestra.run.trial068.stdout: Installing : rbd-fuse-2:20.3.0-5439.ge371433f.el9.x86_64 1/1 2026-02-23T13:05:09.396 INFO:teuthology.orchestra.run.trial068.stdout: Running scriptlet: rbd-fuse-2:20.3.0-5439.ge371433f.el9.x86_64 1/1 2026-02-23T13:05:09.454 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : rbd-fuse-2:20.3.0-5439.ge371433f.el9.x86_64 1/1 2026-02-23T13:05:09.455 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T13:05:09.455 INFO:teuthology.orchestra.run.trial068.stdout:Installed: 2026-02-23T13:05:09.455 INFO:teuthology.orchestra.run.trial068.stdout: rbd-fuse-2:20.3.0-5439.ge371433f.el9.x86_64 2026-02-23T13:05:09.455 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T13:05:09.455 INFO:teuthology.orchestra.run.trial068.stdout:Complete! 2026-02-23T13:05:09.486 DEBUG:teuthology.orchestra.run.trial068:> sudo yum -y install rbd-mirror 2026-02-23T13:05:09.796 INFO:teuthology.orchestra.run.trial068.stdout:Last metadata expiration check: 0:01:19 ago on Mon 23 Feb 2026 01:03:50 PM UTC. 2026-02-23T13:05:09.879 INFO:teuthology.orchestra.run.trial068.stdout:Dependencies resolved. 2026-02-23T13:05:09.879 INFO:teuthology.orchestra.run.trial068.stdout:================================================================================ 2026-02-23T13:05:09.879 INFO:teuthology.orchestra.run.trial068.stdout: Package Arch Version Repo Size 2026-02-23T13:05:09.879 INFO:teuthology.orchestra.run.trial068.stdout:================================================================================ 2026-02-23T13:05:09.879 INFO:teuthology.orchestra.run.trial068.stdout:Installing: 2026-02-23T13:05:09.880 INFO:teuthology.orchestra.run.trial068.stdout: rbd-mirror x86_64 2:20.3.0-5439.ge371433f.el9 ceph 2.6 M 2026-02-23T13:05:09.880 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T13:05:09.880 INFO:teuthology.orchestra.run.trial068.stdout:Transaction Summary 2026-02-23T13:05:09.880 INFO:teuthology.orchestra.run.trial068.stdout:================================================================================ 2026-02-23T13:05:09.880 INFO:teuthology.orchestra.run.trial068.stdout:Install 1 Package 2026-02-23T13:05:09.880 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T13:05:09.880 INFO:teuthology.orchestra.run.trial068.stdout:Total download size: 2.6 M 2026-02-23T13:05:09.880 INFO:teuthology.orchestra.run.trial068.stdout:Installed size: 11 M 2026-02-23T13:05:09.880 INFO:teuthology.orchestra.run.trial068.stdout:Downloading Packages: 2026-02-23T13:05:10.024 INFO:teuthology.orchestra.run.trial068.stdout:rbd-mirror-20.3.0-5439.ge371433f.el9.x86_64.rpm 18 MB/s | 2.6 MB 00:00 2026-02-23T13:05:10.024 INFO:teuthology.orchestra.run.trial068.stdout:-------------------------------------------------------------------------------- 2026-02-23T13:05:10.024 INFO:teuthology.orchestra.run.trial068.stdout:Total 18 MB/s | 2.6 MB 00:00 2026-02-23T13:05:10.024 INFO:teuthology.orchestra.run.trial068.stdout:Running transaction check 2026-02-23T13:05:10.029 INFO:teuthology.orchestra.run.trial068.stdout:Transaction check succeeded. 2026-02-23T13:05:10.029 INFO:teuthology.orchestra.run.trial068.stdout:Running transaction test 2026-02-23T13:05:10.067 INFO:teuthology.orchestra.run.trial068.stdout:Transaction test succeeded. 2026-02-23T13:05:10.067 INFO:teuthology.orchestra.run.trial068.stdout:Running transaction 2026-02-23T13:05:10.230 INFO:teuthology.orchestra.run.trial068.stdout: Preparing : 1/1 2026-02-23T13:05:10.231 INFO:teuthology.orchestra.run.trial068.stdout: Installing : rbd-mirror-2:20.3.0-5439.ge371433f.el9.x86_64 1/1 2026-02-23T13:05:10.254 INFO:teuthology.orchestra.run.trial068.stdout: Running scriptlet: rbd-mirror-2:20.3.0-5439.ge371433f.el9.x86_64 1/1 2026-02-23T13:05:10.254 INFO:teuthology.orchestra.run.trial068.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-23T13:05:10.254 INFO:teuthology.orchestra.run.trial068.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2026-02-23T13:05:10.254 INFO:teuthology.orchestra.run.trial068.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2026-02-23T13:05:10.255 INFO:teuthology.orchestra.run.trial068.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2026-02-23T13:05:10.255 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T13:05:10.540 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : rbd-mirror-2:20.3.0-5439.ge371433f.el9.x86_64 1/1 2026-02-23T13:05:10.540 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T13:05:10.541 INFO:teuthology.orchestra.run.trial068.stdout:Installed: 2026-02-23T13:05:10.541 INFO:teuthology.orchestra.run.trial068.stdout: rbd-mirror-2:20.3.0-5439.ge371433f.el9.x86_64 2026-02-23T13:05:10.541 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T13:05:10.541 INFO:teuthology.orchestra.run.trial068.stdout:Complete! 2026-02-23T13:05:10.568 DEBUG:teuthology.orchestra.run.trial068:> sudo yum -y install rbd-nbd 2026-02-23T13:05:10.882 INFO:teuthology.orchestra.run.trial068.stdout:Last metadata expiration check: 0:01:20 ago on Mon 23 Feb 2026 01:03:50 PM UTC. 2026-02-23T13:05:10.965 INFO:teuthology.orchestra.run.trial068.stdout:Dependencies resolved. 2026-02-23T13:05:10.966 INFO:teuthology.orchestra.run.trial068.stdout:================================================================================ 2026-02-23T13:05:10.966 INFO:teuthology.orchestra.run.trial068.stdout: Package Architecture Version Repository Size 2026-02-23T13:05:10.966 INFO:teuthology.orchestra.run.trial068.stdout:================================================================================ 2026-02-23T13:05:10.966 INFO:teuthology.orchestra.run.trial068.stdout:Installing: 2026-02-23T13:05:10.966 INFO:teuthology.orchestra.run.trial068.stdout: rbd-nbd x86_64 2:20.3.0-5439.ge371433f.el9 ceph 166 k 2026-02-23T13:05:10.966 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T13:05:10.966 INFO:teuthology.orchestra.run.trial068.stdout:Transaction Summary 2026-02-23T13:05:10.966 INFO:teuthology.orchestra.run.trial068.stdout:================================================================================ 2026-02-23T13:05:10.966 INFO:teuthology.orchestra.run.trial068.stdout:Install 1 Package 2026-02-23T13:05:10.967 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T13:05:10.967 INFO:teuthology.orchestra.run.trial068.stdout:Total download size: 166 k 2026-02-23T13:05:10.967 INFO:teuthology.orchestra.run.trial068.stdout:Installed size: 496 k 2026-02-23T13:05:10.967 INFO:teuthology.orchestra.run.trial068.stdout:Downloading Packages: 2026-02-23T13:05:11.053 INFO:teuthology.orchestra.run.trial068.stdout:rbd-nbd-20.3.0-5439.ge371433f.el9.x86_64.rpm 1.9 MB/s | 166 kB 00:00 2026-02-23T13:05:11.054 INFO:teuthology.orchestra.run.trial068.stdout:-------------------------------------------------------------------------------- 2026-02-23T13:05:11.054 INFO:teuthology.orchestra.run.trial068.stdout:Total 1.8 MB/s | 166 kB 00:00 2026-02-23T13:05:11.054 INFO:teuthology.orchestra.run.trial068.stdout:Running transaction check 2026-02-23T13:05:11.059 INFO:teuthology.orchestra.run.trial068.stdout:Transaction check succeeded. 2026-02-23T13:05:11.059 INFO:teuthology.orchestra.run.trial068.stdout:Running transaction test 2026-02-23T13:05:11.090 INFO:teuthology.orchestra.run.trial068.stdout:Transaction test succeeded. 2026-02-23T13:05:11.090 INFO:teuthology.orchestra.run.trial068.stdout:Running transaction 2026-02-23T13:05:11.146 INFO:teuthology.orchestra.run.trial068.stdout: Preparing : 1/1 2026-02-23T13:05:11.162 INFO:teuthology.orchestra.run.trial068.stdout: Installing : rbd-nbd-2:20.3.0-5439.ge371433f.el9.x86_64 1/1 2026-02-23T13:05:11.244 INFO:teuthology.orchestra.run.trial068.stdout: Running scriptlet: rbd-nbd-2:20.3.0-5439.ge371433f.el9.x86_64 1/1 2026-02-23T13:05:11.303 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : rbd-nbd-2:20.3.0-5439.ge371433f.el9.x86_64 1/1 2026-02-23T13:05:11.303 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T13:05:11.303 INFO:teuthology.orchestra.run.trial068.stdout:Installed: 2026-02-23T13:05:11.303 INFO:teuthology.orchestra.run.trial068.stdout: rbd-nbd-2:20.3.0-5439.ge371433f.el9.x86_64 2026-02-23T13:05:11.303 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T13:05:11.303 INFO:teuthology.orchestra.run.trial068.stdout:Complete! 2026-02-23T13:05:11.332 DEBUG:teuthology.orchestra.run.trial068:> sudo yum -y install ceph-osd-crimson 2026-02-23T13:05:11.645 INFO:teuthology.orchestra.run.trial068.stdout:Last metadata expiration check: 0:01:21 ago on Mon 23 Feb 2026 01:03:50 PM UTC. 2026-02-23T13:05:11.710 INFO:teuthology.orchestra.run.trial068.stdout:Package ceph-osd-crimson-2:20.3.0-5439.ge371433f.el9.x86_64 is already installed. 2026-02-23T13:05:11.729 INFO:teuthology.orchestra.run.trial068.stdout:Dependencies resolved. 2026-02-23T13:05:11.729 INFO:teuthology.orchestra.run.trial068.stdout:Nothing to do. 2026-02-23T13:05:11.730 INFO:teuthology.orchestra.run.trial068.stdout:Complete! 2026-02-23T13:05:11.751 DEBUG:teuthology.orchestra.run.trial068:> sudo yum -y install ceph-osd-crimson-debuginfo 2026-02-23T13:05:12.061 INFO:teuthology.orchestra.run.trial068.stdout:Last metadata expiration check: 0:01:22 ago on Mon 23 Feb 2026 01:03:50 PM UTC. 2026-02-23T13:05:12.144 INFO:teuthology.orchestra.run.trial068.stdout:Dependencies resolved. 2026-02-23T13:05:12.144 INFO:teuthology.orchestra.run.trial068.stdout:================================================================================ 2026-02-23T13:05:12.144 INFO:teuthology.orchestra.run.trial068.stdout: Package Arch Version Repo Size 2026-02-23T13:05:12.144 INFO:teuthology.orchestra.run.trial068.stdout:================================================================================ 2026-02-23T13:05:12.144 INFO:teuthology.orchestra.run.trial068.stdout:Installing: 2026-02-23T13:05:12.144 INFO:teuthology.orchestra.run.trial068.stdout: ceph-osd-crimson-debuginfo x86_64 2:20.3.0-5439.ge371433f.el9 ceph 429 M 2026-02-23T13:05:12.144 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T13:05:12.145 INFO:teuthology.orchestra.run.trial068.stdout:Transaction Summary 2026-02-23T13:05:12.145 INFO:teuthology.orchestra.run.trial068.stdout:================================================================================ 2026-02-23T13:05:12.145 INFO:teuthology.orchestra.run.trial068.stdout:Install 1 Package 2026-02-23T13:05:12.145 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T13:05:12.145 INFO:teuthology.orchestra.run.trial068.stdout:Total download size: 429 M 2026-02-23T13:05:12.145 INFO:teuthology.orchestra.run.trial068.stdout:Installed size: 2.6 G 2026-02-23T13:05:12.145 INFO:teuthology.orchestra.run.trial068.stdout:Downloading Packages: 2026-02-23T13:05:20.850 INFO:teuthology.orchestra.run.trial068.stdout:ceph-osd-crimson-debuginfo-20.3.0-5439.ge371433 49 MB/s | 429 MB 00:08 2026-02-23T13:05:20.851 INFO:teuthology.orchestra.run.trial068.stdout:-------------------------------------------------------------------------------- 2026-02-23T13:05:20.852 INFO:teuthology.orchestra.run.trial068.stdout:Total 49 MB/s | 429 MB 00:08 2026-02-23T13:05:20.852 INFO:teuthology.orchestra.run.trial068.stdout:Running transaction check 2026-02-23T13:05:20.853 INFO:teuthology.orchestra.run.trial068.stdout:Transaction check succeeded. 2026-02-23T13:05:20.853 INFO:teuthology.orchestra.run.trial068.stdout:Running transaction test 2026-02-23T13:05:22.132 INFO:teuthology.orchestra.run.trial068.stdout:Transaction test succeeded. 2026-02-23T13:05:22.132 INFO:teuthology.orchestra.run.trial068.stdout:Running transaction 2026-02-23T13:05:29.462 INFO:teuthology.orchestra.run.trial185.stdout: Preparing : 1/1 2026-02-23T13:05:29.478 INFO:teuthology.orchestra.run.trial185.stdout: Installing : ceph-osd-crimson-debuginfo-2:20.3.0-5439.ge371433f.e 1/1 2026-02-23T13:05:31.369 INFO:teuthology.orchestra.run.trial185.stdout: Running scriptlet: ceph-osd-crimson-debuginfo-2:20.3.0-5439.ge371433f.e 1/1 2026-02-23T13:05:31.426 INFO:teuthology.orchestra.run.trial185.stdout: Verifying : ceph-osd-crimson-debuginfo-2:20.3.0-5439.ge371433f.e 1/1 2026-02-23T13:05:31.426 INFO:teuthology.orchestra.run.trial185.stdout: 2026-02-23T13:05:31.427 INFO:teuthology.orchestra.run.trial185.stdout:Installed: 2026-02-23T13:05:31.427 INFO:teuthology.orchestra.run.trial185.stdout: ceph-osd-crimson-debuginfo-2:20.3.0-5439.ge371433f.el9.x86_64 2026-02-23T13:05:31.427 INFO:teuthology.orchestra.run.trial185.stdout: 2026-02-23T13:05:31.427 INFO:teuthology.orchestra.run.trial185.stdout:Complete! 2026-02-23T13:05:31.496 DEBUG:teuthology.parallel:result is None 2026-02-23T13:05:45.504 INFO:teuthology.orchestra.run.trial068.stdout: Preparing : 1/1 2026-02-23T13:05:45.520 INFO:teuthology.orchestra.run.trial068.stdout: Installing : ceph-osd-crimson-debuginfo-2:20.3.0-5439.ge371433f.e 1/1 2026-02-23T13:05:47.494 INFO:teuthology.orchestra.run.trial068.stdout: Running scriptlet: ceph-osd-crimson-debuginfo-2:20.3.0-5439.ge371433f.e 1/1 2026-02-23T13:05:47.553 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : ceph-osd-crimson-debuginfo-2:20.3.0-5439.ge371433f.e 1/1 2026-02-23T13:05:47.553 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T13:05:47.554 INFO:teuthology.orchestra.run.trial068.stdout:Installed: 2026-02-23T13:05:47.554 INFO:teuthology.orchestra.run.trial068.stdout: ceph-osd-crimson-debuginfo-2:20.3.0-5439.ge371433f.el9.x86_64 2026-02-23T13:05:47.554 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T13:05:47.554 INFO:teuthology.orchestra.run.trial068.stdout:Complete! 2026-02-23T13:05:47.621 DEBUG:teuthology.parallel:result is None 2026-02-23T13:05:47.622 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=debug&distros=centos%2F9%2Fx86_64&sha1=e371433fcd1390a0b82cfb18effc422c1f5d8c1d 2026-02-23T13:05:47.749 DEBUG:teuthology.orchestra.run.trial068:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2026-02-23T13:05:47.771 INFO:teuthology.orchestra.run.trial068.stdout:20.3.0-5439.ge371433f.el9 2026-02-23T13:05:47.771 INFO:teuthology.packaging:The installed version of ceph is 20.3.0-5439.ge371433f.el9 2026-02-23T13:05:47.771 INFO:teuthology.task.install:The correct ceph version 20.3.0-5439.ge371433f is installed. 2026-02-23T13:05:47.773 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=debug&distros=centos%2F9%2Fx86_64&sha1=e371433fcd1390a0b82cfb18effc422c1f5d8c1d 2026-02-23T13:05:47.910 DEBUG:teuthology.orchestra.run.trial185:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2026-02-23T13:05:47.931 INFO:teuthology.orchestra.run.trial185.stdout:20.3.0-5439.ge371433f.el9 2026-02-23T13:05:47.931 INFO:teuthology.packaging:The installed version of ceph is 20.3.0-5439.ge371433f.el9 2026-02-23T13:05:47.931 INFO:teuthology.task.install:The correct ceph version 20.3.0-5439.ge371433f is installed. 2026-02-23T13:05:47.933 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2026-02-23T13:05:47.933 DEBUG:teuthology.orchestra.run.trial068:> set -ex 2026-02-23T13:05:47.933 DEBUG:teuthology.orchestra.run.trial068:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2026-02-23T13:05:47.960 DEBUG:teuthology.orchestra.run.trial185:> set -ex 2026-02-23T13:05:47.960 DEBUG:teuthology.orchestra.run.trial185:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2026-02-23T13:05:47.996 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2026-02-23T13:05:47.996 DEBUG:teuthology.orchestra.run.trial068:> set -ex 2026-02-23T13:05:47.996 DEBUG:teuthology.orchestra.run.trial068:> sudo dd of=/usr/bin/daemon-helper 2026-02-23T13:05:48.024 DEBUG:teuthology.orchestra.run.trial068:> sudo chmod a=rx -- /usr/bin/daemon-helper 2026-02-23T13:05:48.091 DEBUG:teuthology.orchestra.run.trial185:> set -ex 2026-02-23T13:05:48.092 DEBUG:teuthology.orchestra.run.trial185:> sudo dd of=/usr/bin/daemon-helper 2026-02-23T13:05:48.119 DEBUG:teuthology.orchestra.run.trial185:> sudo chmod a=rx -- /usr/bin/daemon-helper 2026-02-23T13:05:48.183 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2026-02-23T13:05:48.183 DEBUG:teuthology.orchestra.run.trial068:> set -ex 2026-02-23T13:05:48.183 DEBUG:teuthology.orchestra.run.trial068:> sudo dd of=/usr/bin/adjust-ulimits 2026-02-23T13:05:48.206 DEBUG:teuthology.orchestra.run.trial068:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2026-02-23T13:05:48.270 DEBUG:teuthology.orchestra.run.trial185:> set -ex 2026-02-23T13:05:48.270 DEBUG:teuthology.orchestra.run.trial185:> sudo dd of=/usr/bin/adjust-ulimits 2026-02-23T13:05:48.296 DEBUG:teuthology.orchestra.run.trial185:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2026-02-23T13:05:48.361 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2026-02-23T13:05:48.361 DEBUG:teuthology.orchestra.run.trial068:> set -ex 2026-02-23T13:05:48.361 DEBUG:teuthology.orchestra.run.trial068:> sudo dd of=/usr/bin/stdin-killer 2026-02-23T13:05:48.387 DEBUG:teuthology.orchestra.run.trial068:> sudo chmod a=rx -- /usr/bin/stdin-killer 2026-02-23T13:05:48.451 DEBUG:teuthology.orchestra.run.trial185:> set -ex 2026-02-23T13:05:48.451 DEBUG:teuthology.orchestra.run.trial185:> sudo dd of=/usr/bin/stdin-killer 2026-02-23T13:05:48.476 DEBUG:teuthology.orchestra.run.trial185:> sudo chmod a=rx -- /usr/bin/stdin-killer 2026-02-23T13:05:48.539 INFO:teuthology.run_tasks:Running task pexec... 2026-02-23T13:05:48.552 INFO:teuthology.task.pexec:Executing custom commands... 2026-02-23T13:05:48.552 DEBUG:teuthology.orchestra.run.trial068:> TESTDIR=/home/ubuntu/cephtest bash -s 2026-02-23T13:05:48.553 DEBUG:teuthology.orchestra.run.trial185:> TESTDIR=/home/ubuntu/cephtest bash -s 2026-02-23T13:05:48.556 INFO:teuthology.task.pexec:Running commands on host ubuntu@trial068.front.sepia.ceph.com 2026-02-23T13:05:48.556 INFO:teuthology.task.pexec:sudo update-alternatives --set ceph-osd /usr/bin/ceph-osd-crimson 2026-02-23T13:05:48.582 INFO:teuthology.task.pexec:Running commands on host ubuntu@trial185.front.sepia.ceph.com 2026-02-23T13:05:48.582 INFO:teuthology.task.pexec:sudo update-alternatives --set ceph-osd /usr/bin/ceph-osd-crimson 2026-02-23T13:05:48.607 DEBUG:teuthology.parallel:result is None 2026-02-23T13:05:48.633 DEBUG:teuthology.parallel:result is None 2026-02-23T13:05:48.633 INFO:teuthology.run_tasks:Running task cephadm... 2026-02-23T13:05:48.758 INFO:tasks.cephadm:Config: {'conf': {'mgr': {'debug mgr': 20, 'debug ms': 1, 'debug osd': 10}, 'client.0': {'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'crimson_allow_pg_split': False, 'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc', 'osd_pool_default_min_size': 2, 'osd_pool_default_size': 3}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'osd pool default crimson': True, 'paxos service trim min': 10}, 'osd': {'crimson cpu num': 4, 'crimson osd obc lru size': 10, 'debug ms': 20, 'debug osd': 20, 'debug seastore': 20, 'debug seastore backref': 20, 'debug seastore cache': 20, 'debug seastore cleaner': 20, 'debug seastore device': 20, 'debug seastore epm': 20, 'debug seastore fixedkv tree': 20, 'debug seastore journal': 20, 'debug seastore lba': 20, 'debug seastore odata': 20, 'debug seastore omap': 20, 'debug seastore onode': 20, 'debug seastore t': 20, 'debug seastore tm': 20, 'osd blocked scrub grace period': 3600, 'osd debug reject backfill probability': 0.3, 'osd delete sleep': 1, 'osd max backfills': 3, 'osd max markdown count': 1000, 'osd objectstore': 'seastore', 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd snap trim sleep': 2, 'osd target transaction size': 5, 'seastore cachepin size pershard': '64M', 'seastore main device type': 'RANDOM_BLOCK_SSD', 'seastore max concurrent transactions': 8}}, 'osd_type': 'crimson', 'crimson_compat': True, 'flavor': 'debug', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', 'SLOW_OPS', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', 'mons down', 'mon down', 'out of quorum', 'noscrub', 'nodeep-scrub', 'Degraded data redundancy', 'is down', 'osds down', 'pg .*? is .*?degraded.*?, acting', 'pg .*? is stuck', 'pg degraded', 'PG_BACKFILL_FULL', 'Low space hindering backfill .*? backfill_toofull', 'OSD_ROOT_DOWN', 'pgs degraded', 'pgs undersized', 'is active.*backfill_toofull.*'], 'sha1': 'e371433fcd1390a0b82cfb18effc422c1f5d8c1d'} 2026-02-23T13:05:48.758 INFO:tasks.cephadm:Cluster image is quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d 2026-02-23T13:05:48.761 INFO:tasks.cephadm:Cluster fsid is 5f823084-10b8-11f1-be80-d404e6e7d460 2026-02-23T13:05:48.762 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2026-02-23T13:05:48.762 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '10.20.193.68'} 2026-02-23T13:05:48.762 INFO:tasks.cephadm:First mon is mon.a on trial068 2026-02-23T13:05:48.762 INFO:tasks.cephadm:First mgr is x 2026-02-23T13:05:48.762 INFO:tasks.cephadm:Normalizing hostnames... 2026-02-23T13:05:48.762 DEBUG:teuthology.orchestra.run.trial068:> sudo hostname $(hostname -s) 2026-02-23T13:05:48.786 DEBUG:teuthology.orchestra.run.trial185:> sudo hostname $(hostname -s) 2026-02-23T13:05:48.811 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra 2026-02-23T13:05:48.811 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=debug&distros=centos%2F9%2Fx86_64&sha1=e371433fcd1390a0b82cfb18effc422c1f5d8c1d 2026-02-23T13:05:48.941 INFO:tasks.cephadm:builder_project result: [{'url': 'https://3.chacra.ceph.com/r/ceph/wip-shraddhaag-enable-cephadm-tests/e371433fcd1390a0b82cfb18effc422c1f5d8c1d/centos/9/flavors/debug/', 'chacra_url': 'https://3.chacra.ceph.com/repos/ceph/wip-shraddhaag-enable-cephadm-tests/e371433fcd1390a0b82cfb18effc422c1f5d8c1d/centos/9/flavors/debug/', 'ref': 'wip-shraddhaag-enable-cephadm-tests', 'sha1': 'e371433fcd1390a0b82cfb18effc422c1f5d8c1d', 'distro': 'centos', 'distro_version': '9', 'distro_codename': None, 'modified': '2026-02-23 09:29:09.802677', 'status': 'ready', 'flavor': 'debug', 'project': 'ceph', 'archs': ['source', 'x86_64'], 'extra': {'version': '20.3.0-5439-ge371433f', 'package_manager_version': '20.3.0-5439.ge371433f', 'build_url': 'https://jenkins.ceph.com/job/ceph-dev-pipeline/3195/', 'root_build_cause': '', 'node_name': '10.20.192.97+toko07', 'job_name': 'ceph-dev-pipeline'}}] 2026-02-23T13:05:49.071 INFO:tasks.util.chacra:got chacra host 3.chacra.ceph.com, ref wip-shraddhaag-enable-cephadm-tests, sha1 e371433fcd1390a0b82cfb18effc422c1f5d8c1d from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F9%2Fx86_64&flavor=debug&sha1=e371433fcd1390a0b82cfb18effc422c1f5d8c1d 2026-02-23T13:05:49.073 INFO:tasks.cephadm:Discovered cachra url: https://3.chacra.ceph.com/binaries/ceph/wip-shraddhaag-enable-cephadm-tests/e371433fcd1390a0b82cfb18effc422c1f5d8c1d/centos/9/x86_64/flavors/debug/cephadm 2026-02-23T13:05:49.073 INFO:tasks.cephadm:Downloading cephadm from url: https://3.chacra.ceph.com/binaries/ceph/wip-shraddhaag-enable-cephadm-tests/e371433fcd1390a0b82cfb18effc422c1f5d8c1d/centos/9/x86_64/flavors/debug/cephadm 2026-02-23T13:05:49.073 DEBUG:teuthology.orchestra.run.trial068:> curl --silent -L https://3.chacra.ceph.com/binaries/ceph/wip-shraddhaag-enable-cephadm-tests/e371433fcd1390a0b82cfb18effc422c1f5d8c1d/centos/9/x86_64/flavors/debug/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2026-02-23T13:05:49.956 INFO:teuthology.orchestra.run.trial068.stdout:-rw-r--r--. 1 ubuntu ubuntu 1076284 Feb 23 13:05 /home/ubuntu/cephtest/cephadm 2026-02-23T13:05:49.956 DEBUG:teuthology.orchestra.run.trial185:> curl --silent -L https://3.chacra.ceph.com/binaries/ceph/wip-shraddhaag-enable-cephadm-tests/e371433fcd1390a0b82cfb18effc422c1f5d8c1d/centos/9/x86_64/flavors/debug/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2026-02-23T13:05:50.981 INFO:teuthology.orchestra.run.trial185.stdout:-rw-r--r--. 1 ubuntu ubuntu 1076284 Feb 23 13:05 /home/ubuntu/cephtest/cephadm 2026-02-23T13:05:50.982 DEBUG:teuthology.orchestra.run.trial068:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2026-02-23T13:05:51.001 DEBUG:teuthology.orchestra.run.trial185:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2026-02-23T13:05:51.036 INFO:tasks.cephadm:Pulling image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d on all hosts... 2026-02-23T13:05:51.036 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d pull 2026-02-23T13:05:51.044 DEBUG:teuthology.orchestra.run.trial185:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d pull 2026-02-23T13:05:51.281 INFO:teuthology.orchestra.run.trial068.stderr:Pulling container image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d... 2026-02-23T13:05:51.291 INFO:teuthology.orchestra.run.trial185.stderr:Pulling container image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d... 2026-02-23T13:06:00.472 INFO:teuthology.orchestra.run.trial185.stdout:{ 2026-02-23T13:06:00.472 INFO:teuthology.orchestra.run.trial185.stdout: "ceph_version": "ceph version 20.3.0-5439-ge371433f (e371433fcd1390a0b82cfb18effc422c1f5d8c1d) tentacle (dev)", 2026-02-23T13:06:00.472 INFO:teuthology.orchestra.run.trial185.stdout: "image_id": "63df4b83db6f95a8ba965919b3671cb27173117c23b5c950aaadc8913f56213c", 2026-02-23T13:06:00.472 INFO:teuthology.orchestra.run.trial185.stdout: "repo_digests": [ 2026-02-23T13:06:00.472 INFO:teuthology.orchestra.run.trial185.stdout: "quay.ceph.io/ceph-ci/ceph@sha256:49621faaa845c39b2aae72e4fa24123ee9b2ac3fb8ae834753907068ee2972ea" 2026-02-23T13:06:00.472 INFO:teuthology.orchestra.run.trial185.stdout: ] 2026-02-23T13:06:00.472 INFO:teuthology.orchestra.run.trial185.stdout:} 2026-02-23T13:06:00.625 INFO:teuthology.orchestra.run.trial068.stdout:{ 2026-02-23T13:06:00.625 INFO:teuthology.orchestra.run.trial068.stdout: "ceph_version": "ceph version 20.3.0-5439-ge371433f (e371433fcd1390a0b82cfb18effc422c1f5d8c1d) tentacle (dev)", 2026-02-23T13:06:00.626 INFO:teuthology.orchestra.run.trial068.stdout: "image_id": "63df4b83db6f95a8ba965919b3671cb27173117c23b5c950aaadc8913f56213c", 2026-02-23T13:06:00.626 INFO:teuthology.orchestra.run.trial068.stdout: "repo_digests": [ 2026-02-23T13:06:00.626 INFO:teuthology.orchestra.run.trial068.stdout: "quay.ceph.io/ceph-ci/ceph@sha256:49621faaa845c39b2aae72e4fa24123ee9b2ac3fb8ae834753907068ee2972ea" 2026-02-23T13:06:00.626 INFO:teuthology.orchestra.run.trial068.stdout: ] 2026-02-23T13:06:00.626 INFO:teuthology.orchestra.run.trial068.stdout:} 2026-02-23T13:06:00.641 DEBUG:teuthology.orchestra.run.trial068:> sudo mkdir -p /etc/ceph 2026-02-23T13:06:00.703 DEBUG:teuthology.orchestra.run.trial185:> sudo mkdir -p /etc/ceph 2026-02-23T13:06:00.732 DEBUG:teuthology.orchestra.run.trial068:> sudo chmod 777 /etc/ceph 2026-02-23T13:06:00.756 DEBUG:teuthology.orchestra.run.trial185:> sudo chmod 777 /etc/ceph 2026-02-23T13:06:00.797 INFO:tasks.cephadm:Writing seed config... 2026-02-23T13:06:00.798 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2026-02-23T13:06:00.798 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2026-02-23T13:06:00.798 INFO:tasks.cephadm: override: [mgr] debug osd = 10 2026-02-23T13:06:00.798 INFO:tasks.cephadm: override: [client.0] debug ms = 1 2026-02-23T13:06:00.798 INFO:tasks.cephadm: override: [client.0] debug objecter = 20 2026-02-23T13:06:00.798 INFO:tasks.cephadm: override: [client.0] debug rados = 20 2026-02-23T13:06:00.798 INFO:tasks.cephadm: override: [global] crimson_allow_pg_split = False 2026-02-23T13:06:00.798 INFO:tasks.cephadm: override: [global] enable experimental unrecoverable data corrupting features = crimson 2026-02-23T13:06:00.798 INFO:tasks.cephadm: override: [global] ms client mode = crc 2026-02-23T13:06:00.798 INFO:tasks.cephadm: override: [global] ms cluster mode = crc 2026-02-23T13:06:00.798 INFO:tasks.cephadm: override: [global] ms mon client mode = crc 2026-02-23T13:06:00.798 INFO:tasks.cephadm: override: [global] ms mon cluster mode = crc 2026-02-23T13:06:00.798 INFO:tasks.cephadm: override: [global] ms mon service mode = crc 2026-02-23T13:06:00.798 INFO:tasks.cephadm: override: [global] ms service mode = crc 2026-02-23T13:06:00.798 INFO:tasks.cephadm: override: [global] osd_pool_default_min_size = 2 2026-02-23T13:06:00.798 INFO:tasks.cephadm: override: [global] osd_pool_default_size = 3 2026-02-23T13:06:00.798 INFO:tasks.cephadm: override: [mon] debug mon = 20 2026-02-23T13:06:00.798 INFO:tasks.cephadm: override: [mon] debug ms = 1 2026-02-23T13:06:00.798 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2026-02-23T13:06:00.798 INFO:tasks.cephadm: override: [mon] mon min osdmap epochs = 50 2026-02-23T13:06:00.798 INFO:tasks.cephadm: override: [mon] mon osdmap full prune interval = 2 2026-02-23T13:06:00.798 INFO:tasks.cephadm: override: [mon] mon osdmap full prune min = 15 2026-02-23T13:06:00.798 INFO:tasks.cephadm: override: [mon] mon osdmap full prune txsize = 2 2026-02-23T13:06:00.798 INFO:tasks.cephadm: override: [mon] osd pool default crimson = True 2026-02-23T13:06:00.798 INFO:tasks.cephadm: override: [mon] paxos service trim min = 10 2026-02-23T13:06:00.798 INFO:tasks.cephadm: override: [osd] crimson cpu num = 4 2026-02-23T13:06:00.798 INFO:tasks.cephadm: override: [osd] crimson osd obc lru size = 10 2026-02-23T13:06:00.799 INFO:tasks.cephadm: override: [osd] debug ms = 20 2026-02-23T13:06:00.799 INFO:tasks.cephadm: override: [osd] debug osd = 20 2026-02-23T13:06:00.799 INFO:tasks.cephadm: override: [osd] debug seastore = 20 2026-02-23T13:06:00.799 INFO:tasks.cephadm: override: [osd] debug seastore backref = 20 2026-02-23T13:06:00.799 INFO:tasks.cephadm: override: [osd] debug seastore cache = 20 2026-02-23T13:06:00.799 INFO:tasks.cephadm: override: [osd] debug seastore cleaner = 20 2026-02-23T13:06:00.799 INFO:tasks.cephadm: override: [osd] debug seastore device = 20 2026-02-23T13:06:00.799 INFO:tasks.cephadm: override: [osd] debug seastore epm = 20 2026-02-23T13:06:00.799 INFO:tasks.cephadm: override: [osd] debug seastore fixedkv tree = 20 2026-02-23T13:06:00.799 INFO:tasks.cephadm: override: [osd] debug seastore journal = 20 2026-02-23T13:06:00.799 INFO:tasks.cephadm: override: [osd] debug seastore lba = 20 2026-02-23T13:06:00.799 INFO:tasks.cephadm: override: [osd] debug seastore odata = 20 2026-02-23T13:06:00.799 INFO:tasks.cephadm: override: [osd] debug seastore omap = 20 2026-02-23T13:06:00.799 INFO:tasks.cephadm: override: [osd] debug seastore onode = 20 2026-02-23T13:06:00.799 INFO:tasks.cephadm: override: [osd] debug seastore t = 20 2026-02-23T13:06:00.799 INFO:tasks.cephadm: override: [osd] debug seastore tm = 20 2026-02-23T13:06:00.799 INFO:tasks.cephadm: override: [osd] osd blocked scrub grace period = 3600 2026-02-23T13:06:00.799 INFO:tasks.cephadm: override: [osd] osd debug reject backfill probability = 0.3 2026-02-23T13:06:00.799 INFO:tasks.cephadm: override: [osd] osd delete sleep = 1 2026-02-23T13:06:00.799 INFO:tasks.cephadm: override: [osd] osd max backfills = 3 2026-02-23T13:06:00.799 INFO:tasks.cephadm: override: [osd] osd max markdown count = 1000 2026-02-23T13:06:00.799 INFO:tasks.cephadm: override: [osd] osd objectstore = seastore 2026-02-23T13:06:00.799 INFO:tasks.cephadm: override: [osd] osd scrub max interval = 120 2026-02-23T13:06:00.799 INFO:tasks.cephadm: override: [osd] osd scrub min interval = 60 2026-02-23T13:06:00.799 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2026-02-23T13:06:00.799 INFO:tasks.cephadm: override: [osd] osd snap trim sleep = 2 2026-02-23T13:06:00.799 INFO:tasks.cephadm: override: [osd] osd target transaction size = 5 2026-02-23T13:06:00.799 INFO:tasks.cephadm: override: [osd] seastore cachepin size pershard = 64M 2026-02-23T13:06:00.799 INFO:tasks.cephadm: override: [osd] seastore main device type = RANDOM_BLOCK_SSD 2026-02-23T13:06:00.799 INFO:tasks.cephadm: override: [osd] seastore max concurrent transactions = 8 2026-02-23T13:06:00.800 DEBUG:teuthology.orchestra.run.trial068:> set -ex 2026-02-23T13:06:00.800 DEBUG:teuthology.orchestra.run.trial068:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2026-02-23T13:06:00.816 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=isa technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 5f823084-10b8-11f1-be80-d404e6e7d460 crimson_allow_pg_split = False enable experimental unrecoverable data corrupting features = crimson ms client mode = crc ms cluster mode = crc ms mon client mode = crc ms mon cluster mode = crc ms mon service mode = crc ms service mode = crc osd_pool_default_min_size = 2 osd_pool_default_size = 3 [osd] osd scrub load threshold = 5.0 osd scrub max interval = 120 osd mclock profile = high_recovery_ops osd mclock skip benchmark = true osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true crimson cpu num = 4 crimson osd obc lru size = 10 debug ms = 20 debug osd = 20 debug seastore = 20 debug seastore backref = 20 debug seastore cache = 20 debug seastore cleaner = 20 debug seastore device = 20 debug seastore epm = 20 debug seastore fixedkv tree = 20 debug seastore journal = 20 debug seastore lba = 20 debug seastore odata = 20 debug seastore omap = 20 debug seastore onode = 20 debug seastore t = 20 debug seastore tm = 20 osd blocked scrub grace period = 3600 osd debug reject backfill probability = 0.3 osd delete sleep = 1 osd max backfills = 3 osd max markdown count = 1000 osd objectstore = seastore osd scrub min interval = 60 osd snap trim sleep = 2 osd target transaction size = 5 seastore cachepin size pershard = 64M seastore main device type = RANDOM_BLOCK_SSD seastore max concurrent transactions = 8 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 debug osd = 10 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon min osdmap epochs = 50 mon osdmap full prune interval = 2 mon osdmap full prune min = 15 mon osdmap full prune txsize = 2 osd pool default crimson = True paxos service trim min = 10 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client.0] debug ms = 1 debug objecter = 20 debug rados = 20 2026-02-23T13:06:00.816 DEBUG:teuthology.orchestra.run.trial068:mon.a> sudo journalctl -f -n 0 -u ceph-5f823084-10b8-11f1-be80-d404e6e7d460@mon.a.service 2026-02-23T13:06:00.858 DEBUG:teuthology.orchestra.run.trial068:mgr.x> sudo journalctl -f -n 0 -u ceph-5f823084-10b8-11f1-be80-d404e6e7d460@mgr.x.service 2026-02-23T13:06:00.901 INFO:tasks.cephadm:Bootstrapping... 2026-02-23T13:06:00.902 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d -v bootstrap --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id x --orphan-initial-daemons --skip-monitoring-stack --mon-ip 10.20.193.68 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2026-02-23T13:06:01.043 INFO:teuthology.orchestra.run.trial068.stdout:-------------------------------------------------------------------------------- 2026-02-23T13:06:01.043 INFO:teuthology.orchestra.run.trial068.stdout:cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d', '-v', 'bootstrap', '--fsid', '5f823084-10b8-11f1-be80-d404e6e7d460', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'x', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '10.20.193.68', '--skip-admin-label'] 2026-02-23T13:06:01.044 INFO:teuthology.orchestra.run.trial068.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2026-02-23T13:06:01.044 INFO:teuthology.orchestra.run.trial068.stdout:Verifying podman|docker is present... 2026-02-23T13:06:01.059 INFO:teuthology.orchestra.run.trial068.stdout:/bin/podman: stdout 5.6.0 2026-02-23T13:06:01.059 INFO:teuthology.orchestra.run.trial068.stdout:Verifying lvm2 is present... 2026-02-23T13:06:01.060 INFO:teuthology.orchestra.run.trial068.stdout:Verifying time synchronization is in place... 2026-02-23T13:06:01.066 INFO:teuthology.orchestra.run.trial068.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2026-02-23T13:06:01.066 INFO:teuthology.orchestra.run.trial068.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2026-02-23T13:06:01.072 INFO:teuthology.orchestra.run.trial068.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2026-02-23T13:06:01.072 INFO:teuthology.orchestra.run.trial068.stdout:systemctl: stdout inactive 2026-02-23T13:06:01.078 INFO:teuthology.orchestra.run.trial068.stdout:systemctl: stdout enabled 2026-02-23T13:06:01.083 INFO:teuthology.orchestra.run.trial068.stdout:systemctl: stdout active 2026-02-23T13:06:01.083 INFO:teuthology.orchestra.run.trial068.stdout:Unit chronyd.service is enabled and running 2026-02-23T13:06:01.084 INFO:teuthology.orchestra.run.trial068.stdout:Repeating the final host check... 2026-02-23T13:06:01.099 INFO:teuthology.orchestra.run.trial068.stdout:/bin/podman: stdout 5.6.0 2026-02-23T13:06:01.099 INFO:teuthology.orchestra.run.trial068.stdout:podman (/bin/podman) version 5.6.0 is present 2026-02-23T13:06:01.099 INFO:teuthology.orchestra.run.trial068.stdout:systemctl is present 2026-02-23T13:06:01.099 INFO:teuthology.orchestra.run.trial068.stdout:lvcreate is present 2026-02-23T13:06:01.104 INFO:teuthology.orchestra.run.trial068.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2026-02-23T13:06:01.104 INFO:teuthology.orchestra.run.trial068.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2026-02-23T13:06:01.110 INFO:teuthology.orchestra.run.trial068.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2026-02-23T13:06:01.110 INFO:teuthology.orchestra.run.trial068.stdout:systemctl: stdout inactive 2026-02-23T13:06:01.115 INFO:teuthology.orchestra.run.trial068.stdout:systemctl: stdout enabled 2026-02-23T13:06:01.121 INFO:teuthology.orchestra.run.trial068.stdout:systemctl: stdout active 2026-02-23T13:06:01.121 INFO:teuthology.orchestra.run.trial068.stdout:Unit chronyd.service is enabled and running 2026-02-23T13:06:01.121 INFO:teuthology.orchestra.run.trial068.stdout:Host looks OK 2026-02-23T13:06:01.121 INFO:teuthology.orchestra.run.trial068.stdout:Cluster fsid: 5f823084-10b8-11f1-be80-d404e6e7d460 2026-02-23T13:06:01.121 INFO:teuthology.orchestra.run.trial068.stdout:Acquiring lock 139968539507440 on /run/cephadm/5f823084-10b8-11f1-be80-d404e6e7d460.lock 2026-02-23T13:06:01.121 INFO:teuthology.orchestra.run.trial068.stdout:Lock 139968539507440 acquired on /run/cephadm/5f823084-10b8-11f1-be80-d404e6e7d460.lock 2026-02-23T13:06:01.122 INFO:teuthology.orchestra.run.trial068.stdout:Verifying IP 10.20.193.68 port 3300 ... 2026-02-23T13:06:01.122 INFO:teuthology.orchestra.run.trial068.stdout:Verifying IP 10.20.193.68 port 6789 ... 2026-02-23T13:06:01.122 INFO:teuthology.orchestra.run.trial068.stdout:Base mon IP(s) is [10.20.193.68:3300, 10.20.193.68:6789], mon addrv is [v2:10.20.193.68:3300,v1:10.20.193.68:6789] 2026-02-23T13:06:01.124 INFO:teuthology.orchestra.run.trial068.stdout:/sbin/ip: stdout default via 10.20.192.1 dev enp1s0f0 proto dhcp src 10.20.193.68 metric 102 2026-02-23T13:06:01.125 INFO:teuthology.orchestra.run.trial068.stdout:/sbin/ip: stdout 10.20.192.0/20 dev enp1s0f0 proto kernel scope link src 10.20.193.68 metric 102 2026-02-23T13:06:01.125 INFO:teuthology.orchestra.run.trial068.stdout:/sbin/ip: stdout 169.254.3.0/24 dev enp7s0f4u2u2c2 proto kernel scope link src 169.254.3.1 metric 101 2026-02-23T13:06:01.127 INFO:teuthology.orchestra.run.trial068.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2026-02-23T13:06:01.127 INFO:teuthology.orchestra.run.trial068.stdout:/sbin/ip: stdout fe80::/64 dev enp1s0f0 proto kernel metric 256 pref medium 2026-02-23T13:06:01.127 INFO:teuthology.orchestra.run.trial068.stdout:/sbin/ip: stdout fe80::/64 dev enp7s0f4u2u2c2 proto kernel metric 1024 pref medium 2026-02-23T13:06:01.130 INFO:teuthology.orchestra.run.trial068.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2026-02-23T13:06:01.131 INFO:teuthology.orchestra.run.trial068.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2026-02-23T13:06:01.131 INFO:teuthology.orchestra.run.trial068.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2026-02-23T13:06:01.131 INFO:teuthology.orchestra.run.trial068.stdout:/sbin/ip: stdout 2: enp7s0f4u2u2c2: mtu 1500 state UNKNOWN qlen 1000 2026-02-23T13:06:01.131 INFO:teuthology.orchestra.run.trial068.stdout:/sbin/ip: stdout inet6 fe80::6f0e:d6b8:2198:bb9b/64 scope link noprefixroute 2026-02-23T13:06:01.131 INFO:teuthology.orchestra.run.trial068.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2026-02-23T13:06:01.131 INFO:teuthology.orchestra.run.trial068.stdout:/sbin/ip: stdout 3: enp1s0f0: mtu 1500 state UP qlen 1000 2026-02-23T13:06:01.131 INFO:teuthology.orchestra.run.trial068.stdout:/sbin/ip: stdout inet6 fe80::925a:8ff:fe77:60ea/64 scope link 2026-02-23T13:06:01.131 INFO:teuthology.orchestra.run.trial068.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2026-02-23T13:06:01.131 INFO:teuthology.orchestra.run.trial068.stdout:Mon IP `10.20.193.68` is in CIDR network `10.20.192.0/20` 2026-02-23T13:06:01.131 INFO:teuthology.orchestra.run.trial068.stdout:Mon IP `10.20.193.68` is in CIDR network `10.20.192.0/20` 2026-02-23T13:06:01.131 INFO:teuthology.orchestra.run.trial068.stdout:Inferred mon public CIDR from local network configuration ['10.20.192.0/20', '10.20.192.0/20'] 2026-02-23T13:06:01.132 INFO:teuthology.orchestra.run.trial068.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2026-02-23T13:06:01.132 INFO:teuthology.orchestra.run.trial068.stdout:Pulling container image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d... 2026-02-23T13:06:01.394 INFO:teuthology.orchestra.run.trial068.stdout:/bin/podman: stdout 63df4b83db6f95a8ba965919b3671cb27173117c23b5c950aaadc8913f56213c 2026-02-23T13:06:01.394 INFO:teuthology.orchestra.run.trial068.stdout:/bin/podman: stderr Trying to pull quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d... 2026-02-23T13:06:01.394 INFO:teuthology.orchestra.run.trial068.stdout:/bin/podman: stderr Getting image source signatures 2026-02-23T13:06:01.394 INFO:teuthology.orchestra.run.trial068.stdout:/bin/podman: stderr Copying blob sha256:3c07feed93a72a05ea2a792fe5c53c65e7834d6d603a33f73f74750df5837c83 2026-02-23T13:06:01.394 INFO:teuthology.orchestra.run.trial068.stdout:/bin/podman: stderr Copying blob sha256:5dd85e1a50663785d86adae5ef92579a03e6f64059ca9c005010518acdd62048 2026-02-23T13:06:01.395 INFO:teuthology.orchestra.run.trial068.stdout:/bin/podman: stderr Copying config sha256:63df4b83db6f95a8ba965919b3671cb27173117c23b5c950aaadc8913f56213c 2026-02-23T13:06:01.395 INFO:teuthology.orchestra.run.trial068.stdout:/bin/podman: stderr Writing manifest to image destination 2026-02-23T13:06:01.536 INFO:teuthology.orchestra.run.trial068.stdout:ceph: stdout ceph version 20.3.0-5439-ge371433f (e371433fcd1390a0b82cfb18effc422c1f5d8c1d) tentacle (dev) 2026-02-23T13:06:01.536 INFO:teuthology.orchestra.run.trial068.stdout:Ceph version: ceph version 20.3.0-5439-ge371433f (e371433fcd1390a0b82cfb18effc422c1f5d8c1d) tentacle (dev) 2026-02-23T13:06:01.536 INFO:teuthology.orchestra.run.trial068.stdout:Extracting ceph user uid/gid from container image... 2026-02-23T13:06:01.679 INFO:teuthology.orchestra.run.trial068.stdout:stat: stdout 167 167 2026-02-23T13:06:01.679 INFO:teuthology.orchestra.run.trial068.stdout:Creating initial keys... 2026-02-23T13:06:01.805 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph-authtool: stdout AQC5UJxpo8RRLxAAHzLazgtkTJ3nfaTQ5DtlEQ== 2026-02-23T13:06:01.927 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph-authtool: stdout AQC5UJxpFNiRNhAAg2bHTSxipyodb+y8gctyqA== 2026-02-23T13:06:02.055 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph-authtool: stdout AQC6UJxpkQ2KAhAAfElfDjHMFiVJmC3tOUOnUg== 2026-02-23T13:06:02.056 INFO:teuthology.orchestra.run.trial068.stdout:Creating initial monmap... 2026-02-23T13:06:02.194 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2026-02-23T13:06:02.194 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = tentacle 2026-02-23T13:06:02.194 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to 5f823084-10b8-11f1-be80-d404e6e7d460 2026-02-23T13:06:02.195 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2026-02-23T13:06:02.195 INFO:teuthology.orchestra.run.trial068.stdout:monmaptool for a [v2:10.20.193.68:3300,v1:10.20.193.68:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2026-02-23T13:06:02.195 INFO:teuthology.orchestra.run.trial068.stdout:setting min_mon_release = tentacle 2026-02-23T13:06:02.195 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/monmaptool: set fsid to 5f823084-10b8-11f1-be80-d404e6e7d460 2026-02-23T13:06:02.195 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2026-02-23T13:06:02.195 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T13:06:02.195 INFO:teuthology.orchestra.run.trial068.stdout:Creating mon... 2026-02-23T13:06:02.335 INFO:teuthology.orchestra.run.trial068.stdout:create mon.a on 2026-02-23T13:06:02.497 INFO:teuthology.orchestra.run.trial068.stdout:systemctl: stderr Removed "/etc/systemd/system/multi-user.target.wants/ceph.target". 2026-02-23T13:06:02.596 INFO:teuthology.orchestra.run.trial068.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2026-02-23T13:06:02.699 INFO:teuthology.orchestra.run.trial068.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-5f823084-10b8-11f1-be80-d404e6e7d460.target → /etc/systemd/system/ceph-5f823084-10b8-11f1-be80-d404e6e7d460.target. 2026-02-23T13:06:02.699 INFO:teuthology.orchestra.run.trial068.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-5f823084-10b8-11f1-be80-d404e6e7d460.target → /etc/systemd/system/ceph-5f823084-10b8-11f1-be80-d404e6e7d460.target. 2026-02-23T13:06:02.819 INFO:teuthology.orchestra.run.trial068.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-5f823084-10b8-11f1-be80-d404e6e7d460@mon.a 2026-02-23T13:06:02.820 INFO:teuthology.orchestra.run.trial068.stdout:systemctl: stderr Failed to reset failed state of unit ceph-5f823084-10b8-11f1-be80-d404e6e7d460@mon.a.service: Unit ceph-5f823084-10b8-11f1-be80-d404e6e7d460@mon.a.service not loaded. 2026-02-23T13:06:02.939 INFO:teuthology.orchestra.run.trial068.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-5f823084-10b8-11f1-be80-d404e6e7d460.target.wants/ceph-5f823084-10b8-11f1-be80-d404e6e7d460@mon.a.service → /etc/systemd/system/ceph-5f823084-10b8-11f1-be80-d404e6e7d460@.service. 2026-02-23T13:06:02.957 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:02 trial068 systemd[1]: Starting Ceph mon.a for 5f823084-10b8-11f1-be80-d404e6e7d460... 2026-02-23T13:06:03.087 INFO:teuthology.orchestra.run.trial068.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2026-02-23T13:06:03.087 INFO:teuthology.orchestra.run.trial068.stdout:systemctl: stdout disabled 2026-02-23T13:06:03.093 INFO:teuthology.orchestra.run.trial068.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2026-02-23T13:06:03.094 INFO:teuthology.orchestra.run.trial068.stdout:systemctl: stdout inactive 2026-02-23T13:06:03.094 INFO:teuthology.orchestra.run.trial068.stdout:firewalld.service is not enabled 2026-02-23T13:06:03.094 INFO:teuthology.orchestra.run.trial068.stdout:Not possible to enable service . firewalld.service is not available 2026-02-23T13:06:03.094 INFO:teuthology.orchestra.run.trial068.stdout:Waiting for mon to start... 2026-02-23T13:06:03.095 INFO:teuthology.orchestra.run.trial068.stdout:Waiting for mon... 2026-02-23T13:06:03.211 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:03 trial068 podman[38338]: 2026-02-23 13:06:03.04382825 +0000 UTC m=+0.016273627 container create 161fc0c6fd624d2dae92e00205427f581b8c3b0b2fcbb6627272b40bf6d7b76a (image=quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d, name=ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mon-a, io.buildah.version=1.41.3, CEPH_SHA1=e371433fcd1390a0b82cfb18effc422c1f5d8c1d, CEPH_REF=wip-shraddhaag-enable-cephadm-tests, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.opencontainers.image.documentation=https://docs.ceph.com/, FROM_IMAGE=quay.io/centos/centos:stream9, OSD_FLAVOR=default, org.label-schema.schema-version=1.0, org.opencontainers.image.authors=Ceph Release Team , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfsganesha-5/, org.label-schema.build-date=20260216, CEPH_GIT_REPO=https://github.com/ceph/ceph-ci.git) 2026-02-23T13:06:03.211 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:03 trial068 podman[38338]: 2026-02-23 13:06:03.071342585 +0000 UTC m=+0.043787972 container init 161fc0c6fd624d2dae92e00205427f581b8c3b0b2fcbb6627272b40bf6d7b76a (image=quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d, name=ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mon-a, org.opencontainers.image.documentation=https://docs.ceph.com/, org.opencontainers.image.authors=Ceph Release Team , CEPH_GIT_REPO=https://github.com/ceph/ceph-ci.git, FROM_IMAGE=quay.io/centos/centos:stream9, OSD_FLAVOR=default, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20260216, io.buildah.version=1.41.3, CEPH_REF=wip-shraddhaag-enable-cephadm-tests, org.label-schema.license=GPLv2, ceph=True, org.label-schema.vendor=CentOS, GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfsganesha-5/, CEPH_SHA1=e371433fcd1390a0b82cfb18effc422c1f5d8c1d, org.label-schema.schema-version=1.0) 2026-02-23T13:06:03.212 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:03 trial068 podman[38338]: 2026-02-23 13:06:03.073791582 +0000 UTC m=+0.046236959 container start 161fc0c6fd624d2dae92e00205427f581b8c3b0b2fcbb6627272b40bf6d7b76a (image=quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d, name=ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mon-a, org.opencontainers.image.documentation=https://docs.ceph.com/, io.buildah.version=1.41.3, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_SHA1=e371433fcd1390a0b82cfb18effc422c1f5d8c1d, org.opencontainers.image.authors=Ceph Release Team , CEPH_GIT_REPO=https://github.com/ceph/ceph-ci.git, OSD_FLAVOR=default, org.label-schema.license=GPLv2, ceph=True, CEPH_REF=wip-shraddhaag-enable-cephadm-tests, FROM_IMAGE=quay.io/centos/centos:stream9, GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfsganesha-5/, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20260216) 2026-02-23T13:06:03.212 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:03 trial068 bash[38338]: 161fc0c6fd624d2dae92e00205427f581b8c3b0b2fcbb6627272b40bf6d7b76a 2026-02-23T13:06:03.212 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:03 trial068 podman[38338]: 2026-02-23 13:06:03.037273627 +0000 UTC m=+0.009719014 image pull 63df4b83db6f95a8ba965919b3671cb27173117c23b5c950aaadc8913f56213c quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d 2026-02-23T13:06:03.212 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:03 trial068 systemd[1]: Started Ceph mon.a for 5f823084-10b8-11f1-be80-d404e6e7d460. 2026-02-23T13:06:03.212 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:03 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mon-a[38361]: 2026-02-23T13:06:03.087+0000 7f8805f74c00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:06:03.212 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:03 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mon-a[38361]: 2026-02-23T13:06:03.087+0000 7f8805f74c00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:06:03.212 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:03 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mon-a[38361]: 2026-02-23T13:06:03.087+0000 7f8805f74c00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:06:03.212 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:03 trial068 ceph-mon[38365]: mkfs 5f823084-10b8-11f1-be80-d404e6e7d460 2026-02-23T13:06:03.212 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:03 trial068 ceph-mon[38365]: mon.a is new leader, mons a in quorum (ranks 0) 2026-02-23T13:06:03.242 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stdout cluster: 2026-02-23T13:06:03.242 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stdout id: 5f823084-10b8-11f1-be80-d404e6e7d460 2026-02-23T13:06:03.242 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2026-02-23T13:06:03.242 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stdout 2026-02-23T13:06:03.242 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stdout services: 2026-02-23T13:06:03.242 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 0.124005s) [leader: a] 2026-02-23T13:06:03.242 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2026-02-23T13:06:03.242 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2026-02-23T13:06:03.243 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stdout 2026-02-23T13:06:03.243 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stdout data: 2026-02-23T13:06:03.243 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2026-02-23T13:06:03.243 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2026-02-23T13:06:03.243 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2026-02-23T13:06:03.243 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stdout pgs: 2026-02-23T13:06:03.243 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stdout 2026-02-23T13:06:03.243 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stderr 2026-02-23T13:06:03.185+0000 7f1c21489640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:06:03.243 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stderr 2026-02-23T13:06:03.189+0000 7f1c21489640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:06:03.243 INFO:teuthology.orchestra.run.trial068.stdout:mon is available 2026-02-23T13:06:03.243 INFO:teuthology.orchestra.run.trial068.stdout:Assimilating anything we can from ceph.conf... 2026-02-23T13:06:03.391 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stdout 2026-02-23T13:06:03.392 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stdout [global] 2026-02-23T13:06:03.392 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stdout fsid = 5f823084-10b8-11f1-be80-d404e6e7d460 2026-02-23T13:06:03.392 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stdout mon_cluster_log_file_level = debug 2026-02-23T13:06:03.392 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stdout mon_host = [v2:10.20.193.68:3300,v1:10.20.193.68:6789] 2026-02-23T13:06:03.392 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2026-02-23T13:06:03.392 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2026-02-23T13:06:03.392 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2026-02-23T13:06:03.392 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2026-02-23T13:06:03.392 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stdout 2026-02-23T13:06:03.392 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stdout [mgr] 2026-02-23T13:06:03.393 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2026-02-23T13:06:03.393 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stdout 2026-02-23T13:06:03.393 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stdout [osd] 2026-02-23T13:06:03.393 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2026-02-23T13:06:03.393 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2026-02-23T13:06:03.393 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stderr 2026-02-23T13:06:03.335+0000 7f0a6263f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:06:03.393 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stderr 2026-02-23T13:06:03.337+0000 7f0a6263f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:06:03.393 INFO:teuthology.orchestra.run.trial068.stdout:Generating new minimal ceph.conf... 2026-02-23T13:06:03.558 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stderr 2026-02-23T13:06:03.491+0000 7fba0be4c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:06:03.559 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stderr 2026-02-23T13:06:03.495+0000 7fba0be4c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:06:03.559 INFO:teuthology.orchestra.run.trial068.stdout:Restarting the monitor... 2026-02-23T13:06:03.765 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:03 trial068 systemd[1]: Stopping Ceph mon.a for 5f823084-10b8-11f1-be80-d404e6e7d460... 2026-02-23T13:06:03.765 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:03 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mon-a[38361]: 2026-02-23T13:06:03.637+0000 7f88045ec640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2026-02-23T13:06:03.765 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:03 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mon-a[38361]: 2026-02-23T13:06:03.637+0000 7f88045ec640 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2026-02-23T13:06:03.765 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:03 trial068 podman[38647]: 2026-02-23 13:06:03.729410429 +0000 UTC m=+0.103241556 container died 161fc0c6fd624d2dae92e00205427f581b8c3b0b2fcbb6627272b40bf6d7b76a (image=quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d, name=ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mon-a, FROM_IMAGE=quay.io/centos/centos:stream9, CEPH_GIT_REPO=https://github.com/ceph/ceph-ci.git, CEPH_SHA1=e371433fcd1390a0b82cfb18effc422c1f5d8c1d, io.buildah.version=1.41.3, org.label-schema.name=CentOS Stream 9 Base Image, org.opencontainers.image.authors=Ceph Release Team , org.label-schema.vendor=CentOS, GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfsganesha-5/, CEPH_REF=wip-shraddhaag-enable-cephadm-tests, ceph=True, org.label-schema.license=GPLv2, OSD_FLAVOR=default, org.label-schema.build-date=20260216, org.opencontainers.image.documentation=https://docs.ceph.com/, org.label-schema.schema-version=1.0) 2026-02-23T13:06:03.766 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:03 trial068 podman[38647]: 2026-02-23 13:06:03.737587889 +0000 UTC m=+0.111419006 container remove 161fc0c6fd624d2dae92e00205427f581b8c3b0b2fcbb6627272b40bf6d7b76a (image=quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d, name=ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mon-a, OSD_FLAVOR=default, org.label-schema.license=GPLv2, org.opencontainers.image.authors=Ceph Release Team , org.opencontainers.image.documentation=https://docs.ceph.com/, CEPH_GIT_REPO=https://github.com/ceph/ceph-ci.git, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_REF=wip-shraddhaag-enable-cephadm-tests, FROM_IMAGE=quay.io/centos/centos:stream9, org.label-schema.vendor=CentOS, CEPH_SHA1=e371433fcd1390a0b82cfb18effc422c1f5d8c1d, GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfsganesha-5/, ceph=True, io.buildah.version=1.41.3, org.label-schema.build-date=20260216, org.label-schema.schema-version=1.0) 2026-02-23T13:06:03.766 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:03 trial068 bash[38647]: ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mon-a 2026-02-23T13:06:03.925 INFO:teuthology.orchestra.run.trial068.stdout:Setting public_network to 10.20.192.0/20 in global config section 2026-02-23T13:06:04.068 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:03 trial068 systemd[1]: ceph-5f823084-10b8-11f1-be80-d404e6e7d460@mon.a.service: Deactivated successfully. 2026-02-23T13:06:04.068 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:03 trial068 systemd[1]: Stopped Ceph mon.a for 5f823084-10b8-11f1-be80-d404e6e7d460. 2026-02-23T13:06:04.069 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:03 trial068 systemd[1]: Starting Ceph mon.a for 5f823084-10b8-11f1-be80-d404e6e7d460... 2026-02-23T13:06:04.069 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:03 trial068 podman[38774]: 2026-02-23 13:06:03.888408397 +0000 UTC m=+0.014681198 container create 0619909708becb855ed6fa914645373fae47493db7cf119301891ac1aedcdc96 (image=quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d, name=ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mon-a, ceph=True, org.label-schema.schema-version=1.0, CEPH_GIT_REPO=https://github.com/ceph/ceph-ci.git, org.label-schema.license=GPLv2, org.opencontainers.image.authors=Ceph Release Team , org.label-schema.vendor=CentOS, GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfsganesha-5/, org.opencontainers.image.documentation=https://docs.ceph.com/, org.label-schema.build-date=20260216, CEPH_SHA1=e371433fcd1390a0b82cfb18effc422c1f5d8c1d, FROM_IMAGE=quay.io/centos/centos:stream9, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_REF=wip-shraddhaag-enable-cephadm-tests, OSD_FLAVOR=default, io.buildah.version=1.41.3) 2026-02-23T13:06:04.069 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:03 trial068 podman[38774]: 2026-02-23 13:06:03.917971468 +0000 UTC m=+0.044244279 container init 0619909708becb855ed6fa914645373fae47493db7cf119301891ac1aedcdc96 (image=quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d, name=ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mon-a, CEPH_GIT_REPO=https://github.com/ceph/ceph-ci.git, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.41.3, org.opencontainers.image.documentation=https://docs.ceph.com/, ceph=True, GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfsganesha-5/, CEPH_REF=wip-shraddhaag-enable-cephadm-tests, CEPH_SHA1=e371433fcd1390a0b82cfb18effc422c1f5d8c1d, FROM_IMAGE=quay.io/centos/centos:stream9, org.opencontainers.image.authors=Ceph Release Team , org.label-schema.build-date=20260216, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, OSD_FLAVOR=default, org.label-schema.license=GPLv2) 2026-02-23T13:06:04.069 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:03 trial068 podman[38774]: 2026-02-23 13:06:03.920283828 +0000 UTC m=+0.046556639 container start 0619909708becb855ed6fa914645373fae47493db7cf119301891ac1aedcdc96 (image=quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d, name=ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mon-a, org.opencontainers.image.documentation=https://docs.ceph.com/, CEPH_SHA1=e371433fcd1390a0b82cfb18effc422c1f5d8c1d, FROM_IMAGE=quay.io/centos/centos:stream9, GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfsganesha-5/, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.opencontainers.image.authors=Ceph Release Team , org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.41.3, ceph=True, org.label-schema.vendor=CentOS, CEPH_GIT_REPO=https://github.com/ceph/ceph-ci.git, OSD_FLAVOR=default, org.label-schema.build-date=20260216, CEPH_REF=wip-shraddhaag-enable-cephadm-tests) 2026-02-23T13:06:04.069 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:03 trial068 bash[38774]: 0619909708becb855ed6fa914645373fae47493db7cf119301891ac1aedcdc96 2026-02-23T13:06:04.069 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:03 trial068 podman[38774]: 2026-02-23 13:06:03.882747902 +0000 UTC m=+0.009020703 image pull 63df4b83db6f95a8ba965919b3671cb27173117c23b5c950aaadc8913f56213c quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d 2026-02-23T13:06:04.069 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:03 trial068 systemd[1]: Started Ceph mon.a for 5f823084-10b8-11f1-be80-d404e6e7d460. 2026-02-23T13:06:04.069 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:03 trial068 ceph-mon[38805]: set uid:gid to 167:167 (ceph:ceph) 2026-02-23T13:06:04.069 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:03 trial068 ceph-mon[38805]: ceph version 20.3.0-5439-ge371433f (e371433fcd1390a0b82cfb18effc422c1f5d8c1d) tentacle (dev - RelWithDebInfo), process ceph-mon, pid 2 2026-02-23T13:06:04.069 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:03 trial068 ceph-mon[38805]: pidfile_write: ignore empty --pid-file 2026-02-23T13:06:04.069 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:03 trial068 ceph-mon[38805]: load: jerasure load: lrc 2026-02-23T13:06:04.070 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:03 trial068 ceph-mon[38805]: rocksdb: RocksDB version: 7.9.2 2026-02-23T13:06:04.070 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:03 trial068 ceph-mon[38805]: rocksdb: Git sha 0 2026-02-23T13:06:04.070 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:03 trial068 ceph-mon[38805]: rocksdb: Compile date 2026-02-23 07:43:00 2026-02-23T13:06:04.070 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:03 trial068 ceph-mon[38805]: rocksdb: DB SUMMARY 2026-02-23T13:06:04.070 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:03 trial068 ceph-mon[38805]: rocksdb: DB Session ID: GQRBEIYPCNVFBA5TD0GX 2026-02-23T13:06:04.070 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:03 trial068 ceph-mon[38805]: rocksdb: CURRENT file: CURRENT 2026-02-23T13:06:04.070 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:03 trial068 ceph-mon[38805]: rocksdb: IDENTITY file: IDENTITY 2026-02-23T13:06:04.070 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:03 trial068 ceph-mon[38805]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2026-02-23T13:06:04.070 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:03 trial068 ceph-mon[38805]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2026-02-23T13:06:04.070 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:03 trial068 ceph-mon[38805]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000009.log size: 99658 ; 2026-02-23T13:06:04.070 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:03 trial068 ceph-mon[38805]: rocksdb: Options.error_if_exists: 0 2026-02-23T13:06:04.070 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:03 trial068 ceph-mon[38805]: rocksdb: Options.create_if_missing: 0 2026-02-23T13:06:04.070 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:03 trial068 ceph-mon[38805]: rocksdb: Options.paranoid_checks: 1 2026-02-23T13:06:04.071 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:03 trial068 ceph-mon[38805]: rocksdb: Options.flush_verify_memtable_count: 1 2026-02-23T13:06:04.071 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:03 trial068 ceph-mon[38805]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2026-02-23T13:06:04.071 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:03 trial068 ceph-mon[38805]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2026-02-23T13:06:04.071 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:03 trial068 ceph-mon[38805]: rocksdb: Options.env: 0xed5d60 2026-02-23T13:06:04.071 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:03 trial068 ceph-mon[38805]: rocksdb: Options.fs: PosixFileSystem 2026-02-23T13:06:04.071 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:03 trial068 ceph-mon[38805]: rocksdb: Options.info_log: 0x362cec0 2026-02-23T13:06:04.071 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:03 trial068 ceph-mon[38805]: rocksdb: Options.max_file_opening_threads: 16 2026-02-23T13:06:04.071 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:03 trial068 ceph-mon[38805]: rocksdb: Options.statistics: (nil) 2026-02-23T13:06:04.071 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:03 trial068 ceph-mon[38805]: rocksdb: Options.use_fsync: 0 2026-02-23T13:06:04.071 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:03 trial068 ceph-mon[38805]: rocksdb: Options.max_log_file_size: 0 2026-02-23T13:06:04.071 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:03 trial068 ceph-mon[38805]: rocksdb: Options.max_manifest_file_size: 1073741824 2026-02-23T13:06:04.071 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:03 trial068 ceph-mon[38805]: rocksdb: Options.log_file_time_to_roll: 0 2026-02-23T13:06:04.071 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:03 trial068 ceph-mon[38805]: rocksdb: Options.keep_log_file_num: 1000 2026-02-23T13:06:04.072 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:03 trial068 ceph-mon[38805]: rocksdb: Options.recycle_log_file_num: 0 2026-02-23T13:06:04.072 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:03 trial068 ceph-mon[38805]: rocksdb: Options.allow_fallocate: 1 2026-02-23T13:06:04.072 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:03 trial068 ceph-mon[38805]: rocksdb: Options.allow_mmap_reads: 0 2026-02-23T13:06:04.072 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:03 trial068 ceph-mon[38805]: rocksdb: Options.allow_mmap_writes: 0 2026-02-23T13:06:04.072 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:03 trial068 ceph-mon[38805]: rocksdb: Options.use_direct_reads: 0 2026-02-23T13:06:04.072 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:03 trial068 ceph-mon[38805]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2026-02-23T13:06:04.072 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:03 trial068 ceph-mon[38805]: rocksdb: Options.create_missing_column_families: 0 2026-02-23T13:06:04.072 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:03 trial068 ceph-mon[38805]: rocksdb: Options.db_log_dir: 2026-02-23T13:06:04.072 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:03 trial068 ceph-mon[38805]: rocksdb: Options.wal_dir: 2026-02-23T13:06:04.072 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:03 trial068 ceph-mon[38805]: rocksdb: Options.table_cache_numshardbits: 6 2026-02-23T13:06:04.072 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:03 trial068 ceph-mon[38805]: rocksdb: Options.WAL_ttl_seconds: 0 2026-02-23T13:06:04.072 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:03 trial068 ceph-mon[38805]: rocksdb: Options.WAL_size_limit_MB: 0 2026-02-23T13:06:04.073 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:03 trial068 ceph-mon[38805]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2026-02-23T13:06:04.073 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:03 trial068 ceph-mon[38805]: rocksdb: Options.manifest_preallocation_size: 4194304 2026-02-23T13:06:04.073 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:03 trial068 ceph-mon[38805]: rocksdb: Options.is_fd_close_on_exec: 1 2026-02-23T13:06:04.073 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:03 trial068 ceph-mon[38805]: rocksdb: Options.advise_random_on_open: 1 2026-02-23T13:06:04.073 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:03 trial068 ceph-mon[38805]: rocksdb: Options.db_write_buffer_size: 0 2026-02-23T13:06:04.073 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:03 trial068 ceph-mon[38805]: rocksdb: Options.write_buffer_manager: 0x3628140 2026-02-23T13:06:04.073 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:03 trial068 ceph-mon[38805]: rocksdb: Options.access_hint_on_compaction_start: 1 2026-02-23T13:06:04.073 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:03 trial068 ceph-mon[38805]: rocksdb: Options.random_access_max_buffer_size: 1048576 2026-02-23T13:06:04.073 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:03 trial068 ceph-mon[38805]: rocksdb: Options.use_adaptive_mutex: 0 2026-02-23T13:06:04.073 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:03 trial068 ceph-mon[38805]: rocksdb: Options.rate_limiter: (nil) 2026-02-23T13:06:04.073 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:03 trial068 ceph-mon[38805]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2026-02-23T13:06:04.073 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:03 trial068 ceph-mon[38805]: rocksdb: Options.wal_recovery_mode: 2 2026-02-23T13:06:04.073 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:03 trial068 ceph-mon[38805]: rocksdb: Options.enable_thread_tracking: 0 2026-02-23T13:06:04.074 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:03 trial068 ceph-mon[38805]: rocksdb: Options.enable_pipelined_write: 0 2026-02-23T13:06:04.074 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:03 trial068 ceph-mon[38805]: rocksdb: Options.unordered_write: 0 2026-02-23T13:06:04.074 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:03 trial068 ceph-mon[38805]: rocksdb: Options.allow_concurrent_memtable_write: 1 2026-02-23T13:06:04.074 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:03 trial068 ceph-mon[38805]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2026-02-23T13:06:04.074 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:03 trial068 ceph-mon[38805]: rocksdb: Options.write_thread_max_yield_usec: 100 2026-02-23T13:06:04.074 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:03 trial068 ceph-mon[38805]: rocksdb: Options.write_thread_slow_yield_usec: 3 2026-02-23T13:06:04.074 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:03 trial068 ceph-mon[38805]: rocksdb: Options.row_cache: None 2026-02-23T13:06:04.074 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:03 trial068 ceph-mon[38805]: rocksdb: Options.wal_filter: None 2026-02-23T13:06:04.074 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:03 trial068 ceph-mon[38805]: rocksdb: Options.avoid_flush_during_recovery: 0 2026-02-23T13:06:04.074 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:03 trial068 ceph-mon[38805]: rocksdb: Options.allow_ingest_behind: 0 2026-02-23T13:06:04.074 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:03 trial068 ceph-mon[38805]: rocksdb: Options.two_write_queues: 0 2026-02-23T13:06:04.074 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:03 trial068 ceph-mon[38805]: rocksdb: Options.manual_wal_flush: 0 2026-02-23T13:06:04.074 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:03 trial068 ceph-mon[38805]: rocksdb: Options.wal_compression: 0 2026-02-23T13:06:04.075 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:03 trial068 ceph-mon[38805]: rocksdb: Options.atomic_flush: 0 2026-02-23T13:06:04.075 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:03 trial068 ceph-mon[38805]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2026-02-23T13:06:04.075 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:03 trial068 ceph-mon[38805]: rocksdb: Options.persist_stats_to_disk: 0 2026-02-23T13:06:04.075 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:03 trial068 ceph-mon[38805]: rocksdb: Options.write_dbid_to_manifest: 0 2026-02-23T13:06:04.075 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:03 trial068 ceph-mon[38805]: rocksdb: Options.log_readahead_size: 0 2026-02-23T13:06:04.075 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:03 trial068 ceph-mon[38805]: rocksdb: Options.file_checksum_gen_factory: Unknown 2026-02-23T13:06:04.075 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:03 trial068 ceph-mon[38805]: rocksdb: Options.best_efforts_recovery: 0 2026-02-23T13:06:04.075 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:03 trial068 ceph-mon[38805]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2026-02-23T13:06:04.075 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:03 trial068 ceph-mon[38805]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2026-02-23T13:06:04.075 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:03 trial068 ceph-mon[38805]: rocksdb: Options.allow_data_in_errors: 0 2026-02-23T13:06:04.075 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:03 trial068 ceph-mon[38805]: rocksdb: Options.db_host_id: __hostname__ 2026-02-23T13:06:04.075 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:03 trial068 ceph-mon[38805]: rocksdb: Options.enforce_single_del_contracts: true 2026-02-23T13:06:04.075 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:03 trial068 ceph-mon[38805]: rocksdb: Options.max_background_jobs: 2 2026-02-23T13:06:04.076 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:03 trial068 ceph-mon[38805]: rocksdb: Options.max_background_compactions: -1 2026-02-23T13:06:04.076 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:03 trial068 ceph-mon[38805]: rocksdb: Options.max_subcompactions: 1 2026-02-23T13:06:04.076 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:03 trial068 ceph-mon[38805]: rocksdb: Options.avoid_flush_during_shutdown: 0 2026-02-23T13:06:04.076 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:03 trial068 ceph-mon[38805]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2026-02-23T13:06:04.076 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:03 trial068 ceph-mon[38805]: rocksdb: Options.delayed_write_rate : 16777216 2026-02-23T13:06:04.076 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:03 trial068 ceph-mon[38805]: rocksdb: Options.max_total_wal_size: 0 2026-02-23T13:06:04.076 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:03 trial068 ceph-mon[38805]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2026-02-23T13:06:04.076 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:03 trial068 ceph-mon[38805]: rocksdb: Options.stats_dump_period_sec: 600 2026-02-23T13:06:04.076 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:03 trial068 ceph-mon[38805]: rocksdb: Options.stats_persist_period_sec: 600 2026-02-23T13:06:04.076 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:03 trial068 ceph-mon[38805]: rocksdb: Options.stats_history_buffer_size: 1048576 2026-02-23T13:06:04.076 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:03 trial068 ceph-mon[38805]: rocksdb: Options.max_open_files: -1 2026-02-23T13:06:04.076 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:03 trial068 ceph-mon[38805]: rocksdb: Options.bytes_per_sync: 0 2026-02-23T13:06:04.077 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:03 trial068 ceph-mon[38805]: rocksdb: Options.wal_bytes_per_sync: 0 2026-02-23T13:06:04.077 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:03 trial068 ceph-mon[38805]: rocksdb: Options.strict_bytes_per_sync: 0 2026-02-23T13:06:04.077 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:03 trial068 ceph-mon[38805]: rocksdb: Options.compaction_readahead_size: 0 2026-02-23T13:06:04.077 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:03 trial068 ceph-mon[38805]: rocksdb: Options.max_background_flushes: -1 2026-02-23T13:06:04.077 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:03 trial068 ceph-mon[38805]: rocksdb: Compression algorithms supported: 2026-02-23T13:06:04.077 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:03 trial068 ceph-mon[38805]: rocksdb: kZSTD supported: 0 2026-02-23T13:06:04.077 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:03 trial068 ceph-mon[38805]: rocksdb: kXpressCompression supported: 0 2026-02-23T13:06:04.077 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:03 trial068 ceph-mon[38805]: rocksdb: kBZip2Compression supported: 0 2026-02-23T13:06:04.077 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:03 trial068 ceph-mon[38805]: rocksdb: kZSTDNotFinalCompression supported: 0 2026-02-23T13:06:04.077 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:03 trial068 ceph-mon[38805]: rocksdb: kLZ4Compression supported: 1 2026-02-23T13:06:04.077 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:03 trial068 ceph-mon[38805]: rocksdb: kZlibCompression supported: 1 2026-02-23T13:06:04.077 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:03 trial068 ceph-mon[38805]: rocksdb: kLZ4HCCompression supported: 1 2026-02-23T13:06:04.077 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:03 trial068 ceph-mon[38805]: rocksdb: kSnappyCompression supported: 1 2026-02-23T13:06:04.078 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:03 trial068 ceph-mon[38805]: rocksdb: Fast CRC32 supported: Supported on x86 2026-02-23T13:06:04.078 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:03 trial068 ceph-mon[38805]: rocksdb: DMutex implementation: pthread_mutex_t 2026-02-23T13:06:04.078 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:03 trial068 ceph-mon[38805]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 2026-02-23T13:06:04.078 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:03 trial068 ceph-mon[38805]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2026-02-23T13:06:04.078 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:03 trial068 ceph-mon[38805]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2026-02-23T13:06:04.078 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:03 trial068 ceph-mon[38805]: rocksdb: Options.merge_operator: 2026-02-23T13:06:04.078 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:03 trial068 ceph-mon[38805]: rocksdb: Options.compaction_filter: None 2026-02-23T13:06:04.078 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:03 trial068 ceph-mon[38805]: rocksdb: Options.compaction_filter_factory: None 2026-02-23T13:06:04.078 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:03 trial068 ceph-mon[38805]: rocksdb: Options.sst_partitioner_factory: None 2026-02-23T13:06:04.078 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:03 trial068 ceph-mon[38805]: rocksdb: Options.memtable_factory: SkipListFactory 2026-02-23T13:06:04.078 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:03 trial068 ceph-mon[38805]: rocksdb: Options.table_factory: BlockBasedTable 2026-02-23T13:06:04.078 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:03 trial068 ceph-mon[38805]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x362d600) 2026-02-23T13:06:04.078 INFO:journalctl@ceph.mon.a.trial068.stdout: cache_index_and_filter_blocks: 1 2026-02-23T13:06:04.078 INFO:journalctl@ceph.mon.a.trial068.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2026-02-23T13:06:04.079 INFO:journalctl@ceph.mon.a.trial068.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2026-02-23T13:06:04.079 INFO:journalctl@ceph.mon.a.trial068.stdout: pin_top_level_index_and_filter: 1 2026-02-23T13:06:04.079 INFO:journalctl@ceph.mon.a.trial068.stdout: index_type: 0 2026-02-23T13:06:04.079 INFO:journalctl@ceph.mon.a.trial068.stdout: data_block_index_type: 0 2026-02-23T13:06:04.079 INFO:journalctl@ceph.mon.a.trial068.stdout: index_shortening: 1 2026-02-23T13:06:04.079 INFO:journalctl@ceph.mon.a.trial068.stdout: data_block_hash_table_util_ratio: 0.750000 2026-02-23T13:06:04.079 INFO:journalctl@ceph.mon.a.trial068.stdout: checksum: 4 2026-02-23T13:06:04.079 INFO:journalctl@ceph.mon.a.trial068.stdout: no_block_cache: 0 2026-02-23T13:06:04.079 INFO:journalctl@ceph.mon.a.trial068.stdout: block_cache: 0x35fba50 2026-02-23T13:06:04.079 INFO:journalctl@ceph.mon.a.trial068.stdout: block_cache_name: BinnedLRUCache 2026-02-23T13:06:04.079 INFO:journalctl@ceph.mon.a.trial068.stdout: block_cache_options: 2026-02-23T13:06:04.079 INFO:journalctl@ceph.mon.a.trial068.stdout: capacity : 536870912 2026-02-23T13:06:04.080 INFO:journalctl@ceph.mon.a.trial068.stdout: num_shard_bits : 4 2026-02-23T13:06:04.080 INFO:journalctl@ceph.mon.a.trial068.stdout: strict_capacity_limit : 0 2026-02-23T13:06:04.080 INFO:journalctl@ceph.mon.a.trial068.stdout: high_pri_pool_ratio: 0.000 2026-02-23T13:06:04.080 INFO:journalctl@ceph.mon.a.trial068.stdout: block_cache_compressed: (nil) 2026-02-23T13:06:04.080 INFO:journalctl@ceph.mon.a.trial068.stdout: persistent_cache: (nil) 2026-02-23T13:06:04.080 INFO:journalctl@ceph.mon.a.trial068.stdout: block_size: 4096 2026-02-23T13:06:04.080 INFO:journalctl@ceph.mon.a.trial068.stdout: block_size_deviation: 10 2026-02-23T13:06:04.080 INFO:journalctl@ceph.mon.a.trial068.stdout: block_restart_interval: 16 2026-02-23T13:06:04.080 INFO:journalctl@ceph.mon.a.trial068.stdout: index_block_restart_interval: 1 2026-02-23T13:06:04.080 INFO:journalctl@ceph.mon.a.trial068.stdout: metadata_block_size: 4096 2026-02-23T13:06:04.080 INFO:journalctl@ceph.mon.a.trial068.stdout: partition_filters: 0 2026-02-23T13:06:04.080 INFO:journalctl@ceph.mon.a.trial068.stdout: use_delta_encoding: 1 2026-02-23T13:06:04.080 INFO:journalctl@ceph.mon.a.trial068.stdout: filter_policy: bloomfilter 2026-02-23T13:06:04.081 INFO:journalctl@ceph.mon.a.trial068.stdout: whole_key_filtering: 1 2026-02-23T13:06:04.081 INFO:journalctl@ceph.mon.a.trial068.stdout: verify_compression: 0 2026-02-23T13:06:04.081 INFO:journalctl@ceph.mon.a.trial068.stdout: read_amp_bytes_per_bit: 0 2026-02-23T13:06:04.081 INFO:journalctl@ceph.mon.a.trial068.stdout: format_version: 5 2026-02-23T13:06:04.081 INFO:journalctl@ceph.mon.a.trial068.stdout: enable_index_compression: 1 2026-02-23T13:06:04.081 INFO:journalctl@ceph.mon.a.trial068.stdout: block_align: 0 2026-02-23T13:06:04.081 INFO:journalctl@ceph.mon.a.trial068.stdout: max_auto_readahead_size: 262144 2026-02-23T13:06:04.081 INFO:journalctl@ceph.mon.a.trial068.stdout: prepopulate_block_cache: 0 2026-02-23T13:06:04.081 INFO:journalctl@ceph.mon.a.trial068.stdout: initial_auto_readahead_size: 8192 2026-02-23T13:06:04.081 INFO:journalctl@ceph.mon.a.trial068.stdout: num_file_reads_for_auto_readahead: 2 2026-02-23T13:06:04.081 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:03 trial068 ceph-mon[38805]: rocksdb: Options.write_buffer_size: 33554432 2026-02-23T13:06:04.081 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:03 trial068 ceph-mon[38805]: rocksdb: Options.max_write_buffer_number: 2 2026-02-23T13:06:04.081 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:03 trial068 ceph-mon[38805]: rocksdb: Options.compression: NoCompression 2026-02-23T13:06:04.081 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:03 trial068 ceph-mon[38805]: rocksdb: Options.bottommost_compression: Disabled 2026-02-23T13:06:04.082 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:03 trial068 ceph-mon[38805]: rocksdb: Options.prefix_extractor: nullptr 2026-02-23T13:06:04.082 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:03 trial068 ceph-mon[38805]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2026-02-23T13:06:04.082 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:03 trial068 ceph-mon[38805]: rocksdb: Options.num_levels: 7 2026-02-23T13:06:04.082 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:03 trial068 ceph-mon[38805]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2026-02-23T13:06:04.082 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:03 trial068 ceph-mon[38805]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2026-02-23T13:06:04.082 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:03 trial068 ceph-mon[38805]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2026-02-23T13:06:04.082 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:03 trial068 ceph-mon[38805]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2026-02-23T13:06:04.082 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:03 trial068 ceph-mon[38805]: rocksdb: Options.bottommost_compression_opts.level: 32767 2026-02-23T13:06:04.082 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:03 trial068 ceph-mon[38805]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2026-02-23T13:06:04.082 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:03 trial068 ceph-mon[38805]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2026-02-23T13:06:04.082 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:03 trial068 ceph-mon[38805]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2026-02-23T13:06:04.082 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:03 trial068 ceph-mon[38805]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2026-02-23T13:06:04.082 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:03 trial068 ceph-mon[38805]: rocksdb: Options.bottommost_compression_opts.enabled: false 2026-02-23T13:06:04.083 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:03 trial068 ceph-mon[38805]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2026-02-23T13:06:04.083 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:03 trial068 ceph-mon[38805]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2026-02-23T13:06:04.083 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:03 trial068 ceph-mon[38805]: rocksdb: Options.compression_opts.window_bits: -14 2026-02-23T13:06:04.084 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:03 trial068 ceph-mon[38805]: rocksdb: Options.compression_opts.level: 32767 2026-02-23T13:06:04.084 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:03 trial068 ceph-mon[38805]: rocksdb: Options.compression_opts.strategy: 0 2026-02-23T13:06:04.084 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:03 trial068 ceph-mon[38805]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2026-02-23T13:06:04.084 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:03 trial068 ceph-mon[38805]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2026-02-23T13:06:04.084 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:03 trial068 ceph-mon[38805]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2026-02-23T13:06:04.084 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:03 trial068 ceph-mon[38805]: rocksdb: Options.compression_opts.parallel_threads: 1 2026-02-23T13:06:04.085 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:03 trial068 ceph-mon[38805]: rocksdb: Options.compression_opts.enabled: false 2026-02-23T13:06:04.085 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:03 trial068 ceph-mon[38805]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2026-02-23T13:06:04.085 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:03 trial068 ceph-mon[38805]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2026-02-23T13:06:04.085 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:03 trial068 ceph-mon[38805]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2026-02-23T13:06:04.085 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:03 trial068 ceph-mon[38805]: rocksdb: Options.level0_stop_writes_trigger: 36 2026-02-23T13:06:04.085 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:03 trial068 ceph-mon[38805]: rocksdb: Options.target_file_size_base: 67108864 2026-02-23T13:06:04.085 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:03 trial068 ceph-mon[38805]: rocksdb: Options.target_file_size_multiplier: 1 2026-02-23T13:06:04.085 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:03 trial068 ceph-mon[38805]: rocksdb: Options.max_bytes_for_level_base: 268435456 2026-02-23T13:06:04.085 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:03 trial068 ceph-mon[38805]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2026-02-23T13:06:04.085 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:03 trial068 ceph-mon[38805]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2026-02-23T13:06:04.085 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:03 trial068 ceph-mon[38805]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2026-02-23T13:06:04.085 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:03 trial068 ceph-mon[38805]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2026-02-23T13:06:04.086 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:03 trial068 ceph-mon[38805]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2026-02-23T13:06:04.086 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:03 trial068 ceph-mon[38805]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2026-02-23T13:06:04.086 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:03 trial068 ceph-mon[38805]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2026-02-23T13:06:04.086 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:03 trial068 ceph-mon[38805]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2026-02-23T13:06:04.086 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:03 trial068 ceph-mon[38805]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2026-02-23T13:06:04.086 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:03 trial068 ceph-mon[38805]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2026-02-23T13:06:04.086 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:03 trial068 ceph-mon[38805]: rocksdb: Options.max_compaction_bytes: 1677721600 2026-02-23T13:06:04.086 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:03 trial068 ceph-mon[38805]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2026-02-23T13:06:04.086 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:03 trial068 ceph-mon[38805]: rocksdb: Options.arena_block_size: 1048576 2026-02-23T13:06:04.086 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:03 trial068 ceph-mon[38805]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2026-02-23T13:06:04.086 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:03 trial068 ceph-mon[38805]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2026-02-23T13:06:04.086 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:03 trial068 ceph-mon[38805]: rocksdb: Options.disable_auto_compactions: 0 2026-02-23T13:06:04.086 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:03 trial068 ceph-mon[38805]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2026-02-23T13:06:04.087 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:03 trial068 ceph-mon[38805]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2026-02-23T13:06:04.087 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:03 trial068 ceph-mon[38805]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2026-02-23T13:06:04.087 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:03 trial068 ceph-mon[38805]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2026-02-23T13:06:04.087 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:03 trial068 ceph-mon[38805]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2026-02-23T13:06:04.087 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:03 trial068 ceph-mon[38805]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2026-02-23T13:06:04.087 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:03 trial068 ceph-mon[38805]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2026-02-23T13:06:04.087 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:03 trial068 ceph-mon[38805]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2026-02-23T13:06:04.087 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:03 trial068 ceph-mon[38805]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2026-02-23T13:06:04.087 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:03 trial068 ceph-mon[38805]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2026-02-23T13:06:04.087 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:03 trial068 ceph-mon[38805]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2026-02-23T13:06:04.087 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:03 trial068 ceph-mon[38805]: rocksdb: Options.inplace_update_support: 0 2026-02-23T13:06:04.087 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:03 trial068 ceph-mon[38805]: rocksdb: Options.inplace_update_num_locks: 10000 2026-02-23T13:06:04.088 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:03 trial068 ceph-mon[38805]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2026-02-23T13:06:04.088 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:03 trial068 ceph-mon[38805]: rocksdb: Options.memtable_whole_key_filtering: 0 2026-02-23T13:06:04.088 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:03 trial068 ceph-mon[38805]: rocksdb: Options.memtable_huge_page_size: 0 2026-02-23T13:06:04.088 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:03 trial068 ceph-mon[38805]: rocksdb: Options.bloom_locality: 0 2026-02-23T13:06:04.088 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:03 trial068 ceph-mon[38805]: rocksdb: Options.max_successive_merges: 0 2026-02-23T13:06:04.088 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:03 trial068 ceph-mon[38805]: rocksdb: Options.optimize_filters_for_hits: 0 2026-02-23T13:06:04.088 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:03 trial068 ceph-mon[38805]: rocksdb: Options.paranoid_file_checks: 0 2026-02-23T13:06:04.088 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:03 trial068 ceph-mon[38805]: rocksdb: Options.force_consistency_checks: 1 2026-02-23T13:06:04.088 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:03 trial068 ceph-mon[38805]: rocksdb: Options.report_bg_io_stats: 0 2026-02-23T13:06:04.088 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:03 trial068 ceph-mon[38805]: rocksdb: Options.ttl: 2592000 2026-02-23T13:06:04.088 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:03 trial068 ceph-mon[38805]: rocksdb: Options.periodic_compaction_seconds: 0 2026-02-23T13:06:04.088 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:03 trial068 ceph-mon[38805]: rocksdb: Options.preclude_last_level_data_seconds: 0 2026-02-23T13:06:04.088 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:03 trial068 ceph-mon[38805]: rocksdb: Options.preserve_internal_time_seconds: 0 2026-02-23T13:06:04.089 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:03 trial068 ceph-mon[38805]: rocksdb: Options.enable_blob_files: false 2026-02-23T13:06:04.089 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:03 trial068 ceph-mon[38805]: rocksdb: Options.min_blob_size: 0 2026-02-23T13:06:04.089 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:03 trial068 ceph-mon[38805]: rocksdb: Options.blob_file_size: 268435456 2026-02-23T13:06:04.089 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:03 trial068 ceph-mon[38805]: rocksdb: Options.blob_compression_type: NoCompression 2026-02-23T13:06:04.089 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:03 trial068 ceph-mon[38805]: rocksdb: Options.enable_blob_garbage_collection: false 2026-02-23T13:06:04.089 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:03 trial068 ceph-mon[38805]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2026-02-23T13:06:04.089 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:03 trial068 ceph-mon[38805]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2026-02-23T13:06:04.089 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:03 trial068 ceph-mon[38805]: rocksdb: Options.blob_compaction_readahead_size: 0 2026-02-23T13:06:04.089 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:03 trial068 ceph-mon[38805]: rocksdb: Options.blob_file_starting_level: 0 2026-02-23T13:06:04.089 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:03 trial068 ceph-mon[38805]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2026-02-23T13:06:04.089 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:03 trial068 ceph-mon[38805]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2026-02-23T13:06:04.089 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:03 trial068 ceph-mon[38805]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2026-02-23T13:06:04.089 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:03 trial068 ceph-mon[38805]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: c286f4bd-a62c-459e-b999-4e34fe40bff5 2026-02-23T13:06:04.090 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:03 trial068 ceph-mon[38805]: rocksdb: EVENT_LOG_v1 {"time_micros": 1771851963936910, "job": 1, "event": "recovery_started", "wal_files": [9]} 2026-02-23T13:06:04.090 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:03 trial068 ceph-mon[38805]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2026-02-23T13:06:04.090 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:03 trial068 ceph-mon[38805]: rocksdb: EVENT_LOG_v1 {"time_micros": 1771851963937657, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 94230, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 323, "table_properties": {"data_size": 92173, "index_size": 250, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 773, "raw_key_size": 15582, "raw_average_key_size": 53, "raw_value_size": 84103, "raw_average_value_size": 286, "num_data_blocks": 9, "num_entries": 294, "num_filter_entries": 294, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1771851963, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "c286f4bd-a62c-459e-b999-4e34fe40bff5", "db_session_id": "GQRBEIYPCNVFBA5TD0GX", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2026-02-23T13:06:04.090 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:03 trial068 ceph-mon[38805]: rocksdb: EVENT_LOG_v1 {"time_micros": 1771851963937709, "job": 1, "event": "recovery_finished"} 2026-02-23T13:06:04.090 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:03 trial068 ceph-mon[38805]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2026-02-23T13:06:04.090 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:03 trial068 ceph-mon[38805]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2026-02-23T13:06:04.090 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:03 trial068 ceph-mon[38805]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x3648e00 2026-02-23T13:06:04.090 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:03 trial068 ceph-mon[38805]: rocksdb: DB pointer 0x443a000 2026-02-23T13:06:04.090 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:03 trial068 ceph-mon[38805]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2026-02-23T13:06:04.090 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:03 trial068 ceph-mon[38805]: rocksdb: [db/db_impl/db_impl.cc:1111] 2026-02-23T13:06:04.090 INFO:journalctl@ceph.mon.a.trial068.stdout: ** DB Stats ** 2026-02-23T13:06:04.091 INFO:journalctl@ceph.mon.a.trial068.stdout: Uptime(secs): 0.0 total, 0.0 interval 2026-02-23T13:06:04.091 INFO:journalctl@ceph.mon.a.trial068.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2026-02-23T13:06:04.091 INFO:journalctl@ceph.mon.a.trial068.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2026-02-23T13:06:04.091 INFO:journalctl@ceph.mon.a.trial068.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2026-02-23T13:06:04.091 INFO:journalctl@ceph.mon.a.trial068.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2026-02-23T13:06:04.091 INFO:journalctl@ceph.mon.a.trial068.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2026-02-23T13:06:04.091 INFO:journalctl@ceph.mon.a.trial068.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2026-02-23T13:06:04.091 INFO:journalctl@ceph.mon.a.trial068.stdout: 2026-02-23T13:06:04.091 INFO:journalctl@ceph.mon.a.trial068.stdout: ** Compaction Stats [default] ** 2026-02-23T13:06:04.091 INFO:journalctl@ceph.mon.a.trial068.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2026-02-23T13:06:04.091 INFO:journalctl@ceph.mon.a.trial068.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2026-02-23T13:06:04.091 INFO:journalctl@ceph.mon.a.trial068.stdout: L0 2/0 93.89 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 186.4 0.00 0.00 1 0.000 0 0 0.0 0.0 2026-02-23T13:06:04.092 INFO:journalctl@ceph.mon.a.trial068.stdout: Sum 2/0 93.89 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 186.4 0.00 0.00 1 0.000 0 0 0.0 0.0 2026-02-23T13:06:04.092 INFO:journalctl@ceph.mon.a.trial068.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 186.4 0.00 0.00 1 0.000 0 0 0.0 0.0 2026-02-23T13:06:04.092 INFO:journalctl@ceph.mon.a.trial068.stdout: 2026-02-23T13:06:04.092 INFO:journalctl@ceph.mon.a.trial068.stdout: ** Compaction Stats [default] ** 2026-02-23T13:06:04.092 INFO:journalctl@ceph.mon.a.trial068.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2026-02-23T13:06:04.092 INFO:journalctl@ceph.mon.a.trial068.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2026-02-23T13:06:04.092 INFO:journalctl@ceph.mon.a.trial068.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 186.4 0.00 0.00 1 0.000 0 0 0.0 0.0 2026-02-23T13:06:04.092 INFO:journalctl@ceph.mon.a.trial068.stdout: 2026-02-23T13:06:04.092 INFO:journalctl@ceph.mon.a.trial068.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2026-02-23T13:06:04.092 INFO:journalctl@ceph.mon.a.trial068.stdout: 2026-02-23T13:06:04.092 INFO:journalctl@ceph.mon.a.trial068.stdout: Uptime(secs): 0.0 total, 0.0 interval 2026-02-23T13:06:04.092 INFO:journalctl@ceph.mon.a.trial068.stdout: Flush(GB): cumulative 0.000, interval 0.000 2026-02-23T13:06:04.093 INFO:journalctl@ceph.mon.a.trial068.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2026-02-23T13:06:04.093 INFO:journalctl@ceph.mon.a.trial068.stdout: AddFile(Total Files): cumulative 0, interval 0 2026-02-23T13:06:04.093 INFO:journalctl@ceph.mon.a.trial068.stdout: AddFile(L0 Files): cumulative 0, interval 0 2026-02-23T13:06:04.093 INFO:journalctl@ceph.mon.a.trial068.stdout: AddFile(Keys): cumulative 0, interval 0 2026-02-23T13:06:04.093 INFO:journalctl@ceph.mon.a.trial068.stdout: Cumulative compaction: 0.00 GB write, 37.43 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2026-02-23T13:06:04.093 INFO:journalctl@ceph.mon.a.trial068.stdout: Interval compaction: 0.00 GB write, 37.43 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2026-02-23T13:06:04.093 INFO:journalctl@ceph.mon.a.trial068.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2026-02-23T13:06:04.093 INFO:journalctl@ceph.mon.a.trial068.stdout: Block cache BinnedLRUCache@0x35fba50#2 capacity: 512.00 MB usage: 1.45 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1.5e-05 secs_since: 0 2026-02-23T13:06:04.093 INFO:journalctl@ceph.mon.a.trial068.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,1.02 KB,0.000193715%) IndexBlock(2,0.44 KB,8.34465e-05%) Misc(1,0.00 KB,0%) 2026-02-23T13:06:04.093 INFO:journalctl@ceph.mon.a.trial068.stdout: 2026-02-23T13:06:04.093 INFO:journalctl@ceph.mon.a.trial068.stdout: ** File Read Latency Histogram By Level [default] ** 2026-02-23T13:06:04.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:03 trial068 ceph-mon[38805]: starting mon.a rank 0 at public addrs [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] at bind addrs [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid 5f823084-10b8-11f1-be80-d404e6e7d460 2026-02-23T13:06:04.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:03 trial068 ceph-mon[38805]: mon.a@-1(???) e1 preinit fsid 5f823084-10b8-11f1-be80-d404e6e7d460 2026-02-23T13:06:04.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:03 trial068 ceph-mon[38805]: mon.a@-1(???).mds e1 new map 2026-02-23T13:06:04.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:03 trial068 ceph-mon[38805]: mon.a@-1(???).mds e1 print_map 2026-02-23T13:06:04.094 INFO:journalctl@ceph.mon.a.trial068.stdout: e1 2026-02-23T13:06:04.094 INFO:journalctl@ceph.mon.a.trial068.stdout: btime 2026-02-23T13:06:03:098534+0000 2026-02-23T13:06:04.094 INFO:journalctl@ceph.mon.a.trial068.stdout: enable_multiple, ever_enabled_multiple: 1,1 2026-02-23T13:06:04.094 INFO:journalctl@ceph.mon.a.trial068.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2026-02-23T13:06:04.094 INFO:journalctl@ceph.mon.a.trial068.stdout: legacy client fscid: -1 2026-02-23T13:06:04.094 INFO:journalctl@ceph.mon.a.trial068.stdout: 2026-02-23T13:06:04.094 INFO:journalctl@ceph.mon.a.trial068.stdout: No filesystems configured 2026-02-23T13:06:04.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:03 trial068 ceph-mon[38805]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2026-02-23T13:06:04.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:03 trial068 ceph-mon[38805]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2026-02-23T13:06:04.095 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:03 trial068 ceph-mon[38805]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2026-02-23T13:06:04.095 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:03 trial068 ceph-mon[38805]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2026-02-23T13:06:04.095 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:03 trial068 ceph-mon[38805]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2026-02-23T13:06:04.095 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:03 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mon-a[38796]: 2026-02-23T13:06:03.939+0000 7f0841624c00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:06:04.095 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:03 trial068 ceph-mon[38805]: mon.a is new leader, mons a in quorum (ranks 0) 2026-02-23T13:06:04.095 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:03 trial068 ceph-mon[38805]: monmap epoch 1 2026-02-23T13:06:04.095 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:03 trial068 ceph-mon[38805]: fsid 5f823084-10b8-11f1-be80-d404e6e7d460 2026-02-23T13:06:04.095 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:03 trial068 ceph-mon[38805]: last_changed 2026-02-23T13:06:02.168892+0000 2026-02-23T13:06:04.095 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:03 trial068 ceph-mon[38805]: created 2026-02-23T13:06:02.168892+0000 2026-02-23T13:06:04.095 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:03 trial068 ceph-mon[38805]: min_mon_release 20 (tentacle) 2026-02-23T13:06:04.095 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:03 trial068 ceph-mon[38805]: election_strategy: 1 2026-02-23T13:06:04.095 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:03 trial068 ceph-mon[38805]: 0: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.a 2026-02-23T13:06:04.095 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:03 trial068 ceph-mon[38805]: fsmap 2026-02-23T13:06:04.096 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:03 trial068 ceph-mon[38805]: osdmap e1: 0 total, 0 up, 0 in 2026-02-23T13:06:04.096 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:03 trial068 ceph-mon[38805]: mgrmap e1: no daemons active 2026-02-23T13:06:04.097 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stderr 2026-02-23T13:06:04.022+0000 7f3e5e43b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:06:04.097 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stderr 2026-02-23T13:06:04.028+0000 7f3e5e43b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:06:04.097 INFO:teuthology.orchestra.run.trial068.stdout:Wrote config to /etc/ceph/ceph.conf 2026-02-23T13:06:04.097 INFO:teuthology.orchestra.run.trial068.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2026-02-23T13:06:04.097 INFO:teuthology.orchestra.run.trial068.stdout:Creating mgr... 2026-02-23T13:06:04.097 INFO:teuthology.orchestra.run.trial068.stdout:Verifying port 0.0.0.0:9283 ... 2026-02-23T13:06:04.097 INFO:teuthology.orchestra.run.trial068.stdout:Verifying port 0.0.0.0:8765 ... 2026-02-23T13:06:04.208 INFO:teuthology.orchestra.run.trial068.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-5f823084-10b8-11f1-be80-d404e6e7d460@mgr.x 2026-02-23T13:06:04.209 INFO:teuthology.orchestra.run.trial068.stdout:systemctl: stderr Failed to reset failed state of unit ceph-5f823084-10b8-11f1-be80-d404e6e7d460@mgr.x.service: Unit ceph-5f823084-10b8-11f1-be80-d404e6e7d460@mgr.x.service not loaded. 2026-02-23T13:06:04.313 INFO:teuthology.orchestra.run.trial068.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-5f823084-10b8-11f1-be80-d404e6e7d460.target.wants/ceph-5f823084-10b8-11f1-be80-d404e6e7d460@mgr.x.service → /etc/systemd/system/ceph-5f823084-10b8-11f1-be80-d404e6e7d460@.service. 2026-02-23T13:06:04.456 INFO:teuthology.orchestra.run.trial068.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2026-02-23T13:06:04.456 INFO:teuthology.orchestra.run.trial068.stdout:systemctl: stdout disabled 2026-02-23T13:06:04.461 INFO:teuthology.orchestra.run.trial068.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2026-02-23T13:06:04.461 INFO:teuthology.orchestra.run.trial068.stdout:systemctl: stdout inactive 2026-02-23T13:06:04.462 INFO:teuthology.orchestra.run.trial068.stdout:firewalld.service is not enabled 2026-02-23T13:06:04.462 INFO:teuthology.orchestra.run.trial068.stdout:Not possible to enable service . firewalld.service is not available 2026-02-23T13:06:04.469 INFO:teuthology.orchestra.run.trial068.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2026-02-23T13:06:04.469 INFO:teuthology.orchestra.run.trial068.stdout:systemctl: stdout disabled 2026-02-23T13:06:04.475 INFO:teuthology.orchestra.run.trial068.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2026-02-23T13:06:04.475 INFO:teuthology.orchestra.run.trial068.stdout:systemctl: stdout inactive 2026-02-23T13:06:04.475 INFO:teuthology.orchestra.run.trial068.stdout:firewalld.service is not enabled 2026-02-23T13:06:04.476 INFO:teuthology.orchestra.run.trial068.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2026-02-23T13:06:04.476 INFO:teuthology.orchestra.run.trial068.stdout:Waiting for mgr to start... 2026-02-23T13:06:04.476 INFO:teuthology.orchestra.run.trial068.stdout:Waiting for mgr... 2026-02-23T13:06:04.634 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stdout 2026-02-23T13:06:04.634 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stdout { 2026-02-23T13:06:04.634 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stdout "fsid": "5f823084-10b8-11f1-be80-d404e6e7d460", 2026-02-23T13:06:04.634 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stdout "health": { 2026-02-23T13:06:04.634 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2026-02-23T13:06:04.634 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stdout "checks": {}, 2026-02-23T13:06:04.634 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stdout "mutes": [] 2026-02-23T13:06:04.634 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stdout }, 2026-02-23T13:06:04.635 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2026-02-23T13:06:04.635 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stdout "quorum": [ 2026-02-23T13:06:04.635 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stdout 0 2026-02-23T13:06:04.635 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stdout ], 2026-02-23T13:06:04.635 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2026-02-23T13:06:04.635 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stdout "a" 2026-02-23T13:06:04.635 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stdout ], 2026-02-23T13:06:04.635 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stdout "quorum_age": 0, 2026-02-23T13:06:04.635 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stdout "monmap": { 2026-02-23T13:06:04.635 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stdout "epoch": 1, 2026-02-23T13:06:04.635 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "tentacle", 2026-02-23T13:06:04.635 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stdout "num_mons": 1 2026-02-23T13:06:04.635 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stdout }, 2026-02-23T13:06:04.636 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stdout "osdmap": { 2026-02-23T13:06:04.636 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stdout "epoch": 1, 2026-02-23T13:06:04.636 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2026-02-23T13:06:04.636 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2026-02-23T13:06:04.636 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2026-02-23T13:06:04.636 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2026-02-23T13:06:04.636 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2026-02-23T13:06:04.636 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2026-02-23T13:06:04.636 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stdout }, 2026-02-23T13:06:04.636 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stdout "pgmap": { 2026-02-23T13:06:04.636 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2026-02-23T13:06:04.636 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2026-02-23T13:06:04.637 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2026-02-23T13:06:04.637 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2026-02-23T13:06:04.637 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2026-02-23T13:06:04.637 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2026-02-23T13:06:04.637 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2026-02-23T13:06:04.637 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2026-02-23T13:06:04.637 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stdout }, 2026-02-23T13:06:04.637 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stdout "fsmap": { 2026-02-23T13:06:04.637 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stdout "epoch": 1, 2026-02-23T13:06:04.637 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stdout "btime": "2026-02-23T13:06:03:098534+0000", 2026-02-23T13:06:04.637 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stdout "by_rank": [], 2026-02-23T13:06:04.637 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stdout "up:standby": 0 2026-02-23T13:06:04.637 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stdout }, 2026-02-23T13:06:04.638 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stdout "mgrmap": { 2026-02-23T13:06:04.638 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stdout "available": false, 2026-02-23T13:06:04.638 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2026-02-23T13:06:04.638 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stdout "modules": [ 2026-02-23T13:06:04.638 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stdout "iostat", 2026-02-23T13:06:04.638 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stdout "nfs" 2026-02-23T13:06:04.638 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stdout ], 2026-02-23T13:06:04.638 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stdout "services": {} 2026-02-23T13:06:04.638 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stdout }, 2026-02-23T13:06:04.638 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stdout "servicemap": { 2026-02-23T13:06:04.638 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stdout "epoch": 1, 2026-02-23T13:06:04.638 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stdout "modified": "2026-02-23T13:06:03.098894+0000", 2026-02-23T13:06:04.638 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stdout "services": {} 2026-02-23T13:06:04.639 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stdout }, 2026-02-23T13:06:04.639 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stdout "progress_events": {} 2026-02-23T13:06:04.639 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stdout } 2026-02-23T13:06:04.639 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stderr 2026-02-23T13:06:04.575+0000 7fdcdcfe2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:06:04.639 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stderr 2026-02-23T13:06:04.578+0000 7fdcdcfe2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:06:04.639 INFO:teuthology.orchestra.run.trial068.stdout:mgr not available, waiting (1/15)... 2026-02-23T13:06:05.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:05 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2544866121' entity='client.admin' 2026-02-23T13:06:05.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:05 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/327751432' entity='client.admin' cmd={"prefix": "status", "format": "json-pretty"} : dispatch 2026-02-23T13:06:05.843 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:06:05 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: from numpy import show_config as show_numpy_config 2026-02-23T13:06:06.793 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stdout 2026-02-23T13:06:06.794 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stdout { 2026-02-23T13:06:06.794 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stdout "fsid": "5f823084-10b8-11f1-be80-d404e6e7d460", 2026-02-23T13:06:06.794 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stdout "health": { 2026-02-23T13:06:06.794 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2026-02-23T13:06:06.794 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stdout "checks": {}, 2026-02-23T13:06:06.795 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stdout "mutes": [] 2026-02-23T13:06:06.795 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stdout }, 2026-02-23T13:06:06.795 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2026-02-23T13:06:06.795 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stdout "quorum": [ 2026-02-23T13:06:06.795 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stdout 0 2026-02-23T13:06:06.795 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stdout ], 2026-02-23T13:06:06.795 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2026-02-23T13:06:06.795 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stdout "a" 2026-02-23T13:06:06.796 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stdout ], 2026-02-23T13:06:06.796 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stdout "quorum_age": 2, 2026-02-23T13:06:06.796 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stdout "monmap": { 2026-02-23T13:06:06.796 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stdout "epoch": 1, 2026-02-23T13:06:06.796 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "tentacle", 2026-02-23T13:06:06.796 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stdout "num_mons": 1 2026-02-23T13:06:06.796 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stdout }, 2026-02-23T13:06:06.796 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stdout "osdmap": { 2026-02-23T13:06:06.796 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stdout "epoch": 1, 2026-02-23T13:06:06.797 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2026-02-23T13:06:06.797 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2026-02-23T13:06:06.797 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2026-02-23T13:06:06.797 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2026-02-23T13:06:06.797 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2026-02-23T13:06:06.797 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2026-02-23T13:06:06.797 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stdout }, 2026-02-23T13:06:06.797 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stdout "pgmap": { 2026-02-23T13:06:06.798 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2026-02-23T13:06:06.798 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2026-02-23T13:06:06.798 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2026-02-23T13:06:06.798 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2026-02-23T13:06:06.798 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2026-02-23T13:06:06.798 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2026-02-23T13:06:06.798 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2026-02-23T13:06:06.798 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2026-02-23T13:06:06.799 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stdout }, 2026-02-23T13:06:06.799 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stdout "fsmap": { 2026-02-23T13:06:06.799 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stdout "epoch": 1, 2026-02-23T13:06:06.799 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stdout "btime": "2026-02-23T13:06:03:098534+0000", 2026-02-23T13:06:06.799 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stdout "by_rank": [], 2026-02-23T13:06:06.799 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stdout "up:standby": 0 2026-02-23T13:06:06.799 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stdout }, 2026-02-23T13:06:06.799 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stdout "mgrmap": { 2026-02-23T13:06:06.800 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stdout "available": false, 2026-02-23T13:06:06.800 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2026-02-23T13:06:06.800 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stdout "modules": [ 2026-02-23T13:06:06.800 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stdout "iostat", 2026-02-23T13:06:06.800 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stdout "nfs" 2026-02-23T13:06:06.800 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stdout ], 2026-02-23T13:06:06.800 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stdout "services": {} 2026-02-23T13:06:06.800 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stdout }, 2026-02-23T13:06:06.800 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stdout "servicemap": { 2026-02-23T13:06:06.801 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stdout "epoch": 1, 2026-02-23T13:06:06.801 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stdout "modified": "2026-02-23T13:06:03.098894+0000", 2026-02-23T13:06:06.801 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stdout "services": {} 2026-02-23T13:06:06.801 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stdout }, 2026-02-23T13:06:06.801 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stdout "progress_events": {} 2026-02-23T13:06:06.801 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stdout } 2026-02-23T13:06:06.801 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stderr 2026-02-23T13:06:06.731+0000 7fc4e23fa640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:06:06.801 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stderr 2026-02-23T13:06:06.734+0000 7fc4e23fa640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:06:06.802 INFO:teuthology.orchestra.run.trial068.stdout:mgr not available, waiting (2/15)... 2026-02-23T13:06:07.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:06 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/185345738' entity='client.admin' cmd={"prefix": "status", "format": "json-pretty"} : dispatch 2026-02-23T13:06:08.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:07 trial068 ceph-mon[38805]: Activating manager daemon x 2026-02-23T13:06:08.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:07 trial068 ceph-mon[38805]: mgrmap e2: x(active, starting, since 0.00296081s) 2026-02-23T13:06:08.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:07 trial068 ceph-mon[38805]: from='mgr.14100 10.20.193.68:0/3122043453' entity='mgr.x' cmd={"prefix": "mds metadata"} : dispatch 2026-02-23T13:06:08.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:07 trial068 ceph-mon[38805]: from='mgr.14100 10.20.193.68:0/3122043453' entity='mgr.x' cmd={"prefix": "osd metadata"} : dispatch 2026-02-23T13:06:08.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:07 trial068 ceph-mon[38805]: from='mgr.14100 10.20.193.68:0/3122043453' entity='mgr.x' cmd={"prefix": "mon metadata"} : dispatch 2026-02-23T13:06:08.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:07 trial068 ceph-mon[38805]: from='mgr.14100 10.20.193.68:0/3122043453' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2026-02-23T13:06:08.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:07 trial068 ceph-mon[38805]: from='mgr.14100 10.20.193.68:0/3122043453' entity='mgr.x' cmd={"prefix": "mgr metadata", "who": "x", "id": "x"} : dispatch 2026-02-23T13:06:08.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:07 trial068 ceph-mon[38805]: Manager daemon x is now available 2026-02-23T13:06:08.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:07 trial068 ceph-mon[38805]: from='mgr.14100 10.20.193.68:0/3122043453' entity='mgr.x' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"} : dispatch 2026-02-23T13:06:08.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:07 trial068 ceph-mon[38805]: from='mgr.14100 10.20.193.68:0/3122043453' entity='mgr.x' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"} : dispatch 2026-02-23T13:06:08.095 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:07 trial068 ceph-mon[38805]: from='mgr.14100 10.20.193.68:0/3122043453' entity='mgr.x' 2026-02-23T13:06:08.095 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:07 trial068 ceph-mon[38805]: from='mgr.14100 10.20.193.68:0/3122043453' entity='mgr.x' 2026-02-23T13:06:08.095 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:07 trial068 ceph-mon[38805]: from='mgr.14100 10.20.193.68:0/3122043453' entity='mgr.x' 2026-02-23T13:06:09.070 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stdout 2026-02-23T13:06:09.071 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stdout { 2026-02-23T13:06:09.071 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stdout "fsid": "5f823084-10b8-11f1-be80-d404e6e7d460", 2026-02-23T13:06:09.071 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stdout "health": { 2026-02-23T13:06:09.071 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2026-02-23T13:06:09.071 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stdout "checks": {}, 2026-02-23T13:06:09.072 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stdout "mutes": [] 2026-02-23T13:06:09.072 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stdout }, 2026-02-23T13:06:09.072 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2026-02-23T13:06:09.072 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stdout "quorum": [ 2026-02-23T13:06:09.072 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stdout 0 2026-02-23T13:06:09.072 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stdout ], 2026-02-23T13:06:09.072 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2026-02-23T13:06:09.072 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stdout "a" 2026-02-23T13:06:09.073 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stdout ], 2026-02-23T13:06:09.073 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stdout "quorum_age": 5, 2026-02-23T13:06:09.073 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stdout "monmap": { 2026-02-23T13:06:09.073 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stdout "epoch": 1, 2026-02-23T13:06:09.073 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "tentacle", 2026-02-23T13:06:09.073 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stdout "num_mons": 1 2026-02-23T13:06:09.073 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stdout }, 2026-02-23T13:06:09.073 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stdout "osdmap": { 2026-02-23T13:06:09.074 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stdout "epoch": 1, 2026-02-23T13:06:09.074 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2026-02-23T13:06:09.074 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2026-02-23T13:06:09.074 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2026-02-23T13:06:09.074 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2026-02-23T13:06:09.074 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2026-02-23T13:06:09.074 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2026-02-23T13:06:09.074 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stdout }, 2026-02-23T13:06:09.074 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stdout "pgmap": { 2026-02-23T13:06:09.075 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2026-02-23T13:06:09.075 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2026-02-23T13:06:09.075 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2026-02-23T13:06:09.075 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2026-02-23T13:06:09.075 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2026-02-23T13:06:09.075 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2026-02-23T13:06:09.075 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2026-02-23T13:06:09.075 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2026-02-23T13:06:09.076 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stdout }, 2026-02-23T13:06:09.076 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stdout "fsmap": { 2026-02-23T13:06:09.076 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stdout "epoch": 1, 2026-02-23T13:06:09.076 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stdout "btime": "2026-02-23T13:06:03:098534+0000", 2026-02-23T13:06:09.076 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stdout "by_rank": [], 2026-02-23T13:06:09.076 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stdout "up:standby": 0 2026-02-23T13:06:09.076 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stdout }, 2026-02-23T13:06:09.076 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stdout "mgrmap": { 2026-02-23T13:06:09.077 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stdout "available": true, 2026-02-23T13:06:09.077 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2026-02-23T13:06:09.077 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stdout "modules": [ 2026-02-23T13:06:09.077 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stdout "iostat", 2026-02-23T13:06:09.077 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stdout "nfs" 2026-02-23T13:06:09.077 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stdout ], 2026-02-23T13:06:09.077 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stdout "services": {} 2026-02-23T13:06:09.077 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stdout }, 2026-02-23T13:06:09.077 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stdout "servicemap": { 2026-02-23T13:06:09.078 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stdout "epoch": 1, 2026-02-23T13:06:09.078 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stdout "modified": "2026-02-23T13:06:03.098894+0000", 2026-02-23T13:06:09.078 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stdout "services": {} 2026-02-23T13:06:09.078 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stdout }, 2026-02-23T13:06:09.078 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stdout "progress_events": {} 2026-02-23T13:06:09.078 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stdout } 2026-02-23T13:06:09.078 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stderr 2026-02-23T13:06:08.886+0000 7f1aedd7f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:06:09.078 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stderr 2026-02-23T13:06:08.889+0000 7f1aedd7f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:06:09.078 INFO:teuthology.orchestra.run.trial068.stdout:mgr is available 2026-02-23T13:06:09.287 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stdout 2026-02-23T13:06:09.288 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stdout [global] 2026-02-23T13:06:09.288 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stdout fsid = 5f823084-10b8-11f1-be80-d404e6e7d460 2026-02-23T13:06:09.288 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stdout mon_cluster_log_file_level = debug 2026-02-23T13:06:09.288 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stdout mon_host = [v2:10.20.193.68:3300,v1:10.20.193.68:6789] 2026-02-23T13:06:09.288 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2026-02-23T13:06:09.288 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2026-02-23T13:06:09.288 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2026-02-23T13:06:09.288 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2026-02-23T13:06:09.288 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stdout 2026-02-23T13:06:09.289 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stdout [mgr] 2026-02-23T13:06:09.289 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2026-02-23T13:06:09.289 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stdout 2026-02-23T13:06:09.289 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stdout [osd] 2026-02-23T13:06:09.289 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2026-02-23T13:06:09.289 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2026-02-23T13:06:09.289 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stderr 2026-02-23T13:06:09.151+0000 7f979aa55640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:06:09.289 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stderr 2026-02-23T13:06:09.154+0000 7f979aa55640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:06:09.289 INFO:teuthology.orchestra.run.trial068.stdout:Enabling cephadm module... 2026-02-23T13:06:09.336 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:09 trial068 ceph-mon[38805]: mgrmap e3: x(active, since 1.00506s) 2026-02-23T13:06:09.336 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:09 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2273826008' entity='client.admin' cmd={"prefix": "status", "format": "json-pretty"} : dispatch 2026-02-23T13:06:10.224 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stderr 2026-02-23T13:06:09.375+0000 7fd8f0daf640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:06:10.224 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stderr 2026-02-23T13:06:09.379+0000 7fd8f0daf640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:06:10.460 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:10 trial068 ceph-mon[38805]: mgrmap e4: x(active, since 2s) 2026-02-23T13:06:10.461 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:10 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/230919796' entity='client.admin' cmd={"prefix": "config assimilate-conf"} : dispatch 2026-02-23T13:06:10.461 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:10 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1094995824' entity='client.admin' cmd={"prefix": "mgr module enable", "module": "cephadm"} : dispatch 2026-02-23T13:06:10.461 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:06:10 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ignoring --setuser ceph since I am not root 2026-02-23T13:06:10.461 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:06:10 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ignoring --setgroup ceph since I am not root 2026-02-23T13:06:10.461 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:06:10 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: 2026-02-23T13:06:10.236+0000 7fd0803ebfc0 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:06:10.461 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:06:10 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: 2026-02-23T13:06:10.238+0000 7fd0803ebfc0 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:06:10.461 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:06:10 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: 2026-02-23T13:06:10.238+0000 7fd0803ebfc0 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:06:10.486 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stdout { 2026-02-23T13:06:10.486 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stdout "epoch": 5, 2026-02-23T13:06:10.486 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stdout "available": true, 2026-02-23T13:06:10.486 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stdout "active_name": "x", 2026-02-23T13:06:10.486 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stdout "num_standby": 0 2026-02-23T13:06:10.486 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stdout } 2026-02-23T13:06:10.486 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stderr 2026-02-23T13:06:10.315+0000 7f2bfb58a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:06:10.486 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stderr 2026-02-23T13:06:10.319+0000 7f2bfb58a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:06:10.486 INFO:teuthology.orchestra.run.trial068.stdout:Waiting for the mgr to restart... 2026-02-23T13:06:10.486 INFO:teuthology.orchestra.run.trial068.stdout:Waiting for mgr epoch 5... 2026-02-23T13:06:11.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:11 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1094995824' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2026-02-23T13:06:11.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:11 trial068 ceph-mon[38805]: mgrmap e5: x(active, since 3s) 2026-02-23T13:06:11.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:11 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2352920024' entity='client.admin' cmd={"prefix": "mgr stat"} : dispatch 2026-02-23T13:06:11.593 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:06:11 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: /lib64/python3.9/site-packages/scipy/__init__.py:73: UserWarning: NumPy was imported from a Python sub-interpreter but NumPy does not properly support sub-interpreters. This will likely work for most users but might cause hard to track down issues or subtle bugs. A common user of the rare sub-interpreter feature is wsgi which also allows single-interpreter mode. 2026-02-23T13:06:11.593 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:06:11 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: Improvements in the case of bugs are welcome, but is not on the NumPy roadmap, and full support may require significant effort to achieve. 2026-02-23T13:06:11.593 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:06:11 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: from numpy import show_config as show_numpy_config 2026-02-23T13:06:13.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:13 trial068 ceph-mon[38805]: Active manager daemon x restarted 2026-02-23T13:06:13.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:13 trial068 ceph-mon[38805]: Activating manager daemon x 2026-02-23T13:06:13.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:13 trial068 ceph-mon[38805]: osdmap e2: 0 total, 0 up, 0 in 2026-02-23T13:06:13.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:13 trial068 ceph-mon[38805]: mgrmap e6: x(active, starting, since 0.00263798s) 2026-02-23T13:06:13.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:13 trial068 ceph-mon[38805]: from='mgr.14118 10.20.193.68:0/3047353658' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2026-02-23T13:06:13.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:13 trial068 ceph-mon[38805]: from='mgr.14118 10.20.193.68:0/3047353658' entity='mgr.x' cmd={"prefix": "mgr metadata", "who": "x", "id": "x"} : dispatch 2026-02-23T13:06:13.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:13 trial068 ceph-mon[38805]: from='mgr.14118 10.20.193.68:0/3047353658' entity='mgr.x' cmd={"prefix": "mds metadata"} : dispatch 2026-02-23T13:06:13.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:13 trial068 ceph-mon[38805]: from='mgr.14118 10.20.193.68:0/3047353658' entity='mgr.x' cmd={"prefix": "osd metadata"} : dispatch 2026-02-23T13:06:13.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:13 trial068 ceph-mon[38805]: from='mgr.14118 10.20.193.68:0/3047353658' entity='mgr.x' cmd={"prefix": "mon metadata"} : dispatch 2026-02-23T13:06:13.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:13 trial068 ceph-mon[38805]: Manager daemon x is now available 2026-02-23T13:06:13.987 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stdout { 2026-02-23T13:06:13.988 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2026-02-23T13:06:13.988 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stdout "initialized": true 2026-02-23T13:06:13.988 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stdout } 2026-02-23T13:06:13.988 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stderr 2026-02-23T13:06:10.588+0000 7ffb6f774640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:06:13.988 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stderr 2026-02-23T13:06:10.591+0000 7ffb6f774640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:06:13.988 INFO:teuthology.orchestra.run.trial068.stdout:mgr epoch 5 is available 2026-02-23T13:06:13.988 INFO:teuthology.orchestra.run.trial068.stdout:Verifying orchestrator module is enabled... 2026-02-23T13:06:14.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:14 trial068 ceph-mon[38805]: from='mgr.14118 10.20.193.68:0/3047353658' entity='mgr.x' 2026-02-23T13:06:14.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:14 trial068 ceph-mon[38805]: from='mgr.14118 10.20.193.68:0/3047353658' entity='mgr.x' 2026-02-23T13:06:14.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:14 trial068 ceph-mon[38805]: Found migration_current of "None". Setting to last migration. 2026-02-23T13:06:14.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:14 trial068 ceph-mon[38805]: from='mgr.14118 10.20.193.68:0/3047353658' entity='mgr.x' 2026-02-23T13:06:14.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:14 trial068 ceph-mon[38805]: from='mgr.14118 10.20.193.68:0/3047353658' entity='mgr.x' 2026-02-23T13:06:14.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:14 trial068 ceph-mon[38805]: from='mgr.14118 10.20.193.68:0/3047353658' entity='mgr.x' 2026-02-23T13:06:14.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:14 trial068 ceph-mon[38805]: from='mgr.14118 10.20.193.68:0/3047353658' entity='mgr.x' 2026-02-23T13:06:14.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:14 trial068 ceph-mon[38805]: from='mgr.14118 10.20.193.68:0/3047353658' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2026-02-23T13:06:14.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:14 trial068 ceph-mon[38805]: from='mgr.14118 10.20.193.68:0/3047353658' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2026-02-23T13:06:14.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:14 trial068 ceph-mon[38805]: from='mgr.14118 10.20.193.68:0/3047353658' entity='mgr.x' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"} : dispatch 2026-02-23T13:06:14.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:14 trial068 ceph-mon[38805]: from='mgr.14118 10.20.193.68:0/3047353658' entity='mgr.x' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"} : dispatch 2026-02-23T13:06:14.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:14 trial068 ceph-mon[38805]: from='mgr.14118 10.20.193.68:0/3047353658' entity='mgr.x' 2026-02-23T13:06:14.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:14 trial068 ceph-mon[38805]: from='mgr.14118 10.20.193.68:0/3047353658' entity='mgr.x' 2026-02-23T13:06:14.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:14 trial068 ceph-mon[38805]: mgrmap e7: x(active, since 1.00335s) 2026-02-23T13:06:14.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:14 trial068 ceph-mon[38805]: from='client.14122 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2026-02-23T13:06:14.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:14 trial068 ceph-mon[38805]: from='client.14122 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2026-02-23T13:06:14.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:14 trial068 ceph-mon[38805]: [23/Feb/2026:13:06:14] ENGINE Bus STARTING 2026-02-23T13:06:14.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:14 trial068 ceph-mon[38805]: [23/Feb/2026:13:06:14] ENGINE Serving on https://10.20.193.68:7150 2026-02-23T13:06:14.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:14 trial068 ceph-mon[38805]: [23/Feb/2026:13:06:14] ENGINE Client ('10.20.193.68', 44318) lost — peer dropped the TLS connection suddenly, during handshake: (6, 'TLS/SSL connection has been closed (EOF) (_ssl.c:1147)') 2026-02-23T13:06:14.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:14 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2114182204' entity='client.admin' cmd={"prefix": "mgr module enable", "module": "orchestrator"} : dispatch 2026-02-23T13:06:14.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:14 trial068 ceph-mon[38805]: from='mgr.14118 10.20.193.68:0/3047353658' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2026-02-23T13:06:14.985 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stderr 2026-02-23T13:06:14.074+0000 7f0964c60640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:06:14.985 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stderr 2026-02-23T13:06:14.078+0000 7f0964c60640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:06:14.985 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stderr module 'orchestrator' is already enabled (always-on) 2026-02-23T13:06:14.985 INFO:teuthology.orchestra.run.trial068.stdout:Setting orchestrator backend to cephadm... 2026-02-23T13:06:15.216 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stderr 2026-02-23T13:06:15.071+0000 7fed2b35d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:06:15.217 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stderr 2026-02-23T13:06:15.075+0000 7fed2b35d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:06:15.446 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stdout value unchanged 2026-02-23T13:06:15.446 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stderr 2026-02-23T13:06:15.309+0000 7f4fea533640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:06:15.447 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stderr 2026-02-23T13:06:15.313+0000 7f4fea533640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:06:15.447 INFO:teuthology.orchestra.run.trial068.stdout:Generating ssh key... 2026-02-23T13:06:15.689 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stderr 2026-02-23T13:06:15.532+0000 7f987df07640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:06:15.690 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stderr 2026-02-23T13:06:15.535+0000 7f987df07640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:06:15.736 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:06:15 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: Generating public/private ed25519 key pair. 2026-02-23T13:06:15.736 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:06:15 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: Your identification has been saved in /tmp/tmpwzea9lq9/key 2026-02-23T13:06:15.736 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:06:15 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: Your public key has been saved in /tmp/tmpwzea9lq9/key.pub 2026-02-23T13:06:15.736 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:06:15 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: The key fingerprint is: 2026-02-23T13:06:15.736 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:06:15 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: SHA256:2DwuKnGjTfk33X+sBAkRBTWBFeYUW3odRnPPSt1SDMo ceph-5f823084-10b8-11f1-be80-d404e6e7d460 2026-02-23T13:06:15.736 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:06:15 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: The key's randomart image is: 2026-02-23T13:06:15.736 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:06:15 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: +--[ED25519 256]--+ 2026-02-23T13:06:15.736 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:06:15 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: | +B@++Bo| 2026-02-23T13:06:15.736 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:06:15 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: | .* *.=B| 2026-02-23T13:06:15.736 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:06:15 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: | . E + *| 2026-02-23T13:06:15.737 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:06:15 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: | + . + o | 2026-02-23T13:06:15.737 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:06:15 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: | .. S o . | 2026-02-23T13:06:15.737 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:06:15 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: | . = . . . | 2026-02-23T13:06:15.737 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:06:15 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: | * o. .. . .. | 2026-02-23T13:06:15.737 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:06:15 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: | o ....o . .. o| 2026-02-23T13:06:15.737 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:06:15 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: | .. . . .oo | 2026-02-23T13:06:15.737 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:06:15 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: +----[SHA256]-----+ 2026-02-23T13:06:15.918 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stdout ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIBvqpjHgiqo7XwHkyYf//97+ZJ1I2+OqJ/FUq2ubUEkD ceph-5f823084-10b8-11f1-be80-d404e6e7d460 2026-02-23T13:06:15.918 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stderr 2026-02-23T13:06:15.784+0000 7f594794f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:06:15.918 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stderr 2026-02-23T13:06:15.788+0000 7f594794f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:06:15.919 INFO:teuthology.orchestra.run.trial068.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2026-02-23T13:06:15.919 INFO:teuthology.orchestra.run.trial068.stdout:Adding key to root@localhost authorized_keys... 2026-02-23T13:06:15.920 INFO:teuthology.orchestra.run.trial068.stdout:Adding host trial068... 2026-02-23T13:06:16.009 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:15 trial068 ceph-mon[38805]: [23/Feb/2026:13:06:14] ENGINE Serving on http://10.20.193.68:8765 2026-02-23T13:06:16.009 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:15 trial068 ceph-mon[38805]: [23/Feb/2026:13:06:14] ENGINE Bus STARTED 2026-02-23T13:06:16.009 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:15 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2114182204' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "orchestrator"}]': finished 2026-02-23T13:06:16.009 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:15 trial068 ceph-mon[38805]: mgrmap e8: x(active, since 2s) 2026-02-23T13:06:16.010 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:15 trial068 ceph-mon[38805]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2026-02-23T13:06:16.010 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:15 trial068 ceph-mon[38805]: from='mgr.14118 10.20.193.68:0/3047353658' entity='mgr.x' 2026-02-23T13:06:16.010 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:15 trial068 ceph-mon[38805]: from='mgr.14118 10.20.193.68:0/3047353658' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2026-02-23T13:06:16.010 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:15 trial068 ceph-mon[38805]: from='mgr.14118 10.20.193.68:0/3047353658' entity='mgr.x' 2026-02-23T13:06:16.010 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:15 trial068 ceph-mon[38805]: from='mgr.14118 10.20.193.68:0/3047353658' entity='mgr.x' 2026-02-23T13:06:17.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:16 trial068 ceph-mon[38805]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2026-02-23T13:06:17.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:16 trial068 ceph-mon[38805]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2026-02-23T13:06:17.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:16 trial068 ceph-mon[38805]: Generating ssh key... 2026-02-23T13:06:17.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:16 trial068 ceph-mon[38805]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2026-02-23T13:06:17.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:16 trial068 ceph-mon[38805]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "trial068", "addr": "10.20.193.68", "target": ["mon-mgr", ""]}]: dispatch 2026-02-23T13:06:17.716 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stdout Added host 'trial068' with addr '10.20.193.68' 2026-02-23T13:06:17.717 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stderr 2026-02-23T13:06:16.010+0000 7f552ee09640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:06:17.717 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stderr 2026-02-23T13:06:16.015+0000 7f552ee09640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:06:17.717 INFO:teuthology.orchestra.run.trial068.stdout:Deploying unmanaged mon service... 2026-02-23T13:06:17.967 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2026-02-23T13:06:17.968 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stderr 2026-02-23T13:06:17.822+0000 7f3c5e0fb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:06:17.968 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stderr 2026-02-23T13:06:17.826+0000 7f3c5e0fb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:06:17.968 INFO:teuthology.orchestra.run.trial068.stdout:Deploying unmanaged mgr service... 2026-02-23T13:06:18.217 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:17 trial068 ceph-mon[38805]: Deploying cephadm binary to trial068 2026-02-23T13:06:18.218 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:17 trial068 ceph-mon[38805]: from='mgr.14118 10.20.193.68:0/3047353658' entity='mgr.x' 2026-02-23T13:06:18.218 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:17 trial068 ceph-mon[38805]: from='mgr.14118 10.20.193.68:0/3047353658' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2026-02-23T13:06:18.218 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:17 trial068 ceph-mon[38805]: from='mgr.14118 10.20.193.68:0/3047353658' entity='mgr.x' 2026-02-23T13:06:18.220 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2026-02-23T13:06:18.220 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stderr 2026-02-23T13:06:18.058+0000 7fbdbf6ac640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:06:18.220 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stderr 2026-02-23T13:06:18.062+0000 7fbdbf6ac640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:06:18.451 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stderr 2026-02-23T13:06:18.313+0000 7f0dac965640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:06:18.451 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stderr 2026-02-23T13:06:18.318+0000 7f0dac965640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:06:18.689 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stderr 2026-02-23T13:06:18.543+0000 7f8ecbc7c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:06:18.689 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stderr 2026-02-23T13:06:18.548+0000 7f8ecbc7c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:06:18.689 INFO:teuthology.orchestra.run.trial068.stdout:Enabling the dashboard module... 2026-02-23T13:06:19.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:19 trial068 ceph-mon[38805]: Added host trial068 2026-02-23T13:06:19.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:19 trial068 ceph-mon[38805]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2026-02-23T13:06:19.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:19 trial068 ceph-mon[38805]: Saving service mon spec with placement count:5 2026-02-23T13:06:19.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:19 trial068 ceph-mon[38805]: from='client.14144 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2026-02-23T13:06:19.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:19 trial068 ceph-mon[38805]: Saving service mgr spec with placement count:2 2026-02-23T13:06:19.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:19 trial068 ceph-mon[38805]: from='mgr.14118 10.20.193.68:0/3047353658' entity='mgr.x' 2026-02-23T13:06:19.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:19 trial068 ceph-mon[38805]: from='mgr.14118 10.20.193.68:0/3047353658' entity='mgr.x' 2026-02-23T13:06:19.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:19 trial068 ceph-mon[38805]: from='mgr.14118 10.20.193.68:0/3047353658' entity='mgr.x' 2026-02-23T13:06:19.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:19 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3375122859' entity='client.admin' 2026-02-23T13:06:19.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:19 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3882075708' entity='client.admin' 2026-02-23T13:06:19.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:19 trial068 ceph-mon[38805]: from='mgr.14118 10.20.193.68:0/3047353658' entity='mgr.x' 2026-02-23T13:06:19.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:19 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/306857404' entity='client.admin' cmd={"prefix": "mgr module enable", "module": "dashboard"} : dispatch 2026-02-23T13:06:19.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:19 trial068 ceph-mon[38805]: from='mgr.14118 10.20.193.68:0/3047353658' entity='mgr.x' 2026-02-23T13:06:19.728 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stderr 2026-02-23T13:06:18.784+0000 7f8b1bcb1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:06:19.728 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stderr 2026-02-23T13:06:18.789+0000 7f8b1bcb1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:06:19.969 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:06:19 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ignoring --setuser ceph since I am not root 2026-02-23T13:06:19.970 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:06:19 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ignoring --setgroup ceph since I am not root 2026-02-23T13:06:19.970 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:06:19 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: 2026-02-23T13:06:19.742+0000 7fc1e85cbfc0 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:06:19.970 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:06:19 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: 2026-02-23T13:06:19.745+0000 7fc1e85cbfc0 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:06:19.970 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:06:19 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: 2026-02-23T13:06:19.745+0000 7fc1e85cbfc0 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:06:19.992 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stdout { 2026-02-23T13:06:19.992 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stdout "epoch": 9, 2026-02-23T13:06:19.992 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stdout "available": true, 2026-02-23T13:06:19.992 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stdout "active_name": "x", 2026-02-23T13:06:19.992 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stdout "num_standby": 0 2026-02-23T13:06:19.992 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stdout } 2026-02-23T13:06:19.992 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stderr 2026-02-23T13:06:19.825+0000 7f08b17d2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:06:19.993 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stderr 2026-02-23T13:06:19.829+0000 7f08b17d2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:06:19.993 INFO:teuthology.orchestra.run.trial068.stdout:Waiting for the mgr to restart... 2026-02-23T13:06:19.993 INFO:teuthology.orchestra.run.trial068.stdout:Waiting for mgr epoch 9... 2026-02-23T13:06:21.094 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:06:20 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: /lib64/python3.9/site-packages/scipy/__init__.py:73: UserWarning: NumPy was imported from a Python sub-interpreter but NumPy does not properly support sub-interpreters. This will likely work for most users but might cause hard to track down issues or subtle bugs. A common user of the rare sub-interpreter feature is wsgi which also allows single-interpreter mode. 2026-02-23T13:06:21.094 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:06:20 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: Improvements in the case of bugs are welcome, but is not on the NumPy roadmap, and full support may require significant effort to achieve. 2026-02-23T13:06:21.094 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:06:20 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: from numpy import show_config as show_numpy_config 2026-02-23T13:06:21.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:20 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/306857404' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2026-02-23T13:06:21.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:20 trial068 ceph-mon[38805]: mgrmap e9: x(active, since 6s) 2026-02-23T13:06:21.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:20 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/286476006' entity='client.admin' cmd={"prefix": "mgr stat"} : dispatch 2026-02-23T13:06:22.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:22 trial068 ceph-mon[38805]: Active manager daemon x restarted 2026-02-23T13:06:22.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:22 trial068 ceph-mon[38805]: Activating manager daemon x 2026-02-23T13:06:22.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:22 trial068 ceph-mon[38805]: osdmap e3: 0 total, 0 up, 0 in 2026-02-23T13:06:22.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:22 trial068 ceph-mon[38805]: mgrmap e10: x(active, starting, since 0.00294719s) 2026-02-23T13:06:22.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:22 trial068 ceph-mon[38805]: from='mgr.14152 10.20.193.68:0/3466135759' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2026-02-23T13:06:22.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:22 trial068 ceph-mon[38805]: from='mgr.14152 10.20.193.68:0/3466135759' entity='mgr.x' cmd={"prefix": "mgr metadata", "who": "x", "id": "x"} : dispatch 2026-02-23T13:06:22.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:22 trial068 ceph-mon[38805]: from='mgr.14152 10.20.193.68:0/3466135759' entity='mgr.x' cmd={"prefix": "mds metadata"} : dispatch 2026-02-23T13:06:22.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:22 trial068 ceph-mon[38805]: from='mgr.14152 10.20.193.68:0/3466135759' entity='mgr.x' cmd={"prefix": "osd metadata"} : dispatch 2026-02-23T13:06:22.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:22 trial068 ceph-mon[38805]: from='mgr.14152 10.20.193.68:0/3466135759' entity='mgr.x' cmd={"prefix": "mon metadata"} : dispatch 2026-02-23T13:06:22.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:22 trial068 ceph-mon[38805]: Manager daemon x is now available 2026-02-23T13:06:23.561 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stdout { 2026-02-23T13:06:23.561 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2026-02-23T13:06:23.562 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stdout "initialized": true 2026-02-23T13:06:23.562 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stdout } 2026-02-23T13:06:23.562 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stderr 2026-02-23T13:06:20.094+0000 7fd0128ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:06:23.562 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stderr 2026-02-23T13:06:20.098+0000 7fd0128ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:06:23.562 INFO:teuthology.orchestra.run.trial068.stdout:mgr epoch 9 is available 2026-02-23T13:06:23.562 INFO:teuthology.orchestra.run.trial068.stdout:Using certmgr to generate dashboard self-signed certificate... 2026-02-23T13:06:23.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:23 trial068 ceph-mon[38805]: from='mgr.14152 10.20.193.68:0/3466135759' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2026-02-23T13:06:23.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:23 trial068 ceph-mon[38805]: from='mgr.14152 10.20.193.68:0/3466135759' entity='mgr.x' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"} : dispatch 2026-02-23T13:06:23.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:23 trial068 ceph-mon[38805]: from='mgr.14152 10.20.193.68:0/3466135759' entity='mgr.x' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"} : dispatch 2026-02-23T13:06:23.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:23 trial068 ceph-mon[38805]: from='mgr.14152 10.20.193.68:0/3466135759' entity='mgr.x' 2026-02-23T13:06:23.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:23 trial068 ceph-mon[38805]: from='mgr.14152 10.20.193.68:0/3466135759' entity='mgr.x' 2026-02-23T13:06:23.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:23 trial068 ceph-mon[38805]: mgrmap e11: x(active, since 1.00459s) 2026-02-23T13:06:24.058 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stdout {"cert": "-----BEGIN CERTIFICATE-----\nMIIFHzCCAwegAwIBAgIUAz5nJuvvfNDi4v0VWh3DCuIQZG4wDQYJKoZIhvcNAQEL\nBQAwPDE6MDgGA1UEAwwxY2VwaGFkbS1yb290LTVmODIzMDg0LTEwYjgtMTFmMS1i\nZTgwLWQ0MDRlNmU3ZDQ2MDAeFw0yNjAyMjMxMzA2MjRaFw0yOTAyMjIxMzA2MjRa\nMBcxFTATBgNVBAMMDDEwLjIwLjE5My42ODCCAiIwDQYJKoZIhvcNAQEBBQADggIP\nADCCAgoCggIBAK9gaqi1QIUgeyuLtrE8pFbJgidqVJDWI5TZ0BCPqZMM4zx7Vo0g\nf7CauT6UGBLK0SzBjLCkSK5v0ntWoySh4mhleTPW/OXUv3NWTTjVyqnvk38HHWly\nYNvpJbfVMRSL4eLKFo/rX1rhFJmcyrB6ZgKVlr4wLlp3Y/62P4SFSTYhss6ymfXt\nHGaw5txM7hl/srhsGELF00IoXe1qktyexUVm9Ehw76kVsrx+zZVyvIdsXCCbE20P\n+TiPX5mFooxwwlyLWvrvBr6xaVGvAVvQzve6Rwiqu16dvRtTHa4vS4L8VT5fcPkh\nAwwwDFs5rBkS2RHpufW2ttIg2vsQ84RsMC3+eiprafctrmjk/gqvRznLqOLjAMKx\nY4238hNzbexZUEoPZylVDDMod7jC61UdKUZsr/bk1Yoh2YVHv1zFAaAStUfUh3ih\n+z9CYmt2LhCDIsSXCiP7o+8dBNVGRBe9Y7ZU+sz7Ce2T1wLNeLeGGH/wCXQ9myEA\nkKOrTYksNaHoiurnGvzbxSY9EEDoFS4khcx0JQlGiIt0ZMB9v15BZPEOpGuZfcIC\nbjihLObGZIF7KiRtE2bUy3chm/KSBQm9MVsGM9eqBFNNM9HPkRqwghSRAcP+OFPX\nRk4eIGxiMaoZf5eu87mCAVx0e2kQKcRC1Ji2TAtvdSAAKu1OLqZIbljRAgMBAAGj\nPjA8MCwGA1UdEQQlMCOCCHRyaWFsMDY4ghFkYXNoYm9hcmRfc2VydmVyc4cEChTB\nRDAMBgNVHRMBAf8EAjAAMA0GCSqGSIb3DQEBCwUAA4ICAQB9gqazrFjWcE+DuR+w\nP0ngFt0FZJbDcV3Yf4hr/21U4ZlgvRyaKuQxnhCrneDhNzFaUcsX0P3V/CP6iGZ6\nsWlFzrBxX7Uk2ceUAQqQwOeZvUC5PsCuONvL1KuD9BcMaw6IUeGbF/zjRrWC3R4Q\nR4IQY9rkkRK3b2hyZChyH1k9GoWqOQByCpI79yxfQUdU2dRFTFIo1cX0JniMBqnM\n+khnGrm3YOhc+FXYJ6QUVM4Bjx5v6h4F/4ygpvZfhWtdvuQu/HO4XRQJQjS9Hp5N\nuPlajUh2xq4UwYSl7YMZKozUZzTz32K7LIt8vxXvz8RuchnHqL2Tzj7MpKZsCWWM\n5Z9F4ApJL+vKSjw4A6vWc8aRF5UWGU88pSCe30uMOKHN/d8Az3SAdokbhqoodiiV\ncd7ClryuIQqux62P7SQNxNtnbnTkHHL67q1ZS58p+VTLodPoYXrpX4nDr5m3R/CT\n3leB419xvyv+KWb7I6ibTMaM0CU6i4w+wYWdylmL1CRQlbdJpvFBZI/AtVbo5acn\no7D6wOvvcK+PjQYiaykObSR+wbLyB/Uc06LPbKKKvd6ttzq8qGyBp5j/EviK7yZd\nffkgTReRM1MZuUdSEBMD0kpywYnuAf+7cOVLrLz10dzbrjw+rdZNApeF/XqnxOp8\nnZayptq1ZPfWT17jZfSm10boDg==\n-----END CERTIFICATE-----\n", "key": "-----BEGIN RSA PRIVATE KEY-----\nMIIJJwIBAAKCAgEAr2BqqLVAhSB7K4u2sTykVsmCJ2pUkNYjlNnQEI+pkwzjPHtW\njSB/sJq5PpQYEsrRLMGMsKRIrm/Se1ajJKHiaGV5M9b85dS/c1ZNONXKqe+Tfwcd\naXJg2+klt9UxFIvh4soWj+tfWuEUmZzKsHpmApWWvjAuWndj/rY/hIVJNiGyzrKZ\n9e0cZrDm3EzuGX+yuGwYQsXTQihd7WqS3J7FRWb0SHDvqRWyvH7NlXK8h2xcIJsT\nbQ/5OI9fmYWijHDCXIta+u8GvrFpUa8BW9DO97pHCKq7Xp29G1Mdri9LgvxVPl9w\n+SEDDDAMWzmsGRLZEem59ba20iDa+xDzhGwwLf56Kmtp9y2uaOT+Cq9HOcuo4uMA\nwrFjjbfyE3Nt7FlQSg9nKVUMMyh3uMLrVR0pRmyv9uTViiHZhUe/XMUBoBK1R9SH\neKH7P0Jia3YuEIMixJcKI/uj7x0E1UZEF71jtlT6zPsJ7ZPXAs14t4YYf/AJdD2b\nIQCQo6tNiSw1oeiK6uca/NvFJj0QQOgVLiSFzHQlCUaIi3RkwH2/XkFk8Q6ka5l9\nwgJuOKEs5sZkgXsqJG0TZtTLdyGb8pIFCb0xWwYz16oEU00z0c+RGrCCFJEBw/44\nU9dGTh4gbGIxqhl/l67zuYIBXHR7aRApxELUmLZMC291IAAq7U4upkhuWNECAwEA\nAQKCAf8o91y/pZc5gcNDzyvd1H/CLIM6sc8POVywAjExrSoileAYc+pdc20tOipx\n2iYr1zSfDJc3ElxOYwPHy9uDTLPJrx2gyzEU5TcGuiIciwdl3fcv6EZc9PR4A8r+\n8wEdmFwrndYNyFPkTRXpMoO/6NeE9jlANAEGVBBuCPEz0FUWgy8OTKoQlR9zqkV2\nY7U6pgAZesAwDAXm1LpPVMj5jOm7GVGQZzeUeuN4viYUj7itgzwB+2rBdZNbb7vl\nILikfBjquG+ILk2YPdj11VHiU+RrLOUEh6oJ1miJX+D1aOOhG7bN2xciTfo1EdEE\nSw4bmhhN47FZgSGZ9BLMkvXo+PZHy3Ukt6bHbiR745FUZ+vTc/8h052UozOuJxpK\nHKu0X+OlYptw3wgZQlwPGP5CGJMi7cLiuuSMVVjyDYhyxBlBJIox0H9pHmKr3PPz\nsd1FcvE7pKjvKUt5NsyiLvKGAaK0xG12dJaY3tSYhdirCnTBBIjuO16a75sPwSGi\nNwtsMy5zuP6BzXshexpAlgH2VoC+DhNGt1T/QULUyUOto0UEgqVM5srYb2ZImTid\nLM+hvPyaffaQikvZR8oCNPeJE1a2EXmJGIqxZQtiUa6zgRBLM+u9N4L3cvhNAEEN\nkfU6sHZ6l7ZHCeooRUg0y3RjtHOEJHk+iBvY8uuYI9fBLbQRAoIBAQDeWPTW2ZT/\n9GUB7yawByDxedTZaddVyYUbVUbyK1Uh3bmAtRpUOLvgMNgtTrkCKnE5auTdNVcn\nu+UwjSUW9063RJqFIRDUQkN8EWa0UhBp6Ad5UdJVK+ef9Xi+Vvlc7y/hsLICu//O\nDgGE4zwP+BBp9osPtsQokaGsfV3+wUElOQkVShNhaN9tKPtVK/NWxWl+RgsNX4aq\neFTCD3cCEUju6eE/yKCSAG74YF9BA+UxucWcU6MwC3YJ8d6g87T2qRxkaxIXWjA6\nVGvyJgNewdNixPHYtK7qukK00P/q2GrMaJ3eIu+ekULSBXTbUs4wQbkr/eunYsm1\n+2wm35yq5aJtAoIBAQDJ64i805wMq9POSeNA9Xja8Xj2HwWsKIM9pgk7ORoE9Xkf\nKdAqd/uaOKwcqn94LcjFQnCs9lbTLY4b/0T0eYjVCw8cfWpNT7cSacier5+4bAvy\ng1T79piTAlCeRNk5PvW+D8GSMyl0VEew27pyK5aXrGPBlW86aS+LQhe29zaefHgQ\nHrKyEFxhUANEvckp8NhwOuZJZHeS4Ov1f65NP7owlNJ++5CgLscZXgcb9Y7OEZVG\nMInWJfARgk/kVFxMotfcEKzeiMsoi5UcfdcpqSV/g0bazw9CcAFC+ZLkCZ/ZHD/J\nJhtv4tiglHKX5ZV/Ip9aZwKmBnCVqwnip60OTXF1AoIBAQDL6VsJWtv57GFh57hc\nQP1mV6vrg9SW+UxYJ9n7p9/YQiaose4mhS1PypvG2WZ0ZQYXeT/Q2uVGLFxhLWBx\ncna/PmxTYWK0ttx7AN5mysgAyGlqSq6Rq8UtfoqwTMU+BKGt4kUumslbaJ7eByOD\nJaWv+AnKbHvzFLPno8W7J/Kdt3PsEVY6xFn7G1V74+WzkzJPOt9R8WRoHBjnLBWh\nUMbJTLZlr5IKs9kRchiJYJAmDU/8MPv8yNYFAsuAYmB4wdinA0bBIdYO+HNSzKfU\nk/9PmnyNmA6+d/lLKE07TCSNt/fny7RjgKZ+Q8DKj2NHAE9Y1/LxPHjuq1ihQfvF\nAvn1AoIBABVZ9KGS9sNYQBNmOMObsWXldtb/vMfRmN8h+YPrq9auZlpnP33eJhrO\nt1RoqwX4QWyqc+O2QMQf4xEO0ynO2Cs++iAXGvCF52PpD6/H/mmv8yt3u7rQ/p08\n921k9ltZTv8EB4IAKvgEGXiYI4UDGu966SmFNbfFe6OMH5nXVk03+DIsLNUrablY\njp/sOaNXtmtXrxY/q206n3jcQYAlpviQJBoNW+O0TpVeHf5U3v7nWROM+mZMqLAR\nSYVpbkEkTxMUKFN3v5PP/rp+jAv5uN9c8Fce4gwyb8+3OFMouWBp3kzU3L3eDO4e\nBGgPfNluvc7/aYZI1ArfhTIUbPZANikCggEAYBGTkw0aLbXUKvD/7qhWm53DwPds\nLC0UbTlt69ZS+ZsMI3uW8HSDBLY12hTDy3QkipdfR5aSiBYuMgtDfdbV8la7HBav\nkQMxbBIhwQEUmI15+oS3a/kqzzk6Z4prakJmg7XKCOWxDCfJqwPNRll1lX9F5G7Q\nbcNa8gMYy2SbW4ePEg+hKiBZeNlGU2YPtq4e0QWb4hozCd5Q/b01V0Ca2oBp5Ngs\nQt1ayMwommamlKxkUN5vQ1C1skk6Hoa/mLiW6zJ/FzpNFY65oXO5a4k37IauUnqN\nQ9BWpTvCCbqE7EozfTB05Ca2P+mprFZlHSDEt0goPNrH0IOVUxLAIuds8w==\n-----END RSA PRIVATE KEY-----\n"} 2026-02-23T13:06:24.058 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stderr 2026-02-23T13:06:23.662+0000 7f101a87e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:06:24.059 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stderr 2026-02-23T13:06:23.666+0000 7f101a87e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:06:24.301 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stdout SSL certificate updated 2026-02-23T13:06:24.301 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stderr 2026-02-23T13:06:24.159+0000 7ff1ae785640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:06:24.301 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stderr 2026-02-23T13:06:24.163+0000 7ff1ae785640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:06:24.539 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stdout SSL certificate key updated 2026-02-23T13:06:24.539 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stderr 2026-02-23T13:06:24.398+0000 7faeba816640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:06:24.539 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stderr 2026-02-23T13:06:24.402+0000 7faeba816640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:06:24.539 INFO:teuthology.orchestra.run.trial068.stdout:Creating initial admin user... 2026-02-23T13:06:24.542 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:24 trial068 ceph-mon[38805]: [23/Feb/2026:13:06:22] ENGINE Bus STARTING 2026-02-23T13:06:24.542 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:24 trial068 ceph-mon[38805]: [23/Feb/2026:13:06:22] ENGINE Serving on http://10.20.193.68:8765 2026-02-23T13:06:24.542 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:24 trial068 ceph-mon[38805]: [23/Feb/2026:13:06:23] ENGINE Serving on https://10.20.193.68:7150 2026-02-23T13:06:24.542 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:24 trial068 ceph-mon[38805]: [23/Feb/2026:13:06:23] ENGINE Bus STARTED 2026-02-23T13:06:24.542 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:24 trial068 ceph-mon[38805]: [23/Feb/2026:13:06:23] ENGINE Client ('10.20.193.68', 40184) lost — peer dropped the TLS connection suddenly, during handshake: (6, 'TLS/SSL connection has been closed (EOF) (_ssl.c:1147)') 2026-02-23T13:06:24.542 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:24 trial068 ceph-mon[38805]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2026-02-23T13:06:24.542 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:24 trial068 ceph-mon[38805]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2026-02-23T13:06:24.542 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:24 trial068 ceph-mon[38805]: from='mgr.14152 10.20.193.68:0/3466135759' entity='mgr.x' 2026-02-23T13:06:24.542 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:24 trial068 ceph-mon[38805]: from='mgr.14152 10.20.193.68:0/3466135759' entity='mgr.x' 2026-02-23T13:06:24.971 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$kR6VAqushTPhoe27i5nh4eYNI8ol3wSyfeSqQQldBu0RjwMnRUtGu", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1771851984, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2026-02-23T13:06:24.972 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stderr 2026-02-23T13:06:24.628+0000 7f373a65c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:06:24.972 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stderr 2026-02-23T13:06:24.634+0000 7f373a65c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:06:24.972 INFO:teuthology.orchestra.run.trial068.stdout:Fetching dashboard port number... 2026-02-23T13:06:25.195 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stdout 8443 2026-02-23T13:06:25.195 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stderr 2026-02-23T13:06:25.059+0000 7fada6895640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:06:25.196 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stderr 2026-02-23T13:06:25.064+0000 7fada6895640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:06:25.204 INFO:teuthology.orchestra.run.trial068.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2026-02-23T13:06:25.204 INFO:teuthology.orchestra.run.trial068.stdout:systemctl: stdout disabled 2026-02-23T13:06:25.210 INFO:teuthology.orchestra.run.trial068.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2026-02-23T13:06:25.210 INFO:teuthology.orchestra.run.trial068.stdout:systemctl: stdout inactive 2026-02-23T13:06:25.210 INFO:teuthology.orchestra.run.trial068.stdout:firewalld.service is not enabled 2026-02-23T13:06:25.210 INFO:teuthology.orchestra.run.trial068.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2026-02-23T13:06:25.211 INFO:teuthology.orchestra.run.trial068.stdout:Ceph Dashboard is now available at: 2026-02-23T13:06:25.211 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T13:06:25.211 INFO:teuthology.orchestra.run.trial068.stdout: URL: https://trial068:8443/ 2026-02-23T13:06:25.211 INFO:teuthology.orchestra.run.trial068.stdout: User: admin 2026-02-23T13:06:25.212 INFO:teuthology.orchestra.run.trial068.stdout: Password: 0upr8tg2al 2026-02-23T13:06:25.212 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T13:06:25.212 INFO:teuthology.orchestra.run.trial068.stdout:Saving cluster configuration to /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/config directory 2026-02-23T13:06:25.492 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stderr 2026-02-23T13:06:25.304+0000 7fdce12b6640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:06:25.492 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stderr 2026-02-23T13:06:25.308+0000 7fdce12b6640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:06:25.492 INFO:teuthology.orchestra.run.trial068.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2026-02-23T13:06:25.492 INFO:teuthology.orchestra.run.trial068.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2026-02-23T13:06:25.492 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T13:06:25.492 INFO:teuthology.orchestra.run.trial068.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2026-02-23T13:06:25.492 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T13:06:25.492 INFO:teuthology.orchestra.run.trial068.stdout:Or, if you are only running a single cluster on this host: 2026-02-23T13:06:25.493 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T13:06:25.493 INFO:teuthology.orchestra.run.trial068.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2026-02-23T13:06:25.493 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T13:06:25.493 INFO:teuthology.orchestra.run.trial068.stdout:Please consider enabling telemetry to help improve Ceph: 2026-02-23T13:06:25.493 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T13:06:25.493 INFO:teuthology.orchestra.run.trial068.stdout: ceph telemetry on 2026-02-23T13:06:25.493 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T13:06:25.493 INFO:teuthology.orchestra.run.trial068.stdout:For more information see: 2026-02-23T13:06:25.493 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T13:06:25.493 INFO:teuthology.orchestra.run.trial068.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2026-02-23T13:06:25.493 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T13:06:25.493 INFO:teuthology.orchestra.run.trial068.stdout:Bootstrap complete. 2026-02-23T13:06:25.499 INFO:teuthology.orchestra.run.trial068.stdout:systemctl: stdout static 2026-02-23T13:06:25.504 INFO:teuthology.orchestra.run.trial068.stdout:Non-zero exit code 3 from systemctl is-active logrotate 2026-02-23T13:06:25.504 INFO:teuthology.orchestra.run.trial068.stdout:systemctl: stdout inactive 2026-02-23T13:06:25.504 INFO:teuthology.orchestra.run.trial068.stdout:Enabling the logrotate.timer service to perform daily log rotation. 2026-02-23T13:06:25.661 INFO:tasks.cephadm:Fetching config... 2026-02-23T13:06:25.662 DEBUG:teuthology.orchestra.run.trial068:> set -ex 2026-02-23T13:06:25.662 DEBUG:teuthology.orchestra.run.trial068:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2026-02-23T13:06:25.679 INFO:tasks.cephadm:Fetching client.admin keyring... 2026-02-23T13:06:25.680 DEBUG:teuthology.orchestra.run.trial068:> set -ex 2026-02-23T13:06:25.680 DEBUG:teuthology.orchestra.run.trial068:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2026-02-23T13:06:25.737 INFO:tasks.cephadm:Fetching mon keyring... 2026-02-23T13:06:25.737 DEBUG:teuthology.orchestra.run.trial068:> set -ex 2026-02-23T13:06:25.737 DEBUG:teuthology.orchestra.run.trial068:> sudo dd if=/var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/keyring of=/dev/stdout 2026-02-23T13:06:25.784 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:25 trial068 ceph-mon[38805]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "orch certmgr generate-certificates", "module_name": "dashboard", "target": ["mon-mgr", ""]}]: dispatch 2026-02-23T13:06:25.784 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:25 trial068 ceph-mon[38805]: from='client.14166 -' entity='client.admin' cmd=[{"prefix": "dashboard set-ssl-certificate", "target": ["mon-mgr", ""]}]: dispatch 2026-02-23T13:06:25.785 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:25 trial068 ceph-mon[38805]: from='client.14168 -' entity='client.admin' cmd=[{"prefix": "dashboard set-ssl-certificate-key", "target": ["mon-mgr", ""]}]: dispatch 2026-02-23T13:06:25.785 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:25 trial068 ceph-mon[38805]: from='mgr.14152 10.20.193.68:0/3466135759' entity='mgr.x' 2026-02-23T13:06:25.785 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:25 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2504188007' entity='client.admin' cmd={"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} : dispatch 2026-02-23T13:06:25.785 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:25 trial068 ceph-mon[38805]: mgrmap e12: x(active, since 2s) 2026-02-23T13:06:25.785 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:25 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1582250250' entity='client.admin' 2026-02-23T13:06:25.804 INFO:tasks.cephadm:Fetching pub ssh key... 2026-02-23T13:06:25.804 DEBUG:teuthology.orchestra.run.trial068:> set -ex 2026-02-23T13:06:25.804 DEBUG:teuthology.orchestra.run.trial068:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2026-02-23T13:06:25.860 INFO:tasks.cephadm:Installing pub ssh key for root users... 2026-02-23T13:06:25.861 DEBUG:teuthology.orchestra.run.trial068:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIBvqpjHgiqo7XwHkyYf//97+ZJ1I2+OqJ/FUq2ubUEkD ceph-5f823084-10b8-11f1-be80-d404e6e7d460' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2026-02-23T13:06:25.934 INFO:teuthology.orchestra.run.trial068.stdout:ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIBvqpjHgiqo7XwHkyYf//97+ZJ1I2+OqJ/FUq2ubUEkD ceph-5f823084-10b8-11f1-be80-d404e6e7d460 2026-02-23T13:06:25.944 DEBUG:teuthology.orchestra.run.trial185:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIBvqpjHgiqo7XwHkyYf//97+ZJ1I2+OqJ/FUq2ubUEkD ceph-5f823084-10b8-11f1-be80-d404e6e7d460' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2026-02-23T13:06:25.980 INFO:teuthology.orchestra.run.trial185.stdout:ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIBvqpjHgiqo7XwHkyYf//97+ZJ1I2+OqJ/FUq2ubUEkD ceph-5f823084-10b8-11f1-be80-d404e6e7d460 2026-02-23T13:06:25.990 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2026-02-23T13:06:26.104 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:06:26.211 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:06:26.209+0000 7f459e429640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:06:26.211 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:06:26.209+0000 7f459e429640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:06:26.361 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2026-02-23T13:06:26.361 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph orch client-keyring set client.admin '*' --mode 0755 2026-02-23T13:06:26.481 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:06:26.592 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:06:26.591+0000 7fd799e4c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:06:26.593 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:06:26.591+0000 7fd799e4c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:06:26.658 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:26 trial068 ceph-mon[38805]: from='client.14170 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2026-02-23T13:06:26.659 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:26 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1832672821' entity='client.admin' 2026-02-23T13:06:26.765 INFO:tasks.cephadm:Writing (initial) conf and keyring to trial185 2026-02-23T13:06:26.766 DEBUG:teuthology.orchestra.run.trial185:> set -ex 2026-02-23T13:06:26.766 DEBUG:teuthology.orchestra.run.trial185:> dd of=/etc/ceph/ceph.conf 2026-02-23T13:06:26.784 DEBUG:teuthology.orchestra.run.trial185:> set -ex 2026-02-23T13:06:26.784 DEBUG:teuthology.orchestra.run.trial185:> dd of=/etc/ceph/ceph.client.admin.keyring 2026-02-23T13:06:26.840 INFO:tasks.cephadm:Adding host trial185 to orchestrator... 2026-02-23T13:06:26.840 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph orch host add trial185 2026-02-23T13:06:26.958 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:06:27.065 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:06:27.063+0000 7f04753c8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:06:27.065 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:06:27.063+0000 7f04753c8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:06:28.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:27 trial068 ceph-mon[38805]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2026-02-23T13:06:28.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:27 trial068 ceph-mon[38805]: from='mgr.14152 10.20.193.68:0/3466135759' entity='mgr.x' 2026-02-23T13:06:28.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:27 trial068 ceph-mon[38805]: from='mgr.14152 10.20.193.68:0/3466135759' entity='mgr.x' 2026-02-23T13:06:28.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:27 trial068 ceph-mon[38805]: from='mgr.14152 10.20.193.68:0/3466135759' entity='mgr.x' 2026-02-23T13:06:28.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:27 trial068 ceph-mon[38805]: from='mgr.14152 10.20.193.68:0/3466135759' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd/host:trial068", "name": "osd_memory_target"} : dispatch 2026-02-23T13:06:28.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:27 trial068 ceph-mon[38805]: from='mgr.14152 10.20.193.68:0/3466135759' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-23T13:06:28.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:27 trial068 ceph-mon[38805]: from='mgr.14152 10.20.193.68:0/3466135759' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2026-02-23T13:06:28.896 INFO:teuthology.orchestra.run.trial068.stdout:Added host 'trial185' with addr '10.20.193.185' 2026-02-23T13:06:29.047 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:28 trial068 ceph-mon[38805]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "trial185", "target": ["mon-mgr", ""]}]: dispatch 2026-02-23T13:06:29.047 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:28 trial068 ceph-mon[38805]: Updating trial068:/etc/ceph/ceph.conf 2026-02-23T13:06:29.047 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:28 trial068 ceph-mon[38805]: Updating trial068:/var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/config/ceph.conf 2026-02-23T13:06:29.047 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:28 trial068 ceph-mon[38805]: Deploying cephadm binary to trial185 2026-02-23T13:06:29.047 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:28 trial068 ceph-mon[38805]: Updating trial068:/etc/ceph/ceph.client.admin.keyring 2026-02-23T13:06:29.047 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:28 trial068 ceph-mon[38805]: Updating trial068:/var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/config/ceph.client.admin.keyring 2026-02-23T13:06:29.047 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:28 trial068 ceph-mon[38805]: from='mgr.14152 10.20.193.68:0/3466135759' entity='mgr.x' 2026-02-23T13:06:29.047 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:28 trial068 ceph-mon[38805]: from='mgr.14152 10.20.193.68:0/3466135759' entity='mgr.x' 2026-02-23T13:06:29.047 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:28 trial068 ceph-mon[38805]: from='mgr.14152 10.20.193.68:0/3466135759' entity='mgr.x' 2026-02-23T13:06:29.047 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:28 trial068 ceph-mon[38805]: from='mgr.14152 10.20.193.68:0/3466135759' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2026-02-23T13:06:29.047 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:28 trial068 ceph-mon[38805]: from='mgr.14152 10.20.193.68:0/3466135759' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-23T13:06:29.047 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:28 trial068 ceph-mon[38805]: from='mgr.14152 10.20.193.68:0/3466135759' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2026-02-23T13:06:29.047 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:28 trial068 ceph-mon[38805]: from='mgr.14152 10.20.193.68:0/3466135759' entity='mgr.x' 2026-02-23T13:06:29.069 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph orch host ls --format=json 2026-02-23T13:06:29.186 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:06:29.289 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:06:29.288+0000 7f0b06da8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:06:29.289 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:06:29.288+0000 7f0b06da8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:06:29.400 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T13:06:29.400 INFO:teuthology.orchestra.run.trial068.stdout:[{"addr": "10.20.193.68", "hostname": "trial068", "labels": [], "status": ""}, {"addr": "10.20.193.185", "hostname": "trial185", "labels": [], "status": ""}] 2026-02-23T13:06:29.436 INFO:tasks.cephadm:Setting crush tunables to default 2026-02-23T13:06:29.436 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd crush tunables default 2026-02-23T13:06:29.551 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:06:29.652 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:06:29.651+0000 7f985ba1a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:06:29.653 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:06:29.651+0000 7f985ba1a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:06:29.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:29 trial068 ceph-mon[38805]: Certificate for "cephadm-signed_agent_cert (trial068)" is still valid for 1824 days. 2026-02-23T13:06:29.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:29 trial068 ceph-mon[38805]: from='mgr.14152 10.20.193.68:0/3466135759' entity='mgr.x' 2026-02-23T13:06:29.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:29 trial068 ceph-mon[38805]: from='mgr.14152 10.20.193.68:0/3466135759' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2026-02-23T13:06:29.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:29 trial068 ceph-mon[38805]: from='mgr.14152 10.20.193.68:0/3466135759' entity='mgr.x' 2026-02-23T13:06:29.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:29 trial068 ceph-mon[38805]: mgrmap e13: x(active, since 6s) 2026-02-23T13:06:29.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:29 trial068 ceph-mon[38805]: from='mgr.14152 10.20.193.68:0/3466135759' entity='mgr.x' 2026-02-23T13:06:30.707 INFO:teuthology.orchestra.run.trial068.stderr:adjusted tunables profile to default 2026-02-23T13:06:30.747 INFO:tasks.cephadm:Adding mon.a on trial068 2026-02-23T13:06:30.747 DEBUG:teuthology.orchestra.run.trial185:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph orch apply mon '1;trial068:10.20.193.68=a' 2026-02-23T13:06:30.867 INFO:teuthology.orchestra.run.trial185.stderr:Inferring config /etc/ceph/ceph.conf 2026-02-23T13:06:30.867 INFO:teuthology.orchestra.run.trial185.stderr:Inferring config /etc/ceph/ceph.conf 2026-02-23T13:06:30.990 INFO:teuthology.orchestra.run.trial185.stderr:2026-02-23T13:06:30.989+0000 7f83e8017640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:06:30.991 INFO:teuthology.orchestra.run.trial185.stderr:2026-02-23T13:06:30.989+0000 7f83e8017640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:06:31.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:30 trial068 ceph-mon[38805]: Added host trial185 2026-02-23T13:06:31.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:30 trial068 ceph-mon[38805]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2026-02-23T13:06:31.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:30 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3533231451' entity='client.admin' cmd={"prefix": "osd crush tunables", "profile": "default"} : dispatch 2026-02-23T13:06:31.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:30 trial068 ceph-mon[38805]: from='mgr.14152 10.20.193.68:0/3466135759' entity='mgr.x' 2026-02-23T13:06:31.112 INFO:teuthology.orchestra.run.trial185.stdout:Scheduled mon update... 2026-02-23T13:06:31.154 INFO:tasks.cephadm:Waiting for 1 mons in monmap... 2026-02-23T13:06:31.154 DEBUG:teuthology.orchestra.run.trial185:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph mon dump -f json 2026-02-23T13:06:31.336 INFO:teuthology.orchestra.run.trial185.stderr:Inferring config /etc/ceph/ceph.conf 2026-02-23T13:06:31.336 INFO:teuthology.orchestra.run.trial185.stderr:Inferring config /etc/ceph/ceph.conf 2026-02-23T13:06:31.452 INFO:teuthology.orchestra.run.trial185.stderr:2026-02-23T13:06:31.451+0000 7f6b14edf640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:06:31.453 INFO:teuthology.orchestra.run.trial185.stderr:2026-02-23T13:06:31.451+0000 7f6b14edf640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:06:31.606 INFO:teuthology.orchestra.run.trial185.stdout: 2026-02-23T13:06:31.606 INFO:teuthology.orchestra.run.trial185.stdout:{"epoch":1,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","modified":"2026-02-23T13:06:02.168892Z","created":"2026-02-23T13:06:02.168892Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:3300","nonce":0},{"type":"v1","addr":"10.20.193.68:6789","nonce":0}]},"addr":"10.20.193.68:6789/0","public_addr":"10.20.193.68:6789/0","priority":0,"weight":0,"time_added":"2026-02-23T13:06:02:168908+0000","crush_location":[]}],"quorum":[0]} 2026-02-23T13:06:31.607 INFO:teuthology.orchestra.run.trial185.stderr:dumped monmap epoch 1 2026-02-23T13:06:31.666 INFO:tasks.cephadm:Generating final ceph.conf file... 2026-02-23T13:06:31.667 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph config generate-minimal-conf 2026-02-23T13:06:31.782 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:06:31.877 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:06:31.876+0000 7fcffd0ff640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:06:31.877 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:06:31.876+0000 7fcffd0ff640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:06:31.987 INFO:teuthology.orchestra.run.trial068.stdout:# minimal ceph.conf for 5f823084-10b8-11f1-be80-d404e6e7d460 2026-02-23T13:06:31.987 INFO:teuthology.orchestra.run.trial068.stdout:[global] 2026-02-23T13:06:31.987 INFO:teuthology.orchestra.run.trial068.stdout: fsid = 5f823084-10b8-11f1-be80-d404e6e7d460 2026-02-23T13:06:31.987 INFO:teuthology.orchestra.run.trial068.stdout: mon_host = [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] 2026-02-23T13:06:31.988 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:31 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3533231451' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2026-02-23T13:06:31.988 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:31 trial068 ceph-mon[38805]: osdmap e4: 0 total, 0 up, 0 in 2026-02-23T13:06:31.988 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:31 trial068 ceph-mon[38805]: from='mgr.14152 10.20.193.68:0/3466135759' entity='mgr.x' 2026-02-23T13:06:31.988 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:31 trial068 ceph-mon[38805]: from='mgr.14152 10.20.193.68:0/3466135759' entity='mgr.x' 2026-02-23T13:06:31.988 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:31 trial068 ceph-mon[38805]: from='mgr.14152 10.20.193.68:0/3466135759' entity='mgr.x' 2026-02-23T13:06:31.988 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:31 trial068 ceph-mon[38805]: from='mgr.14152 10.20.193.68:0/3466135759' entity='mgr.x' 2026-02-23T13:06:31.989 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:31 trial068 ceph-mon[38805]: from='mgr.14152 10.20.193.68:0/3466135759' entity='mgr.x' 2026-02-23T13:06:31.989 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:31 trial068 ceph-mon[38805]: from='mgr.14152 10.20.193.68:0/3466135759' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd/host:trial185", "name": "osd_memory_target"} : dispatch 2026-02-23T13:06:31.989 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:31 trial068 ceph-mon[38805]: from='mgr.14152 10.20.193.68:0/3466135759' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-23T13:06:31.989 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:31 trial068 ceph-mon[38805]: from='mgr.14152 10.20.193.68:0/3466135759' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2026-02-23T13:06:31.989 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:31 trial068 ceph-mon[38805]: from='mgr.14152 10.20.193.68:0/3466135759' entity='mgr.x' 2026-02-23T13:06:31.989 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:31 trial068 ceph-mon[38805]: from='mgr.14152 10.20.193.68:0/3466135759' entity='mgr.x' 2026-02-23T13:06:31.990 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:31 trial068 ceph-mon[38805]: from='mgr.14152 10.20.193.68:0/3466135759' entity='mgr.x' 2026-02-23T13:06:31.990 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:31 trial068 ceph-mon[38805]: from='mgr.14152 10.20.193.68:0/3466135759' entity='mgr.x' 2026-02-23T13:06:31.990 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:31 trial068 ceph-mon[38805]: from='client.? 10.20.193.185:0/2171352220' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2026-02-23T13:06:31.990 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:31 trial068 ceph-mon[38805]: from='mgr.14152 10.20.193.68:0/3466135759' entity='mgr.x' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2026-02-23T13:06:31.990 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:31 trial068 ceph-mon[38805]: from='mgr.14152 10.20.193.68:0/3466135759' entity='mgr.x' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2026-02-23T13:06:31.990 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:31 trial068 ceph-mon[38805]: from='mgr.14152 10.20.193.68:0/3466135759' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-23T13:06:32.024 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2026-02-23T13:06:32.024 DEBUG:teuthology.orchestra.run.trial068:> set -ex 2026-02-23T13:06:32.024 DEBUG:teuthology.orchestra.run.trial068:> sudo dd of=/etc/ceph/ceph.conf 2026-02-23T13:06:32.049 DEBUG:teuthology.orchestra.run.trial068:> set -ex 2026-02-23T13:06:32.049 DEBUG:teuthology.orchestra.run.trial068:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2026-02-23T13:06:32.113 DEBUG:teuthology.orchestra.run.trial185:> set -ex 2026-02-23T13:06:32.114 DEBUG:teuthology.orchestra.run.trial185:> sudo dd of=/etc/ceph/ceph.conf 2026-02-23T13:06:32.141 DEBUG:teuthology.orchestra.run.trial185:> set -ex 2026-02-23T13:06:32.141 DEBUG:teuthology.orchestra.run.trial185:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2026-02-23T13:06:32.205 INFO:tasks.cephadm:Adding mgr.x on trial068 2026-02-23T13:06:32.205 DEBUG:teuthology.orchestra.run.trial185:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph orch apply mgr '1;trial068=x' 2026-02-23T13:06:32.353 INFO:teuthology.orchestra.run.trial185.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/config/ceph.conf 2026-02-23T13:06:32.469 INFO:teuthology.orchestra.run.trial185.stderr:2026-02-23T13:06:32.468+0000 7feba0a8d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:06:32.469 INFO:teuthology.orchestra.run.trial185.stderr:2026-02-23T13:06:32.468+0000 7feba0a8d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:06:32.589 INFO:teuthology.orchestra.run.trial185.stdout:Scheduled mgr update... 2026-02-23T13:06:32.646 DEBUG:tasks.cephadm:set 0 configs 2026-02-23T13:06:32.647 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph config dump 2026-02-23T13:06:32.763 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:06:32.866 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:06:32.864+0000 7f1a0e6c8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:06:32.866 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:06:32.864+0000 7f1a0e6c8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:06:32.977 INFO:teuthology.orchestra.run.trial068.stdout:WHO MASK LEVEL OPTION VALUE RO 2026-02-23T13:06:32.977 INFO:teuthology.orchestra.run.trial068.stdout:global dev auth_debug true 2026-02-23T13:06:32.977 INFO:teuthology.orchestra.run.trial068.stdout:global basic container_image quay.ceph.io/ceph-ci/ceph@sha256:49621faaa845c39b2aae72e4fa24123ee9b2ac3fb8ae834753907068ee2972ea * 2026-02-23T13:06:32.977 INFO:teuthology.orchestra.run.trial068.stdout:global advanced crimson_allow_pg_split false 2026-02-23T13:06:32.977 INFO:teuthology.orchestra.run.trial068.stdout:global dev debug_asserts_on_shutdown true 2026-02-23T13:06:32.977 INFO:teuthology.orchestra.run.trial068.stdout:global advanced enable_experimental_unrecoverable_data_corrupting_features crimson 2026-02-23T13:06:32.977 INFO:teuthology.orchestra.run.trial068.stdout:global basic log_to_file true 2026-02-23T13:06:32.977 INFO:teuthology.orchestra.run.trial068.stdout:global basic log_to_journald false 2026-02-23T13:06:32.978 INFO:teuthology.orchestra.run.trial068.stdout:global basic log_to_stderr false 2026-02-23T13:06:32.978 INFO:teuthology.orchestra.run.trial068.stdout:global advanced mon_allow_pool_delete true 2026-02-23T13:06:32.978 INFO:teuthology.orchestra.run.trial068.stdout:global advanced mon_clock_drift_allowed 1.000000 2026-02-23T13:06:32.978 INFO:teuthology.orchestra.run.trial068.stdout:global advanced mon_cluster_log_to_file true 2026-02-23T13:06:32.978 INFO:teuthology.orchestra.run.trial068.stdout:global advanced mon_max_pg_per_osd 10000 2026-02-23T13:06:32.978 INFO:teuthology.orchestra.run.trial068.stdout:global advanced mon_pg_warn_max_object_skew 0.000000 2026-02-23T13:06:32.978 INFO:teuthology.orchestra.run.trial068.stdout:global advanced mon_warn_on_crush_straw_calc_version_zero false 2026-02-23T13:06:32.978 INFO:teuthology.orchestra.run.trial068.stdout:global advanced mon_warn_on_legacy_crush_tunables false 2026-02-23T13:06:32.978 INFO:teuthology.orchestra.run.trial068.stdout:global advanced mon_warn_on_osd_down_out_interval_zero false 2026-02-23T13:06:32.978 INFO:teuthology.orchestra.run.trial068.stdout:global dev mon_warn_on_pool_pg_num_not_power_of_two false 2026-02-23T13:06:32.978 INFO:teuthology.orchestra.run.trial068.stdout:global advanced mon_warn_on_too_few_osds false 2026-02-23T13:06:32.978 INFO:teuthology.orchestra.run.trial068.stdout:global basic ms_client_mode crc * 2026-02-23T13:06:32.979 INFO:teuthology.orchestra.run.trial068.stdout:global basic ms_cluster_mode crc * 2026-02-23T13:06:32.979 INFO:teuthology.orchestra.run.trial068.stdout:global dev ms_die_on_bug true 2026-02-23T13:06:32.979 INFO:teuthology.orchestra.run.trial068.stdout:global dev ms_die_on_old_message true 2026-02-23T13:06:32.979 INFO:teuthology.orchestra.run.trial068.stdout:global basic ms_mon_client_mode crc * 2026-02-23T13:06:32.979 INFO:teuthology.orchestra.run.trial068.stdout:global basic ms_mon_cluster_mode crc * 2026-02-23T13:06:32.979 INFO:teuthology.orchestra.run.trial068.stdout:global basic ms_mon_service_mode crc * 2026-02-23T13:06:32.979 INFO:teuthology.orchestra.run.trial068.stdout:global basic ms_service_mode crc * 2026-02-23T13:06:32.979 INFO:teuthology.orchestra.run.trial068.stdout:global advanced osd_pool_default_erasure_code_profile plugin=isa technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2026-02-23T13:06:32.979 INFO:teuthology.orchestra.run.trial068.stdout:global advanced osd_pool_default_min_size 2 2026-02-23T13:06:32.979 INFO:teuthology.orchestra.run.trial068.stdout:global advanced osd_pool_default_pg_autoscale_mode off 2026-02-23T13:06:32.979 INFO:teuthology.orchestra.run.trial068.stdout:global advanced osd_pool_default_size 3 2026-02-23T13:06:32.979 INFO:teuthology.orchestra.run.trial068.stdout:global advanced public_network 10.20.192.0/20 * 2026-02-23T13:06:32.979 INFO:teuthology.orchestra.run.trial068.stdout:mon advanced auth_allow_insecure_global_id_reclaim false 2026-02-23T13:06:32.980 INFO:teuthology.orchestra.run.trial068.stdout:mon advanced auth_mon_ticket_ttl 660.000000 2026-02-23T13:06:32.980 INFO:teuthology.orchestra.run.trial068.stdout:mon advanced auth_service_ticket_ttl 240.000000 2026-02-23T13:06:32.980 INFO:teuthology.orchestra.run.trial068.stdout:mon advanced debug_mon 20/20 2026-02-23T13:06:32.980 INFO:teuthology.orchestra.run.trial068.stdout:mon advanced debug_ms 1/1 2026-02-23T13:06:32.980 INFO:teuthology.orchestra.run.trial068.stdout:mon advanced debug_paxos 20/20 2026-02-23T13:06:32.980 INFO:teuthology.orchestra.run.trial068.stdout:mon advanced mon_data_avail_warn 5 2026-02-23T13:06:32.980 INFO:teuthology.orchestra.run.trial068.stdout:mon advanced mon_mgr_mkfs_grace 240 2026-02-23T13:06:32.980 INFO:teuthology.orchestra.run.trial068.stdout:mon advanced mon_min_osdmap_epochs 50 2026-02-23T13:06:32.980 INFO:teuthology.orchestra.run.trial068.stdout:mon dev mon_osd_prime_pg_temp true 2026-02-23T13:06:32.980 INFO:teuthology.orchestra.run.trial068.stdout:mon advanced mon_osd_reporter_subtree_level osd 2026-02-23T13:06:32.981 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:32 trial068 ceph-mon[38805]: from='client.14186 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "1;trial068:10.20.193.68=a", "target": ["mon-mgr", ""]}]: dispatch 2026-02-23T13:06:32.981 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:32 trial068 ceph-mon[38805]: Saving service mon spec with placement trial068:10.20.193.68=a;count:1 2026-02-23T13:06:32.981 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:32 trial068 ceph-mon[38805]: Updating trial185:/etc/ceph/ceph.conf 2026-02-23T13:06:32.981 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:32 trial068 ceph-mon[38805]: Updating trial185:/var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/config/ceph.conf 2026-02-23T13:06:32.981 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:32 trial068 ceph-mon[38805]: Updating trial185:/etc/ceph/ceph.client.admin.keyring 2026-02-23T13:06:32.981 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:32 trial068 ceph-mon[38805]: Updating trial185:/var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/config/ceph.client.admin.keyring 2026-02-23T13:06:32.981 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:32 trial068 ceph-mon[38805]: Reconfiguring mon.a (unknown last config time)... 2026-02-23T13:06:32.981 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:32 trial068 ceph-mon[38805]: Reconfiguring daemon mon.a on trial068 2026-02-23T13:06:32.981 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:32 trial068 ceph-mon[38805]: from='mgr.14152 10.20.193.68:0/3466135759' entity='mgr.x' 2026-02-23T13:06:32.982 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:32 trial068 ceph-mon[38805]: from='mgr.14152 10.20.193.68:0/3466135759' entity='mgr.x' 2026-02-23T13:06:32.982 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:32 trial068 ceph-mon[38805]: from='mgr.14152 10.20.193.68:0/3466135759' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2026-02-23T13:06:32.982 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:32 trial068 ceph-mon[38805]: from='mgr.14152 10.20.193.68:0/3466135759' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-23T13:06:32.982 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:32 trial068 ceph-mon[38805]: from='mgr.14152 10.20.193.68:0/3466135759' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2026-02-23T13:06:32.982 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:32 trial068 ceph-mon[38805]: from='mgr.14152 10.20.193.68:0/3466135759' entity='mgr.x' 2026-02-23T13:06:32.982 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:32 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2668120708' entity='client.admin' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-23T13:06:32.982 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:32 trial068 ceph-mon[38805]: from='mgr.14152 10.20.193.68:0/3466135759' entity='mgr.x' 2026-02-23T13:06:32.982 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:32 trial068 ceph-mon[38805]: from='mgr.14152 10.20.193.68:0/3466135759' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2026-02-23T13:06:32.982 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:32 trial068 ceph-mon[38805]: from='mgr.14152 10.20.193.68:0/3466135759' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-23T13:06:32.982 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:32 trial068 ceph-mon[38805]: from='mgr.14152 10.20.193.68:0/3466135759' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2026-02-23T13:06:32.982 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:32 trial068 ceph-mon[38805]: from='mgr.14152 10.20.193.68:0/3466135759' entity='mgr.x' 2026-02-23T13:06:32.982 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:32 trial068 ceph-mon[38805]: from='mgr.14152 10.20.193.68:0/3466135759' entity='mgr.x' 2026-02-23T13:06:32.983 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:32 trial068 ceph-mon[38805]: from='mgr.14152 10.20.193.68:0/3466135759' entity='mgr.x' cmd={"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]} : dispatch 2026-02-23T13:06:32.983 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:32 trial068 ceph-mon[38805]: from='mgr.14152 10.20.193.68:0/3466135759' entity='mgr.x' cmd={"prefix": "mgr services"} : dispatch 2026-02-23T13:06:32.983 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:32 trial068 ceph-mon[38805]: from='mgr.14152 10.20.193.68:0/3466135759' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-23T13:06:32.983 INFO:teuthology.orchestra.run.trial068.stdout:mon advanced mon_osdmap_full_prune_interval 2 2026-02-23T13:06:32.983 INFO:teuthology.orchestra.run.trial068.stdout:mon advanced mon_osdmap_full_prune_min 15 2026-02-23T13:06:32.983 INFO:teuthology.orchestra.run.trial068.stdout:mon advanced mon_osdmap_full_prune_txsize 2 2026-02-23T13:06:32.984 INFO:teuthology.orchestra.run.trial068.stdout:mon advanced mon_reweight_min_bytes_per_osd 10 2026-02-23T13:06:32.984 INFO:teuthology.orchestra.run.trial068.stdout:mon advanced mon_reweight_min_pgs_per_osd 4 2026-02-23T13:06:32.984 INFO:teuthology.orchestra.run.trial068.stdout:mon advanced mon_warn_on_insecure_global_id_reclaim false 2026-02-23T13:06:32.984 INFO:teuthology.orchestra.run.trial068.stdout:mon advanced mon_warn_on_insecure_global_id_reclaim_allowed false 2026-02-23T13:06:32.984 INFO:teuthology.orchestra.run.trial068.stdout:mon advanced osd_pool_default_crimson true 2026-02-23T13:06:32.984 INFO:teuthology.orchestra.run.trial068.stdout:mon advanced paxos_service_trim_min 10 2026-02-23T13:06:32.984 INFO:teuthology.orchestra.run.trial068.stdout:mgr advanced debug_mgr 20/20 2026-02-23T13:06:32.984 INFO:teuthology.orchestra.run.trial068.stdout:mgr advanced debug_ms 1/1 2026-02-23T13:06:32.984 INFO:teuthology.orchestra.run.trial068.stdout:mgr advanced debug_osd 10/10 2026-02-23T13:06:32.984 INFO:teuthology.orchestra.run.trial068.stdout:mgr advanced mgr/cephadm/allow_ptrace true * 2026-02-23T13:06:32.984 INFO:teuthology.orchestra.run.trial068.stdout:mgr advanced mgr/cephadm/container_init True * 2026-02-23T13:06:32.984 INFO:teuthology.orchestra.run.trial068.stdout:mgr advanced mgr/cephadm/migration_current 8 * 2026-02-23T13:06:32.985 INFO:teuthology.orchestra.run.trial068.stdout:mgr advanced mgr/dashboard/ssl_server_port 8443 * 2026-02-23T13:06:32.985 INFO:teuthology.orchestra.run.trial068.stdout:mgr advanced mgr/orchestrator/orchestrator cephadm 2026-02-23T13:06:32.985 INFO:teuthology.orchestra.run.trial068.stdout:mgr advanced mon_reweight_min_bytes_per_osd 10 2026-02-23T13:06:32.985 INFO:teuthology.orchestra.run.trial068.stdout:mgr advanced mon_reweight_min_pgs_per_osd 4 2026-02-23T13:06:32.985 INFO:teuthology.orchestra.run.trial068.stdout:osd dev bdev_debug_aio true 2026-02-23T13:06:32.985 INFO:teuthology.orchestra.run.trial068.stdout:osd advanced crimson_cpu_num 4 * 2026-02-23T13:06:32.985 INFO:teuthology.orchestra.run.trial068.stdout:osd advanced crimson_osd_obc_lru_size 10 2026-02-23T13:06:32.985 INFO:teuthology.orchestra.run.trial068.stdout:osd advanced debug_ms 20/20 2026-02-23T13:06:32.985 INFO:teuthology.orchestra.run.trial068.stdout:osd advanced debug_osd 20/20 2026-02-23T13:06:32.985 INFO:teuthology.orchestra.run.trial068.stdout:osd advanced debug_seastore 20/20 2026-02-23T13:06:32.985 INFO:teuthology.orchestra.run.trial068.stdout:osd advanced debug_seastore_backref 20/20 2026-02-23T13:06:32.985 INFO:teuthology.orchestra.run.trial068.stdout:osd advanced debug_seastore_cache 20/20 2026-02-23T13:06:32.986 INFO:teuthology.orchestra.run.trial068.stdout:osd advanced debug_seastore_cleaner 20/20 2026-02-23T13:06:32.986 INFO:teuthology.orchestra.run.trial068.stdout:osd advanced debug_seastore_device 20/20 2026-02-23T13:06:32.986 INFO:teuthology.orchestra.run.trial068.stdout:osd advanced debug_seastore_epm 20/20 2026-02-23T13:06:32.986 INFO:teuthology.orchestra.run.trial068.stdout:osd advanced debug_seastore_fixedkv_tree 20/20 2026-02-23T13:06:32.986 INFO:teuthology.orchestra.run.trial068.stdout:osd advanced debug_seastore_journal 20/20 2026-02-23T13:06:32.986 INFO:teuthology.orchestra.run.trial068.stdout:osd advanced debug_seastore_lba 20/20 2026-02-23T13:06:32.986 INFO:teuthology.orchestra.run.trial068.stdout:osd advanced debug_seastore_odata 20/20 2026-02-23T13:06:32.986 INFO:teuthology.orchestra.run.trial068.stdout:osd advanced debug_seastore_omap 20/20 2026-02-23T13:06:32.986 INFO:teuthology.orchestra.run.trial068.stdout:osd advanced debug_seastore_onode 20/20 2026-02-23T13:06:32.986 INFO:teuthology.orchestra.run.trial068.stdout:osd advanced debug_seastore_t 20/20 2026-02-23T13:06:32.986 INFO:teuthology.orchestra.run.trial068.stdout:osd advanced debug_seastore_tm 20/20 2026-02-23T13:06:32.986 INFO:teuthology.orchestra.run.trial068.stdout:osd advanced osd_blocked_scrub_grace_period 3600 2026-02-23T13:06:32.986 INFO:teuthology.orchestra.run.trial068.stdout:osd dev osd_debug_misdirected_ops true 2026-02-23T13:06:32.987 INFO:teuthology.orchestra.run.trial068.stdout:osd dev osd_debug_op_order true 2026-02-23T13:06:32.987 INFO:teuthology.orchestra.run.trial068.stdout:osd dev osd_debug_pg_log_writeout true 2026-02-23T13:06:32.987 INFO:teuthology.orchestra.run.trial068.stdout:osd dev osd_debug_reject_backfill_probability 0.300000 2026-02-23T13:06:32.987 INFO:teuthology.orchestra.run.trial068.stdout:osd dev osd_debug_shutdown true 2026-02-23T13:06:32.987 INFO:teuthology.orchestra.run.trial068.stdout:osd dev osd_debug_verify_cached_snaps true 2026-02-23T13:06:32.987 INFO:teuthology.orchestra.run.trial068.stdout:osd dev osd_debug_verify_missing_on_start true 2026-02-23T13:06:32.987 INFO:teuthology.orchestra.run.trial068.stdout:osd dev osd_debug_verify_stray_on_activate true 2026-02-23T13:06:32.987 INFO:teuthology.orchestra.run.trial068.stdout:osd advanced osd_deep_scrub_update_digest_min_age 30 2026-02-23T13:06:32.987 INFO:teuthology.orchestra.run.trial068.stdout:osd advanced osd_delete_sleep 1.000000 2026-02-23T13:06:32.987 INFO:teuthology.orchestra.run.trial068.stdout:osd advanced osd_max_backfills 3 2026-02-23T13:06:32.987 INFO:teuthology.orchestra.run.trial068.stdout:osd advanced osd_max_markdown_count 1000 2026-02-23T13:06:32.987 INFO:teuthology.orchestra.run.trial068.stdout:osd advanced osd_mclock_profile high_recovery_ops 2026-02-23T13:06:32.988 INFO:teuthology.orchestra.run.trial068.stdout:osd dev osd_mclock_skip_benchmark true 2026-02-23T13:06:32.988 INFO:teuthology.orchestra.run.trial068.stdout:osd advanced osd_memory_target_autotune true 2026-02-23T13:06:32.988 INFO:teuthology.orchestra.run.trial068.stdout:osd advanced osd_objectstore seastore * 2026-02-23T13:06:32.988 INFO:teuthology.orchestra.run.trial068.stdout:osd advanced osd_op_queue debug_random * 2026-02-23T13:06:32.988 INFO:teuthology.orchestra.run.trial068.stdout:osd advanced osd_op_queue_cut_off debug_random * 2026-02-23T13:06:32.988 INFO:teuthology.orchestra.run.trial068.stdout:osd advanced osd_recover_clone_overlap true 2026-02-23T13:06:32.988 INFO:teuthology.orchestra.run.trial068.stdout:osd advanced osd_recovery_max_chunk 1048576 2026-02-23T13:06:32.988 INFO:teuthology.orchestra.run.trial068.stdout:osd advanced osd_scrub_load_threshold 5.000000 2026-02-23T13:06:32.988 INFO:teuthology.orchestra.run.trial068.stdout:osd advanced osd_scrub_max_interval 120.000000 2026-02-23T13:06:32.988 INFO:teuthology.orchestra.run.trial068.stdout:osd advanced osd_scrub_min_interval 60.000000 2026-02-23T13:06:32.988 INFO:teuthology.orchestra.run.trial068.stdout:osd advanced osd_shutdown_pgref_assert true 2026-02-23T13:06:32.988 INFO:teuthology.orchestra.run.trial068.stdout:osd advanced osd_snap_trim_sleep 2.000000 2026-02-23T13:06:32.988 INFO:teuthology.orchestra.run.trial068.stdout:osd advanced osd_target_transaction_size 5 2026-02-23T13:06:32.989 INFO:teuthology.orchestra.run.trial068.stdout:osd advanced seastore_cachepin_size_pershard 67108864 2026-02-23T13:06:32.989 INFO:teuthology.orchestra.run.trial068.stdout:osd dev seastore_main_device_type RANDOM_BLOCK_SSD * 2026-02-23T13:06:32.989 INFO:teuthology.orchestra.run.trial068.stdout:osd advanced seastore_max_concurrent_transactions 8 2026-02-23T13:06:32.989 INFO:teuthology.orchestra.run.trial068.stdout:client.0 advanced debug_ms 1/1 2026-02-23T13:06:32.989 INFO:teuthology.orchestra.run.trial068.stdout:client.0 advanced debug_objecter 20/20 2026-02-23T13:06:32.989 INFO:teuthology.orchestra.run.trial068.stdout:client.0 advanced debug_rados 20/20 2026-02-23T13:06:32.989 INFO:teuthology.orchestra.run.trial068.stdout:client.rgw advanced rgw_cache_enabled true 2026-02-23T13:06:32.989 INFO:teuthology.orchestra.run.trial068.stdout:client.rgw advanced rgw_enable_ops_log true 2026-02-23T13:06:32.989 INFO:teuthology.orchestra.run.trial068.stdout:client.rgw advanced rgw_enable_usage_log true 2026-02-23T13:06:33.013 INFO:tasks.cephadm:Enabling crimson flags... 2026-02-23T13:06:33.013 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd set-allow-crimson --yes-i-really-mean-it 2026-02-23T13:06:33.129 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:06:33.227 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:06:33.226+0000 7f77b3257640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:06:33.227 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:06:33.226+0000 7f77b3257640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:06:33.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:33 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mon-a[38796]: 2026-02-23T13:06:33.340+0000 7f083a491640 -1 WARNING: experimental feature 'crimson' is enabled 2026-02-23T13:06:33.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:33 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mon-a[38796]: Please be aware that this feature is experimental, untested, 2026-02-23T13:06:33.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:33 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mon-a[38796]: unsupported, and may result in data corruption, data loss, 2026-02-23T13:06:33.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:33 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mon-a[38796]: and/or irreparable damage to your cluster. Do not use 2026-02-23T13:06:33.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:33 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mon-a[38796]: feature with important data. 2026-02-23T13:06:33.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:33 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mon-a[38796]: 2026-02-23T13:06:33.936 INFO:tasks.cephadm:Deploying OSDs... 2026-02-23T13:06:33.936 DEBUG:teuthology.orchestra.run.trial068:> set -ex 2026-02-23T13:06:33.936 DEBUG:teuthology.orchestra.run.trial068:> dd if=/scratch_devs of=/dev/stdout 2026-02-23T13:06:33.954 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2026-02-23T13:06:33.954 DEBUG:teuthology.orchestra.run.trial068:> stat /dev/vg_nvme/lv_1 2026-02-23T13:06:34.011 INFO:teuthology.orchestra.run.trial068.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2026-02-23T13:06:34.011 INFO:teuthology.orchestra.run.trial068.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2026-02-23T13:06:34.011 INFO:teuthology.orchestra.run.trial068.stdout:Device: 6h/6d Inode: 903 Links: 1 2026-02-23T13:06:34.011 INFO:teuthology.orchestra.run.trial068.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2026-02-23T13:06:34.011 INFO:teuthology.orchestra.run.trial068.stdout:Context: system_u:object_r:device_t:s0 2026-02-23T13:06:34.012 INFO:teuthology.orchestra.run.trial068.stdout:Access: 2026-02-23 13:06:27.108648125 +0000 2026-02-23T13:06:34.012 INFO:teuthology.orchestra.run.trial068.stdout:Modify: 2026-02-23 13:03:15.460547070 +0000 2026-02-23T13:06:34.012 INFO:teuthology.orchestra.run.trial068.stdout:Change: 2026-02-23 13:03:15.460547070 +0000 2026-02-23T13:06:34.012 INFO:teuthology.orchestra.run.trial068.stdout: Birth: 2026-02-23 13:03:15.460547070 +0000 2026-02-23T13:06:34.012 DEBUG:teuthology.orchestra.run.trial068:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2026-02-23T13:06:34.072 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:33 trial068 ceph-mon[38805]: from='client.14192 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "1;trial068=x", "target": ["mon-mgr", ""]}]: dispatch 2026-02-23T13:06:34.073 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:33 trial068 ceph-mon[38805]: Saving service mgr spec with placement trial068=x;count:1 2026-02-23T13:06:34.073 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:33 trial068 ceph-mon[38805]: Reconfiguring mgr.x (unknown last config time)... 2026-02-23T13:06:34.073 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:33 trial068 ceph-mon[38805]: Reconfiguring daemon mgr.x on trial068 2026-02-23T13:06:34.073 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:33 trial068 ceph-mon[38805]: from='mgr.14152 10.20.193.68:0/3466135759' entity='mgr.x' 2026-02-23T13:06:34.073 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:33 trial068 ceph-mon[38805]: from='mgr.14152 10.20.193.68:0/3466135759' entity='mgr.x' 2026-02-23T13:06:34.073 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:33 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2660362852' entity='client.admin' cmd={"prefix": "config dump"} : dispatch 2026-02-23T13:06:34.073 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:33 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1927989447' entity='client.admin' cmd={"prefix": "osd set-allow-crimson", "yes_i_really_mean_it": true} : dispatch 2026-02-23T13:06:34.075 INFO:teuthology.orchestra.run.trial068.stderr:1+0 records in 2026-02-23T13:06:34.075 INFO:teuthology.orchestra.run.trial068.stderr:1+0 records out 2026-02-23T13:06:34.075 INFO:teuthology.orchestra.run.trial068.stderr:512 bytes copied, 0.000134412 s, 3.8 MB/s 2026-02-23T13:06:34.076 DEBUG:teuthology.orchestra.run.trial068:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2026-02-23T13:06:34.134 DEBUG:teuthology.orchestra.run.trial068:> stat /dev/vg_nvme/lv_2 2026-02-23T13:06:34.191 INFO:teuthology.orchestra.run.trial068.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2026-02-23T13:06:34.191 INFO:teuthology.orchestra.run.trial068.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2026-02-23T13:06:34.191 INFO:teuthology.orchestra.run.trial068.stdout:Device: 6h/6d Inode: 936 Links: 1 2026-02-23T13:06:34.191 INFO:teuthology.orchestra.run.trial068.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2026-02-23T13:06:34.191 INFO:teuthology.orchestra.run.trial068.stdout:Context: system_u:object_r:device_t:s0 2026-02-23T13:06:34.192 INFO:teuthology.orchestra.run.trial068.stdout:Access: 2026-02-23 13:06:27.137648275 +0000 2026-02-23T13:06:34.192 INFO:teuthology.orchestra.run.trial068.stdout:Modify: 2026-02-23 13:03:15.692591692 +0000 2026-02-23T13:06:34.192 INFO:teuthology.orchestra.run.trial068.stdout:Change: 2026-02-23 13:03:15.692591692 +0000 2026-02-23T13:06:34.192 INFO:teuthology.orchestra.run.trial068.stdout: Birth: 2026-02-23 13:03:15.692591692 +0000 2026-02-23T13:06:34.192 DEBUG:teuthology.orchestra.run.trial068:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2026-02-23T13:06:34.255 INFO:teuthology.orchestra.run.trial068.stderr:1+0 records in 2026-02-23T13:06:34.255 INFO:teuthology.orchestra.run.trial068.stderr:1+0 records out 2026-02-23T13:06:34.256 INFO:teuthology.orchestra.run.trial068.stderr:512 bytes copied, 0.000137108 s, 3.7 MB/s 2026-02-23T13:06:34.257 DEBUG:teuthology.orchestra.run.trial068:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2026-02-23T13:06:34.315 DEBUG:teuthology.orchestra.run.trial068:> stat /dev/vg_nvme/lv_3 2026-02-23T13:06:34.372 INFO:teuthology.orchestra.run.trial068.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2026-02-23T13:06:34.372 INFO:teuthology.orchestra.run.trial068.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2026-02-23T13:06:34.372 INFO:teuthology.orchestra.run.trial068.stdout:Device: 6h/6d Inode: 967 Links: 1 2026-02-23T13:06:34.372 INFO:teuthology.orchestra.run.trial068.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2026-02-23T13:06:34.372 INFO:teuthology.orchestra.run.trial068.stdout:Context: system_u:object_r:device_t:s0 2026-02-23T13:06:34.373 INFO:teuthology.orchestra.run.trial068.stdout:Access: 2026-02-23 13:06:27.168648434 +0000 2026-02-23T13:06:34.373 INFO:teuthology.orchestra.run.trial068.stdout:Modify: 2026-02-23 13:03:15.941639583 +0000 2026-02-23T13:06:34.373 INFO:teuthology.orchestra.run.trial068.stdout:Change: 2026-02-23 13:03:15.941639583 +0000 2026-02-23T13:06:34.373 INFO:teuthology.orchestra.run.trial068.stdout: Birth: 2026-02-23 13:03:15.941639583 +0000 2026-02-23T13:06:34.373 DEBUG:teuthology.orchestra.run.trial068:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2026-02-23T13:06:34.435 INFO:teuthology.orchestra.run.trial068.stderr:1+0 records in 2026-02-23T13:06:34.435 INFO:teuthology.orchestra.run.trial068.stderr:1+0 records out 2026-02-23T13:06:34.435 INFO:teuthology.orchestra.run.trial068.stderr:512 bytes copied, 0.00013826 s, 3.7 MB/s 2026-02-23T13:06:34.437 DEBUG:teuthology.orchestra.run.trial068:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2026-02-23T13:06:34.493 DEBUG:teuthology.orchestra.run.trial068:> stat /dev/vg_nvme/lv_4 2026-02-23T13:06:34.549 INFO:teuthology.orchestra.run.trial068.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2026-02-23T13:06:34.549 INFO:teuthology.orchestra.run.trial068.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2026-02-23T13:06:34.549 INFO:teuthology.orchestra.run.trial068.stdout:Device: 6h/6d Inode: 995 Links: 1 2026-02-23T13:06:34.549 INFO:teuthology.orchestra.run.trial068.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2026-02-23T13:06:34.550 INFO:teuthology.orchestra.run.trial068.stdout:Context: system_u:object_r:device_t:s0 2026-02-23T13:06:34.550 INFO:teuthology.orchestra.run.trial068.stdout:Access: 2026-02-23 13:06:27.198648589 +0000 2026-02-23T13:06:34.550 INFO:teuthology.orchestra.run.trial068.stdout:Modify: 2026-02-23 13:03:16.207690743 +0000 2026-02-23T13:06:34.550 INFO:teuthology.orchestra.run.trial068.stdout:Change: 2026-02-23 13:03:16.207690743 +0000 2026-02-23T13:06:34.550 INFO:teuthology.orchestra.run.trial068.stdout: Birth: 2026-02-23 13:03:16.207690743 +0000 2026-02-23T13:06:34.550 DEBUG:teuthology.orchestra.run.trial068:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2026-02-23T13:06:34.613 INFO:teuthology.orchestra.run.trial068.stderr:1+0 records in 2026-02-23T13:06:34.613 INFO:teuthology.orchestra.run.trial068.stderr:1+0 records out 2026-02-23T13:06:34.613 INFO:teuthology.orchestra.run.trial068.stderr:512 bytes copied, 0.000145663 s, 3.5 MB/s 2026-02-23T13:06:34.615 DEBUG:teuthology.orchestra.run.trial068:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2026-02-23T13:06:34.671 DEBUG:teuthology.orchestra.run.trial185:> set -ex 2026-02-23T13:06:34.671 DEBUG:teuthology.orchestra.run.trial185:> dd if=/scratch_devs of=/dev/stdout 2026-02-23T13:06:34.688 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2026-02-23T13:06:34.688 DEBUG:teuthology.orchestra.run.trial185:> stat /dev/vg_nvme/lv_1 2026-02-23T13:06:34.743 INFO:teuthology.orchestra.run.trial185.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2026-02-23T13:06:34.744 INFO:teuthology.orchestra.run.trial185.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2026-02-23T13:06:34.744 INFO:teuthology.orchestra.run.trial185.stdout:Device: 6h/6d Inode: 905 Links: 1 2026-02-23T13:06:34.744 INFO:teuthology.orchestra.run.trial185.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2026-02-23T13:06:34.744 INFO:teuthology.orchestra.run.trial185.stdout:Context: system_u:object_r:device_t:s0 2026-02-23T13:06:34.744 INFO:teuthology.orchestra.run.trial185.stdout:Access: 2026-02-23 13:06:30.904625075 +0000 2026-02-23T13:06:34.744 INFO:teuthology.orchestra.run.trial185.stdout:Modify: 2026-02-23 13:03:16.682218279 +0000 2026-02-23T13:06:34.744 INFO:teuthology.orchestra.run.trial185.stdout:Change: 2026-02-23 13:03:16.682218279 +0000 2026-02-23T13:06:34.744 INFO:teuthology.orchestra.run.trial185.stdout: Birth: 2026-02-23 13:03:16.682218279 +0000 2026-02-23T13:06:34.744 DEBUG:teuthology.orchestra.run.trial185:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2026-02-23T13:06:34.807 INFO:teuthology.orchestra.run.trial185.stderr:1+0 records in 2026-02-23T13:06:34.807 INFO:teuthology.orchestra.run.trial185.stderr:1+0 records out 2026-02-23T13:06:34.807 INFO:teuthology.orchestra.run.trial185.stderr:512 bytes copied, 0.000182989 s, 2.8 MB/s 2026-02-23T13:06:34.808 DEBUG:teuthology.orchestra.run.trial185:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2026-02-23T13:06:34.864 DEBUG:teuthology.orchestra.run.trial185:> stat /dev/vg_nvme/lv_2 2026-02-23T13:06:34.920 INFO:teuthology.orchestra.run.trial185.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2026-02-23T13:06:34.920 INFO:teuthology.orchestra.run.trial185.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2026-02-23T13:06:34.920 INFO:teuthology.orchestra.run.trial185.stdout:Device: 6h/6d Inode: 935 Links: 1 2026-02-23T13:06:34.921 INFO:teuthology.orchestra.run.trial185.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2026-02-23T13:06:34.921 INFO:teuthology.orchestra.run.trial185.stdout:Context: system_u:object_r:device_t:s0 2026-02-23T13:06:34.921 INFO:teuthology.orchestra.run.trial185.stdout:Access: 2026-02-23 13:06:30.932625828 +0000 2026-02-23T13:06:34.921 INFO:teuthology.orchestra.run.trial185.stdout:Modify: 2026-02-23 13:03:16.910216333 +0000 2026-02-23T13:06:34.921 INFO:teuthology.orchestra.run.trial185.stdout:Change: 2026-02-23 13:03:16.910216333 +0000 2026-02-23T13:06:34.921 INFO:teuthology.orchestra.run.trial185.stdout: Birth: 2026-02-23 13:03:16.910216333 +0000 2026-02-23T13:06:34.921 DEBUG:teuthology.orchestra.run.trial185:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2026-02-23T13:06:34.984 INFO:teuthology.orchestra.run.trial185.stderr:1+0 records in 2026-02-23T13:06:34.984 INFO:teuthology.orchestra.run.trial185.stderr:1+0 records out 2026-02-23T13:06:34.984 INFO:teuthology.orchestra.run.trial185.stderr:512 bytes copied, 0.000138794 s, 3.7 MB/s 2026-02-23T13:06:34.985 DEBUG:teuthology.orchestra.run.trial185:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2026-02-23T13:06:35.043 DEBUG:teuthology.orchestra.run.trial185:> stat /dev/vg_nvme/lv_3 2026-02-23T13:06:35.099 INFO:teuthology.orchestra.run.trial185.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2026-02-23T13:06:35.099 INFO:teuthology.orchestra.run.trial185.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2026-02-23T13:06:35.099 INFO:teuthology.orchestra.run.trial185.stdout:Device: 6h/6d Inode: 965 Links: 1 2026-02-23T13:06:35.099 INFO:teuthology.orchestra.run.trial185.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2026-02-23T13:06:35.099 INFO:teuthology.orchestra.run.trial185.stdout:Context: system_u:object_r:device_t:s0 2026-02-23T13:06:35.099 INFO:teuthology.orchestra.run.trial185.stdout:Access: 2026-02-23 13:06:30.959626555 +0000 2026-02-23T13:06:35.100 INFO:teuthology.orchestra.run.trial185.stdout:Modify: 2026-02-23 13:03:17.162214184 +0000 2026-02-23T13:06:35.100 INFO:teuthology.orchestra.run.trial185.stdout:Change: 2026-02-23 13:03:17.162214184 +0000 2026-02-23T13:06:35.100 INFO:teuthology.orchestra.run.trial185.stdout: Birth: 2026-02-23 13:03:17.162214184 +0000 2026-02-23T13:06:35.100 DEBUG:teuthology.orchestra.run.trial185:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2026-02-23T13:06:35.163 INFO:teuthology.orchestra.run.trial185.stderr:1+0 records in 2026-02-23T13:06:35.163 INFO:teuthology.orchestra.run.trial185.stderr:1+0 records out 2026-02-23T13:06:35.163 INFO:teuthology.orchestra.run.trial185.stderr:512 bytes copied, 0.000130449 s, 3.9 MB/s 2026-02-23T13:06:35.165 DEBUG:teuthology.orchestra.run.trial185:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2026-02-23T13:06:35.221 DEBUG:teuthology.orchestra.run.trial185:> stat /dev/vg_nvme/lv_4 2026-02-23T13:06:35.276 INFO:teuthology.orchestra.run.trial185.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2026-02-23T13:06:35.276 INFO:teuthology.orchestra.run.trial185.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2026-02-23T13:06:35.276 INFO:teuthology.orchestra.run.trial185.stdout:Device: 6h/6d Inode: 992 Links: 1 2026-02-23T13:06:35.276 INFO:teuthology.orchestra.run.trial185.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2026-02-23T13:06:35.276 INFO:teuthology.orchestra.run.trial185.stdout:Context: system_u:object_r:device_t:s0 2026-02-23T13:06:35.276 INFO:teuthology.orchestra.run.trial185.stdout:Access: 2026-02-23 13:06:30.987627310 +0000 2026-02-23T13:06:35.276 INFO:teuthology.orchestra.run.trial185.stdout:Modify: 2026-02-23 13:03:17.390212239 +0000 2026-02-23T13:06:35.276 INFO:teuthology.orchestra.run.trial185.stdout:Change: 2026-02-23 13:03:17.390212239 +0000 2026-02-23T13:06:35.276 INFO:teuthology.orchestra.run.trial185.stdout: Birth: 2026-02-23 13:03:17.390212239 +0000 2026-02-23T13:06:35.276 DEBUG:teuthology.orchestra.run.trial185:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2026-02-23T13:06:35.339 INFO:teuthology.orchestra.run.trial185.stderr:1+0 records in 2026-02-23T13:06:35.340 INFO:teuthology.orchestra.run.trial185.stderr:1+0 records out 2026-02-23T13:06:35.340 INFO:teuthology.orchestra.run.trial185.stderr:512 bytes copied, 0.000142952 s, 3.6 MB/s 2026-02-23T13:06:35.341 DEBUG:teuthology.orchestra.run.trial185:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2026-02-23T13:06:35.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:34 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1927989447' entity='client.admin' cmd='[{"prefix": "osd set-allow-crimson", "yes_i_really_mean_it": true}]': finished 2026-02-23T13:06:35.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:34 trial068 ceph-mon[38805]: osdmap e5: 0 total, 0 up, 0 in 2026-02-23T13:06:35.399 INFO:tasks.cephadm:Deploying osd.0 on trial068 with /dev/vg_nvme/lv_4... 2026-02-23T13:06:35.400 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- lvm zap /dev/vg_nvme/lv_4 2026-02-23T13:06:35.515 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:06:35.946 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T13:06:35.962 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph orch daemon add osd trial068:vg_nvme/lv_4 --skip-validation 2026-02-23T13:06:36.078 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:06:36.189 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:06:36.187+0000 7f8e8660b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:06:36.189 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:06:36.187+0000 7f8e8660b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:06:37.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:37 trial068 ceph-mon[38805]: from='client.14198 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "trial068:vg_nvme/lv_4", "skip_validation": true, "target": ["mon-mgr", ""]}]: dispatch 2026-02-23T13:06:37.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:37 trial068 ceph-mon[38805]: osd.default does not exist. Creating it now. 2026-02-23T13:06:37.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:37 trial068 ceph-mon[38805]: Creating OSDs with service ID: default on trial068:['vg_nvme/lv_4'] 2026-02-23T13:06:37.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:37 trial068 ceph-mon[38805]: from='mgr.14152 10.20.193.68:0/3466135759' entity='mgr.x' 2026-02-23T13:06:37.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:37 trial068 ceph-mon[38805]: Marking host: trial068 for OSDSpec preview refresh. 2026-02-23T13:06:37.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:37 trial068 ceph-mon[38805]: Saving service osd.default spec with placement trial068 2026-02-23T13:06:37.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:37 trial068 ceph-mon[38805]: from='mgr.14152 10.20.193.68:0/3466135759' entity='mgr.x' 2026-02-23T13:06:37.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:37 trial068 ceph-mon[38805]: from='mgr.14152 10.20.193.68:0/3466135759' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2026-02-23T13:06:37.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:37 trial068 ceph-mon[38805]: from='mgr.14152 10.20.193.68:0/3466135759' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2026-02-23T13:06:37.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:37 trial068 ceph-mon[38805]: from='mgr.14152 10.20.193.68:0/3466135759' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2026-02-23T13:06:37.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:37 trial068 ceph-mon[38805]: from='mgr.14152 10.20.193.68:0/3466135759' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-23T13:06:37.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:37 trial068 ceph-mon[38805]: from='mgr.14152 10.20.193.68:0/3466135759' entity='mgr.x' 2026-02-23T13:06:37.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:37 trial068 ceph-mon[38805]: from='mgr.14152 10.20.193.68:0/3466135759' entity='mgr.x' 2026-02-23T13:06:37.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:37 trial068 ceph-mon[38805]: from='mgr.14152 10.20.193.68:0/3466135759' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-23T13:06:37.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:37 trial068 ceph-mon[38805]: from='mgr.14152 10.20.193.68:0/3466135759' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2026-02-23T13:06:37.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:37 trial068 ceph-mon[38805]: from='mgr.14152 10.20.193.68:0/3466135759' entity='mgr.x' 2026-02-23T13:06:37.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:37 trial068 ceph-mon[38805]: from='mgr.14152 10.20.193.68:0/3466135759' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2026-02-23T13:06:37.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:37 trial068 ceph-mon[38805]: from='mgr.14152 10.20.193.68:0/3466135759' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2026-02-23T13:06:37.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:37 trial068 ceph-mon[38805]: from='mgr.14152 10.20.193.68:0/3466135759' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-23T13:06:37.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:37 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/4122944276' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "5e7d2db8-8070-4095-a077-d0f8420f7d9d"} : dispatch 2026-02-23T13:06:37.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:37 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/4122944276' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5e7d2db8-8070-4095-a077-d0f8420f7d9d"}]': finished 2026-02-23T13:06:37.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:37 trial068 ceph-mon[38805]: osdmap e6: 1 total, 0 up, 1 in 2026-02-23T13:06:37.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:37 trial068 ceph-mon[38805]: from='mgr.14152 10.20.193.68:0/3466135759' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2026-02-23T13:06:37.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:37 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1339136732' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2026-02-23T13:06:39.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:39 trial068 ceph-mon[38805]: from='mgr.14152 10.20.193.68:0/3466135759' entity='mgr.x' cmd={"prefix": "auth get", "entity": "osd.0"} : dispatch 2026-02-23T13:06:39.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:39 trial068 ceph-mon[38805]: from='mgr.14152 10.20.193.68:0/3466135759' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-23T13:06:40.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:40 trial068 ceph-mon[38805]: Deploying daemon osd.0 on trial068 2026-02-23T13:06:40.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:40 trial068 ceph-mon[38805]: from='mgr.14152 10.20.193.68:0/3466135759' entity='mgr.x' cmd={"prefix": "auth get", "entity": "osd.0"} : dispatch 2026-02-23T13:06:40.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:40 trial068 ceph-mon[38805]: from='mgr.14152 10.20.193.68:0/3466135759' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-23T13:06:40.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:40 trial068 ceph-mon[38805]: Deploying daemon osd.0 on trial068 2026-02-23T13:06:42.023 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:41 trial068 ceph-mon[38805]: from='mgr.14152 10.20.193.68:0/3466135759' entity='mgr.x' 2026-02-23T13:06:42.023 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:41 trial068 ceph-mon[38805]: from='mgr.14152 10.20.193.68:0/3466135759' entity='mgr.x' 2026-02-23T13:06:43.693 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:43 trial068 ceph-mon[38805]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2026-02-23T13:06:43.693 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:43 trial068 ceph-mon[38805]: from='mgr.14152 10.20.193.68:0/3466135759' entity='mgr.x' 2026-02-23T13:06:43.693 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:43 trial068 ceph-mon[38805]: from='mgr.14152 10.20.193.68:0/3466135759' entity='mgr.x' 2026-02-23T13:06:44.329 INFO:teuthology.orchestra.run.trial068.stdout:Created osd(s) 0 on host 'trial068' 2026-02-23T13:06:44.375 DEBUG:teuthology.orchestra.run.trial068:osd.0> sudo journalctl -f -n 0 -u ceph-5f823084-10b8-11f1-be80-d404e6e7d460@osd.0.service 2026-02-23T13:06:44.378 INFO:tasks.cephadm:Deploying osd.1 on trial068 with /dev/vg_nvme/lv_3... 2026-02-23T13:06:44.378 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- lvm zap /dev/vg_nvme/lv_3 2026-02-23T13:06:44.532 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:06:44.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:44 trial068 ceph-mon[38805]: from='mgr.14152 10.20.193.68:0/3466135759' entity='mgr.x' 2026-02-23T13:06:44.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:44 trial068 ceph-mon[38805]: from='mgr.14152 10.20.193.68:0/3466135759' entity='mgr.x' 2026-02-23T13:06:44.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:44 trial068 ceph-mon[38805]: from='mgr.14152 10.20.193.68:0/3466135759' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2026-02-23T13:06:44.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:44 trial068 ceph-mon[38805]: from='osd.0 [v2:10.20.193.68:6802/1216023594,v1:10.20.193.68:6803/1216023594]' entity='osd.0' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]} : dispatch 2026-02-23T13:06:44.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:44 trial068 ceph-mon[38805]: from='mgr.14152 10.20.193.68:0/3466135759' entity='mgr.x' 2026-02-23T13:06:44.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:44 trial068 ceph-mon[38805]: from='mgr.14152 10.20.193.68:0/3466135759' entity='mgr.x' 2026-02-23T13:06:44.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:44 trial068 ceph-mon[38805]: from='mgr.14152 10.20.193.68:0/3466135759' entity='mgr.x' 2026-02-23T13:06:44.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:44 trial068 ceph-mon[38805]: from='mgr.14152 10.20.193.68:0/3466135759' entity='mgr.x' 2026-02-23T13:06:45.612 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T13:06:45.628 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph orch daemon add osd trial068:vg_nvme/lv_3 --skip-validation 2026-02-23T13:06:45.744 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:06:45.857 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:06:45.855+0000 7ffb683ed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:06:45.857 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:06:45.856+0000 7ffb683ed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:06:45.995 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:45 trial068 ceph-mon[38805]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2026-02-23T13:06:45.995 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:45 trial068 ceph-mon[38805]: from='osd.0 [v2:10.20.193.68:6802/1216023594,v1:10.20.193.68:6803/1216023594]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2026-02-23T13:06:45.995 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:45 trial068 ceph-mon[38805]: osdmap e7: 1 total, 0 up, 1 in 2026-02-23T13:06:45.996 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:45 trial068 ceph-mon[38805]: from='mgr.14152 10.20.193.68:0/3466135759' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2026-02-23T13:06:45.996 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:45 trial068 ceph-mon[38805]: from='osd.0 [v2:10.20.193.68:6802/1216023594,v1:10.20.193.68:6803/1216023594]' entity='osd.0' cmd={"prefix": "osd crush create-or-move", "id": 0, "weight":0.6836, "args": ["host=trial068", "root=default"]} : dispatch 2026-02-23T13:06:45.996 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:45 trial068 ceph-mon[38805]: from='mgr.14152 10.20.193.68:0/3466135759' entity='mgr.x' 2026-02-23T13:06:45.996 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:45 trial068 ceph-mon[38805]: from='mgr.14152 10.20.193.68:0/3466135759' entity='mgr.x' 2026-02-23T13:06:45.996 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:45 trial068 ceph-mon[38805]: from='mgr.14152 10.20.193.68:0/3466135759' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2026-02-23T13:06:45.996 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:45 trial068 ceph-mon[38805]: from='mgr.14152 10.20.193.68:0/3466135759' entity='mgr.x' 2026-02-23T13:06:45.997 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:45 trial068 ceph-mon[38805]: from='mgr.14152 10.20.193.68:0/3466135759' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-23T13:06:45.997 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:45 trial068 ceph-mon[38805]: from='mgr.14152 10.20.193.68:0/3466135759' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2026-02-23T13:06:45.997 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:45 trial068 ceph-mon[38805]: from='mgr.14152 10.20.193.68:0/3466135759' entity='mgr.x' 2026-02-23T13:06:45.997 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:45 trial068 ceph-mon[38805]: from='mgr.14152 10.20.193.68:0/3466135759' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2026-02-23T13:06:45.997 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:45 trial068 ceph-mon[38805]: from='mgr.14152 10.20.193.68:0/3466135759' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2026-02-23T13:06:45.997 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:45 trial068 ceph-mon[38805]: from='mgr.14152 10.20.193.68:0/3466135759' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-23T13:06:45.998 INFO:journalctl@ceph.osd.0.trial068.stdout:Feb 23 13:06:45 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-0[54959]: 2026-02-23T13:06:45.748+0000 7f0a65852640 -1 osd.0 0 waiting for initial osdmap 2026-02-23T13:06:45.998 INFO:journalctl@ceph.osd.0.trial068.stdout:Feb 23 13:06:45 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-0[54959]: 2026-02-23T13:06:45.750+0000 7f0a61719640 -1 osd.0 8 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2026-02-23T13:06:46.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:46 trial068 ceph-mon[38805]: Detected new or changed devices on trial068 2026-02-23T13:06:46.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:46 trial068 ceph-mon[38805]: Adjusting osd_memory_target on trial068 to 84047M 2026-02-23T13:06:46.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:46 trial068 ceph-mon[38805]: from='osd.0 [v2:10.20.193.68:6802/1216023594,v1:10.20.193.68:6803/1216023594]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.6836, "args": ["host=trial068", "root=default"]}]': finished 2026-02-23T13:06:46.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:46 trial068 ceph-mon[38805]: osdmap e8: 1 total, 0 up, 1 in 2026-02-23T13:06:46.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:46 trial068 ceph-mon[38805]: from='mgr.14152 10.20.193.68:0/3466135759' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2026-02-23T13:06:46.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:46 trial068 ceph-mon[38805]: from='mgr.14152 10.20.193.68:0/3466135759' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2026-02-23T13:06:46.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:46 trial068 ceph-mon[38805]: from='mgr.14152 10.20.193.68:0/3466135759' entity='mgr.x' 2026-02-23T13:06:46.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:46 trial068 ceph-mon[38805]: from='mgr.14152 10.20.193.68:0/3466135759' entity='mgr.x' 2026-02-23T13:06:46.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:46 trial068 ceph-mon[38805]: from='mgr.14152 10.20.193.68:0/3466135759' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2026-02-23T13:06:46.845 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:46 trial068 ceph-mon[38805]: from='mgr.14152 10.20.193.68:0/3466135759' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2026-02-23T13:06:46.845 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:46 trial068 ceph-mon[38805]: from='mgr.14152 10.20.193.68:0/3466135759' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-23T13:06:48.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:47 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T13:06:48.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:47 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T13:06:48.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:47 trial068 ceph-mon[38805]: from='client.14208 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "trial068:vg_nvme/lv_3", "skip_validation": true, "target": ["mon-mgr", ""]}]: dispatch 2026-02-23T13:06:48.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:47 trial068 ceph-mon[38805]: osd.default does not exist. Creating it now. 2026-02-23T13:06:48.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:47 trial068 ceph-mon[38805]: Creating OSDs with service ID: default on trial068:['vg_nvme/lv_3'] 2026-02-23T13:06:48.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:47 trial068 ceph-mon[38805]: Marking host: trial068 for OSDSpec preview refresh. 2026-02-23T13:06:48.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:47 trial068 ceph-mon[38805]: Saving service osd.default spec with placement trial068 2026-02-23T13:06:48.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:47 trial068 ceph-mon[38805]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2026-02-23T13:06:48.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:47 trial068 ceph-mon[38805]: osd.0 [v2:10.20.193.68:6802/1216023594,v1:10.20.193.68:6803/1216023594] boot 2026-02-23T13:06:48.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:47 trial068 ceph-mon[38805]: osdmap e9: 1 total, 1 up, 1 in 2026-02-23T13:06:48.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:47 trial068 ceph-mon[38805]: from='mgr.14152 10.20.193.68:0/3466135759' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2026-02-23T13:06:48.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:47 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1838031167' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "c6854288-3ff8-4295-80a0-8fcaafedb28f"} : dispatch 2026-02-23T13:06:48.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:47 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1838031167' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c6854288-3ff8-4295-80a0-8fcaafedb28f"}]': finished 2026-02-23T13:06:48.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:47 trial068 ceph-mon[38805]: osdmap e10: 2 total, 1 up, 2 in 2026-02-23T13:06:48.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:47 trial068 ceph-mon[38805]: from='mgr.14152 10.20.193.68:0/3466135759' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2026-02-23T13:06:48.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:47 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/14294090' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2026-02-23T13:06:49.999 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:49 trial068 ceph-mon[38805]: pgmap v13: 0 pgs: ; 0 B data, 26 MiB used, 700 GiB / 700 GiB avail 2026-02-23T13:06:50.000 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:49 trial068 ceph-mon[38805]: from='mgr.14152 10.20.193.68:0/3466135759' entity='mgr.x' 2026-02-23T13:06:50.000 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:49 trial068 ceph-mon[38805]: from='mgr.14152 10.20.193.68:0/3466135759' entity='mgr.x' 2026-02-23T13:06:50.000 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:49 trial068 ceph-mon[38805]: from='mgr.14152 10.20.193.68:0/3466135759' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2026-02-23T13:06:50.000 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:49 trial068 ceph-mon[38805]: from='mgr.14152 10.20.193.68:0/3466135759' entity='mgr.x' cmd={"prefix": "auth get", "entity": "osd.1"} : dispatch 2026-02-23T13:06:50.000 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:49 trial068 ceph-mon[38805]: from='mgr.14152 10.20.193.68:0/3466135759' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-23T13:06:50.000 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:49 trial068 ceph-mon[38805]: from='mgr.14152 10.20.193.68:0/3466135759' entity='mgr.x' 2026-02-23T13:06:50.000 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:49 trial068 ceph-mon[38805]: from='mgr.14152 10.20.193.68:0/3466135759' entity='mgr.x' 2026-02-23T13:06:50.961 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:50 trial068 ceph-mon[38805]: Deploying daemon osd.1 on trial068 2026-02-23T13:06:51.944 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:51 trial068 ceph-mon[38805]: pgmap v14: 0 pgs: ; 0 B data, 26 MiB used, 700 GiB / 700 GiB avail 2026-02-23T13:06:51.944 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:51 trial068 ceph-mon[38805]: from='mgr.14152 10.20.193.68:0/3466135759' entity='mgr.x' 2026-02-23T13:06:51.944 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:51 trial068 ceph-mon[38805]: from='mgr.14152 10.20.193.68:0/3466135759' entity='mgr.x' 2026-02-23T13:06:51.944 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:51 trial068 ceph-mon[38805]: from='osd.1 [v2:10.20.193.68:6810/1086778192,v1:10.20.193.68:6811/1086778192]' entity='osd.1' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]} : dispatch 2026-02-23T13:06:52.327 INFO:teuthology.orchestra.run.trial068.stdout:Created osd(s) 1 on host 'trial068' 2026-02-23T13:06:52.367 DEBUG:teuthology.orchestra.run.trial068:osd.1> sudo journalctl -f -n 0 -u ceph-5f823084-10b8-11f1-be80-d404e6e7d460@osd.1.service 2026-02-23T13:06:52.369 INFO:tasks.cephadm:Deploying osd.2 on trial185 with /dev/vg_nvme/lv_4... 2026-02-23T13:06:52.369 DEBUG:teuthology.orchestra.run.trial185:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- lvm zap /dev/vg_nvme/lv_4 2026-02-23T13:06:52.485 INFO:teuthology.orchestra.run.trial185.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/config/ceph.conf 2026-02-23T13:06:53.014 INFO:teuthology.orchestra.run.trial185.stdout: 2026-02-23T13:06:53.030 DEBUG:teuthology.orchestra.run.trial185:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph orch daemon add osd trial185:vg_nvme/lv_4 --skip-validation 2026-02-23T13:06:53.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:52 trial068 ceph-mon[38805]: Detected new or changed devices on trial068 2026-02-23T13:06:53.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:52 trial068 ceph-mon[38805]: from='mgr.14152 10.20.193.68:0/3466135759' entity='mgr.x' 2026-02-23T13:06:53.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:52 trial068 ceph-mon[38805]: from='mgr.14152 10.20.193.68:0/3466135759' entity='mgr.x' 2026-02-23T13:06:53.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:52 trial068 ceph-mon[38805]: from='mgr.14152 10.20.193.68:0/3466135759' entity='mgr.x' 2026-02-23T13:06:53.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:52 trial068 ceph-mon[38805]: from='mgr.14152 10.20.193.68:0/3466135759' entity='mgr.x' 2026-02-23T13:06:53.095 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:52 trial068 ceph-mon[38805]: from='mgr.14152 10.20.193.68:0/3466135759' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-23T13:06:53.095 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:52 trial068 ceph-mon[38805]: from='mgr.14152 10.20.193.68:0/3466135759' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2026-02-23T13:06:53.095 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:52 trial068 ceph-mon[38805]: from='mgr.14152 10.20.193.68:0/3466135759' entity='mgr.x' 2026-02-23T13:06:53.095 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:52 trial068 ceph-mon[38805]: from='mgr.14152 10.20.193.68:0/3466135759' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2026-02-23T13:06:53.095 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:52 trial068 ceph-mon[38805]: from='mgr.14152 10.20.193.68:0/3466135759' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2026-02-23T13:06:53.095 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:52 trial068 ceph-mon[38805]: from='mgr.14152 10.20.193.68:0/3466135759' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-23T13:06:53.095 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:52 trial068 ceph-mon[38805]: from='osd.1 [v2:10.20.193.68:6810/1086778192,v1:10.20.193.68:6811/1086778192]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2026-02-23T13:06:53.096 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:52 trial068 ceph-mon[38805]: osdmap e11: 2 total, 1 up, 2 in 2026-02-23T13:06:53.096 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:52 trial068 ceph-mon[38805]: from='mgr.14152 10.20.193.68:0/3466135759' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2026-02-23T13:06:53.096 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:52 trial068 ceph-mon[38805]: from='osd.1 [v2:10.20.193.68:6810/1086778192,v1:10.20.193.68:6811/1086778192]' entity='osd.1' cmd={"prefix": "osd crush create-or-move", "id": 1, "weight":0.6836, "args": ["host=trial068", "root=default"]} : dispatch 2026-02-23T13:06:53.096 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:52 trial068 ceph-mon[38805]: from='mgr.14152 10.20.193.68:0/3466135759' entity='mgr.x' 2026-02-23T13:06:53.096 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:52 trial068 ceph-mon[38805]: from='mgr.14152 10.20.193.68:0/3466135759' entity='mgr.x' 2026-02-23T13:06:53.096 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:52 trial068 ceph-mon[38805]: pgmap v16: 0 pgs: ; 0 B data, 26 MiB used, 700 GiB / 700 GiB avail 2026-02-23T13:06:53.097 INFO:journalctl@ceph.osd.1.trial068.stdout:Feb 23 13:06:53 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-1[62702]: 2026-02-23T13:06:53.001+0000 7f72d9bc9640 -1 osd.1 0 waiting for initial osdmap 2026-02-23T13:06:53.097 INFO:journalctl@ceph.osd.1.trial068.stdout:Feb 23 13:06:53 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-1[62702]: 2026-02-23T13:06:53.003+0000 7f72d5a90640 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2026-02-23T13:06:53.180 INFO:teuthology.orchestra.run.trial185.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/config/ceph.conf 2026-02-23T13:06:53.303 INFO:teuthology.orchestra.run.trial185.stderr:2026-02-23T13:06:53.302+0000 7f66d99dc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:06:53.303 INFO:teuthology.orchestra.run.trial185.stderr:2026-02-23T13:06:53.303+0000 7f66d99dc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:06:54.322 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:54 trial068 ceph-mon[38805]: from='osd.1 [v2:10.20.193.68:6810/1086778192,v1:10.20.193.68:6811/1086778192]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.6836, "args": ["host=trial068", "root=default"]}]': finished 2026-02-23T13:06:54.322 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:54 trial068 ceph-mon[38805]: osdmap e12: 2 total, 1 up, 2 in 2026-02-23T13:06:54.322 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:54 trial068 ceph-mon[38805]: from='mgr.14152 10.20.193.68:0/3466135759' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2026-02-23T13:06:54.322 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:54 trial068 ceph-mon[38805]: from='mgr.14152 10.20.193.68:0/3466135759' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2026-02-23T13:06:54.322 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:54 trial068 ceph-mon[38805]: from='client.14217 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "trial185:vg_nvme/lv_4", "skip_validation": true, "target": ["mon-mgr", ""]}]: dispatch 2026-02-23T13:06:54.322 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:54 trial068 ceph-mon[38805]: osd.default does not exist. Creating it now. 2026-02-23T13:06:54.322 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:54 trial068 ceph-mon[38805]: Creating OSDs with service ID: default on trial185:['vg_nvme/lv_4'] 2026-02-23T13:06:54.322 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:54 trial068 ceph-mon[38805]: from='mgr.14152 10.20.193.68:0/3466135759' entity='mgr.x' 2026-02-23T13:06:54.322 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:54 trial068 ceph-mon[38805]: Marking host: trial185 for OSDSpec preview refresh. 2026-02-23T13:06:54.322 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:54 trial068 ceph-mon[38805]: Saving service osd.default spec with placement trial185 2026-02-23T13:06:54.322 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:54 trial068 ceph-mon[38805]: from='mgr.14152 10.20.193.68:0/3466135759' entity='mgr.x' 2026-02-23T13:06:54.323 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:54 trial068 ceph-mon[38805]: from='mgr.14152 10.20.193.68:0/3466135759' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2026-02-23T13:06:54.323 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:54 trial068 ceph-mon[38805]: from='mgr.14152 10.20.193.68:0/3466135759' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2026-02-23T13:06:54.323 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:54 trial068 ceph-mon[38805]: from='mgr.14152 10.20.193.68:0/3466135759' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-23T13:06:54.323 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:54 trial068 ceph-mon[38805]: from='mgr.14152 10.20.193.68:0/3466135759' entity='mgr.x' 2026-02-23T13:06:54.323 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:54 trial068 ceph-mon[38805]: from='mgr.14152 10.20.193.68:0/3466135759' entity='mgr.x' 2026-02-23T13:06:54.323 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:54 trial068 ceph-mon[38805]: from='mgr.14152 10.20.193.68:0/3466135759' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2026-02-23T13:06:54.323 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:54 trial068 ceph-mon[38805]: from='mgr.14152 10.20.193.68:0/3466135759' entity='mgr.x' 2026-02-23T13:06:54.323 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:54 trial068 ceph-mon[38805]: from='mgr.14152 10.20.193.68:0/3466135759' entity='mgr.x' 2026-02-23T13:06:55.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:55 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T13:06:55.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:55 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T13:06:55.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:55 trial068 ceph-mon[38805]: osd.1 [v2:10.20.193.68:6810/1086778192,v1:10.20.193.68:6811/1086778192] boot 2026-02-23T13:06:55.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:55 trial068 ceph-mon[38805]: osdmap e13: 2 total, 2 up, 2 in 2026-02-23T13:06:55.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:55 trial068 ceph-mon[38805]: from='mgr.14152 10.20.193.68:0/3466135759' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2026-02-23T13:06:55.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:55 trial068 ceph-mon[38805]: from='client.? 10.20.193.185:0/2828895033' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "25faf9eb-f475-49fe-8eeb-0663163402e8"} : dispatch 2026-02-23T13:06:55.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:55 trial068 ceph-mon[38805]: from='client.? 10.20.193.185:0/2828895033' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "25faf9eb-f475-49fe-8eeb-0663163402e8"}]': finished 2026-02-23T13:06:55.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:55 trial068 ceph-mon[38805]: osdmap e14: 3 total, 2 up, 3 in 2026-02-23T13:06:55.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:55 trial068 ceph-mon[38805]: from='mgr.14152 10.20.193.68:0/3466135759' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2026-02-23T13:06:55.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:55 trial068 ceph-mon[38805]: from='mgr.14152 10.20.193.68:0/3466135759' entity='mgr.x' 2026-02-23T13:06:55.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:55 trial068 ceph-mon[38805]: from='mgr.14152 10.20.193.68:0/3466135759' entity='mgr.x' 2026-02-23T13:06:55.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:55 trial068 ceph-mon[38805]: from='client.? 10.20.193.185:0/1269908645' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2026-02-23T13:06:55.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:55 trial068 ceph-mon[38805]: pgmap v20: 0 pgs: ; 0 B data, 53 MiB used, 1.4 TiB / 1.4 TiB avail 2026-02-23T13:06:56.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:56 trial068 ceph-mon[38805]: from='mgr.14152 10.20.193.68:0/3466135759' entity='mgr.x' 2026-02-23T13:06:56.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:56 trial068 ceph-mon[38805]: from='mgr.14152 10.20.193.68:0/3466135759' entity='mgr.x' 2026-02-23T13:06:56.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:56 trial068 ceph-mon[38805]: from='mgr.14152 10.20.193.68:0/3466135759' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2026-02-23T13:06:56.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:56 trial068 ceph-mon[38805]: from='mgr.14152 10.20.193.68:0/3466135759' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2026-02-23T13:06:56.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:56 trial068 ceph-mon[38805]: Adjusting osd_memory_target on trial068 to 42023M 2026-02-23T13:06:56.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:56 trial068 ceph-mon[38805]: from='mgr.14152 10.20.193.68:0/3466135759' entity='mgr.x' 2026-02-23T13:06:56.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:56 trial068 ceph-mon[38805]: from='mgr.14152 10.20.193.68:0/3466135759' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-23T13:06:56.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:56 trial068 ceph-mon[38805]: from='mgr.14152 10.20.193.68:0/3466135759' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2026-02-23T13:06:56.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:56 trial068 ceph-mon[38805]: from='mgr.14152 10.20.193.68:0/3466135759' entity='mgr.x' 2026-02-23T13:06:56.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:56 trial068 ceph-mon[38805]: from='mgr.14152 10.20.193.68:0/3466135759' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2026-02-23T13:06:56.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:56 trial068 ceph-mon[38805]: from='mgr.14152 10.20.193.68:0/3466135759' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2026-02-23T13:06:56.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:56 trial068 ceph-mon[38805]: from='mgr.14152 10.20.193.68:0/3466135759' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-23T13:06:56.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:56 trial068 ceph-mon[38805]: osdmap e15: 3 total, 2 up, 3 in 2026-02-23T13:06:56.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:56 trial068 ceph-mon[38805]: from='mgr.14152 10.20.193.68:0/3466135759' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2026-02-23T13:06:57.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:57 trial068 ceph-mon[38805]: from='mgr.14152 10.20.193.68:0/3466135759' entity='mgr.x' cmd={"prefix": "auth get", "entity": "osd.2"} : dispatch 2026-02-23T13:06:57.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:57 trial068 ceph-mon[38805]: from='mgr.14152 10.20.193.68:0/3466135759' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-23T13:06:57.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:57 trial068 ceph-mon[38805]: Deploying daemon osd.2 on trial185 2026-02-23T13:06:57.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:57 trial068 ceph-mon[38805]: from='mgr.14152 10.20.193.68:0/3466135759' entity='mgr.x' cmd={"prefix": "auth get", "entity": "osd.2"} : dispatch 2026-02-23T13:06:57.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:57 trial068 ceph-mon[38805]: from='mgr.14152 10.20.193.68:0/3466135759' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-23T13:06:57.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:57 trial068 ceph-mon[38805]: Deploying daemon osd.2 on trial185 2026-02-23T13:06:57.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:57 trial068 ceph-mon[38805]: pgmap v22: 0 pgs: ; 0 B data, 53 MiB used, 1.4 TiB / 1.4 TiB avail 2026-02-23T13:06:59.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:59 trial068 ceph-mon[38805]: from='mgr.14152 10.20.193.68:0/3466135759' entity='mgr.x' 2026-02-23T13:06:59.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:59 trial068 ceph-mon[38805]: from='mgr.14152 10.20.193.68:0/3466135759' entity='mgr.x' 2026-02-23T13:06:59.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:06:59 trial068 ceph-mon[38805]: pgmap v23: 0 pgs: ; 0 B data, 53 MiB used, 1.4 TiB / 1.4 TiB avail 2026-02-23T13:07:00.679 INFO:teuthology.orchestra.run.trial185.stdout:Created osd(s) 2 on host 'trial185' 2026-02-23T13:07:00.719 DEBUG:teuthology.orchestra.run.trial185:osd.2> sudo journalctl -f -n 0 -u ceph-5f823084-10b8-11f1-be80-d404e6e7d460@osd.2.service 2026-02-23T13:07:00.722 INFO:tasks.cephadm:Deploying osd.3 on trial185 with /dev/vg_nvme/lv_3... 2026-02-23T13:07:00.722 DEBUG:teuthology.orchestra.run.trial185:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- lvm zap /dev/vg_nvme/lv_3 2026-02-23T13:07:00.883 INFO:teuthology.orchestra.run.trial185.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/config/ceph.conf 2026-02-23T13:07:00.905 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:07:00 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-2[45891]: 2026-02-23T13:07:00.793+0000 7fd5bc8dd780 -1 osd.2 0 log_to_monitors true 2026-02-23T13:07:01.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:01 trial068 ceph-mon[38805]: from='mgr.14152 10.20.193.68:0/3466135759' entity='mgr.x' 2026-02-23T13:07:01.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:01 trial068 ceph-mon[38805]: from='mgr.14152 10.20.193.68:0/3466135759' entity='mgr.x' 2026-02-23T13:07:01.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:01 trial068 ceph-mon[38805]: pgmap v24: 0 pgs: ; 0 B data, 53 MiB used, 1.4 TiB / 1.4 TiB avail 2026-02-23T13:07:01.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:01 trial068 ceph-mon[38805]: from='mgr.14152 10.20.193.68:0/3466135759' entity='mgr.x' 2026-02-23T13:07:01.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:01 trial068 ceph-mon[38805]: from='mgr.14152 10.20.193.68:0/3466135759' entity='mgr.x' 2026-02-23T13:07:01.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:01 trial068 ceph-mon[38805]: from='osd.2 [v2:10.20.193.185:6800/1978920641,v1:10.20.193.185:6801/1978920641]' entity='osd.2' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]} : dispatch 2026-02-23T13:07:01.654 INFO:teuthology.orchestra.run.trial185.stdout: 2026-02-23T13:07:01.673 DEBUG:teuthology.orchestra.run.trial185:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph orch daemon add osd trial185:vg_nvme/lv_3 --skip-validation 2026-02-23T13:07:01.792 INFO:teuthology.orchestra.run.trial185.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/config/ceph.conf 2026-02-23T13:07:01.901 INFO:teuthology.orchestra.run.trial185.stderr:2026-02-23T13:07:01.899+0000 7fda93f4d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:07:01.901 INFO:teuthology.orchestra.run.trial185.stderr:2026-02-23T13:07:01.900+0000 7fda93f4d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:07:02.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:02 trial068 ceph-mon[38805]: from='mgr.14152 10.20.193.68:0/3466135759' entity='mgr.x' 2026-02-23T13:07:02.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:02 trial068 ceph-mon[38805]: from='mgr.14152 10.20.193.68:0/3466135759' entity='mgr.x' 2026-02-23T13:07:02.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:02 trial068 ceph-mon[38805]: from='mgr.14152 10.20.193.68:0/3466135759' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2026-02-23T13:07:02.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:02 trial068 ceph-mon[38805]: from='mgr.14152 10.20.193.68:0/3466135759' entity='mgr.x' 2026-02-23T13:07:02.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:02 trial068 ceph-mon[38805]: from='mgr.14152 10.20.193.68:0/3466135759' entity='mgr.x' 2026-02-23T13:07:02.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:02 trial068 ceph-mon[38805]: from='osd.2 [v2:10.20.193.185:6800/1978920641,v1:10.20.193.185:6801/1978920641]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2026-02-23T13:07:02.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:02 trial068 ceph-mon[38805]: osdmap e16: 3 total, 2 up, 3 in 2026-02-23T13:07:02.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:02 trial068 ceph-mon[38805]: from='mgr.14152 10.20.193.68:0/3466135759' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2026-02-23T13:07:02.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:02 trial068 ceph-mon[38805]: from='osd.2 [v2:10.20.193.185:6800/1978920641,v1:10.20.193.185:6801/1978920641]' entity='osd.2' cmd={"prefix": "osd crush create-or-move", "id": 2, "weight":0.6836, "args": ["host=trial185", "root=default"]} : dispatch 2026-02-23T13:07:02.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:02 trial068 ceph-mon[38805]: from='mgr.14152 10.20.193.68:0/3466135759' entity='mgr.x' 2026-02-23T13:07:02.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:02 trial068 ceph-mon[38805]: from='mgr.14152 10.20.193.68:0/3466135759' entity='mgr.x' 2026-02-23T13:07:02.595 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:02 trial068 ceph-mon[38805]: from='mgr.14152 10.20.193.68:0/3466135759' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2026-02-23T13:07:02.595 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:02 trial068 ceph-mon[38805]: from='mgr.14152 10.20.193.68:0/3466135759' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2026-02-23T13:07:02.595 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:02 trial068 ceph-mon[38805]: from='mgr.14152 10.20.193.68:0/3466135759' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-23T13:07:02.825 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:07:02 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-2[45891]: 2026-02-23T13:07:02.681+0000 7fd5b885f640 -1 osd.2 0 waiting for initial osdmap 2026-02-23T13:07:02.825 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:07:02 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-2[45891]: 2026-02-23T13:07:02.683+0000 7fd5b071e640 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2026-02-23T13:07:04.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:03 trial068 ceph-mon[38805]: from='client.14227 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "trial185:vg_nvme/lv_3", "skip_validation": true, "target": ["mon-mgr", ""]}]: dispatch 2026-02-23T13:07:04.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:03 trial068 ceph-mon[38805]: osd.default does not exist. Creating it now. 2026-02-23T13:07:04.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:03 trial068 ceph-mon[38805]: Creating OSDs with service ID: default on trial185:['vg_nvme/lv_3'] 2026-02-23T13:07:04.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:03 trial068 ceph-mon[38805]: Marking host: trial185 for OSDSpec preview refresh. 2026-02-23T13:07:04.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:03 trial068 ceph-mon[38805]: Saving service osd.default spec with placement trial185 2026-02-23T13:07:04.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:03 trial068 ceph-mon[38805]: pgmap v26: 0 pgs: ; 0 B data, 53 MiB used, 1.4 TiB / 1.4 TiB avail 2026-02-23T13:07:04.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:03 trial068 ceph-mon[38805]: Detected new or changed devices on trial185 2026-02-23T13:07:04.095 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:03 trial068 ceph-mon[38805]: from='mgr.14152 10.20.193.68:0/3466135759' entity='mgr.x' 2026-02-23T13:07:04.095 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:03 trial068 ceph-mon[38805]: from='mgr.14152 10.20.193.68:0/3466135759' entity='mgr.x' 2026-02-23T13:07:04.095 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:03 trial068 ceph-mon[38805]: from='mgr.14152 10.20.193.68:0/3466135759' entity='mgr.x' 2026-02-23T13:07:04.095 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:03 trial068 ceph-mon[38805]: from='mgr.14152 10.20.193.68:0/3466135759' entity='mgr.x' 2026-02-23T13:07:04.095 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:03 trial068 ceph-mon[38805]: from='mgr.14152 10.20.193.68:0/3466135759' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"} : dispatch 2026-02-23T13:07:04.095 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:03 trial068 ceph-mon[38805]: Adjusting osd_memory_target on trial185 to 89167M 2026-02-23T13:07:04.095 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:03 trial068 ceph-mon[38805]: from='mgr.14152 10.20.193.68:0/3466135759' entity='mgr.x' 2026-02-23T13:07:04.095 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:03 trial068 ceph-mon[38805]: from='mgr.14152 10.20.193.68:0/3466135759' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-23T13:07:04.095 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:03 trial068 ceph-mon[38805]: from='mgr.14152 10.20.193.68:0/3466135759' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2026-02-23T13:07:04.095 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:03 trial068 ceph-mon[38805]: from='mgr.14152 10.20.193.68:0/3466135759' entity='mgr.x' 2026-02-23T13:07:04.095 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:03 trial068 ceph-mon[38805]: from='mgr.14152 10.20.193.68:0/3466135759' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2026-02-23T13:07:04.095 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:03 trial068 ceph-mon[38805]: from='mgr.14152 10.20.193.68:0/3466135759' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2026-02-23T13:07:04.096 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:03 trial068 ceph-mon[38805]: from='mgr.14152 10.20.193.68:0/3466135759' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-23T13:07:04.096 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:03 trial068 ceph-mon[38805]: from='osd.2 [v2:10.20.193.185:6800/1978920641,v1:10.20.193.185:6801/1978920641]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.6836, "args": ["host=trial185", "root=default"]}]': finished 2026-02-23T13:07:04.096 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:03 trial068 ceph-mon[38805]: osdmap e17: 3 total, 2 up, 3 in 2026-02-23T13:07:04.096 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:03 trial068 ceph-mon[38805]: from='mgr.14152 10.20.193.68:0/3466135759' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2026-02-23T13:07:04.096 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:03 trial068 ceph-mon[38805]: from='mgr.14152 10.20.193.68:0/3466135759' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2026-02-23T13:07:04.096 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:03 trial068 ceph-mon[38805]: from='client.? 10.20.193.185:0/1782739403' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "e606eee6-fc74-4c50-bfdc-a6a2adf9a945"} : dispatch 2026-02-23T13:07:04.096 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:03 trial068 ceph-mon[38805]: from='client.? 10.20.193.185:0/1782739403' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e606eee6-fc74-4c50-bfdc-a6a2adf9a945"}]': finished 2026-02-23T13:07:04.096 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:03 trial068 ceph-mon[38805]: osd.2 [v2:10.20.193.185:6800/1978920641,v1:10.20.193.185:6801/1978920641] boot 2026-02-23T13:07:04.096 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:03 trial068 ceph-mon[38805]: osdmap e18: 4 total, 3 up, 4 in 2026-02-23T13:07:04.096 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:03 trial068 ceph-mon[38805]: from='mgr.14152 10.20.193.68:0/3466135759' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2026-02-23T13:07:04.096 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:03 trial068 ceph-mon[38805]: from='mgr.14152 10.20.193.68:0/3466135759' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2026-02-23T13:07:04.096 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:03 trial068 ceph-mon[38805]: from='client.? 10.20.193.185:0/3194590679' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2026-02-23T13:07:05.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:04 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T13:07:05.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:04 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T13:07:06.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:05 trial068 ceph-mon[38805]: pgmap v29: 0 pgs: ; 0 B data, 79 MiB used, 2.1 TiB / 2.1 TiB avail 2026-02-23T13:07:06.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:05 trial068 ceph-mon[38805]: osdmap e19: 4 total, 3 up, 4 in 2026-02-23T13:07:06.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:05 trial068 ceph-mon[38805]: from='mgr.14152 10.20.193.68:0/3466135759' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2026-02-23T13:07:06.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:05 trial068 ceph-mon[38805]: from='mgr.14152 10.20.193.68:0/3466135759' entity='mgr.x' cmd={"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true} : dispatch 2026-02-23T13:07:06.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:05 trial068 ceph-mon[38805]: from='mgr.14152 10.20.193.68:0/3466135759' entity='mgr.x' cmd={"prefix": "auth get", "entity": "osd.3"} : dispatch 2026-02-23T13:07:06.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:05 trial068 ceph-mon[38805]: from='mgr.14152 10.20.193.68:0/3466135759' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-23T13:07:06.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:05 trial068 ceph-mon[38805]: from='mgr.14152 10.20.193.68:0/3466135759' entity='mgr.x' cmd={"prefix": "auth get", "entity": "osd.3"} : dispatch 2026-02-23T13:07:06.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:05 trial068 ceph-mon[38805]: from='mgr.14152 10.20.193.68:0/3466135759' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-23T13:07:06.881 INFO:journalctl@ceph.osd.1.trial068.stdout:Feb 23 13:07:06 trial068 sudo[70739]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme2n1 2026-02-23T13:07:06.881 INFO:journalctl@ceph.osd.1.trial068.stdout:Feb 23 13:07:06 trial068 sudo[70739]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2026-02-23T13:07:06.881 INFO:journalctl@ceph.osd.1.trial068.stdout:Feb 23 13:07:06 trial068 sudo[70739]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2026-02-23T13:07:06.881 INFO:journalctl@ceph.osd.1.trial068.stdout:Feb 23 13:07:06 trial068 sudo[70739]: pam_unix(sudo:session): session closed for user root 2026-02-23T13:07:06.881 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:06 trial068 ceph-mon[38805]: Deploying daemon osd.3 on trial185 2026-02-23T13:07:06.882 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:06 trial068 ceph-mon[38805]: Deploying daemon osd.3 on trial185 2026-02-23T13:07:06.882 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:06 trial068 ceph-mon[38805]: from='mgr.14152 10.20.193.68:0/3466135759' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2026-02-23T13:07:06.882 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:06 trial068 ceph-mon[38805]: osdmap e20: 4 total, 3 up, 4 in 2026-02-23T13:07:06.882 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:06 trial068 ceph-mon[38805]: from='mgr.14152 10.20.193.68:0/3466135759' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2026-02-23T13:07:06.882 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:06 trial068 ceph-mon[38805]: from='mgr.14152 10.20.193.68:0/3466135759' entity='mgr.x' cmd={"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true} : dispatch 2026-02-23T13:07:06.882 INFO:journalctl@ceph.osd.0.trial068.stdout:Feb 23 13:07:06 trial068 sudo[70731]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme2n1 2026-02-23T13:07:06.882 INFO:journalctl@ceph.osd.0.trial068.stdout:Feb 23 13:07:06 trial068 sudo[70731]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2026-02-23T13:07:06.882 INFO:journalctl@ceph.osd.0.trial068.stdout:Feb 23 13:07:06 trial068 sudo[70731]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2026-02-23T13:07:06.883 INFO:journalctl@ceph.osd.0.trial068.stdout:Feb 23 13:07:06 trial068 sudo[70731]: pam_unix(sudo:session): session closed for user root 2026-02-23T13:07:06.883 INFO:journalctl@ceph.osd.0.trial068.stdout:Feb 23 13:07:06 trial068 sudo[70735]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme mtfdkcc1t9tgp-1bk1jabyy smart-log-add --json /dev/nvme2n1 2026-02-23T13:07:06.883 INFO:journalctl@ceph.osd.0.trial068.stdout:Feb 23 13:07:06 trial068 sudo[70735]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2026-02-23T13:07:06.883 INFO:journalctl@ceph.osd.0.trial068.stdout:Feb 23 13:07:06 trial068 sudo[70735]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2026-02-23T13:07:06.883 INFO:journalctl@ceph.osd.0.trial068.stdout:Feb 23 13:07:06 trial068 sudo[70735]: pam_unix(sudo:session): session closed for user root 2026-02-23T13:07:07.067 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:07:06 trial185 sudo[53145]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme2n1 2026-02-23T13:07:07.068 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:07:06 trial185 sudo[53145]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2026-02-23T13:07:07.068 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:07:06 trial185 sudo[53145]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2026-02-23T13:07:07.068 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:07:06 trial185 sudo[53145]: pam_unix(sudo:session): session closed for user root 2026-02-23T13:07:07.068 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:07:06 trial185 sudo[53170]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme mtfdkcc1t9tgp-1bk1jabyy smart-log-add --json /dev/nvme2n1 2026-02-23T13:07:07.068 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:07:06 trial185 sudo[53170]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2026-02-23T13:07:07.068 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:07:06 trial185 sudo[53170]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2026-02-23T13:07:07.068 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:07:06 trial185 sudo[53170]: pam_unix(sudo:session): session closed for user root 2026-02-23T13:07:07.343 INFO:journalctl@ceph.osd.1.trial068.stdout:Feb 23 13:07:06 trial068 sudo[70743]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme mtfdkcc1t9tgp-1bk1jabyy smart-log-add --json /dev/nvme2n1 2026-02-23T13:07:07.344 INFO:journalctl@ceph.osd.1.trial068.stdout:Feb 23 13:07:06 trial068 sudo[70743]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2026-02-23T13:07:07.344 INFO:journalctl@ceph.osd.1.trial068.stdout:Feb 23 13:07:06 trial068 sudo[70743]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2026-02-23T13:07:07.344 INFO:journalctl@ceph.osd.1.trial068.stdout:Feb 23 13:07:06 trial068 sudo[70743]: pam_unix(sudo:session): session closed for user root 2026-02-23T13:07:07.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:06 trial068 sudo[70747]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme1n1 2026-02-23T13:07:07.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:06 trial068 sudo[70747]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2026-02-23T13:07:07.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:06 trial068 sudo[70747]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2026-02-23T13:07:07.345 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:06 trial068 sudo[70747]: pam_unix(sudo:session): session closed for user root 2026-02-23T13:07:07.345 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:06 trial068 sudo[70751]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme mtfdkcc1t9tgp-1bk1jabyy smart-log-add --json /dev/nvme1n1 2026-02-23T13:07:07.345 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:06 trial068 sudo[70751]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2026-02-23T13:07:07.345 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:06 trial068 sudo[70751]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2026-02-23T13:07:07.345 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:06 trial068 sudo[70751]: pam_unix(sudo:session): session closed for user root 2026-02-23T13:07:08.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:07 trial068 ceph-mon[38805]: pgmap v32: 1 pgs: 1 unknown; 0 B data, 79 MiB used, 2.1 TiB / 2.1 TiB avail 2026-02-23T13:07:08.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:07 trial068 ceph-mon[38805]: from='mgr.14152 10.20.193.68:0/3466135759' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2026-02-23T13:07:08.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:07 trial068 ceph-mon[38805]: osdmap e21: 4 total, 3 up, 4 in 2026-02-23T13:07:08.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:07 trial068 ceph-mon[38805]: from='mgr.14152 10.20.193.68:0/3466135759' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2026-02-23T13:07:08.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:07 trial068 ceph-mon[38805]: from='mgr.14152 10.20.193.68:0/3466135759' entity='mgr.x' 2026-02-23T13:07:08.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:07 trial068 ceph-mon[38805]: from='mgr.14152 10.20.193.68:0/3466135759' entity='mgr.x' 2026-02-23T13:07:08.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:07 trial068 ceph-mon[38805]: from='mgr.14152 10.20.193.68:0/3466135759' entity='mgr.x' 2026-02-23T13:07:08.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:07 trial068 ceph-mon[38805]: from='mgr.14152 10.20.193.68:0/3466135759' entity='mgr.x' 2026-02-23T13:07:08.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:07 trial068 ceph-mon[38805]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2026-02-23T13:07:08.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:07 trial068 ceph-mon[38805]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2026-02-23T13:07:08.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:07 trial068 ceph-mon[38805]: from='mgr.14152 10.20.193.68:0/3466135759' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2026-02-23T13:07:08.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:07 trial068 ceph-mon[38805]: from='mgr.14152 10.20.193.68:0/3466135759' entity='mgr.x' 2026-02-23T13:07:09.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:08 trial068 ceph-mon[38805]: osdmap e22: 4 total, 3 up, 4 in 2026-02-23T13:07:09.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:08 trial068 ceph-mon[38805]: from='mgr.14152 10.20.193.68:0/3466135759' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2026-02-23T13:07:09.209 INFO:teuthology.orchestra.run.trial185.stdout:Created osd(s) 3 on host 'trial185' 2026-02-23T13:07:09.249 DEBUG:teuthology.orchestra.run.trial185:osd.3> sudo journalctl -f -n 0 -u ceph-5f823084-10b8-11f1-be80-d404e6e7d460@osd.3.service 2026-02-23T13:07:09.251 INFO:tasks.cephadm:Waiting for 4 OSDs to come up... 2026-02-23T13:07:09.251 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd stat -f json 2026-02-23T13:07:09.374 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:07:09.473 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:07:09.471+0000 7f5a78b4c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:07:09.473 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:07:09.472+0000 7f5a78b4c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:07:09.581 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T13:07:09.605 INFO:journalctl@ceph.osd.3.trial185.stdout:Feb 23 13:07:09 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-3[54020]: 2026-02-23T13:07:09.323+0000 7fad71761780 -1 osd.3 0 log_to_monitors true 2026-02-23T13:07:09.627 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":22,"num_osds":4,"num_up_osds":3,"osd_up_since":1771852023,"num_in_osds":4,"osd_in_since":1771852023,"num_remapped_pgs":0} 2026-02-23T13:07:09.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:09 trial068 ceph-mon[38805]: pgmap v35: 1 pgs: 1 unknown; 0 B data, 79 MiB used, 2.1 TiB / 2.1 TiB avail 2026-02-23T13:07:09.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:09 trial068 ceph-mon[38805]: mgrmap e14: x(active, since 46s) 2026-02-23T13:07:09.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:09 trial068 ceph-mon[38805]: from='mgr.14152 10.20.193.68:0/3466135759' entity='mgr.x' 2026-02-23T13:07:09.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:09 trial068 ceph-mon[38805]: from='mgr.14152 10.20.193.68:0/3466135759' entity='mgr.x' 2026-02-23T13:07:09.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:09 trial068 ceph-mon[38805]: from='mgr.14152 10.20.193.68:0/3466135759' entity='mgr.x' 2026-02-23T13:07:09.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:09 trial068 ceph-mon[38805]: from='mgr.14152 10.20.193.68:0/3466135759' entity='mgr.x' 2026-02-23T13:07:09.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:09 trial068 ceph-mon[38805]: from='osd.3 [v2:10.20.193.185:6808/3135022860,v1:10.20.193.185:6809/3135022860]' entity='osd.3' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]} : dispatch 2026-02-23T13:07:09.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:09 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2638495466' entity='client.admin' cmd={"prefix": "osd stat", "format": "json"} : dispatch 2026-02-23T13:07:10.628 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd stat -f json 2026-02-23T13:07:10.748 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:07:10.844 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:07:10.842+0000 7fea37f2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:07:10.844 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:07:10.842+0000 7fea37f2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:07:10.950 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T13:07:10.951 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:10 trial068 ceph-mon[38805]: from='mgr.14152 10.20.193.68:0/3466135759' entity='mgr.x' 2026-02-23T13:07:10.951 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:10 trial068 ceph-mon[38805]: from='mgr.14152 10.20.193.68:0/3466135759' entity='mgr.x' 2026-02-23T13:07:10.951 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:10 trial068 ceph-mon[38805]: from='mgr.14152 10.20.193.68:0/3466135759' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2026-02-23T13:07:10.951 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:10 trial068 ceph-mon[38805]: from='mgr.14152 10.20.193.68:0/3466135759' entity='mgr.x' 2026-02-23T13:07:10.951 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:10 trial068 ceph-mon[38805]: from='mgr.14152 10.20.193.68:0/3466135759' entity='mgr.x' 2026-02-23T13:07:10.951 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:10 trial068 ceph-mon[38805]: from='osd.3 [v2:10.20.193.185:6808/3135022860,v1:10.20.193.185:6809/3135022860]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2026-02-23T13:07:10.951 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:10 trial068 ceph-mon[38805]: osdmap e23: 4 total, 3 up, 4 in 2026-02-23T13:07:10.951 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:10 trial068 ceph-mon[38805]: from='mgr.14152 10.20.193.68:0/3466135759' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2026-02-23T13:07:10.951 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:10 trial068 ceph-mon[38805]: from='osd.3 [v2:10.20.193.185:6808/3135022860,v1:10.20.193.185:6809/3135022860]' entity='osd.3' cmd={"prefix": "osd crush create-or-move", "id": 3, "weight":0.6836, "args": ["host=trial185", "root=default"]} : dispatch 2026-02-23T13:07:10.987 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":23,"num_osds":4,"num_up_osds":3,"osd_up_since":1771852023,"num_in_osds":4,"osd_in_since":1771852023,"num_remapped_pgs":0} 2026-02-23T13:07:11.597 INFO:journalctl@ceph.osd.3.trial185.stdout:Feb 23 13:07:11 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-3[54020]: 2026-02-23T13:07:11.212+0000 7fad6d6e3640 -1 osd.3 0 waiting for initial osdmap 2026-02-23T13:07:11.598 INFO:journalctl@ceph.osd.3.trial185.stdout:Feb 23 13:07:11 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-3[54020]: 2026-02-23T13:07:11.215+0000 7fad655a2640 -1 osd.3 24 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2026-02-23T13:07:11.988 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd stat -f json 2026-02-23T13:07:12.014 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:11 trial068 ceph-mon[38805]: pgmap v37: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 2.1 TiB / 2.1 TiB avail 2026-02-23T13:07:12.014 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:11 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3547759313' entity='client.admin' cmd={"prefix": "osd stat", "format": "json"} : dispatch 2026-02-23T13:07:12.014 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:11 trial068 ceph-mon[38805]: from='mgr.14152 10.20.193.68:0/3466135759' entity='mgr.x' 2026-02-23T13:07:12.014 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:11 trial068 ceph-mon[38805]: from='mgr.14152 10.20.193.68:0/3466135759' entity='mgr.x' 2026-02-23T13:07:12.014 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:11 trial068 ceph-mon[38805]: from='mgr.14152 10.20.193.68:0/3466135759' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"} : dispatch 2026-02-23T13:07:12.014 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:11 trial068 ceph-mon[38805]: from='mgr.14152 10.20.193.68:0/3466135759' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"} : dispatch 2026-02-23T13:07:12.014 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:11 trial068 ceph-mon[38805]: from='mgr.14152 10.20.193.68:0/3466135759' entity='mgr.x' 2026-02-23T13:07:12.015 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:11 trial068 ceph-mon[38805]: from='mgr.14152 10.20.193.68:0/3466135759' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-23T13:07:12.015 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:11 trial068 ceph-mon[38805]: from='mgr.14152 10.20.193.68:0/3466135759' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2026-02-23T13:07:12.015 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:11 trial068 ceph-mon[38805]: from='mgr.14152 10.20.193.68:0/3466135759' entity='mgr.x' 2026-02-23T13:07:12.015 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:11 trial068 ceph-mon[38805]: from='mgr.14152 10.20.193.68:0/3466135759' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2026-02-23T13:07:12.015 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:11 trial068 ceph-mon[38805]: from='mgr.14152 10.20.193.68:0/3466135759' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2026-02-23T13:07:12.015 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:11 trial068 ceph-mon[38805]: from='mgr.14152 10.20.193.68:0/3466135759' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-23T13:07:12.015 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:11 trial068 ceph-mon[38805]: from='osd.3 [v2:10.20.193.185:6808/3135022860,v1:10.20.193.185:6809/3135022860]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.6836, "args": ["host=trial185", "root=default"]}]': finished 2026-02-23T13:07:12.015 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:11 trial068 ceph-mon[38805]: osdmap e24: 4 total, 3 up, 4 in 2026-02-23T13:07:12.015 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:11 trial068 ceph-mon[38805]: from='mgr.14152 10.20.193.68:0/3466135759' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2026-02-23T13:07:12.015 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:11 trial068 ceph-mon[38805]: from='mgr.14152 10.20.193.68:0/3466135759' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2026-02-23T13:07:12.107 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:07:12.208 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:07:12.207+0000 7f7b56d2a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:07:12.209 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:07:12.207+0000 7f7b56d2a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:07:12.318 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T13:07:12.367 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":25,"num_osds":4,"num_up_osds":4,"osd_up_since":1771852032,"num_in_osds":4,"osd_in_since":1771852023,"num_remapped_pgs":1} 2026-02-23T13:07:12.367 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T13:07:12.522 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:07:12.620 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:07:12.619+0000 7fe0f87d9640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:07:12.620 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:07:12.619+0000 7fe0f87d9640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:07:12.730 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T13:07:12.730 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":25,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T13:07:12.211946+0000","last_up_change":"2026-02-23T13:07:12.211946+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":10,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.75,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":1216023594},{"type":"v1","addr":"10.20.193.68:6803","nonce":1216023594}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":1216023594},{"type":"v1","addr":"10.20.193.68:6805","nonce":1216023594}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":1216023594},{"type":"v1","addr":"10.20.193.68:6809","nonce":1216023594}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":1216023594},{"type":"v1","addr":"10.20.193.68:6807","nonce":1216023594}]},"public_addr":"10.20.193.68:6803/1216023594","cluster_addr":"10.20.193.68:6805/1216023594","heartbeat_back_addr":"10.20.193.68:6809/1216023594","heartbeat_front_addr":"10.20.193.68:6807/1216023594","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":1086778192},{"type":"v1","addr":"10.20.193.68:6811","nonce":1086778192}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":1086778192},{"type":"v1","addr":"10.20.193.68:6813","nonce":1086778192}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":1086778192},{"type":"v1","addr":"10.20.193.68:6817","nonce":1086778192}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":1086778192},{"type":"v1","addr":"10.20.193.68:6815","nonce":1086778192}]},"public_addr":"10.20.193.68:6811/1086778192","cluster_addr":"10.20.193.68:6813/1086778192","heartbeat_back_addr":"10.20.193.68:6817/1086778192","heartbeat_front_addr":"10.20.193.68:6815/1086778192","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":1978920641},{"type":"v1","addr":"10.20.193.185:6801","nonce":1978920641}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":1978920641},{"type":"v1","addr":"10.20.193.185:6803","nonce":1978920641}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":1978920641},{"type":"v1","addr":"10.20.193.185:6807","nonce":1978920641}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":1978920641},{"type":"v1","addr":"10.20.193.185:6805","nonce":1978920641}]},"public_addr":"10.20.193.185:6801/1978920641","cluster_addr":"10.20.193.185:6803/1978920641","heartbeat_back_addr":"10.20.193.185:6807/1978920641","heartbeat_front_addr":"10.20.193.185:6805/1978920641","state":["exists","up"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3135022860},{"type":"v1","addr":"10.20.193.185:6809","nonce":3135022860}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3135022860},{"type":"v1","addr":"10.20.193.185:6811","nonce":3135022860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3135022860},{"type":"v1","addr":"10.20.193.185:6815","nonce":3135022860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3135022860},{"type":"v1","addr":"10.20.193.185:6813","nonce":3135022860}]},"public_addr":"10.20.193.185:6809/3135022860","cluster_addr":"10.20.193.185:6811/3135022860","heartbeat_back_addr":"10.20.193.185:6815/3135022860","heartbeat_front_addr":"10.20.193.185:6813/3135022860","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:06:44.828141+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:06:52.665438+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:07:01.832977+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[{"pgid":"1.0","osds":[0,2,1]}],"primary_temp":[],"blocklist":{"10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T13:07:12.767 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2026-02-23T13:07:04.695348+0000', 'flags': 262145, 'flags_names': 'hashpspool,crimson', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'nonprimary_shards': '{}', 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 4, 'score_stable': 4, 'optimal_score': 0.75, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2026-02-23T13:07:12.768 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd pool get .mgr pg_num 2026-02-23T13:07:12.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:12 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T13:07:12.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:12 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T13:07:12.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:12 trial068 ceph-mon[38805]: Detected new or changed devices on trial185 2026-02-23T13:07:12.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:12 trial068 ceph-mon[38805]: Adjusting osd_memory_target on trial185 to 44583M 2026-02-23T13:07:12.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:12 trial068 ceph-mon[38805]: osd.3 [v2:10.20.193.185:6808/3135022860,v1:10.20.193.185:6809/3135022860] boot 2026-02-23T13:07:12.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:12 trial068 ceph-mon[38805]: osdmap e25: 4 total, 4 up, 4 in 2026-02-23T13:07:12.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:12 trial068 ceph-mon[38805]: from='mgr.14152 10.20.193.68:0/3466135759' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2026-02-23T13:07:12.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:12 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/723613819' entity='client.admin' cmd={"prefix": "osd stat", "format": "json"} : dispatch 2026-02-23T13:07:12.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:12 trial068 ceph-mon[38805]: from='mgr.14152 10.20.193.68:0/3466135759' entity='mgr.x' 2026-02-23T13:07:12.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:12 trial068 ceph-mon[38805]: from='mgr.14152 10.20.193.68:0/3466135759' entity='mgr.x' 2026-02-23T13:07:12.884 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:07:12.989 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:07:12.987+0000 7f556e03f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:07:12.989 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:07:12.987+0000 7f556e03f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:07:13.097 INFO:teuthology.orchestra.run.trial068.stdout:pg_num: 1 2026-02-23T13:07:13.134 INFO:tasks.cephadm:Adding prometheus.a on trial185 2026-02-23T13:07:13.134 DEBUG:teuthology.orchestra.run.trial185:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph orch apply prometheus '1;trial185=a' 2026-02-23T13:07:13.252 INFO:teuthology.orchestra.run.trial185.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/config/ceph.conf 2026-02-23T13:07:13.358 INFO:teuthology.orchestra.run.trial185.stderr:2026-02-23T13:07:13.357+0000 7fb52154a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:07:13.358 INFO:teuthology.orchestra.run.trial185.stderr:2026-02-23T13:07:13.358+0000 7fb52154a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:07:13.480 INFO:teuthology.orchestra.run.trial185.stdout:Scheduled prometheus update... 2026-02-23T13:07:13.519 DEBUG:teuthology.orchestra.run.trial185:prometheus.a> sudo journalctl -f -n 0 -u ceph-5f823084-10b8-11f1-be80-d404e6e7d460@prometheus.a.service 2026-02-23T13:07:13.521 INFO:tasks.cephadm:Adding node-exporter.a on trial068 2026-02-23T13:07:13.521 INFO:tasks.cephadm:Adding node-exporter.b on trial185 2026-02-23T13:07:13.521 DEBUG:teuthology.orchestra.run.trial185:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph orch apply node-exporter '2;trial068=a;trial185=b' 2026-02-23T13:07:13.674 INFO:teuthology.orchestra.run.trial185.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/config/ceph.conf 2026-02-23T13:07:13.783 INFO:teuthology.orchestra.run.trial185.stderr:2026-02-23T13:07:13.782+0000 7f619f755640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:07:13.783 INFO:teuthology.orchestra.run.trial185.stderr:2026-02-23T13:07:13.782+0000 7f619f755640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:07:13.903 INFO:teuthology.orchestra.run.trial185.stdout:Scheduled node-exporter update... 2026-02-23T13:07:13.942 DEBUG:teuthology.orchestra.run.trial068:node-exporter.a> sudo journalctl -f -n 0 -u ceph-5f823084-10b8-11f1-be80-d404e6e7d460@node-exporter.a.service 2026-02-23T13:07:13.945 DEBUG:teuthology.orchestra.run.trial185:node-exporter.b> sudo journalctl -f -n 0 -u ceph-5f823084-10b8-11f1-be80-d404e6e7d460@node-exporter.b.service 2026-02-23T13:07:13.948 INFO:tasks.cephadm:Setting up client nodes... 2026-02-23T13:07:13.949 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2026-02-23T13:07:13.967 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:13 trial068 ceph-mon[38805]: pgmap v40: 1 pgs: 1 unknown; 577 KiB data, 108 MiB used, 2.7 TiB / 2.7 TiB avail 2026-02-23T13:07:13.967 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:13 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1524360151' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T13:07:13.967 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:13 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2758273267' entity='client.admin' cmd={"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} : dispatch 2026-02-23T13:07:13.967 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:13 trial068 ceph-mon[38805]: osdmap e26: 4 total, 4 up, 4 in 2026-02-23T13:07:13.967 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:13 trial068 ceph-mon[38805]: from='mgr.14152 10.20.193.68:0/3466135759' entity='mgr.x' 2026-02-23T13:07:13.967 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:13 trial068 ceph-mon[38805]: from='mgr.14152 10.20.193.68:0/3466135759' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2026-02-23T13:07:14.081 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:07:14.194 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:07:14.193+0000 7f596dbc0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:07:14.194 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:07:14.193+0000 7f596dbc0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:07:14.341 INFO:teuthology.orchestra.run.trial068.stdout:[client.0] 2026-02-23T13:07:14.341 INFO:teuthology.orchestra.run.trial068.stdout: key = AQACUZxphhFOFBAAfC6krhSiFod+uTzBDTh1jg== 2026-02-23T13:07:14.378 DEBUG:teuthology.orchestra.run.trial068:> set -ex 2026-02-23T13:07:14.378 DEBUG:teuthology.orchestra.run.trial068:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2026-02-23T13:07:14.378 DEBUG:teuthology.orchestra.run.trial068:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2026-02-23T13:07:14.414 DEBUG:teuthology.orchestra.run.trial185:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph auth get-or-create client.1 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2026-02-23T13:07:14.532 INFO:teuthology.orchestra.run.trial185.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/config/ceph.conf 2026-02-23T13:07:14.651 INFO:teuthology.orchestra.run.trial185.stderr:2026-02-23T13:07:14.650+0000 7f35f6995640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:07:14.651 INFO:teuthology.orchestra.run.trial185.stderr:2026-02-23T13:07:14.650+0000 7f35f6995640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:07:14.803 INFO:teuthology.orchestra.run.trial185.stdout:[client.1] 2026-02-23T13:07:14.804 INFO:teuthology.orchestra.run.trial185.stdout: key = AQACUZxpiLzcLxAA/4+ltePqICCVrI4gXD6ZYg== 2026-02-23T13:07:14.844 DEBUG:teuthology.orchestra.run.trial185:> set -ex 2026-02-23T13:07:14.844 DEBUG:teuthology.orchestra.run.trial185:> sudo dd of=/etc/ceph/ceph.client.1.keyring 2026-02-23T13:07:14.844 DEBUG:teuthology.orchestra.run.trial185:> sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2026-02-23T13:07:14.882 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph config log 1 --format=json 2026-02-23T13:07:14.996 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:07:15.019 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:14 trial068 ceph-mon[38805]: from='client.14250 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;trial185=a", "target": ["mon-mgr", ""]}]: dispatch 2026-02-23T13:07:15.019 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:14 trial068 ceph-mon[38805]: Saving service prometheus spec with placement trial185=a;count:1 2026-02-23T13:07:15.019 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:14 trial068 ceph-mon[38805]: from='mgr.14152 10.20.193.68:0/3466135759' entity='mgr.x' 2026-02-23T13:07:15.019 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:14 trial068 ceph-mon[38805]: from='mgr.14152 10.20.193.68:0/3466135759' entity='mgr.x' 2026-02-23T13:07:15.019 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:14 trial068 ceph-mon[38805]: from='mgr.14152 10.20.193.68:0/3466135759' entity='mgr.x' 2026-02-23T13:07:15.019 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:14 trial068 ceph-mon[38805]: osdmap e27: 4 total, 4 up, 4 in 2026-02-23T13:07:15.019 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:14 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2223386902' entity='client.admin' cmd={"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} : dispatch 2026-02-23T13:07:15.019 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:14 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2223386902' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2026-02-23T13:07:15.093 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:07:15.091+0000 7f2908195640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:07:15.093 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:07:15.092+0000 7f2908195640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:07:15.202 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T13:07:15.239 INFO:teuthology.orchestra.run.trial068.stdout:[{"version":13,"timestamp":"2026-02-23T13:07:10.985165+0000","name":"","changes":[{"name":"osd/host:trial185/osd_memory_target","previous_value":"93498635059","new_value":"46749317529"}]}] 2026-02-23T13:07:15.239 INFO:tasks.ceph_manager:config epoch is 13 2026-02-23T13:07:15.240 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2026-02-23T13:07:15.240 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2026-02-23T13:07:15.240 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph mgr dump --format=json 2026-02-23T13:07:15.357 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:07:15.456 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:07:15.455+0000 7febab8f9640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:07:15.457 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:07:15.455+0000 7febab8f9640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:07:15.596 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T13:07:15.631 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":14,"flags":0,"active_gid":14152,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6800","nonce":2332248704},{"type":"v1","addr":"10.20.193.68:6801","nonce":2332248704}]},"active_addr":"10.20.193.68:6801/2332248704","active_change":"2026-02-23T13:06:22.483109+0000","active_mgr_features":4544132024016699391,"available":true,"standbys":[],"modules":["cephadm","dashboard","iostat","nfs"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to, use commas to separate multiple","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_pg_upmap_activity":{"name":"update_pg_upmap_activity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Updates pg_upmap activity stats to be used in `balancer status detail`","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often the agent on each host will gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"The first TCP port the agent will try to bind to. Up to 1000 subsequent ports will be attempted if this port cannot be bound","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this option can facilitate debugging daemons that encounter runtime problems.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"How frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"Fraction of total system memory to divide among autotuned daemons. Converged systems hosting both Ceph and compute should set a lower value; systems with more than 10 GiB of memory per OSD may benefit from a slightly larger value so that the abundance of RAM can be fully utilized","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command persistent logging","long_desc":"","tags":[],"see_also":[]},"certificate_automated_rotation_enabled":{"name":"certificate_automated_rotation_enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"This flag controls whether cephadm automatically rotates certificates upon expiration.","long_desc":"","tags":[],"see_also":[]},"certificate_check_debug_mode":{"name":"certificate_check_debug_mode","type":"bool","level":"dev","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"FOR TESTING ONLY: This flag forces the certificate check instead of waiting for certificate_check_period.","long_desc":"","tags":[],"see_also":[]},"certificate_check_period":{"name":"certificate_check_period","type":"int","level":"advanced","flags":0,"default_value":"1","min":"0","max":"30","enum_allowed":[],"desc":"Specifies how often (in days) the certificate should be checked for validity.","long_desc":"","tags":[],"see_also":[]},"certificate_duration_days":{"name":"certificate_duration_days","type":"int","level":"advanced","flags":0,"default_value":"1095","min":"90","max":"3650","enum_allowed":[],"desc":"Specifies the duration of self-signed certificates generated and signed by the cephadm root CA","long_desc":"","tags":[],"see_also":[]},"certificate_renewal_threshold_days":{"name":"certificate_renewal_threshold_days","type":"int","level":"advanced","flags":0,"default_value":"30","min":"10","max":"90","enum_allowed":[],"desc":"Specifies the lead time in days to initiate certificate renewal before expiration.","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently Podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.28.1","min":"","max":"","enum_allowed":[],"desc":"Alertmanager container image","long_desc":"","tags":[],"see_also":[]},"container_image_alloy":{"name":"container_image_alloy","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/alloy:latest","min":"","max":"","enum_allowed":[],"desc":"Alloy container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"Elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:12.3.1","min":"","max":"","enum_allowed":[],"desc":"Grafana container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"Haproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_nginx":{"name":"container_image_nginx","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nginx:sclorg-nginx-126","min":"","max":"","enum_allowed":[],"desc":"Nginx container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.9.1","min":"","max":"","enum_allowed":[],"desc":"Node exporter container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.6","min":"","max":"","enum_allowed":[],"desc":"Nvmeof container image","long_desc":"","tags":[],"see_also":[]},"container_image_oauth2_proxy":{"name":"container_image_oauth2_proxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/oauth2-proxy/oauth2-proxy:v7.6.0","min":"","max":"","enum_allowed":[],"desc":"Oauth2 proxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v3.6.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba_metrics":{"name":"container_image_samba_metrics","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-metrics:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba metrics container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"Snmp gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"Seconds for which to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"quay.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"Seconds for which to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"Seconds for which to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"Location of dashboards to include in Grafana","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"How frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether the hardware monitoring daemon be deployed on every host?","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at the debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"Max number of daemons per service per host: guard against mistakes and runaways","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"Max number of OSDs that will be drained simultaneously when OSDs are removed","long_desc":"","tags":[],"see_also":[]},"max_parallel_osd_upgrades":{"name":"max_parallel_osd_upgrades","type":"int","level":"advanced","flags":0,"default_value":"16","min":"","max":"","enum_allowed":[],"desc":"Maximum number of OSD daemons upgraded in parallel.","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"Mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default IP address for RedFish API (OOB management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"Location of alerts to include in Prometheus Alertmanager","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry URL for login purposes. This is not the default registry.","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"Cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Customized SSH config file to distribute to managed hosts","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_count_max":{"name":"ssh_keepalive_count_max","type":"int","level":"advanced","flags":0,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"How many sequential SSH connection liveness check failures before the host is marked offline","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_interval":{"name":"ssh_keepalive_interval","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"How often SSH connections are checked for liveness","long_desc":"","tags":[],"see_also":[]},"stray_daemon_check_interval":{"name":"stray_daemon_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"How frequently cephadm should check for the presence of stray daemons","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use the cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest to ensure that all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROM_ALERT_CREDENTIAL_CACHE_TTL":{"name":"PROM_ALERT_CREDENTIAL_CACHE_TTL","type":"int","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_HOSTNAME_PER_DAEMON":{"name":"RGW_HOSTNAME_PER_DAEMON","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crypto_caller":{"name":"crypto_caller","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sso_oauth2":{"name":"sso_oauth2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"healthcheck_history_max_entries":{"name":"healthcheck_history_max_entries","type":"int","level":"advanced","flags":1,"default_value":"1000","min":"","max":"","enum_allowed":[],"desc":"Maximum number of health check history entries to keep","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"prometheus_tls_secret_name":{"name":"prometheus_tls_secret_name","type":"str","level":"advanced","flags":0,"default_value":"rook-ceph-prometheus-server-tls","min":"","max":"","enum_allowed":[],"desc":"name of tls secret in k8s for prometheus","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"smb","can_run":true,"error_string":"","module_options":{"internal_store_backend":{"name":"internal_store_backend","type":"str","level":"dev","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"set internal store backend. for develoment and testing only","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_orchestration":{"name":"update_orchestration","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically update orchestration when smb resources are changed","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"pause_cloning":{"name":"pause_cloning","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Pause asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"pause_purging":{"name":"pause_purging","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Pause asynchronous subvolume purge threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://10.20.193.68:8443/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"tentacle":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"force_disabled_modules":{},"last_failure_osd_epoch":3,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"10.20.193.68:0","nonce":1895012493}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"10.20.193.68:0","nonce":2848886808}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"10.20.193.68:0","nonce":2910016328}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"10.20.193.68:0","nonce":2086895042}]}]} 2026-02-23T13:07:15.639 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2026-02-23T13:07:15.639 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2026-02-23T13:07:15.640 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T13:07:15.755 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:07:15.855 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:07:15.854+0000 7f0bfea4c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:07:15.855 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:07:15.854+0000 7f0bfea4c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:07:15.966 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T13:07:15.967 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":27,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T13:07:14.215964+0000","last_up_change":"2026-02-23T13:07:12.211946+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":10,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.75,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":1216023594},{"type":"v1","addr":"10.20.193.68:6803","nonce":1216023594}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":1216023594},{"type":"v1","addr":"10.20.193.68:6805","nonce":1216023594}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":1216023594},{"type":"v1","addr":"10.20.193.68:6809","nonce":1216023594}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":1216023594},{"type":"v1","addr":"10.20.193.68:6807","nonce":1216023594}]},"public_addr":"10.20.193.68:6803/1216023594","cluster_addr":"10.20.193.68:6805/1216023594","heartbeat_back_addr":"10.20.193.68:6809/1216023594","heartbeat_front_addr":"10.20.193.68:6807/1216023594","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":1086778192},{"type":"v1","addr":"10.20.193.68:6811","nonce":1086778192}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":1086778192},{"type":"v1","addr":"10.20.193.68:6813","nonce":1086778192}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":1086778192},{"type":"v1","addr":"10.20.193.68:6817","nonce":1086778192}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":1086778192},{"type":"v1","addr":"10.20.193.68:6815","nonce":1086778192}]},"public_addr":"10.20.193.68:6811/1086778192","cluster_addr":"10.20.193.68:6813/1086778192","heartbeat_back_addr":"10.20.193.68:6817/1086778192","heartbeat_front_addr":"10.20.193.68:6815/1086778192","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":1978920641},{"type":"v1","addr":"10.20.193.185:6801","nonce":1978920641}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":1978920641},{"type":"v1","addr":"10.20.193.185:6803","nonce":1978920641}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":1978920641},{"type":"v1","addr":"10.20.193.185:6807","nonce":1978920641}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":1978920641},{"type":"v1","addr":"10.20.193.185:6805","nonce":1978920641}]},"public_addr":"10.20.193.185:6801/1978920641","cluster_addr":"10.20.193.185:6803/1978920641","heartbeat_back_addr":"10.20.193.185:6807/1978920641","heartbeat_front_addr":"10.20.193.185:6805/1978920641","state":["exists","up"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3135022860},{"type":"v1","addr":"10.20.193.185:6809","nonce":3135022860}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3135022860},{"type":"v1","addr":"10.20.193.185:6811","nonce":3135022860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3135022860},{"type":"v1","addr":"10.20.193.185:6815","nonce":3135022860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3135022860},{"type":"v1","addr":"10.20.193.185:6813","nonce":3135022860}]},"public_addr":"10.20.193.185:6809/3135022860","cluster_addr":"10.20.193.185:6811/3135022860","heartbeat_back_addr":"10.20.193.185:6815/3135022860","heartbeat_front_addr":"10.20.193.185:6813/3135022860","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:06:44.828141+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:06:52.665438+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:07:01.832977+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:07:10.302688+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T13:07:15.967 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:15 trial068 ceph-mon[38805]: from='client.14252 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;trial068=a;trial185=b", "target": ["mon-mgr", ""]}]: dispatch 2026-02-23T13:07:15.967 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:15 trial068 ceph-mon[38805]: Saving service node-exporter spec with placement trial068=a;trial185=b;count:2 2026-02-23T13:07:15.967 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:15 trial068 ceph-mon[38805]: pgmap v43: 1 pgs: 1 unknown; 577 KiB data, 108 MiB used, 2.7 TiB / 2.7 TiB avail 2026-02-23T13:07:15.967 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:15 trial068 ceph-mon[38805]: from='client.? 10.20.193.185:0/1749319782' entity='client.admin' cmd={"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} : dispatch 2026-02-23T13:07:15.967 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:15 trial068 ceph-mon[38805]: from='client.? 10.20.193.185:0/1749319782' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2026-02-23T13:07:15.967 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:15 trial068 ceph-mon[38805]: from='mgr.14152 10.20.193.68:0/3466135759' entity='mgr.x' 2026-02-23T13:07:15.967 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:15 trial068 ceph-mon[38805]: from='mgr.14152 10.20.193.68:0/3466135759' entity='mgr.x' 2026-02-23T13:07:15.967 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:15 trial068 ceph-mon[38805]: from='mgr.14152 10.20.193.68:0/3466135759' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-23T13:07:15.967 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:15 trial068 ceph-mon[38805]: from='mgr.14152 10.20.193.68:0/3466135759' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2026-02-23T13:07:15.967 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:15 trial068 ceph-mon[38805]: from='mgr.14152 10.20.193.68:0/3466135759' entity='mgr.x' 2026-02-23T13:07:15.967 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:15 trial068 ceph-mon[38805]: from='mgr.14152 10.20.193.68:0/3466135759' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2026-02-23T13:07:15.967 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:15 trial068 ceph-mon[38805]: from='mgr.14152 10.20.193.68:0/3466135759' entity='mgr.x' 2026-02-23T13:07:15.967 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:15 trial068 ceph-mon[38805]: from='mgr.14152 10.20.193.68:0/3466135759' entity='mgr.x' 2026-02-23T13:07:15.967 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:15 trial068 ceph-mon[38805]: from='mgr.14152 10.20.193.68:0/3466135759' entity='mgr.x' 2026-02-23T13:07:15.967 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:15 trial068 ceph-mon[38805]: from='mgr.14152 10.20.193.68:0/3466135759' entity='mgr.x' 2026-02-23T13:07:15.967 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:15 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/4010122718' entity='client.admin' cmd={"prefix": "config log", "num": 1, "format": "json"} : dispatch 2026-02-23T13:07:15.967 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:15 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2568751940' entity='client.admin' cmd={"prefix": "mgr dump", "format": "json"} : dispatch 2026-02-23T13:07:16.003 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2026-02-23T13:07:16.003 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T13:07:16.118 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:07:16.217 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:07:16.215+0000 7f8c7310b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:07:16.217 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:07:16.215+0000 7f8c7310b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:07:16.326 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T13:07:16.327 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":27,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T13:07:14.215964+0000","last_up_change":"2026-02-23T13:07:12.211946+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":10,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.75,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":1216023594},{"type":"v1","addr":"10.20.193.68:6803","nonce":1216023594}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":1216023594},{"type":"v1","addr":"10.20.193.68:6805","nonce":1216023594}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":1216023594},{"type":"v1","addr":"10.20.193.68:6809","nonce":1216023594}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":1216023594},{"type":"v1","addr":"10.20.193.68:6807","nonce":1216023594}]},"public_addr":"10.20.193.68:6803/1216023594","cluster_addr":"10.20.193.68:6805/1216023594","heartbeat_back_addr":"10.20.193.68:6809/1216023594","heartbeat_front_addr":"10.20.193.68:6807/1216023594","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":1086778192},{"type":"v1","addr":"10.20.193.68:6811","nonce":1086778192}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":1086778192},{"type":"v1","addr":"10.20.193.68:6813","nonce":1086778192}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":1086778192},{"type":"v1","addr":"10.20.193.68:6817","nonce":1086778192}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":1086778192},{"type":"v1","addr":"10.20.193.68:6815","nonce":1086778192}]},"public_addr":"10.20.193.68:6811/1086778192","cluster_addr":"10.20.193.68:6813/1086778192","heartbeat_back_addr":"10.20.193.68:6817/1086778192","heartbeat_front_addr":"10.20.193.68:6815/1086778192","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":1978920641},{"type":"v1","addr":"10.20.193.185:6801","nonce":1978920641}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":1978920641},{"type":"v1","addr":"10.20.193.185:6803","nonce":1978920641}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":1978920641},{"type":"v1","addr":"10.20.193.185:6807","nonce":1978920641}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":1978920641},{"type":"v1","addr":"10.20.193.185:6805","nonce":1978920641}]},"public_addr":"10.20.193.185:6801/1978920641","cluster_addr":"10.20.193.185:6803/1978920641","heartbeat_back_addr":"10.20.193.185:6807/1978920641","heartbeat_front_addr":"10.20.193.185:6805/1978920641","state":["exists","up"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3135022860},{"type":"v1","addr":"10.20.193.185:6809","nonce":3135022860}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3135022860},{"type":"v1","addr":"10.20.193.185:6811","nonce":3135022860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3135022860},{"type":"v1","addr":"10.20.193.185:6815","nonce":3135022860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3135022860},{"type":"v1","addr":"10.20.193.185:6813","nonce":3135022860}]},"public_addr":"10.20.193.185:6809/3135022860","cluster_addr":"10.20.193.185:6811/3135022860","heartbeat_back_addr":"10.20.193.185:6815/3135022860","heartbeat_front_addr":"10.20.193.185:6813/3135022860","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:06:44.828141+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:06:52.665438+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:07:01.832977+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:07:10.302688+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T13:07:16.379 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph tell osd.0 flush_pg_stats 2026-02-23T13:07:16.380 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph tell osd.1 flush_pg_stats 2026-02-23T13:07:16.380 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph tell osd.2 flush_pg_stats 2026-02-23T13:07:16.381 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph tell osd.3 flush_pg_stats 2026-02-23T13:07:16.499 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:07:16.500 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:07:16.506 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:07:16.509 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:07:16.609 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:07:16.608+0000 7f9a51098640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:07:16.610 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:07:16.608+0000 7f9de494e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:07:16.610 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:07:16.608+0000 7f9a51098640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:07:16.610 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:07:16.608+0000 7f9de494e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:07:16.612 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:07:16.610+0000 7fa60c5f0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:07:16.612 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:07:16.611+0000 7fa60c5f0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:07:16.620 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:07:16.619+0000 7f7950a6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:07:16.620 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:07:16.619+0000 7f7950a6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:07:16.660 INFO:teuthology.orchestra.run.trial068.stdout:77309411333 2026-02-23T13:07:16.661 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd last-stat-seq osd.2 2026-02-23T13:07:16.661 INFO:teuthology.orchestra.run.trial068.stdout:38654705672 2026-02-23T13:07:16.661 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd last-stat-seq osd.0 2026-02-23T13:07:16.675 INFO:teuthology.orchestra.run.trial068.stdout:107374182402 2026-02-23T13:07:16.675 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd last-stat-seq osd.3 2026-02-23T13:07:16.679 INFO:teuthology.orchestra.run.trial068.stdout:55834574855 2026-02-23T13:07:16.680 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd last-stat-seq osd.1 2026-02-23T13:07:16.791 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:07:16.791 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:07:16.813 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:16 trial068 ceph-mon[38805]: Deploying daemon prometheus.a on trial185 2026-02-23T13:07:16.813 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:16 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/658025333' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T13:07:16.813 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:16 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/4003420073' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T13:07:16.839 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:07:16.841 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:07:16.895 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:07:16.894+0000 7ff0e3dda640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:07:16.895 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:07:16.893+0000 7f468d377640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:07:16.895 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:07:16.894+0000 7f468d377640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:07:16.895 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:07:16.894+0000 7ff0e3dda640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:07:16.946 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:07:16.944+0000 7fcbd407e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:07:16.946 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:07:16.944+0000 7fcbd407e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:07:16.947 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:07:16.945+0000 7fb390aac640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:07:16.947 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:07:16.945+0000 7fb390aac640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:07:17.007 INFO:teuthology.orchestra.run.trial068.stdout:77309411332 2026-02-23T13:07:17.009 INFO:teuthology.orchestra.run.trial068.stdout:38654705671 2026-02-23T13:07:17.051 INFO:tasks.cephadm.ceph_manager.ceph:need seq 38654705672 got 38654705671 for osd.0 2026-02-23T13:07:17.051 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411333 got 77309411332 for osd.2 2026-02-23T13:07:17.061 INFO:teuthology.orchestra.run.trial068.stdout:55834574854 2026-02-23T13:07:17.061 INFO:teuthology.orchestra.run.trial068.stdout:107374182401 2026-02-23T13:07:17.101 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574855 got 55834574854 for osd.1 2026-02-23T13:07:17.104 INFO:tasks.cephadm.ceph_manager.ceph:need seq 107374182402 got 107374182401 for osd.3 2026-02-23T13:07:18.052 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd last-stat-seq osd.0 2026-02-23T13:07:18.052 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd last-stat-seq osd.2 2026-02-23T13:07:18.077 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:17 trial068 ceph-mon[38805]: pgmap v44: 1 pgs: 1 unknown; 577 KiB data, 108 MiB used, 2.7 TiB / 2.7 TiB avail 2026-02-23T13:07:18.077 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:17 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2089145810' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 2} : dispatch 2026-02-23T13:07:18.077 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:17 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3690854661' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 0} : dispatch 2026-02-23T13:07:18.077 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:17 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1129820889' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 1} : dispatch 2026-02-23T13:07:18.077 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:17 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3629433065' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 3} : dispatch 2026-02-23T13:07:18.102 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd last-stat-seq osd.1 2026-02-23T13:07:18.105 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd last-stat-seq osd.3 2026-02-23T13:07:18.171 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:07:18.171 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:07:18.234 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:07:18.237 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:07:18.276 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:07:18.275+0000 7fad1bfff640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:07:18.276 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:07:18.275+0000 7fad1bfff640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:07:18.278 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:07:18.277+0000 7f5ccd88b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:07:18.278 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:07:18.277+0000 7f5ccd88b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:07:18.345 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:07:18.344+0000 7f49f9751640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:07:18.346 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:07:18.344+0000 7fe86b0eb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:07:18.346 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:07:18.344+0000 7f49f9751640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:07:18.346 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:07:18.344+0000 7fe86b0eb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:07:18.390 INFO:teuthology.orchestra.run.trial068.stdout:38654705671 2026-02-23T13:07:18.390 INFO:teuthology.orchestra.run.trial068.stdout:77309411332 2026-02-23T13:07:18.430 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411333 got 77309411332 for osd.2 2026-02-23T13:07:18.434 INFO:tasks.cephadm.ceph_manager.ceph:need seq 38654705672 got 38654705671 for osd.0 2026-02-23T13:07:18.458 INFO:teuthology.orchestra.run.trial068.stdout:55834574854 2026-02-23T13:07:18.466 INFO:teuthology.orchestra.run.trial068.stdout:107374182401 2026-02-23T13:07:18.497 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574855 got 55834574854 for osd.1 2026-02-23T13:07:18.505 INFO:tasks.cephadm.ceph_manager.ceph:need seq 107374182402 got 107374182401 for osd.3 2026-02-23T13:07:19.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:18 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/4163349735' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 0} : dispatch 2026-02-23T13:07:19.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:18 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1653636250' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 2} : dispatch 2026-02-23T13:07:19.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:18 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/789734880' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 1} : dispatch 2026-02-23T13:07:19.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:18 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2949625945' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 3} : dispatch 2026-02-23T13:07:19.431 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd last-stat-seq osd.2 2026-02-23T13:07:19.435 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd last-stat-seq osd.0 2026-02-23T13:07:19.498 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd last-stat-seq osd.1 2026-02-23T13:07:19.506 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd last-stat-seq osd.3 2026-02-23T13:07:19.549 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:07:19.591 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:07:19.635 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:07:19.635 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:07:19.657 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:07:19.656+0000 7f5c84c9b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:07:19.657 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:07:19.656+0000 7f5c84c9b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:07:19.709 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:07:19.707+0000 7f4c75689640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:07:19.709 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:19 trial068 ceph-mon[38805]: pgmap v45: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 2.7 TiB / 2.7 TiB avail; 92 KiB/s, 0 objects/s recovering 2026-02-23T13:07:19.709 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:07:19.707+0000 7f4c75689640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:07:19.725 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:07:19.723+0000 7f186fad7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:07:19.725 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:07:19.724+0000 7f186fad7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:07:19.726 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:07:19.724+0000 7f75f9394640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:07:19.726 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:07:19.724+0000 7f75f9394640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:07:19.772 INFO:teuthology.orchestra.run.trial068.stdout:77309411333 2026-02-23T13:07:19.810 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411333 got 77309411333 for osd.2 2026-02-23T13:07:19.810 DEBUG:teuthology.parallel:result is None 2026-02-23T13:07:19.823 INFO:teuthology.orchestra.run.trial068.stdout:38654705673 2026-02-23T13:07:19.839 INFO:teuthology.orchestra.run.trial068.stdout:55834574855 2026-02-23T13:07:19.839 INFO:teuthology.orchestra.run.trial068.stdout:107374182403 2026-02-23T13:07:19.862 INFO:tasks.cephadm.ceph_manager.ceph:need seq 38654705672 got 38654705673 for osd.0 2026-02-23T13:07:19.862 DEBUG:teuthology.parallel:result is None 2026-02-23T13:07:19.878 INFO:tasks.cephadm.ceph_manager.ceph:need seq 107374182402 got 107374182403 for osd.3 2026-02-23T13:07:19.878 DEBUG:teuthology.parallel:result is None 2026-02-23T13:07:19.895 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574855 got 55834574855 for osd.1 2026-02-23T13:07:19.896 DEBUG:teuthology.parallel:result is None 2026-02-23T13:07:19.896 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2026-02-23T13:07:19.896 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph pg dump --format=json 2026-02-23T13:07:20.049 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:07:20.148 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:07:20.147+0000 7f833f450640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:07:20.149 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:07:20.147+0000 7f833f450640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:07:20.254 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T13:07:20.255 INFO:teuthology.orchestra.run.trial068.stderr:dumped all 2026-02-23T13:07:20.290 INFO:teuthology.orchestra.run.trial068.stdout:{"pg_ready":true,"pg_map":{"version":45,"stamp":"2026-02-23T13:07:18.492189+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":4,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":3,"kb":2936012800,"kb_used":110492,"kb_used_data":2636,"kb_used_omap":33,"kb_used_meta":107742,"kb_avail":2935902308,"statfs":{"total":3006477107200,"available":3006363963392,"internally_reserved":0,"allocated":2699264,"data_stored":2079736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":34472,"internal_metadata":110328152},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.279456"},"pg_stats":[{"pgid":"1.0","version":"21'92","reported_seq":18,"reported_epoch":27,"state":"active+clean","last_fresh":"2026-02-23T13:07:14.224912+0000","last_change":"2026-02-23T13:07:14.224739+0000","last_active":"2026-02-23T13:07:14.224912+0000","last_peered":"2026-02-23T13:07:14.224912+0000","last_clean":"2026-02-23T13:07:14.224912+0000","last_became_active":"2026-02-23T13:07:14.220178+0000","last_became_peered":"2026-02-23T13:07:14.220178+0000","last_unstale":"2026-02-23T13:07:14.224912+0000","last_undegraded":"2026-02-23T13:07:14.224912+0000","last_fullsized":"2026-02-23T13:07:14.224912+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:07:05.619208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:07:05.619208+0000","last_clean_scrub_stamp":"2026-02-23T13:07:05.619208+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T13:08:06.145941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":25,"seq":107374182403,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":734003200,"kb_used":27768,"kb_used_data":804,"kb_used_omap":5,"kb_used_meta":26938,"kb_avail":733975432,"statfs":{"total":751619276800,"available":751590842368,"internally_reserved":0,"allocated":823296,"data_stored":667526,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5517,"internal_metadata":27585139},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411333,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":734003200,"kb_used":27768,"kb_used_data":804,"kb_used_omap":6,"kb_used_meta":26937,"kb_avail":733975432,"statfs":{"total":751619276800,"available":751590842368,"internally_reserved":0,"allocated":823296,"data_stored":667526,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6175,"internal_metadata":27584481},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574855,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":734003200,"kb_used":27188,"kb_used_data":224,"kb_used_omap":9,"kb_used_meta":26934,"kb_avail":733976012,"statfs":{"total":751619276800,"available":751591436288,"internally_reserved":0,"allocated":229376,"data_stored":77158,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9437,"internal_metadata":27581219},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705673,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":734003200,"kb_used":27768,"kb_used_data":804,"kb_used_omap":13,"kb_used_meta":26930,"kb_avail":733975432,"statfs":{"total":751619276800,"available":751590842368,"internally_reserved":0,"allocated":823296,"data_stored":667526,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13343,"internal_metadata":27577313},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-23T13:07:20.291 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph pg dump --format=json 2026-02-23T13:07:20.405 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:07:20 trial185 systemd[1]: Started Ceph prometheus.a for 5f823084-10b8-11f1-be80-d404e6e7d460. 2026-02-23T13:07:20.408 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:07:20.508 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:07:20.506+0000 7f553a3a3640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:07:20.508 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:07:20.506+0000 7f553a3a3640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:07:20.614 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T13:07:20.614 INFO:teuthology.orchestra.run.trial068.stderr:dumped all 2026-02-23T13:07:20.651 INFO:teuthology.orchestra.run.trial068.stdout:{"pg_ready":true,"pg_map":{"version":46,"stamp":"2026-02-23T13:07:20.492438+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":4,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":3,"kb":2936012800,"kb_used":110492,"kb_used_data":2636,"kb_used_omap":33,"kb_used_meta":107742,"kb_avail":2935902308,"statfs":{"total":3006477107200,"available":3006363963392,"internally_reserved":0,"allocated":2699264,"data_stored":2079736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":34472,"internal_metadata":110328152},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001093"},"pg_stats":[{"pgid":"1.0","version":"21'92","reported_seq":18,"reported_epoch":27,"state":"active+clean","last_fresh":"2026-02-23T13:07:14.224912+0000","last_change":"2026-02-23T13:07:14.224739+0000","last_active":"2026-02-23T13:07:14.224912+0000","last_peered":"2026-02-23T13:07:14.224912+0000","last_clean":"2026-02-23T13:07:14.224912+0000","last_became_active":"2026-02-23T13:07:14.220178+0000","last_became_peered":"2026-02-23T13:07:14.220178+0000","last_unstale":"2026-02-23T13:07:14.224912+0000","last_undegraded":"2026-02-23T13:07:14.224912+0000","last_fullsized":"2026-02-23T13:07:14.224912+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:07:05.619208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:07:05.619208+0000","last_clean_scrub_stamp":"2026-02-23T13:07:05.619208+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T13:08:06.145941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":25,"seq":107374182403,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":734003200,"kb_used":27768,"kb_used_data":804,"kb_used_omap":5,"kb_used_meta":26938,"kb_avail":733975432,"statfs":{"total":751619276800,"available":751590842368,"internally_reserved":0,"allocated":823296,"data_stored":667526,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5517,"internal_metadata":27585139},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411334,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":734003200,"kb_used":27768,"kb_used_data":804,"kb_used_omap":6,"kb_used_meta":26937,"kb_avail":733975432,"statfs":{"total":751619276800,"available":751590842368,"internally_reserved":0,"allocated":823296,"data_stored":667526,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6175,"internal_metadata":27584481},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574856,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":734003200,"kb_used":27188,"kb_used_data":224,"kb_used_omap":9,"kb_used_meta":26934,"kb_avail":733976012,"statfs":{"total":751619276800,"available":751591436288,"internally_reserved":0,"allocated":229376,"data_stored":77158,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9437,"internal_metadata":27581219},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705673,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":734003200,"kb_used":27768,"kb_used_data":804,"kb_used_omap":13,"kb_used_meta":26930,"kb_avail":733975432,"statfs":{"total":751619276800,"available":751590842368,"internally_reserved":0,"allocated":823296,"data_stored":667526,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13343,"internal_metadata":27577313},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-23T13:07:20.652 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2026-02-23T13:07:20.652 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2026-02-23T13:07:20.652 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2026-02-23T13:07:20.652 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph health --format=json 2026-02-23T13:07:20.767 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:07:20.856 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:07:20 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[63174]: time=2026-02-23T13:07:20.404Z level=INFO source=main.go:1545 msg="updated GOGC" old=100 new=75 2026-02-23T13:07:20.856 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:07:20 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[63174]: time=2026-02-23T13:07:20.405Z level=INFO source=main.go:677 msg="Leaving GOMAXPROCS=24: CPU quota undefined" component=automaxprocs 2026-02-23T13:07:20.856 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:07:20 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[63174]: time=2026-02-23T13:07:20.405Z level=INFO source=memlimit.go:198 msg="GOMEMLIMIT is updated" component=automemlimit package=github.com/KimMachineGun/automemlimit/memlimit GOMEMLIMIT=120212530790 previous=9223372036854775807 2026-02-23T13:07:20.856 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:07:20 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[63174]: time=2026-02-23T13:07:20.405Z level=INFO source=main.go:770 msg="Starting Prometheus Server" mode=server version="(version=3.6.0, branch=HEAD, revision=491734606591ef8c188f5489c70380805bc868c0)" 2026-02-23T13:07:20.857 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:07:20 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[63174]: time=2026-02-23T13:07:20.405Z level=INFO source=main.go:775 msg="operational information" build_context="(go=go1.24.7, platform=linux/amd64, user=root@bfdf1aa26e4d, date=20250921-19:51:16, tags=netgo,builtinassets)" host_details="(Linux 5.14.0-681.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Wed Feb 11 20:19:22 UTC 2026 x86_64 trial185 (none))" fd_limits="(soft=1048576, hard=1048576)" vm_limits="(soft=unlimited, hard=unlimited)" 2026-02-23T13:07:20.857 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:07:20 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[63174]: time=2026-02-23T13:07:20.406Z level=INFO source=web.go:659 msg="Start listening for connections" component=web address=:9095 2026-02-23T13:07:20.857 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:07:20 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[63174]: time=2026-02-23T13:07:20.407Z level=INFO source=main.go:1289 msg="Starting TSDB ..." 2026-02-23T13:07:20.857 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:07:20 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[63174]: time=2026-02-23T13:07:20.408Z level=INFO source=tls_config.go:347 msg="Listening on" component=web address=[::]:9095 2026-02-23T13:07:20.857 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:07:20 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[63174]: time=2026-02-23T13:07:20.408Z level=INFO source=tls_config.go:350 msg="TLS is disabled." component=web http2=false address=[::]:9095 2026-02-23T13:07:20.857 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:07:20 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[63174]: time=2026-02-23T13:07:20.409Z level=INFO source=head.go:667 msg="Replaying on-disk memory mappable chunks if any" component=tsdb 2026-02-23T13:07:20.857 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:07:20 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[63174]: time=2026-02-23T13:07:20.409Z level=INFO source=head.go:754 msg="On-disk memory mappable chunks replay completed" component=tsdb duration=601ns 2026-02-23T13:07:20.857 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:07:20 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[63174]: time=2026-02-23T13:07:20.409Z level=INFO source=head.go:762 msg="Replaying WAL, this may take a while" component=tsdb 2026-02-23T13:07:20.857 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:07:20 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[63174]: time=2026-02-23T13:07:20.410Z level=INFO source=head.go:835 msg="WAL segment loaded" component=tsdb segment=0 maxSegment=0 duration=1.255962ms 2026-02-23T13:07:20.857 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:07:20 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[63174]: time=2026-02-23T13:07:20.410Z level=INFO source=head.go:872 msg="WAL replay completed" component=tsdb checkpoint_replay_duration=12.694µs wal_replay_duration=1.278113ms wbl_replay_duration=141ns chunk_snapshot_load_duration=0s mmap_chunk_replay_duration=601ns total_replay_duration=1.320071ms 2026-02-23T13:07:20.858 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:07:20 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[63174]: time=2026-02-23T13:07:20.411Z level=INFO source=main.go:1310 msg="filesystem information" fs_type=XFS_SUPER_MAGIC 2026-02-23T13:07:20.858 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:07:20 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[63174]: time=2026-02-23T13:07:20.411Z level=INFO source=main.go:1313 msg="TSDB started" 2026-02-23T13:07:20.858 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:07:20 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[63174]: time=2026-02-23T13:07:20.411Z level=INFO source=main.go:1498 msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2026-02-23T13:07:20.858 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:07:20 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[63174]: time=2026-02-23T13:07:20.423Z level=INFO source=main.go:1538 msg="Completed loading of configuration file" db_storage=591ns remote_storage=1.112µs web_handler=210ns query_engine=441ns scrape=362.297µs scrape_sd=57.566µs notify=922ns notify_sd=481ns rules=11.408586ms tracing=5.821µs filename=/etc/prometheus/prometheus.yml totalDuration=11.985539ms 2026-02-23T13:07:20.858 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:07:20 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[63174]: time=2026-02-23T13:07:20.423Z level=INFO source=main.go:1274 msg="Server is ready to receive web requests." 2026-02-23T13:07:20.858 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:07:20 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[63174]: time=2026-02-23T13:07:20.423Z level=INFO source=manager.go:176 msg="Starting rule manager..." component="rule manager" 2026-02-23T13:07:20.870 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:07:20.869+0000 7fc8b8170640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:07:20.871 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:07:20.869+0000 7fc8b8170640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:07:20.871 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:20 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/146621217' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 2} : dispatch 2026-02-23T13:07:20.871 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:20 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/663439802' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 0} : dispatch 2026-02-23T13:07:20.871 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:20 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/4195640669' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 1} : dispatch 2026-02-23T13:07:20.871 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:20 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/695931213' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 3} : dispatch 2026-02-23T13:07:20.871 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:20 trial068 ceph-mon[38805]: from='mgr.14152 10.20.193.68:0/3466135759' entity='mgr.x' 2026-02-23T13:07:20.871 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:20 trial068 ceph-mon[38805]: from='mgr.14152 10.20.193.68:0/3466135759' entity='mgr.x' 2026-02-23T13:07:20.871 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:20 trial068 ceph-mon[38805]: from='mgr.14152 10.20.193.68:0/3466135759' entity='mgr.x' 2026-02-23T13:07:20.871 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:20 trial068 ceph-mon[38805]: from='mgr.14152 10.20.193.68:0/3466135759' entity='mgr.x' cmd={"prefix": "mgr module enable", "module": "prometheus"} : dispatch 2026-02-23T13:07:21.005 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T13:07:21.005 INFO:teuthology.orchestra.run.trial068.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2026-02-23T13:07:21.041 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2026-02-23T13:07:21.041 INFO:tasks.cephadm:Setup complete, yielding 2026-02-23T13:07:21.042 INFO:teuthology.run_tasks:Running task cephadm.shell... 2026-02-23T13:07:21.051 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@trial068.front.sepia.ceph.com 2026-02-23T13:07:21.052 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- bash -c 'ceph orch status' 2026-02-23T13:07:21.166 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:07:21.269 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:07:21.267+0000 7f75be760640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:07:21.269 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:07:21.268+0000 7f75be760640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:07:21.384 INFO:teuthology.orchestra.run.trial068.stdout:Backend: cephadm 2026-02-23T13:07:21.384 INFO:teuthology.orchestra.run.trial068.stdout:Available: Yes 2026-02-23T13:07:21.385 INFO:teuthology.orchestra.run.trial068.stdout:Paused: No 2026-02-23T13:07:21.420 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- bash -c 'ceph orch ps' 2026-02-23T13:07:21.540 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:07:21.648 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:07:21.646+0000 7f8679b04640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:07:21.648 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:07:21.646+0000 7f8679b04640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:07:21.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:21 trial068 ceph-mon[38805]: from='client.14298 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2026-02-23T13:07:21.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:21 trial068 ceph-mon[38805]: pgmap v46: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 2.7 TiB / 2.7 TiB avail; 72 KiB/s, 0 objects/s recovering 2026-02-23T13:07:21.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:21 trial068 ceph-mon[38805]: from='client.14300 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2026-02-23T13:07:21.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:21 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1393843347' entity='client.admin' cmd={"prefix": "health", "format": "json"} : dispatch 2026-02-23T13:07:21.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:21 trial068 ceph-mon[38805]: from='mgr.14152 10.20.193.68:0/3466135759' entity='mgr.x' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2026-02-23T13:07:21.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:21 trial068 ceph-mon[38805]: mgrmap e15: x(active, since 58s) 2026-02-23T13:07:21.843 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:07:21 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ignoring --setuser ceph since I am not root 2026-02-23T13:07:21.843 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:07:21 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ignoring --setgroup ceph since I am not root 2026-02-23T13:07:21.843 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:07:21 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: 2026-02-23T13:07:21.463+0000 7f879e41afc0 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:07:21.843 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:07:21 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: 2026-02-23T13:07:21.465+0000 7f879e41afc0 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:07:21.844 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:07:21 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: 2026-02-23T13:07:21.465+0000 7f879e41afc0 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:07:22.843 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:07:22 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: /lib64/python3.9/site-packages/scipy/__init__.py:73: UserWarning: NumPy was imported from a Python sub-interpreter but NumPy does not properly support sub-interpreters. This will likely work for most users but might cause hard to track down issues or subtle bugs. A common user of the rare sub-interpreter feature is wsgi which also allows single-interpreter mode. 2026-02-23T13:07:22.844 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:07:22 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: Improvements in the case of bugs are welcome, but is not on the NumPy roadmap, and full support may require significant effort to achieve. 2026-02-23T13:07:22.844 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:07:22 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: from numpy import show_config as show_numpy_config 2026-02-23T13:07:25.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:24 trial068 ceph-mon[38805]: Active manager daemon x restarted 2026-02-23T13:07:25.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:24 trial068 ceph-mon[38805]: Activating manager daemon x 2026-02-23T13:07:25.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:24 trial068 ceph-mon[38805]: osdmap e28: 4 total, 4 up, 4 in 2026-02-23T13:07:25.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:24 trial068 ceph-mon[38805]: mgrmap e16: x(active, starting, since 0.00320144s) 2026-02-23T13:07:25.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:24 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2026-02-23T13:07:25.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:24 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "mgr metadata", "who": "x", "id": "x"} : dispatch 2026-02-23T13:07:25.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:24 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2026-02-23T13:07:25.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:24 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2026-02-23T13:07:25.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:24 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2026-02-23T13:07:25.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:24 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2026-02-23T13:07:25.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:24 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "mds metadata"} : dispatch 2026-02-23T13:07:25.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:24 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "osd metadata"} : dispatch 2026-02-23T13:07:25.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:24 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "mon metadata"} : dispatch 2026-02-23T13:07:25.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:24 trial068 ceph-mon[38805]: Manager daemon x is now available 2026-02-23T13:07:25.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:24 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2026-02-23T13:07:25.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:24 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T13:07:25.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:24 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2026-02-23T13:07:25.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:24 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "orch get-security-config"} : dispatch 2026-02-23T13:07:25.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:24 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"} : dispatch 2026-02-23T13:07:25.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:24 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"} : dispatch 2026-02-23T13:07:25.226 INFO:teuthology.orchestra.run.trial068.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2026-02-23T13:07:25.227 INFO:teuthology.orchestra.run.trial068.stdout:mgr.x trial068 *:9283,8765 running (80s) 0s ago 80s 510M - 20.3.0-5439-ge371433f 63df4b83db6f e6ebb07ef74a 2026-02-23T13:07:25.227 INFO:teuthology.orchestra.run.trial068.stdout:mon.a trial068 running (81s) 0s ago 82s 46.4M 2048M 20.3.0-5439-ge371433f 63df4b83db6f 0619909708be 2026-02-23T13:07:25.227 INFO:teuthology.orchestra.run.trial068.stdout:osd.0 trial068 running (42s) 0s ago 44s 39.5M 41.0G 20.3.0-5439-ge371433f 63df4b83db6f b1da7a11e45f 2026-02-23T13:07:25.227 INFO:teuthology.orchestra.run.trial068.stdout:osd.1 trial068 running (34s) 0s ago 34s 38.7M 41.0G 20.3.0-5439-ge371433f 63df4b83db6f ab0fbdb05622 2026-02-23T13:07:25.227 INFO:teuthology.orchestra.run.trial068.stdout:osd.2 trial185 running (25s) 0s ago 27s 39.9M 43.5G 20.3.0-5439-ge371433f 63df4b83db6f fec9fcc326bc 2026-02-23T13:07:25.227 INFO:teuthology.orchestra.run.trial068.stdout:osd.3 trial185 running (16s) 0s ago 18s 38.3M 43.5G 20.3.0-5439-ge371433f 63df4b83db6f 5cbc736f6ef2 2026-02-23T13:07:25.227 INFO:teuthology.orchestra.run.trial068.stdout:prometheus.a trial185 *:9095 running (4s) 0s ago 4s 35.5M - 3.6.0 4fcecf061b74 9f48b6754b00 2026-02-23T13:07:25.278 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- bash -c 'ceph orch ls' 2026-02-23T13:07:25.397 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:07:25.511 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:07:25.509+0000 7fd1a7a42640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:07:25.511 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:07:25.509+0000 7fd1a7a42640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:07:25.512 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:07:25 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: [23/Feb/2026:13:07:25] ENGINE Bus STARTING 2026-02-23T13:07:25.512 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:07:25 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: CherryPy Checker: 2026-02-23T13:07:25.512 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:07:25 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: The Application mounted at '' has an empty config. 2026-02-23T13:07:25.512 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:07:25 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: 2026-02-23T13:07:25.512 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:07:25 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: [23/Feb/2026:13:07:25] ENGINE Serving on http://:::9283 2026-02-23T13:07:25.512 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:07:25 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: [23/Feb/2026:13:07:25] ENGINE Bus STARTED 2026-02-23T13:07:25.624 INFO:teuthology.orchestra.run.trial068.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2026-02-23T13:07:25.625 INFO:teuthology.orchestra.run.trial068.stdout:mgr 1/1 0s ago 53s trial068=x;count:1 2026-02-23T13:07:25.625 INFO:teuthology.orchestra.run.trial068.stdout:mon 1/1 0s ago 54s trial068:10.20.193.68=a;count:1 2026-02-23T13:07:25.625 INFO:teuthology.orchestra.run.trial068.stdout:node-exporter ?:9100 0/2 - 11s trial068=a;trial185=b;count:2 2026-02-23T13:07:25.625 INFO:teuthology.orchestra.run.trial068.stdout:osd.default 4 0s ago 23s trial185 2026-02-23T13:07:25.625 INFO:teuthology.orchestra.run.trial068.stdout:prometheus ?:9095 1/1 0s ago 12s trial185=a;count:1 2026-02-23T13:07:25.665 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- bash -c 'ceph orch host ls' 2026-02-23T13:07:25.783 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:07:25.888 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:07:25.886+0000 7fad73ddd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:07:25.888 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:07:25.886+0000 7fad73ddd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:07:26.001 INFO:teuthology.orchestra.run.trial068.stdout:HOST ADDR LABELS STATUS 2026-02-23T13:07:26.002 INFO:teuthology.orchestra.run.trial068.stdout:trial068 10.20.193.68 2026-02-23T13:07:26.002 INFO:teuthology.orchestra.run.trial068.stdout:trial185 10.20.193.185 2026-02-23T13:07:26.002 INFO:teuthology.orchestra.run.trial068.stdout:2 hosts in cluster 2026-02-23T13:07:26.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:25 trial068 ceph-mon[38805]: [23/Feb/2026:13:07:24] ENGINE Bus STARTING 2026-02-23T13:07:26.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:25 trial068 ceph-mon[38805]: [23/Feb/2026:13:07:24] ENGINE Serving on https://10.20.193.68:7150 2026-02-23T13:07:26.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:25 trial068 ceph-mon[38805]: [23/Feb/2026:13:07:24] ENGINE Client ('10.20.193.68', 47392) lost — peer dropped the TLS connection suddenly, during handshake: (6, 'TLS/SSL connection has been closed (EOF) (_ssl.c:1147)') 2026-02-23T13:07:26.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:25 trial068 ceph-mon[38805]: [23/Feb/2026:13:07:24] ENGINE Serving on http://10.20.193.68:8765 2026-02-23T13:07:26.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:25 trial068 ceph-mon[38805]: [23/Feb/2026:13:07:24] ENGINE Bus STARTED 2026-02-23T13:07:26.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:25 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T13:07:26.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:25 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T13:07:26.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:25 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T13:07:26.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:25 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T13:07:26.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:25 trial068 ceph-mon[38805]: from='client.14308 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2026-02-23T13:07:26.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:25 trial068 ceph-mon[38805]: mgrmap e17: x(active, since 1.00535s) 2026-02-23T13:07:26.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:25 trial068 ceph-mon[38805]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "orch get-security-config"}]: dispatch 2026-02-23T13:07:26.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:25 trial068 ceph-mon[38805]: pgmap v3: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 2.7 TiB / 2.7 TiB avail 2026-02-23T13:07:26.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:25 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T13:07:26.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:25 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T13:07:26.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:25 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"} : dispatch 2026-02-23T13:07:26.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:25 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"} : dispatch 2026-02-23T13:07:26.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:25 trial068 ceph-mon[38805]: Adjusting osd_memory_target on trial185 to 44071M 2026-02-23T13:07:26.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:25 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T13:07:26.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:25 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T13:07:26.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:25 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T13:07:26.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:25 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-23T13:07:26.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:25 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2026-02-23T13:07:26.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:25 trial068 ceph-mon[38805]: Updating trial068:/etc/ceph/ceph.conf 2026-02-23T13:07:26.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:25 trial068 ceph-mon[38805]: Updating trial185:/etc/ceph/ceph.conf 2026-02-23T13:07:26.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:25 trial068 ceph-mon[38805]: Updating trial185:/var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/config/ceph.conf 2026-02-23T13:07:26.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:25 trial068 ceph-mon[38805]: Updating trial068:/var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/config/ceph.conf 2026-02-23T13:07:26.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:25 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T13:07:26.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:25 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T13:07:26.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:25 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T13:07:26.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:25 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T13:07:26.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:25 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T13:07:26.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:25 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2026-02-23T13:07:26.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:25 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T13:07:26.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:25 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T13:07:26.155 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- bash -c 'ceph orch device ls' 2026-02-23T13:07:26.364 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:07:26.387 INFO:journalctl@ceph.node-exporter.a.trial068.stdout:Feb 23 13:07:26 trial068 bash[75890]: Trying to pull quay.io/prometheus/node-exporter:v1.9.1... 2026-02-23T13:07:26.477 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:07:26.475+0000 7f8dd18eb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:07:26.477 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:07:26.475+0000 7f8dd18eb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:07:26.591 INFO:teuthology.orchestra.run.trial068.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2026-02-23T13:07:26.591 INFO:teuthology.orchestra.run.trial068.stdout:trial068 /dev/nvme1n1 ssd MTFDKCC1T9TGP-1BK1JABYY_012510B0041B 1788G No 31s ago Has a FileSystem, LVM detected 2026-02-23T13:07:26.591 INFO:teuthology.orchestra.run.trial068.stdout:trial068 /dev/nvme2n1 ssd MTFDKCC1T9TGP-1BK1JABYY_012510B0041A 1788G No 31s ago Has a FileSystem, LVM detected 2026-02-23T13:07:26.591 INFO:teuthology.orchestra.run.trial068.stdout:trial185 /dev/nvme1n1 ssd MTFDKCC1T9TGP-1BK1JABYY_112510AB533E 1788G No 11s ago Has a FileSystem, LVM detected 2026-02-23T13:07:26.591 INFO:teuthology.orchestra.run.trial068.stdout:trial185 /dev/nvme2n1 ssd MTFDKCC1T9TGP-1BK1JABYY_112510AB5806 1788G No 11s ago Has a FileSystem, LVM detected 2026-02-23T13:07:26.628 INFO:teuthology.run_tasks:Running task thrashosds... 2026-02-23T13:07:26.641 INFO:tasks.thrashosds:config is {'ceph_objectstore_tool': False, 'chance_inject_pause_short': 0, 'chance_reset_purged_snaps_last': 0, 'chance_thrash_cluster_full': 0, 'chance_thrash_pg_upmap': 0, 'chance_trim_stale_osdmaps': 0, 'dump_ops_enable': False, 'min_in': 3, 'noscrub_toggle_delay': 0, 'reweight_osd': 0, 'sighup_delay': 0, 'thrash_primary_affinity': False, 'timeout': 2400, 'optrack_toggle_delay': 2.0, 'random_eio': 0.0} 2026-02-23T13:07:26.642 INFO:tasks.thrashosds:overrides is {'admin_socket': {'branch': 'main'}, 'ceph': {'conf': {'client.0': {'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'crimson_allow_pg_split': False, 'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc', 'osd_pool_default_min_size': 2, 'osd_pool_default_size': 3}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'osd pool default crimson': True, 'paxos service trim min': 10}, 'osd': {'crimson cpu num': 4, 'crimson osd obc lru size': 10, 'debug ms': 20, 'debug osd': 20, 'debug seastore': 20, 'debug seastore backref': 20, 'debug seastore cache': 20, 'debug seastore cleaner': 20, 'debug seastore device': 20, 'debug seastore epm': 20, 'debug seastore fixedkv tree': 20, 'debug seastore journal': 20, 'debug seastore lba': 20, 'debug seastore odata': 20, 'debug seastore omap': 20, 'debug seastore onode': 20, 'debug seastore t': 20, 'debug seastore tm': 20, 'osd blocked scrub grace period': 3600, 'osd debug reject backfill probability': 0.3, 'osd delete sleep': 1, 'osd max backfills': 3, 'osd max markdown count': 1000, 'osd objectstore': 'seastore', 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd snap trim sleep': 2, 'osd target transaction size': 5, 'seastore cachepin size pershard': '64M', 'seastore main device type': 'RANDOM_BLOCK_SSD', 'seastore max concurrent transactions': 8}}, 'crimson_compat': True, 'flavor': 'debug', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', 'SLOW_OPS', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', 'mons down', 'mon down', 'out of quorum', 'noscrub', 'nodeep-scrub', 'Degraded data redundancy', 'is down', 'osds down', 'pg .*? is .*?degraded.*?, acting', 'pg .*? is stuck', 'pg degraded', 'PG_BACKFILL_FULL', 'Low space hindering backfill .*? backfill_toofull', 'OSD_ROOT_DOWN', 'pgs degraded', 'pgs undersized', 'is active.*backfill_toofull.*'], 'sha1': 'e371433fcd1390a0b82cfb18effc422c1f5d8c1d'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'debug', 'sha1': 'e371433fcd1390a0b82cfb18effc422c1f5d8c1d'}}, 'thrashosds': {'min_in': 4}, 'workunit': {'branch': 'wip-shraddhaag-enable-cephadm-tests', 'env': {'CRIMSON_COMPAT': '1'}, 'sha1': 'a78040ddef3ba5a489697ca2fe95df59a7d8bd08'}} 2026-02-23T13:07:26.642 INFO:tasks.thrashosds:config is {'ceph_objectstore_tool': False, 'chance_inject_pause_short': 0, 'chance_reset_purged_snaps_last': 0, 'chance_thrash_cluster_full': 0, 'chance_thrash_pg_upmap': 0, 'chance_trim_stale_osdmaps': 0, 'dump_ops_enable': False, 'min_in': 4, 'noscrub_toggle_delay': 0, 'reweight_osd': 0, 'sighup_delay': 0, 'thrash_primary_affinity': False, 'timeout': 2400, 'optrack_toggle_delay': 2.0, 'random_eio': 0.0} 2026-02-23T13:07:26.642 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph config set mgr mgr_debug_aggressive_pg_num_changes true 2026-02-23T13:07:26.853 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:07:26.919 INFO:journalctl@ceph.node-exporter.a.trial068.stdout:Feb 23 13:07:26 trial068 bash[75890]: Getting image source signatures 2026-02-23T13:07:26.920 INFO:journalctl@ceph.node-exporter.a.trial068.stdout:Feb 23 13:07:26 trial068 bash[75890]: Copying blob sha256:c6e37428e3b36731b2b7a99979e0dbb27ce0849df0c23fb2b73e99bdbcd7ca33 2026-02-23T13:07:26.920 INFO:journalctl@ceph.node-exporter.a.trial068.stdout:Feb 23 13:07:26 trial068 bash[75890]: Copying blob sha256:9fa9226be034e47923c0457d916aa68474cdfb23af8d4525e9baeebc4760977a 2026-02-23T13:07:26.920 INFO:journalctl@ceph.node-exporter.a.trial068.stdout:Feb 23 13:07:26 trial068 bash[75890]: Copying blob sha256:1617e25568b2231fdd0d5caff63b06f6f7738d8d961f031c80e47d35aaec9733 2026-02-23T13:07:26.920 INFO:journalctl@ceph.node-exporter.a.trial068.stdout:Feb 23 13:07:26 trial068 bash[75890]: Copying config sha256:255ec253085fb105696533b31864a5e3b64f1f1e2147b5ec368100cbf3ec9b80 2026-02-23T13:07:26.920 INFO:journalctl@ceph.node-exporter.a.trial068.stdout:Feb 23 13:07:26 trial068 bash[75890]: Writing manifest to image destination 2026-02-23T13:07:27.004 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:07:27.002+0000 7f7c43316640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:07:27.004 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:07:27.003+0000 7f7c43316640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:07:27.156 INFO:tasks.thrashosds:Beginning thrashosds... 2026-02-23T13:07:27.156 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2026-02-23T13:07:27.156 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph pg dump --format=json 2026-02-23T13:07:27.179 INFO:journalctl@ceph.node-exporter.a.trial068.stdout:Feb 23 13:07:26 trial068 podman[75890]: 2026-02-23 13:07:26.92154904 +0000 UTC m=+0.555762599 container create cd5d7c98a1b39969436917b6ed45aae7fd1fe252bf0e22781c4114afa55db253 (image=quay.io/prometheus/node-exporter:v1.9.1, name=ceph-5f823084-10b8-11f1-be80-d404e6e7d460-node-exporter-a, maintainer=The Prometheus Authors ) 2026-02-23T13:07:27.180 INFO:journalctl@ceph.node-exporter.a.trial068.stdout:Feb 23 13:07:26 trial068 podman[75890]: 2026-02-23 13:07:26.950180662 +0000 UTC m=+0.584394231 container init cd5d7c98a1b39969436917b6ed45aae7fd1fe252bf0e22781c4114afa55db253 (image=quay.io/prometheus/node-exporter:v1.9.1, name=ceph-5f823084-10b8-11f1-be80-d404e6e7d460-node-exporter-a, maintainer=The Prometheus Authors ) 2026-02-23T13:07:27.180 INFO:journalctl@ceph.node-exporter.a.trial068.stdout:Feb 23 13:07:26 trial068 podman[75890]: 2026-02-23 13:07:26.952402646 +0000 UTC m=+0.586616205 container start cd5d7c98a1b39969436917b6ed45aae7fd1fe252bf0e22781c4114afa55db253 (image=quay.io/prometheus/node-exporter:v1.9.1, name=ceph-5f823084-10b8-11f1-be80-d404e6e7d460-node-exporter-a, maintainer=The Prometheus Authors ) 2026-02-23T13:07:27.180 INFO:journalctl@ceph.node-exporter.a.trial068.stdout:Feb 23 13:07:26 trial068 bash[75890]: cd5d7c98a1b39969436917b6ed45aae7fd1fe252bf0e22781c4114afa55db253 2026-02-23T13:07:27.180 INFO:journalctl@ceph.node-exporter.a.trial068.stdout:Feb 23 13:07:26 trial068 podman[75890]: 2026-02-23 13:07:26.915105636 +0000 UTC m=+0.549319205 image pull 255ec253085fb105696533b31864a5e3b64f1f1e2147b5ec368100cbf3ec9b80 quay.io/prometheus/node-exporter:v1.9.1 2026-02-23T13:07:27.180 INFO:journalctl@ceph.node-exporter.a.trial068.stdout:Feb 23 13:07:26 trial068 systemd[1]: Started Ceph node-exporter.a for 5f823084-10b8-11f1-be80-d404e6e7d460. 2026-02-23T13:07:27.180 INFO:journalctl@ceph.node-exporter.a.trial068.stdout:Feb 23 13:07:26 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-node-exporter-a[76135]: time=2026-02-23T13:07:26.955Z level=INFO source=node_exporter.go:216 msg="Starting node_exporter" version="(version=1.9.1, branch=HEAD, revision=f2ec547b49af53815038a50265aa2adcd1275959)" 2026-02-23T13:07:27.180 INFO:journalctl@ceph.node-exporter.a.trial068.stdout:Feb 23 13:07:26 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-node-exporter-a[76135]: time=2026-02-23T13:07:26.955Z level=INFO source=node_exporter.go:217 msg="Build context" build_context="(go=go1.23.7, platform=linux/amd64, user=root@7023beaa563a, date=20250401-15:19:01, tags=unknown)" 2026-02-23T13:07:27.180 INFO:journalctl@ceph.node-exporter.a.trial068.stdout:Feb 23 13:07:26 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-node-exporter-a[76135]: time=2026-02-23T13:07:26.956Z level=INFO source=diskstats_common.go:110 msg="Parsed flag --collector.diskstats.device-exclude" collector=diskstats flag=^(z?ram|loop|fd|(h|s|v|xv)d[a-z]|nvme\d+n\d+p)\d+$ 2026-02-23T13:07:27.180 INFO:journalctl@ceph.node-exporter.a.trial068.stdout:Feb 23 13:07:26 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-node-exporter-a[76135]: time=2026-02-23T13:07:26.956Z level=ERROR source=diskstats_linux.go:264 msg="Failed to open directory, disabling udev device properties" collector=diskstats path=/run/udev/data 2026-02-23T13:07:27.180 INFO:journalctl@ceph.node-exporter.a.trial068.stdout:Feb 23 13:07:26 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-node-exporter-a[76135]: time=2026-02-23T13:07:26.956Z level=INFO source=filesystem_common.go:265 msg="Parsed flag --collector.filesystem.mount-points-exclude" collector=filesystem flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+|var/lib/containers/storage/.+)($|/) 2026-02-23T13:07:27.181 INFO:journalctl@ceph.node-exporter.a.trial068.stdout:Feb 23 13:07:26 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-node-exporter-a[76135]: time=2026-02-23T13:07:26.956Z level=INFO source=filesystem_common.go:294 msg="Parsed flag --collector.filesystem.fs-types-exclude" collector=filesystem flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2026-02-23T13:07:27.181 INFO:journalctl@ceph.node-exporter.a.trial068.stdout:Feb 23 13:07:26 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-node-exporter-a[76135]: time=2026-02-23T13:07:26.956Z level=INFO source=node_exporter.go:135 msg="Enabled collectors" 2026-02-23T13:07:27.181 INFO:journalctl@ceph.node-exporter.a.trial068.stdout:Feb 23 13:07:26 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-node-exporter-a[76135]: time=2026-02-23T13:07:26.956Z level=INFO source=node_exporter.go:141 msg=arp 2026-02-23T13:07:27.181 INFO:journalctl@ceph.node-exporter.a.trial068.stdout:Feb 23 13:07:26 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-node-exporter-a[76135]: time=2026-02-23T13:07:26.956Z level=INFO source=node_exporter.go:141 msg=bcache 2026-02-23T13:07:27.181 INFO:journalctl@ceph.node-exporter.a.trial068.stdout:Feb 23 13:07:26 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-node-exporter-a[76135]: time=2026-02-23T13:07:26.956Z level=INFO source=node_exporter.go:141 msg=bonding 2026-02-23T13:07:27.181 INFO:journalctl@ceph.node-exporter.a.trial068.stdout:Feb 23 13:07:26 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-node-exporter-a[76135]: time=2026-02-23T13:07:26.956Z level=INFO source=node_exporter.go:141 msg=btrfs 2026-02-23T13:07:27.181 INFO:journalctl@ceph.node-exporter.a.trial068.stdout:Feb 23 13:07:26 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-node-exporter-a[76135]: time=2026-02-23T13:07:26.956Z level=INFO source=node_exporter.go:141 msg=conntrack 2026-02-23T13:07:27.181 INFO:journalctl@ceph.node-exporter.a.trial068.stdout:Feb 23 13:07:26 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-node-exporter-a[76135]: time=2026-02-23T13:07:26.956Z level=INFO source=node_exporter.go:141 msg=cpu 2026-02-23T13:07:27.181 INFO:journalctl@ceph.node-exporter.a.trial068.stdout:Feb 23 13:07:26 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-node-exporter-a[76135]: time=2026-02-23T13:07:26.956Z level=INFO source=node_exporter.go:141 msg=cpufreq 2026-02-23T13:07:27.181 INFO:journalctl@ceph.node-exporter.a.trial068.stdout:Feb 23 13:07:26 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-node-exporter-a[76135]: time=2026-02-23T13:07:26.956Z level=INFO source=node_exporter.go:141 msg=diskstats 2026-02-23T13:07:27.181 INFO:journalctl@ceph.node-exporter.a.trial068.stdout:Feb 23 13:07:26 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-node-exporter-a[76135]: time=2026-02-23T13:07:26.956Z level=INFO source=node_exporter.go:141 msg=dmi 2026-02-23T13:07:27.181 INFO:journalctl@ceph.node-exporter.a.trial068.stdout:Feb 23 13:07:26 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-node-exporter-a[76135]: time=2026-02-23T13:07:26.956Z level=INFO source=node_exporter.go:141 msg=edac 2026-02-23T13:07:27.182 INFO:journalctl@ceph.node-exporter.a.trial068.stdout:Feb 23 13:07:26 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-node-exporter-a[76135]: time=2026-02-23T13:07:26.956Z level=INFO source=node_exporter.go:141 msg=entropy 2026-02-23T13:07:27.182 INFO:journalctl@ceph.node-exporter.a.trial068.stdout:Feb 23 13:07:26 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-node-exporter-a[76135]: time=2026-02-23T13:07:26.956Z level=INFO source=node_exporter.go:141 msg=fibrechannel 2026-02-23T13:07:27.182 INFO:journalctl@ceph.node-exporter.a.trial068.stdout:Feb 23 13:07:26 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-node-exporter-a[76135]: time=2026-02-23T13:07:26.956Z level=INFO source=node_exporter.go:141 msg=filefd 2026-02-23T13:07:27.182 INFO:journalctl@ceph.node-exporter.a.trial068.stdout:Feb 23 13:07:26 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-node-exporter-a[76135]: time=2026-02-23T13:07:26.956Z level=INFO source=node_exporter.go:141 msg=filesystem 2026-02-23T13:07:27.182 INFO:journalctl@ceph.node-exporter.a.trial068.stdout:Feb 23 13:07:26 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-node-exporter-a[76135]: time=2026-02-23T13:07:26.956Z level=INFO source=node_exporter.go:141 msg=hwmon 2026-02-23T13:07:27.182 INFO:journalctl@ceph.node-exporter.a.trial068.stdout:Feb 23 13:07:26 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-node-exporter-a[76135]: time=2026-02-23T13:07:26.956Z level=INFO source=node_exporter.go:141 msg=infiniband 2026-02-23T13:07:27.182 INFO:journalctl@ceph.node-exporter.a.trial068.stdout:Feb 23 13:07:26 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-node-exporter-a[76135]: time=2026-02-23T13:07:26.956Z level=INFO source=node_exporter.go:141 msg=ipvs 2026-02-23T13:07:27.182 INFO:journalctl@ceph.node-exporter.a.trial068.stdout:Feb 23 13:07:26 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-node-exporter-a[76135]: time=2026-02-23T13:07:26.956Z level=INFO source=node_exporter.go:141 msg=loadavg 2026-02-23T13:07:27.182 INFO:journalctl@ceph.node-exporter.a.trial068.stdout:Feb 23 13:07:26 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-node-exporter-a[76135]: time=2026-02-23T13:07:26.956Z level=INFO source=node_exporter.go:141 msg=mdadm 2026-02-23T13:07:27.182 INFO:journalctl@ceph.node-exporter.a.trial068.stdout:Feb 23 13:07:26 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-node-exporter-a[76135]: time=2026-02-23T13:07:26.956Z level=INFO source=node_exporter.go:141 msg=meminfo 2026-02-23T13:07:27.182 INFO:journalctl@ceph.node-exporter.a.trial068.stdout:Feb 23 13:07:26 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-node-exporter-a[76135]: time=2026-02-23T13:07:26.956Z level=INFO source=node_exporter.go:141 msg=netclass 2026-02-23T13:07:27.182 INFO:journalctl@ceph.node-exporter.a.trial068.stdout:Feb 23 13:07:26 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-node-exporter-a[76135]: time=2026-02-23T13:07:26.956Z level=INFO source=node_exporter.go:141 msg=netdev 2026-02-23T13:07:27.182 INFO:journalctl@ceph.node-exporter.a.trial068.stdout:Feb 23 13:07:26 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-node-exporter-a[76135]: time=2026-02-23T13:07:26.956Z level=INFO source=node_exporter.go:141 msg=netstat 2026-02-23T13:07:27.183 INFO:journalctl@ceph.node-exporter.a.trial068.stdout:Feb 23 13:07:26 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-node-exporter-a[76135]: time=2026-02-23T13:07:26.956Z level=INFO source=node_exporter.go:141 msg=nfs 2026-02-23T13:07:27.183 INFO:journalctl@ceph.node-exporter.a.trial068.stdout:Feb 23 13:07:26 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-node-exporter-a[76135]: time=2026-02-23T13:07:26.956Z level=INFO source=node_exporter.go:141 msg=nfsd 2026-02-23T13:07:27.183 INFO:journalctl@ceph.node-exporter.a.trial068.stdout:Feb 23 13:07:26 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-node-exporter-a[76135]: time=2026-02-23T13:07:26.956Z level=INFO source=node_exporter.go:141 msg=nvme 2026-02-23T13:07:27.183 INFO:journalctl@ceph.node-exporter.a.trial068.stdout:Feb 23 13:07:26 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-node-exporter-a[76135]: time=2026-02-23T13:07:26.956Z level=INFO source=node_exporter.go:141 msg=os 2026-02-23T13:07:27.183 INFO:journalctl@ceph.node-exporter.a.trial068.stdout:Feb 23 13:07:26 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-node-exporter-a[76135]: time=2026-02-23T13:07:26.956Z level=INFO source=node_exporter.go:141 msg=powersupplyclass 2026-02-23T13:07:27.183 INFO:journalctl@ceph.node-exporter.a.trial068.stdout:Feb 23 13:07:26 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-node-exporter-a[76135]: time=2026-02-23T13:07:26.956Z level=INFO source=node_exporter.go:141 msg=pressure 2026-02-23T13:07:27.183 INFO:journalctl@ceph.node-exporter.a.trial068.stdout:Feb 23 13:07:26 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-node-exporter-a[76135]: time=2026-02-23T13:07:26.956Z level=INFO source=node_exporter.go:141 msg=rapl 2026-02-23T13:07:27.183 INFO:journalctl@ceph.node-exporter.a.trial068.stdout:Feb 23 13:07:26 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-node-exporter-a[76135]: time=2026-02-23T13:07:26.956Z level=INFO source=node_exporter.go:141 msg=schedstat 2026-02-23T13:07:27.183 INFO:journalctl@ceph.node-exporter.a.trial068.stdout:Feb 23 13:07:26 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-node-exporter-a[76135]: time=2026-02-23T13:07:26.956Z level=INFO source=node_exporter.go:141 msg=selinux 2026-02-23T13:07:27.183 INFO:journalctl@ceph.node-exporter.a.trial068.stdout:Feb 23 13:07:26 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-node-exporter-a[76135]: time=2026-02-23T13:07:26.956Z level=INFO source=node_exporter.go:141 msg=sockstat 2026-02-23T13:07:27.183 INFO:journalctl@ceph.node-exporter.a.trial068.stdout:Feb 23 13:07:26 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-node-exporter-a[76135]: time=2026-02-23T13:07:26.956Z level=INFO source=node_exporter.go:141 msg=softnet 2026-02-23T13:07:27.183 INFO:journalctl@ceph.node-exporter.a.trial068.stdout:Feb 23 13:07:26 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-node-exporter-a[76135]: time=2026-02-23T13:07:26.956Z level=INFO source=node_exporter.go:141 msg=stat 2026-02-23T13:07:27.183 INFO:journalctl@ceph.node-exporter.a.trial068.stdout:Feb 23 13:07:26 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-node-exporter-a[76135]: time=2026-02-23T13:07:26.956Z level=INFO source=node_exporter.go:141 msg=tapestats 2026-02-23T13:07:27.184 INFO:journalctl@ceph.node-exporter.a.trial068.stdout:Feb 23 13:07:26 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-node-exporter-a[76135]: time=2026-02-23T13:07:26.956Z level=INFO source=node_exporter.go:141 msg=textfile 2026-02-23T13:07:27.184 INFO:journalctl@ceph.node-exporter.a.trial068.stdout:Feb 23 13:07:26 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-node-exporter-a[76135]: time=2026-02-23T13:07:26.956Z level=INFO source=node_exporter.go:141 msg=thermal_zone 2026-02-23T13:07:27.184 INFO:journalctl@ceph.node-exporter.a.trial068.stdout:Feb 23 13:07:26 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-node-exporter-a[76135]: time=2026-02-23T13:07:26.956Z level=INFO source=node_exporter.go:141 msg=time 2026-02-23T13:07:27.184 INFO:journalctl@ceph.node-exporter.a.trial068.stdout:Feb 23 13:07:26 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-node-exporter-a[76135]: time=2026-02-23T13:07:26.956Z level=INFO source=node_exporter.go:141 msg=udp_queues 2026-02-23T13:07:27.184 INFO:journalctl@ceph.node-exporter.a.trial068.stdout:Feb 23 13:07:26 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-node-exporter-a[76135]: time=2026-02-23T13:07:26.956Z level=INFO source=node_exporter.go:141 msg=uname 2026-02-23T13:07:27.184 INFO:journalctl@ceph.node-exporter.a.trial068.stdout:Feb 23 13:07:26 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-node-exporter-a[76135]: time=2026-02-23T13:07:26.956Z level=INFO source=node_exporter.go:141 msg=vmstat 2026-02-23T13:07:27.184 INFO:journalctl@ceph.node-exporter.a.trial068.stdout:Feb 23 13:07:26 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-node-exporter-a[76135]: time=2026-02-23T13:07:26.956Z level=INFO source=node_exporter.go:141 msg=watchdog 2026-02-23T13:07:27.184 INFO:journalctl@ceph.node-exporter.a.trial068.stdout:Feb 23 13:07:26 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-node-exporter-a[76135]: time=2026-02-23T13:07:26.956Z level=INFO source=node_exporter.go:141 msg=xfs 2026-02-23T13:07:27.184 INFO:journalctl@ceph.node-exporter.a.trial068.stdout:Feb 23 13:07:26 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-node-exporter-a[76135]: time=2026-02-23T13:07:26.956Z level=INFO source=node_exporter.go:141 msg=zfs 2026-02-23T13:07:27.184 INFO:journalctl@ceph.node-exporter.a.trial068.stdout:Feb 23 13:07:26 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-node-exporter-a[76135]: time=2026-02-23T13:07:26.957Z level=INFO source=tls_config.go:347 msg="Listening on" address=[::]:9100 2026-02-23T13:07:27.184 INFO:journalctl@ceph.node-exporter.a.trial068.stdout:Feb 23 13:07:26 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-node-exporter-a[76135]: time=2026-02-23T13:07:26.957Z level=INFO source=tls_config.go:350 msg="TLS is disabled." http2=false address=[::]:9100 2026-02-23T13:07:27.274 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:07:27.381 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:07:27.379+0000 7fe76695c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:07:27.381 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:07:27.379+0000 7fe76695c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:07:27.444 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:07:27 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: [23/Feb/2026:13:07:27] ENGINE Bus STOPPING 2026-02-23T13:07:27.444 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:07:27 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: [23/Feb/2026:13:07:27] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2026-02-23T13:07:27.445 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:07:27 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: [23/Feb/2026:13:07:27] ENGINE Bus STOPPED 2026-02-23T13:07:27.445 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:07:27 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: [23/Feb/2026:13:07:27] ENGINE Bus STARTING 2026-02-23T13:07:27.445 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:07:27 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: [23/Feb/2026:13:07:27] ENGINE Serving on http://:::9283 2026-02-23T13:07:27.445 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:07:27 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: [23/Feb/2026:13:07:27] ENGINE Bus STARTED 2026-02-23T13:07:27.445 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:27 trial068 ceph-mon[38805]: Updating trial068:/etc/ceph/ceph.client.admin.keyring 2026-02-23T13:07:27.445 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:27 trial068 ceph-mon[38805]: Updating trial185:/etc/ceph/ceph.client.admin.keyring 2026-02-23T13:07:27.445 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:27 trial068 ceph-mon[38805]: from='client.14316 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2026-02-23T13:07:27.445 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:27 trial068 ceph-mon[38805]: Updating trial068:/var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/config/ceph.client.admin.keyring 2026-02-23T13:07:27.446 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:27 trial068 ceph-mon[38805]: Updating trial185:/var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/config/ceph.client.admin.keyring 2026-02-23T13:07:27.446 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:27 trial068 ceph-mon[38805]: Deploying daemon node-exporter.a on trial068 2026-02-23T13:07:27.446 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:27 trial068 ceph-mon[38805]: from='client.14318 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2026-02-23T13:07:27.446 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:27 trial068 ceph-mon[38805]: pgmap v4: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 2.7 TiB / 2.7 TiB avail 2026-02-23T13:07:27.446 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:27 trial068 ceph-mon[38805]: mgrmap e18: x(active, since 2s) 2026-02-23T13:07:27.446 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:27 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T13:07:27.446 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:27 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T13:07:27.446 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:27 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T13:07:27.446 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:27 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1231207730' entity='client.admin' 2026-02-23T13:07:27.489 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T13:07:27.489 INFO:teuthology.orchestra.run.trial068.stderr:dumped all 2026-02-23T13:07:27.524 INFO:teuthology.orchestra.run.trial068.stdout:{"pg_ready":true,"pg_map":{"version":4,"stamp":"2026-02-23T13:07:26.223101+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":3,"kb":2936012800,"kb_used":110516,"kb_used_data":2660,"kb_used_omap":34,"kb_used_meta":107741,"kb_avail":2935902284,"statfs":{"total":3006477107200,"available":3006363938816,"internally_reserved":0,"allocated":2723840,"data_stored":2092609,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":35123,"internal_metadata":110327501},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.997931"},"pg_stats":[{"pgid":"1.0","version":"21'92","reported_seq":18,"reported_epoch":27,"state":"active+clean","last_fresh":"2026-02-23T13:07:14.224912+0000","last_change":"2026-02-23T13:07:14.224739+0000","last_active":"2026-02-23T13:07:14.224912+0000","last_peered":"2026-02-23T13:07:14.224912+0000","last_clean":"2026-02-23T13:07:14.224912+0000","last_became_active":"2026-02-23T13:07:14.220178+0000","last_became_peered":"2026-02-23T13:07:14.220178+0000","last_unstale":"2026-02-23T13:07:14.224912+0000","last_undegraded":"2026-02-23T13:07:14.224912+0000","last_fullsized":"2026-02-23T13:07:14.224912+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:07:05.619208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:07:05.619208+0000","last_clean_scrub_stamp":"2026-02-23T13:07:05.619208+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T13:08:06.145941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":25,"seq":107374182404,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":734003200,"kb_used":27768,"kb_used_data":804,"kb_used_omap":5,"kb_used_meta":26938,"kb_avail":733975432,"statfs":{"total":751619276800,"available":751590842368,"internally_reserved":0,"allocated":823296,"data_stored":667526,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5517,"internal_metadata":27585139},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411335,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":734003200,"kb_used":27776,"kb_used_data":812,"kb_used_omap":6,"kb_used_meta":26937,"kb_avail":733975424,"statfs":{"total":751619276800,"available":751590834176,"internally_reserved":0,"allocated":831488,"data_stored":671817,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6175,"internal_metadata":27584481},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574857,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":734003200,"kb_used":27196,"kb_used_data":232,"kb_used_omap":9,"kb_used_meta":26934,"kb_avail":733976004,"statfs":{"total":751619276800,"available":751591428096,"internally_reserved":0,"allocated":237568,"data_stored":81449,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9437,"internal_metadata":27581219},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705674,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":734003200,"kb_used":27776,"kb_used_data":812,"kb_used_omap":13,"kb_used_meta":26930,"kb_avail":733975424,"statfs":{"total":751619276800,"available":751590834176,"internally_reserved":0,"allocated":831488,"data_stored":671817,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13994,"internal_metadata":27576662},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-23T13:07:27.525 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph pg dump --format=json 2026-02-23T13:07:27.681 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:07:27.780 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:07:27.778+0000 7f43d1e17640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:07:27.780 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:07:27.779+0000 7f43d1e17640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:07:27.819 INFO:journalctl@ceph.node-exporter.b.trial185.stdout:Feb 23 13:07:27 trial185 bash[64572]: Getting image source signatures 2026-02-23T13:07:27.820 INFO:journalctl@ceph.node-exporter.b.trial185.stdout:Feb 23 13:07:27 trial185 bash[64572]: Copying blob sha256:9fa9226be034e47923c0457d916aa68474cdfb23af8d4525e9baeebc4760977a 2026-02-23T13:07:27.820 INFO:journalctl@ceph.node-exporter.b.trial185.stdout:Feb 23 13:07:27 trial185 bash[64572]: Copying blob sha256:c6e37428e3b36731b2b7a99979e0dbb27ce0849df0c23fb2b73e99bdbcd7ca33 2026-02-23T13:07:27.820 INFO:journalctl@ceph.node-exporter.b.trial185.stdout:Feb 23 13:07:27 trial185 bash[64572]: Copying blob sha256:1617e25568b2231fdd0d5caff63b06f6f7738d8d961f031c80e47d35aaec9733 2026-02-23T13:07:27.887 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T13:07:27.888 INFO:teuthology.orchestra.run.trial068.stderr:dumped all 2026-02-23T13:07:27.925 INFO:teuthology.orchestra.run.trial068.stdout:{"pg_ready":true,"pg_map":{"version":4,"stamp":"2026-02-23T13:07:26.223101+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":3,"kb":2936012800,"kb_used":110516,"kb_used_data":2660,"kb_used_omap":34,"kb_used_meta":107741,"kb_avail":2935902284,"statfs":{"total":3006477107200,"available":3006363938816,"internally_reserved":0,"allocated":2723840,"data_stored":2092609,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":35123,"internal_metadata":110327501},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.997931"},"pg_stats":[{"pgid":"1.0","version":"21'92","reported_seq":18,"reported_epoch":27,"state":"active+clean","last_fresh":"2026-02-23T13:07:14.224912+0000","last_change":"2026-02-23T13:07:14.224739+0000","last_active":"2026-02-23T13:07:14.224912+0000","last_peered":"2026-02-23T13:07:14.224912+0000","last_clean":"2026-02-23T13:07:14.224912+0000","last_became_active":"2026-02-23T13:07:14.220178+0000","last_became_peered":"2026-02-23T13:07:14.220178+0000","last_unstale":"2026-02-23T13:07:14.224912+0000","last_undegraded":"2026-02-23T13:07:14.224912+0000","last_fullsized":"2026-02-23T13:07:14.224912+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:07:05.619208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:07:05.619208+0000","last_clean_scrub_stamp":"2026-02-23T13:07:05.619208+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T13:08:06.145941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":25,"seq":107374182404,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":734003200,"kb_used":27768,"kb_used_data":804,"kb_used_omap":5,"kb_used_meta":26938,"kb_avail":733975432,"statfs":{"total":751619276800,"available":751590842368,"internally_reserved":0,"allocated":823296,"data_stored":667526,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5517,"internal_metadata":27585139},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411335,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":734003200,"kb_used":27776,"kb_used_data":812,"kb_used_omap":6,"kb_used_meta":26937,"kb_avail":733975424,"statfs":{"total":751619276800,"available":751590834176,"internally_reserved":0,"allocated":831488,"data_stored":671817,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6175,"internal_metadata":27584481},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574857,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":734003200,"kb_used":27196,"kb_used_data":232,"kb_used_omap":9,"kb_used_meta":26934,"kb_avail":733976004,"statfs":{"total":751619276800,"available":751591428096,"internally_reserved":0,"allocated":237568,"data_stored":81449,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9437,"internal_metadata":27581219},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705674,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":734003200,"kb_used":27776,"kb_used_data":812,"kb_used_omap":13,"kb_used_meta":26930,"kb_avail":733975424,"statfs":{"total":751619276800,"available":751590834176,"internally_reserved":0,"allocated":831488,"data_stored":671817,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13994,"internal_metadata":27576662},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-23T13:07:27.926 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2026-02-23T13:07:27.926 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump 2026-02-23T13:07:28.042 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:07:28.106 INFO:journalctl@ceph.node-exporter.b.trial185.stdout:Feb 23 13:07:27 trial185 bash[64572]: Copying config sha256:255ec253085fb105696533b31864a5e3b64f1f1e2147b5ec368100cbf3ec9b80 2026-02-23T13:07:28.106 INFO:journalctl@ceph.node-exporter.b.trial185.stdout:Feb 23 13:07:27 trial185 bash[64572]: Writing manifest to image destination 2026-02-23T13:07:28.107 INFO:journalctl@ceph.node-exporter.b.trial185.stdout:Feb 23 13:07:27 trial185 podman[64572]: 2026-02-23 13:07:27.974114068 +0000 UTC m=+0.457790806 container create ea6b1a2e8cb2afc027f5c26aaf1bfbe314defb0cf5759a73695bc7c91d16629b (image=quay.io/prometheus/node-exporter:v1.9.1, name=ceph-5f823084-10b8-11f1-be80-d404e6e7d460-node-exporter-b, maintainer=The Prometheus Authors ) 2026-02-23T13:07:28.107 INFO:journalctl@ceph.node-exporter.b.trial185.stdout:Feb 23 13:07:28 trial185 podman[64572]: 2026-02-23 13:07:28.003815227 +0000 UTC m=+0.487491905 container init ea6b1a2e8cb2afc027f5c26aaf1bfbe314defb0cf5759a73695bc7c91d16629b (image=quay.io/prometheus/node-exporter:v1.9.1, name=ceph-5f823084-10b8-11f1-be80-d404e6e7d460-node-exporter-b, maintainer=The Prometheus Authors ) 2026-02-23T13:07:28.107 INFO:journalctl@ceph.node-exporter.b.trial185.stdout:Feb 23 13:07:28 trial185 podman[64572]: 2026-02-23 13:07:28.006073305 +0000 UTC m=+0.489749983 container start ea6b1a2e8cb2afc027f5c26aaf1bfbe314defb0cf5759a73695bc7c91d16629b (image=quay.io/prometheus/node-exporter:v1.9.1, name=ceph-5f823084-10b8-11f1-be80-d404e6e7d460-node-exporter-b, maintainer=The Prometheus Authors ) 2026-02-23T13:07:28.107 INFO:journalctl@ceph.node-exporter.b.trial185.stdout:Feb 23 13:07:28 trial185 bash[64572]: ea6b1a2e8cb2afc027f5c26aaf1bfbe314defb0cf5759a73695bc7c91d16629b 2026-02-23T13:07:28.107 INFO:journalctl@ceph.node-exporter.b.trial185.stdout:Feb 23 13:07:28 trial185 podman[64572]: 2026-02-23 13:07:27.968561614 +0000 UTC m=+0.452238292 image pull 255ec253085fb105696533b31864a5e3b64f1f1e2147b5ec368100cbf3ec9b80 quay.io/prometheus/node-exporter:v1.9.1 2026-02-23T13:07:28.107 INFO:journalctl@ceph.node-exporter.b.trial185.stdout:Feb 23 13:07:28 trial185 systemd[1]: Started Ceph node-exporter.b for 5f823084-10b8-11f1-be80-d404e6e7d460. 2026-02-23T13:07:28.107 INFO:journalctl@ceph.node-exporter.b.trial185.stdout:Feb 23 13:07:28 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-node-exporter-b[64620]: time=2026-02-23T13:07:28.009Z level=INFO source=node_exporter.go:216 msg="Starting node_exporter" version="(version=1.9.1, branch=HEAD, revision=f2ec547b49af53815038a50265aa2adcd1275959)" 2026-02-23T13:07:28.107 INFO:journalctl@ceph.node-exporter.b.trial185.stdout:Feb 23 13:07:28 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-node-exporter-b[64620]: time=2026-02-23T13:07:28.009Z level=INFO source=node_exporter.go:217 msg="Build context" build_context="(go=go1.23.7, platform=linux/amd64, user=root@7023beaa563a, date=20250401-15:19:01, tags=unknown)" 2026-02-23T13:07:28.107 INFO:journalctl@ceph.node-exporter.b.trial185.stdout:Feb 23 13:07:28 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-node-exporter-b[64620]: time=2026-02-23T13:07:28.009Z level=INFO source=diskstats_common.go:110 msg="Parsed flag --collector.diskstats.device-exclude" collector=diskstats flag=^(z?ram|loop|fd|(h|s|v|xv)d[a-z]|nvme\d+n\d+p)\d+$ 2026-02-23T13:07:28.107 INFO:journalctl@ceph.node-exporter.b.trial185.stdout:Feb 23 13:07:28 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-node-exporter-b[64620]: time=2026-02-23T13:07:28.009Z level=ERROR source=diskstats_linux.go:264 msg="Failed to open directory, disabling udev device properties" collector=diskstats path=/run/udev/data 2026-02-23T13:07:28.108 INFO:journalctl@ceph.node-exporter.b.trial185.stdout:Feb 23 13:07:28 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-node-exporter-b[64620]: time=2026-02-23T13:07:28.009Z level=INFO source=filesystem_common.go:265 msg="Parsed flag --collector.filesystem.mount-points-exclude" collector=filesystem flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+|var/lib/containers/storage/.+)($|/) 2026-02-23T13:07:28.108 INFO:journalctl@ceph.node-exporter.b.trial185.stdout:Feb 23 13:07:28 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-node-exporter-b[64620]: time=2026-02-23T13:07:28.009Z level=INFO source=filesystem_common.go:294 msg="Parsed flag --collector.filesystem.fs-types-exclude" collector=filesystem flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2026-02-23T13:07:28.108 INFO:journalctl@ceph.node-exporter.b.trial185.stdout:Feb 23 13:07:28 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-node-exporter-b[64620]: time=2026-02-23T13:07:28.010Z level=INFO source=node_exporter.go:135 msg="Enabled collectors" 2026-02-23T13:07:28.108 INFO:journalctl@ceph.node-exporter.b.trial185.stdout:Feb 23 13:07:28 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-node-exporter-b[64620]: time=2026-02-23T13:07:28.010Z level=INFO source=node_exporter.go:141 msg=arp 2026-02-23T13:07:28.108 INFO:journalctl@ceph.node-exporter.b.trial185.stdout:Feb 23 13:07:28 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-node-exporter-b[64620]: time=2026-02-23T13:07:28.010Z level=INFO source=node_exporter.go:141 msg=bcache 2026-02-23T13:07:28.108 INFO:journalctl@ceph.node-exporter.b.trial185.stdout:Feb 23 13:07:28 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-node-exporter-b[64620]: time=2026-02-23T13:07:28.010Z level=INFO source=node_exporter.go:141 msg=bonding 2026-02-23T13:07:28.108 INFO:journalctl@ceph.node-exporter.b.trial185.stdout:Feb 23 13:07:28 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-node-exporter-b[64620]: time=2026-02-23T13:07:28.010Z level=INFO source=node_exporter.go:141 msg=btrfs 2026-02-23T13:07:28.108 INFO:journalctl@ceph.node-exporter.b.trial185.stdout:Feb 23 13:07:28 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-node-exporter-b[64620]: time=2026-02-23T13:07:28.010Z level=INFO source=node_exporter.go:141 msg=conntrack 2026-02-23T13:07:28.108 INFO:journalctl@ceph.node-exporter.b.trial185.stdout:Feb 23 13:07:28 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-node-exporter-b[64620]: time=2026-02-23T13:07:28.010Z level=INFO source=node_exporter.go:141 msg=cpu 2026-02-23T13:07:28.108 INFO:journalctl@ceph.node-exporter.b.trial185.stdout:Feb 23 13:07:28 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-node-exporter-b[64620]: time=2026-02-23T13:07:28.010Z level=INFO source=node_exporter.go:141 msg=cpufreq 2026-02-23T13:07:28.108 INFO:journalctl@ceph.node-exporter.b.trial185.stdout:Feb 23 13:07:28 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-node-exporter-b[64620]: time=2026-02-23T13:07:28.010Z level=INFO source=node_exporter.go:141 msg=diskstats 2026-02-23T13:07:28.108 INFO:journalctl@ceph.node-exporter.b.trial185.stdout:Feb 23 13:07:28 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-node-exporter-b[64620]: time=2026-02-23T13:07:28.010Z level=INFO source=node_exporter.go:141 msg=dmi 2026-02-23T13:07:28.109 INFO:journalctl@ceph.node-exporter.b.trial185.stdout:Feb 23 13:07:28 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-node-exporter-b[64620]: time=2026-02-23T13:07:28.010Z level=INFO source=node_exporter.go:141 msg=edac 2026-02-23T13:07:28.109 INFO:journalctl@ceph.node-exporter.b.trial185.stdout:Feb 23 13:07:28 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-node-exporter-b[64620]: time=2026-02-23T13:07:28.010Z level=INFO source=node_exporter.go:141 msg=entropy 2026-02-23T13:07:28.109 INFO:journalctl@ceph.node-exporter.b.trial185.stdout:Feb 23 13:07:28 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-node-exporter-b[64620]: time=2026-02-23T13:07:28.010Z level=INFO source=node_exporter.go:141 msg=fibrechannel 2026-02-23T13:07:28.109 INFO:journalctl@ceph.node-exporter.b.trial185.stdout:Feb 23 13:07:28 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-node-exporter-b[64620]: time=2026-02-23T13:07:28.010Z level=INFO source=node_exporter.go:141 msg=filefd 2026-02-23T13:07:28.109 INFO:journalctl@ceph.node-exporter.b.trial185.stdout:Feb 23 13:07:28 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-node-exporter-b[64620]: time=2026-02-23T13:07:28.010Z level=INFO source=node_exporter.go:141 msg=filesystem 2026-02-23T13:07:28.109 INFO:journalctl@ceph.node-exporter.b.trial185.stdout:Feb 23 13:07:28 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-node-exporter-b[64620]: time=2026-02-23T13:07:28.010Z level=INFO source=node_exporter.go:141 msg=hwmon 2026-02-23T13:07:28.109 INFO:journalctl@ceph.node-exporter.b.trial185.stdout:Feb 23 13:07:28 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-node-exporter-b[64620]: time=2026-02-23T13:07:28.010Z level=INFO source=node_exporter.go:141 msg=infiniband 2026-02-23T13:07:28.109 INFO:journalctl@ceph.node-exporter.b.trial185.stdout:Feb 23 13:07:28 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-node-exporter-b[64620]: time=2026-02-23T13:07:28.010Z level=INFO source=node_exporter.go:141 msg=ipvs 2026-02-23T13:07:28.109 INFO:journalctl@ceph.node-exporter.b.trial185.stdout:Feb 23 13:07:28 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-node-exporter-b[64620]: time=2026-02-23T13:07:28.010Z level=INFO source=node_exporter.go:141 msg=loadavg 2026-02-23T13:07:28.109 INFO:journalctl@ceph.node-exporter.b.trial185.stdout:Feb 23 13:07:28 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-node-exporter-b[64620]: time=2026-02-23T13:07:28.010Z level=INFO source=node_exporter.go:141 msg=mdadm 2026-02-23T13:07:28.109 INFO:journalctl@ceph.node-exporter.b.trial185.stdout:Feb 23 13:07:28 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-node-exporter-b[64620]: time=2026-02-23T13:07:28.010Z level=INFO source=node_exporter.go:141 msg=meminfo 2026-02-23T13:07:28.109 INFO:journalctl@ceph.node-exporter.b.trial185.stdout:Feb 23 13:07:28 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-node-exporter-b[64620]: time=2026-02-23T13:07:28.010Z level=INFO source=node_exporter.go:141 msg=netclass 2026-02-23T13:07:28.109 INFO:journalctl@ceph.node-exporter.b.trial185.stdout:Feb 23 13:07:28 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-node-exporter-b[64620]: time=2026-02-23T13:07:28.010Z level=INFO source=node_exporter.go:141 msg=netdev 2026-02-23T13:07:28.110 INFO:journalctl@ceph.node-exporter.b.trial185.stdout:Feb 23 13:07:28 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-node-exporter-b[64620]: time=2026-02-23T13:07:28.010Z level=INFO source=node_exporter.go:141 msg=netstat 2026-02-23T13:07:28.110 INFO:journalctl@ceph.node-exporter.b.trial185.stdout:Feb 23 13:07:28 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-node-exporter-b[64620]: time=2026-02-23T13:07:28.010Z level=INFO source=node_exporter.go:141 msg=nfs 2026-02-23T13:07:28.110 INFO:journalctl@ceph.node-exporter.b.trial185.stdout:Feb 23 13:07:28 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-node-exporter-b[64620]: time=2026-02-23T13:07:28.010Z level=INFO source=node_exporter.go:141 msg=nfsd 2026-02-23T13:07:28.110 INFO:journalctl@ceph.node-exporter.b.trial185.stdout:Feb 23 13:07:28 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-node-exporter-b[64620]: time=2026-02-23T13:07:28.010Z level=INFO source=node_exporter.go:141 msg=nvme 2026-02-23T13:07:28.110 INFO:journalctl@ceph.node-exporter.b.trial185.stdout:Feb 23 13:07:28 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-node-exporter-b[64620]: time=2026-02-23T13:07:28.010Z level=INFO source=node_exporter.go:141 msg=os 2026-02-23T13:07:28.110 INFO:journalctl@ceph.node-exporter.b.trial185.stdout:Feb 23 13:07:28 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-node-exporter-b[64620]: time=2026-02-23T13:07:28.010Z level=INFO source=node_exporter.go:141 msg=powersupplyclass 2026-02-23T13:07:28.110 INFO:journalctl@ceph.node-exporter.b.trial185.stdout:Feb 23 13:07:28 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-node-exporter-b[64620]: time=2026-02-23T13:07:28.010Z level=INFO source=node_exporter.go:141 msg=pressure 2026-02-23T13:07:28.110 INFO:journalctl@ceph.node-exporter.b.trial185.stdout:Feb 23 13:07:28 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-node-exporter-b[64620]: time=2026-02-23T13:07:28.010Z level=INFO source=node_exporter.go:141 msg=rapl 2026-02-23T13:07:28.110 INFO:journalctl@ceph.node-exporter.b.trial185.stdout:Feb 23 13:07:28 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-node-exporter-b[64620]: time=2026-02-23T13:07:28.010Z level=INFO source=node_exporter.go:141 msg=schedstat 2026-02-23T13:07:28.110 INFO:journalctl@ceph.node-exporter.b.trial185.stdout:Feb 23 13:07:28 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-node-exporter-b[64620]: time=2026-02-23T13:07:28.010Z level=INFO source=node_exporter.go:141 msg=selinux 2026-02-23T13:07:28.110 INFO:journalctl@ceph.node-exporter.b.trial185.stdout:Feb 23 13:07:28 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-node-exporter-b[64620]: time=2026-02-23T13:07:28.010Z level=INFO source=node_exporter.go:141 msg=sockstat 2026-02-23T13:07:28.110 INFO:journalctl@ceph.node-exporter.b.trial185.stdout:Feb 23 13:07:28 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-node-exporter-b[64620]: time=2026-02-23T13:07:28.010Z level=INFO source=node_exporter.go:141 msg=softnet 2026-02-23T13:07:28.111 INFO:journalctl@ceph.node-exporter.b.trial185.stdout:Feb 23 13:07:28 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-node-exporter-b[64620]: time=2026-02-23T13:07:28.010Z level=INFO source=node_exporter.go:141 msg=stat 2026-02-23T13:07:28.111 INFO:journalctl@ceph.node-exporter.b.trial185.stdout:Feb 23 13:07:28 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-node-exporter-b[64620]: time=2026-02-23T13:07:28.010Z level=INFO source=node_exporter.go:141 msg=tapestats 2026-02-23T13:07:28.111 INFO:journalctl@ceph.node-exporter.b.trial185.stdout:Feb 23 13:07:28 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-node-exporter-b[64620]: time=2026-02-23T13:07:28.010Z level=INFO source=node_exporter.go:141 msg=textfile 2026-02-23T13:07:28.111 INFO:journalctl@ceph.node-exporter.b.trial185.stdout:Feb 23 13:07:28 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-node-exporter-b[64620]: time=2026-02-23T13:07:28.010Z level=INFO source=node_exporter.go:141 msg=thermal_zone 2026-02-23T13:07:28.111 INFO:journalctl@ceph.node-exporter.b.trial185.stdout:Feb 23 13:07:28 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-node-exporter-b[64620]: time=2026-02-23T13:07:28.010Z level=INFO source=node_exporter.go:141 msg=time 2026-02-23T13:07:28.111 INFO:journalctl@ceph.node-exporter.b.trial185.stdout:Feb 23 13:07:28 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-node-exporter-b[64620]: time=2026-02-23T13:07:28.010Z level=INFO source=node_exporter.go:141 msg=udp_queues 2026-02-23T13:07:28.111 INFO:journalctl@ceph.node-exporter.b.trial185.stdout:Feb 23 13:07:28 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-node-exporter-b[64620]: time=2026-02-23T13:07:28.010Z level=INFO source=node_exporter.go:141 msg=uname 2026-02-23T13:07:28.111 INFO:journalctl@ceph.node-exporter.b.trial185.stdout:Feb 23 13:07:28 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-node-exporter-b[64620]: time=2026-02-23T13:07:28.010Z level=INFO source=node_exporter.go:141 msg=vmstat 2026-02-23T13:07:28.111 INFO:journalctl@ceph.node-exporter.b.trial185.stdout:Feb 23 13:07:28 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-node-exporter-b[64620]: time=2026-02-23T13:07:28.010Z level=INFO source=node_exporter.go:141 msg=watchdog 2026-02-23T13:07:28.111 INFO:journalctl@ceph.node-exporter.b.trial185.stdout:Feb 23 13:07:28 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-node-exporter-b[64620]: time=2026-02-23T13:07:28.010Z level=INFO source=node_exporter.go:141 msg=xfs 2026-02-23T13:07:28.111 INFO:journalctl@ceph.node-exporter.b.trial185.stdout:Feb 23 13:07:28 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-node-exporter-b[64620]: time=2026-02-23T13:07:28.010Z level=INFO source=node_exporter.go:141 msg=zfs 2026-02-23T13:07:28.111 INFO:journalctl@ceph.node-exporter.b.trial185.stdout:Feb 23 13:07:28 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-node-exporter-b[64620]: time=2026-02-23T13:07:28.010Z level=INFO source=tls_config.go:347 msg="Listening on" address=[::]:9100 2026-02-23T13:07:28.112 INFO:journalctl@ceph.node-exporter.b.trial185.stdout:Feb 23 13:07:28 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-node-exporter-b[64620]: time=2026-02-23T13:07:28.010Z level=INFO source=tls_config.go:350 msg="TLS is disabled." http2=false address=[::]:9100 2026-02-23T13:07:28.144 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:07:28.142+0000 7fb74a509640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:07:28.144 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:07:28.143+0000 7fb74a509640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:07:28.258 INFO:teuthology.orchestra.run.trial068.stdout:epoch 28 2026-02-23T13:07:28.258 INFO:teuthology.orchestra.run.trial068.stdout:fsid 5f823084-10b8-11f1-be80-d404e6e7d460 2026-02-23T13:07:28.258 INFO:teuthology.orchestra.run.trial068.stdout:created 2026-02-23T13:06:03.098686+0000 2026-02-23T13:07:28.258 INFO:teuthology.orchestra.run.trial068.stdout:modified 2026-02-23T13:07:24.217971+0000 2026-02-23T13:07:28.258 INFO:teuthology.orchestra.run.trial068.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2026-02-23T13:07:28.258 INFO:teuthology.orchestra.run.trial068.stdout:crush_version 10 2026-02-23T13:07:28.258 INFO:teuthology.orchestra.run.trial068.stdout:full_ratio 0.95 2026-02-23T13:07:28.258 INFO:teuthology.orchestra.run.trial068.stdout:backfillfull_ratio 0.9 2026-02-23T13:07:28.259 INFO:teuthology.orchestra.run.trial068.stdout:nearfull_ratio 0.85 2026-02-23T13:07:28.259 INFO:teuthology.orchestra.run.trial068.stdout:require_min_compat_client luminous 2026-02-23T13:07:28.259 INFO:teuthology.orchestra.run.trial068.stdout:min_compat_client jewel 2026-02-23T13:07:28.259 INFO:teuthology.orchestra.run.trial068.stdout:require_osd_release tentacle 2026-02-23T13:07:28.259 INFO:teuthology.orchestra.run.trial068.stdout:stretch_mode_enabled false 2026-02-23T13:07:28.259 INFO:teuthology.orchestra.run.trial068.stdout:allow_crimson=true 2026-02-23T13:07:28.259 INFO:teuthology.orchestra.run.trial068.stdout:pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 22 flags hashpspool,crimson stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 4.00 2026-02-23T13:07:28.259 INFO:teuthology.orchestra.run.trial068.stdout:max_osd 4 2026-02-23T13:07:28.259 INFO:teuthology.orchestra.run.trial068.stdout:osd.0 up in weight 1 up_from 9 up_thru 25 down_at 0 last_clean_interval [0,0) [v2:10.20.193.68:6802/1216023594,v1:10.20.193.68:6803/1216023594] [v2:10.20.193.68:6804/1216023594,v1:10.20.193.68:6805/1216023594] exists,up 5e7d2db8-8070-4095-a077-d0f8420f7d9d 2026-02-23T13:07:28.259 INFO:teuthology.orchestra.run.trial068.stdout:osd.1 up in weight 1 up_from 13 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:10.20.193.68:6810/1086778192,v1:10.20.193.68:6811/1086778192] [v2:10.20.193.68:6812/1086778192,v1:10.20.193.68:6813/1086778192] exists,up c6854288-3ff8-4295-80a0-8fcaafedb28f 2026-02-23T13:07:28.259 INFO:teuthology.orchestra.run.trial068.stdout:osd.2 up in weight 1 up_from 18 up_thru 20 down_at 0 last_clean_interval [0,0) [v2:10.20.193.185:6800/1978920641,v1:10.20.193.185:6801/1978920641] [v2:10.20.193.185:6802/1978920641,v1:10.20.193.185:6803/1978920641] exists,up 25faf9eb-f475-49fe-8eeb-0663163402e8 2026-02-23T13:07:28.260 INFO:teuthology.orchestra.run.trial068.stdout:osd.3 up in weight 1 up_from 25 up_thru 26 down_at 0 last_clean_interval [0,0) [v2:10.20.193.185:6808/3135022860,v1:10.20.193.185:6809/3135022860] [v2:10.20.193.185:6810/3135022860,v1:10.20.193.185:6811/3135022860] exists,up e606eee6-fc74-4c50-bfdc-a6a2adf9a945 2026-02-23T13:07:28.260 INFO:teuthology.orchestra.run.trial068.stdout:blocklist 10.20.193.68:0/2086895042 expires 2026-02-24T13:07:24.217938+0000 2026-02-23T13:07:28.260 INFO:teuthology.orchestra.run.trial068.stdout:blocklist 10.20.193.68:6801/2332248704 expires 2026-02-24T13:07:24.217938+0000 2026-02-23T13:07:28.260 INFO:teuthology.orchestra.run.trial068.stdout:blocklist 10.20.193.68:0/1209278317 expires 2026-02-24T13:06:12.960687+0000 2026-02-23T13:07:28.260 INFO:teuthology.orchestra.run.trial068.stdout:blocklist 10.20.193.68:0/3495130282 expires 2026-02-24T13:06:22.483019+0000 2026-02-23T13:07:28.260 INFO:teuthology.orchestra.run.trial068.stdout:blocklist 10.20.193.68:6800/2061884286 expires 2026-02-24T13:06:12.960687+0000 2026-02-23T13:07:28.260 INFO:teuthology.orchestra.run.trial068.stdout:blocklist 10.20.193.68:6801/2061884286 expires 2026-02-24T13:06:12.960687+0000 2026-02-23T13:07:28.260 INFO:teuthology.orchestra.run.trial068.stdout:blocklist 10.20.193.68:0/3103836131 expires 2026-02-24T13:06:12.960687+0000 2026-02-23T13:07:28.260 INFO:teuthology.orchestra.run.trial068.stdout:blocklist 10.20.193.68:0/2255147122 expires 2026-02-24T13:06:12.960687+0000 2026-02-23T13:07:28.260 INFO:teuthology.orchestra.run.trial068.stdout:blocklist 10.20.193.68:0/2848886808 expires 2026-02-24T13:07:24.217938+0000 2026-02-23T13:07:28.260 INFO:teuthology.orchestra.run.trial068.stdout:blocklist 10.20.193.68:6800/1870073644 expires 2026-02-24T13:06:22.483019+0000 2026-02-23T13:07:28.260 INFO:teuthology.orchestra.run.trial068.stdout:blocklist 10.20.193.68:0/55681093 expires 2026-02-24T13:06:22.483019+0000 2026-02-23T13:07:28.260 INFO:teuthology.orchestra.run.trial068.stdout:blocklist 10.20.193.68:6800/2332248704 expires 2026-02-24T13:07:24.217938+0000 2026-02-23T13:07:28.261 INFO:teuthology.orchestra.run.trial068.stdout:blocklist 10.20.193.68:6801/1870073644 expires 2026-02-24T13:06:22.483019+0000 2026-02-23T13:07:28.261 INFO:teuthology.orchestra.run.trial068.stdout:blocklist 10.20.193.68:0/1895012493 expires 2026-02-24T13:07:24.217938+0000 2026-02-23T13:07:28.261 INFO:teuthology.orchestra.run.trial068.stdout:blocklist 10.20.193.68:0/2910016328 expires 2026-02-24T13:07:24.217938+0000 2026-02-23T13:07:28.261 INFO:teuthology.orchestra.run.trial068.stdout:blocklist 10.20.193.68:0/2415220450 expires 2026-02-24T13:06:22.483019+0000 2026-02-23T13:07:28.297 INFO:tasks.cephadm.ceph_manager.ceph:['osd.0 up in weight 1 up_from 9 up_thru 25 down_at 0 last_clean_interval [0,0) [v2:10.20.193.68:6802/1216023594,v1:10.20.193.68:6803/1216023594] [v2:10.20.193.68:6804/1216023594,v1:10.20.193.68:6805/1216023594] exists,up 5e7d2db8-8070-4095-a077-d0f8420f7d9d', 'osd.1 up in weight 1 up_from 13 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:10.20.193.68:6810/1086778192,v1:10.20.193.68:6811/1086778192] [v2:10.20.193.68:6812/1086778192,v1:10.20.193.68:6813/1086778192] exists,up c6854288-3ff8-4295-80a0-8fcaafedb28f', 'osd.2 up in weight 1 up_from 18 up_thru 20 down_at 0 last_clean_interval [0,0) [v2:10.20.193.185:6800/1978920641,v1:10.20.193.185:6801/1978920641] [v2:10.20.193.185:6802/1978920641,v1:10.20.193.185:6803/1978920641] exists,up 25faf9eb-f475-49fe-8eeb-0663163402e8', 'osd.3 up in weight 1 up_from 25 up_thru 26 down_at 0 last_clean_interval [0,0) [v2:10.20.193.185:6808/3135022860,v1:10.20.193.185:6809/3135022860] [v2:10.20.193.185:6810/3135022860,v1:10.20.193.185:6811/3135022860] exists,up e606eee6-fc74-4c50-bfdc-a6a2adf9a945'] 2026-02-23T13:07:28.298 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph daemon mon.a config show 2026-02-23T13:07:28.319 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:28 trial068 ceph-mon[38805]: from='client.14322 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2026-02-23T13:07:28.319 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:28 trial068 ceph-mon[38805]: Deploying daemon node-exporter.b on trial185 2026-02-23T13:07:28.319 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:28 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "orch get-security-config"} : dispatch 2026-02-23T13:07:28.320 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:28 trial068 ceph-mon[38805]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "orch get-security-config"}]: dispatch 2026-02-23T13:07:28.320 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:28 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T13:07:28.320 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:28 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T13:07:28.320 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:28 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T13:07:28.320 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:28 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T13:07:28.320 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:28 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T13:07:28.320 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:28 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2026-02-23T13:07:28.413 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:07:28.529 INFO:teuthology.orchestra.run.trial068.stdout:{ 2026-02-23T13:07:28.530 INFO:teuthology.orchestra.run.trial068.stdout: "name": "mon.a", 2026-02-23T13:07:28.530 INFO:teuthology.orchestra.run.trial068.stdout: "cluster": "ceph", 2026-02-23T13:07:28.530 INFO:teuthology.orchestra.run.trial068.stdout: "admin_socket": "/var/run/ceph/ceph-mon.a.asok", 2026-02-23T13:07:28.530 INFO:teuthology.orchestra.run.trial068.stdout: "admin_socket_mode": "", 2026-02-23T13:07:28.530 INFO:teuthology.orchestra.run.trial068.stdout: "auth_allow_insecure_global_id_reclaim": "false", 2026-02-23T13:07:28.531 INFO:teuthology.orchestra.run.trial068.stdout: "auth_client_required": "cephx, none", 2026-02-23T13:07:28.531 INFO:teuthology.orchestra.run.trial068.stdout: "auth_cluster_required": "cephx", 2026-02-23T13:07:28.531 INFO:teuthology.orchestra.run.trial068.stdout: "auth_debug": "true", 2026-02-23T13:07:28.531 INFO:teuthology.orchestra.run.trial068.stdout: "auth_expose_insecure_global_id_reclaim": "true", 2026-02-23T13:07:28.531 INFO:teuthology.orchestra.run.trial068.stdout: "auth_mon_ticket_ttl": "660.000000", 2026-02-23T13:07:28.531 INFO:teuthology.orchestra.run.trial068.stdout: "auth_service_required": "cephx", 2026-02-23T13:07:28.531 INFO:teuthology.orchestra.run.trial068.stdout: "auth_service_ticket_ttl": "240.000000", 2026-02-23T13:07:28.531 INFO:teuthology.orchestra.run.trial068.stdout: "auth_supported": "", 2026-02-23T13:07:28.531 INFO:teuthology.orchestra.run.trial068.stdout: "bdev_aio": "true", 2026-02-23T13:07:28.531 INFO:teuthology.orchestra.run.trial068.stdout: "bdev_aio_max_queue_depth": "1024", 2026-02-23T13:07:28.531 INFO:teuthology.orchestra.run.trial068.stdout: "bdev_aio_poll_ms": "250", 2026-02-23T13:07:28.532 INFO:teuthology.orchestra.run.trial068.stdout: "bdev_aio_reap_max": "16", 2026-02-23T13:07:28.532 INFO:teuthology.orchestra.run.trial068.stdout: "bdev_aio_submit_retry_initial_delay_us": "125", 2026-02-23T13:07:28.532 INFO:teuthology.orchestra.run.trial068.stdout: "bdev_aio_submit_retry_max": "16", 2026-02-23T13:07:28.532 INFO:teuthology.orchestra.run.trial068.stdout: "bdev_async_discard": "false", 2026-02-23T13:07:28.532 INFO:teuthology.orchestra.run.trial068.stdout: "bdev_async_discard_max_pending": "1000000", 2026-02-23T13:07:28.532 INFO:teuthology.orchestra.run.trial068.stdout: "bdev_async_discard_threads": "0", 2026-02-23T13:07:28.532 INFO:teuthology.orchestra.run.trial068.stdout: "bdev_block_size": "4096", 2026-02-23T13:07:28.532 INFO:teuthology.orchestra.run.trial068.stdout: "bdev_debug_aio": "false", 2026-02-23T13:07:28.532 INFO:teuthology.orchestra.run.trial068.stdout: "bdev_debug_aio_log_age": "5.000000", 2026-02-23T13:07:28.532 INFO:teuthology.orchestra.run.trial068.stdout: "bdev_debug_aio_suicide_timeout": "60.000000", 2026-02-23T13:07:28.532 INFO:teuthology.orchestra.run.trial068.stdout: "bdev_debug_discard_sleep": "0", 2026-02-23T13:07:28.533 INFO:teuthology.orchestra.run.trial068.stdout: "bdev_debug_inflight_ios": "false", 2026-02-23T13:07:28.533 INFO:teuthology.orchestra.run.trial068.stdout: "bdev_discard_max_bytes": "10737418240", 2026-02-23T13:07:28.533 INFO:teuthology.orchestra.run.trial068.stdout: "bdev_enable_discard": "false", 2026-02-23T13:07:28.533 INFO:teuthology.orchestra.run.trial068.stdout: "bdev_flock_retry": "3", 2026-02-23T13:07:28.533 INFO:teuthology.orchestra.run.trial068.stdout: "bdev_flock_retry_interval": "0.100000", 2026-02-23T13:07:28.533 INFO:teuthology.orchestra.run.trial068.stdout: "bdev_inject_crash": "0", 2026-02-23T13:07:28.533 INFO:teuthology.orchestra.run.trial068.stdout: "bdev_inject_crash_flush_delay": "2", 2026-02-23T13:07:28.533 INFO:teuthology.orchestra.run.trial068.stdout: "bdev_ioring": "false", 2026-02-23T13:07:28.533 INFO:teuthology.orchestra.run.trial068.stdout: "bdev_ioring_hipri": "false", 2026-02-23T13:07:28.533 INFO:teuthology.orchestra.run.trial068.stdout: "bdev_ioring_sqthread_poll": "false", 2026-02-23T13:07:28.533 INFO:teuthology.orchestra.run.trial068.stdout: "bdev_max_discard_length": "2147483648", 2026-02-23T13:07:28.533 INFO:teuthology.orchestra.run.trial068.stdout: "bdev_nvme_unbind_from_kernel": "false", 2026-02-23T13:07:28.534 INFO:teuthology.orchestra.run.trial068.stdout: "bdev_read_buffer_alignment": "4096", 2026-02-23T13:07:28.534 INFO:teuthology.orchestra.run.trial068.stdout: "bdev_read_preallocated_huge_buffers": "", 2026-02-23T13:07:28.534 INFO:teuthology.orchestra.run.trial068.stdout: "bdev_stalled_read_warn_lifetime": "86400", 2026-02-23T13:07:28.534 INFO:teuthology.orchestra.run.trial068.stdout: "bdev_stalled_read_warn_threshold": "1", 2026-02-23T13:07:28.534 INFO:teuthology.orchestra.run.trial068.stdout: "bdev_type": "", 2026-02-23T13:07:28.534 INFO:teuthology.orchestra.run.trial068.stdout: "bluefs_alloc_size": "1048576", 2026-02-23T13:07:28.534 INFO:teuthology.orchestra.run.trial068.stdout: "bluefs_allocator": "hybrid", 2026-02-23T13:07:28.534 INFO:teuthology.orchestra.run.trial068.stdout: "bluefs_buffered_io": "true", 2026-02-23T13:07:28.534 INFO:teuthology.orchestra.run.trial068.stdout: "bluefs_check_for_zeros": "false", 2026-02-23T13:07:28.534 INFO:teuthology.orchestra.run.trial068.stdout: "bluefs_check_volume_selector_often": "false", 2026-02-23T13:07:28.534 INFO:teuthology.orchestra.run.trial068.stdout: "bluefs_check_volume_selector_on_umount": "false", 2026-02-23T13:07:28.535 INFO:teuthology.orchestra.run.trial068.stdout: "bluefs_compact_log_sync": "false", 2026-02-23T13:07:28.535 INFO:teuthology.orchestra.run.trial068.stdout: "bluefs_failed_shared_alloc_cooldown": "600.000000", 2026-02-23T13:07:28.535 INFO:teuthology.orchestra.run.trial068.stdout: "bluefs_log_compact_min_ratio": "5.000000", 2026-02-23T13:07:28.535 INFO:teuthology.orchestra.run.trial068.stdout: "bluefs_log_compact_min_size": "16777216", 2026-02-23T13:07:28.535 INFO:teuthology.orchestra.run.trial068.stdout: "bluefs_log_replay_check_allocations": "true", 2026-02-23T13:07:28.535 INFO:teuthology.orchestra.run.trial068.stdout: "bluefs_max_log_runway": "4194304", 2026-02-23T13:07:28.535 INFO:teuthology.orchestra.run.trial068.stdout: "bluefs_max_prefetch": "1048576", 2026-02-23T13:07:28.535 INFO:teuthology.orchestra.run.trial068.stdout: "bluefs_min_flush_size": "524288", 2026-02-23T13:07:28.535 INFO:teuthology.orchestra.run.trial068.stdout: "bluefs_min_log_runway": "1048576", 2026-02-23T13:07:28.535 INFO:teuthology.orchestra.run.trial068.stdout: "bluefs_replay_recovery": "false", 2026-02-23T13:07:28.535 INFO:teuthology.orchestra.run.trial068.stdout: "bluefs_replay_recovery_disable_compact": "false", 2026-02-23T13:07:28.536 INFO:teuthology.orchestra.run.trial068.stdout: "bluefs_shared_alloc_size": "65536", 2026-02-23T13:07:28.536 INFO:teuthology.orchestra.run.trial068.stdout: "bluefs_sync_write": "false", 2026-02-23T13:07:28.536 INFO:teuthology.orchestra.run.trial068.stdout: "bluefs_wal_envelope_mode": "true", 2026-02-23T13:07:28.536 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_2q_cache_kin_ratio": "0.500000", 2026-02-23T13:07:28.536 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_2q_cache_kout_ratio": "0.500000", 2026-02-23T13:07:28.536 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_alloc_stats_dump_interval": "86400.000000", 2026-02-23T13:07:28.536 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_allocation_from_file": "true", 2026-02-23T13:07:28.536 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_allocator": "hybrid", 2026-02-23T13:07:28.536 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_allocator_lookup_policy": "auto", 2026-02-23T13:07:28.536 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_async_db_compaction": "true", 2026-02-23T13:07:28.536 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_avl_alloc_bf_free_pct": "4", 2026-02-23T13:07:28.537 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_avl_alloc_bf_threshold": "131072", 2026-02-23T13:07:28.537 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_avl_alloc_ff_max_search_bytes": "16777216", 2026-02-23T13:07:28.537 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_avl_alloc_ff_max_search_count": "100", 2026-02-23T13:07:28.537 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_bdev_label_multi": "true", 2026-02-23T13:07:28.537 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_bdev_label_multi_upgrade": "false", 2026-02-23T13:07:28.537 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_bdev_label_require_all": "true", 2026-02-23T13:07:28.537 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_bitmapallocator_blocks_per_zone": "1024", 2026-02-23T13:07:28.537 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_bitmapallocator_span_size": "1024", 2026-02-23T13:07:28.537 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_blobid_prealloc": "10240", 2026-02-23T13:07:28.537 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_block_create": "true", 2026-02-23T13:07:28.537 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_block_db_create": "false", 2026-02-23T13:07:28.537 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_block_db_path": "", 2026-02-23T13:07:28.538 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_block_db_size": "0", 2026-02-23T13:07:28.538 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_block_path": "", 2026-02-23T13:07:28.538 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_block_preallocate_file": "false", 2026-02-23T13:07:28.538 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_block_size": "107374182400", 2026-02-23T13:07:28.538 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_block_wal_create": "false", 2026-02-23T13:07:28.538 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_block_wal_path": "", 2026-02-23T13:07:28.538 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_block_wal_size": "100663296", 2026-02-23T13:07:28.538 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_bluefs": "true", 2026-02-23T13:07:28.538 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_bluefs_alloc_failure_dump_interval": "0.000000", 2026-02-23T13:07:28.538 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_bluefs_env_mirror": "false", 2026-02-23T13:07:28.538 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_bluefs_max_free": "10737418240", 2026-02-23T13:07:28.538 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_btree2_alloc_weight_factor": "2.000000", 2026-02-23T13:07:28.539 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_cache_age_bin_interval": "1.000000", 2026-02-23T13:07:28.539 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_cache_age_bins_data": "1 2 6 24 120 720 0 0 0 0", 2026-02-23T13:07:28.539 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_cache_age_bins_kv": "1 2 6 24 120 720 0 0 0 0", 2026-02-23T13:07:28.539 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_cache_age_bins_kv_onode": "0 0 0 0 0 0 0 0 0 720", 2026-02-23T13:07:28.539 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_cache_age_bins_meta": "1 2 6 24 120 720 0 0 0 0", 2026-02-23T13:07:28.539 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_cache_autotune": "true", 2026-02-23T13:07:28.539 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_cache_autotune_interval": "5.000000", 2026-02-23T13:07:28.539 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_cache_kv_onode_ratio": "0.040000", 2026-02-23T13:07:28.539 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_cache_kv_ratio": "0.450000", 2026-02-23T13:07:28.539 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_cache_meta_ratio": "0.450000", 2026-02-23T13:07:28.539 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_cache_size": "0", 2026-02-23T13:07:28.540 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_cache_size_hdd": "1073741824", 2026-02-23T13:07:28.540 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_cache_size_ssd": "3221225472", 2026-02-23T13:07:28.540 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_cache_trim_interval": "0.050000", 2026-02-23T13:07:28.540 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_cache_trim_max_skip_pinned": "1000", 2026-02-23T13:07:28.540 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_cache_type": "2q", 2026-02-23T13:07:28.540 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_cleaner_sleep_interval": "5.000000", 2026-02-23T13:07:28.540 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_clone_cow": "true", 2026-02-23T13:07:28.540 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_compression_algorithm": "snappy", 2026-02-23T13:07:28.540 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_compression_max_blob_size": "0", 2026-02-23T13:07:28.540 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_compression_max_blob_size_hdd": "65536", 2026-02-23T13:07:28.540 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_compression_max_blob_size_ssd": "65536", 2026-02-23T13:07:28.540 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_compression_min_blob_size": "0", 2026-02-23T13:07:28.541 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_compression_min_blob_size_hdd": "8192", 2026-02-23T13:07:28.541 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_compression_min_blob_size_ssd": "65536", 2026-02-23T13:07:28.541 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_compression_mode": "none", 2026-02-23T13:07:28.541 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_compression_required_ratio": "0.875000", 2026-02-23T13:07:28.541 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_csum_type": "crc32c", 2026-02-23T13:07:28.541 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_debug_enforce_min_alloc_size": "0", 2026-02-23T13:07:28.541 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_debug_enforce_settings": "default", 2026-02-23T13:07:28.541 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_debug_extent_map_encode_check": "false", 2026-02-23T13:07:28.541 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_debug_freelist": "false", 2026-02-23T13:07:28.541 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_debug_fsck_abort": "false", 2026-02-23T13:07:28.541 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_debug_inject_allocation_from_file_failure": "0.000000", 2026-02-23T13:07:28.541 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_debug_inject_csum_err_probability": "0.000000", 2026-02-23T13:07:28.542 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_debug_inject_read_err": "false", 2026-02-23T13:07:28.542 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_debug_legacy_omap": "false", 2026-02-23T13:07:28.542 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_debug_no_reuse_blocks": "false", 2026-02-23T13:07:28.542 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_debug_omit_block_device_write": "false", 2026-02-23T13:07:28.542 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_debug_omit_kv_commit": "false", 2026-02-23T13:07:28.542 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_debug_onode_segmentation_random": "false", 2026-02-23T13:07:28.542 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_debug_permit_any_bdev_label": "false", 2026-02-23T13:07:28.542 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_debug_prefragment_max": "1048576", 2026-02-23T13:07:28.542 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_debug_random_read_err": "0.000000", 2026-02-23T13:07:28.542 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_debug_randomize_serial_transaction": "0", 2026-02-23T13:07:28.542 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_debug_small_allocations": "0", 2026-02-23T13:07:28.542 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_debug_too_many_blobs_threshold": "24576", 2026-02-23T13:07:28.543 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_default_buffered_read": "true", 2026-02-23T13:07:28.543 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_default_buffered_write": "false", 2026-02-23T13:07:28.543 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_deferred_batch_ops": "0", 2026-02-23T13:07:28.543 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_deferred_batch_ops_hdd": "64", 2026-02-23T13:07:28.543 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_deferred_batch_ops_ssd": "16", 2026-02-23T13:07:28.543 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_discard_on_mkfs": "true", 2026-02-23T13:07:28.543 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_elastic_shared_blobs": "true", 2026-02-23T13:07:28.543 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_extent_map_inline_shard_prealloc_size": "256", 2026-02-23T13:07:28.543 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_extent_map_shard_max_size": "1200", 2026-02-23T13:07:28.543 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_extent_map_shard_min_size": "150", 2026-02-23T13:07:28.543 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_extent_map_shard_target_size": "500", 2026-02-23T13:07:28.544 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_extent_map_shard_target_size_slop": "0.200000", 2026-02-23T13:07:28.544 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_fail_eio": "false", 2026-02-23T13:07:28.544 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_fragmentation_check_period": "3600", 2026-02-23T13:07:28.544 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_freelist_blocks_per_key": "128", 2026-02-23T13:07:28.544 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_fsck_error_on_no_per_pg_omap": "false", 2026-02-23T13:07:28.544 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_fsck_error_on_no_per_pool_omap": "false", 2026-02-23T13:07:28.544 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_fsck_error_on_no_per_pool_stats": "false", 2026-02-23T13:07:28.544 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_fsck_on_mkfs": "true", 2026-02-23T13:07:28.544 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_fsck_on_mkfs_deep": "false", 2026-02-23T13:07:28.544 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_fsck_on_mount": "false", 2026-02-23T13:07:28.544 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_fsck_on_mount_deep": "false", 2026-02-23T13:07:28.544 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_fsck_on_umount": "false", 2026-02-23T13:07:28.545 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_fsck_on_umount_deep": "false", 2026-02-23T13:07:28.545 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_fsck_quick_fix_on_mount": "false", 2026-02-23T13:07:28.545 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_fsck_quick_fix_threads": "2", 2026-02-23T13:07:28.545 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_fsck_read_bytes_cap": "67108864", 2026-02-23T13:07:28.545 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_fsck_shared_blob_tracker_size": "0.031250", 2026-02-23T13:07:28.545 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_gc_enable_blob_threshold": "0", 2026-02-23T13:07:28.545 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_gc_enable_total_threshold": "0", 2026-02-23T13:07:28.545 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_hybrid_alloc_mem_cap": "67108864", 2026-02-23T13:07:28.545 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_ignore_data_csum": "false", 2026-02-23T13:07:28.545 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_kv_sync_util_logging_s": "10.000000", 2026-02-23T13:07:28.545 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_kvbackend": "rocksdb", 2026-02-23T13:07:28.545 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_log_collection_list_age": "60.000000", 2026-02-23T13:07:28.546 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_log_omap_iterator_age": "5.000000", 2026-02-23T13:07:28.546 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_log_op_age": "5.000000", 2026-02-23T13:07:28.547 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_max_alloc_size": "0", 2026-02-23T13:07:28.547 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_max_blob_size": "0", 2026-02-23T13:07:28.548 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_max_blob_size_hdd": "65536", 2026-02-23T13:07:28.548 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_max_blob_size_ssd": "65536", 2026-02-23T13:07:28.548 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_max_defer_interval": "3.000000", 2026-02-23T13:07:28.548 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_max_deferred_txc": "32", 2026-02-23T13:07:28.548 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_min_alloc_size": "0", 2026-02-23T13:07:28.548 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_min_alloc_size_hdd": "4096", 2026-02-23T13:07:28.548 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_min_alloc_size_ssd": "4096", 2026-02-23T13:07:28.548 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_nid_prealloc": "1024", 2026-02-23T13:07:28.548 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_onode_segment_size": "0", 2026-02-23T13:07:28.548 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_prefer_deferred_size": "0", 2026-02-23T13:07:28.548 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_prefer_deferred_size_hdd": "65536", 2026-02-23T13:07:28.549 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_prefer_deferred_size_ssd": "0", 2026-02-23T13:07:28.549 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_qfsck_on_mount": "true", 2026-02-23T13:07:28.549 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_recompression_min_gain": "1.200000", 2026-02-23T13:07:28.549 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_retry_disk_reads": "3", 2026-02-23T13:07:28.549 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_rocksdb_cf": "true", 2026-02-23T13:07:28.549 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_rocksdb_cfs": "m(3) p(3,0-12) O(3,0-13)=block_cache={type=binned_lru} L=min_write_buffer_number_to_merge=32 P=min_write_buffer_number_to_merge=32", 2026-02-23T13:07:28.549 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_rocksdb_options": "compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0", 2026-02-23T13:07:28.549 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_rocksdb_options_annex": "", 2026-02-23T13:07:28.549 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_slow_ops_warn_lifetime": "86400", 2026-02-23T13:07:28.549 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_slow_ops_warn_threshold": "1", 2026-02-23T13:07:28.549 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_spdk_coremask": "0x1", 2026-02-23T13:07:28.550 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_spdk_io_sleep": "5", 2026-02-23T13:07:28.550 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_spdk_max_io_completion": "0", 2026-02-23T13:07:28.550 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_spdk_mem": "512", 2026-02-23T13:07:28.550 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_sync_submit_transaction": "false", 2026-02-23T13:07:28.550 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_throttle_bytes": "67108864", 2026-02-23T13:07:28.550 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_throttle_cost_per_io": "0", 2026-02-23T13:07:28.550 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_throttle_cost_per_io_hdd": "670000", 2026-02-23T13:07:28.550 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_throttle_cost_per_io_ssd": "4000", 2026-02-23T13:07:28.550 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_throttle_deferred_bytes": "134217728", 2026-02-23T13:07:28.550 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_throttle_trace_rate": "0.000000", 2026-02-23T13:07:28.550 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_tracing": "false", 2026-02-23T13:07:28.551 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_use_optimal_io_size_for_min_alloc_size": "false", 2026-02-23T13:07:28.551 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_volume_selection_policy": "use_some_extra", 2026-02-23T13:07:28.551 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_volume_selection_reserved": "0", 2026-02-23T13:07:28.551 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_volume_selection_reserved_factor": "2.000000", 2026-02-23T13:07:28.551 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_warn_on_bluefs_spillover": "true", 2026-02-23T13:07:28.551 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_warn_on_free_fragmentation": "0.800000", 2026-02-23T13:07:28.551 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_warn_on_legacy_statfs": "true", 2026-02-23T13:07:28.551 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_warn_on_no_per_pg_omap": "false", 2026-02-23T13:07:28.551 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_warn_on_no_per_pool_omap": "true", 2026-02-23T13:07:28.551 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_warn_on_spurious_read_errors": "true", 2026-02-23T13:07:28.551 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_write_v2": "false", 2026-02-23T13:07:28.551 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_write_v2_random": "false", 2026-02-23T13:07:28.552 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_zero_block_detection": "false", 2026-02-23T13:07:28.552 INFO:teuthology.orchestra.run.trial068.stdout: "breakpad": "false", 2026-02-23T13:07:28.552 INFO:teuthology.orchestra.run.trial068.stdout: "ceph_assert_supresssions": "", 2026-02-23T13:07:28.552 INFO:teuthology.orchestra.run.trial068.stdout: "cephadm_path": "/usr/sbin/cephadm", 2026-02-23T13:07:28.552 INFO:teuthology.orchestra.run.trial068.stdout: "cephfs_mirror_action_update_interval": "2", 2026-02-23T13:07:28.552 INFO:teuthology.orchestra.run.trial068.stdout: "cephfs_mirror_directory_scan_interval": "10", 2026-02-23T13:07:28.552 INFO:teuthology.orchestra.run.trial068.stdout: "cephfs_mirror_max_concurrent_directory_syncs": "3", 2026-02-23T13:07:28.552 INFO:teuthology.orchestra.run.trial068.stdout: "cephfs_mirror_max_consecutive_failures_per_directory": "10", 2026-02-23T13:07:28.552 INFO:teuthology.orchestra.run.trial068.stdout: "cephfs_mirror_max_snapshot_sync_per_cycle": "3", 2026-02-23T13:07:28.552 INFO:teuthology.orchestra.run.trial068.stdout: "cephfs_mirror_mount_timeout": "10", 2026-02-23T13:07:28.552 INFO:teuthology.orchestra.run.trial068.stdout: "cephfs_mirror_perf_stats_prio": "5", 2026-02-23T13:07:28.553 INFO:teuthology.orchestra.run.trial068.stdout: "cephfs_mirror_restart_mirror_on_blocklist_interval": "30", 2026-02-23T13:07:28.553 INFO:teuthology.orchestra.run.trial068.stdout: "cephfs_mirror_restart_mirror_on_failure_interval": "20", 2026-02-23T13:07:28.553 INFO:teuthology.orchestra.run.trial068.stdout: "cephfs_mirror_retry_failed_directories_interval": "60", 2026-02-23T13:07:28.553 INFO:teuthology.orchestra.run.trial068.stdout: "cephsqlite_blocklist_dead_locker": "true", 2026-02-23T13:07:28.553 INFO:teuthology.orchestra.run.trial068.stdout: "cephsqlite_lock_renewal_interval": "2000", 2026-02-23T13:07:28.553 INFO:teuthology.orchestra.run.trial068.stdout: "cephsqlite_lock_renewal_timeout": "30000", 2026-02-23T13:07:28.553 INFO:teuthology.orchestra.run.trial068.stdout: "cephx_cluster_require_signatures": "false", 2026-02-23T13:07:28.553 INFO:teuthology.orchestra.run.trial068.stdout: "cephx_cluster_require_version": "2", 2026-02-23T13:07:28.553 INFO:teuthology.orchestra.run.trial068.stdout: "cephx_require_signatures": "false", 2026-02-23T13:07:28.553 INFO:teuthology.orchestra.run.trial068.stdout: "cephx_require_version": "2", 2026-02-23T13:07:28.553 INFO:teuthology.orchestra.run.trial068.stdout: "cephx_service_require_signatures": "false", 2026-02-23T13:07:28.554 INFO:teuthology.orchestra.run.trial068.stdout: "cephx_service_require_version": "2", 2026-02-23T13:07:28.554 INFO:teuthology.orchestra.run.trial068.stdout: "cephx_sign_messages": "true", 2026-02-23T13:07:28.554 INFO:teuthology.orchestra.run.trial068.stdout: "chdir": "", 2026-02-23T13:07:28.554 INFO:teuthology.orchestra.run.trial068.stdout: "client_acl_type": "", 2026-02-23T13:07:28.554 INFO:teuthology.orchestra.run.trial068.stdout: "client_asio_thread_count": "2", 2026-02-23T13:07:28.554 INFO:teuthology.orchestra.run.trial068.stdout: "client_cache_mid": "0.750000", 2026-02-23T13:07:28.554 INFO:teuthology.orchestra.run.trial068.stdout: "client_cache_size": "16384", 2026-02-23T13:07:28.554 INFO:teuthology.orchestra.run.trial068.stdout: "client_caps_release_delay": "5", 2026-02-23T13:07:28.554 INFO:teuthology.orchestra.run.trial068.stdout: "client_check_pool_perm": "true", 2026-02-23T13:07:28.554 INFO:teuthology.orchestra.run.trial068.stdout: "client_collect_and_send_global_metrics": "false", 2026-02-23T13:07:28.554 INFO:teuthology.orchestra.run.trial068.stdout: "client_debug_force_sync_read": "false", 2026-02-23T13:07:28.555 INFO:teuthology.orchestra.run.trial068.stdout: "client_debug_getattr_caps": "false", 2026-02-23T13:07:28.555 INFO:teuthology.orchestra.run.trial068.stdout: "client_debug_inject_features": "", 2026-02-23T13:07:28.555 INFO:teuthology.orchestra.run.trial068.stdout: "client_debug_inject_tick_delay": "0", 2026-02-23T13:07:28.555 INFO:teuthology.orchestra.run.trial068.stdout: "client_die_on_failed_dentry_invalidate": "true", 2026-02-23T13:07:28.555 INFO:teuthology.orchestra.run.trial068.stdout: "client_die_on_failed_remount": "false", 2026-02-23T13:07:28.555 INFO:teuthology.orchestra.run.trial068.stdout: "client_dirsize_rbytes": "true", 2026-02-23T13:07:28.555 INFO:teuthology.orchestra.run.trial068.stdout: "client_file_blockdiff_max_concurrent_object_scans": "16", 2026-02-23T13:07:28.555 INFO:teuthology.orchestra.run.trial068.stdout: "client_force_lazyio": "false", 2026-02-23T13:07:28.555 INFO:teuthology.orchestra.run.trial068.stdout: "client_fs": "", 2026-02-23T13:07:28.555 INFO:teuthology.orchestra.run.trial068.stdout: "client_fscrypt_as": "true", 2026-02-23T13:07:28.555 INFO:teuthology.orchestra.run.trial068.stdout: "client_fscrypt_dummy_encryption": "false", 2026-02-23T13:07:28.555 INFO:teuthology.orchestra.run.trial068.stdout: "client_inject_fixed_oldest_tid": "false", 2026-02-23T13:07:28.556 INFO:teuthology.orchestra.run.trial068.stdout: "client_inject_release_failure": "false", 2026-02-23T13:07:28.556 INFO:teuthology.orchestra.run.trial068.stdout: "client_inject_write_delay_secs": "0", 2026-02-23T13:07:28.556 INFO:teuthology.orchestra.run.trial068.stdout: "client_max_inline_size": "4096", 2026-02-23T13:07:28.556 INFO:teuthology.orchestra.run.trial068.stdout: "client_max_retries_on_remount_failure": "5", 2026-02-23T13:07:28.556 INFO:teuthology.orchestra.run.trial068.stdout: "client_mds_namespace": "", 2026-02-23T13:07:28.556 INFO:teuthology.orchestra.run.trial068.stdout: "client_metadata": "", 2026-02-23T13:07:28.556 INFO:teuthology.orchestra.run.trial068.stdout: "client_mount_gid": "-1", 2026-02-23T13:07:28.556 INFO:teuthology.orchestra.run.trial068.stdout: "client_mount_timeout": "300", 2026-02-23T13:07:28.556 INFO:teuthology.orchestra.run.trial068.stdout: "client_mount_uid": "-1", 2026-02-23T13:07:28.556 INFO:teuthology.orchestra.run.trial068.stdout: "client_mountpoint": "/", 2026-02-23T13:07:28.556 INFO:teuthology.orchestra.run.trial068.stdout: "client_notify_timeout": "10", 2026-02-23T13:07:28.556 INFO:teuthology.orchestra.run.trial068.stdout: "client_oc": "true", 2026-02-23T13:07:28.557 INFO:teuthology.orchestra.run.trial068.stdout: "client_oc_max_dirty": "104857600", 2026-02-23T13:07:28.557 INFO:teuthology.orchestra.run.trial068.stdout: "client_oc_max_dirty_age": "5.000000", 2026-02-23T13:07:28.557 INFO:teuthology.orchestra.run.trial068.stdout: "client_oc_max_objects": "1000", 2026-02-23T13:07:28.557 INFO:teuthology.orchestra.run.trial068.stdout: "client_oc_size": "209715200", 2026-02-23T13:07:28.557 INFO:teuthology.orchestra.run.trial068.stdout: "client_oc_target_dirty": "8388608", 2026-02-23T13:07:28.557 INFO:teuthology.orchestra.run.trial068.stdout: "client_permissions": "true", 2026-02-23T13:07:28.557 INFO:teuthology.orchestra.run.trial068.stdout: "client_quota": "true", 2026-02-23T13:07:28.557 INFO:teuthology.orchestra.run.trial068.stdout: "client_quota_df": "true", 2026-02-23T13:07:28.557 INFO:teuthology.orchestra.run.trial068.stdout: "client_readahead_max_bytes": "0", 2026-02-23T13:07:28.557 INFO:teuthology.orchestra.run.trial068.stdout: "client_readahead_max_periods": "4", 2026-02-23T13:07:28.557 INFO:teuthology.orchestra.run.trial068.stdout: "client_readahead_min": "131072", 2026-02-23T13:07:28.557 INFO:teuthology.orchestra.run.trial068.stdout: "client_reconnect_stale": "false", 2026-02-23T13:07:28.558 INFO:teuthology.orchestra.run.trial068.stdout: "client_respect_subvolume_snapshot_visibility": "false", 2026-02-23T13:07:28.558 INFO:teuthology.orchestra.run.trial068.stdout: "client_shutdown_timeout": "30", 2026-02-23T13:07:28.558 INFO:teuthology.orchestra.run.trial068.stdout: "client_snapdir": ".snap", 2026-02-23T13:07:28.558 INFO:teuthology.orchestra.run.trial068.stdout: "client_tick_interval": "1", 2026-02-23T13:07:28.558 INFO:teuthology.orchestra.run.trial068.stdout: "client_trace": "", 2026-02-23T13:07:28.558 INFO:teuthology.orchestra.run.trial068.stdout: "client_try_dentry_invalidate": "false", 2026-02-23T13:07:28.558 INFO:teuthology.orchestra.run.trial068.stdout: "client_use_faked_inos": "false", 2026-02-23T13:07:28.558 INFO:teuthology.orchestra.run.trial068.stdout: "client_use_random_mds": "false", 2026-02-23T13:07:28.558 INFO:teuthology.orchestra.run.trial068.stdout: "clog_to_graylog": "false", 2026-02-23T13:07:28.558 INFO:teuthology.orchestra.run.trial068.stdout: "clog_to_graylog_host": "127.0.0.1", 2026-02-23T13:07:28.558 INFO:teuthology.orchestra.run.trial068.stdout: "clog_to_graylog_port": "12201", 2026-02-23T13:07:28.558 INFO:teuthology.orchestra.run.trial068.stdout: "clog_to_monitors": "default=true", 2026-02-23T13:07:28.559 INFO:teuthology.orchestra.run.trial068.stdout: "clog_to_syslog": "false", 2026-02-23T13:07:28.559 INFO:teuthology.orchestra.run.trial068.stdout: "clog_to_syslog_facility": "default=daemon audit=local0", 2026-02-23T13:07:28.559 INFO:teuthology.orchestra.run.trial068.stdout: "clog_to_syslog_level": "info", 2026-02-23T13:07:28.559 INFO:teuthology.orchestra.run.trial068.stdout: "cluster_addr": "-", 2026-02-23T13:07:28.559 INFO:teuthology.orchestra.run.trial068.stdout: "cluster_network": "", 2026-02-23T13:07:28.559 INFO:teuthology.orchestra.run.trial068.stdout: "cluster_network_interface": "", 2026-02-23T13:07:28.559 INFO:teuthology.orchestra.run.trial068.stdout: "compressor_zlib_isal": "false", 2026-02-23T13:07:28.559 INFO:teuthology.orchestra.run.trial068.stdout: "compressor_zlib_level": "5", 2026-02-23T13:07:28.559 INFO:teuthology.orchestra.run.trial068.stdout: "compressor_zlib_winsize": "-15", 2026-02-23T13:07:28.559 INFO:teuthology.orchestra.run.trial068.stdout: "compressor_zstd_level": "1", 2026-02-23T13:07:28.559 INFO:teuthology.orchestra.run.trial068.stdout: "container_image": "quay.ceph.io/ceph-ci/ceph@sha256:49621faaa845c39b2aae72e4fa24123ee9b2ac3fb8ae834753907068ee2972ea", 2026-02-23T13:07:28.559 INFO:teuthology.orchestra.run.trial068.stdout: "crash_dir": "/var/lib/ceph/crash", 2026-02-23T13:07:28.560 INFO:teuthology.orchestra.run.trial068.stdout: "crimson_allow_pg_split": "false", 2026-02-23T13:07:28.560 INFO:teuthology.orchestra.run.trial068.stdout: "crimson_bluestore_cpu_set": "", 2026-02-23T13:07:28.560 INFO:teuthology.orchestra.run.trial068.stdout: "crimson_bluestore_num_threads": "6", 2026-02-23T13:07:28.560 INFO:teuthology.orchestra.run.trial068.stdout: "crimson_cpu_num": "0", 2026-02-23T13:07:28.560 INFO:teuthology.orchestra.run.trial068.stdout: "crimson_cpu_set": "", 2026-02-23T13:07:28.560 INFO:teuthology.orchestra.run.trial068.stdout: "crimson_osd_obc_lru_size": "512", 2026-02-23T13:07:28.560 INFO:teuthology.orchestra.run.trial068.stdout: "crimson_osd_scheduler_concurrency": "0", 2026-02-23T13:07:28.560 INFO:teuthology.orchestra.run.trial068.stdout: "crimson_osd_stat_interval": "0", 2026-02-23T13:07:28.560 INFO:teuthology.orchestra.run.trial068.stdout: "crimson_poll_mode": "false", 2026-02-23T13:07:28.560 INFO:teuthology.orchestra.run.trial068.stdout: "crimson_reactor_backend": "", 2026-02-23T13:07:28.560 INFO:teuthology.orchestra.run.trial068.stdout: "crimson_reactor_idle_poll_time_us": "200", 2026-02-23T13:07:28.560 INFO:teuthology.orchestra.run.trial068.stdout: "crimson_reactor_io_latency_goal_ms": "0.000000", 2026-02-23T13:07:28.561 INFO:teuthology.orchestra.run.trial068.stdout: "crimson_reactor_task_quota_ms": "0.500000", 2026-02-23T13:07:28.561 INFO:teuthology.orchestra.run.trial068.stdout: "crush_location": "", 2026-02-23T13:07:28.561 INFO:teuthology.orchestra.run.trial068.stdout: "crush_location_hook": "", 2026-02-23T13:07:28.561 INFO:teuthology.orchestra.run.trial068.stdout: "crush_location_hook_timeout": "10", 2026-02-23T13:07:28.561 INFO:teuthology.orchestra.run.trial068.stdout: "d4n_writecache_enabled": "false", 2026-02-23T13:07:28.561 INFO:teuthology.orchestra.run.trial068.stdout: "daemonize": "false", 2026-02-23T13:07:28.561 INFO:teuthology.orchestra.run.trial068.stdout: "daos_pool": "tank", 2026-02-23T13:07:28.561 INFO:teuthology.orchestra.run.trial068.stdout: "dbstore_config_uri": "file:/var/lib/ceph/radosgw/dbstore-config.db", 2026-02-23T13:07:28.561 INFO:teuthology.orchestra.run.trial068.stdout: "dbstore_db_dir": "/var/lib/ceph/radosgw", 2026-02-23T13:07:28.561 INFO:teuthology.orchestra.run.trial068.stdout: "dbstore_db_name_prefix": "dbstore", 2026-02-23T13:07:28.561 INFO:teuthology.orchestra.run.trial068.stdout: "debug_alienstore": "0/5", 2026-02-23T13:07:28.561 INFO:teuthology.orchestra.run.trial068.stdout: "debug_allow_any_pool_priority": "false", 2026-02-23T13:07:28.562 INFO:teuthology.orchestra.run.trial068.stdout: "debug_asok": "1/5", 2026-02-23T13:07:28.562 INFO:teuthology.orchestra.run.trial068.stdout: "debug_asok_assert_abort": "false", 2026-02-23T13:07:28.562 INFO:teuthology.orchestra.run.trial068.stdout: "debug_asserts_on_shutdown": "true", 2026-02-23T13:07:28.562 INFO:teuthology.orchestra.run.trial068.stdout: "debug_auth": "1/5", 2026-02-23T13:07:28.562 INFO:teuthology.orchestra.run.trial068.stdout: "debug_bdev": "1/3", 2026-02-23T13:07:28.562 INFO:teuthology.orchestra.run.trial068.stdout: "debug_bluefs": "1/5", 2026-02-23T13:07:28.562 INFO:teuthology.orchestra.run.trial068.stdout: "debug_bluestore": "1/5", 2026-02-23T13:07:28.562 INFO:teuthology.orchestra.run.trial068.stdout: "debug_bluestore_compression": "1/5", 2026-02-23T13:07:28.562 INFO:teuthology.orchestra.run.trial068.stdout: "debug_buffer": "0/1", 2026-02-23T13:07:28.562 INFO:teuthology.orchestra.run.trial068.stdout: "debug_ceph_dedup": "0/5", 2026-02-23T13:07:28.562 INFO:teuthology.orchestra.run.trial068.stdout: "debug_ceph_exporter": "1/5", 2026-02-23T13:07:28.562 INFO:teuthology.orchestra.run.trial068.stdout: "debug_cephfs_mirror": "0/5", 2026-02-23T13:07:28.562 INFO:teuthology.orchestra.run.trial068.stdout: "debug_cephsqlite": "0/5", 2026-02-23T13:07:28.563 INFO:teuthology.orchestra.run.trial068.stdout: "debug_client": "0/5", 2026-02-23T13:07:28.563 INFO:teuthology.orchestra.run.trial068.stdout: "debug_compressor": "1/5", 2026-02-23T13:07:28.563 INFO:teuthology.orchestra.run.trial068.stdout: "debug_context": "0/1", 2026-02-23T13:07:28.563 INFO:teuthology.orchestra.run.trial068.stdout: "debug_crimson_interrupt": "0/5", 2026-02-23T13:07:28.563 INFO:teuthology.orchestra.run.trial068.stdout: "debug_crush": "1/1", 2026-02-23T13:07:28.563 INFO:teuthology.orchestra.run.trial068.stdout: "debug_crypto": "1/5", 2026-02-23T13:07:28.563 INFO:teuthology.orchestra.run.trial068.stdout: "debug_cyanstore": "0/5", 2026-02-23T13:07:28.563 INFO:teuthology.orchestra.run.trial068.stdout: "debug_deliberately_leak_memory": "false", 2026-02-23T13:07:28.563 INFO:teuthology.orchestra.run.trial068.stdout: "debug_disable_randomized_ping": "false", 2026-02-23T13:07:28.563 INFO:teuthology.orchestra.run.trial068.stdout: "debug_dpdk": "1/5", 2026-02-23T13:07:28.563 INFO:teuthology.orchestra.run.trial068.stdout: "debug_eventtrace": "1/5", 2026-02-23T13:07:28.563 INFO:teuthology.orchestra.run.trial068.stdout: "debug_filer": "0/1", 2026-02-23T13:07:28.564 INFO:teuthology.orchestra.run.trial068.stdout: "debug_filestore": "1/3", 2026-02-23T13:07:28.564 INFO:teuthology.orchestra.run.trial068.stdout: "debug_finisher": "1/1", 2026-02-23T13:07:28.564 INFO:teuthology.orchestra.run.trial068.stdout: "debug_fuse": "1/5", 2026-02-23T13:07:28.564 INFO:teuthology.orchestra.run.trial068.stdout: "debug_heartbeat_testing_span": "0", 2026-02-23T13:07:28.564 INFO:teuthology.orchestra.run.trial068.stdout: "debug_heartbeatmap": "1/5", 2026-02-23T13:07:28.564 INFO:teuthology.orchestra.run.trial068.stdout: "debug_immutable_obj_cache": "0/5", 2026-02-23T13:07:28.564 INFO:teuthology.orchestra.run.trial068.stdout: "debug_javaclient": "1/5", 2026-02-23T13:07:28.564 INFO:teuthology.orchestra.run.trial068.stdout: "debug_journal": "1/3", 2026-02-23T13:07:28.564 INFO:teuthology.orchestra.run.trial068.stdout: "debug_journaler": "0/5", 2026-02-23T13:07:28.564 INFO:teuthology.orchestra.run.trial068.stdout: "debug_kstore": "1/5", 2026-02-23T13:07:28.564 INFO:teuthology.orchestra.run.trial068.stdout: "debug_lockdep": "0/1", 2026-02-23T13:07:28.564 INFO:teuthology.orchestra.run.trial068.stdout: "debug_mclock": "1/5", 2026-02-23T13:07:28.564 INFO:teuthology.orchestra.run.trial068.stdout: "debug_mds": "1/5", 2026-02-23T13:07:28.564 INFO:teuthology.orchestra.run.trial068.stdout: "debug_mds_balancer": "1/5", 2026-02-23T13:07:28.565 INFO:teuthology.orchestra.run.trial068.stdout: "debug_mds_locker": "1/5", 2026-02-23T13:07:28.565 INFO:teuthology.orchestra.run.trial068.stdout: "debug_mds_log": "1/5", 2026-02-23T13:07:28.565 INFO:teuthology.orchestra.run.trial068.stdout: "debug_mds_log_expire": "1/5", 2026-02-23T13:07:28.565 INFO:teuthology.orchestra.run.trial068.stdout: "debug_mds_migrator": "1/5", 2026-02-23T13:07:28.565 INFO:teuthology.orchestra.run.trial068.stdout: "debug_mds_quiesce": "3/5", 2026-02-23T13:07:28.565 INFO:teuthology.orchestra.run.trial068.stdout: "debug_memstore": "1/5", 2026-02-23T13:07:28.565 INFO:teuthology.orchestra.run.trial068.stdout: "debug_mgr": "2/5", 2026-02-23T13:07:28.565 INFO:teuthology.orchestra.run.trial068.stdout: "debug_mgrc": "1/5", 2026-02-23T13:07:28.565 INFO:teuthology.orchestra.run.trial068.stdout: "debug_mon": "20/20", 2026-02-23T13:07:28.565 INFO:teuthology.orchestra.run.trial068.stdout: "debug_monc": "0/10", 2026-02-23T13:07:28.565 INFO:teuthology.orchestra.run.trial068.stdout: "debug_ms": "1/1", 2026-02-23T13:07:28.565 INFO:teuthology.orchestra.run.trial068.stdout: "debug_none": "0/5", 2026-02-23T13:07:28.566 INFO:teuthology.orchestra.run.trial068.stdout: "debug_objclass": "0/5", 2026-02-23T13:07:28.566 INFO:teuthology.orchestra.run.trial068.stdout: "debug_objectcacher": "0/5", 2026-02-23T13:07:28.566 INFO:teuthology.orchestra.run.trial068.stdout: "debug_objecter": "0/1", 2026-02-23T13:07:28.566 INFO:teuthology.orchestra.run.trial068.stdout: "debug_optracker": "0/5", 2026-02-23T13:07:28.566 INFO:teuthology.orchestra.run.trial068.stdout: "debug_osd": "1/5", 2026-02-23T13:07:28.566 INFO:teuthology.orchestra.run.trial068.stdout: "debug_paxos": "20/20", 2026-02-23T13:07:28.566 INFO:teuthology.orchestra.run.trial068.stdout: "debug_perfcounter": "1/5", 2026-02-23T13:07:28.566 INFO:teuthology.orchestra.run.trial068.stdout: "debug_prioritycache": "1/5", 2026-02-23T13:07:28.566 INFO:teuthology.orchestra.run.trial068.stdout: "debug_rados": "0/5", 2026-02-23T13:07:28.566 INFO:teuthology.orchestra.run.trial068.stdout: "debug_rbd": "0/5", 2026-02-23T13:07:28.566 INFO:teuthology.orchestra.run.trial068.stdout: "debug_rbd_mirror": "0/5", 2026-02-23T13:07:28.566 INFO:teuthology.orchestra.run.trial068.stdout: "debug_rbd_pwl": "0/5", 2026-02-23T13:07:28.566 INFO:teuthology.orchestra.run.trial068.stdout: "debug_rbd_replay": "0/5", 2026-02-23T13:07:28.566 INFO:teuthology.orchestra.run.trial068.stdout: "debug_refs": "0/0", 2026-02-23T13:07:28.566 INFO:teuthology.orchestra.run.trial068.stdout: "debug_reserver": "1/1", 2026-02-23T13:07:28.566 INFO:teuthology.orchestra.run.trial068.stdout: "debug_rgw": "1/5", 2026-02-23T13:07:28.566 INFO:teuthology.orchestra.run.trial068.stdout: "debug_rgw_access": "1/5", 2026-02-23T13:07:28.566 INFO:teuthology.orchestra.run.trial068.stdout: "debug_rgw_bucket_logging": "1/5", 2026-02-23T13:07:28.566 INFO:teuthology.orchestra.run.trial068.stdout: "debug_rgw_datacache": "1/5", 2026-02-23T13:07:28.566 INFO:teuthology.orchestra.run.trial068.stdout: "debug_rgw_dbstore": "1/5", 2026-02-23T13:07:28.566 INFO:teuthology.orchestra.run.trial068.stdout: "debug_rgw_dedup": "1/5", 2026-02-23T13:07:28.566 INFO:teuthology.orchestra.run.trial068.stdout: "debug_rgw_flight": "1/5", 2026-02-23T13:07:28.566 INFO:teuthology.orchestra.run.trial068.stdout: "debug_rgw_lifecycle": "1/5", 2026-02-23T13:07:28.566 INFO:teuthology.orchestra.run.trial068.stdout: "debug_rgw_notification": "1/5", 2026-02-23T13:07:28.566 INFO:teuthology.orchestra.run.trial068.stdout: "debug_rgw_restore": "1/5", 2026-02-23T13:07:28.566 INFO:teuthology.orchestra.run.trial068.stdout: "debug_rgw_sync": "1/5", 2026-02-23T13:07:28.566 INFO:teuthology.orchestra.run.trial068.stdout: "debug_rocksdb": "4/5", 2026-02-23T13:07:28.566 INFO:teuthology.orchestra.run.trial068.stdout: "debug_seastore": "0/5", 2026-02-23T13:07:28.566 INFO:teuthology.orchestra.run.trial068.stdout: "debug_seastore_backref": "0/5", 2026-02-23T13:07:28.566 INFO:teuthology.orchestra.run.trial068.stdout: "debug_seastore_cache": "0/5", 2026-02-23T13:07:28.566 INFO:teuthology.orchestra.run.trial068.stdout: "debug_seastore_cleaner": "0/5", 2026-02-23T13:07:28.566 INFO:teuthology.orchestra.run.trial068.stdout: "debug_seastore_device": "0/5", 2026-02-23T13:07:28.566 INFO:teuthology.orchestra.run.trial068.stdout: "debug_seastore_epm": "0/5", 2026-02-23T13:07:28.566 INFO:teuthology.orchestra.run.trial068.stdout: "debug_seastore_fixedkv_tree": "0/5", 2026-02-23T13:07:28.567 INFO:teuthology.orchestra.run.trial068.stdout: "debug_seastore_journal": "0/5", 2026-02-23T13:07:28.567 INFO:teuthology.orchestra.run.trial068.stdout: "debug_seastore_lba": "0/5", 2026-02-23T13:07:28.567 INFO:teuthology.orchestra.run.trial068.stdout: "debug_seastore_odata": "0/5", 2026-02-23T13:07:28.567 INFO:teuthology.orchestra.run.trial068.stdout: "debug_seastore_omap": "0/5", 2026-02-23T13:07:28.567 INFO:teuthology.orchestra.run.trial068.stdout: "debug_seastore_onode": "0/5", 2026-02-23T13:07:28.567 INFO:teuthology.orchestra.run.trial068.stdout: "debug_seastore_t": "0/5", 2026-02-23T13:07:28.567 INFO:teuthology.orchestra.run.trial068.stdout: "debug_seastore_tm": "0/5", 2026-02-23T13:07:28.567 INFO:teuthology.orchestra.run.trial068.stdout: "debug_striper": "0/1", 2026-02-23T13:07:28.567 INFO:teuthology.orchestra.run.trial068.stdout: "debug_test": "0/5", 2026-02-23T13:07:28.567 INFO:teuthology.orchestra.run.trial068.stdout: "debug_throttle": "1/1", 2026-02-23T13:07:28.567 INFO:teuthology.orchestra.run.trial068.stdout: "debug_timer": "0/1", 2026-02-23T13:07:28.567 INFO:teuthology.orchestra.run.trial068.stdout: "debug_tp": "0/5", 2026-02-23T13:07:28.567 INFO:teuthology.orchestra.run.trial068.stdout: "debug_trace": "1/5", 2026-02-23T13:07:28.567 INFO:teuthology.orchestra.run.trial068.stdout: "defer_client_eviction_on_laggy_osds": "false", 2026-02-23T13:07:28.567 INFO:teuthology.orchestra.run.trial068.stdout: "device_failure_prediction_mode": "none", 2026-02-23T13:07:28.567 INFO:teuthology.orchestra.run.trial068.stdout: "ec_extent_cache_size": "10485760", 2026-02-23T13:07:28.567 INFO:teuthology.orchestra.run.trial068.stdout: "ec_pdw_write_mode": "0", 2026-02-23T13:07:28.567 INFO:teuthology.orchestra.run.trial068.stdout: "enable_availability_tracking": "false", 2026-02-23T13:07:28.567 INFO:teuthology.orchestra.run.trial068.stdout: "enable_experimental_unrecoverable_data_corrupting_features": "crimson", 2026-02-23T13:07:28.567 INFO:teuthology.orchestra.run.trial068.stdout: "erasure_code_dir": "/usr/lib64/ceph/erasure-code", 2026-02-23T13:07:28.567 INFO:teuthology.orchestra.run.trial068.stdout: "err_to_graylog": "false", 2026-02-23T13:07:28.567 INFO:teuthology.orchestra.run.trial068.stdout: "err_to_journald": "false", 2026-02-23T13:07:28.567 INFO:teuthology.orchestra.run.trial068.stdout: "err_to_stderr": "true", 2026-02-23T13:07:28.567 INFO:teuthology.orchestra.run.trial068.stdout: "err_to_syslog": "false", 2026-02-23T13:07:28.567 INFO:teuthology.orchestra.run.trial068.stdout: "event_tracing": "false", 2026-02-23T13:07:28.567 INFO:teuthology.orchestra.run.trial068.stdout: "exporter_addr": "0.0.0.0", 2026-02-23T13:07:28.567 INFO:teuthology.orchestra.run.trial068.stdout: "exporter_cert_file": "", 2026-02-23T13:07:28.567 INFO:teuthology.orchestra.run.trial068.stdout: "exporter_http_port": "9926", 2026-02-23T13:07:28.567 INFO:teuthology.orchestra.run.trial068.stdout: "exporter_key_file": "", 2026-02-23T13:07:28.567 INFO:teuthology.orchestra.run.trial068.stdout: "exporter_prio_limit": "5", 2026-02-23T13:07:28.567 INFO:teuthology.orchestra.run.trial068.stdout: "exporter_sock_dir": "/var/run/ceph/", 2026-02-23T13:07:28.567 INFO:teuthology.orchestra.run.trial068.stdout: "exporter_sort_metrics": "true", 2026-02-23T13:07:28.567 INFO:teuthology.orchestra.run.trial068.stdout: "exporter_stats_period": "5", 2026-02-23T13:07:28.567 INFO:teuthology.orchestra.run.trial068.stdout: "fake_statfs_for_testing": "0", 2026-02-23T13:07:28.567 INFO:teuthology.orchestra.run.trial068.stdout: "fatal_signal_handlers": "true", 2026-02-23T13:07:28.567 INFO:teuthology.orchestra.run.trial068.stdout: "filer_max_purge_ops": "10", 2026-02-23T13:07:28.567 INFO:teuthology.orchestra.run.trial068.stdout: "filer_max_truncate_ops": "128", 2026-02-23T13:07:28.567 INFO:teuthology.orchestra.run.trial068.stdout: "filestore_apply_finisher_threads": "1", 2026-02-23T13:07:28.567 INFO:teuthology.orchestra.run.trial068.stdout: "filestore_blackhole": "false", 2026-02-23T13:07:28.568 INFO:teuthology.orchestra.run.trial068.stdout: "filestore_btrfs_clone_range": "true", 2026-02-23T13:07:28.568 INFO:teuthology.orchestra.run.trial068.stdout: "filestore_btrfs_snap": "true", 2026-02-23T13:07:28.568 INFO:teuthology.orchestra.run.trial068.stdout: "filestore_caller_concurrency": "10", 2026-02-23T13:07:28.568 INFO:teuthology.orchestra.run.trial068.stdout: "filestore_collect_device_partition_information": "true", 2026-02-23T13:07:28.568 INFO:teuthology.orchestra.run.trial068.stdout: "filestore_commit_timeout": "600.000000", 2026-02-23T13:07:28.568 INFO:teuthology.orchestra.run.trial068.stdout: "filestore_debug_inject_read_err": "false", 2026-02-23T13:07:28.568 INFO:teuthology.orchestra.run.trial068.stdout: "filestore_debug_omap_check": "false", 2026-02-23T13:07:28.568 INFO:teuthology.orchestra.run.trial068.stdout: "filestore_debug_verify_split": "false", 2026-02-23T13:07:28.568 INFO:teuthology.orchestra.run.trial068.stdout: "filestore_dump_file": "", 2026-02-23T13:07:28.568 INFO:teuthology.orchestra.run.trial068.stdout: "filestore_expected_throughput_bytes": "209715200.000000", 2026-02-23T13:07:28.568 INFO:teuthology.orchestra.run.trial068.stdout: "filestore_expected_throughput_ops": "200.000000", 2026-02-23T13:07:28.568 INFO:teuthology.orchestra.run.trial068.stdout: "filestore_fadvise": "true", 2026-02-23T13:07:28.568 INFO:teuthology.orchestra.run.trial068.stdout: "filestore_fail_eio": "true", 2026-02-23T13:07:28.568 INFO:teuthology.orchestra.run.trial068.stdout: "filestore_fd_cache_shards": "16", 2026-02-23T13:07:28.568 INFO:teuthology.orchestra.run.trial068.stdout: "filestore_fd_cache_size": "128", 2026-02-23T13:07:28.568 INFO:teuthology.orchestra.run.trial068.stdout: "filestore_fiemap": "false", 2026-02-23T13:07:28.568 INFO:teuthology.orchestra.run.trial068.stdout: "filestore_fiemap_threshold": "4096", 2026-02-23T13:07:28.568 INFO:teuthology.orchestra.run.trial068.stdout: "filestore_fsync_flushes_journal_data": "false", 2026-02-23T13:07:28.568 INFO:teuthology.orchestra.run.trial068.stdout: "filestore_index_retry_probability": "0.000000", 2026-02-23T13:07:28.568 INFO:teuthology.orchestra.run.trial068.stdout: "filestore_inject_stall": "0", 2026-02-23T13:07:28.568 INFO:teuthology.orchestra.run.trial068.stdout: "filestore_journal_parallel": "false", 2026-02-23T13:07:28.568 INFO:teuthology.orchestra.run.trial068.stdout: "filestore_journal_trailing": "false", 2026-02-23T13:07:28.568 INFO:teuthology.orchestra.run.trial068.stdout: "filestore_journal_writeahead": "false", 2026-02-23T13:07:28.568 INFO:teuthology.orchestra.run.trial068.stdout: "filestore_kill_at": "0", 2026-02-23T13:07:28.568 INFO:teuthology.orchestra.run.trial068.stdout: "filestore_max_alloc_hint_size": "1048576", 2026-02-23T13:07:28.568 INFO:teuthology.orchestra.run.trial068.stdout: "filestore_max_inline_xattr_size": "0", 2026-02-23T13:07:28.568 INFO:teuthology.orchestra.run.trial068.stdout: "filestore_max_inline_xattr_size_btrfs": "2048", 2026-02-23T13:07:28.568 INFO:teuthology.orchestra.run.trial068.stdout: "filestore_max_inline_xattr_size_other": "512", 2026-02-23T13:07:28.568 INFO:teuthology.orchestra.run.trial068.stdout: "filestore_max_inline_xattr_size_xfs": "65536", 2026-02-23T13:07:28.568 INFO:teuthology.orchestra.run.trial068.stdout: "filestore_max_inline_xattrs": "0", 2026-02-23T13:07:28.568 INFO:teuthology.orchestra.run.trial068.stdout: "filestore_max_inline_xattrs_btrfs": "10", 2026-02-23T13:07:28.568 INFO:teuthology.orchestra.run.trial068.stdout: "filestore_max_inline_xattrs_other": "2", 2026-02-23T13:07:28.568 INFO:teuthology.orchestra.run.trial068.stdout: "filestore_max_inline_xattrs_xfs": "10", 2026-02-23T13:07:28.568 INFO:teuthology.orchestra.run.trial068.stdout: "filestore_max_sync_interval": "5.000000", 2026-02-23T13:07:28.568 INFO:teuthology.orchestra.run.trial068.stdout: "filestore_max_xattr_value_size": "0", 2026-02-23T13:07:28.568 INFO:teuthology.orchestra.run.trial068.stdout: "filestore_max_xattr_value_size_btrfs": "65536", 2026-02-23T13:07:28.568 INFO:teuthology.orchestra.run.trial068.stdout: "filestore_max_xattr_value_size_other": "1024", 2026-02-23T13:07:28.568 INFO:teuthology.orchestra.run.trial068.stdout: "filestore_max_xattr_value_size_xfs": "65536", 2026-02-23T13:07:28.568 INFO:teuthology.orchestra.run.trial068.stdout: "filestore_merge_threshold": "-10", 2026-02-23T13:07:28.569 INFO:teuthology.orchestra.run.trial068.stdout: "filestore_min_sync_interval": "0.010000", 2026-02-23T13:07:28.569 INFO:teuthology.orchestra.run.trial068.stdout: "filestore_odsync_write": "false", 2026-02-23T13:07:28.569 INFO:teuthology.orchestra.run.trial068.stdout: "filestore_omap_backend": "rocksdb", 2026-02-23T13:07:28.569 INFO:teuthology.orchestra.run.trial068.stdout: "filestore_omap_backend_path": "", 2026-02-23T13:07:28.569 INFO:teuthology.orchestra.run.trial068.stdout: "filestore_omap_header_cache_size": "1024", 2026-02-23T13:07:28.569 INFO:teuthology.orchestra.run.trial068.stdout: "filestore_ondisk_finisher_threads": "1", 2026-02-23T13:07:28.569 INFO:teuthology.orchestra.run.trial068.stdout: "filestore_op_thread_suicide_timeout": "180", 2026-02-23T13:07:28.569 INFO:teuthology.orchestra.run.trial068.stdout: "filestore_op_thread_timeout": "60", 2026-02-23T13:07:28.569 INFO:teuthology.orchestra.run.trial068.stdout: "filestore_op_threads": "2", 2026-02-23T13:07:28.569 INFO:teuthology.orchestra.run.trial068.stdout: "filestore_punch_hole": "false", 2026-02-23T13:07:28.569 INFO:teuthology.orchestra.run.trial068.stdout: "filestore_queue_high_delay_multiple": "0.000000", 2026-02-23T13:07:28.569 INFO:teuthology.orchestra.run.trial068.stdout: "filestore_queue_high_delay_multiple_bytes": "0.000000", 2026-02-23T13:07:28.569 INFO:teuthology.orchestra.run.trial068.stdout: "filestore_queue_high_delay_multiple_ops": "0.000000", 2026-02-23T13:07:28.569 INFO:teuthology.orchestra.run.trial068.stdout: "filestore_queue_high_threshhold": "0.900000", 2026-02-23T13:07:28.569 INFO:teuthology.orchestra.run.trial068.stdout: "filestore_queue_low_threshhold": "0.300000", 2026-02-23T13:07:28.569 INFO:teuthology.orchestra.run.trial068.stdout: "filestore_queue_max_bytes": "104857600", 2026-02-23T13:07:28.569 INFO:teuthology.orchestra.run.trial068.stdout: "filestore_queue_max_delay_multiple": "0.000000", 2026-02-23T13:07:28.569 INFO:teuthology.orchestra.run.trial068.stdout: "filestore_queue_max_delay_multiple_bytes": "0.000000", 2026-02-23T13:07:28.569 INFO:teuthology.orchestra.run.trial068.stdout: "filestore_queue_max_delay_multiple_ops": "0.000000", 2026-02-23T13:07:28.569 INFO:teuthology.orchestra.run.trial068.stdout: "filestore_queue_max_ops": "50", 2026-02-23T13:07:28.569 INFO:teuthology.orchestra.run.trial068.stdout: "filestore_rocksdb_options": "max_background_jobs=10,compaction_readahead_size=2097152,compression=kNoCompression", 2026-02-23T13:07:28.569 INFO:teuthology.orchestra.run.trial068.stdout: "filestore_seek_data_hole": "false", 2026-02-23T13:07:28.569 INFO:teuthology.orchestra.run.trial068.stdout: "filestore_sloppy_crc": "false", 2026-02-23T13:07:28.569 INFO:teuthology.orchestra.run.trial068.stdout: "filestore_sloppy_crc_block_size": "65536", 2026-02-23T13:07:28.569 INFO:teuthology.orchestra.run.trial068.stdout: "filestore_splice": "false", 2026-02-23T13:07:28.569 INFO:teuthology.orchestra.run.trial068.stdout: "filestore_split_multiple": "2", 2026-02-23T13:07:28.569 INFO:teuthology.orchestra.run.trial068.stdout: "filestore_split_rand_factor": "20", 2026-02-23T13:07:28.569 INFO:teuthology.orchestra.run.trial068.stdout: "filestore_update_to": "1000", 2026-02-23T13:07:28.569 INFO:teuthology.orchestra.run.trial068.stdout: "filestore_wbthrottle_btrfs_bytes_hard_limit": "419430400", 2026-02-23T13:07:28.569 INFO:teuthology.orchestra.run.trial068.stdout: "filestore_wbthrottle_btrfs_bytes_start_flusher": "41943040", 2026-02-23T13:07:28.569 INFO:teuthology.orchestra.run.trial068.stdout: "filestore_wbthrottle_btrfs_inodes_hard_limit": "5000", 2026-02-23T13:07:28.569 INFO:teuthology.orchestra.run.trial068.stdout: "filestore_wbthrottle_btrfs_inodes_start_flusher": "500", 2026-02-23T13:07:28.569 INFO:teuthology.orchestra.run.trial068.stdout: "filestore_wbthrottle_btrfs_ios_hard_limit": "5000", 2026-02-23T13:07:28.569 INFO:teuthology.orchestra.run.trial068.stdout: "filestore_wbthrottle_btrfs_ios_start_flusher": "500", 2026-02-23T13:07:28.569 INFO:teuthology.orchestra.run.trial068.stdout: "filestore_wbthrottle_enable": "true", 2026-02-23T13:07:28.569 INFO:teuthology.orchestra.run.trial068.stdout: "filestore_wbthrottle_xfs_bytes_hard_limit": "419430400", 2026-02-23T13:07:28.569 INFO:teuthology.orchestra.run.trial068.stdout: "filestore_wbthrottle_xfs_bytes_start_flusher": "41943040", 2026-02-23T13:07:28.569 INFO:teuthology.orchestra.run.trial068.stdout: "filestore_wbthrottle_xfs_inodes_hard_limit": "5000", 2026-02-23T13:07:28.569 INFO:teuthology.orchestra.run.trial068.stdout: "filestore_wbthrottle_xfs_inodes_start_flusher": "500", 2026-02-23T13:07:28.569 INFO:teuthology.orchestra.run.trial068.stdout: "filestore_wbthrottle_xfs_ios_hard_limit": "5000", 2026-02-23T13:07:28.569 INFO:teuthology.orchestra.run.trial068.stdout: "filestore_wbthrottle_xfs_ios_start_flusher": "500", 2026-02-23T13:07:28.570 INFO:teuthology.orchestra.run.trial068.stdout: "filestore_xfs_extsize": "false", 2026-02-23T13:07:28.570 INFO:teuthology.orchestra.run.trial068.stdout: "filestore_zfs_snap": "false", 2026-02-23T13:07:28.570 INFO:teuthology.orchestra.run.trial068.stdout: "fio_dir": "/tmp/fio", 2026-02-23T13:07:28.570 INFO:teuthology.orchestra.run.trial068.stdout: "fsid": "5f823084-10b8-11f1-be80-d404e6e7d460", 2026-02-23T13:07:28.570 INFO:teuthology.orchestra.run.trial068.stdout: "fuse_allow_other": "true", 2026-02-23T13:07:28.570 INFO:teuthology.orchestra.run.trial068.stdout: "fuse_atomic_o_trunc": "true", 2026-02-23T13:07:28.570 INFO:teuthology.orchestra.run.trial068.stdout: "fuse_big_writes": "true", 2026-02-23T13:07:28.570 INFO:teuthology.orchestra.run.trial068.stdout: "fuse_debug": "false", 2026-02-23T13:07:28.570 INFO:teuthology.orchestra.run.trial068.stdout: "fuse_default_permissions": "false", 2026-02-23T13:07:28.570 INFO:teuthology.orchestra.run.trial068.stdout: "fuse_disable_pagecache": "false", 2026-02-23T13:07:28.570 INFO:teuthology.orchestra.run.trial068.stdout: "fuse_max_write": "0", 2026-02-23T13:07:28.570 INFO:teuthology.orchestra.run.trial068.stdout: "fuse_multithreaded": "true", 2026-02-23T13:07:28.570 INFO:teuthology.orchestra.run.trial068.stdout: "fuse_require_active_mds": "true", 2026-02-23T13:07:28.570 INFO:teuthology.orchestra.run.trial068.stdout: "fuse_set_user_groups": "true", 2026-02-23T13:07:28.570 INFO:teuthology.orchestra.run.trial068.stdout: "fuse_splice_move": "true", 2026-02-23T13:07:28.570 INFO:teuthology.orchestra.run.trial068.stdout: "fuse_splice_read": "true", 2026-02-23T13:07:28.570 INFO:teuthology.orchestra.run.trial068.stdout: "fuse_splice_write": "true", 2026-02-23T13:07:28.570 INFO:teuthology.orchestra.run.trial068.stdout: "fuse_syncfs_on_mksnap": "true", 2026-02-23T13:07:28.570 INFO:teuthology.orchestra.run.trial068.stdout: "fuse_use_invalidate_cb": "true", 2026-02-23T13:07:28.570 INFO:teuthology.orchestra.run.trial068.stdout: "gss_ktab_client_file": "/var/lib/ceph/mon.a/gss_client_mon.a.ktab", 2026-02-23T13:07:28.570 INFO:teuthology.orchestra.run.trial068.stdout: "gss_target_name": "ceph", 2026-02-23T13:07:28.570 INFO:teuthology.orchestra.run.trial068.stdout: "heartbeat_file": "", 2026-02-23T13:07:28.570 INFO:teuthology.orchestra.run.trial068.stdout: "heartbeat_inject_failure": "0", 2026-02-23T13:07:28.570 INFO:teuthology.orchestra.run.trial068.stdout: "heartbeat_interval": "5", 2026-02-23T13:07:28.570 INFO:teuthology.orchestra.run.trial068.stdout: "host": "trial068", 2026-02-23T13:07:28.570 INFO:teuthology.orchestra.run.trial068.stdout: "immutable_object_cache_client_dedicated_thread_num": "2", 2026-02-23T13:07:28.570 INFO:teuthology.orchestra.run.trial068.stdout: "immutable_object_cache_max_inflight_ops": "128", 2026-02-23T13:07:28.570 INFO:teuthology.orchestra.run.trial068.stdout: "immutable_object_cache_max_size": "1073741824", 2026-02-23T13:07:28.570 INFO:teuthology.orchestra.run.trial068.stdout: "immutable_object_cache_path": "/tmp/ceph_immutable_object_cache", 2026-02-23T13:07:28.570 INFO:teuthology.orchestra.run.trial068.stdout: "immutable_object_cache_qos_bps_burst": "0", 2026-02-23T13:07:28.570 INFO:teuthology.orchestra.run.trial068.stdout: "immutable_object_cache_qos_bps_burst_seconds": "1", 2026-02-23T13:07:28.570 INFO:teuthology.orchestra.run.trial068.stdout: "immutable_object_cache_qos_bps_limit": "0", 2026-02-23T13:07:28.570 INFO:teuthology.orchestra.run.trial068.stdout: "immutable_object_cache_qos_iops_burst": "0", 2026-02-23T13:07:28.570 INFO:teuthology.orchestra.run.trial068.stdout: "immutable_object_cache_qos_iops_burst_seconds": "1", 2026-02-23T13:07:28.570 INFO:teuthology.orchestra.run.trial068.stdout: "immutable_object_cache_qos_iops_limit": "0", 2026-02-23T13:07:28.570 INFO:teuthology.orchestra.run.trial068.stdout: "immutable_object_cache_qos_schedule_tick_min": "50", 2026-02-23T13:07:28.570 INFO:teuthology.orchestra.run.trial068.stdout: "immutable_object_cache_sock": "/var/run/ceph/immutable_object_cache_sock", 2026-02-23T13:07:28.570 INFO:teuthology.orchestra.run.trial068.stdout: "immutable_object_cache_watermark": "0.900000", 2026-02-23T13:07:28.571 INFO:teuthology.orchestra.run.trial068.stdout: "inject_early_sigterm": "false", 2026-02-23T13:07:28.571 INFO:teuthology.orchestra.run.trial068.stdout: "jaeger_agent_port": "6799", 2026-02-23T13:07:28.571 INFO:teuthology.orchestra.run.trial068.stdout: "jaeger_tracing_enable": "false", 2026-02-23T13:07:28.571 INFO:teuthology.orchestra.run.trial068.stdout: "journal_aio": "true", 2026-02-23T13:07:28.571 INFO:teuthology.orchestra.run.trial068.stdout: "journal_align_min_size": "65536", 2026-02-23T13:07:28.571 INFO:teuthology.orchestra.run.trial068.stdout: "journal_block_align": "true", 2026-02-23T13:07:28.571 INFO:teuthology.orchestra.run.trial068.stdout: "journal_block_size": "4096", 2026-02-23T13:07:28.571 INFO:teuthology.orchestra.run.trial068.stdout: "journal_dio": "true", 2026-02-23T13:07:28.571 INFO:teuthology.orchestra.run.trial068.stdout: "journal_discard": "false", 2026-02-23T13:07:28.571 INFO:teuthology.orchestra.run.trial068.stdout: "journal_force_aio": "false", 2026-02-23T13:07:28.571 INFO:teuthology.orchestra.run.trial068.stdout: "journal_ignore_corruption": "false", 2026-02-23T13:07:28.571 INFO:teuthology.orchestra.run.trial068.stdout: "journal_max_write_bytes": "10485760", 2026-02-23T13:07:28.571 INFO:teuthology.orchestra.run.trial068.stdout: "journal_max_write_entries": "100", 2026-02-23T13:07:28.571 INFO:teuthology.orchestra.run.trial068.stdout: "journal_replay_from": "0", 2026-02-23T13:07:28.571 INFO:teuthology.orchestra.run.trial068.stdout: "journal_throttle_high_multiple": "0.000000", 2026-02-23T13:07:28.571 INFO:teuthology.orchestra.run.trial068.stdout: "journal_throttle_high_threshhold": "0.900000", 2026-02-23T13:07:28.571 INFO:teuthology.orchestra.run.trial068.stdout: "journal_throttle_low_threshhold": "0.600000", 2026-02-23T13:07:28.571 INFO:teuthology.orchestra.run.trial068.stdout: "journal_throttle_max_multiple": "0.000000", 2026-02-23T13:07:28.571 INFO:teuthology.orchestra.run.trial068.stdout: "journal_write_header_frequency": "0", 2026-02-23T13:07:28.571 INFO:teuthology.orchestra.run.trial068.stdout: "journal_zero_on_create": "false", 2026-02-23T13:07:28.571 INFO:teuthology.orchestra.run.trial068.stdout: "journaler_prefetch_periods": "10", 2026-02-23T13:07:28.571 INFO:teuthology.orchestra.run.trial068.stdout: "journaler_prezero_periods": "5", 2026-02-23T13:07:28.571 INFO:teuthology.orchestra.run.trial068.stdout: "journaler_write_head_interval": "15", 2026-02-23T13:07:28.571 INFO:teuthology.orchestra.run.trial068.stdout: "key": "", 2026-02-23T13:07:28.571 INFO:teuthology.orchestra.run.trial068.stdout: "keyfile": "", 2026-02-23T13:07:28.571 INFO:teuthology.orchestra.run.trial068.stdout: "keyring": "/var/lib/ceph/mon/ceph-a/keyring", 2026-02-23T13:07:28.571 INFO:teuthology.orchestra.run.trial068.stdout: "librados_thread_count": "2", 2026-02-23T13:07:28.571 INFO:teuthology.orchestra.run.trial068.stdout: "lockdep": "false", 2026-02-23T13:07:28.571 INFO:teuthology.orchestra.run.trial068.stdout: "lockdep_force_backtrace": "false", 2026-02-23T13:07:28.571 INFO:teuthology.orchestra.run.trial068.stdout: "log_coarse_timestamps": "true", 2026-02-23T13:07:28.571 INFO:teuthology.orchestra.run.trial068.stdout: "log_file": "/var/log/ceph/ceph-mon.a.log", 2026-02-23T13:07:28.571 INFO:teuthology.orchestra.run.trial068.stdout: "log_flush_on_exit": "false", 2026-02-23T13:07:28.571 INFO:teuthology.orchestra.run.trial068.stdout: "log_graylog_host": "127.0.0.1", 2026-02-23T13:07:28.571 INFO:teuthology.orchestra.run.trial068.stdout: "log_graylog_port": "12201", 2026-02-23T13:07:28.571 INFO:teuthology.orchestra.run.trial068.stdout: "log_max_new": "1000", 2026-02-23T13:07:28.571 INFO:teuthology.orchestra.run.trial068.stdout: "log_max_recent": "10000", 2026-02-23T13:07:28.571 INFO:teuthology.orchestra.run.trial068.stdout: "log_stderr_prefix": "", 2026-02-23T13:07:28.571 INFO:teuthology.orchestra.run.trial068.stdout: "log_stop_at_utilization": "0.970000", 2026-02-23T13:07:28.571 INFO:teuthology.orchestra.run.trial068.stdout: "log_to_file": "true", 2026-02-23T13:07:28.571 INFO:teuthology.orchestra.run.trial068.stdout: "log_to_graylog": "false", 2026-02-23T13:07:28.571 INFO:teuthology.orchestra.run.trial068.stdout: "log_to_journald": "false", 2026-02-23T13:07:28.571 INFO:teuthology.orchestra.run.trial068.stdout: "log_to_stderr": "false", 2026-02-23T13:07:28.571 INFO:teuthology.orchestra.run.trial068.stdout: "log_to_syslog": "false", 2026-02-23T13:07:28.572 INFO:teuthology.orchestra.run.trial068.stdout: "max_rotating_auth_attempts": "10", 2026-02-23T13:07:28.572 INFO:teuthology.orchestra.run.trial068.stdout: "mds_abort_on_newly_corrupt_dentry": "true", 2026-02-23T13:07:28.572 INFO:teuthology.orchestra.run.trial068.stdout: "mds_action_on_write_error": "1", 2026-02-23T13:07:28.572 INFO:teuthology.orchestra.run.trial068.stdout: "mds_allow_async_dirops": "true", 2026-02-23T13:07:28.572 INFO:teuthology.orchestra.run.trial068.stdout: "mds_allow_batched_ops": "true", 2026-02-23T13:07:28.572 INFO:teuthology.orchestra.run.trial068.stdout: "mds_alternate_name_max": "8192", 2026-02-23T13:07:28.572 INFO:teuthology.orchestra.run.trial068.stdout: "mds_asio_thread_count": "2", 2026-02-23T13:07:28.572 INFO:teuthology.orchestra.run.trial068.stdout: "mds_bal_export_pin": "true", 2026-02-23T13:07:28.572 INFO:teuthology.orchestra.run.trial068.stdout: "mds_bal_fragment_dirs": "true", 2026-02-23T13:07:28.572 INFO:teuthology.orchestra.run.trial068.stdout: "mds_bal_fragment_fast_factor": "1.500000", 2026-02-23T13:07:28.572 INFO:teuthology.orchestra.run.trial068.stdout: "mds_bal_fragment_interval": "5", 2026-02-23T13:07:28.572 INFO:teuthology.orchestra.run.trial068.stdout: "mds_bal_fragment_size_max": "100000", 2026-02-23T13:07:28.572 INFO:teuthology.orchestra.run.trial068.stdout: "mds_bal_idle_threshold": "0.000000", 2026-02-23T13:07:28.572 INFO:teuthology.orchestra.run.trial068.stdout: "mds_bal_interval": "10", 2026-02-23T13:07:28.572 INFO:teuthology.orchestra.run.trial068.stdout: "mds_bal_max": "-1", 2026-02-23T13:07:28.572 INFO:teuthology.orchestra.run.trial068.stdout: "mds_bal_max_until": "-1", 2026-02-23T13:07:28.572 INFO:teuthology.orchestra.run.trial068.stdout: "mds_bal_merge_size": "50", 2026-02-23T13:07:28.572 INFO:teuthology.orchestra.run.trial068.stdout: "mds_bal_midchunk": "0.300000", 2026-02-23T13:07:28.572 INFO:teuthology.orchestra.run.trial068.stdout: "mds_bal_min_rebalance": "0.100000", 2026-02-23T13:07:28.572 INFO:teuthology.orchestra.run.trial068.stdout: "mds_bal_min_start": "0.200000", 2026-02-23T13:07:28.572 INFO:teuthology.orchestra.run.trial068.stdout: "mds_bal_minchunk": "0.001000", 2026-02-23T13:07:28.572 INFO:teuthology.orchestra.run.trial068.stdout: "mds_bal_mode": "0", 2026-02-23T13:07:28.572 INFO:teuthology.orchestra.run.trial068.stdout: "mds_bal_need_max": "1.200000", 2026-02-23T13:07:28.572 INFO:teuthology.orchestra.run.trial068.stdout: "mds_bal_need_min": "0.800000", 2026-02-23T13:07:28.572 INFO:teuthology.orchestra.run.trial068.stdout: "mds_bal_overload_epochs": "2", 2026-02-23T13:07:28.572 INFO:teuthology.orchestra.run.trial068.stdout: "mds_bal_replicate_threshold": "8000.000000", 2026-02-23T13:07:28.572 INFO:teuthology.orchestra.run.trial068.stdout: "mds_bal_sample_interval": "3.000000", 2026-02-23T13:07:28.572 INFO:teuthology.orchestra.run.trial068.stdout: "mds_bal_split_bits": "3", 2026-02-23T13:07:28.572 INFO:teuthology.orchestra.run.trial068.stdout: "mds_bal_split_rd": "25000.000000", 2026-02-23T13:07:28.572 INFO:teuthology.orchestra.run.trial068.stdout: "mds_bal_split_size": "10000", 2026-02-23T13:07:28.572 INFO:teuthology.orchestra.run.trial068.stdout: "mds_bal_split_wr": "10000.000000", 2026-02-23T13:07:28.572 INFO:teuthology.orchestra.run.trial068.stdout: "mds_bal_target_decay": "10.000000", 2026-02-23T13:07:28.572 INFO:teuthology.orchestra.run.trial068.stdout: "mds_bal_unreplicate_threshold": "0.000000", 2026-02-23T13:07:28.572 INFO:teuthology.orchestra.run.trial068.stdout: "mds_beacon_grace": "15.000000", 2026-02-23T13:07:28.572 INFO:teuthology.orchestra.run.trial068.stdout: "mds_beacon_interval": "4.000000", 2026-02-23T13:07:28.572 INFO:teuthology.orchestra.run.trial068.stdout: "mds_beacon_mon_down_grace": "60", 2026-02-23T13:07:28.572 INFO:teuthology.orchestra.run.trial068.stdout: "mds_cache_memory_limit": "4294967296", 2026-02-23T13:07:28.572 INFO:teuthology.orchestra.run.trial068.stdout: "mds_cache_mid": "0.700000", 2026-02-23T13:07:28.572 INFO:teuthology.orchestra.run.trial068.stdout: "mds_cache_quiesce_decay_rate": "1.000000", 2026-02-23T13:07:28.572 INFO:teuthology.orchestra.run.trial068.stdout: "mds_cache_quiesce_delay": "0", 2026-02-23T13:07:28.572 INFO:teuthology.orchestra.run.trial068.stdout: "mds_cache_quiesce_sleep": "200", 2026-02-23T13:07:28.572 INFO:teuthology.orchestra.run.trial068.stdout: "mds_cache_quiesce_splitauth": "true", 2026-02-23T13:07:28.572 INFO:teuthology.orchestra.run.trial068.stdout: "mds_cache_quiesce_threshold": "524288", 2026-02-23T13:07:28.572 INFO:teuthology.orchestra.run.trial068.stdout: "mds_cache_release_free_interval": "10", 2026-02-23T13:07:28.573 INFO:teuthology.orchestra.run.trial068.stdout: "mds_cache_reservation": "0.050000", 2026-02-23T13:07:28.573 INFO:teuthology.orchestra.run.trial068.stdout: "mds_cache_trim_decay_rate": "1.000000", 2026-02-23T13:07:28.573 INFO:teuthology.orchestra.run.trial068.stdout: "mds_cache_trim_interval": "1", 2026-02-23T13:07:28.573 INFO:teuthology.orchestra.run.trial068.stdout: "mds_cache_trim_threshold": "262144", 2026-02-23T13:07:28.573 INFO:teuthology.orchestra.run.trial068.stdout: "mds_cap_acquisition_throttle_retry_request_timeout": "0.500000", 2026-02-23T13:07:28.573 INFO:teuthology.orchestra.run.trial068.stdout: "mds_cap_revoke_eviction_timeout": "0.000000", 2026-02-23T13:07:28.573 INFO:teuthology.orchestra.run.trial068.stdout: "mds_client_delegate_inos_pct": "50", 2026-02-23T13:07:28.573 INFO:teuthology.orchestra.run.trial068.stdout: "mds_client_prealloc_inos": "1000", 2026-02-23T13:07:28.573 INFO:teuthology.orchestra.run.trial068.stdout: "mds_client_writeable_range_max_inc_objs": "1024", 2026-02-23T13:07:28.573 INFO:teuthology.orchestra.run.trial068.stdout: "mds_connect_bootstrapping": "false", 2026-02-23T13:07:28.573 INFO:teuthology.orchestra.run.trial068.stdout: "mds_damage_table_max_entries": "10000", 2026-02-23T13:07:28.573 INFO:teuthology.orchestra.run.trial068.stdout: "mds_data": "/var/lib/ceph/mds/ceph-a", 2026-02-23T13:07:28.573 INFO:teuthology.orchestra.run.trial068.stdout: "mds_debug_auth_pins": "false", 2026-02-23T13:07:28.573 INFO:teuthology.orchestra.run.trial068.stdout: "mds_debug_frag": "false", 2026-02-23T13:07:28.573 INFO:teuthology.orchestra.run.trial068.stdout: "mds_debug_scatterstat": "false", 2026-02-23T13:07:28.573 INFO:teuthology.orchestra.run.trial068.stdout: "mds_debug_subtrees": "false", 2026-02-23T13:07:28.573 INFO:teuthology.orchestra.run.trial068.stdout: "mds_decay_halflife": "5.000000", 2026-02-23T13:07:28.573 INFO:teuthology.orchestra.run.trial068.stdout: "mds_default_dir_hash": "2", 2026-02-23T13:07:28.573 INFO:teuthology.orchestra.run.trial068.stdout: "mds_defer_session_stale": "true", 2026-02-23T13:07:28.573 INFO:teuthology.orchestra.run.trial068.stdout: "mds_delay_journal_replay_for_testing": "0", 2026-02-23T13:07:28.573 INFO:teuthology.orchestra.run.trial068.stdout: "mds_deny_all_reconnect": "false", 2026-02-23T13:07:28.573 INFO:teuthology.orchestra.run.trial068.stdout: "mds_dir_keys_per_op": "16384", 2026-02-23T13:07:28.573 INFO:teuthology.orchestra.run.trial068.stdout: "mds_dir_max_commit_size": "10", 2026-02-23T13:07:28.573 INFO:teuthology.orchestra.run.trial068.stdout: "mds_dir_max_entries": "0", 2026-02-23T13:07:28.573 INFO:teuthology.orchestra.run.trial068.stdout: "mds_dir_prefetch": "true", 2026-02-23T13:07:28.573 INFO:teuthology.orchestra.run.trial068.stdout: "mds_dirstat_min_interval": "1.000000", 2026-02-23T13:07:28.573 INFO:teuthology.orchestra.run.trial068.stdout: "mds_dump_cache_after_rejoin": "false", 2026-02-23T13:07:28.573 INFO:teuthology.orchestra.run.trial068.stdout: "mds_dump_cache_on_map": "false", 2026-02-23T13:07:28.573 INFO:teuthology.orchestra.run.trial068.stdout: "mds_dump_cache_threshold_file": "0", 2026-02-23T13:07:28.573 INFO:teuthology.orchestra.run.trial068.stdout: "mds_dump_cache_threshold_formatter": "1073741824", 2026-02-23T13:07:28.573 INFO:teuthology.orchestra.run.trial068.stdout: "mds_early_reply": "true", 2026-02-23T13:07:28.573 INFO:teuthology.orchestra.run.trial068.stdout: "mds_enable_op_tracker": "true", 2026-02-23T13:07:28.573 INFO:teuthology.orchestra.run.trial068.stdout: "mds_enforce_unique_name": "true", 2026-02-23T13:07:28.573 INFO:teuthology.orchestra.run.trial068.stdout: "mds_export_ephemeral_distributed": "true", 2026-02-23T13:07:28.573 INFO:teuthology.orchestra.run.trial068.stdout: "mds_export_ephemeral_distributed_factor": "2.000000", 2026-02-23T13:07:28.573 INFO:teuthology.orchestra.run.trial068.stdout: "mds_export_ephemeral_random": "true", 2026-02-23T13:07:28.573 INFO:teuthology.orchestra.run.trial068.stdout: "mds_export_ephemeral_random_max": "0.010000", 2026-02-23T13:07:28.573 INFO:teuthology.orchestra.run.trial068.stdout: "mds_extraordinary_events_dump_interval": "0", 2026-02-23T13:07:28.573 INFO:teuthology.orchestra.run.trial068.stdout: "mds_file_blockdiff_max_concurrent_object_scans": "16", 2026-02-23T13:07:28.573 INFO:teuthology.orchestra.run.trial068.stdout: "mds_forward_all_requests_to_auth": "false", 2026-02-23T13:07:28.573 INFO:teuthology.orchestra.run.trial068.stdout: "mds_freeze_tree_timeout": "30.000000", 2026-02-23T13:07:28.573 INFO:teuthology.orchestra.run.trial068.stdout: "mds_fscrypt_last_block_max_size": "4096", 2026-02-23T13:07:28.574 INFO:teuthology.orchestra.run.trial068.stdout: "mds_go_bad_corrupt_dentry": "true", 2026-02-23T13:07:28.574 INFO:teuthology.orchestra.run.trial068.stdout: "mds_hack_allow_loading_invalid_metadata": "false", 2026-02-23T13:07:28.574 INFO:teuthology.orchestra.run.trial068.stdout: "mds_health_cache_threshold": "1.500000", 2026-02-23T13:07:28.574 INFO:teuthology.orchestra.run.trial068.stdout: "mds_health_summarize_threshold": "10", 2026-02-23T13:07:28.574 INFO:teuthology.orchestra.run.trial068.stdout: "mds_heartbeat_grace": "15.000000", 2026-02-23T13:07:28.574 INFO:teuthology.orchestra.run.trial068.stdout: "mds_heartbeat_reset_grace": "1000", 2026-02-23T13:07:28.574 INFO:teuthology.orchestra.run.trial068.stdout: "mds_inject_health_dummy": "false", 2026-02-23T13:07:28.574 INFO:teuthology.orchestra.run.trial068.stdout: "mds_inject_journal_corrupt_dentry_first": "0.000000", 2026-02-23T13:07:28.574 INFO:teuthology.orchestra.run.trial068.stdout: "mds_inject_migrator_session_race": "false", 2026-02-23T13:07:28.574 INFO:teuthology.orchestra.run.trial068.stdout: "mds_inject_rename_corrupt_dentry_first": "0.000000", 2026-02-23T13:07:28.574 INFO:teuthology.orchestra.run.trial068.stdout: "mds_inject_skip_replaying_inotable": "false", 2026-02-23T13:07:28.574 INFO:teuthology.orchestra.run.trial068.stdout: "mds_inject_traceless_reply_probability": "0.000000", 2026-02-23T13:07:28.574 INFO:teuthology.orchestra.run.trial068.stdout: "mds_join_fs": "", 2026-02-23T13:07:28.574 INFO:teuthology.orchestra.run.trial068.stdout: "mds_journal_format": "1", 2026-02-23T13:07:28.574 INFO:teuthology.orchestra.run.trial068.stdout: "mds_kill_after_journal_logs_flushed": "false", 2026-02-23T13:07:28.574 INFO:teuthology.orchestra.run.trial068.stdout: "mds_kill_create_at": "0", 2026-02-23T13:07:28.574 INFO:teuthology.orchestra.run.trial068.stdout: "mds_kill_dirfrag_at": "0", 2026-02-23T13:07:28.574 INFO:teuthology.orchestra.run.trial068.stdout: "mds_kill_export_at": "0", 2026-02-23T13:07:28.574 INFO:teuthology.orchestra.run.trial068.stdout: "mds_kill_import_at": "0", 2026-02-23T13:07:28.574 INFO:teuthology.orchestra.run.trial068.stdout: "mds_kill_journal_at": "0", 2026-02-23T13:07:28.574 INFO:teuthology.orchestra.run.trial068.stdout: "mds_kill_journal_expire_at": "0", 2026-02-23T13:07:28.574 INFO:teuthology.orchestra.run.trial068.stdout: "mds_kill_journal_replay_at": "0", 2026-02-23T13:07:28.574 INFO:teuthology.orchestra.run.trial068.stdout: "mds_kill_link_at": "0", 2026-02-23T13:07:28.574 INFO:teuthology.orchestra.run.trial068.stdout: "mds_kill_mdstable_at": "0", 2026-02-23T13:07:28.574 INFO:teuthology.orchestra.run.trial068.stdout: "mds_kill_openc_at": "0", 2026-02-23T13:07:28.574 INFO:teuthology.orchestra.run.trial068.stdout: "mds_kill_rename_at": "0", 2026-02-23T13:07:28.574 INFO:teuthology.orchestra.run.trial068.stdout: "mds_kill_shutdown_at": "0", 2026-02-23T13:07:28.574 INFO:teuthology.orchestra.run.trial068.stdout: "mds_log_event_large_threshold": "524288", 2026-02-23T13:07:28.574 INFO:teuthology.orchestra.run.trial068.stdout: "mds_log_events_per_segment": "1024", 2026-02-23T13:07:28.574 INFO:teuthology.orchestra.run.trial068.stdout: "mds_log_max_events": "-1", 2026-02-23T13:07:28.574 INFO:teuthology.orchestra.run.trial068.stdout: "mds_log_max_segments": "128", 2026-02-23T13:07:28.574 INFO:teuthology.orchestra.run.trial068.stdout: "mds_log_minor_segments_per_major_segment": "16", 2026-02-23T13:07:28.574 INFO:teuthology.orchestra.run.trial068.stdout: "mds_log_pause": "false", 2026-02-23T13:07:28.574 INFO:teuthology.orchestra.run.trial068.stdout: "mds_log_segment_size": "0", 2026-02-23T13:07:28.574 INFO:teuthology.orchestra.run.trial068.stdout: "mds_log_skip_corrupt_events": "false", 2026-02-23T13:07:28.574 INFO:teuthology.orchestra.run.trial068.stdout: "mds_log_skip_unbounded_events": "false", 2026-02-23T13:07:28.574 INFO:teuthology.orchestra.run.trial068.stdout: "mds_log_trim_decay_rate": "1.000000", 2026-02-23T13:07:28.574 INFO:teuthology.orchestra.run.trial068.stdout: "mds_log_trim_threshold": "128", 2026-02-23T13:07:28.574 INFO:teuthology.orchestra.run.trial068.stdout: "mds_log_trim_upkeep_interval": "1000", 2026-02-23T13:07:28.574 INFO:teuthology.orchestra.run.trial068.stdout: "mds_log_warn_factor": "2.000000", 2026-02-23T13:07:28.574 INFO:teuthology.orchestra.run.trial068.stdout: "mds_max_caps_per_client": "1048576", 2026-02-23T13:07:28.574 INFO:teuthology.orchestra.run.trial068.stdout: "mds_max_completed_flushes": "100000", 2026-02-23T13:07:28.574 INFO:teuthology.orchestra.run.trial068.stdout: "mds_max_completed_requests": "100000", 2026-02-23T13:07:28.574 INFO:teuthology.orchestra.run.trial068.stdout: "mds_max_export_size": "20971520", 2026-02-23T13:07:28.575 INFO:teuthology.orchestra.run.trial068.stdout: "mds_max_file_recover": "32", 2026-02-23T13:07:28.575 INFO:teuthology.orchestra.run.trial068.stdout: "mds_max_purge_files": "64", 2026-02-23T13:07:28.575 INFO:teuthology.orchestra.run.trial068.stdout: "mds_max_purge_ops": "8192", 2026-02-23T13:07:28.575 INFO:teuthology.orchestra.run.trial068.stdout: "mds_max_purge_ops_per_pg": "0.500000", 2026-02-23T13:07:28.575 INFO:teuthology.orchestra.run.trial068.stdout: "mds_max_scrub_ops_in_progress": "5", 2026-02-23T13:07:28.575 INFO:teuthology.orchestra.run.trial068.stdout: "mds_max_snaps_per_dir": "100", 2026-02-23T13:07:28.575 INFO:teuthology.orchestra.run.trial068.stdout: "mds_metrics_update_interval": "2", 2026-02-23T13:07:28.575 INFO:teuthology.orchestra.run.trial068.stdout: "mds_min_caps_per_client": "100", 2026-02-23T13:07:28.575 INFO:teuthology.orchestra.run.trial068.stdout: "mds_min_caps_working_set": "10000", 2026-02-23T13:07:28.575 INFO:teuthology.orchestra.run.trial068.stdout: "mds_mon_shutdown_timeout": "5.000000", 2026-02-23T13:07:28.575 INFO:teuthology.orchestra.run.trial068.stdout: "mds_numa_node": "-1", 2026-02-23T13:07:28.575 INFO:teuthology.orchestra.run.trial068.stdout: "mds_oft_prefetch_dirfrags": "false", 2026-02-23T13:07:28.575 INFO:teuthology.orchestra.run.trial068.stdout: "mds_op_complaint_time": "30.000000", 2026-02-23T13:07:28.575 INFO:teuthology.orchestra.run.trial068.stdout: "mds_op_history_duration": "600", 2026-02-23T13:07:28.575 INFO:teuthology.orchestra.run.trial068.stdout: "mds_op_history_size": "20", 2026-02-23T13:07:28.575 INFO:teuthology.orchestra.run.trial068.stdout: "mds_op_history_slow_op_size": "20", 2026-02-23T13:07:28.575 INFO:teuthology.orchestra.run.trial068.stdout: "mds_op_history_slow_op_threshold": "10", 2026-02-23T13:07:28.575 INFO:teuthology.orchestra.run.trial068.stdout: "mds_op_log_threshold": "5", 2026-02-23T13:07:28.575 INFO:teuthology.orchestra.run.trial068.stdout: "mds_ping_grace": "15", 2026-02-23T13:07:28.575 INFO:teuthology.orchestra.run.trial068.stdout: "mds_ping_interval": "5", 2026-02-23T13:07:28.575 INFO:teuthology.orchestra.run.trial068.stdout: "mds_purge_queue_busy_flush_period": "1.000000", 2026-02-23T13:07:28.575 INFO:teuthology.orchestra.run.trial068.stdout: "mds_recall_global_max_decay_threshold": "131072", 2026-02-23T13:07:28.575 INFO:teuthology.orchestra.run.trial068.stdout: "mds_recall_max_caps": "30000", 2026-02-23T13:07:28.575 INFO:teuthology.orchestra.run.trial068.stdout: "mds_recall_max_decay_rate": "1.500000", 2026-02-23T13:07:28.575 INFO:teuthology.orchestra.run.trial068.stdout: "mds_recall_max_decay_threshold": "131072", 2026-02-23T13:07:28.575 INFO:teuthology.orchestra.run.trial068.stdout: "mds_recall_warning_decay_rate": "60.000000", 2026-02-23T13:07:28.575 INFO:teuthology.orchestra.run.trial068.stdout: "mds_recall_warning_threshold": "262144", 2026-02-23T13:07:28.575 INFO:teuthology.orchestra.run.trial068.stdout: "mds_reconnect_timeout": "45.000000", 2026-02-23T13:07:28.575 INFO:teuthology.orchestra.run.trial068.stdout: "mds_replay_interval": "1.000000", 2026-02-23T13:07:28.575 INFO:teuthology.orchestra.run.trial068.stdout: "mds_replay_unsafe_with_closed_session": "false", 2026-02-23T13:07:28.575 INFO:teuthology.orchestra.run.trial068.stdout: "mds_request_load_average_decay_rate": "60.000000", 2026-02-23T13:07:28.575 INFO:teuthology.orchestra.run.trial068.stdout: "mds_root_ino_gid": "0", 2026-02-23T13:07:28.575 INFO:teuthology.orchestra.run.trial068.stdout: "mds_root_ino_uid": "0", 2026-02-23T13:07:28.575 INFO:teuthology.orchestra.run.trial068.stdout: "mds_scatter_nudge_interval": "5.000000", 2026-02-23T13:07:28.575 INFO:teuthology.orchestra.run.trial068.stdout: "mds_scrub_stats_review_period": "1", 2026-02-23T13:07:28.575 INFO:teuthology.orchestra.run.trial068.stdout: "mds_server_dispatch_client_request_delay": "0", 2026-02-23T13:07:28.575 INFO:teuthology.orchestra.run.trial068.stdout: "mds_server_dispatch_killpoint_random": "0.000000", 2026-02-23T13:07:28.575 INFO:teuthology.orchestra.run.trial068.stdout: "mds_session_blocklist_on_evict": "true", 2026-02-23T13:07:28.575 INFO:teuthology.orchestra.run.trial068.stdout: "mds_session_blocklist_on_timeout": "true", 2026-02-23T13:07:28.575 INFO:teuthology.orchestra.run.trial068.stdout: "mds_session_cache_liveness_decay_rate": "300.000000", 2026-02-23T13:07:28.575 INFO:teuthology.orchestra.run.trial068.stdout: "mds_session_cache_liveness_magnitude": "10", 2026-02-23T13:07:28.576 INFO:teuthology.orchestra.run.trial068.stdout: "mds_session_cap_acquisition_decay_rate": "30.000000", 2026-02-23T13:07:28.576 INFO:teuthology.orchestra.run.trial068.stdout: "mds_session_cap_acquisition_throttle": "100000", 2026-02-23T13:07:28.576 INFO:teuthology.orchestra.run.trial068.stdout: "mds_session_max_caps_throttle_ratio": "1.100000", 2026-02-23T13:07:28.576 INFO:teuthology.orchestra.run.trial068.stdout: "mds_session_metadata_threshold": "16777216", 2026-02-23T13:07:28.576 INFO:teuthology.orchestra.run.trial068.stdout: "mds_sessionmap_keys_per_op": "1024", 2026-02-23T13:07:28.576 INFO:teuthology.orchestra.run.trial068.stdout: "mds_shutdown_check": "0", 2026-02-23T13:07:28.576 INFO:teuthology.orchestra.run.trial068.stdout: "mds_skip_ino": "0", 2026-02-23T13:07:28.576 INFO:teuthology.orchestra.run.trial068.stdout: "mds_sleep_rank_change": "0.000000", 2026-02-23T13:07:28.576 INFO:teuthology.orchestra.run.trial068.stdout: "mds_snap_max_uid": "4294967294", 2026-02-23T13:07:28.576 INFO:teuthology.orchestra.run.trial068.stdout: "mds_snap_min_uid": "0", 2026-02-23T13:07:28.576 INFO:teuthology.orchestra.run.trial068.stdout: "mds_snap_rstat": "false", 2026-02-23T13:07:28.576 INFO:teuthology.orchestra.run.trial068.stdout: "mds_standby_replay_damaged": "false", 2026-02-23T13:07:28.576 INFO:teuthology.orchestra.run.trial068.stdout: "mds_symlink_recovery": "true", 2026-02-23T13:07:28.576 INFO:teuthology.orchestra.run.trial068.stdout: "mds_task_status_update_interval": "2.000000", 2026-02-23T13:07:28.576 INFO:teuthology.orchestra.run.trial068.stdout: "mds_thrash_exports": "0", 2026-02-23T13:07:28.576 INFO:teuthology.orchestra.run.trial068.stdout: "mds_thrash_fragments": "0", 2026-02-23T13:07:28.576 INFO:teuthology.orchestra.run.trial068.stdout: "mds_tick_interval": "5.000000", 2026-02-23T13:07:28.576 INFO:teuthology.orchestra.run.trial068.stdout: "mds_use_global_snaprealm_seq_for_subvol": "true", 2026-02-23T13:07:28.576 INFO:teuthology.orchestra.run.trial068.stdout: "mds_valgrind_exit": "false", 2026-02-23T13:07:28.576 INFO:teuthology.orchestra.run.trial068.stdout: "mds_verify_backtrace": "1", 2026-02-23T13:07:28.576 INFO:teuthology.orchestra.run.trial068.stdout: "mds_verify_scatter": "false", 2026-02-23T13:07:28.576 INFO:teuthology.orchestra.run.trial068.stdout: "mds_wipe_ino_prealloc": "false", 2026-02-23T13:07:28.576 INFO:teuthology.orchestra.run.trial068.stdout: "mds_wipe_sessions": "false", 2026-02-23T13:07:28.576 INFO:teuthology.orchestra.run.trial068.stdout: "mempool_debug": "false", 2026-02-23T13:07:28.576 INFO:teuthology.orchestra.run.trial068.stdout: "memstore_debug_omit_block_device_write": "false", 2026-02-23T13:07:28.576 INFO:teuthology.orchestra.run.trial068.stdout: "memstore_device_bytes": "1073741824", 2026-02-23T13:07:28.576 INFO:teuthology.orchestra.run.trial068.stdout: "memstore_page_set": "false", 2026-02-23T13:07:28.576 INFO:teuthology.orchestra.run.trial068.stdout: "memstore_page_size": "65536", 2026-02-23T13:07:28.576 INFO:teuthology.orchestra.run.trial068.stdout: "mgr_client_bytes": "134217728", 2026-02-23T13:07:28.576 INFO:teuthology.orchestra.run.trial068.stdout: "mgr_client_messages": "512", 2026-02-23T13:07:28.576 INFO:teuthology.orchestra.run.trial068.stdout: "mgr_client_service_daemon_unregister_timeout": "1.000000", 2026-02-23T13:07:28.576 INFO:teuthology.orchestra.run.trial068.stdout: "mgr_connect_retry_interval": "1.000000", 2026-02-23T13:07:28.576 INFO:teuthology.orchestra.run.trial068.stdout: "mgr_data": "/var/lib/ceph/mgr/ceph-a", 2026-02-23T13:07:28.576 INFO:teuthology.orchestra.run.trial068.stdout: "mgr_debug_aggressive_pg_num_changes": "false", 2026-02-23T13:07:28.576 INFO:teuthology.orchestra.run.trial068.stdout: "mgr_disabled_modules": "", 2026-02-23T13:07:28.576 INFO:teuthology.orchestra.run.trial068.stdout: "mgr_enable_op_tracker": "true", 2026-02-23T13:07:28.576 INFO:teuthology.orchestra.run.trial068.stdout: "mgr_initial_modules": "iostat nfs", 2026-02-23T13:07:28.576 INFO:teuthology.orchestra.run.trial068.stdout: "mgr_max_pg_creating": "1024", 2026-02-23T13:07:28.576 INFO:teuthology.orchestra.run.trial068.stdout: "mgr_max_pg_num_change": "128", 2026-02-23T13:07:28.576 INFO:teuthology.orchestra.run.trial068.stdout: "mgr_mds_bytes": "134217728", 2026-02-23T13:07:28.576 INFO:teuthology.orchestra.run.trial068.stdout: "mgr_mds_messages": "128", 2026-02-23T13:07:28.576 INFO:teuthology.orchestra.run.trial068.stdout: "mgr_module_path": "/usr/share/ceph/mgr", 2026-02-23T13:07:28.577 INFO:teuthology.orchestra.run.trial068.stdout: "mgr_mon_bytes": "134217728", 2026-02-23T13:07:28.577 INFO:teuthology.orchestra.run.trial068.stdout: "mgr_mon_messages": "128", 2026-02-23T13:07:28.577 INFO:teuthology.orchestra.run.trial068.stdout: "mgr_num_op_tracker_shard": "32", 2026-02-23T13:07:28.577 INFO:teuthology.orchestra.run.trial068.stdout: "mgr_op_complaint_time": "30.000000", 2026-02-23T13:07:28.577 INFO:teuthology.orchestra.run.trial068.stdout: "mgr_op_history_duration": "600", 2026-02-23T13:07:28.577 INFO:teuthology.orchestra.run.trial068.stdout: "mgr_op_history_size": "20", 2026-02-23T13:07:28.577 INFO:teuthology.orchestra.run.trial068.stdout: "mgr_op_history_slow_op_size": "20", 2026-02-23T13:07:28.577 INFO:teuthology.orchestra.run.trial068.stdout: "mgr_op_history_slow_op_threshold": "10.000000", 2026-02-23T13:07:28.577 INFO:teuthology.orchestra.run.trial068.stdout: "mgr_op_log_threshold": "5", 2026-02-23T13:07:28.577 INFO:teuthology.orchestra.run.trial068.stdout: "mgr_osd_bytes": "536870912", 2026-02-23T13:07:28.577 INFO:teuthology.orchestra.run.trial068.stdout: "mgr_osd_messages": "8192", 2026-02-23T13:07:28.577 INFO:teuthology.orchestra.run.trial068.stdout: "mgr_pool": "true", 2026-02-23T13:07:28.577 INFO:teuthology.orchestra.run.trial068.stdout: "mgr_service_beacon_grace": "60.000000", 2026-02-23T13:07:28.577 INFO:teuthology.orchestra.run.trial068.stdout: "mgr_standby_modules": "true", 2026-02-23T13:07:28.577 INFO:teuthology.orchestra.run.trial068.stdout: "mgr_stats_period": "5", 2026-02-23T13:07:28.577 INFO:teuthology.orchestra.run.trial068.stdout: "mgr_stats_threshold": "5", 2026-02-23T13:07:28.577 INFO:teuthology.orchestra.run.trial068.stdout: "mgr_tick_period": "2", 2026-02-23T13:07:28.577 INFO:teuthology.orchestra.run.trial068.stdout: "mgr_ttl_cache_expire_seconds": "0", 2026-02-23T13:07:28.577 INFO:teuthology.orchestra.run.trial068.stdout: "mon_accept_timeout_factor": "2.000000", 2026-02-23T13:07:28.577 INFO:teuthology.orchestra.run.trial068.stdout: "mon_allow_pool_delete": "true", 2026-02-23T13:07:28.577 INFO:teuthology.orchestra.run.trial068.stdout: "mon_allow_pool_size_one": "false", 2026-02-23T13:07:28.577 INFO:teuthology.orchestra.run.trial068.stdout: "mon_auth_validate_all_caps": "true", 2026-02-23T13:07:28.577 INFO:teuthology.orchestra.run.trial068.stdout: "mon_cache_target_full_warn_ratio": "0.660000", 2026-02-23T13:07:28.577 INFO:teuthology.orchestra.run.trial068.stdout: "mon_clean_pg_upmaps_per_chunk": "256", 2026-02-23T13:07:28.577 INFO:teuthology.orchestra.run.trial068.stdout: "mon_client_bytes": "104857600", 2026-02-23T13:07:28.577 INFO:teuthology.orchestra.run.trial068.stdout: "mon_client_directed_command_retry": "2", 2026-02-23T13:07:28.577 INFO:teuthology.orchestra.run.trial068.stdout: "mon_client_hunt_interval": "3.000000", 2026-02-23T13:07:28.577 INFO:teuthology.orchestra.run.trial068.stdout: "mon_client_hunt_interval_backoff": "1.500000", 2026-02-23T13:07:28.577 INFO:teuthology.orchestra.run.trial068.stdout: "mon_client_hunt_interval_max_multiple": "10.000000", 2026-02-23T13:07:28.577 INFO:teuthology.orchestra.run.trial068.stdout: "mon_client_hunt_interval_min_multiple": "1.000000", 2026-02-23T13:07:28.577 INFO:teuthology.orchestra.run.trial068.stdout: "mon_client_hunt_on_resend": "true", 2026-02-23T13:07:28.577 INFO:teuthology.orchestra.run.trial068.stdout: "mon_client_hunt_parallel": "3", 2026-02-23T13:07:28.577 INFO:teuthology.orchestra.run.trial068.stdout: "mon_client_log_interval": "1.000000", 2026-02-23T13:07:28.577 INFO:teuthology.orchestra.run.trial068.stdout: "mon_client_max_log_entries_per_message": "1000", 2026-02-23T13:07:28.577 INFO:teuthology.orchestra.run.trial068.stdout: "mon_client_ping_interval": "10.000000", 2026-02-23T13:07:28.577 INFO:teuthology.orchestra.run.trial068.stdout: "mon_client_ping_timeout": "30.000000", 2026-02-23T13:07:28.577 INFO:teuthology.orchestra.run.trial068.stdout: "mon_client_target_rank": "-1", 2026-02-23T13:07:28.577 INFO:teuthology.orchestra.run.trial068.stdout: "mon_clock_drift_allowed": "1.000000", 2026-02-23T13:07:28.577 INFO:teuthology.orchestra.run.trial068.stdout: "mon_clock_drift_warn_backoff": "5.000000", 2026-02-23T13:07:28.577 INFO:teuthology.orchestra.run.trial068.stdout: "mon_cluster_log_file": "default=/var/log/ceph/ceph.$channel.log cluster=/var/log/ceph/ceph.log", 2026-02-23T13:07:28.577 INFO:teuthology.orchestra.run.trial068.stdout: "mon_cluster_log_level": "debug", 2026-02-23T13:07:28.578 INFO:teuthology.orchestra.run.trial068.stdout: "mon_cluster_log_to_file": "true", 2026-02-23T13:07:28.578 INFO:teuthology.orchestra.run.trial068.stdout: "mon_cluster_log_to_graylog": "false", 2026-02-23T13:07:28.578 INFO:teuthology.orchestra.run.trial068.stdout: "mon_cluster_log_to_graylog_host": "127.0.0.1", 2026-02-23T13:07:28.578 INFO:teuthology.orchestra.run.trial068.stdout: "mon_cluster_log_to_graylog_port": "12201", 2026-02-23T13:07:28.578 INFO:teuthology.orchestra.run.trial068.stdout: "mon_cluster_log_to_journald": "true", 2026-02-23T13:07:28.578 INFO:teuthology.orchestra.run.trial068.stdout: "mon_cluster_log_to_stderr": "false", 2026-02-23T13:07:28.578 INFO:teuthology.orchestra.run.trial068.stdout: "mon_cluster_log_to_syslog": "default=false", 2026-02-23T13:07:28.578 INFO:teuthology.orchestra.run.trial068.stdout: "mon_cluster_log_to_syslog_facility": "daemon", 2026-02-23T13:07:28.578 INFO:teuthology.orchestra.run.trial068.stdout: "mon_compact_on_bootstrap": "false", 2026-02-23T13:07:28.578 INFO:teuthology.orchestra.run.trial068.stdout: "mon_compact_on_start": "false", 2026-02-23T13:07:28.578 INFO:teuthology.orchestra.run.trial068.stdout: "mon_compact_on_trim": "true", 2026-02-23T13:07:28.578 INFO:teuthology.orchestra.run.trial068.stdout: "mon_con_tracker_persist_interval": "10", 2026-02-23T13:07:28.578 INFO:teuthology.orchestra.run.trial068.stdout: "mon_con_tracker_score_halflife": "43200", 2026-02-23T13:07:28.578 INFO:teuthology.orchestra.run.trial068.stdout: "mon_config_key_max_entry_size": "65536", 2026-02-23T13:07:28.578 INFO:teuthology.orchestra.run.trial068.stdout: "mon_cpu_threads": "4", 2026-02-23T13:07:28.578 INFO:teuthology.orchestra.run.trial068.stdout: "mon_crush_min_required_version": "hammer", 2026-02-23T13:07:28.578 INFO:teuthology.orchestra.run.trial068.stdout: "mon_daemon_bytes": "419430400", 2026-02-23T13:07:28.578 INFO:teuthology.orchestra.run.trial068.stdout: "mon_data": "/var/lib/ceph/mon/ceph-a", 2026-02-23T13:07:28.578 INFO:teuthology.orchestra.run.trial068.stdout: "mon_data_avail_crit": "5", 2026-02-23T13:07:28.578 INFO:teuthology.orchestra.run.trial068.stdout: "mon_data_avail_warn": "5", 2026-02-23T13:07:28.578 INFO:teuthology.orchestra.run.trial068.stdout: "mon_data_size_warn": "16106127360", 2026-02-23T13:07:28.578 INFO:teuthology.orchestra.run.trial068.stdout: "mon_debug_block_osdmap_trim": "false", 2026-02-23T13:07:28.578 INFO:teuthology.orchestra.run.trial068.stdout: "mon_debug_deprecated_as_obsolete": "false", 2026-02-23T13:07:28.578 INFO:teuthology.orchestra.run.trial068.stdout: "mon_debug_dump_json": "false", 2026-02-23T13:07:28.578 INFO:teuthology.orchestra.run.trial068.stdout: "mon_debug_dump_location": "/var/log/ceph/ceph-mon.a.tdump", 2026-02-23T13:07:28.578 INFO:teuthology.orchestra.run.trial068.stdout: "mon_debug_dump_transactions": "false", 2026-02-23T13:07:28.578 INFO:teuthology.orchestra.run.trial068.stdout: "mon_debug_extra_checks": "false", 2026-02-23T13:07:28.578 INFO:teuthology.orchestra.run.trial068.stdout: "mon_debug_no_initial_persistent_features": "false", 2026-02-23T13:07:28.578 INFO:teuthology.orchestra.run.trial068.stdout: "mon_debug_no_require_bluestore_for_ec_overwrites": "false", 2026-02-23T13:07:28.578 INFO:teuthology.orchestra.run.trial068.stdout: "mon_debug_no_require_squid": "false", 2026-02-23T13:07:28.578 INFO:teuthology.orchestra.run.trial068.stdout: "mon_debug_no_require_tentacle": "false", 2026-02-23T13:07:28.578 INFO:teuthology.orchestra.run.trial068.stdout: "mon_debug_unsafe_allow_tier_with_nonempty_snaps": "false", 2026-02-23T13:07:28.578 INFO:teuthology.orchestra.run.trial068.stdout: "mon_delta_reset_interval": "10.000000", 2026-02-23T13:07:28.578 INFO:teuthology.orchestra.run.trial068.stdout: "mon_dns_srv_name": "ceph-mon", 2026-02-23T13:07:28.578 INFO:teuthology.orchestra.run.trial068.stdout: "mon_down_added_grace": "180", 2026-02-23T13:07:28.578 INFO:teuthology.orchestra.run.trial068.stdout: "mon_down_mkfs_grace": "60", 2026-02-23T13:07:28.578 INFO:teuthology.orchestra.run.trial068.stdout: "mon_down_uptime_grace": "60", 2026-02-23T13:07:28.578 INFO:teuthology.orchestra.run.trial068.stdout: "mon_election_default_strategy": "1", 2026-02-23T13:07:28.578 INFO:teuthology.orchestra.run.trial068.stdout: "mon_election_timeout": "5.000000", 2026-02-23T13:07:28.578 INFO:teuthology.orchestra.run.trial068.stdout: "mon_elector_ignore_propose_margin": "0.000500", 2026-02-23T13:07:28.578 INFO:teuthology.orchestra.run.trial068.stdout: "mon_elector_ping_divisor": "2", 2026-02-23T13:07:28.578 INFO:teuthology.orchestra.run.trial068.stdout: "mon_elector_ping_timeout": "2.000000", 2026-02-23T13:07:28.579 INFO:teuthology.orchestra.run.trial068.stdout: "mon_enable_op_tracker": "true", 2026-02-23T13:07:28.579 INFO:teuthology.orchestra.run.trial068.stdout: "mon_fake_pool_delete": "false", 2026-02-23T13:07:28.579 INFO:teuthology.orchestra.run.trial068.stdout: "mon_force_quorum_join": "false", 2026-02-23T13:07:28.579 INFO:teuthology.orchestra.run.trial068.stdout: "mon_fsmap_prune_threshold": "300", 2026-02-23T13:07:28.579 INFO:teuthology.orchestra.run.trial068.stdout: "mon_globalid_prealloc": "10000", 2026-02-23T13:07:28.579 INFO:teuthology.orchestra.run.trial068.stdout: "mon_health_detail_to_clog": "true", 2026-02-23T13:07:28.579 INFO:teuthology.orchestra.run.trial068.stdout: "mon_health_log_update_period": "5", 2026-02-23T13:07:28.579 INFO:teuthology.orchestra.run.trial068.stdout: "mon_health_max_detail": "50", 2026-02-23T13:07:28.579 INFO:teuthology.orchestra.run.trial068.stdout: "mon_health_to_clog": "true", 2026-02-23T13:07:28.579 INFO:teuthology.orchestra.run.trial068.stdout: "mon_health_to_clog_interval": "600", 2026-02-23T13:07:28.579 INFO:teuthology.orchestra.run.trial068.stdout: "mon_health_to_clog_tick_interval": "60.000000", 2026-02-23T13:07:28.579 INFO:teuthology.orchestra.run.trial068.stdout: "mon_host": "[v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0]", 2026-02-23T13:07:28.579 INFO:teuthology.orchestra.run.trial068.stdout: "mon_host_override": "", 2026-02-23T13:07:28.579 INFO:teuthology.orchestra.run.trial068.stdout: "mon_initial_members": "", 2026-02-23T13:07:28.579 INFO:teuthology.orchestra.run.trial068.stdout: "mon_inject_pg_merge_bounce_probability": "0.000000", 2026-02-23T13:07:28.579 INFO:teuthology.orchestra.run.trial068.stdout: "mon_inject_sync_get_chunk_delay": "0.000000", 2026-02-23T13:07:28.579 INFO:teuthology.orchestra.run.trial068.stdout: "mon_inject_transaction_delay_max": "10.000000", 2026-02-23T13:07:28.579 INFO:teuthology.orchestra.run.trial068.stdout: "mon_inject_transaction_delay_probability": "0.000000", 2026-02-23T13:07:28.579 INFO:teuthology.orchestra.run.trial068.stdout: "mon_keyvaluedb": "rocksdb", 2026-02-23T13:07:28.579 INFO:teuthology.orchestra.run.trial068.stdout: "mon_lease": "5.000000", 2026-02-23T13:07:28.579 INFO:teuthology.orchestra.run.trial068.stdout: "mon_lease_ack_timeout_factor": "2.000000", 2026-02-23T13:07:28.579 INFO:teuthology.orchestra.run.trial068.stdout: "mon_lease_renew_interval_factor": "0.600000", 2026-02-23T13:07:28.579 INFO:teuthology.orchestra.run.trial068.stdout: "mon_log_full_interval": "50", 2026-02-23T13:07:28.579 INFO:teuthology.orchestra.run.trial068.stdout: "mon_log_max": "10000", 2026-02-23T13:07:28.579 INFO:teuthology.orchestra.run.trial068.stdout: "mon_log_max_summary": "50", 2026-02-23T13:07:28.579 INFO:teuthology.orchestra.run.trial068.stdout: "mon_max_log_entries_per_event": "4096", 2026-02-23T13:07:28.579 INFO:teuthology.orchestra.run.trial068.stdout: "mon_max_log_epochs": "500", 2026-02-23T13:07:28.579 INFO:teuthology.orchestra.run.trial068.stdout: "mon_max_mdsmap_epochs": "500", 2026-02-23T13:07:28.579 INFO:teuthology.orchestra.run.trial068.stdout: "mon_max_mgrmap_epochs": "500", 2026-02-23T13:07:28.579 INFO:teuthology.orchestra.run.trial068.stdout: "mon_max_nvmeof_epochs": "500", 2026-02-23T13:07:28.579 INFO:teuthology.orchestra.run.trial068.stdout: "mon_max_osd": "10000", 2026-02-23T13:07:28.579 INFO:teuthology.orchestra.run.trial068.stdout: "mon_max_pg_per_osd": "10000", 2026-02-23T13:07:28.579 INFO:teuthology.orchestra.run.trial068.stdout: "mon_max_pool_pg_num": "65536", 2026-02-23T13:07:28.579 INFO:teuthology.orchestra.run.trial068.stdout: "mon_max_snap_prune_per_epoch": "100", 2026-02-23T13:07:28.579 INFO:teuthology.orchestra.run.trial068.stdout: "mon_mds_blocklist_interval": "86400.000000", 2026-02-23T13:07:28.579 INFO:teuthology.orchestra.run.trial068.stdout: "mon_mds_force_trim_to": "0", 2026-02-23T13:07:28.579 INFO:teuthology.orchestra.run.trial068.stdout: "mon_mds_skip_sanity": "false", 2026-02-23T13:07:28.579 INFO:teuthology.orchestra.run.trial068.stdout: "mon_memory_autotune": "true", 2026-02-23T13:07:28.579 INFO:teuthology.orchestra.run.trial068.stdout: "mon_memory_target": "2147483648", 2026-02-23T13:07:28.579 INFO:teuthology.orchestra.run.trial068.stdout: "mon_mgr_beacon_grace": "30", 2026-02-23T13:07:28.579 INFO:teuthology.orchestra.run.trial068.stdout: "mon_mgr_blocklist_interval": "86400.000000", 2026-02-23T13:07:28.579 INFO:teuthology.orchestra.run.trial068.stdout: "mon_mgr_digest_period": "5", 2026-02-23T13:07:28.579 INFO:teuthology.orchestra.run.trial068.stdout: "mon_mgr_inactive_grace": "60", 2026-02-23T13:07:28.579 INFO:teuthology.orchestra.run.trial068.stdout: "mon_mgr_mkfs_grace": "240", 2026-02-23T13:07:28.580 INFO:teuthology.orchestra.run.trial068.stdout: "mon_mgr_proxy_client_bytes_ratio": "0.300000", 2026-02-23T13:07:28.580 INFO:teuthology.orchestra.run.trial068.stdout: "mon_min_osdmap_epochs": "50", 2026-02-23T13:07:28.580 INFO:teuthology.orchestra.run.trial068.stdout: "mon_netsplit_grace_period": "9", 2026-02-23T13:07:28.580 INFO:teuthology.orchestra.run.trial068.stdout: "mon_nvmeofgw_beacon_grace": "7", 2026-02-23T13:07:28.580 INFO:teuthology.orchestra.run.trial068.stdout: "mon_nvmeofgw_beacons_till_ack": "15", 2026-02-23T13:07:28.580 INFO:teuthology.orchestra.run.trial068.stdout: "mon_nvmeofgw_delete_grace": "900", 2026-02-23T13:07:28.580 INFO:teuthology.orchestra.run.trial068.stdout: "mon_nvmeofgw_set_group_id_retry": "1000", 2026-02-23T13:07:28.580 INFO:teuthology.orchestra.run.trial068.stdout: "mon_nvmeofgw_skip_failovers_interval": "16", 2026-02-23T13:07:28.580 INFO:teuthology.orchestra.run.trial068.stdout: "mon_nvmeofgw_wrong_map_ignore_sec": "15", 2026-02-23T13:07:28.580 INFO:teuthology.orchestra.run.trial068.stdout: "mon_op_complaint_time": "30", 2026-02-23T13:07:28.580 INFO:teuthology.orchestra.run.trial068.stdout: "mon_op_history_duration": "600", 2026-02-23T13:07:28.580 INFO:teuthology.orchestra.run.trial068.stdout: "mon_op_history_size": "20", 2026-02-23T13:07:28.580 INFO:teuthology.orchestra.run.trial068.stdout: "mon_op_history_slow_op_size": "20", 2026-02-23T13:07:28.580 INFO:teuthology.orchestra.run.trial068.stdout: "mon_op_history_slow_op_threshold": "10", 2026-02-23T13:07:28.580 INFO:teuthology.orchestra.run.trial068.stdout: "mon_op_log_threshold": "5", 2026-02-23T13:07:28.580 INFO:teuthology.orchestra.run.trial068.stdout: "mon_osd_adjust_down_out_interval": "true", 2026-02-23T13:07:28.580 INFO:teuthology.orchestra.run.trial068.stdout: "mon_osd_adjust_heartbeat_grace": "true", 2026-02-23T13:07:28.580 INFO:teuthology.orchestra.run.trial068.stdout: "mon_osd_auto_mark_auto_out_in": "true", 2026-02-23T13:07:28.580 INFO:teuthology.orchestra.run.trial068.stdout: "mon_osd_auto_mark_in": "false", 2026-02-23T13:07:28.580 INFO:teuthology.orchestra.run.trial068.stdout: "mon_osd_auto_mark_new_in": "true", 2026-02-23T13:07:28.580 INFO:teuthology.orchestra.run.trial068.stdout: "mon_osd_backfillfull_ratio": "0.900000", 2026-02-23T13:07:28.580 INFO:teuthology.orchestra.run.trial068.stdout: "mon_osd_blocklist_default_expire": "3600.000000", 2026-02-23T13:07:28.580 INFO:teuthology.orchestra.run.trial068.stdout: "mon_osd_cache_size": "500", 2026-02-23T13:07:28.580 INFO:teuthology.orchestra.run.trial068.stdout: "mon_osd_cache_size_min": "134217728", 2026-02-23T13:07:28.580 INFO:teuthology.orchestra.run.trial068.stdout: "mon_osd_crush_smoke_test": "true", 2026-02-23T13:07:28.580 INFO:teuthology.orchestra.run.trial068.stdout: "mon_osd_destroyed_out_interval": "600", 2026-02-23T13:07:28.580 INFO:teuthology.orchestra.run.trial068.stdout: "mon_osd_down_out_interval": "600", 2026-02-23T13:07:28.580 INFO:teuthology.orchestra.run.trial068.stdout: "mon_osd_down_out_subtree_limit": "rack", 2026-02-23T13:07:28.580 INFO:teuthology.orchestra.run.trial068.stdout: "mon_osd_err_op_age_ratio": "128.000000", 2026-02-23T13:07:28.580 INFO:teuthology.orchestra.run.trial068.stdout: "mon_osd_force_trim_to": "0", 2026-02-23T13:07:28.580 INFO:teuthology.orchestra.run.trial068.stdout: "mon_osd_full_ratio": "0.950000", 2026-02-23T13:07:28.580 INFO:teuthology.orchestra.run.trial068.stdout: "mon_osd_initial_require_min_compat_client": "luminous", 2026-02-23T13:07:28.580 INFO:teuthology.orchestra.run.trial068.stdout: "mon_osd_laggy_halflife": "3600", 2026-02-23T13:07:28.580 INFO:teuthology.orchestra.run.trial068.stdout: "mon_osd_laggy_max_interval": "300", 2026-02-23T13:07:28.580 INFO:teuthology.orchestra.run.trial068.stdout: "mon_osd_laggy_weight": "0.300000", 2026-02-23T13:07:28.580 INFO:teuthology.orchestra.run.trial068.stdout: "mon_osd_mapping_pgs_per_chunk": "4096", 2026-02-23T13:07:28.580 INFO:teuthology.orchestra.run.trial068.stdout: "mon_osd_max_initial_pgs": "1024", 2026-02-23T13:07:28.580 INFO:teuthology.orchestra.run.trial068.stdout: "mon_osd_min_down_reporters": "2", 2026-02-23T13:07:28.580 INFO:teuthology.orchestra.run.trial068.stdout: "mon_osd_min_in_ratio": "0.750000", 2026-02-23T13:07:28.580 INFO:teuthology.orchestra.run.trial068.stdout: "mon_osd_min_up_ratio": "0.300000", 2026-02-23T13:07:28.580 INFO:teuthology.orchestra.run.trial068.stdout: "mon_osd_nearfull_ratio": "0.850000", 2026-02-23T13:07:28.580 INFO:teuthology.orchestra.run.trial068.stdout: "mon_osd_prime_pg_temp": "true", 2026-02-23T13:07:28.580 INFO:teuthology.orchestra.run.trial068.stdout: "mon_osd_prime_pg_temp_max_estimate": "0.250000", 2026-02-23T13:07:28.581 INFO:teuthology.orchestra.run.trial068.stdout: "mon_osd_prime_pg_temp_max_time": "0.500000", 2026-02-23T13:07:28.581 INFO:teuthology.orchestra.run.trial068.stdout: "mon_osd_report_timeout": "900", 2026-02-23T13:07:28.581 INFO:teuthology.orchestra.run.trial068.stdout: "mon_osd_reporter_subtree_level": "osd", 2026-02-23T13:07:28.581 INFO:teuthology.orchestra.run.trial068.stdout: "mon_osd_snap_trim_queue_warn_on": "32768", 2026-02-23T13:07:28.581 INFO:teuthology.orchestra.run.trial068.stdout: "mon_osd_warn_num_repaired": "10", 2026-02-23T13:07:28.581 INFO:teuthology.orchestra.run.trial068.stdout: "mon_osd_warn_op_age": "32.000000", 2026-02-23T13:07:28.581 INFO:teuthology.orchestra.run.trial068.stdout: "mon_osdmap_full_prune_enabled": "true", 2026-02-23T13:07:28.581 INFO:teuthology.orchestra.run.trial068.stdout: "mon_osdmap_full_prune_interval": "2", 2026-02-23T13:07:28.581 INFO:teuthology.orchestra.run.trial068.stdout: "mon_osdmap_full_prune_min": "15", 2026-02-23T13:07:28.581 INFO:teuthology.orchestra.run.trial068.stdout: "mon_osdmap_full_prune_txsize": "2", 2026-02-23T13:07:28.581 INFO:teuthology.orchestra.run.trial068.stdout: "mon_pg_check_down_all_threshold": "0.500000", 2026-02-23T13:07:28.581 INFO:teuthology.orchestra.run.trial068.stdout: "mon_pg_stuck_threshold": "60", 2026-02-23T13:07:28.581 INFO:teuthology.orchestra.run.trial068.stdout: "mon_pg_warn_max_object_skew": "0.000000", 2026-02-23T13:07:28.581 INFO:teuthology.orchestra.run.trial068.stdout: "mon_pg_warn_min_objects": "10000", 2026-02-23T13:07:28.581 INFO:teuthology.orchestra.run.trial068.stdout: "mon_pg_warn_min_per_osd": "0", 2026-02-23T13:07:28.581 INFO:teuthology.orchestra.run.trial068.stdout: "mon_pg_warn_min_pool_objects": "1000", 2026-02-23T13:07:28.581 INFO:teuthology.orchestra.run.trial068.stdout: "mon_pool_quota_crit_threshold": "0", 2026-02-23T13:07:28.581 INFO:teuthology.orchestra.run.trial068.stdout: "mon_pool_quota_warn_threshold": "0", 2026-02-23T13:07:28.581 INFO:teuthology.orchestra.run.trial068.stdout: "mon_probe_timeout": "2.000000", 2026-02-23T13:07:28.581 INFO:teuthology.orchestra.run.trial068.stdout: "mon_reweight_max_change": "0.050000", 2026-02-23T13:07:28.581 INFO:teuthology.orchestra.run.trial068.stdout: "mon_reweight_max_osds": "4", 2026-02-23T13:07:28.581 INFO:teuthology.orchestra.run.trial068.stdout: "mon_reweight_min_bytes_per_osd": "10", 2026-02-23T13:07:28.581 INFO:teuthology.orchestra.run.trial068.stdout: "mon_reweight_min_pgs_per_osd": "4", 2026-02-23T13:07:28.581 INFO:teuthology.orchestra.run.trial068.stdout: "mon_rocksdb_options": "write_buffer_size=33554432,compression=kNoCompression,level_compaction_dynamic_level_bytes=true", 2026-02-23T13:07:28.581 INFO:teuthology.orchestra.run.trial068.stdout: "mon_scrub_inject_crc_mismatch": "0.000000", 2026-02-23T13:07:28.581 INFO:teuthology.orchestra.run.trial068.stdout: "mon_scrub_inject_missing_keys": "0.000000", 2026-02-23T13:07:28.581 INFO:teuthology.orchestra.run.trial068.stdout: "mon_scrub_interval": "86400", 2026-02-23T13:07:28.581 INFO:teuthology.orchestra.run.trial068.stdout: "mon_scrub_max_keys": "100", 2026-02-23T13:07:28.581 INFO:teuthology.orchestra.run.trial068.stdout: "mon_scrub_timeout": "300", 2026-02-23T13:07:28.581 INFO:teuthology.orchestra.run.trial068.stdout: "mon_session_timeout": "300", 2026-02-23T13:07:28.581 INFO:teuthology.orchestra.run.trial068.stdout: "mon_smart_report_timeout": "5", 2026-02-23T13:07:28.581 INFO:teuthology.orchestra.run.trial068.stdout: "mon_stat_smooth_intervals": "6", 2026-02-23T13:07:28.581 INFO:teuthology.orchestra.run.trial068.stdout: "mon_stretch_cluster_recovery_ratio": "0.600000", 2026-02-23T13:07:28.581 INFO:teuthology.orchestra.run.trial068.stdout: "mon_stretch_pool_min_size": "2", 2026-02-23T13:07:28.581 INFO:teuthology.orchestra.run.trial068.stdout: "mon_stretch_pool_size": "4", 2026-02-23T13:07:28.581 INFO:teuthology.orchestra.run.trial068.stdout: "mon_stretch_recovery_min_wait": "15.000000", 2026-02-23T13:07:28.581 INFO:teuthology.orchestra.run.trial068.stdout: "mon_subscribe_interval": "86400.000000", 2026-02-23T13:07:28.581 INFO:teuthology.orchestra.run.trial068.stdout: "mon_sync_debug": "false", 2026-02-23T13:07:28.581 INFO:teuthology.orchestra.run.trial068.stdout: "mon_sync_max_payload_keys": "2000", 2026-02-23T13:07:28.581 INFO:teuthology.orchestra.run.trial068.stdout: "mon_sync_max_payload_size": "1048576", 2026-02-23T13:07:28.581 INFO:teuthology.orchestra.run.trial068.stdout: "mon_sync_provider_kill_at": "0", 2026-02-23T13:07:28.581 INFO:teuthology.orchestra.run.trial068.stdout: "mon_sync_requester_kill_at": "0", 2026-02-23T13:07:28.582 INFO:teuthology.orchestra.run.trial068.stdout: "mon_sync_timeout": "60.000000", 2026-02-23T13:07:28.582 INFO:teuthology.orchestra.run.trial068.stdout: "mon_target_pg_per_osd": "100", 2026-02-23T13:07:28.582 INFO:teuthology.orchestra.run.trial068.stdout: "mon_tick_interval": "5", 2026-02-23T13:07:28.582 INFO:teuthology.orchestra.run.trial068.stdout: "mon_timecheck_interval": "300.000000", 2026-02-23T13:07:28.582 INFO:teuthology.orchestra.run.trial068.stdout: "mon_timecheck_skew_interval": "30.000000", 2026-02-23T13:07:28.582 INFO:teuthology.orchestra.run.trial068.stdout: "mon_use_min_delay_socket": "false", 2026-02-23T13:07:28.582 INFO:teuthology.orchestra.run.trial068.stdout: "mon_warn_older_version_delay": "604800", 2026-02-23T13:07:28.582 INFO:teuthology.orchestra.run.trial068.stdout: "mon_warn_on_cache_pools_without_hit_sets": "true", 2026-02-23T13:07:28.582 INFO:teuthology.orchestra.run.trial068.stdout: "mon_warn_on_crush_straw_calc_version_zero": "false", 2026-02-23T13:07:28.582 INFO:teuthology.orchestra.run.trial068.stdout: "mon_warn_on_degraded_stretch_mode": "true", 2026-02-23T13:07:28.582 INFO:teuthology.orchestra.run.trial068.stdout: "mon_warn_on_filestore_osds": "true", 2026-02-23T13:07:28.582 INFO:teuthology.orchestra.run.trial068.stdout: "mon_warn_on_insecure_global_id_reclaim": "false", 2026-02-23T13:07:28.582 INFO:teuthology.orchestra.run.trial068.stdout: "mon_warn_on_insecure_global_id_reclaim_allowed": "false", 2026-02-23T13:07:28.582 INFO:teuthology.orchestra.run.trial068.stdout: "mon_warn_on_legacy_crush_tunables": "false", 2026-02-23T13:07:28.582 INFO:teuthology.orchestra.run.trial068.stdout: "mon_warn_on_misplaced": "false", 2026-02-23T13:07:28.582 INFO:teuthology.orchestra.run.trial068.stdout: "mon_warn_on_msgr2_not_enabled": "true", 2026-02-23T13:07:28.582 INFO:teuthology.orchestra.run.trial068.stdout: "mon_warn_on_older_version": "true", 2026-02-23T13:07:28.582 INFO:teuthology.orchestra.run.trial068.stdout: "mon_warn_on_osd_down_out_interval_zero": "false", 2026-02-23T13:07:28.582 INFO:teuthology.orchestra.run.trial068.stdout: "mon_warn_on_pool_no_app": "true", 2026-02-23T13:07:28.582 INFO:teuthology.orchestra.run.trial068.stdout: "mon_warn_on_pool_no_app_grace": "300", 2026-02-23T13:07:28.582 INFO:teuthology.orchestra.run.trial068.stdout: "mon_warn_on_pool_no_redundancy": "true", 2026-02-23T13:07:28.582 INFO:teuthology.orchestra.run.trial068.stdout: "mon_warn_on_pool_pg_num_not_power_of_two": "false", 2026-02-23T13:07:28.582 INFO:teuthology.orchestra.run.trial068.stdout: "mon_warn_on_slow_ping_ratio": "0.050000", 2026-02-23T13:07:28.582 INFO:teuthology.orchestra.run.trial068.stdout: "mon_warn_on_slow_ping_time": "0.000000", 2026-02-23T13:07:28.582 INFO:teuthology.orchestra.run.trial068.stdout: "mon_warn_on_too_few_osds": "false", 2026-02-23T13:07:28.582 INFO:teuthology.orchestra.run.trial068.stdout: "mon_warn_pg_not_deep_scrubbed_ratio": "0.750000", 2026-02-23T13:07:28.582 INFO:teuthology.orchestra.run.trial068.stdout: "mon_warn_pg_not_scrubbed_ratio": "0.500000", 2026-02-23T13:07:28.582 INFO:teuthology.orchestra.run.trial068.stdout: "monmap": "", 2026-02-23T13:07:28.582 INFO:teuthology.orchestra.run.trial068.stdout: "motr_admin_endpoint": "192.168.180.182@tcp:12345:4:1", 2026-02-23T13:07:28.582 INFO:teuthology.orchestra.run.trial068.stdout: "motr_admin_fid": "0x7200000000000001:0x0", 2026-02-23T13:07:28.582 INFO:teuthology.orchestra.run.trial068.stdout: "motr_ha_endpoint": "192.168.180.182@tcp:12345:1:1", 2026-02-23T13:07:28.582 INFO:teuthology.orchestra.run.trial068.stdout: "motr_my_endpoint": "192.168.180.182@tcp:12345:4:1", 2026-02-23T13:07:28.582 INFO:teuthology.orchestra.run.trial068.stdout: "motr_my_fid": "0x7200000000000001:0x0", 2026-02-23T13:07:28.582 INFO:teuthology.orchestra.run.trial068.stdout: "motr_profile_fid": "0x7000000000000001:0x0", 2026-02-23T13:07:28.582 INFO:teuthology.orchestra.run.trial068.stdout: "motr_tracing_enabled": "false", 2026-02-23T13:07:28.582 INFO:teuthology.orchestra.run.trial068.stdout: "ms_async_op_threads": "3", 2026-02-23T13:07:28.582 INFO:teuthology.orchestra.run.trial068.stdout: "ms_async_rdma_buffer_size": "131072", 2026-02-23T13:07:28.582 INFO:teuthology.orchestra.run.trial068.stdout: "ms_async_rdma_cm": "false", 2026-02-23T13:07:28.582 INFO:teuthology.orchestra.run.trial068.stdout: "ms_async_rdma_device_name": "", 2026-02-23T13:07:28.582 INFO:teuthology.orchestra.run.trial068.stdout: "ms_async_rdma_dscp": "96", 2026-02-23T13:07:28.582 INFO:teuthology.orchestra.run.trial068.stdout: "ms_async_rdma_enable_hugepage": "false", 2026-02-23T13:07:28.582 INFO:teuthology.orchestra.run.trial068.stdout: "ms_async_rdma_gid_idx": "0", 2026-02-23T13:07:28.582 INFO:teuthology.orchestra.run.trial068.stdout: "ms_async_rdma_local_gid": "", 2026-02-23T13:07:28.582 INFO:teuthology.orchestra.run.trial068.stdout: "ms_async_rdma_polling_us": "1000", 2026-02-23T13:07:28.583 INFO:teuthology.orchestra.run.trial068.stdout: "ms_async_rdma_port_num": "1", 2026-02-23T13:07:28.583 INFO:teuthology.orchestra.run.trial068.stdout: "ms_async_rdma_receive_buffers": "32768", 2026-02-23T13:07:28.583 INFO:teuthology.orchestra.run.trial068.stdout: "ms_async_rdma_receive_queue_len": "4096", 2026-02-23T13:07:28.583 INFO:teuthology.orchestra.run.trial068.stdout: "ms_async_rdma_roce_ver": "1", 2026-02-23T13:07:28.583 INFO:teuthology.orchestra.run.trial068.stdout: "ms_async_rdma_send_buffers": "1024", 2026-02-23T13:07:28.583 INFO:teuthology.orchestra.run.trial068.stdout: "ms_async_rdma_sl": "3", 2026-02-23T13:07:28.583 INFO:teuthology.orchestra.run.trial068.stdout: "ms_async_rdma_support_srq": "true", 2026-02-23T13:07:28.583 INFO:teuthology.orchestra.run.trial068.stdout: "ms_async_rdma_type": "ib", 2026-02-23T13:07:28.583 INFO:teuthology.orchestra.run.trial068.stdout: "ms_async_reap_threshold": "5", 2026-02-23T13:07:28.583 INFO:teuthology.orchestra.run.trial068.stdout: "ms_bind_before_connect": "false", 2026-02-23T13:07:28.583 INFO:teuthology.orchestra.run.trial068.stdout: "ms_bind_ipv4": "true", 2026-02-23T13:07:28.583 INFO:teuthology.orchestra.run.trial068.stdout: "ms_bind_ipv6": "false", 2026-02-23T13:07:28.583 INFO:teuthology.orchestra.run.trial068.stdout: "ms_bind_msgr1": "true", 2026-02-23T13:07:28.583 INFO:teuthology.orchestra.run.trial068.stdout: "ms_bind_msgr2": "true", 2026-02-23T13:07:28.583 INFO:teuthology.orchestra.run.trial068.stdout: "ms_bind_port_max": "7568", 2026-02-23T13:07:28.583 INFO:teuthology.orchestra.run.trial068.stdout: "ms_bind_port_min": "6800", 2026-02-23T13:07:28.583 INFO:teuthology.orchestra.run.trial068.stdout: "ms_bind_prefer_ipv4": "false", 2026-02-23T13:07:28.583 INFO:teuthology.orchestra.run.trial068.stdout: "ms_bind_retry_count": "3", 2026-02-23T13:07:28.583 INFO:teuthology.orchestra.run.trial068.stdout: "ms_bind_retry_delay": "5", 2026-02-23T13:07:28.583 INFO:teuthology.orchestra.run.trial068.stdout: "ms_blackhole_client": "false", 2026-02-23T13:07:28.583 INFO:teuthology.orchestra.run.trial068.stdout: "ms_blackhole_mds": "false", 2026-02-23T13:07:28.583 INFO:teuthology.orchestra.run.trial068.stdout: "ms_blackhole_mgr": "false", 2026-02-23T13:07:28.583 INFO:teuthology.orchestra.run.trial068.stdout: "ms_blackhole_mon": "false", 2026-02-23T13:07:28.583 INFO:teuthology.orchestra.run.trial068.stdout: "ms_blackhole_osd": "false", 2026-02-23T13:07:28.583 INFO:teuthology.orchestra.run.trial068.stdout: "ms_client_mode": "crc", 2026-02-23T13:07:28.583 INFO:teuthology.orchestra.run.trial068.stdout: "ms_client_throttle_retry_time_interval": "5000", 2026-02-23T13:07:28.583 INFO:teuthology.orchestra.run.trial068.stdout: "ms_cluster_mode": "crc", 2026-02-23T13:07:28.583 INFO:teuthology.orchestra.run.trial068.stdout: "ms_cluster_type": "", 2026-02-23T13:07:28.583 INFO:teuthology.orchestra.run.trial068.stdout: "ms_compress_secure": "false", 2026-02-23T13:07:28.583 INFO:teuthology.orchestra.run.trial068.stdout: "ms_connection_idle_timeout": "900", 2026-02-23T13:07:28.583 INFO:teuthology.orchestra.run.trial068.stdout: "ms_connection_ready_timeout": "10", 2026-02-23T13:07:28.583 INFO:teuthology.orchestra.run.trial068.stdout: "ms_crc_data": "true", 2026-02-23T13:07:28.583 INFO:teuthology.orchestra.run.trial068.stdout: "ms_crc_header": "true", 2026-02-23T13:07:28.583 INFO:teuthology.orchestra.run.trial068.stdout: "ms_die_on_bad_msg": "false", 2026-02-23T13:07:28.583 INFO:teuthology.orchestra.run.trial068.stdout: "ms_die_on_bug": "true", 2026-02-23T13:07:28.583 INFO:teuthology.orchestra.run.trial068.stdout: "ms_die_on_old_message": "true", 2026-02-23T13:07:28.583 INFO:teuthology.orchestra.run.trial068.stdout: "ms_die_on_skipped_message": "false", 2026-02-23T13:07:28.583 INFO:teuthology.orchestra.run.trial068.stdout: "ms_die_on_unhandled_msg": "false", 2026-02-23T13:07:28.583 INFO:teuthology.orchestra.run.trial068.stdout: "ms_dispatch_throttle_bytes": "104857600", 2026-02-23T13:07:28.583 INFO:teuthology.orchestra.run.trial068.stdout: "ms_dpdk_coremask": "0xF", 2026-02-23T13:07:28.583 INFO:teuthology.orchestra.run.trial068.stdout: "ms_dpdk_debug_allow_loopback": "false", 2026-02-23T13:07:28.583 INFO:teuthology.orchestra.run.trial068.stdout: "ms_dpdk_devs_allowlist": "", 2026-02-23T13:07:28.583 INFO:teuthology.orchestra.run.trial068.stdout: "ms_dpdk_enable_tso": "true", 2026-02-23T13:07:28.584 INFO:teuthology.orchestra.run.trial068.stdout: "ms_dpdk_gateway_ipv4_addr": "", 2026-02-23T13:07:28.584 INFO:teuthology.orchestra.run.trial068.stdout: "ms_dpdk_host_ipv4_addr": "", 2026-02-23T13:07:28.584 INFO:teuthology.orchestra.run.trial068.stdout: "ms_dpdk_hugepages": "", 2026-02-23T13:07:28.584 INFO:teuthology.orchestra.run.trial068.stdout: "ms_dpdk_hw_flow_control": "true", 2026-02-23T13:07:28.584 INFO:teuthology.orchestra.run.trial068.stdout: "ms_dpdk_hw_queue_weight": "1.000000", 2026-02-23T13:07:28.584 INFO:teuthology.orchestra.run.trial068.stdout: "ms_dpdk_lro": "true", 2026-02-23T13:07:28.584 INFO:teuthology.orchestra.run.trial068.stdout: "ms_dpdk_memory_channel": "4", 2026-02-23T13:07:28.584 INFO:teuthology.orchestra.run.trial068.stdout: "ms_dpdk_netmask_ipv4_addr": "", 2026-02-23T13:07:28.584 INFO:teuthology.orchestra.run.trial068.stdout: "ms_dpdk_pmd": "", 2026-02-23T13:07:28.584 INFO:teuthology.orchestra.run.trial068.stdout: "ms_dpdk_port_id": "0", 2026-02-23T13:07:28.584 INFO:teuthology.orchestra.run.trial068.stdout: "ms_dpdk_rx_buffer_count_per_core": "8192", 2026-02-23T13:07:28.584 INFO:teuthology.orchestra.run.trial068.stdout: "ms_dump_corrupt_message_level": "1", 2026-02-23T13:07:28.584 INFO:teuthology.orchestra.run.trial068.stdout: "ms_dump_on_send": "false", 2026-02-23T13:07:28.584 INFO:teuthology.orchestra.run.trial068.stdout: "ms_initial_backoff": "0.200000", 2026-02-23T13:07:28.584 INFO:teuthology.orchestra.run.trial068.stdout: "ms_inject_delay_max": "1.000000", 2026-02-23T13:07:28.584 INFO:teuthology.orchestra.run.trial068.stdout: "ms_inject_delay_probability": "0.000000", 2026-02-23T13:07:28.584 INFO:teuthology.orchestra.run.trial068.stdout: "ms_inject_delay_type": "", 2026-02-23T13:07:28.584 INFO:teuthology.orchestra.run.trial068.stdout: "ms_inject_internal_delays": "0.000000", 2026-02-23T13:07:28.584 INFO:teuthology.orchestra.run.trial068.stdout: "ms_inject_network_congestion": "0", 2026-02-23T13:07:28.584 INFO:teuthology.orchestra.run.trial068.stdout: "ms_inject_socket_failures": "0", 2026-02-23T13:07:28.584 INFO:teuthology.orchestra.run.trial068.stdout: "ms_learn_addr_from_peer": "true", 2026-02-23T13:07:28.584 INFO:teuthology.orchestra.run.trial068.stdout: "ms_max_accept_failures": "4", 2026-02-23T13:07:28.584 INFO:teuthology.orchestra.run.trial068.stdout: "ms_max_backoff": "15.000000", 2026-02-23T13:07:28.584 INFO:teuthology.orchestra.run.trial068.stdout: "ms_mon_client_mode": "crc", 2026-02-23T13:07:28.584 INFO:teuthology.orchestra.run.trial068.stdout: "ms_mon_cluster_mode": "crc", 2026-02-23T13:07:28.584 INFO:teuthology.orchestra.run.trial068.stdout: "ms_mon_service_mode": "crc", 2026-02-23T13:07:28.584 INFO:teuthology.orchestra.run.trial068.stdout: "ms_osd_compress_min_size": "1024", 2026-02-23T13:07:28.584 INFO:teuthology.orchestra.run.trial068.stdout: "ms_osd_compress_mode": "none", 2026-02-23T13:07:28.584 INFO:teuthology.orchestra.run.trial068.stdout: "ms_osd_compression_algorithm": "snappy", 2026-02-23T13:07:28.584 INFO:teuthology.orchestra.run.trial068.stdout: "ms_pq_max_tokens_per_priority": "16777216", 2026-02-23T13:07:28.584 INFO:teuthology.orchestra.run.trial068.stdout: "ms_pq_min_cost": "65536", 2026-02-23T13:07:28.584 INFO:teuthology.orchestra.run.trial068.stdout: "ms_public_type": "", 2026-02-23T13:07:28.584 INFO:teuthology.orchestra.run.trial068.stdout: "ms_service_mode": "crc", 2026-02-23T13:07:28.584 INFO:teuthology.orchestra.run.trial068.stdout: "ms_tcp_listen_backlog": "512", 2026-02-23T13:07:28.584 INFO:teuthology.orchestra.run.trial068.stdout: "ms_tcp_nodelay": "true", 2026-02-23T13:07:28.584 INFO:teuthology.orchestra.run.trial068.stdout: "ms_tcp_prefetch_max_size": "65536", 2026-02-23T13:07:28.584 INFO:teuthology.orchestra.run.trial068.stdout: "ms_tcp_rcvbuf": "0", 2026-02-23T13:07:28.584 INFO:teuthology.orchestra.run.trial068.stdout: "ms_time_events_min_wait_interval": "1000", 2026-02-23T13:07:28.584 INFO:teuthology.orchestra.run.trial068.stdout: "ms_type": "async+posix", 2026-02-23T13:07:28.584 INFO:teuthology.orchestra.run.trial068.stdout: "no_config_file": "false", 2026-02-23T13:07:28.584 INFO:teuthology.orchestra.run.trial068.stdout: "nvmeof_mon_client_connect_panic": "30", 2026-02-23T13:07:28.584 INFO:teuthology.orchestra.run.trial068.stdout: "nvmeof_mon_client_disconnect_panic": "100", 2026-02-23T13:07:28.584 INFO:teuthology.orchestra.run.trial068.stdout: "nvmeof_mon_client_tick_period": "1", 2026-02-23T13:07:28.585 INFO:teuthology.orchestra.run.trial068.stdout: "objecter_completion_locks_per_session": "32", 2026-02-23T13:07:28.585 INFO:teuthology.orchestra.run.trial068.stdout: "objecter_debug_inject_relock_delay": "false", 2026-02-23T13:07:28.585 INFO:teuthology.orchestra.run.trial068.stdout: "objecter_inflight_op_bytes": "104857600", 2026-02-23T13:07:28.585 INFO:teuthology.orchestra.run.trial068.stdout: "objecter_inflight_ops": "1024", 2026-02-23T13:07:28.585 INFO:teuthology.orchestra.run.trial068.stdout: "objecter_inject_no_watch_ping": "false", 2026-02-23T13:07:28.585 INFO:teuthology.orchestra.run.trial068.stdout: "objecter_retry_writes_after_first_reply": "false", 2026-02-23T13:07:28.585 INFO:teuthology.orchestra.run.trial068.stdout: "objecter_tick_interval": "5.000000", 2026-02-23T13:07:28.585 INFO:teuthology.orchestra.run.trial068.stdout: "objecter_timeout": "10.000000", 2026-02-23T13:07:28.585 INFO:teuthology.orchestra.run.trial068.stdout: "objectstore_blackhole": "false", 2026-02-23T13:07:28.585 INFO:teuthology.orchestra.run.trial068.stdout: "objectstore_debug_throw_on_failed_txc": "false", 2026-02-23T13:07:28.585 INFO:teuthology.orchestra.run.trial068.stdout: "openssl_conf": "", 2026-02-23T13:07:28.585 INFO:teuthology.orchestra.run.trial068.stdout: "osd_agent_delay_time": "5.000000", 2026-02-23T13:07:28.585 INFO:teuthology.orchestra.run.trial068.stdout: "osd_agent_hist_halflife": "1000", 2026-02-23T13:07:28.585 INFO:teuthology.orchestra.run.trial068.stdout: "osd_agent_max_low_ops": "2", 2026-02-23T13:07:28.585 INFO:teuthology.orchestra.run.trial068.stdout: "osd_agent_max_ops": "4", 2026-02-23T13:07:28.585 INFO:teuthology.orchestra.run.trial068.stdout: "osd_agent_min_evict_effort": "0.100000", 2026-02-23T13:07:28.585 INFO:teuthology.orchestra.run.trial068.stdout: "osd_agent_quantize_effort": "0.100000", 2026-02-23T13:07:28.585 INFO:teuthology.orchestra.run.trial068.stdout: "osd_agent_slop": "0.020000", 2026-02-23T13:07:28.585 INFO:teuthology.orchestra.run.trial068.stdout: "osd_aggregated_slow_ops_logging": "true", 2026-02-23T13:07:28.585 INFO:teuthology.orchestra.run.trial068.stdout: "osd_allow_recovery_below_min_size": "true", 2026-02-23T13:07:28.585 INFO:teuthology.orchestra.run.trial068.stdout: "osd_asio_thread_count": "2", 2026-02-23T13:07:28.585 INFO:teuthology.orchestra.run.trial068.stdout: "osd_async_recovery_min_cost": "100", 2026-02-23T13:07:28.585 INFO:teuthology.orchestra.run.trial068.stdout: "osd_auto_mark_unfound_lost": "false", 2026-02-23T13:07:28.585 INFO:teuthology.orchestra.run.trial068.stdout: "osd_backfill_retry_interval": "30.000000", 2026-02-23T13:07:28.585 INFO:teuthology.orchestra.run.trial068.stdout: "osd_backfill_scan_max": "512", 2026-02-23T13:07:28.585 INFO:teuthology.orchestra.run.trial068.stdout: "osd_backfill_scan_min": "64", 2026-02-23T13:07:28.585 INFO:teuthology.orchestra.run.trial068.stdout: "osd_backoff_on_degraded": "false", 2026-02-23T13:07:28.585 INFO:teuthology.orchestra.run.trial068.stdout: "osd_backoff_on_peering": "false", 2026-02-23T13:07:28.585 INFO:teuthology.orchestra.run.trial068.stdout: "osd_backoff_on_unfound": "true", 2026-02-23T13:07:28.585 INFO:teuthology.orchestra.run.trial068.stdout: "osd_beacon_report_interval": "300", 2026-02-23T13:07:28.585 INFO:teuthology.orchestra.run.trial068.stdout: "osd_bench_duration": "30", 2026-02-23T13:07:28.585 INFO:teuthology.orchestra.run.trial068.stdout: "osd_bench_large_size_max_throughput": "104857600", 2026-02-23T13:07:28.585 INFO:teuthology.orchestra.run.trial068.stdout: "osd_bench_max_block_size": "67108864", 2026-02-23T13:07:28.585 INFO:teuthology.orchestra.run.trial068.stdout: "osd_bench_small_size_max_iops": "100", 2026-02-23T13:07:28.585 INFO:teuthology.orchestra.run.trial068.stdout: "osd_blkin_trace_all": "false", 2026-02-23T13:07:28.585 INFO:teuthology.orchestra.run.trial068.stdout: "osd_blocked_scrub_grace_period": "120", 2026-02-23T13:07:28.585 INFO:teuthology.orchestra.run.trial068.stdout: "osd_calc_pg_upmaps_aggressively": "true", 2026-02-23T13:07:28.585 INFO:teuthology.orchestra.run.trial068.stdout: "osd_calc_pg_upmaps_aggressively_fast": "true", 2026-02-23T13:07:28.585 INFO:teuthology.orchestra.run.trial068.stdout: "osd_calc_pg_upmaps_local_fallback_retries": "100", 2026-02-23T13:07:28.585 INFO:teuthology.orchestra.run.trial068.stdout: "osd_check_for_log_corruption": "false", 2026-02-23T13:07:28.585 INFO:teuthology.orchestra.run.trial068.stdout: "osd_check_max_object_name_len_on_startup": "true", 2026-02-23T13:07:28.585 INFO:teuthology.orchestra.run.trial068.stdout: "osd_class_default_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo sem_set", 2026-02-23T13:07:28.585 INFO:teuthology.orchestra.run.trial068.stdout: "osd_class_dir": "/usr/lib64/rados-classes", 2026-02-23T13:07:28.586 INFO:teuthology.orchestra.run.trial068.stdout: "osd_class_load_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo sem_set", 2026-02-23T13:07:28.586 INFO:teuthology.orchestra.run.trial068.stdout: "osd_class_update_on_start": "true", 2026-02-23T13:07:28.586 INFO:teuthology.orchestra.run.trial068.stdout: "osd_client_message_cap": "256", 2026-02-23T13:07:28.586 INFO:teuthology.orchestra.run.trial068.stdout: "osd_client_message_size_cap": "524288000", 2026-02-23T13:07:28.586 INFO:teuthology.orchestra.run.trial068.stdout: "osd_client_op_priority": "63", 2026-02-23T13:07:28.586 INFO:teuthology.orchestra.run.trial068.stdout: "osd_client_watch_timeout": "30", 2026-02-23T13:07:28.586 INFO:teuthology.orchestra.run.trial068.stdout: "osd_command_max_records": "256", 2026-02-23T13:07:28.586 INFO:teuthology.orchestra.run.trial068.stdout: "osd_compact_on_start": "false", 2026-02-23T13:07:28.586 INFO:teuthology.orchestra.run.trial068.stdout: "osd_copyfrom_max_chunk": "8388608", 2026-02-23T13:07:28.586 INFO:teuthology.orchestra.run.trial068.stdout: "osd_crush_chooseleaf_type": "1", 2026-02-23T13:07:28.586 INFO:teuthology.orchestra.run.trial068.stdout: "osd_crush_initial_weight": "-1.000000", 2026-02-23T13:07:28.586 INFO:teuthology.orchestra.run.trial068.stdout: "osd_crush_update_on_start": "true", 2026-02-23T13:07:28.586 INFO:teuthology.orchestra.run.trial068.stdout: "osd_crush_update_weight_set": "true", 2026-02-23T13:07:28.586 INFO:teuthology.orchestra.run.trial068.stdout: "osd_data": "/var/lib/ceph/osd/ceph-a", 2026-02-23T13:07:28.586 INFO:teuthology.orchestra.run.trial068.stdout: "osd_debug_crash_on_ignored_backoff": "false", 2026-02-23T13:07:28.586 INFO:teuthology.orchestra.run.trial068.stdout: "osd_debug_deep_scrub_sleep": "0.000000", 2026-02-23T13:07:28.586 INFO:teuthology.orchestra.run.trial068.stdout: "osd_debug_drop_ping_duration": "0", 2026-02-23T13:07:28.586 INFO:teuthology.orchestra.run.trial068.stdout: "osd_debug_drop_ping_probability": "0.000000", 2026-02-23T13:07:28.586 INFO:teuthology.orchestra.run.trial068.stdout: "osd_debug_feed_pullee": "-1", 2026-02-23T13:07:28.586 INFO:teuthology.orchestra.run.trial068.stdout: "osd_debug_inject_copyfrom_error": "false", 2026-02-23T13:07:28.586 INFO:teuthology.orchestra.run.trial068.stdout: "osd_debug_inject_dispatch_delay_duration": "0.100000", 2026-02-23T13:07:28.586 INFO:teuthology.orchestra.run.trial068.stdout: "osd_debug_inject_dispatch_delay_probability": "0.000000", 2026-02-23T13:07:28.586 INFO:teuthology.orchestra.run.trial068.stdout: "osd_debug_misdirected_ops": "false", 2026-02-23T13:07:28.586 INFO:teuthology.orchestra.run.trial068.stdout: "osd_debug_no_acting_change": "false", 2026-02-23T13:07:28.586 INFO:teuthology.orchestra.run.trial068.stdout: "osd_debug_no_purge_strays": "false", 2026-02-23T13:07:28.586 INFO:teuthology.orchestra.run.trial068.stdout: "osd_debug_op_order": "false", 2026-02-23T13:07:28.586 INFO:teuthology.orchestra.run.trial068.stdout: "osd_debug_pg_log_writeout": "false", 2026-02-23T13:07:28.586 INFO:teuthology.orchestra.run.trial068.stdout: "osd_debug_pretend_recovery_active": "false", 2026-02-23T13:07:28.586 INFO:teuthology.orchestra.run.trial068.stdout: "osd_debug_random_push_read_error": "0.000000", 2026-02-23T13:07:28.586 INFO:teuthology.orchestra.run.trial068.stdout: "osd_debug_reject_backfill_probability": "0.000000", 2026-02-23T13:07:28.586 INFO:teuthology.orchestra.run.trial068.stdout: "osd_debug_shutdown": "false", 2026-02-23T13:07:28.586 INFO:teuthology.orchestra.run.trial068.stdout: "osd_debug_skip_full_check_in_backfill_reservation": "false", 2026-02-23T13:07:28.586 INFO:teuthology.orchestra.run.trial068.stdout: "osd_debug_skip_full_check_in_recovery": "false", 2026-02-23T13:07:28.586 INFO:teuthology.orchestra.run.trial068.stdout: "osd_debug_trim_objects": "false", 2026-02-23T13:07:28.586 INFO:teuthology.orchestra.run.trial068.stdout: "osd_debug_verify_cached_snaps": "false", 2026-02-23T13:07:28.586 INFO:teuthology.orchestra.run.trial068.stdout: "osd_debug_verify_missing_on_start": "false", 2026-02-23T13:07:28.586 INFO:teuthology.orchestra.run.trial068.stdout: "osd_debug_verify_snaps": "false", 2026-02-23T13:07:28.586 INFO:teuthology.orchestra.run.trial068.stdout: "osd_debug_verify_stray_on_activate": "false", 2026-02-23T13:07:28.586 INFO:teuthology.orchestra.run.trial068.stdout: "osd_deep_scrub_interval": "604800.000000", 2026-02-23T13:07:28.586 INFO:teuthology.orchestra.run.trial068.stdout: "osd_deep_scrub_interval_cv": "0.200000", 2026-02-23T13:07:28.586 INFO:teuthology.orchestra.run.trial068.stdout: "osd_deep_scrub_keys": "1024", 2026-02-23T13:07:28.586 INFO:teuthology.orchestra.run.trial068.stdout: "osd_deep_scrub_large_omap_object_key_threshold": "200000", 2026-02-23T13:07:28.587 INFO:teuthology.orchestra.run.trial068.stdout: "osd_deep_scrub_large_omap_object_value_sum_threshold": "1073741824", 2026-02-23T13:07:28.587 INFO:teuthology.orchestra.run.trial068.stdout: "osd_deep_scrub_randomize_ratio": "0.150000", 2026-02-23T13:07:28.587 INFO:teuthology.orchestra.run.trial068.stdout: "osd_deep_scrub_stride": "4194304", 2026-02-23T13:07:28.587 INFO:teuthology.orchestra.run.trial068.stdout: "osd_deep_scrub_update_digest_min_age": "7200", 2026-02-23T13:07:28.587 INFO:teuthology.orchestra.run.trial068.stdout: "osd_default_data_pool_replay_window": "45", 2026-02-23T13:07:28.587 INFO:teuthology.orchestra.run.trial068.stdout: "osd_default_notify_timeout": "30", 2026-02-23T13:07:28.587 INFO:teuthology.orchestra.run.trial068.stdout: "osd_delete_sleep": "0.000000", 2026-02-23T13:07:28.587 INFO:teuthology.orchestra.run.trial068.stdout: "osd_delete_sleep_hdd": "5.000000", 2026-02-23T13:07:28.587 INFO:teuthology.orchestra.run.trial068.stdout: "osd_delete_sleep_hybrid": "1.000000", 2026-02-23T13:07:28.587 INFO:teuthology.orchestra.run.trial068.stdout: "osd_delete_sleep_ssd": "1.000000", 2026-02-23T13:07:28.587 INFO:teuthology.orchestra.run.trial068.stdout: "osd_discard_disconnected_ops": "true", 2026-02-23T13:07:28.587 INFO:teuthology.orchestra.run.trial068.stdout: "osd_ec_partial_reads": "true", 2026-02-23T13:07:28.587 INFO:teuthology.orchestra.run.trial068.stdout: "osd_enable_op_tracker": "true", 2026-02-23T13:07:28.587 INFO:teuthology.orchestra.run.trial068.stdout: "osd_erasure_code_plugins": "jerasure lrc", 2026-02-23T13:07:28.587 INFO:teuthology.orchestra.run.trial068.stdout: "osd_extblkdev_plugins": "vdo", 2026-02-23T13:07:28.587 INFO:teuthology.orchestra.run.trial068.stdout: "osd_failsafe_full_ratio": "0.970000", 2026-02-23T13:07:28.587 INFO:teuthology.orchestra.run.trial068.stdout: "osd_fast_fail_on_connection_refused": "true", 2026-02-23T13:07:28.587 INFO:teuthology.orchestra.run.trial068.stdout: "osd_fast_info": "true", 2026-02-23T13:07:28.587 INFO:teuthology.orchestra.run.trial068.stdout: "osd_fast_shutdown": "true", 2026-02-23T13:07:28.587 INFO:teuthology.orchestra.run.trial068.stdout: "osd_fast_shutdown_notify_mon": "true", 2026-02-23T13:07:28.587 INFO:teuthology.orchestra.run.trial068.stdout: "osd_fast_shutdown_timeout": "15", 2026-02-23T13:07:28.587 INFO:teuthology.orchestra.run.trial068.stdout: "osd_find_best_info_ignore_history_les": "false", 2026-02-23T13:07:28.587 INFO:teuthology.orchestra.run.trial068.stdout: "osd_force_auth_primary_missing_objects": "100", 2026-02-23T13:07:28.587 INFO:teuthology.orchestra.run.trial068.stdout: "osd_force_recovery_pg_log_entries_factor": "1.300000", 2026-02-23T13:07:28.587 INFO:teuthology.orchestra.run.trial068.stdout: "osd_function_tracing": "false", 2026-02-23T13:07:28.587 INFO:teuthology.orchestra.run.trial068.stdout: "osd_heartbeat_grace": "20", 2026-02-23T13:07:28.587 INFO:teuthology.orchestra.run.trial068.stdout: "osd_heartbeat_interval": "6", 2026-02-23T13:07:28.587 INFO:teuthology.orchestra.run.trial068.stdout: "osd_heartbeat_min_healthy_ratio": "0.330000", 2026-02-23T13:07:28.587 INFO:teuthology.orchestra.run.trial068.stdout: "osd_heartbeat_min_peers": "10", 2026-02-23T13:07:28.587 INFO:teuthology.orchestra.run.trial068.stdout: "osd_heartbeat_min_size": "2000", 2026-02-23T13:07:28.587 INFO:teuthology.orchestra.run.trial068.stdout: "osd_heartbeat_stale": "600", 2026-02-23T13:07:28.587 INFO:teuthology.orchestra.run.trial068.stdout: "osd_heartbeat_use_min_delay_socket": "false", 2026-02-23T13:07:28.587 INFO:teuthology.orchestra.run.trial068.stdout: "osd_hit_set_max_size": "100000", 2026-02-23T13:07:28.587 INFO:teuthology.orchestra.run.trial068.stdout: "osd_hit_set_min_size": "1000", 2026-02-23T13:07:28.587 INFO:teuthology.orchestra.run.trial068.stdout: "osd_hit_set_namespace": ".ceph-internal", 2026-02-23T13:07:28.587 INFO:teuthology.orchestra.run.trial068.stdout: "osd_ignore_stale_divergent_priors": "false", 2026-02-23T13:07:28.587 INFO:teuthology.orchestra.run.trial068.stdout: "osd_inject_bad_map_crc_probability": "0.000000", 2026-02-23T13:07:28.587 INFO:teuthology.orchestra.run.trial068.stdout: "osd_inject_failure_on_pg_removal": "false", 2026-02-23T13:07:28.587 INFO:teuthology.orchestra.run.trial068.stdout: "osd_journal": "/var/lib/ceph/osd/ceph-a/journal", 2026-02-23T13:07:28.587 INFO:teuthology.orchestra.run.trial068.stdout: "osd_journal_flush_on_shutdown": "true", 2026-02-23T13:07:28.587 INFO:teuthology.orchestra.run.trial068.stdout: "osd_journal_size": "5120", 2026-02-23T13:07:28.587 INFO:teuthology.orchestra.run.trial068.stdout: "osd_kill_backfill_at": "0", 2026-02-23T13:07:28.587 INFO:teuthology.orchestra.run.trial068.stdout: "osd_loop_before_reset_tphandle": "64", 2026-02-23T13:07:28.587 INFO:teuthology.orchestra.run.trial068.stdout: "osd_map_cache_size": "50", 2026-02-23T13:07:28.587 INFO:teuthology.orchestra.run.trial068.stdout: "osd_map_dedup": "true", 2026-02-23T13:07:28.588 INFO:teuthology.orchestra.run.trial068.stdout: "osd_map_message_max": "40", 2026-02-23T13:07:28.588 INFO:teuthology.orchestra.run.trial068.stdout: "osd_map_message_max_bytes": "10485760", 2026-02-23T13:07:28.588 INFO:teuthology.orchestra.run.trial068.stdout: "osd_map_share_max_epochs": "40", 2026-02-23T13:07:28.588 INFO:teuthology.orchestra.run.trial068.stdout: "osd_max_attr_name_len": "100", 2026-02-23T13:07:28.588 INFO:teuthology.orchestra.run.trial068.stdout: "osd_max_attr_size": "0", 2026-02-23T13:07:28.588 INFO:teuthology.orchestra.run.trial068.stdout: "osd_max_backfills": "1", 2026-02-23T13:07:28.588 INFO:teuthology.orchestra.run.trial068.stdout: "osd_max_markdown_count": "5", 2026-02-23T13:07:28.588 INFO:teuthology.orchestra.run.trial068.stdout: "osd_max_markdown_period": "600", 2026-02-23T13:07:28.588 INFO:teuthology.orchestra.run.trial068.stdout: "osd_max_object_name_len": "2048", 2026-02-23T13:07:28.588 INFO:teuthology.orchestra.run.trial068.stdout: "osd_max_object_namespace_len": "256", 2026-02-23T13:07:28.588 INFO:teuthology.orchestra.run.trial068.stdout: "osd_max_object_size": "134217728", 2026-02-23T13:07:28.588 INFO:teuthology.orchestra.run.trial068.stdout: "osd_max_omap_bytes_per_request": "1073741824", 2026-02-23T13:07:28.588 INFO:teuthology.orchestra.run.trial068.stdout: "osd_max_omap_entries_per_request": "1024", 2026-02-23T13:07:28.588 INFO:teuthology.orchestra.run.trial068.stdout: "osd_max_pg_blocked_by": "16", 2026-02-23T13:07:28.588 INFO:teuthology.orchestra.run.trial068.stdout: "osd_max_pg_log_entries": "10000", 2026-02-23T13:07:28.588 INFO:teuthology.orchestra.run.trial068.stdout: "osd_max_pg_per_osd_hard_ratio": "3.000000", 2026-02-23T13:07:28.588 INFO:teuthology.orchestra.run.trial068.stdout: "osd_max_pgls": "1024", 2026-02-23T13:07:28.588 INFO:teuthology.orchestra.run.trial068.stdout: "osd_max_push_cost": "8388608", 2026-02-23T13:07:28.588 INFO:teuthology.orchestra.run.trial068.stdout: "osd_max_push_objects": "10", 2026-02-23T13:07:28.588 INFO:teuthology.orchestra.run.trial068.stdout: "osd_max_scrubs": "3", 2026-02-23T13:07:28.588 INFO:teuthology.orchestra.run.trial068.stdout: "osd_max_snap_prune_intervals_per_epoch": "512", 2026-02-23T13:07:28.588 INFO:teuthology.orchestra.run.trial068.stdout: "osd_max_trimming_pgs": "2", 2026-02-23T13:07:28.588 INFO:teuthology.orchestra.run.trial068.stdout: "osd_max_write_op_reply_len": "64", 2026-02-23T13:07:28.588 INFO:teuthology.orchestra.run.trial068.stdout: "osd_max_write_size": "90", 2026-02-23T13:07:28.588 INFO:teuthology.orchestra.run.trial068.stdout: "osd_mclock_force_run_benchmark_on_init": "false", 2026-02-23T13:07:28.588 INFO:teuthology.orchestra.run.trial068.stdout: "osd_mclock_iops_capacity_low_threshold_hdd": "50.000000", 2026-02-23T13:07:28.588 INFO:teuthology.orchestra.run.trial068.stdout: "osd_mclock_iops_capacity_low_threshold_ssd": "1000.000000", 2026-02-23T13:07:28.588 INFO:teuthology.orchestra.run.trial068.stdout: "osd_mclock_iops_capacity_threshold_hdd": "500.000000", 2026-02-23T13:07:28.588 INFO:teuthology.orchestra.run.trial068.stdout: "osd_mclock_iops_capacity_threshold_ssd": "80000.000000", 2026-02-23T13:07:28.588 INFO:teuthology.orchestra.run.trial068.stdout: "osd_mclock_max_capacity_iops_hdd": "315.000000", 2026-02-23T13:07:28.588 INFO:teuthology.orchestra.run.trial068.stdout: "osd_mclock_max_capacity_iops_ssd": "21500.000000", 2026-02-23T13:07:28.588 INFO:teuthology.orchestra.run.trial068.stdout: "osd_mclock_max_sequential_bandwidth_hdd": "157286400", 2026-02-23T13:07:28.588 INFO:teuthology.orchestra.run.trial068.stdout: "osd_mclock_max_sequential_bandwidth_ssd": "1258291200", 2026-02-23T13:07:28.588 INFO:teuthology.orchestra.run.trial068.stdout: "osd_mclock_override_recovery_settings": "false", 2026-02-23T13:07:28.588 INFO:teuthology.orchestra.run.trial068.stdout: "osd_mclock_profile": "balanced", 2026-02-23T13:07:28.588 INFO:teuthology.orchestra.run.trial068.stdout: "osd_mclock_scheduler_anticipation_timeout": "0.000000", 2026-02-23T13:07:28.588 INFO:teuthology.orchestra.run.trial068.stdout: "osd_mclock_scheduler_background_best_effort_lim": "0.000000", 2026-02-23T13:07:28.588 INFO:teuthology.orchestra.run.trial068.stdout: "osd_mclock_scheduler_background_best_effort_res": "0.000000", 2026-02-23T13:07:28.588 INFO:teuthology.orchestra.run.trial068.stdout: "osd_mclock_scheduler_background_best_effort_wgt": "1", 2026-02-23T13:07:28.588 INFO:teuthology.orchestra.run.trial068.stdout: "osd_mclock_scheduler_background_recovery_lim": "0.000000", 2026-02-23T13:07:28.588 INFO:teuthology.orchestra.run.trial068.stdout: "osd_mclock_scheduler_background_recovery_res": "0.000000", 2026-02-23T13:07:28.588 INFO:teuthology.orchestra.run.trial068.stdout: "osd_mclock_scheduler_background_recovery_wgt": "1", 2026-02-23T13:07:28.588 INFO:teuthology.orchestra.run.trial068.stdout: "osd_mclock_scheduler_client_lim": "0.000000", 2026-02-23T13:07:28.588 INFO:teuthology.orchestra.run.trial068.stdout: "osd_mclock_scheduler_client_res": "0.000000", 2026-02-23T13:07:28.589 INFO:teuthology.orchestra.run.trial068.stdout: "osd_mclock_scheduler_client_wgt": "1", 2026-02-23T13:07:28.589 INFO:teuthology.orchestra.run.trial068.stdout: "osd_mclock_skip_benchmark": "false", 2026-02-23T13:07:28.589 INFO:teuthology.orchestra.run.trial068.stdout: "osd_memory_base": "805306368", 2026-02-23T13:07:28.589 INFO:teuthology.orchestra.run.trial068.stdout: "osd_memory_cache_min": "134217728", 2026-02-23T13:07:28.589 INFO:teuthology.orchestra.run.trial068.stdout: "osd_memory_cache_resize_interval": "1.000000", 2026-02-23T13:07:28.589 INFO:teuthology.orchestra.run.trial068.stdout: "osd_memory_expected_fragmentation": "0.150000", 2026-02-23T13:07:28.589 INFO:teuthology.orchestra.run.trial068.stdout: "osd_memory_target": "4294967296", 2026-02-23T13:07:28.589 INFO:teuthology.orchestra.run.trial068.stdout: "osd_memory_target_autotune": "false", 2026-02-23T13:07:28.589 INFO:teuthology.orchestra.run.trial068.stdout: "osd_memory_target_cgroup_limit_ratio": "0.800000", 2026-02-23T13:07:28.589 INFO:teuthology.orchestra.run.trial068.stdout: "osd_min_pg_log_entries": "250", 2026-02-23T13:07:28.589 INFO:teuthology.orchestra.run.trial068.stdout: "osd_min_recovery_priority": "0", 2026-02-23T13:07:28.589 INFO:teuthology.orchestra.run.trial068.stdout: "osd_mon_heartbeat_interval": "30", 2026-02-23T13:07:28.589 INFO:teuthology.orchestra.run.trial068.stdout: "osd_mon_heartbeat_stat_stale": "3600", 2026-02-23T13:07:28.589 INFO:teuthology.orchestra.run.trial068.stdout: "osd_mon_report_interval": "5", 2026-02-23T13:07:28.589 INFO:teuthology.orchestra.run.trial068.stdout: "osd_mon_report_max_in_flight": "2", 2026-02-23T13:07:28.589 INFO:teuthology.orchestra.run.trial068.stdout: "osd_mon_shutdown_timeout": "5.000000", 2026-02-23T13:07:28.589 INFO:teuthology.orchestra.run.trial068.stdout: "osd_num_cache_shards": "32", 2026-02-23T13:07:28.589 INFO:teuthology.orchestra.run.trial068.stdout: "osd_num_op_tracker_shard": "32", 2026-02-23T13:07:28.589 INFO:teuthology.orchestra.run.trial068.stdout: "osd_numa_auto_affinity": "true", 2026-02-23T13:07:28.589 INFO:teuthology.orchestra.run.trial068.stdout: "osd_numa_node": "-1", 2026-02-23T13:07:28.589 INFO:teuthology.orchestra.run.trial068.stdout: "osd_numa_prefer_iface": "true", 2026-02-23T13:07:28.589 INFO:teuthology.orchestra.run.trial068.stdout: "osd_object_clean_region_max_num_intervals": "10", 2026-02-23T13:07:28.589 INFO:teuthology.orchestra.run.trial068.stdout: "osd_objecter_finishers": "1", 2026-02-23T13:07:28.589 INFO:teuthology.orchestra.run.trial068.stdout: "osd_objectstore": "bluestore", 2026-02-23T13:07:28.589 INFO:teuthology.orchestra.run.trial068.stdout: "osd_objectstore_fuse": "false", 2026-02-23T13:07:28.589 INFO:teuthology.orchestra.run.trial068.stdout: "osd_objectstore_ideal_list_max": "64", 2026-02-23T13:07:28.589 INFO:teuthology.orchestra.run.trial068.stdout: "osd_objectstore_tracing": "false", 2026-02-23T13:07:28.589 INFO:teuthology.orchestra.run.trial068.stdout: "osd_op_complaint_time": "30.000000", 2026-02-23T13:07:28.589 INFO:teuthology.orchestra.run.trial068.stdout: "osd_op_history_duration": "600", 2026-02-23T13:07:28.589 INFO:teuthology.orchestra.run.trial068.stdout: "osd_op_history_size": "20", 2026-02-23T13:07:28.589 INFO:teuthology.orchestra.run.trial068.stdout: "osd_op_history_slow_op_size": "20", 2026-02-23T13:07:28.589 INFO:teuthology.orchestra.run.trial068.stdout: "osd_op_history_slow_op_threshold": "10.000000", 2026-02-23T13:07:28.589 INFO:teuthology.orchestra.run.trial068.stdout: "osd_op_log_threshold": "5", 2026-02-23T13:07:28.589 INFO:teuthology.orchestra.run.trial068.stdout: "osd_op_num_shards": "0", 2026-02-23T13:07:28.589 INFO:teuthology.orchestra.run.trial068.stdout: "osd_op_num_shards_hdd": "1", 2026-02-23T13:07:28.589 INFO:teuthology.orchestra.run.trial068.stdout: "osd_op_num_shards_ssd": "8", 2026-02-23T13:07:28.589 INFO:teuthology.orchestra.run.trial068.stdout: "osd_op_num_threads_per_shard": "0", 2026-02-23T13:07:28.589 INFO:teuthology.orchestra.run.trial068.stdout: "osd_op_num_threads_per_shard_hdd": "5", 2026-02-23T13:07:28.589 INFO:teuthology.orchestra.run.trial068.stdout: "osd_op_num_threads_per_shard_ssd": "2", 2026-02-23T13:07:28.589 INFO:teuthology.orchestra.run.trial068.stdout: "osd_op_pq_max_tokens_per_priority": "4194304", 2026-02-23T13:07:28.589 INFO:teuthology.orchestra.run.trial068.stdout: "osd_op_pq_min_cost": "65536", 2026-02-23T13:07:28.589 INFO:teuthology.orchestra.run.trial068.stdout: "osd_op_queue": "mclock_scheduler", 2026-02-23T13:07:28.590 INFO:teuthology.orchestra.run.trial068.stdout: "osd_op_queue_cut_off": "high", 2026-02-23T13:07:28.590 INFO:teuthology.orchestra.run.trial068.stdout: "osd_op_thread_suicide_timeout": "150", 2026-02-23T13:07:28.590 INFO:teuthology.orchestra.run.trial068.stdout: "osd_op_thread_timeout": "15", 2026-02-23T13:07:28.590 INFO:teuthology.orchestra.run.trial068.stdout: "osd_open_classes_on_start": "true", 2026-02-23T13:07:28.590 INFO:teuthology.orchestra.run.trial068.stdout: "osd_os_flags": "0", 2026-02-23T13:07:28.590 INFO:teuthology.orchestra.run.trial068.stdout: "osd_peering_op_priority": "255", 2026-02-23T13:07:28.590 INFO:teuthology.orchestra.run.trial068.stdout: "osd_pg_delete_cost": "1048576", 2026-02-23T13:07:28.590 INFO:teuthology.orchestra.run.trial068.stdout: "osd_pg_delete_priority": "5", 2026-02-23T13:07:28.590 INFO:teuthology.orchestra.run.trial068.stdout: "osd_pg_epoch_max_lag_factor": "2.000000", 2026-02-23T13:07:28.590 INFO:teuthology.orchestra.run.trial068.stdout: "osd_pg_epoch_persisted_max_stale": "40", 2026-02-23T13:07:28.590 INFO:teuthology.orchestra.run.trial068.stdout: "osd_pg_log_dups_tracked": "3000", 2026-02-23T13:07:28.590 INFO:teuthology.orchestra.run.trial068.stdout: "osd_pg_log_trim_max": "10000", 2026-02-23T13:07:28.590 INFO:teuthology.orchestra.run.trial068.stdout: "osd_pg_log_trim_min": "100", 2026-02-23T13:07:28.590 INFO:teuthology.orchestra.run.trial068.stdout: "osd_pg_max_concurrent_snap_trims": "2", 2026-02-23T13:07:28.590 INFO:teuthology.orchestra.run.trial068.stdout: "osd_pg_object_context_cache_count": "64", 2026-02-23T13:07:28.590 INFO:teuthology.orchestra.run.trial068.stdout: "osd_pg_stat_report_interval_max_epochs": "500", 2026-02-23T13:07:28.590 INFO:teuthology.orchestra.run.trial068.stdout: "osd_pg_stat_report_interval_max_seconds": "5", 2026-02-23T13:07:28.590 INFO:teuthology.orchestra.run.trial068.stdout: "osd_pool_default_cache_max_evict_check_size": "10", 2026-02-23T13:07:28.590 INFO:teuthology.orchestra.run.trial068.stdout: "osd_pool_default_cache_min_evict_age": "0", 2026-02-23T13:07:28.590 INFO:teuthology.orchestra.run.trial068.stdout: "osd_pool_default_cache_min_flush_age": "0", 2026-02-23T13:07:28.590 INFO:teuthology.orchestra.run.trial068.stdout: "osd_pool_default_cache_target_dirty_high_ratio": "0.600000", 2026-02-23T13:07:28.590 INFO:teuthology.orchestra.run.trial068.stdout: "osd_pool_default_cache_target_dirty_ratio": "0.400000", 2026-02-23T13:07:28.590 INFO:teuthology.orchestra.run.trial068.stdout: "osd_pool_default_cache_target_full_ratio": "0.800000", 2026-02-23T13:07:28.590 INFO:teuthology.orchestra.run.trial068.stdout: "osd_pool_default_crimson": "true", 2026-02-23T13:07:28.590 INFO:teuthology.orchestra.run.trial068.stdout: "osd_pool_default_crush_rule": "-1", 2026-02-23T13:07:28.590 INFO:teuthology.orchestra.run.trial068.stdout: "osd_pool_default_ec_fast_read": "false", 2026-02-23T13:07:28.590 INFO:teuthology.orchestra.run.trial068.stdout: "osd_pool_default_erasure_code_profile": "plugin=isa technique=reed_sol_van k=2 m=1 crush-failure-domain=osd", 2026-02-23T13:07:28.590 INFO:teuthology.orchestra.run.trial068.stdout: "osd_pool_default_flag_bulk": "false", 2026-02-23T13:07:28.590 INFO:teuthology.orchestra.run.trial068.stdout: "osd_pool_default_flag_ec_optimizations": "false", 2026-02-23T13:07:28.590 INFO:teuthology.orchestra.run.trial068.stdout: "osd_pool_default_flag_hashpspool": "true", 2026-02-23T13:07:28.590 INFO:teuthology.orchestra.run.trial068.stdout: "osd_pool_default_flag_nodelete": "false", 2026-02-23T13:07:28.590 INFO:teuthology.orchestra.run.trial068.stdout: "osd_pool_default_flag_nopgchange": "false", 2026-02-23T13:07:28.590 INFO:teuthology.orchestra.run.trial068.stdout: "osd_pool_default_flag_nosizechange": "false", 2026-02-23T13:07:28.590 INFO:teuthology.orchestra.run.trial068.stdout: "osd_pool_default_flags": "0", 2026-02-23T13:07:28.590 INFO:teuthology.orchestra.run.trial068.stdout: "osd_pool_default_hit_set_bloom_fpp": "0.050000", 2026-02-23T13:07:28.590 INFO:teuthology.orchestra.run.trial068.stdout: "osd_pool_default_min_size": "2", 2026-02-23T13:07:28.590 INFO:teuthology.orchestra.run.trial068.stdout: "osd_pool_default_pg_autoscale_mode": "off", 2026-02-23T13:07:28.590 INFO:teuthology.orchestra.run.trial068.stdout: "osd_pool_default_pg_num": "32", 2026-02-23T13:07:28.590 INFO:teuthology.orchestra.run.trial068.stdout: "osd_pool_default_pgp_num": "0", 2026-02-23T13:07:28.590 INFO:teuthology.orchestra.run.trial068.stdout: "osd_pool_default_read_lease_ratio": "0.800000", 2026-02-23T13:07:28.590 INFO:teuthology.orchestra.run.trial068.stdout: "osd_pool_default_read_ratio": "70", 2026-02-23T13:07:28.590 INFO:teuthology.orchestra.run.trial068.stdout: "osd_pool_default_size": "3", 2026-02-23T13:07:28.590 INFO:teuthology.orchestra.run.trial068.stdout: "osd_pool_default_type": "replicated", 2026-02-23T13:07:28.590 INFO:teuthology.orchestra.run.trial068.stdout: "osd_pool_erasure_code_stripe_unit": "0", 2026-02-23T13:07:28.591 INFO:teuthology.orchestra.run.trial068.stdout: "osd_pool_use_gmt_hitset": "true", 2026-02-23T13:07:28.591 INFO:teuthology.orchestra.run.trial068.stdout: "osd_push_per_object_cost": "1000", 2026-02-23T13:07:28.591 INFO:teuthology.orchestra.run.trial068.stdout: "osd_read_ec_check_for_errors": "false", 2026-02-23T13:07:28.591 INFO:teuthology.orchestra.run.trial068.stdout: "osd_recover_clone_overlap": "true", 2026-02-23T13:07:28.591 INFO:teuthology.orchestra.run.trial068.stdout: "osd_recover_clone_overlap_limit": "10", 2026-02-23T13:07:28.591 INFO:teuthology.orchestra.run.trial068.stdout: "osd_recovery_cost": "20971520", 2026-02-23T13:07:28.591 INFO:teuthology.orchestra.run.trial068.stdout: "osd_recovery_delay_start": "0.000000", 2026-02-23T13:07:28.591 INFO:teuthology.orchestra.run.trial068.stdout: "osd_recovery_max_active": "0", 2026-02-23T13:07:28.591 INFO:teuthology.orchestra.run.trial068.stdout: "osd_recovery_max_active_hdd": "3", 2026-02-23T13:07:28.591 INFO:teuthology.orchestra.run.trial068.stdout: "osd_recovery_max_active_ssd": "10", 2026-02-23T13:07:28.591 INFO:teuthology.orchestra.run.trial068.stdout: "osd_recovery_max_chunk": "8388608", 2026-02-23T13:07:28.591 INFO:teuthology.orchestra.run.trial068.stdout: "osd_recovery_max_omap_entries_per_chunk": "8096", 2026-02-23T13:07:28.591 INFO:teuthology.orchestra.run.trial068.stdout: "osd_recovery_max_single_start": "1", 2026-02-23T13:07:28.591 INFO:teuthology.orchestra.run.trial068.stdout: "osd_recovery_op_priority": "3", 2026-02-23T13:07:28.591 INFO:teuthology.orchestra.run.trial068.stdout: "osd_recovery_op_warn_multiple": "16", 2026-02-23T13:07:28.591 INFO:teuthology.orchestra.run.trial068.stdout: "osd_recovery_priority": "5", 2026-02-23T13:07:28.591 INFO:teuthology.orchestra.run.trial068.stdout: "osd_recovery_retry_interval": "30.000000", 2026-02-23T13:07:28.591 INFO:teuthology.orchestra.run.trial068.stdout: "osd_recovery_sleep": "0.000000", 2026-02-23T13:07:28.591 INFO:teuthology.orchestra.run.trial068.stdout: "osd_recovery_sleep_degraded": "0.000000", 2026-02-23T13:07:28.591 INFO:teuthology.orchestra.run.trial068.stdout: "osd_recovery_sleep_degraded_hdd": "0.100000", 2026-02-23T13:07:28.591 INFO:teuthology.orchestra.run.trial068.stdout: "osd_recovery_sleep_degraded_hybrid": "0.025000", 2026-02-23T13:07:28.591 INFO:teuthology.orchestra.run.trial068.stdout: "osd_recovery_sleep_degraded_ssd": "0.000000", 2026-02-23T13:07:28.591 INFO:teuthology.orchestra.run.trial068.stdout: "osd_recovery_sleep_hdd": "0.100000", 2026-02-23T13:07:28.591 INFO:teuthology.orchestra.run.trial068.stdout: "osd_recovery_sleep_hybrid": "0.025000", 2026-02-23T13:07:28.591 INFO:teuthology.orchestra.run.trial068.stdout: "osd_recovery_sleep_ssd": "0.000000", 2026-02-23T13:07:28.591 INFO:teuthology.orchestra.run.trial068.stdout: "osd_requested_scrub_priority": "5", 2026-02-23T13:07:28.591 INFO:teuthology.orchestra.run.trial068.stdout: "osd_rocksdb_iterator_bounds_enabled": "true", 2026-02-23T13:07:28.591 INFO:teuthology.orchestra.run.trial068.stdout: "osd_rollback_to_cluster_snap": "", 2026-02-23T13:07:28.591 INFO:teuthology.orchestra.run.trial068.stdout: "osd_scrub_auto_repair": "false", 2026-02-23T13:07:28.591 INFO:teuthology.orchestra.run.trial068.stdout: "osd_scrub_auto_repair_num_errors": "5", 2026-02-23T13:07:28.591 INFO:teuthology.orchestra.run.trial068.stdout: "osd_scrub_backoff_ratio": "0.660000", 2026-02-23T13:07:28.591 INFO:teuthology.orchestra.run.trial068.stdout: "osd_scrub_begin_hour": "0", 2026-02-23T13:07:28.591 INFO:teuthology.orchestra.run.trial068.stdout: "osd_scrub_begin_week_day": "0", 2026-02-23T13:07:28.591 INFO:teuthology.orchestra.run.trial068.stdout: "osd_scrub_chunk_max": "15", 2026-02-23T13:07:28.591 INFO:teuthology.orchestra.run.trial068.stdout: "osd_scrub_chunk_min": "5", 2026-02-23T13:07:28.591 INFO:teuthology.orchestra.run.trial068.stdout: "osd_scrub_cost": "52428800", 2026-02-23T13:07:28.591 INFO:teuthology.orchestra.run.trial068.stdout: "osd_scrub_disable_reservation_queuing": "false", 2026-02-23T13:07:28.591 INFO:teuthology.orchestra.run.trial068.stdout: "osd_scrub_during_recovery": "false", 2026-02-23T13:07:28.591 INFO:teuthology.orchestra.run.trial068.stdout: "osd_scrub_end_hour": "0", 2026-02-23T13:07:28.591 INFO:teuthology.orchestra.run.trial068.stdout: "osd_scrub_end_week_day": "0", 2026-02-23T13:07:28.591 INFO:teuthology.orchestra.run.trial068.stdout: "osd_scrub_event_cost": "4096", 2026-02-23T13:07:28.591 INFO:teuthology.orchestra.run.trial068.stdout: "osd_scrub_extended_sleep": "0.000000", 2026-02-23T13:07:28.591 INFO:teuthology.orchestra.run.trial068.stdout: "osd_scrub_interval_randomize_ratio": "0.500000", 2026-02-23T13:07:28.591 INFO:teuthology.orchestra.run.trial068.stdout: "osd_scrub_invalid_stats": "true", 2026-02-23T13:07:28.592 INFO:teuthology.orchestra.run.trial068.stdout: "osd_scrub_load_threshold": "10.000000", 2026-02-23T13:07:28.592 INFO:teuthology.orchestra.run.trial068.stdout: "osd_scrub_max_interval": "604800.000000", 2026-02-23T13:07:28.592 INFO:teuthology.orchestra.run.trial068.stdout: "osd_scrub_max_preemptions": "5", 2026-02-23T13:07:28.592 INFO:teuthology.orchestra.run.trial068.stdout: "osd_scrub_min_interval": "86400.000000", 2026-02-23T13:07:28.592 INFO:teuthology.orchestra.run.trial068.stdout: "osd_scrub_priority": "5", 2026-02-23T13:07:28.592 INFO:teuthology.orchestra.run.trial068.stdout: "osd_scrub_retry_after_noscrub": "60", 2026-02-23T13:07:28.592 INFO:teuthology.orchestra.run.trial068.stdout: "osd_scrub_retry_delay": "30", 2026-02-23T13:07:28.592 INFO:teuthology.orchestra.run.trial068.stdout: "osd_scrub_retry_new_interval": "10", 2026-02-23T13:07:28.592 INFO:teuthology.orchestra.run.trial068.stdout: "osd_scrub_retry_pg_state": "60", 2026-02-23T13:07:28.592 INFO:teuthology.orchestra.run.trial068.stdout: "osd_scrub_retry_trimming": "10", 2026-02-23T13:07:28.592 INFO:teuthology.orchestra.run.trial068.stdout: "osd_scrub_sleep": "0.000000", 2026-02-23T13:07:28.592 INFO:teuthology.orchestra.run.trial068.stdout: "osd_shallow_scrub_chunk_max": "100", 2026-02-23T13:07:28.592 INFO:teuthology.orchestra.run.trial068.stdout: "osd_shallow_scrub_chunk_min": "50", 2026-02-23T13:07:28.592 INFO:teuthology.orchestra.run.trial068.stdout: "osd_shutdown_pgref_assert": "false", 2026-02-23T13:07:28.592 INFO:teuthology.orchestra.run.trial068.stdout: "osd_skip_check_past_interval_bounds": "false", 2026-02-23T13:07:28.592 INFO:teuthology.orchestra.run.trial068.stdout: "osd_skip_data_digest": "false", 2026-02-23T13:07:28.592 INFO:teuthology.orchestra.run.trial068.stdout: "osd_smart_report_timeout": "5", 2026-02-23T13:07:28.592 INFO:teuthology.orchestra.run.trial068.stdout: "osd_snap_trim_cost": "1048576", 2026-02-23T13:07:28.592 INFO:teuthology.orchestra.run.trial068.stdout: "osd_snap_trim_priority": "5", 2026-02-23T13:07:28.592 INFO:teuthology.orchestra.run.trial068.stdout: "osd_snap_trim_sleep": "0.000000", 2026-02-23T13:07:28.592 INFO:teuthology.orchestra.run.trial068.stdout: "osd_snap_trim_sleep_hdd": "5.000000", 2026-02-23T13:07:28.592 INFO:teuthology.orchestra.run.trial068.stdout: "osd_snap_trim_sleep_hybrid": "2.000000", 2026-02-23T13:07:28.592 INFO:teuthology.orchestra.run.trial068.stdout: "osd_snap_trim_sleep_ssd": "0.000000", 2026-02-23T13:07:28.592 INFO:teuthology.orchestra.run.trial068.stdout: "osd_stats_update_period_not_scrubbing": "120", 2026-02-23T13:07:28.592 INFO:teuthology.orchestra.run.trial068.stdout: "osd_stats_update_period_scrubbing": "15", 2026-02-23T13:07:28.592 INFO:teuthology.orchestra.run.trial068.stdout: "osd_target_pg_log_entries_per_osd": "300000", 2026-02-23T13:07:28.592 INFO:teuthology.orchestra.run.trial068.stdout: "osd_target_transaction_size": "30", 2026-02-23T13:07:28.592 INFO:teuthology.orchestra.run.trial068.stdout: "osd_tier_default_cache_hit_set_count": "4", 2026-02-23T13:07:28.592 INFO:teuthology.orchestra.run.trial068.stdout: "osd_tier_default_cache_hit_set_grade_decay_rate": "20", 2026-02-23T13:07:28.592 INFO:teuthology.orchestra.run.trial068.stdout: "osd_tier_default_cache_hit_set_period": "1200", 2026-02-23T13:07:28.592 INFO:teuthology.orchestra.run.trial068.stdout: "osd_tier_default_cache_hit_set_search_last_n": "1", 2026-02-23T13:07:28.592 INFO:teuthology.orchestra.run.trial068.stdout: "osd_tier_default_cache_hit_set_type": "bloom", 2026-02-23T13:07:28.592 INFO:teuthology.orchestra.run.trial068.stdout: "osd_tier_default_cache_min_read_recency_for_promote": "1", 2026-02-23T13:07:28.592 INFO:teuthology.orchestra.run.trial068.stdout: "osd_tier_default_cache_min_write_recency_for_promote": "1", 2026-02-23T13:07:28.592 INFO:teuthology.orchestra.run.trial068.stdout: "osd_tier_default_cache_mode": "writeback", 2026-02-23T13:07:28.592 INFO:teuthology.orchestra.run.trial068.stdout: "osd_tier_promote_max_bytes_sec": "5242880", 2026-02-23T13:07:28.592 INFO:teuthology.orchestra.run.trial068.stdout: "osd_tier_promote_max_objects_sec": "25", 2026-02-23T13:07:28.592 INFO:teuthology.orchestra.run.trial068.stdout: "osd_tracing": "false", 2026-02-23T13:07:28.592 INFO:teuthology.orchestra.run.trial068.stdout: "osd_use_stale_snap": "false", 2026-02-23T13:07:28.592 INFO:teuthology.orchestra.run.trial068.stdout: "osd_uuid": "00000000-0000-0000-0000-000000000000", 2026-02-23T13:07:28.592 INFO:teuthology.orchestra.run.trial068.stdout: "osdc_blkin_trace_all": "false", 2026-02-23T13:07:28.592 INFO:teuthology.orchestra.run.trial068.stdout: "paxos_kill_at": "0", 2026-02-23T13:07:28.592 INFO:teuthology.orchestra.run.trial068.stdout: "paxos_max_join_drift": "10", 2026-02-23T13:07:28.592 INFO:teuthology.orchestra.run.trial068.stdout: "paxos_min": "500", 2026-02-23T13:07:28.593 INFO:teuthology.orchestra.run.trial068.stdout: "paxos_min_wait": "0.050000", 2026-02-23T13:07:28.593 INFO:teuthology.orchestra.run.trial068.stdout: "paxos_propose_interval": "1.000000", 2026-02-23T13:07:28.593 INFO:teuthology.orchestra.run.trial068.stdout: "paxos_service_trim_max": "500", 2026-02-23T13:07:28.593 INFO:teuthology.orchestra.run.trial068.stdout: "paxos_service_trim_max_multiplier": "20", 2026-02-23T13:07:28.593 INFO:teuthology.orchestra.run.trial068.stdout: "paxos_service_trim_min": "10", 2026-02-23T13:07:28.593 INFO:teuthology.orchestra.run.trial068.stdout: "paxos_stash_full_interval": "25", 2026-02-23T13:07:28.593 INFO:teuthology.orchestra.run.trial068.stdout: "paxos_trim_max": "500", 2026-02-23T13:07:28.593 INFO:teuthology.orchestra.run.trial068.stdout: "paxos_trim_min": "250", 2026-02-23T13:07:28.593 INFO:teuthology.orchestra.run.trial068.stdout: "perf": "true", 2026-02-23T13:07:28.593 INFO:teuthology.orchestra.run.trial068.stdout: "pid_file": "", 2026-02-23T13:07:28.593 INFO:teuthology.orchestra.run.trial068.stdout: "plugin_crypto_accelerator": "crypto_isal", 2026-02-23T13:07:28.593 INFO:teuthology.orchestra.run.trial068.stdout: "plugin_dir": "/usr/lib64/ceph", 2026-02-23T13:07:28.593 INFO:teuthology.orchestra.run.trial068.stdout: "pool_availability_update_interval": "1.000000", 2026-02-23T13:07:28.593 INFO:teuthology.orchestra.run.trial068.stdout: "public_addr": "-", 2026-02-23T13:07:28.593 INFO:teuthology.orchestra.run.trial068.stdout: "public_addrv": "", 2026-02-23T13:07:28.593 INFO:teuthology.orchestra.run.trial068.stdout: "public_bind_addr": "-", 2026-02-23T13:07:28.593 INFO:teuthology.orchestra.run.trial068.stdout: "public_network": "", 2026-02-23T13:07:28.593 INFO:teuthology.orchestra.run.trial068.stdout: "public_network_interface": "", 2026-02-23T13:07:28.593 INFO:teuthology.orchestra.run.trial068.stdout: "qat_compressor_busy_polling": "false", 2026-02-23T13:07:28.593 INFO:teuthology.orchestra.run.trial068.stdout: "qat_compressor_enabled": "false", 2026-02-23T13:07:28.593 INFO:teuthology.orchestra.run.trial068.stdout: "qat_compressor_session_max_number": "256", 2026-02-23T13:07:28.594 INFO:teuthology.orchestra.run.trial068.stdout: "rados_mon_op_timeout": "0", 2026-02-23T13:07:28.594 INFO:teuthology.orchestra.run.trial068.stdout: "rados_osd_op_timeout": "0", 2026-02-23T13:07:28.594 INFO:teuthology.orchestra.run.trial068.stdout: "rados_replica_read_policy": "default", 2026-02-23T13:07:28.594 INFO:teuthology.orchestra.run.trial068.stdout: "rados_replica_read_policy_on_objclass": "false", 2026-02-23T13:07:28.594 INFO:teuthology.orchestra.run.trial068.stdout: "rados_tracing": "false", 2026-02-23T13:07:28.594 INFO:teuthology.orchestra.run.trial068.stdout: "rbd_atime_update_interval": "60", 2026-02-23T13:07:28.594 INFO:teuthology.orchestra.run.trial068.stdout: "rbd_auto_exclusive_lock_until_manual_request": "true", 2026-02-23T13:07:28.594 INFO:teuthology.orchestra.run.trial068.stdout: "rbd_balance_parent_reads": "false", 2026-02-23T13:07:28.594 INFO:teuthology.orchestra.run.trial068.stdout: "rbd_balance_snap_reads": "false", 2026-02-23T13:07:28.594 INFO:teuthology.orchestra.run.trial068.stdout: "rbd_blkin_trace_all": "false", 2026-02-23T13:07:28.594 INFO:teuthology.orchestra.run.trial068.stdout: "rbd_blocklist_expire_seconds": "0", 2026-02-23T13:07:28.594 INFO:teuthology.orchestra.run.trial068.stdout: "rbd_blocklist_on_break_lock": "true", 2026-02-23T13:07:28.594 INFO:teuthology.orchestra.run.trial068.stdout: "rbd_cache": "true", 2026-02-23T13:07:28.594 INFO:teuthology.orchestra.run.trial068.stdout: "rbd_cache_block_writes_upfront": "false", 2026-02-23T13:07:28.594 INFO:teuthology.orchestra.run.trial068.stdout: "rbd_cache_max_dirty": "25165824", 2026-02-23T13:07:28.594 INFO:teuthology.orchestra.run.trial068.stdout: "rbd_cache_max_dirty_age": "1.000000", 2026-02-23T13:07:28.594 INFO:teuthology.orchestra.run.trial068.stdout: "rbd_cache_max_dirty_object": "0", 2026-02-23T13:07:28.594 INFO:teuthology.orchestra.run.trial068.stdout: "rbd_cache_policy": "writearound", 2026-02-23T13:07:28.594 INFO:teuthology.orchestra.run.trial068.stdout: "rbd_cache_size": "33554432", 2026-02-23T13:07:28.594 INFO:teuthology.orchestra.run.trial068.stdout: "rbd_cache_target_dirty": "16777216", 2026-02-23T13:07:28.594 INFO:teuthology.orchestra.run.trial068.stdout: "rbd_cache_writethrough_until_flush": "true", 2026-02-23T13:07:28.594 INFO:teuthology.orchestra.run.trial068.stdout: "rbd_clone_copy_on_read": "false", 2026-02-23T13:07:28.594 INFO:teuthology.orchestra.run.trial068.stdout: "rbd_compression_hint": "none", 2026-02-23T13:07:28.594 INFO:teuthology.orchestra.run.trial068.stdout: "rbd_concurrent_management_ops": "10", 2026-02-23T13:07:28.594 INFO:teuthology.orchestra.run.trial068.stdout: "rbd_config_pool_override_update_timestamp": "0", 2026-02-23T13:07:28.594 INFO:teuthology.orchestra.run.trial068.stdout: "rbd_default_clone_format": "auto", 2026-02-23T13:07:28.594 INFO:teuthology.orchestra.run.trial068.stdout: "rbd_default_data_pool": "", 2026-02-23T13:07:28.594 INFO:teuthology.orchestra.run.trial068.stdout: "rbd_default_features": "61", 2026-02-23T13:07:28.594 INFO:teuthology.orchestra.run.trial068.stdout: "rbd_default_format": "2", 2026-02-23T13:07:28.594 INFO:teuthology.orchestra.run.trial068.stdout: "rbd_default_map_options": "", 2026-02-23T13:07:28.594 INFO:teuthology.orchestra.run.trial068.stdout: "rbd_default_order": "22", 2026-02-23T13:07:28.594 INFO:teuthology.orchestra.run.trial068.stdout: "rbd_default_pool": "rbd", 2026-02-23T13:07:28.594 INFO:teuthology.orchestra.run.trial068.stdout: "rbd_default_snapshot_quiesce_mode": "required", 2026-02-23T13:07:28.595 INFO:teuthology.orchestra.run.trial068.stdout: "rbd_default_stripe_count": "0", 2026-02-23T13:07:28.595 INFO:teuthology.orchestra.run.trial068.stdout: "rbd_default_stripe_unit": "0", 2026-02-23T13:07:28.595 INFO:teuthology.orchestra.run.trial068.stdout: "rbd_disable_zero_copy_writes": "true", 2026-02-23T13:07:28.595 INFO:teuthology.orchestra.run.trial068.stdout: "rbd_discard_granularity_bytes": "65536", 2026-02-23T13:07:28.595 INFO:teuthology.orchestra.run.trial068.stdout: "rbd_discard_on_zeroed_write_same": "true", 2026-02-23T13:07:28.595 INFO:teuthology.orchestra.run.trial068.stdout: "rbd_enable_alloc_hint": "true", 2026-02-23T13:07:28.595 INFO:teuthology.orchestra.run.trial068.stdout: "rbd_invalidate_object_map_on_timeout": "true", 2026-02-23T13:07:28.595 INFO:teuthology.orchestra.run.trial068.stdout: "rbd_io_scheduler": "simple", 2026-02-23T13:07:28.595 INFO:teuthology.orchestra.run.trial068.stdout: "rbd_io_scheduler_simple_max_delay": "0", 2026-02-23T13:07:28.595 INFO:teuthology.orchestra.run.trial068.stdout: "rbd_journal_commit_age": "5.000000", 2026-02-23T13:07:28.595 INFO:teuthology.orchestra.run.trial068.stdout: "rbd_journal_max_concurrent_object_sets": "0", 2026-02-23T13:07:28.595 INFO:teuthology.orchestra.run.trial068.stdout: "rbd_journal_max_payload_bytes": "16384", 2026-02-23T13:07:28.595 INFO:teuthology.orchestra.run.trial068.stdout: "rbd_journal_object_flush_age": "0.000000", 2026-02-23T13:07:28.595 INFO:teuthology.orchestra.run.trial068.stdout: "rbd_journal_object_flush_bytes": "1048576", 2026-02-23T13:07:28.595 INFO:teuthology.orchestra.run.trial068.stdout: "rbd_journal_object_flush_interval": "0", 2026-02-23T13:07:28.595 INFO:teuthology.orchestra.run.trial068.stdout: "rbd_journal_object_max_in_flight_appends": "0", 2026-02-23T13:07:28.595 INFO:teuthology.orchestra.run.trial068.stdout: "rbd_journal_object_writethrough_until_flush": "true", 2026-02-23T13:07:28.595 INFO:teuthology.orchestra.run.trial068.stdout: "rbd_journal_order": "24", 2026-02-23T13:07:28.595 INFO:teuthology.orchestra.run.trial068.stdout: "rbd_journal_pool": "", 2026-02-23T13:07:28.595 INFO:teuthology.orchestra.run.trial068.stdout: "rbd_journal_splay_width": "4", 2026-02-23T13:07:28.595 INFO:teuthology.orchestra.run.trial068.stdout: "rbd_localize_parent_reads": "false", 2026-02-23T13:07:28.595 INFO:teuthology.orchestra.run.trial068.stdout: "rbd_localize_snap_reads": "false", 2026-02-23T13:07:28.595 INFO:teuthology.orchestra.run.trial068.stdout: "rbd_mirror_concurrent_image_deletions": "1", 2026-02-23T13:07:28.595 INFO:teuthology.orchestra.run.trial068.stdout: "rbd_mirror_concurrent_image_syncs": "5", 2026-02-23T13:07:28.595 INFO:teuthology.orchestra.run.trial068.stdout: "rbd_mirror_delete_retry_interval": "30.000000", 2026-02-23T13:07:28.595 INFO:teuthology.orchestra.run.trial068.stdout: "rbd_mirror_image_perf_stats_prio": "5", 2026-02-23T13:07:28.595 INFO:teuthology.orchestra.run.trial068.stdout: "rbd_mirror_image_policy_migration_throttle": "300", 2026-02-23T13:07:28.595 INFO:teuthology.orchestra.run.trial068.stdout: "rbd_mirror_image_policy_rebalance_timeout": "0.000000", 2026-02-23T13:07:28.595 INFO:teuthology.orchestra.run.trial068.stdout: "rbd_mirror_image_policy_type": "simple", 2026-02-23T13:07:28.595 INFO:teuthology.orchestra.run.trial068.stdout: "rbd_mirror_image_policy_update_throttle_interval": "1.000000", 2026-02-23T13:07:28.595 INFO:teuthology.orchestra.run.trial068.stdout: "rbd_mirror_image_state_check_interval": "30", 2026-02-23T13:07:28.595 INFO:teuthology.orchestra.run.trial068.stdout: "rbd_mirror_journal_commit_age": "5.000000", 2026-02-23T13:07:28.595 INFO:teuthology.orchestra.run.trial068.stdout: "rbd_mirror_journal_poll_age": "5.000000", 2026-02-23T13:07:28.595 INFO:teuthology.orchestra.run.trial068.stdout: "rbd_mirror_leader_heartbeat_interval": "5", 2026-02-23T13:07:28.595 INFO:teuthology.orchestra.run.trial068.stdout: "rbd_mirror_leader_max_acquire_attempts_before_break": "3", 2026-02-23T13:07:28.595 INFO:teuthology.orchestra.run.trial068.stdout: "rbd_mirror_leader_max_missed_heartbeats": "2", 2026-02-23T13:07:28.595 INFO:teuthology.orchestra.run.trial068.stdout: "rbd_mirror_memory_autotune": "true", 2026-02-23T13:07:28.595 INFO:teuthology.orchestra.run.trial068.stdout: "rbd_mirror_memory_base": "805306368", 2026-02-23T13:07:28.595 INFO:teuthology.orchestra.run.trial068.stdout: "rbd_mirror_memory_cache_autotune_interval": "30.000000", 2026-02-23T13:07:28.595 INFO:teuthology.orchestra.run.trial068.stdout: "rbd_mirror_memory_cache_min": "134217728", 2026-02-23T13:07:28.595 INFO:teuthology.orchestra.run.trial068.stdout: "rbd_mirror_memory_cache_resize_interval": "5.000000", 2026-02-23T13:07:28.595 INFO:teuthology.orchestra.run.trial068.stdout: "rbd_mirror_memory_expected_fragmentation": "0.150000", 2026-02-23T13:07:28.596 INFO:teuthology.orchestra.run.trial068.stdout: "rbd_mirror_memory_target": "4294967296", 2026-02-23T13:07:28.596 INFO:teuthology.orchestra.run.trial068.stdout: "rbd_mirror_perf_stats_prio": "5", 2026-02-23T13:07:28.596 INFO:teuthology.orchestra.run.trial068.stdout: "rbd_mirror_pool_replayers_refresh_interval": "30", 2026-02-23T13:07:28.596 INFO:teuthology.orchestra.run.trial068.stdout: "rbd_mirror_sync_point_update_age": "30.000000", 2026-02-23T13:07:28.596 INFO:teuthology.orchestra.run.trial068.stdout: "rbd_mirroring_delete_delay": "0", 2026-02-23T13:07:28.596 INFO:teuthology.orchestra.run.trial068.stdout: "rbd_mirroring_max_mirroring_snapshots": "5", 2026-02-23T13:07:28.596 INFO:teuthology.orchestra.run.trial068.stdout: "rbd_mirroring_replay_delay": "0", 2026-02-23T13:07:28.596 INFO:teuthology.orchestra.run.trial068.stdout: "rbd_mirroring_resync_after_disconnect": "false", 2026-02-23T13:07:28.596 INFO:teuthology.orchestra.run.trial068.stdout: "rbd_move_parent_to_trash_on_remove": "false", 2026-02-23T13:07:28.596 INFO:teuthology.orchestra.run.trial068.stdout: "rbd_move_to_trash_on_remove": "false", 2026-02-23T13:07:28.596 INFO:teuthology.orchestra.run.trial068.stdout: "rbd_move_to_trash_on_remove_expire_seconds": "0", 2026-02-23T13:07:28.596 INFO:teuthology.orchestra.run.trial068.stdout: "rbd_mtime_update_interval": "60", 2026-02-23T13:07:28.596 INFO:teuthology.orchestra.run.trial068.stdout: "rbd_non_blocking_aio": "true", 2026-02-23T13:07:28.596 INFO:teuthology.orchestra.run.trial068.stdout: "rbd_op_thread_timeout": "60", 2026-02-23T13:07:28.596 INFO:teuthology.orchestra.run.trial068.stdout: "rbd_op_threads": "1", 2026-02-23T13:07:28.596 INFO:teuthology.orchestra.run.trial068.stdout: "rbd_parent_cache_enabled": "false", 2026-02-23T13:07:28.596 INFO:teuthology.orchestra.run.trial068.stdout: "rbd_persistent_cache_mode": "disabled", 2026-02-23T13:07:28.596 INFO:teuthology.orchestra.run.trial068.stdout: "rbd_persistent_cache_path": "/tmp", 2026-02-23T13:07:28.596 INFO:teuthology.orchestra.run.trial068.stdout: "rbd_persistent_cache_size": "1073741824", 2026-02-23T13:07:28.596 INFO:teuthology.orchestra.run.trial068.stdout: "rbd_plugins": "", 2026-02-23T13:07:28.596 INFO:teuthology.orchestra.run.trial068.stdout: "rbd_qos_bps_burst": "0", 2026-02-23T13:07:28.596 INFO:teuthology.orchestra.run.trial068.stdout: "rbd_qos_bps_burst_seconds": "1", 2026-02-23T13:07:28.596 INFO:teuthology.orchestra.run.trial068.stdout: "rbd_qos_bps_limit": "0", 2026-02-23T13:07:28.596 INFO:teuthology.orchestra.run.trial068.stdout: "rbd_qos_exclude_ops": "0", 2026-02-23T13:07:28.596 INFO:teuthology.orchestra.run.trial068.stdout: "rbd_qos_iops_burst": "0", 2026-02-23T13:07:28.596 INFO:teuthology.orchestra.run.trial068.stdout: "rbd_qos_iops_burst_seconds": "1", 2026-02-23T13:07:28.596 INFO:teuthology.orchestra.run.trial068.stdout: "rbd_qos_iops_limit": "0", 2026-02-23T13:07:28.596 INFO:teuthology.orchestra.run.trial068.stdout: "rbd_qos_read_bps_burst": "0", 2026-02-23T13:07:28.596 INFO:teuthology.orchestra.run.trial068.stdout: "rbd_qos_read_bps_burst_seconds": "1", 2026-02-23T13:07:28.596 INFO:teuthology.orchestra.run.trial068.stdout: "rbd_qos_read_bps_limit": "0", 2026-02-23T13:07:28.596 INFO:teuthology.orchestra.run.trial068.stdout: "rbd_qos_read_iops_burst": "0", 2026-02-23T13:07:28.596 INFO:teuthology.orchestra.run.trial068.stdout: "rbd_qos_read_iops_burst_seconds": "1", 2026-02-23T13:07:28.596 INFO:teuthology.orchestra.run.trial068.stdout: "rbd_qos_read_iops_limit": "0", 2026-02-23T13:07:28.596 INFO:teuthology.orchestra.run.trial068.stdout: "rbd_qos_schedule_tick_min": "50", 2026-02-23T13:07:28.596 INFO:teuthology.orchestra.run.trial068.stdout: "rbd_qos_write_bps_burst": "0", 2026-02-23T13:07:28.596 INFO:teuthology.orchestra.run.trial068.stdout: "rbd_qos_write_bps_burst_seconds": "1", 2026-02-23T13:07:28.596 INFO:teuthology.orchestra.run.trial068.stdout: "rbd_qos_write_bps_limit": "0", 2026-02-23T13:07:28.596 INFO:teuthology.orchestra.run.trial068.stdout: "rbd_qos_write_iops_burst": "0", 2026-02-23T13:07:28.596 INFO:teuthology.orchestra.run.trial068.stdout: "rbd_qos_write_iops_burst_seconds": "1", 2026-02-23T13:07:28.596 INFO:teuthology.orchestra.run.trial068.stdout: "rbd_qos_write_iops_limit": "0", 2026-02-23T13:07:28.596 INFO:teuthology.orchestra.run.trial068.stdout: "rbd_quiesce_notification_attempts": "10", 2026-02-23T13:07:28.596 INFO:teuthology.orchestra.run.trial068.stdout: "rbd_read_from_replica_policy": "default", 2026-02-23T13:07:28.596 INFO:teuthology.orchestra.run.trial068.stdout: "rbd_readahead_disable_after_bytes": "52428800", 2026-02-23T13:07:28.597 INFO:teuthology.orchestra.run.trial068.stdout: "rbd_readahead_max_bytes": "524288", 2026-02-23T13:07:28.597 INFO:teuthology.orchestra.run.trial068.stdout: "rbd_readahead_trigger_requests": "10", 2026-02-23T13:07:28.597 INFO:teuthology.orchestra.run.trial068.stdout: "rbd_request_timed_out_seconds": "30", 2026-02-23T13:07:28.597 INFO:teuthology.orchestra.run.trial068.stdout: "rbd_skip_partial_discard": "true", 2026-02-23T13:07:28.597 INFO:teuthology.orchestra.run.trial068.stdout: "rbd_sparse_read_threshold_bytes": "65536", 2026-02-23T13:07:28.597 INFO:teuthology.orchestra.run.trial068.stdout: "rbd_tracing": "false", 2026-02-23T13:07:28.597 INFO:teuthology.orchestra.run.trial068.stdout: "rbd_validate_names": "true", 2026-02-23T13:07:28.597 INFO:teuthology.orchestra.run.trial068.stdout: "rbd_validate_pool": "true", 2026-02-23T13:07:28.597 INFO:teuthology.orchestra.run.trial068.stdout: "restapi_base_url": "", 2026-02-23T13:07:28.597 INFO:teuthology.orchestra.run.trial068.stdout: "restapi_log_level": "", 2026-02-23T13:07:28.597 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_account_default_quota_max_objects": "-1", 2026-02-23T13:07:28.597 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_account_default_quota_max_size": "-1", 2026-02-23T13:07:28.597 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_acl_grants_max_num": "100", 2026-02-23T13:07:28.597 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_admin_entry": "admin", 2026-02-23T13:07:28.597 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_allow_notification_secrets_in_cleartext": "false", 2026-02-23T13:07:28.597 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_asio_assert_yielding": "false", 2026-02-23T13:07:28.597 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_backend_store": "rados", 2026-02-23T13:07:28.597 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_barbican_url": "", 2026-02-23T13:07:28.597 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_beast_enable_async": "true", 2026-02-23T13:07:28.597 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_bucket_counters_cache": "false", 2026-02-23T13:07:28.597 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_bucket_counters_cache_size": "10000", 2026-02-23T13:07:28.597 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_bucket_default_quota_max_objects": "-1", 2026-02-23T13:07:28.597 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_bucket_default_quota_max_size": "-1", 2026-02-23T13:07:28.597 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_bucket_eexist_override": "false", 2026-02-23T13:07:28.597 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_bucket_index_max_aio": "128", 2026-02-23T13:07:28.597 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_bucket_index_transaction_instrumentation": "false", 2026-02-23T13:07:28.597 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_bucket_logging_obj_roll_time": "300", 2026-02-23T13:07:28.597 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_bucket_persistent_notif_num_shards": "11", 2026-02-23T13:07:28.597 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_bucket_quota_cache_size": "10000", 2026-02-23T13:07:28.597 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_bucket_quota_ttl": "600", 2026-02-23T13:07:28.597 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_bucket_sync_spawn_window": "20", 2026-02-23T13:07:28.597 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_cache_enabled": "true", 2026-02-23T13:07:28.597 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_cache_expiry_interval": "900", 2026-02-23T13:07:28.597 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_cache_lru_size": "25000", 2026-02-23T13:07:28.597 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_config_store": "rados", 2026-02-23T13:07:28.597 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_content_length_compat": "false", 2026-02-23T13:07:28.597 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_copy_obj_progress": "true", 2026-02-23T13:07:28.597 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_copy_obj_progress_every_bytes": "1048576", 2026-02-23T13:07:28.597 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_cors_rules_max_num": "100", 2026-02-23T13:07:28.597 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_cross_domain_policy": "", 2026-02-23T13:07:28.597 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_crypt_default_encryption_key": "", 2026-02-23T13:07:28.597 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_crypt_kmip_addr": "", 2026-02-23T13:07:28.598 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_crypt_kmip_ca_path": "", 2026-02-23T13:07:28.598 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_crypt_kmip_client_cert": "", 2026-02-23T13:07:28.598 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_crypt_kmip_client_key": "", 2026-02-23T13:07:28.598 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_crypt_kmip_kms_key_template": "", 2026-02-23T13:07:28.598 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_crypt_kmip_password": "", 2026-02-23T13:07:28.598 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_crypt_kmip_s3_key_template": "$keyid", 2026-02-23T13:07:28.598 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_crypt_kmip_username": "", 2026-02-23T13:07:28.598 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_crypt_require_ssl": "true", 2026-02-23T13:07:28.598 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_crypt_s3_kms_backend": "barbican", 2026-02-23T13:07:28.598 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_crypt_s3_kms_encryption_keys": "", 2026-02-23T13:07:28.598 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_crypt_sse_s3_backend": "vault", 2026-02-23T13:07:28.598 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_crypt_sse_s3_key_template": "%bucket_id", 2026-02-23T13:07:28.598 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_crypt_sse_s3_vault_addr": "", 2026-02-23T13:07:28.598 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_crypt_sse_s3_vault_auth": "token", 2026-02-23T13:07:28.598 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_crypt_sse_s3_vault_namespace": "", 2026-02-23T13:07:28.598 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_crypt_sse_s3_vault_prefix": "", 2026-02-23T13:07:28.598 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_crypt_sse_s3_vault_secret_engine": "transit", 2026-02-23T13:07:28.598 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_crypt_sse_s3_vault_ssl_cacert": "", 2026-02-23T13:07:28.598 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_crypt_sse_s3_vault_ssl_clientcert": "", 2026-02-23T13:07:28.598 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_crypt_sse_s3_vault_ssl_clientkey": "", 2026-02-23T13:07:28.598 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_crypt_sse_s3_vault_token_file": "", 2026-02-23T13:07:28.598 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_crypt_sse_s3_vault_verify_ssl": "true", 2026-02-23T13:07:28.598 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_crypt_suppress_logs": "true", 2026-02-23T13:07:28.598 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_crypt_vault_addr": "", 2026-02-23T13:07:28.598 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_crypt_vault_auth": "token", 2026-02-23T13:07:28.598 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_crypt_vault_namespace": "", 2026-02-23T13:07:28.598 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_crypt_vault_prefix": "", 2026-02-23T13:07:28.598 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_crypt_vault_secret_engine": "transit", 2026-02-23T13:07:28.598 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_crypt_vault_ssl_cacert": "", 2026-02-23T13:07:28.598 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_crypt_vault_ssl_clientcert": "", 2026-02-23T13:07:28.598 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_crypt_vault_ssl_clientkey": "", 2026-02-23T13:07:28.598 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_crypt_vault_token_file": "", 2026-02-23T13:07:28.598 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_crypt_vault_verify_ssl": "true", 2026-02-23T13:07:28.598 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_curl_buffersize": "524288", 2026-02-23T13:07:28.598 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_curl_low_speed_limit": "1024", 2026-02-23T13:07:28.598 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_curl_low_speed_time": "300", 2026-02-23T13:07:28.598 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_curl_tcp_keepalive": "0", 2026-02-23T13:07:28.598 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_curl_wait_timeout_ms": "1000", 2026-02-23T13:07:28.598 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_d3n_l1_datacache_persistent_path": "/tmp/rgw_datacache/", 2026-02-23T13:07:28.598 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_d3n_l1_datacache_size": "1073741824", 2026-02-23T13:07:28.598 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_d3n_l1_evict_cache_on_start": "true", 2026-02-23T13:07:28.598 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_d3n_l1_eviction_policy": "lru", 2026-02-23T13:07:28.598 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_d3n_l1_fadvise": "4", 2026-02-23T13:07:28.598 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_d3n_l1_local_datacache_enabled": "false", 2026-02-23T13:07:28.599 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_d3n_libaio_aio_num": "64", 2026-02-23T13:07:28.599 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_d3n_libaio_aio_threads": "20", 2026-02-23T13:07:28.599 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_d4n_address": "127.0.0.1:6379", 2026-02-23T13:07:28.599 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_d4n_backend_address": "127.0.0.1:6379", 2026-02-23T13:07:28.599 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_d4n_cache_cleaning_interval": "1000", 2026-02-23T13:07:28.599 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_d4n_l1_datacache_address": "127.0.0.1:6379", 2026-02-23T13:07:28.599 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_d4n_l1_datacache_disk_reserve": "1073741824", 2026-02-23T13:07:28.599 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_d4n_l1_datacache_persistent_path": "/tmp/rgw_d4n_datacache/", 2026-02-23T13:07:28.599 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_d4n_l1_evict_cache_on_start": "true", 2026-02-23T13:07:28.599 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_d4n_l1_fadvise": "4", 2026-02-23T13:07:28.599 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_d4n_l1_write_open_flags": "4096", 2026-02-23T13:07:28.599 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_d4n_libaio_aio_num": "64", 2026-02-23T13:07:28.599 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_d4n_libaio_aio_threads": "20", 2026-02-23T13:07:28.599 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_d4n_local_rgw_address": "127.0.0.1:8000", 2026-02-23T13:07:28.599 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_data": "/var/lib/ceph/radosgw/ceph-a", 2026-02-23T13:07:28.599 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_data_log_changes_size": "1000", 2026-02-23T13:07:28.599 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_data_log_num_shards": "128", 2026-02-23T13:07:28.599 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_data_log_window": "30", 2026-02-23T13:07:28.599 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_data_notify_interval_msec": "0", 2026-02-23T13:07:28.599 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_data_sync_poll_interval": "20", 2026-02-23T13:07:28.599 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_data_sync_spawn_window": "20", 2026-02-23T13:07:28.599 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_debug_inject_latency_bi_unlink": "0", 2026-02-23T13:07:28.599 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_debug_inject_olh_cancel_modification_err": "false", 2026-02-23T13:07:28.599 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_debug_inject_set_olh_err": "0", 2026-02-23T13:07:28.599 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_default_data_log_backing": "fifo", 2026-02-23T13:07:28.599 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_default_realm_info_oid": "default.realm", 2026-02-23T13:07:28.599 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_default_region_info_oid": "default.region", 2026-02-23T13:07:28.599 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_default_zone_info_oid": "default.zone", 2026-02-23T13:07:28.599 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_default_zonegroup_info_oid": "default.zonegroup", 2026-02-23T13:07:28.599 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_defer_to_bucket_acls": "", 2026-02-23T13:07:28.599 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_delete_multi_obj_max_num": "1000", 2026-02-23T13:07:28.599 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_disable_s3select": "false", 2026-02-23T13:07:28.599 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_dmclock_admin_lim": "0.000000", 2026-02-23T13:07:28.599 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_dmclock_admin_res": "100.000000", 2026-02-23T13:07:28.599 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_dmclock_admin_wgt": "100.000000", 2026-02-23T13:07:28.599 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_dmclock_auth_lim": "0.000000", 2026-02-23T13:07:28.599 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_dmclock_auth_res": "200.000000", 2026-02-23T13:07:28.599 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_dmclock_auth_wgt": "100.000000", 2026-02-23T13:07:28.599 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_dmclock_data_lim": "0.000000", 2026-02-23T13:07:28.599 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_dmclock_data_res": "500.000000", 2026-02-23T13:07:28.599 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_dmclock_data_wgt": "500.000000", 2026-02-23T13:07:28.599 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_dmclock_metadata_lim": "0.000000", 2026-02-23T13:07:28.600 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_dmclock_metadata_res": "500.000000", 2026-02-23T13:07:28.600 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_dmclock_metadata_wgt": "500.000000", 2026-02-23T13:07:28.600 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_dns_name": "", 2026-02-23T13:07:28.600 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_dns_s3website_name": "", 2026-02-23T13:07:28.600 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_dynamic_resharding": "true", 2026-02-23T13:07:28.600 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_dynamic_resharding_may_reduce": "true", 2026-02-23T13:07:28.600 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_dynamic_resharding_reduction_wait": "120", 2026-02-23T13:07:28.600 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_enable_apis": "s3, s3website, swift, swift_auth, admin, sts, iam, notifications", 2026-02-23T13:07:28.600 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_enable_gc_threads": "true", 2026-02-23T13:07:28.600 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_enable_jwks_url_verification": "false", 2026-02-23T13:07:28.600 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_enable_lc_threads": "true", 2026-02-23T13:07:28.600 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_enable_mdsearch": "true", 2026-02-23T13:07:28.600 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_enable_ops_log": "false", 2026-02-23T13:07:28.600 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_enable_quota_threads": "true", 2026-02-23T13:07:28.600 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_enable_restore_threads": "true", 2026-02-23T13:07:28.600 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_enable_static_website": "false", 2026-02-23T13:07:28.600 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_enable_usage_log": "false", 2026-02-23T13:07:28.600 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_enforce_swift_acls": "true", 2026-02-23T13:07:28.600 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_exit_timeout_secs": "120", 2026-02-23T13:07:28.600 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_expose_bucket": "false", 2026-02-23T13:07:28.600 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_extended_http_attrs": "", 2026-02-23T13:07:28.600 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_filter": "none", 2026-02-23T13:07:28.600 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_frontend_defaults": "beast ssl_certificate=config://rgw/cert/$realm/$zone.crt ssl_private_key=config://rgw/cert/$realm/$zone.key", 2026-02-23T13:07:28.600 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_frontends": "beast port=7480", 2026-02-23T13:07:28.600 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_gc_max_concurrent_io": "10", 2026-02-23T13:07:28.600 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_gc_max_deferred": "50", 2026-02-23T13:07:28.600 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_gc_max_deferred_entries_size": "3072", 2026-02-23T13:07:28.600 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_gc_max_objs": "32", 2026-02-23T13:07:28.600 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_gc_max_queue_size": "134213632", 2026-02-23T13:07:28.600 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_gc_max_trim_chunk": "16", 2026-02-23T13:07:28.600 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_gc_obj_min_wait": "7200", 2026-02-23T13:07:28.600 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_gc_processor_max_time": "3600", 2026-02-23T13:07:28.600 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_gc_processor_period": "3600", 2026-02-23T13:07:28.600 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_get_obj_max_req_size": "4194304", 2026-02-23T13:07:28.600 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_get_obj_window_size": "16777216", 2026-02-23T13:07:28.600 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_graceful_stop": "false", 2026-02-23T13:07:28.600 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_healthcheck_disabling_path": "", 2026-02-23T13:07:28.600 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_http_notif_connection_timeout": "5", 2026-02-23T13:07:28.600 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_http_notif_max_inflight": "8192", 2026-02-23T13:07:28.600 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_http_notif_message_timeout": "10", 2026-02-23T13:07:28.600 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_ignore_get_invalid_range": "false", 2026-02-23T13:07:28.600 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_init_timeout": "300", 2026-02-23T13:07:28.600 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_inject_notify_timeout_probability": "0.000000", 2026-02-23T13:07:28.601 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_json_config": "/var/lib/ceph/radosgw/config.json", 2026-02-23T13:07:28.601 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_kafka_connection_idle": "300", 2026-02-23T13:07:28.601 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_kafka_message_timeout": "5000", 2026-02-23T13:07:28.601 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_kafka_sleep_timeout": "10", 2026-02-23T13:07:28.601 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_keystone_accepted_admin_roles": "", 2026-02-23T13:07:28.601 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_keystone_accepted_reader_roles": "", 2026-02-23T13:07:28.601 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_keystone_accepted_roles": "Member, admin", 2026-02-23T13:07:28.601 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_keystone_admin_domain": "", 2026-02-23T13:07:28.601 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_keystone_admin_password": "", 2026-02-23T13:07:28.601 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_keystone_admin_password_path": "", 2026-02-23T13:07:28.601 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_keystone_admin_project": "", 2026-02-23T13:07:28.601 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_keystone_admin_tenant": "", 2026-02-23T13:07:28.601 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_keystone_admin_user": "", 2026-02-23T13:07:28.601 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_keystone_barbican_domain": "", 2026-02-23T13:07:28.601 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_keystone_barbican_password": "", 2026-02-23T13:07:28.601 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_keystone_barbican_project": "", 2026-02-23T13:07:28.601 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_keystone_barbican_tenant": "", 2026-02-23T13:07:28.601 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_keystone_barbican_user": "", 2026-02-23T13:07:28.601 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_keystone_expired_token_cache_expiration": "3600", 2026-02-23T13:07:28.601 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_keystone_implicit_tenants": "false", 2026-02-23T13:07:28.601 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_keystone_service_token_accepted_roles": "admin", 2026-02-23T13:07:28.601 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_keystone_service_token_enabled": "false", 2026-02-23T13:07:28.601 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_keystone_token_cache_size": "10000", 2026-02-23T13:07:28.601 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_keystone_token_cache_ttl": "300", 2026-02-23T13:07:28.601 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_keystone_url": "", 2026-02-23T13:07:28.601 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_keystone_verify_ssl": "true", 2026-02-23T13:07:28.601 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_lc_debug_interval": "-1", 2026-02-23T13:07:28.601 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_lc_list_cnt": "1000", 2026-02-23T13:07:28.601 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_lc_lock_max_time": "90", 2026-02-23T13:07:28.601 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_lc_max_objs": "32", 2026-02-23T13:07:28.601 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_lc_max_rules": "1000", 2026-02-23T13:07:28.601 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_lc_max_worker": "3", 2026-02-23T13:07:28.601 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_lc_max_wp_worker": "128", 2026-02-23T13:07:28.601 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_lc_ordered_list_threshold": "500", 2026-02-23T13:07:28.601 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_lc_thread_delay": "0", 2026-02-23T13:07:28.601 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_ldap_binddn": "uid=admin,cn=users,dc=example,dc=com", 2026-02-23T13:07:28.601 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_ldap_dnattr": "uid", 2026-02-23T13:07:28.601 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_ldap_searchdn": "cn=users,cn=accounts,dc=example,dc=com", 2026-02-23T13:07:28.601 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_ldap_searchfilter": "", 2026-02-23T13:07:28.601 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_ldap_secret": "/etc/openldap/secret", 2026-02-23T13:07:28.601 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_ldap_uri": "", 2026-02-23T13:07:28.601 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_lfuda_sync_frequency": "60", 2026-02-23T13:07:28.601 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_lifecycle_work_time": "00:00-06:00", 2026-02-23T13:07:28.601 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_list_bucket_min_readahead": "1000", 2026-02-23T13:07:28.602 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_list_buckets_max_chunk": "1000", 2026-02-23T13:07:28.602 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_log_http_headers": "", 2026-02-23T13:07:28.602 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_log_nonexistent_bucket": "false", 2026-02-23T13:07:28.602 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_log_object_name": "%Y-%m-%d-%H-%i-%n", 2026-02-23T13:07:28.602 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_log_object_name_utc": "false", 2026-02-23T13:07:28.602 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_lua_enable": "true", 2026-02-23T13:07:28.602 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_lua_max_memory_per_state": "128000", 2026-02-23T13:07:28.602 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_lua_max_runtime_per_state": "1000", 2026-02-23T13:07:28.602 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_luarocks_location": "/tmp/rgw_luarocks/mon.a", 2026-02-23T13:07:28.602 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_max_attr_name_len": "0", 2026-02-23T13:07:28.602 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_max_attr_size": "0", 2026-02-23T13:07:28.602 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_max_attrs_num_in_req": "0", 2026-02-23T13:07:28.602 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_max_chunk_size": "4194304", 2026-02-23T13:07:28.602 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_max_concurrent_requests": "1024", 2026-02-23T13:07:28.602 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_max_control_aio": "8", 2026-02-23T13:07:28.602 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_max_copy_obj_concurrent_io": "10", 2026-02-23T13:07:28.602 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_max_dynamic_shards": "1999", 2026-02-23T13:07:28.602 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_max_listing_results": "5000", 2026-02-23T13:07:28.602 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_max_notify_retries": "10", 2026-02-23T13:07:28.602 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_max_objs_per_shard": "100000", 2026-02-23T13:07:28.602 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_max_put_param_size": "1048576", 2026-02-23T13:07:28.602 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_max_put_size": "5368709120", 2026-02-23T13:07:28.602 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_max_slo_entries": "1000", 2026-02-23T13:07:28.602 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_md_log_max_shards": "64", 2026-02-23T13:07:28.602 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_md_notify_interval_msec": "200", 2026-02-23T13:07:28.602 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_meta_sync_poll_interval": "20", 2026-02-23T13:07:28.602 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_meta_sync_spawn_window": "20", 2026-02-23T13:07:28.602 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_mime_types_file": "/etc/mime.types", 2026-02-23T13:07:28.602 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_mp_lock_max_time": "600", 2026-02-23T13:07:28.602 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_multi_obj_del_max_aio": "16", 2026-02-23T13:07:28.602 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_multipart_min_part_size": "5242880", 2026-02-23T13:07:28.602 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_multipart_part_upload_limit": "10000", 2026-02-23T13:07:28.602 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_nfs_fhcache_partitions": "3", 2026-02-23T13:07:28.602 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_nfs_fhcache_size": "2017", 2026-02-23T13:07:28.602 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_nfs_frontends": "rgw-nfs", 2026-02-23T13:07:28.602 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_nfs_lru_lane_hiwat": "911", 2026-02-23T13:07:28.602 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_nfs_lru_lanes": "5", 2026-02-23T13:07:28.602 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_nfs_max_gc": "300", 2026-02-23T13:07:28.602 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_nfs_namespace_expire_secs": "300", 2026-02-23T13:07:28.602 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_nfs_run_gc_threads": "false", 2026-02-23T13:07:28.602 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_nfs_run_lc_threads": "false", 2026-02-23T13:07:28.602 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_nfs_run_quota_threads": "false", 2026-02-23T13:07:28.602 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_nfs_run_restore_threads": "false", 2026-02-23T13:07:28.602 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_nfs_run_sync_thread": "false", 2026-02-23T13:07:28.603 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_nfs_s3_fast_attrs": "false", 2026-02-23T13:07:28.603 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_nfs_write_completion_interval_s": "10", 2026-02-23T13:07:28.603 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_num_async_rados_threads": "32", 2026-02-23T13:07:28.603 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_num_control_oids": "8", 2026-02-23T13:07:28.603 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_numa_node": "-1", 2026-02-23T13:07:28.603 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_obj_stripe_size": "4194304", 2026-02-23T13:07:28.603 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_obj_tombstone_cache_size": "1000", 2026-02-23T13:07:28.603 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_objexp_chunk_size": "100", 2026-02-23T13:07:28.603 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_objexp_gc_interval": "600", 2026-02-23T13:07:28.603 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_objexp_hints_num_shards": "127", 2026-02-23T13:07:28.603 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_olh_pending_timeout_sec": "3600", 2026-02-23T13:07:28.603 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_op_thread_suicide_timeout": "0", 2026-02-23T13:07:28.603 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_op_thread_timeout": "600", 2026-02-23T13:07:28.603 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_op_tracing": "false", 2026-02-23T13:07:28.603 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_opa_token": "", 2026-02-23T13:07:28.603 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_opa_url": "", 2026-02-23T13:07:28.603 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_opa_verify_ssl": "true", 2026-02-23T13:07:28.603 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_ops_log_data_backlog": "5242880", 2026-02-23T13:07:28.603 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_ops_log_file_path": "/var/log/ceph/ops-log-ceph-mon.a.log", 2026-02-23T13:07:28.603 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_ops_log_rados": "false", 2026-02-23T13:07:28.603 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_ops_log_socket_path": "", 2026-02-23T13:07:28.603 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_override_bucket_index_max_shards": "0", 2026-02-23T13:07:28.603 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_parquet_buffer_size": "16777216", 2026-02-23T13:07:28.603 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_pending_bucket_index_op_expiration": "120", 2026-02-23T13:07:28.603 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_period_latest_epoch_info_oid": ".latest_epoch", 2026-02-23T13:07:28.603 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_period_push_interval": "2.000000", 2026-02-23T13:07:28.603 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_period_push_interval_max": "30.000000", 2026-02-23T13:07:28.603 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_period_root_pool": ".rgw.root", 2026-02-23T13:07:28.603 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_policy_reject_invalid_principals": "true", 2026-02-23T13:07:28.603 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_posix_base_path": "/tmp/rgw_posix_driver", 2026-02-23T13:07:28.603 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_posix_cache_lanes": "3", 2026-02-23T13:07:28.603 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_posix_cache_lmdb_count": "3", 2026-02-23T13:07:28.603 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_posix_cache_max_buckets": "100", 2026-02-23T13:07:28.603 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_posix_cache_partitions": "3", 2026-02-23T13:07:28.603 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_posix_database_root": "/var/lib/ceph/radosgw", 2026-02-23T13:07:28.603 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_posix_userdb_dir": "/var/lib/ceph/radosgw", 2026-02-23T13:07:28.603 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_print_continue": "true", 2026-02-23T13:07:28.603 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_print_prohibited_content_length": "false", 2026-02-23T13:07:28.603 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_put_obj_max_window_size": "67108864", 2026-02-23T13:07:28.603 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_put_obj_min_window_size": "16777216", 2026-02-23T13:07:28.603 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_rados_pool_autoscale_bias": "4.000000", 2026-02-23T13:07:28.604 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_rados_pool_recovery_priority": "5", 2026-02-23T13:07:28.604 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_rados_tracing": "false", 2026-02-23T13:07:28.604 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_ratelimit_interval": "60", 2026-02-23T13:07:28.604 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_realm": "", 2026-02-23T13:07:28.604 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_realm_id": "", 2026-02-23T13:07:28.604 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_realm_root_pool": ".rgw.root", 2026-02-23T13:07:28.604 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_redis_connection_pool_size": "512", 2026-02-23T13:07:28.604 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_region": "", 2026-02-23T13:07:28.604 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_region_root_pool": ".rgw.root", 2026-02-23T13:07:28.604 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_relaxed_region_enforcement": "false", 2026-02-23T13:07:28.604 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_relaxed_s3_bucket_names": "false", 2026-02-23T13:07:28.604 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_remote_addr_param": "REMOTE_ADDR", 2026-02-23T13:07:28.604 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_request_uri": "", 2026-02-23T13:07:28.604 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_reshard_batch_size": "64", 2026-02-23T13:07:28.604 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_reshard_bucket_lock_duration": "360", 2026-02-23T13:07:28.604 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_reshard_debug_interval": "-1", 2026-02-23T13:07:28.604 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_reshard_max_aio": "128", 2026-02-23T13:07:28.604 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_reshard_num_logs": "16", 2026-02-23T13:07:28.604 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_reshard_progress_judge_interval": "120", 2026-02-23T13:07:28.604 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_reshard_progress_judge_ratio": "0.500000", 2026-02-23T13:07:28.604 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_reshard_thread_interval": "600", 2026-02-23T13:07:28.604 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_reshardlog_threshold": "30000", 2026-02-23T13:07:28.604 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_resolve_cname": "false", 2026-02-23T13:07:28.604 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_rest_getusage_op_compat": "false", 2026-02-23T13:07:28.604 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_restore_debug_interval": "-1", 2026-02-23T13:07:28.604 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_restore_lock_max_time": "90", 2026-02-23T13:07:28.604 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_restore_max_objs": "32", 2026-02-23T13:07:28.604 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_restore_processor_period": "900", 2026-02-23T13:07:28.604 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_run_sync_thread": "true", 2026-02-23T13:07:28.604 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_s3_auth_disable_signature_url": "false", 2026-02-23T13:07:28.604 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_s3_auth_order": "sts, external, local", 2026-02-23T13:07:28.604 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_s3_auth_use_keystone": "false", 2026-02-23T13:07:28.604 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_s3_auth_use_ldap": "false", 2026-02-23T13:07:28.604 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_s3_auth_use_rados": "true", 2026-02-23T13:07:28.604 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_s3_auth_use_sts": "false", 2026-02-23T13:07:28.604 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_s3_client_max_sig_ver": "-1", 2026-02-23T13:07:28.604 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_s3_success_create_obj_status": "0", 2026-02-23T13:07:28.604 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_safe_max_objects_per_shard": "102400", 2026-02-23T13:07:28.604 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_scheduler_type": "throttler", 2026-02-23T13:07:28.604 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_script_uri": "", 2026-02-23T13:07:28.604 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_service_provider_name": "", 2026-02-23T13:07:28.604 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_shard_warning_threshold": "90.000000", 2026-02-23T13:07:28.604 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_sts_client_id": "", 2026-02-23T13:07:28.605 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_sts_client_secret": "", 2026-02-23T13:07:28.605 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_sts_entry": "sts", 2026-02-23T13:07:28.605 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_sts_key": "", 2026-02-23T13:07:28.605 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_sts_max_session_duration": "43200", 2026-02-23T13:07:28.605 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_sts_min_session_duration": "900", 2026-02-23T13:07:28.605 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_sts_token_introspection_url": "", 2026-02-23T13:07:28.605 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_swift_account_in_url": "false", 2026-02-23T13:07:28.605 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_swift_auth_entry": "auth", 2026-02-23T13:07:28.605 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_swift_auth_url": "", 2026-02-23T13:07:28.605 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_swift_custom_header": "", 2026-02-23T13:07:28.605 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_swift_enforce_content_length": "false", 2026-02-23T13:07:28.605 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_swift_need_stats": "true", 2026-02-23T13:07:28.605 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_swift_tenant_name": "", 2026-02-23T13:07:28.605 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_swift_token_expiration": "86400", 2026-02-23T13:07:28.605 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_swift_url": "", 2026-02-23T13:07:28.605 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_swift_url_prefix": "swift", 2026-02-23T13:07:28.605 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_swift_versioning_enabled": "false", 2026-02-23T13:07:28.605 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_sync_data_full_inject_err_probability": "0.000000", 2026-02-23T13:07:28.605 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_sync_data_inject_err_probability": "0.000000", 2026-02-23T13:07:28.605 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_sync_lease_period": "120", 2026-02-23T13:07:28.605 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_sync_log_trim_concurrent_buckets": "4", 2026-02-23T13:07:28.605 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_sync_log_trim_interval": "1200", 2026-02-23T13:07:28.605 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_sync_log_trim_max_buckets": "16", 2026-02-23T13:07:28.605 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_sync_log_trim_min_cold_buckets": "4", 2026-02-23T13:07:28.605 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_sync_meta_inject_err_probability": "0.000000", 2026-02-23T13:07:28.605 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_sync_obj_etag_verify": "false", 2026-02-23T13:07:28.605 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_sync_trace_history_size": "4096", 2026-02-23T13:07:28.605 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_sync_trace_per_node_log_size": "32", 2026-02-23T13:07:28.605 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_sync_trace_servicemap_update_interval": "10", 2026-02-23T13:07:28.605 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_thread_pool_size": "512", 2026-02-23T13:07:28.605 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_topic_persistency_max_retries": "0", 2026-02-23T13:07:28.605 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_topic_persistency_sleep_duration": "0", 2026-02-23T13:07:28.605 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_topic_persistency_time_to_live": "0", 2026-02-23T13:07:28.605 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_topic_require_publish_policy": "false", 2026-02-23T13:07:28.605 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_torrent_comment": "", 2026-02-23T13:07:28.605 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_torrent_createby": "", 2026-02-23T13:07:28.605 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_torrent_encoding": "", 2026-02-23T13:07:28.605 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_torrent_flag": "false", 2026-02-23T13:07:28.605 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_torrent_max_size": "5368709120", 2026-02-23T13:07:28.605 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_torrent_origin": "", 2026-02-23T13:07:28.605 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_torrent_sha_unit": "524288", 2026-02-23T13:07:28.605 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_torrent_tracker": "", 2026-02-23T13:07:28.605 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_trust_forwarded_https": "false", 2026-02-23T13:07:28.605 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_usage_log_flush_threshold": "1024", 2026-02-23T13:07:28.606 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_usage_log_key_transition": "true", 2026-02-23T13:07:28.606 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_usage_log_tick_interval": "30", 2026-02-23T13:07:28.606 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_usage_max_shards": "32", 2026-02-23T13:07:28.606 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_usage_max_user_shards": "1", 2026-02-23T13:07:28.606 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_use_opa_authz": "false", 2026-02-23T13:07:28.606 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_user_counters_cache": "false", 2026-02-23T13:07:28.606 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_user_counters_cache_size": "10000", 2026-02-23T13:07:28.606 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_user_default_quota_max_objects": "-1", 2026-02-23T13:07:28.606 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_user_default_quota_max_size": "-1", 2026-02-23T13:07:28.606 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_user_max_buckets": "1000", 2026-02-23T13:07:28.606 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_user_policies_max_num": "100", 2026-02-23T13:07:28.606 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_user_quota_bucket_sync_interval": "180", 2026-02-23T13:07:28.606 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_user_quota_sync_idle_users": "false", 2026-02-23T13:07:28.606 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_user_quota_sync_interval": "86400", 2026-02-23T13:07:28.606 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_user_quota_sync_wait_time": "86400", 2026-02-23T13:07:28.606 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_user_unique_email": "true", 2026-02-23T13:07:28.606 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_verify_ssl": "true", 2026-02-23T13:07:28.606 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_website_routing_rules_max_num": "50", 2026-02-23T13:07:28.606 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_zone": "", 2026-02-23T13:07:28.606 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_zone_id": "", 2026-02-23T13:07:28.606 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_zone_root_pool": ".rgw.root", 2026-02-23T13:07:28.606 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_zonegroup": "", 2026-02-23T13:07:28.606 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_zonegroup_id": "", 2026-02-23T13:07:28.606 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_zonegroup_root_pool": ".rgw.root", 2026-02-23T13:07:28.606 INFO:teuthology.orchestra.run.trial068.stdout: "rgwlc_auto_session_clear": "true", 2026-02-23T13:07:28.606 INFO:teuthology.orchestra.run.trial068.stdout: "rgwlc_skip_bucket_step": "false", 2026-02-23T13:07:28.606 INFO:teuthology.orchestra.run.trial068.stdout: "rocksdb_block_size": "4096", 2026-02-23T13:07:28.606 INFO:teuthology.orchestra.run.trial068.stdout: "rocksdb_bloom_bits_per_key": "20", 2026-02-23T13:07:28.606 INFO:teuthology.orchestra.run.trial068.stdout: "rocksdb_cache_index_and_filter_blocks": "true", 2026-02-23T13:07:28.606 INFO:teuthology.orchestra.run.trial068.stdout: "rocksdb_cache_index_and_filter_blocks_with_high_priority": "false", 2026-02-23T13:07:28.606 INFO:teuthology.orchestra.run.trial068.stdout: "rocksdb_cache_row_ratio": "0.000000", 2026-02-23T13:07:28.606 INFO:teuthology.orchestra.run.trial068.stdout: "rocksdb_cache_shard_bits": "4", 2026-02-23T13:07:28.606 INFO:teuthology.orchestra.run.trial068.stdout: "rocksdb_cache_size": "536870912", 2026-02-23T13:07:28.606 INFO:teuthology.orchestra.run.trial068.stdout: "rocksdb_cache_type": "binned_lru", 2026-02-23T13:07:28.606 INFO:teuthology.orchestra.run.trial068.stdout: "rocksdb_cf_compact_on_deletion": "true", 2026-02-23T13:07:28.606 INFO:teuthology.orchestra.run.trial068.stdout: "rocksdb_cf_compact_on_deletion_sliding_window": "32768", 2026-02-23T13:07:28.606 INFO:teuthology.orchestra.run.trial068.stdout: "rocksdb_cf_compact_on_deletion_trigger": "16384", 2026-02-23T13:07:28.606 INFO:teuthology.orchestra.run.trial068.stdout: "rocksdb_collect_compaction_stats": "false", 2026-02-23T13:07:28.606 INFO:teuthology.orchestra.run.trial068.stdout: "rocksdb_collect_extended_stats": "false", 2026-02-23T13:07:28.606 INFO:teuthology.orchestra.run.trial068.stdout: "rocksdb_collect_memory_stats": "false", 2026-02-23T13:07:28.606 INFO:teuthology.orchestra.run.trial068.stdout: "rocksdb_delete_range_threshold": "1048576", 2026-02-23T13:07:28.606 INFO:teuthology.orchestra.run.trial068.stdout: "rocksdb_index_type": "binary_search", 2026-02-23T13:07:28.606 INFO:teuthology.orchestra.run.trial068.stdout: "rocksdb_log_to_ceph_log": "true", 2026-02-23T13:07:28.606 INFO:teuthology.orchestra.run.trial068.stdout: "rocksdb_metadata_block_size": "4096", 2026-02-23T13:07:28.607 INFO:teuthology.orchestra.run.trial068.stdout: "rocksdb_partition_filters": "false", 2026-02-23T13:07:28.607 INFO:teuthology.orchestra.run.trial068.stdout: "rocksdb_perf": "false", 2026-02-23T13:07:28.607 INFO:teuthology.orchestra.run.trial068.stdout: "rocksdb_pin_l0_filter_and_index_blocks_in_cache": "false", 2026-02-23T13:07:28.607 INFO:teuthology.orchestra.run.trial068.stdout: "rotating_keys_bootstrap_timeout": "30", 2026-02-23T13:07:28.607 INFO:teuthology.orchestra.run.trial068.stdout: "rotating_keys_renewal_timeout": "10", 2026-02-23T13:07:28.607 INFO:teuthology.orchestra.run.trial068.stdout: "run_dir": "/var/run/ceph", 2026-02-23T13:07:28.607 INFO:teuthology.orchestra.run.trial068.stdout: "seastore_block_create": "true", 2026-02-23T13:07:28.607 INFO:teuthology.orchestra.run.trial068.stdout: "seastore_cachepin_2q_in_ratio": "0.500000", 2026-02-23T13:07:28.607 INFO:teuthology.orchestra.run.trial068.stdout: "seastore_cachepin_2q_out_ratio": "0.500000", 2026-02-23T13:07:28.607 INFO:teuthology.orchestra.run.trial068.stdout: "seastore_cachepin_size_pershard": "2147483648", 2026-02-23T13:07:28.607 INFO:teuthology.orchestra.run.trial068.stdout: "seastore_cachepin_type": "LRU", 2026-02-23T13:07:28.607 INFO:teuthology.orchestra.run.trial068.stdout: "seastore_cbjournal_size": "5368709120", 2026-02-23T13:07:28.607 INFO:teuthology.orchestra.run.trial068.stdout: "seastore_cold_tier_generations": "3", 2026-02-23T13:07:28.607 INFO:teuthology.orchestra.run.trial068.stdout: "seastore_data_delta_based_overwrite": "0", 2026-02-23T13:07:28.607 INFO:teuthology.orchestra.run.trial068.stdout: "seastore_default_max_object_size": "16777216", 2026-02-23T13:07:28.607 INFO:teuthology.orchestra.run.trial068.stdout: "seastore_default_object_metadata_reservation": "16777216", 2026-02-23T13:07:28.607 INFO:teuthology.orchestra.run.trial068.stdout: "seastore_device_size": "53687091200", 2026-02-23T13:07:28.607 INFO:teuthology.orchestra.run.trial068.stdout: "seastore_disable_end_to_end_data_protection": "true", 2026-02-23T13:07:28.607 INFO:teuthology.orchestra.run.trial068.stdout: "seastore_full_integrity_check": "false", 2026-02-23T13:07:28.607 INFO:teuthology.orchestra.run.trial068.stdout: "seastore_hot_tier_generations": "5", 2026-02-23T13:07:28.607 INFO:teuthology.orchestra.run.trial068.stdout: "seastore_journal_batch_capacity": "16", 2026-02-23T13:07:28.607 INFO:teuthology.orchestra.run.trial068.stdout: "seastore_journal_batch_flush_size": "16777216", 2026-02-23T13:07:28.607 INFO:teuthology.orchestra.run.trial068.stdout: "seastore_journal_batch_preferred_fullness": "0.950000", 2026-02-23T13:07:28.607 INFO:teuthology.orchestra.run.trial068.stdout: "seastore_journal_iodepth_limit": "5", 2026-02-23T13:07:28.607 INFO:teuthology.orchestra.run.trial068.stdout: "seastore_main_device_type": "SSD", 2026-02-23T13:07:28.607 INFO:teuthology.orchestra.run.trial068.stdout: "seastore_max_concurrent_transactions": "128", 2026-02-23T13:07:28.607 INFO:teuthology.orchestra.run.trial068.stdout: "seastore_max_data_allocation_size": "0", 2026-02-23T13:07:28.607 INFO:teuthology.orchestra.run.trial068.stdout: "seastore_multiple_tiers_default_evict_ratio": "0.600000", 2026-02-23T13:07:28.607 INFO:teuthology.orchestra.run.trial068.stdout: "seastore_multiple_tiers_fast_evict_ratio": "0.700000", 2026-02-23T13:07:28.607 INFO:teuthology.orchestra.run.trial068.stdout: "seastore_multiple_tiers_stop_evict_ratio": "0.500000", 2026-02-23T13:07:28.607 INFO:teuthology.orchestra.run.trial068.stdout: "seastore_segment_cleaner_gc_formula": "cost_benefit", 2026-02-23T13:07:28.607 INFO:teuthology.orchestra.run.trial068.stdout: "seastore_segment_size": "67108864", 2026-02-23T13:07:28.607 INFO:teuthology.orchestra.run.trial068.stdout: "service_unique_id": "", 2026-02-23T13:07:28.607 INFO:teuthology.orchestra.run.trial068.stdout: "set_keepcaps": "false", 2026-02-23T13:07:28.607 INFO:teuthology.orchestra.run.trial068.stdout: "setgroup": "ceph", 2026-02-23T13:07:28.607 INFO:teuthology.orchestra.run.trial068.stdout: "setuser": "ceph", 2026-02-23T13:07:28.607 INFO:teuthology.orchestra.run.trial068.stdout: "setuser_match_path": "", 2026-02-23T13:07:28.607 INFO:teuthology.orchestra.run.trial068.stdout: "subv_metrics_window_interval": "30", 2026-02-23T13:07:28.607 INFO:teuthology.orchestra.run.trial068.stdout: "target_max_misplaced_ratio": "0.050000", 2026-02-23T13:07:28.607 INFO:teuthology.orchestra.run.trial068.stdout: "thp": "false", 2026-02-23T13:07:28.607 INFO:teuthology.orchestra.run.trial068.stdout: "threadpool_default_timeout": "60", 2026-02-23T13:07:28.607 INFO:teuthology.orchestra.run.trial068.stdout: "threadpool_empty_queue_max_wait": "2", 2026-02-23T13:07:28.607 INFO:teuthology.orchestra.run.trial068.stdout: "throttler_perf_counter": "true", 2026-02-23T13:07:28.608 INFO:teuthology.orchestra.run.trial068.stdout: "tmp_dir": "/tmp", 2026-02-23T13:07:28.608 INFO:teuthology.orchestra.run.trial068.stdout: "tmp_file_template": "/tmp/ceph-mon.a.XXXXXX", 2026-02-23T13:07:28.608 INFO:teuthology.orchestra.run.trial068.stdout: "uadk_compressor_enabled": "false", 2026-02-23T13:07:28.608 INFO:teuthology.orchestra.run.trial068.stdout: "uadk_wd_sync_ctx_num": "2" 2026-02-23T13:07:28.608 INFO:teuthology.orchestra.run.trial068.stdout:} 2026-02-23T13:07:28.608 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'mon.*' injectargs --mon_osd_down_out_interval=0 2026-02-23T13:07:28.750 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:07:28.842 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:07:28.841+0000 7f0324515640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:07:28.842 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:07:28.841+0000 7f0324515640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:07:28.849 INFO:teuthology.orchestra.run.trial068.stdout:{} 2026-02-23T13:07:28.849 INFO:teuthology.orchestra.run.trial068.stderr:mon_osd_down_out_interval = '' (not observed, change may require restart) 2026-02-23T13:07:28.882 INFO:teuthology.run_tasks:Running task full_sequential... 2026-02-23T13:07:28.898 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2026-02-23T13:07:28.902 INFO:tasks.radosbench:Beginning radosbench... 2026-02-23T13:07:28.902 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_0 2026-02-23T13:07:28.902 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd pool create unique_pool_0 16 2026-02-23T13:07:28.903 INFO:tasks.thrashosds.thrasher:starting do_thrash 2026-02-23T13:07:28.903 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2026-02-23T13:07:28.903 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2026-02-23T13:07:28.904 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph pg dump --format=json 2026-02-23T13:07:28.904 INFO:tasks.thrashosds.thrasher:starting do_optrack_toggle with a delay of 2.0 2026-02-23T13:07:28.904 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:07:29.021 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:07:29.021 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:07:29.022 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:07:29.096 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:07:28 trial185 systemd[1]: Stopping Ceph prometheus.a for 5f823084-10b8-11f1-be80-d404e6e7d460... 2026-02-23T13:07:29.096 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:07:29 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[63174]: time=2026-02-23T13:07:29.024Z level=WARN source=main.go:1076 msg="Received an OS signal, exiting gracefully..." signal=terminated 2026-02-23T13:07:29.096 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:07:29 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[63174]: time=2026-02-23T13:07:29.025Z level=INFO source=main.go:1101 msg="Stopping scrape discovery manager..." 2026-02-23T13:07:29.096 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:07:29 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[63174]: time=2026-02-23T13:07:29.025Z level=INFO source=main.go:1115 msg="Stopping notify discovery manager..." 2026-02-23T13:07:29.096 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:07:29 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[63174]: time=2026-02-23T13:07:29.025Z level=INFO source=manager.go:190 msg="Stopping rule manager..." component="rule manager" 2026-02-23T13:07:29.096 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:07:29 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[63174]: time=2026-02-23T13:07:29.025Z level=INFO source=main.go:1097 msg="Scrape discovery manager stopped" 2026-02-23T13:07:29.096 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:07:29 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[63174]: time=2026-02-23T13:07:29.025Z level=INFO source=main.go:1111 msg="Notify discovery manager stopped" 2026-02-23T13:07:29.096 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:07:29 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[63174]: time=2026-02-23T13:07:29.025Z level=INFO source=manager.go:206 msg="Rule manager stopped" component="rule manager" 2026-02-23T13:07:29.096 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:07:29 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[63174]: time=2026-02-23T13:07:29.025Z level=INFO source=main.go:1152 msg="Stopping scrape manager..." 2026-02-23T13:07:29.096 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:07:29 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[63174]: time=2026-02-23T13:07:29.025Z level=INFO source=main.go:1144 msg="Scrape manager stopped" 2026-02-23T13:07:29.096 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:07:29 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[63174]: time=2026-02-23T13:07:29.025Z level=INFO source=manager.go:543 msg="Stopping notification manager..." component=notifier 2026-02-23T13:07:29.096 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:07:29 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[63174]: time=2026-02-23T13:07:29.025Z level=INFO source=manager.go:285 msg="Draining any remaining notifications..." component=notifier 2026-02-23T13:07:29.096 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:07:29 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[63174]: time=2026-02-23T13:07:29.025Z level=INFO source=manager.go:291 msg="Remaining notifications drained" component=notifier 2026-02-23T13:07:29.097 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:07:29 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[63174]: time=2026-02-23T13:07:29.025Z level=INFO source=manager.go:218 msg="Notification manager stopped" component=notifier 2026-02-23T13:07:29.097 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:07:29 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[63174]: time=2026-02-23T13:07:29.025Z level=INFO source=main.go:1422 msg="Notifier manager stopped" 2026-02-23T13:07:29.097 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:07:29 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[63174]: time=2026-02-23T13:07:29.025Z level=INFO source=main.go:1436 msg="See you next time!" 2026-02-23T13:07:29.097 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:07:29 trial185 podman[65285]: 2026-02-23 13:07:29.034973899 +0000 UTC m=+0.022734873 container died 9f48b6754b0044b536409459bd794b8fbb02a1c6a843c8056e9ba520a84d3b00 (image=quay.io/prometheus/prometheus:v3.6.0, name=ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a, maintainer=The Prometheus Authors , org.opencontainers.image.source=https://github.com/prometheus/prometheus) 2026-02-23T13:07:29.097 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:07:29 trial185 podman[65285]: 2026-02-23 13:07:29.042630317 +0000 UTC m=+0.030391291 container remove 9f48b6754b0044b536409459bd794b8fbb02a1c6a843c8056e9ba520a84d3b00 (image=quay.io/prometheus/prometheus:v3.6.0, name=ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a, maintainer=The Prometheus Authors , org.opencontainers.image.source=https://github.com/prometheus/prometheus) 2026-02-23T13:07:29.097 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:07:29 trial185 bash[65285]: ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a 2026-02-23T13:07:29.127 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:07:29.125+0000 7f503bdf0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:07:29.127 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:07:29.126+0000 7fd7a6773640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:07:29.127 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:07:29.126+0000 7f38075e2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:07:29.128 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:07:29.126+0000 7f503bdf0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:07:29.128 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:07:29.126+0000 7fd7a6773640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:07:29.128 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:07:29.126+0000 7f38075e2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:07:29.139 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:07:29.139 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:07:29.149 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:07:29.150 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:07:29.161 INFO:teuthology.orchestra.run.trial068.stdout:osd.2: {} 2026-02-23T13:07:29.161 INFO:teuthology.orchestra.run.trial068.stderr:osd.2: osd_enable_op_tracker = '' 2026-02-23T13:07:29.172 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:07:29.172 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:07:29.243 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T13:07:29.244 INFO:teuthology.orchestra.run.trial068.stderr:dumped all 2026-02-23T13:07:29.280 INFO:teuthology.orchestra.run.trial068.stdout:{"pg_ready":true,"pg_map":{"version":5,"stamp":"2026-02-23T13:07:28.223310+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":3,"kb":2936012800,"kb_used":110516,"kb_used_data":2660,"kb_used_omap":34,"kb_used_meta":107741,"kb_avail":2935902284,"statfs":{"total":3006477107200,"available":3006363938816,"internally_reserved":0,"allocated":2723840,"data_stored":2092609,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":35123,"internal_metadata":110327501},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.998139"},"pg_stats":[{"pgid":"1.0","version":"21'92","reported_seq":18,"reported_epoch":27,"state":"active+clean","last_fresh":"2026-02-23T13:07:14.224912+0000","last_change":"2026-02-23T13:07:14.224739+0000","last_active":"2026-02-23T13:07:14.224912+0000","last_peered":"2026-02-23T13:07:14.224912+0000","last_clean":"2026-02-23T13:07:14.224912+0000","last_became_active":"2026-02-23T13:07:14.220178+0000","last_became_peered":"2026-02-23T13:07:14.220178+0000","last_unstale":"2026-02-23T13:07:14.224912+0000","last_undegraded":"2026-02-23T13:07:14.224912+0000","last_fullsized":"2026-02-23T13:07:14.224912+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:07:05.619208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:07:05.619208+0000","last_clean_scrub_stamp":"2026-02-23T13:07:05.619208+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T13:08:06.145941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":25,"seq":107374182404,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":734003200,"kb_used":27768,"kb_used_data":804,"kb_used_omap":5,"kb_used_meta":26938,"kb_avail":733975432,"statfs":{"total":751619276800,"available":751590842368,"internally_reserved":0,"allocated":823296,"data_stored":667526,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5517,"internal_metadata":27585139},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411335,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":734003200,"kb_used":27776,"kb_used_data":812,"kb_used_omap":6,"kb_used_meta":26937,"kb_avail":733975424,"statfs":{"total":751619276800,"available":751590834176,"internally_reserved":0,"allocated":831488,"data_stored":671817,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6175,"internal_metadata":27584481},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574857,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":734003200,"kb_used":27196,"kb_used_data":232,"kb_used_omap":9,"kb_used_meta":26934,"kb_avail":733976004,"statfs":{"total":751619276800,"available":751591428096,"internally_reserved":0,"allocated":237568,"data_stored":81449,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9437,"internal_metadata":27581219},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705674,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":734003200,"kb_used":27776,"kb_used_data":812,"kb_used_omap":13,"kb_used_meta":26930,"kb_avail":733975424,"statfs":{"total":751619276800,"available":751590834176,"internally_reserved":0,"allocated":831488,"data_stored":671817,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13994,"internal_metadata":27576662},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-23T13:07:29.445 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:29 trial068 ceph-mon[38805]: from='client.14326 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2026-02-23T13:07:29.446 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:29 trial068 ceph-mon[38805]: from='client.14328 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2026-02-23T13:07:29.446 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:29 trial068 ceph-mon[38805]: pgmap v5: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 2.7 TiB / 2.7 TiB avail 2026-02-23T13:07:29.446 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:29 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1248759181' entity='client.admin' cmd={"prefix": "osd dump"} : dispatch 2026-02-23T13:07:29.446 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:29 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T13:07:29.446 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:29 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T13:07:29.446 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:29 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T13:07:29.446 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:29 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T13:07:29.446 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:29 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-23T13:07:29.446 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:29 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2026-02-23T13:07:29.447 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:29 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T13:07:29.447 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:29 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2026-02-23T13:07:29.447 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:29 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1242366319' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2026-02-23T13:07:29.447 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:29 trial068 ceph-mon[38805]: mgrmap e19: x(active, since 4s) 2026-02-23T13:07:29.447 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:29 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3240374258' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:07:29.606 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:07:29 trial185 systemd[1]: ceph-5f823084-10b8-11f1-be80-d404e6e7d460@prometheus.a.service: Deactivated successfully. 2026-02-23T13:07:29.606 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:07:29 trial185 systemd[1]: Stopped Ceph prometheus.a for 5f823084-10b8-11f1-be80-d404e6e7d460. 2026-02-23T13:07:29.607 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:07:29 trial185 systemd[1]: Starting Ceph prometheus.a for 5f823084-10b8-11f1-be80-d404e6e7d460... 2026-02-23T13:07:29.607 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:07:29 trial185 podman[65412]: 2026-02-23 13:07:29.194064764 +0000 UTC m=+0.016690914 container create 477d6cafbae98db5f536d28ef7974a3a593c52419f623a87b51a3f77aee46fc3 (image=quay.io/prometheus/prometheus:v3.6.0, name=ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a, maintainer=The Prometheus Authors , org.opencontainers.image.source=https://github.com/prometheus/prometheus) 2026-02-23T13:07:29.607 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:07:29 trial185 podman[65412]: 2026-02-23 13:07:29.217146365 +0000 UTC m=+0.039772515 container init 477d6cafbae98db5f536d28ef7974a3a593c52419f623a87b51a3f77aee46fc3 (image=quay.io/prometheus/prometheus:v3.6.0, name=ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a, maintainer=The Prometheus Authors , org.opencontainers.image.source=https://github.com/prometheus/prometheus) 2026-02-23T13:07:29.607 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:07:29 trial185 podman[65412]: 2026-02-23 13:07:29.219352447 +0000 UTC m=+0.041978577 container start 477d6cafbae98db5f536d28ef7974a3a593c52419f623a87b51a3f77aee46fc3 (image=quay.io/prometheus/prometheus:v3.6.0, name=ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a, maintainer=The Prometheus Authors , org.opencontainers.image.source=https://github.com/prometheus/prometheus) 2026-02-23T13:07:29.607 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:07:29 trial185 bash[65412]: 477d6cafbae98db5f536d28ef7974a3a593c52419f623a87b51a3f77aee46fc3 2026-02-23T13:07:29.607 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:07:29 trial185 podman[65412]: 2026-02-23 13:07:29.187072008 +0000 UTC m=+0.009698158 image pull 4fcecf061b747d0b5dc55bc7efdba26f87768d058d403b5bcc3f0904cf49f973 quay.io/prometheus/prometheus:v3.6.0 2026-02-23T13:07:29.607 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:07:29 trial185 systemd[1]: Started Ceph prometheus.a for 5f823084-10b8-11f1-be80-d404e6e7d460. 2026-02-23T13:07:29.607 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:07:29 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:07:29.243Z level=INFO source=main.go:1545 msg="updated GOGC" old=100 new=75 2026-02-23T13:07:29.607 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:07:29 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:07:29.244Z level=INFO source=main.go:677 msg="Leaving GOMAXPROCS=24: CPU quota undefined" component=automaxprocs 2026-02-23T13:07:29.607 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:07:29 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:07:29.245Z level=INFO source=memlimit.go:198 msg="GOMEMLIMIT is updated" component=automemlimit package=github.com/KimMachineGun/automemlimit/memlimit GOMEMLIMIT=120212530790 previous=9223372036854775807 2026-02-23T13:07:29.608 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:07:29 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:07:29.245Z level=INFO source=main.go:770 msg="Starting Prometheus Server" mode=server version="(version=3.6.0, branch=HEAD, revision=491734606591ef8c188f5489c70380805bc868c0)" 2026-02-23T13:07:29.608 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:07:29 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:07:29.245Z level=INFO source=main.go:775 msg="operational information" build_context="(go=go1.24.7, platform=linux/amd64, user=root@bfdf1aa26e4d, date=20250921-19:51:16, tags=netgo,builtinassets)" host_details="(Linux 5.14.0-681.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Wed Feb 11 20:19:22 UTC 2026 x86_64 trial185 (none))" fd_limits="(soft=1048576, hard=1048576)" vm_limits="(soft=unlimited, hard=unlimited)" 2026-02-23T13:07:29.608 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:07:29 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:07:29.246Z level=INFO source=web.go:659 msg="Start listening for connections" component=web address=:9095 2026-02-23T13:07:29.608 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:07:29 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:07:29.246Z level=INFO source=main.go:1289 msg="Starting TSDB ..." 2026-02-23T13:07:29.608 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:07:29 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:07:29.248Z level=INFO source=tls_config.go:347 msg="Listening on" component=web address=[::]:9095 2026-02-23T13:07:29.608 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:07:29 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:07:29.248Z level=INFO source=tls_config.go:350 msg="TLS is disabled." component=web http2=false address=[::]:9095 2026-02-23T13:07:29.608 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:07:29 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:07:29.249Z level=INFO source=head.go:667 msg="Replaying on-disk memory mappable chunks if any" component=tsdb 2026-02-23T13:07:29.608 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:07:29 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:07:29.249Z level=INFO source=head.go:754 msg="On-disk memory mappable chunks replay completed" component=tsdb duration=741ns 2026-02-23T13:07:29.608 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:07:29 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:07:29.249Z level=INFO source=head.go:762 msg="Replaying WAL, this may take a while" component=tsdb 2026-02-23T13:07:29.608 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:07:29 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:07:29.250Z level=INFO source=head.go:835 msg="WAL segment loaded" component=tsdb segment=0 maxSegment=1 duration=374.088µs 2026-02-23T13:07:29.608 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:07:29 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:07:29.250Z level=INFO source=head.go:835 msg="WAL segment loaded" component=tsdb segment=1 maxSegment=1 duration=309.31µs 2026-02-23T13:07:29.609 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:07:29 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:07:29.250Z level=INFO source=head.go:872 msg="WAL replay completed" component=tsdb checkpoint_replay_duration=13.254µs wal_replay_duration=698.816µs wbl_replay_duration=120ns chunk_snapshot_load_duration=0s mmap_chunk_replay_duration=741ns total_replay_duration=735.714µs 2026-02-23T13:07:29.609 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:07:29 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:07:29.251Z level=INFO source=main.go:1310 msg="filesystem information" fs_type=XFS_SUPER_MAGIC 2026-02-23T13:07:29.609 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:07:29 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:07:29.251Z level=INFO source=main.go:1313 msg="TSDB started" 2026-02-23T13:07:29.609 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:07:29 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:07:29.251Z level=INFO source=main.go:1498 msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2026-02-23T13:07:29.609 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:07:29 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:07:29.261Z level=INFO source=main.go:1538 msg="Completed loading of configuration file" db_storage=751ns remote_storage=1.012µs web_handler=220ns query_engine=591ns scrape=495.973µs scrape_sd=62.956µs notify=892ns notify_sd=681ns rules=9.213495ms tracing=3.776µs filename=/etc/prometheus/prometheus.yml totalDuration=10.024739ms 2026-02-23T13:07:29.609 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:07:29 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:07:29.261Z level=INFO source=main.go:1274 msg="Server is ready to receive web requests." 2026-02-23T13:07:29.609 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:07:29 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:07:29.261Z level=INFO source=manager.go:176 msg="Starting rule manager..." component="rule manager" 2026-02-23T13:07:29.843 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:07:29 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: [23/Feb/2026:13:07:29] ENGINE Bus STOPPING 2026-02-23T13:07:29.844 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:07:29 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: [23/Feb/2026:13:07:29] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2026-02-23T13:07:29.844 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:07:29 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: [23/Feb/2026:13:07:29] ENGINE Bus STOPPED 2026-02-23T13:07:29.844 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:07:29 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: [23/Feb/2026:13:07:29] ENGINE Bus STARTING 2026-02-23T13:07:29.844 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:07:29 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: [23/Feb/2026:13:07:29] ENGINE Serving on http://:::9283 2026-02-23T13:07:29.844 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:07:29 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: [23/Feb/2026:13:07:29] ENGINE Bus STARTED 2026-02-23T13:07:30.226 INFO:teuthology.orchestra.run.trial068.stderr:pool 'unique_pool_0' created 2026-02-23T13:07:30.270 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd pool application enable unique_pool_0 rados --yes-i-really-mean-it || true 2026-02-23T13:07:30.385 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:07:30.489 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:07:30.488+0000 7fad277c4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:07:30.490 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:30 trial068 ceph-mon[38805]: Reconfiguring prometheus.a deps ['8765', 'alertmanager_configured:False', 'ceph-exporter_configured:False', 'ingress_configured:False', 'mgmt-gateway_configured:False', 'mgr.x', 'nfs_configured:False', 'node-exporter_configured:False', 'nvmeof_configured:False', 'oauth2-proxy_configured:False', 'secure_monitoring_stack:False', 'smb_configured:False'] -> ['8765', 'alertmanager_configured:False', 'ceph-exporter_configured:False', 'ingress_configured:False', 'mgmt-gateway_configured:False', 'mgr.x', 'nfs_configured:False', 'node-exporter_configured:True', 'nvmeof_configured:False', 'oauth2-proxy_configured:False', 'secure_monitoring_stack:False', 'smb_configured:False'] (diff {'node-exporter_configured:False', 'node-exporter_configured:True'}) 2026-02-23T13:07:30.490 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:30 trial068 ceph-mon[38805]: Reconfiguring daemon prometheus.a on trial185 2026-02-23T13:07:30.490 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:30 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3179039402' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16} : dispatch 2026-02-23T13:07:30.490 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:30 trial068 ceph-mon[38805]: from='client.14339 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2026-02-23T13:07:30.491 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:30 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T13:07:30.491 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:30 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T13:07:30.491 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:30 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T13:07:30.491 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:30 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T13:07:30.491 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:30 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "dashboard get-prometheus-api-host"} : dispatch 2026-02-23T13:07:30.491 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:30 trial068 ceph-mon[38805]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2026-02-23T13:07:30.491 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:30 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "dashboard set-prometheus-api-host", "value": "http://trial185:9095"} : dispatch 2026-02-23T13:07:30.491 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:30 trial068 ceph-mon[38805]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://trial185:9095"}]: dispatch 2026-02-23T13:07:30.491 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:30 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T13:07:30.491 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:30 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T13:07:30.491 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:30 trial068 ceph-mon[38805]: Certificate for "cephadm-signed_agent_cert (trial068)" is still valid for 1824 days. 2026-02-23T13:07:30.491 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:30 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2026-02-23T13:07:30.492 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:30 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "orch get-security-config"} : dispatch 2026-02-23T13:07:30.492 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:30 trial068 ceph-mon[38805]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "orch get-security-config"}]: dispatch 2026-02-23T13:07:30.492 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:30 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T13:07:30.492 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:30 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T13:07:30.492 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:30 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-23T13:07:30.492 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:30 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2026-02-23T13:07:30.492 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:30 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T13:07:30.492 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:30 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2026-02-23T13:07:30.492 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:07:30.488+0000 7fad277c4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:07:31.207 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:07:31.229 INFO:teuthology.orchestra.run.trial068.stderr:enabled application 'rados' on pool 'unique_pool_0' 2026-02-23T13:07:31.329 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:07:31.429 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:07:31.428+0000 7f5a04cbe640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:07:31.429 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:07:31.428+0000 7f5a04cbe640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:07:31.442 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:07:31.442 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:07:31.453 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:07:31.453 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:07:31.465 INFO:teuthology.orchestra.run.trial068.stdout:osd.2: {} 2026-02-23T13:07:31.465 INFO:teuthology.orchestra.run.trial068.stderr:osd.2: osd_enable_op_tracker = '' 2026-02-23T13:07:31.477 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:07:31.477 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:07:31.477 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:31 trial068 ceph-mon[38805]: pgmap v6: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 2.7 TiB / 2.7 TiB avail 2026-02-23T13:07:31.477 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:31 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3179039402' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]': finished 2026-02-23T13:07:31.477 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:31 trial068 ceph-mon[38805]: osdmap e29: 4 total, 4 up, 4 in 2026-02-23T13:07:31.477 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:31 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3311706147' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2026-02-23T13:07:32.271 DEBUG:teuthology.orchestra.run.trial068:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -t 128 -b 8192 --object-size 65536 -p unique_pool_0 bench 90 write' 2026-02-23T13:07:32.274 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2026-02-23T13:07:32.274 INFO:teuthology.orchestra.run:waiting for 3000 2026-02-23T13:07:32.321 INFO:tasks.radosbench.radosbench.0.trial068.stdout:hints = 1 2026-02-23T13:07:32.321 INFO:tasks.radosbench.radosbench.0.trial068.stdout:Maintaining 128 concurrent writes of 8192 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2026-02-23T13:07:32.321 INFO:tasks.radosbench.radosbench.0.trial068.stdout:Object prefix: benchmark_data_trial068_77673 2026-02-23T13:07:32.322 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T13:07:32.322 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 0 0 0 0 0 0 - 0 2026-02-23T13:07:32.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:32 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3311706147' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]': finished 2026-02-23T13:07:32.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:32 trial068 ceph-mon[38805]: osdmap e30: 4 total, 4 up, 4 in 2026-02-23T13:07:32.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:32 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2295485579' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:07:33.322 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1 127 9135 9008 70.3693 70.375 0.0138736 0.0140678 2026-02-23T13:07:33.514 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:07:33.537 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:33 trial068 ceph-mon[38805]: pgmap v9: 17 pgs: 16 unknown, 1 active+clean; 577 KiB data, 108 MiB used, 2.7 TiB / 2.7 TiB avail; 25 KiB/s rd, 0 B/s wr, 10 op/s 2026-02-23T13:07:33.537 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:33 trial068 ceph-mon[38805]: osdmap e31: 4 total, 4 up, 4 in 2026-02-23T13:07:33.655 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:07:33.783 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:07:33.782+0000 7f8028685640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:07:33.783 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:07:33.782+0000 7f8028685640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:07:33.800 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:07:33.800 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:07:33.813 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:07:33.813 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:07:33.825 INFO:teuthology.orchestra.run.trial068.stdout:osd.2: {} 2026-02-23T13:07:33.825 INFO:teuthology.orchestra.run.trial068.stderr:osd.2: osd_enable_op_tracker = '' 2026-02-23T13:07:33.838 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:07:33.838 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:07:34.281 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2026-02-23T13:07:34.281 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2026-02-23T13:07:34.282 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph pg dump --format=json 2026-02-23T13:07:34.322 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2 128 17913 17785 69.4672 68.5703 0.0122204 0.0143168 2026-02-23T13:07:34.416 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:07:34.481 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:34 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1640443065' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:07:34.541 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:07:34.539+0000 7f50bbca4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:07:34.541 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:07:34.539+0000 7f50bbca4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:07:34.675 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T13:07:34.675 INFO:teuthology.orchestra.run.trial068.stderr:dumped all 2026-02-23T13:07:34.720 INFO:teuthology.orchestra.run.trial068.stdout:{"pg_ready":true,"pg_map":{"version":11,"stamp":"2026-02-23T13:07:34.224027+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":240,"num_write":135,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":3,"kb":2936012800,"kb_used":110524,"kb_used_data":2668,"kb_used_omap":34,"kb_used_meta":107741,"kb_avail":2935902276,"statfs":{"total":3006477107200,"available":3006363930624,"internally_reserved":0,"allocated":2732032,"data_stored":2096900,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":35772,"internal_metadata":110326852},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":60,"num_read_kb":176,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.000717"},"pg_stats":[{"pgid":"2.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2026-02-23T13:07:30.224555+0000","last_change":"2026-02-23T13:07:30.224555+0000","last_active":"2026-02-23T13:07:30.224555+0000","last_peered":"2026-02-23T13:07:30.224555+0000","last_clean":"2026-02-23T13:07:30.224555+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2026-02-23T13:07:30.224555+0000","last_undegraded":"2026-02-23T13:07:30.224555+0000","last_fullsized":"2026-02-23T13:07:30.224555+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_clean_scrub_stamp":"2026-02-23T13:07:30.224555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2026-02-23T13:07:30.224555+0000","last_change":"2026-02-23T13:07:30.224555+0000","last_active":"2026-02-23T13:07:30.224555+0000","last_peered":"2026-02-23T13:07:30.224555+0000","last_clean":"2026-02-23T13:07:30.224555+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2026-02-23T13:07:30.224555+0000","last_undegraded":"2026-02-23T13:07:30.224555+0000","last_fullsized":"2026-02-23T13:07:30.224555+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_clean_scrub_stamp":"2026-02-23T13:07:30.224555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2026-02-23T13:07:30.224555+0000","last_change":"2026-02-23T13:07:30.224555+0000","last_active":"2026-02-23T13:07:30.224555+0000","last_peered":"2026-02-23T13:07:30.224555+0000","last_clean":"2026-02-23T13:07:30.224555+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2026-02-23T13:07:30.224555+0000","last_undegraded":"2026-02-23T13:07:30.224555+0000","last_fullsized":"2026-02-23T13:07:30.224555+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_clean_scrub_stamp":"2026-02-23T13:07:30.224555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2026-02-23T13:07:30.224555+0000","last_change":"2026-02-23T13:07:30.224555+0000","last_active":"2026-02-23T13:07:30.224555+0000","last_peered":"2026-02-23T13:07:30.224555+0000","last_clean":"2026-02-23T13:07:30.224555+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2026-02-23T13:07:30.224555+0000","last_undegraded":"2026-02-23T13:07:30.224555+0000","last_fullsized":"2026-02-23T13:07:30.224555+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_clean_scrub_stamp":"2026-02-23T13:07:30.224555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"1.0","version":"28'99","reported_seq":45,"reported_epoch":28,"state":"active+clean","last_fresh":"2026-02-23T13:07:26.430875+0000","last_change":"2026-02-23T13:07:14.224739+0000","last_active":"2026-02-23T13:07:26.430875+0000","last_peered":"2026-02-23T13:07:26.430875+0000","last_clean":"2026-02-23T13:07:26.430875+0000","last_became_active":"2026-02-23T13:07:14.220178+0000","last_became_peered":"2026-02-23T13:07:14.220178+0000","last_unstale":"2026-02-23T13:07:26.430875+0000","last_undegraded":"2026-02-23T13:07:26.430875+0000","last_fullsized":"2026-02-23T13:07:26.430875+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:07:05.619208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:07:05.619208+0000","last_clean_scrub_stamp":"2026-02-23T13:07:05.619208+0000","objects_scrubbed":0,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T13:08:06.145941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":240,"num_write":135,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2026-02-23T13:07:30.224555+0000","last_change":"2026-02-23T13:07:30.224555+0000","last_active":"2026-02-23T13:07:30.224555+0000","last_peered":"2026-02-23T13:07:30.224555+0000","last_clean":"2026-02-23T13:07:30.224555+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2026-02-23T13:07:30.224555+0000","last_undegraded":"2026-02-23T13:07:30.224555+0000","last_fullsized":"2026-02-23T13:07:30.224555+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_clean_scrub_stamp":"2026-02-23T13:07:30.224555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2026-02-23T13:07:30.224555+0000","last_change":"2026-02-23T13:07:30.224555+0000","last_active":"2026-02-23T13:07:30.224555+0000","last_peered":"2026-02-23T13:07:30.224555+0000","last_clean":"2026-02-23T13:07:30.224555+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2026-02-23T13:07:30.224555+0000","last_undegraded":"2026-02-23T13:07:30.224555+0000","last_fullsized":"2026-02-23T13:07:30.224555+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_clean_scrub_stamp":"2026-02-23T13:07:30.224555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2026-02-23T13:07:30.224555+0000","last_change":"2026-02-23T13:07:30.224555+0000","last_active":"2026-02-23T13:07:30.224555+0000","last_peered":"2026-02-23T13:07:30.224555+0000","last_clean":"2026-02-23T13:07:30.224555+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2026-02-23T13:07:30.224555+0000","last_undegraded":"2026-02-23T13:07:30.224555+0000","last_fullsized":"2026-02-23T13:07:30.224555+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_clean_scrub_stamp":"2026-02-23T13:07:30.224555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2026-02-23T13:07:30.224555+0000","last_change":"2026-02-23T13:07:30.224555+0000","last_active":"2026-02-23T13:07:30.224555+0000","last_peered":"2026-02-23T13:07:30.224555+0000","last_clean":"2026-02-23T13:07:30.224555+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2026-02-23T13:07:30.224555+0000","last_undegraded":"2026-02-23T13:07:30.224555+0000","last_fullsized":"2026-02-23T13:07:30.224555+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_clean_scrub_stamp":"2026-02-23T13:07:30.224555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2026-02-23T13:07:30.224555+0000","last_change":"2026-02-23T13:07:30.224555+0000","last_active":"2026-02-23T13:07:30.224555+0000","last_peered":"2026-02-23T13:07:30.224555+0000","last_clean":"2026-02-23T13:07:30.224555+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2026-02-23T13:07:30.224555+0000","last_undegraded":"2026-02-23T13:07:30.224555+0000","last_fullsized":"2026-02-23T13:07:30.224555+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_clean_scrub_stamp":"2026-02-23T13:07:30.224555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2026-02-23T13:07:30.224555+0000","last_change":"2026-02-23T13:07:30.224555+0000","last_active":"2026-02-23T13:07:30.224555+0000","last_peered":"2026-02-23T13:07:30.224555+0000","last_clean":"2026-02-23T13:07:30.224555+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2026-02-23T13:07:30.224555+0000","last_undegraded":"2026-02-23T13:07:30.224555+0000","last_fullsized":"2026-02-23T13:07:30.224555+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_clean_scrub_stamp":"2026-02-23T13:07:30.224555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2026-02-23T13:07:30.224555+0000","last_change":"2026-02-23T13:07:30.224555+0000","last_active":"2026-02-23T13:07:30.224555+0000","last_peered":"2026-02-23T13:07:30.224555+0000","last_clean":"2026-02-23T13:07:30.224555+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2026-02-23T13:07:30.224555+0000","last_undegraded":"2026-02-23T13:07:30.224555+0000","last_fullsized":"2026-02-23T13:07:30.224555+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_clean_scrub_stamp":"2026-02-23T13:07:30.224555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2026-02-23T13:07:30.224555+0000","last_change":"2026-02-23T13:07:30.224555+0000","last_active":"2026-02-23T13:07:30.224555+0000","last_peered":"2026-02-23T13:07:30.224555+0000","last_clean":"2026-02-23T13:07:30.224555+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2026-02-23T13:07:30.224555+0000","last_undegraded":"2026-02-23T13:07:30.224555+0000","last_fullsized":"2026-02-23T13:07:30.224555+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_clean_scrub_stamp":"2026-02-23T13:07:30.224555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2026-02-23T13:07:30.224555+0000","last_change":"2026-02-23T13:07:30.224555+0000","last_active":"2026-02-23T13:07:30.224555+0000","last_peered":"2026-02-23T13:07:30.224555+0000","last_clean":"2026-02-23T13:07:30.224555+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2026-02-23T13:07:30.224555+0000","last_undegraded":"2026-02-23T13:07:30.224555+0000","last_fullsized":"2026-02-23T13:07:30.224555+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_clean_scrub_stamp":"2026-02-23T13:07:30.224555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2026-02-23T13:07:30.224555+0000","last_change":"2026-02-23T13:07:30.224555+0000","last_active":"2026-02-23T13:07:30.224555+0000","last_peered":"2026-02-23T13:07:30.224555+0000","last_clean":"2026-02-23T13:07:30.224555+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2026-02-23T13:07:30.224555+0000","last_undegraded":"2026-02-23T13:07:30.224555+0000","last_fullsized":"2026-02-23T13:07:30.224555+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_clean_scrub_stamp":"2026-02-23T13:07:30.224555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2026-02-23T13:07:30.224555+0000","last_change":"2026-02-23T13:07:30.224555+0000","last_active":"2026-02-23T13:07:30.224555+0000","last_peered":"2026-02-23T13:07:30.224555+0000","last_clean":"2026-02-23T13:07:30.224555+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2026-02-23T13:07:30.224555+0000","last_undegraded":"2026-02-23T13:07:30.224555+0000","last_fullsized":"2026-02-23T13:07:30.224555+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_clean_scrub_stamp":"2026-02-23T13:07:30.224555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2026-02-23T13:07:30.224555+0000","last_change":"2026-02-23T13:07:30.224555+0000","last_active":"2026-02-23T13:07:30.224555+0000","last_peered":"2026-02-23T13:07:30.224555+0000","last_clean":"2026-02-23T13:07:30.224555+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2026-02-23T13:07:30.224555+0000","last_undegraded":"2026-02-23T13:07:30.224555+0000","last_fullsized":"2026-02-23T13:07:30.224555+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_clean_scrub_stamp":"2026-02-23T13:07:30.224555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":240,"num_write":135,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":25,"seq":107374182405,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":734003200,"kb_used":27776,"kb_used_data":812,"kb_used_omap":6,"kb_used_meta":26937,"kb_avail":733975424,"statfs":{"total":751619276800,"available":751590834176,"internally_reserved":0,"allocated":831488,"data_stored":671817,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6166,"internal_metadata":27584490},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411336,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":734003200,"kb_used":27776,"kb_used_data":812,"kb_used_omap":6,"kb_used_meta":26937,"kb_avail":733975424,"statfs":{"total":751619276800,"available":751590834176,"internally_reserved":0,"allocated":831488,"data_stored":671817,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6175,"internal_metadata":27584481},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574858,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":734003200,"kb_used":27196,"kb_used_data":232,"kb_used_omap":9,"kb_used_meta":26934,"kb_avail":733976004,"statfs":{"total":751619276800,"available":751591428096,"internally_reserved":0,"allocated":237568,"data_stored":81449,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9437,"internal_metadata":27581219},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705675,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":734003200,"kb_used":27776,"kb_used_data":812,"kb_used_omap":13,"kb_used_meta":26930,"kb_avail":733975424,"statfs":{"total":751619276800,"available":751590834176,"internally_reserved":0,"allocated":831488,"data_stored":671817,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13994,"internal_metadata":27576662},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-23T13:07:35.322 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3 128 26431 26303 68.4921 66.5469 0.0175143 0.0145628 2026-02-23T13:07:35.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:35 trial068 ceph-mon[38805]: pgmap v11: 17 pgs: 16 unknown, 1 active+clean; 577 KiB data, 108 MiB used, 2.7 TiB / 2.7 TiB avail; 29 KiB/s rd, 0 B/s wr, 11 op/s 2026-02-23T13:07:35.880 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:07:36.028 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:07:36.152 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:07:36.151+0000 7f4615863640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:07:36.152 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:07:36.151+0000 7f4615863640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:07:36.167 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:07:36.167 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:07:36.182 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:07:36.182 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:07:36.195 INFO:teuthology.orchestra.run.trial068.stdout:osd.2: {} 2026-02-23T13:07:36.195 INFO:teuthology.orchestra.run.trial068.stderr:osd.2: osd_enable_op_tracker = '' 2026-02-23T13:07:36.209 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:07:36.209 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:07:36.322 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4 128 34726 34598 67.5689 64.8047 0.00803512 0.0147627 2026-02-23T13:07:36.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:36 trial068 ceph-mon[38805]: from='client.14351 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2026-02-23T13:07:36.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:36 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3161951009' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:07:37.322 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5 127 43394 43267 67.5998 67.7266 0.03153 0.0147631 2026-02-23T13:07:37.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:37 trial068 ceph-mon[38805]: pgmap v12: 17 pgs: 17 active+clean; 200 MiB data, 582 MiB used, 2.7 TiB / 2.7 TiB avail; 29 KiB/s rd, 33 MiB/s wr, 4.26k op/s 2026-02-23T13:07:38.254 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:07:38.322 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6 128 51689 51561 67.1318 64.7969 0.0073375 0.0148652 2026-02-23T13:07:38.395 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:07:38.531 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:07:38.530+0000 7fde6ff48640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:07:38.532 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:07:38.530+0000 7fde6ff48640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:07:38.546 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:07:38.546 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:07:38.560 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:07:38.560 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:07:38.573 INFO:teuthology.orchestra.run.trial068.stdout:osd.2: {} 2026-02-23T13:07:38.573 INFO:teuthology.orchestra.run.trial068.stderr:osd.2: osd_enable_op_tracker = '' 2026-02-23T13:07:38.588 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:07:38.589 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:07:39.322 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7 127 59204 59077 65.9293 58.7188 0.0311639 0.0151403 2026-02-23T13:07:39.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:39 trial068 ceph-mon[38805]: pgmap v13: 17 pgs: 17 active+clean; 200 MiB data, 582 MiB used, 2.7 TiB / 2.7 TiB avail; 25 MiB/s wr, 3.19k op/s 2026-02-23T13:07:39.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:39 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/393626579' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:07:39.723 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2026-02-23T13:07:39.723 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2026-02-23T13:07:39.723 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph pg dump --format=json 2026-02-23T13:07:39.867 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:07:39.994 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:07:39.992+0000 7fd577010640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:07:39.994 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:07:39.992+0000 7fd577010640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:07:40.156 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T13:07:40.157 INFO:teuthology.orchestra.run.trial068.stderr:dumped all 2026-02-23T13:07:40.203 INFO:teuthology.orchestra.run.trial068.stdout:{"pg_ready":true,"pg_map":{"version":13,"stamp":"2026-02-23T13:07:38.224488+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":209519136,"num_objects":3191,"num_object_clones":0,"num_object_copies":9573,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3191,"num_whiteouts":0,"num_read":136,"num_read_kb":240,"num_write":25639,"num_write_kb":206182,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":25603,"ondisk_log_size":25603,"up":51,"acting":51,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":51,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":2936012800,"kb_used":596116,"kb_used_data":488260,"kb_used_omap":43,"kb_used_meta":107732,"kb_avail":2935416684,"statfs":{"total":3006477107200,"available":3005866684416,"internally_reserved":0,"allocated":499978240,"data_stored":499252752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":44897,"internal_metadata":110317727},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[23,18,19,44,16],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":208928768,"num_objects":3189,"num_object_clones":0,"num_object_copies":9567,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3189,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":25504,"num_write_kb":204032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.997979"},"pg_stats":[{"pgid":"2.f","version":"31'1560","reported_seq":1574,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-23T13:07:35.214547+0000","last_change":"2026-02-23T13:07:31.232168+0000","last_active":"2026-02-23T13:07:35.214547+0000","last_peered":"2026-02-23T13:07:35.214547+0000","last_clean":"2026-02-23T13:07:35.214547+0000","last_became_active":"2026-02-23T13:07:31.231983+0000","last_became_peered":"2026-02-23T13:07:31.231983+0000","last_unstale":"2026-02-23T13:07:35.214547+0000","last_undegraded":"2026-02-23T13:07:35.214547+0000","last_fullsized":"2026-02-23T13:07:35.214547+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_clean_scrub_stamp":"2026-02-23T13:07:30.224555+0000","objects_scrubbed":0,"log_size":1560,"log_dups_size":0,"ondisk_log_size":1560,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T13:08:32.236459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12779520,"num_objects":195,"num_object_clones":0,"num_object_copies":585,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":195,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1560,"num_write_kb":12480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.e","version":"31'1448","reported_seq":1462,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-23T13:07:35.223965+0000","last_change":"2026-02-23T13:07:31.233069+0000","last_active":"2026-02-23T13:07:35.223965+0000","last_peered":"2026-02-23T13:07:35.223965+0000","last_clean":"2026-02-23T13:07:35.223965+0000","last_became_active":"2026-02-23T13:07:31.232454+0000","last_became_peered":"2026-02-23T13:07:31.232454+0000","last_unstale":"2026-02-23T13:07:35.223965+0000","last_undegraded":"2026-02-23T13:07:35.223965+0000","last_fullsized":"2026-02-23T13:07:35.223965+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_clean_scrub_stamp":"2026-02-23T13:07:30.224555+0000","objects_scrubbed":0,"log_size":1448,"log_dups_size":0,"ondisk_log_size":1448,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T13:08:31.879194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11862016,"num_objects":181,"num_object_clones":0,"num_object_copies":543,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":181,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1448,"num_write_kb":11584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.d","version":"31'1352","reported_seq":1366,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-23T13:07:35.210945+0000","last_change":"2026-02-23T13:07:31.233445+0000","last_active":"2026-02-23T13:07:35.210945+0000","last_peered":"2026-02-23T13:07:35.210945+0000","last_clean":"2026-02-23T13:07:35.210945+0000","last_became_active":"2026-02-23T13:07:31.232915+0000","last_became_peered":"2026-02-23T13:07:31.232915+0000","last_unstale":"2026-02-23T13:07:35.210945+0000","last_undegraded":"2026-02-23T13:07:35.210945+0000","last_fullsized":"2026-02-23T13:07:35.210945+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_clean_scrub_stamp":"2026-02-23T13:07:30.224555+0000","objects_scrubbed":0,"log_size":1352,"log_dups_size":0,"ondisk_log_size":1352,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T13:08:35.423973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11075584,"num_objects":169,"num_object_clones":0,"num_object_copies":507,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":169,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1352,"num_write_kb":10816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"31'1712","reported_seq":1710,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-23T13:07:35.210203+0000","last_change":"2026-02-23T13:07:31.233497+0000","last_active":"2026-02-23T13:07:35.210203+0000","last_peered":"2026-02-23T13:07:35.210203+0000","last_clean":"2026-02-23T13:07:35.210203+0000","last_became_active":"2026-02-23T13:07:31.233152+0000","last_became_peered":"2026-02-23T13:07:31.233152+0000","last_unstale":"2026-02-23T13:07:35.210203+0000","last_undegraded":"2026-02-23T13:07:35.210203+0000","last_fullsized":"2026-02-23T13:07:35.210203+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_clean_scrub_stamp":"2026-02-23T13:07:30.224555+0000","objects_scrubbed":0,"log_size":1712,"log_dups_size":0,"ondisk_log_size":1712,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T13:08:57.971970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14024704,"num_objects":214,"num_object_clones":0,"num_object_copies":642,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":214,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1712,"num_write_kb":13696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"28'99","reported_seq":51,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-23T13:07:32.230297+0000","last_change":"2026-02-23T13:07:14.224739+0000","last_active":"2026-02-23T13:07:32.230297+0000","last_peered":"2026-02-23T13:07:32.230297+0000","last_clean":"2026-02-23T13:07:32.230297+0000","last_became_active":"2026-02-23T13:07:14.220178+0000","last_became_peered":"2026-02-23T13:07:14.220178+0000","last_unstale":"2026-02-23T13:07:32.230297+0000","last_undegraded":"2026-02-23T13:07:32.230297+0000","last_fullsized":"2026-02-23T13:07:32.230297+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:07:05.619208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:07:05.619208+0000","last_clean_scrub_stamp":"2026-02-23T13:07:05.619208+0000","objects_scrubbed":0,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T13:08:06.145941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":240,"num_write":135,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"31'1717","reported_seq":1718,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-23T13:07:35.222680+0000","last_change":"2026-02-23T13:07:31.233851+0000","last_active":"2026-02-23T13:07:35.222680+0000","last_peered":"2026-02-23T13:07:35.222680+0000","last_clean":"2026-02-23T13:07:35.222680+0000","last_became_active":"2026-02-23T13:07:31.233470+0000","last_became_peered":"2026-02-23T13:07:31.233470+0000","last_unstale":"2026-02-23T13:07:35.222680+0000","last_undegraded":"2026-02-23T13:07:35.222680+0000","last_fullsized":"2026-02-23T13:07:35.222680+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_clean_scrub_stamp":"2026-02-23T13:07:30.224555+0000","objects_scrubbed":0,"log_size":1717,"log_dups_size":0,"ondisk_log_size":1717,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T13:08:34.869333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14065664,"num_objects":215,"num_object_clones":0,"num_object_copies":645,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":215,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1717,"num_write_kb":13736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"31'1582","reported_seq":1590,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-23T13:07:35.200081+0000","last_change":"2026-02-23T13:07:31.232349+0000","last_active":"2026-02-23T13:07:35.200081+0000","last_peered":"2026-02-23T13:07:35.200081+0000","last_clean":"2026-02-23T13:07:35.200081+0000","last_became_active":"2026-02-23T13:07:31.232045+0000","last_became_peered":"2026-02-23T13:07:31.232045+0000","last_unstale":"2026-02-23T13:07:35.200081+0000","last_undegraded":"2026-02-23T13:07:35.200081+0000","last_fullsized":"2026-02-23T13:07:35.200081+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_clean_scrub_stamp":"2026-02-23T13:07:30.224555+0000","objects_scrubbed":0,"log_size":1582,"log_dups_size":0,"ondisk_log_size":1582,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T13:08:54.216320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12959744,"num_objects":198,"num_object_clones":0,"num_object_copies":594,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":198,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1582,"num_write_kb":12656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"31'1616","reported_seq":1622,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-23T13:07:35.209622+0000","last_change":"2026-02-23T13:07:31.233117+0000","last_active":"2026-02-23T13:07:35.209622+0000","last_peered":"2026-02-23T13:07:35.209622+0000","last_clean":"2026-02-23T13:07:35.209622+0000","last_became_active":"2026-02-23T13:07:31.232562+0000","last_became_peered":"2026-02-23T13:07:31.232562+0000","last_unstale":"2026-02-23T13:07:35.209622+0000","last_undegraded":"2026-02-23T13:07:35.209622+0000","last_fullsized":"2026-02-23T13:07:35.209622+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_clean_scrub_stamp":"2026-02-23T13:07:30.224555+0000","objects_scrubbed":0,"log_size":1616,"log_dups_size":0,"ondisk_log_size":1616,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T13:08:51.508423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13238272,"num_objects":202,"num_object_clones":0,"num_object_copies":606,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":202,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1616,"num_write_kb":12928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"31'1677","reported_seq":1684,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-23T13:07:35.223413+0000","last_change":"2026-02-23T13:07:31.234086+0000","last_active":"2026-02-23T13:07:35.223413+0000","last_peered":"2026-02-23T13:07:35.223413+0000","last_clean":"2026-02-23T13:07:35.223413+0000","last_became_active":"2026-02-23T13:07:31.233796+0000","last_became_peered":"2026-02-23T13:07:31.233796+0000","last_unstale":"2026-02-23T13:07:35.223413+0000","last_undegraded":"2026-02-23T13:07:35.223413+0000","last_fullsized":"2026-02-23T13:07:35.223413+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_clean_scrub_stamp":"2026-02-23T13:07:30.224555+0000","objects_scrubbed":0,"log_size":1677,"log_dups_size":0,"ondisk_log_size":1677,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T13:08:52.748856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13737984,"num_objects":210,"num_object_clones":0,"num_object_copies":630,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":210,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1677,"num_write_kb":13416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"31'1392","reported_seq":1406,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-23T13:07:35.221763+0000","last_change":"2026-02-23T13:07:31.234133+0000","last_active":"2026-02-23T13:07:35.221763+0000","last_peered":"2026-02-23T13:07:35.221763+0000","last_clean":"2026-02-23T13:07:35.221763+0000","last_became_active":"2026-02-23T13:07:31.233956+0000","last_became_peered":"2026-02-23T13:07:31.233956+0000","last_unstale":"2026-02-23T13:07:35.221763+0000","last_undegraded":"2026-02-23T13:07:35.221763+0000","last_fullsized":"2026-02-23T13:07:35.221763+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_clean_scrub_stamp":"2026-02-23T13:07:30.224555+0000","objects_scrubbed":0,"log_size":1392,"log_dups_size":0,"ondisk_log_size":1392,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T13:08:47.743470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11403264,"num_objects":174,"num_object_clones":0,"num_object_copies":522,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":174,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1392,"num_write_kb":11136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"31'1696","reported_seq":1710,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-23T13:07:35.198492+0000","last_change":"2026-02-23T13:07:31.232991+0000","last_active":"2026-02-23T13:07:35.198492+0000","last_peered":"2026-02-23T13:07:35.198492+0000","last_clean":"2026-02-23T13:07:35.198492+0000","last_became_active":"2026-02-23T13:07:31.232761+0000","last_became_peered":"2026-02-23T13:07:31.232761+0000","last_unstale":"2026-02-23T13:07:35.198492+0000","last_undegraded":"2026-02-23T13:07:35.198492+0000","last_fullsized":"2026-02-23T13:07:35.198492+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_clean_scrub_stamp":"2026-02-23T13:07:30.224555+0000","objects_scrubbed":0,"log_size":1696,"log_dups_size":0,"ondisk_log_size":1696,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T13:08:39.392260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13893632,"num_objects":212,"num_object_clones":0,"num_object_copies":636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":212,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1696,"num_write_kb":13568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"31'1584","reported_seq":1591,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-23T13:07:35.219168+0000","last_change":"2026-02-23T13:07:31.233397+0000","last_active":"2026-02-23T13:07:35.219168+0000","last_peered":"2026-02-23T13:07:35.219168+0000","last_clean":"2026-02-23T13:07:35.219168+0000","last_became_active":"2026-02-23T13:07:31.232889+0000","last_became_peered":"2026-02-23T13:07:31.232889+0000","last_unstale":"2026-02-23T13:07:35.219168+0000","last_undegraded":"2026-02-23T13:07:35.219168+0000","last_fullsized":"2026-02-23T13:07:35.219168+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_clean_scrub_stamp":"2026-02-23T13:07:30.224555+0000","objects_scrubbed":0,"log_size":1584,"log_dups_size":0,"ondisk_log_size":1584,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T13:08:53.706564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12976128,"num_objects":198,"num_object_clones":0,"num_object_copies":594,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":198,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1584,"num_write_kb":12672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"31'1584","reported_seq":1598,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-23T13:07:35.218320+0000","last_change":"2026-02-23T13:07:31.233949+0000","last_active":"2026-02-23T13:07:35.218320+0000","last_peered":"2026-02-23T13:07:35.218320+0000","last_clean":"2026-02-23T13:07:35.218320+0000","last_became_active":"2026-02-23T13:07:31.233614+0000","last_became_peered":"2026-02-23T13:07:31.233614+0000","last_unstale":"2026-02-23T13:07:35.218320+0000","last_undegraded":"2026-02-23T13:07:35.218320+0000","last_fullsized":"2026-02-23T13:07:35.218320+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_clean_scrub_stamp":"2026-02-23T13:07:30.224555+0000","objects_scrubbed":0,"log_size":1584,"log_dups_size":0,"ondisk_log_size":1584,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T13:08:59.147129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12976128,"num_objects":198,"num_object_clones":0,"num_object_copies":594,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":198,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1584,"num_write_kb":12672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.8","version":"31'1688","reported_seq":1702,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-23T13:07:35.216222+0000","last_change":"2026-02-23T13:07:31.232555+0000","last_active":"2026-02-23T13:07:35.216222+0000","last_peered":"2026-02-23T13:07:35.216222+0000","last_clean":"2026-02-23T13:07:35.216222+0000","last_became_active":"2026-02-23T13:07:31.232268+0000","last_became_peered":"2026-02-23T13:07:31.232268+0000","last_unstale":"2026-02-23T13:07:35.216222+0000","last_undegraded":"2026-02-23T13:07:35.216222+0000","last_fullsized":"2026-02-23T13:07:35.216222+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_clean_scrub_stamp":"2026-02-23T13:07:30.224555+0000","objects_scrubbed":0,"log_size":1688,"log_dups_size":0,"ondisk_log_size":1688,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T13:08:52.518681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13828096,"num_objects":211,"num_object_clones":0,"num_object_copies":633,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":211,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1688,"num_write_kb":13504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"31'1624","reported_seq":1630,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-23T13:07:35.212095+0000","last_change":"2026-02-23T13:07:31.233167+0000","last_active":"2026-02-23T13:07:35.212095+0000","last_peered":"2026-02-23T13:07:35.212095+0000","last_clean":"2026-02-23T13:07:35.212095+0000","last_became_active":"2026-02-23T13:07:31.232674+0000","last_became_peered":"2026-02-23T13:07:31.232674+0000","last_unstale":"2026-02-23T13:07:35.212095+0000","last_undegraded":"2026-02-23T13:07:35.212095+0000","last_fullsized":"2026-02-23T13:07:35.212095+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_clean_scrub_stamp":"2026-02-23T13:07:30.224555+0000","objects_scrubbed":0,"log_size":1624,"log_dups_size":0,"ondisk_log_size":1624,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T13:08:51.864124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13303808,"num_objects":203,"num_object_clones":0,"num_object_copies":609,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":203,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1624,"num_write_kb":12992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.a","version":"31'1688","reported_seq":1686,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-23T13:07:35.212454+0000","last_change":"2026-02-23T13:07:31.233403+0000","last_active":"2026-02-23T13:07:35.212454+0000","last_peered":"2026-02-23T13:07:35.212454+0000","last_clean":"2026-02-23T13:07:35.212454+0000","last_became_active":"2026-02-23T13:07:31.233076+0000","last_became_peered":"2026-02-23T13:07:31.233076+0000","last_unstale":"2026-02-23T13:07:35.212454+0000","last_undegraded":"2026-02-23T13:07:35.212454+0000","last_fullsized":"2026-02-23T13:07:35.212454+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_clean_scrub_stamp":"2026-02-23T13:07:30.224555+0000","objects_scrubbed":0,"log_size":1688,"log_dups_size":0,"ondisk_log_size":1688,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T13:08:42.126151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13828096,"num_objects":211,"num_object_clones":0,"num_object_copies":633,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":211,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1688,"num_write_kb":13504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.b","version":"31'1584","reported_seq":1598,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-23T13:07:35.221819+0000","last_change":"2026-02-23T13:07:31.232618+0000","last_active":"2026-02-23T13:07:35.221819+0000","last_peered":"2026-02-23T13:07:35.221819+0000","last_clean":"2026-02-23T13:07:35.221819+0000","last_became_active":"2026-02-23T13:07:31.232339+0000","last_became_peered":"2026-02-23T13:07:31.232339+0000","last_unstale":"2026-02-23T13:07:35.221819+0000","last_undegraded":"2026-02-23T13:07:35.221819+0000","last_fullsized":"2026-02-23T13:07:35.221819+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_clean_scrub_stamp":"2026-02-23T13:07:30.224555+0000","objects_scrubbed":0,"log_size":1584,"log_dups_size":0,"ondisk_log_size":1584,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T13:08:51.536080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12976128,"num_objects":198,"num_object_clones":0,"num_object_copies":594,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":198,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1584,"num_write_kb":12672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":208928768,"num_objects":3189,"num_object_clones":0,"num_object_copies":9567,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3189,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":25504,"num_write_kb":204032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":626466816,"data_stored":626466816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":25504,"ondisk_log_size":25504,"up":48,"acting":48,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":240,"num_write":135,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":25,"seq":107374182406,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":734003200,"kb_used":156548,"kb_used_data":129584,"kb_used_omap":7,"kb_used_meta":26936,"kb_avail":733846652,"statfs":{"total":751619276800,"available":751458971648,"internally_reserved":0,"allocated":132694016,"data_stored":132511756,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8123,"internal_metadata":27582533},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,7,0,24,16],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411337,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":734003200,"kb_used":142988,"kb_used_data":116024,"kb_used_omap":8,"kb_used_meta":26935,"kb_avail":733860212,"statfs":{"total":751619276800,"available":751472857088,"internally_reserved":0,"allocated":118808576,"data_stored":118626316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8786,"internal_metadata":27581870},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[6,4],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574859,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":734003200,"kb_used":155128,"kb_used_data":128164,"kb_used_omap":11,"kb_used_meta":26932,"kb_avail":733848072,"statfs":{"total":751619276800,"available":751460425728,"internally_reserved":0,"allocated":131239936,"data_stored":131061228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12041,"internal_metadata":27578615},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[14,7,19,20],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705676,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":734003200,"kb_used":141452,"kb_used_data":114488,"kb_used_omap":15,"kb_used_meta":26928,"kb_avail":733861748,"statfs":{"total":751619276800,"available":751474429952,"internally_reserved":0,"allocated":117235712,"data_stored":117053452,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15947,"internal_metadata":27574709},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":141295616,"data_stored":141295616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":155238400,"data_stored":155238400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":144252928,"data_stored":144252928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":185679872,"data_stored":185679872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-23T13:07:40.322 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8 128 66617 66489 64.9259 57.9062 0.0264808 0.0153799 2026-02-23T13:07:40.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:40 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2026-02-23T13:07:40.632 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:07:40.777 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:07:40.913 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:07:40.911+0000 7f1c0d891640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:07:40.913 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:07:40.911+0000 7f1c0d891640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:07:40.929 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:07:40.930 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:07:40.944 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:07:40.944 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:07:40.960 INFO:teuthology.orchestra.run.trial068.stdout:osd.2: {} 2026-02-23T13:07:40.961 INFO:teuthology.orchestra.run.trial068.stderr:osd.2: osd_enable_op_tracker = '' 2026-02-23T13:07:40.974 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:07:40.975 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:07:41.322 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9 128 73748 73620 63.9015 55.7109 0.0128494 0.0156275 2026-02-23T13:07:41.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:41 trial068 ceph-mon[38805]: from='client.14357 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2026-02-23T13:07:41.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:41 trial068 ceph-mon[38805]: pgmap v14: 17 pgs: 17 active+clean; 200 MiB data, 582 MiB used, 2.7 TiB / 2.7 TiB avail; 22 MiB/s wr, 2.83k op/s 2026-02-23T13:07:41.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:41 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/763467828' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:07:42.322 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 10 128 80510 80382 62.7937 52.8281 0.0164898 0.0158998 2026-02-23T13:07:43.023 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:07:43.183 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:07:43.301 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:43 trial068 ceph-mon[38805]: pgmap v15: 17 pgs: 17 active+clean; 515 MiB data, 1.7 GiB used, 2.7 TiB / 2.7 TiB avail; 51 MiB/s wr, 6.58k op/s 2026-02-23T13:07:43.322 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 11 127 87025 86898 61.7127 50.9062 0.00733674 0.0161794 2026-02-23T13:07:43.623 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:07:43.578+0000 7ff8594c5640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:07:43.623 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:07:43.578+0000 7ff8594c5640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:07:43.623 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:07:43.623 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:07:43.623 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:07:43.624 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:07:43.628 INFO:teuthology.orchestra.run.trial068.stdout:osd.2: {} 2026-02-23T13:07:43.628 INFO:teuthology.orchestra.run.trial068.stderr:osd.2: osd_enable_op_tracker = '' 2026-02-23T13:07:43.645 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:07:43.645 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:07:44.195 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:07:43 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:07:43] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:07:44.323 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 12 128 91644 91516 59.5763 36.0781 0.0751856 0.0167519 2026-02-23T13:07:44.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:07:43 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:07:43.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:07:44.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:44 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1411400168' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:07:45.207 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2026-02-23T13:07:45.207 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2026-02-23T13:07:45.208 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph pg dump --format=json 2026-02-23T13:07:45.323 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 13 128 97651 97523 58.6032 46.9297 0.0149604 0.0170463 2026-02-23T13:07:45.362 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:07:45.500 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:45 trial068 ceph-mon[38805]: pgmap v16: 17 pgs: 17 active+clean; 515 MiB data, 1.7 GiB used, 2.7 TiB / 2.7 TiB avail; 43 MiB/s wr, 5.49k op/s 2026-02-23T13:07:45.780 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:07:45.734+0000 7f01388d0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:07:45.780 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:07:45.735+0000 7f01388d0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:07:45.883 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T13:07:45.884 INFO:teuthology.orchestra.run.trial068.stderr:dumped all 2026-02-23T13:07:45.934 INFO:teuthology.orchestra.run.trial068.stdout:{"pg_ready":true,"pg_map":{"version":16,"stamp":"2026-02-23T13:07:44.225153+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":539959840,"num_objects":8233,"num_object_clones":0,"num_object_copies":24699,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8233,"num_whiteouts":0,"num_read":136,"num_read_kb":240,"num_write":65976,"num_write_kb":528878,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":65940,"ondisk_log_size":65940,"up":51,"acting":51,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":51,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":2936012800,"kb_used":1745268,"kb_used_data":1474980,"kb_used_omap":43,"kb_used_meta":270164,"kb_avail":2934267532,"statfs":{"total":3006477107200,"available":3004689952768,"internally_reserved":0,"allocated":1510379520,"data_stored":1509654032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":44897,"internal_metadata":276648095},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[9,0,0,10,20,4],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":539369472,"num_objects":8231,"num_object_clones":0,"num_object_copies":24693,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8231,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":65841,"num_write_kb":526728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"11.995848"},"pg_stats":[{"pgid":"2.f","version":"31'4392","reported_seq":4406,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-23T13:07:40.202281+0000","last_change":"2026-02-23T13:07:31.232168+0000","last_active":"2026-02-23T13:07:40.202281+0000","last_peered":"2026-02-23T13:07:40.202281+0000","last_clean":"2026-02-23T13:07:40.202281+0000","last_became_active":"2026-02-23T13:07:31.231983+0000","last_became_peered":"2026-02-23T13:07:31.231983+0000","last_unstale":"2026-02-23T13:07:40.202281+0000","last_undegraded":"2026-02-23T13:07:40.202281+0000","last_fullsized":"2026-02-23T13:07:40.202281+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_clean_scrub_stamp":"2026-02-23T13:07:30.224555+0000","objects_scrubbed":0,"log_size":4392,"log_dups_size":0,"ondisk_log_size":4392,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T13:08:32.236459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35979264,"num_objects":549,"num_object_clones":0,"num_object_copies":1647,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":549,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4392,"num_write_kb":35136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.e","version":"31'3936","reported_seq":3950,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-23T13:07:40.225222+0000","last_change":"2026-02-23T13:07:31.233069+0000","last_active":"2026-02-23T13:07:40.225222+0000","last_peered":"2026-02-23T13:07:40.225222+0000","last_clean":"2026-02-23T13:07:40.225222+0000","last_became_active":"2026-02-23T13:07:31.232454+0000","last_became_peered":"2026-02-23T13:07:31.232454+0000","last_unstale":"2026-02-23T13:07:40.225222+0000","last_undegraded":"2026-02-23T13:07:40.225222+0000","last_fullsized":"2026-02-23T13:07:40.225222+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_clean_scrub_stamp":"2026-02-23T13:07:30.224555+0000","objects_scrubbed":0,"log_size":3936,"log_dups_size":0,"ondisk_log_size":3936,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T13:08:31.879194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32243712,"num_objects":492,"num_object_clones":0,"num_object_copies":1476,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":492,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3936,"num_write_kb":31488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.d","version":"31'3808","reported_seq":3822,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-23T13:07:40.218891+0000","last_change":"2026-02-23T13:07:31.233445+0000","last_active":"2026-02-23T13:07:40.218891+0000","last_peered":"2026-02-23T13:07:40.218891+0000","last_clean":"2026-02-23T13:07:40.218891+0000","last_became_active":"2026-02-23T13:07:31.232915+0000","last_became_peered":"2026-02-23T13:07:31.232915+0000","last_unstale":"2026-02-23T13:07:40.218891+0000","last_undegraded":"2026-02-23T13:07:40.218891+0000","last_fullsized":"2026-02-23T13:07:40.218891+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_clean_scrub_stamp":"2026-02-23T13:07:30.224555+0000","objects_scrubbed":0,"log_size":3808,"log_dups_size":0,"ondisk_log_size":3808,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T13:08:35.423973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31195136,"num_objects":476,"num_object_clones":0,"num_object_copies":1428,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":476,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3808,"num_write_kb":30464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"31'4064","reported_seq":4073,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-23T13:07:40.223907+0000","last_change":"2026-02-23T13:07:31.233497+0000","last_active":"2026-02-23T13:07:40.223907+0000","last_peered":"2026-02-23T13:07:40.223907+0000","last_clean":"2026-02-23T13:07:40.223907+0000","last_became_active":"2026-02-23T13:07:31.233152+0000","last_became_peered":"2026-02-23T13:07:31.233152+0000","last_unstale":"2026-02-23T13:07:40.223907+0000","last_undegraded":"2026-02-23T13:07:40.223907+0000","last_fullsized":"2026-02-23T13:07:40.223907+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_clean_scrub_stamp":"2026-02-23T13:07:30.224555+0000","objects_scrubbed":0,"log_size":4064,"log_dups_size":0,"ondisk_log_size":4064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T13:08:57.971970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33292288,"num_objects":508,"num_object_clones":0,"num_object_copies":1524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":508,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4064,"num_write_kb":32512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"28'99","reported_seq":51,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-23T13:07:32.230297+0000","last_change":"2026-02-23T13:07:14.224739+0000","last_active":"2026-02-23T13:07:32.230297+0000","last_peered":"2026-02-23T13:07:32.230297+0000","last_clean":"2026-02-23T13:07:32.230297+0000","last_became_active":"2026-02-23T13:07:14.220178+0000","last_became_peered":"2026-02-23T13:07:14.220178+0000","last_unstale":"2026-02-23T13:07:32.230297+0000","last_undegraded":"2026-02-23T13:07:32.230297+0000","last_fullsized":"2026-02-23T13:07:32.230297+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:07:05.619208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:07:05.619208+0000","last_clean_scrub_stamp":"2026-02-23T13:07:05.619208+0000","objects_scrubbed":0,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T13:08:06.145941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":240,"num_write":135,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"31'4152","reported_seq":4166,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-23T13:07:40.226694+0000","last_change":"2026-02-23T13:07:31.233851+0000","last_active":"2026-02-23T13:07:40.226694+0000","last_peered":"2026-02-23T13:07:40.226694+0000","last_clean":"2026-02-23T13:07:40.226694+0000","last_became_active":"2026-02-23T13:07:31.233470+0000","last_became_peered":"2026-02-23T13:07:31.233470+0000","last_unstale":"2026-02-23T13:07:40.226694+0000","last_undegraded":"2026-02-23T13:07:40.226694+0000","last_fullsized":"2026-02-23T13:07:40.226694+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_clean_scrub_stamp":"2026-02-23T13:07:30.224555+0000","objects_scrubbed":0,"log_size":4152,"log_dups_size":0,"ondisk_log_size":4152,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T13:08:34.869333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34013184,"num_objects":519,"num_object_clones":0,"num_object_copies":1557,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":519,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4152,"num_write_kb":33216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"31'4112","reported_seq":4122,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-23T13:07:40.230655+0000","last_change":"2026-02-23T13:07:31.232349+0000","last_active":"2026-02-23T13:07:40.230655+0000","last_peered":"2026-02-23T13:07:40.230655+0000","last_clean":"2026-02-23T13:07:40.230655+0000","last_became_active":"2026-02-23T13:07:31.232045+0000","last_became_peered":"2026-02-23T13:07:31.232045+0000","last_unstale":"2026-02-23T13:07:40.230655+0000","last_undegraded":"2026-02-23T13:07:40.230655+0000","last_fullsized":"2026-02-23T13:07:40.230655+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_clean_scrub_stamp":"2026-02-23T13:07:30.224555+0000","objects_scrubbed":0,"log_size":4112,"log_dups_size":0,"ondisk_log_size":4112,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T13:08:54.216320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33685504,"num_objects":514,"num_object_clones":0,"num_object_copies":1542,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":514,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4112,"num_write_kb":32896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"31'3904","reported_seq":3914,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-23T13:07:40.223507+0000","last_change":"2026-02-23T13:07:31.233117+0000","last_active":"2026-02-23T13:07:40.223507+0000","last_peered":"2026-02-23T13:07:40.223507+0000","last_clean":"2026-02-23T13:07:40.223507+0000","last_became_active":"2026-02-23T13:07:31.232562+0000","last_became_peered":"2026-02-23T13:07:31.232562+0000","last_unstale":"2026-02-23T13:07:40.223507+0000","last_undegraded":"2026-02-23T13:07:40.223507+0000","last_fullsized":"2026-02-23T13:07:40.223507+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_clean_scrub_stamp":"2026-02-23T13:07:30.224555+0000","objects_scrubbed":0,"log_size":3904,"log_dups_size":0,"ondisk_log_size":3904,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T13:08:51.508423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31981568,"num_objects":488,"num_object_clones":0,"num_object_copies":1464,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":488,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3904,"num_write_kb":31232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"31'4288","reported_seq":4302,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-23T13:07:40.225754+0000","last_change":"2026-02-23T13:07:31.234086+0000","last_active":"2026-02-23T13:07:40.225754+0000","last_peered":"2026-02-23T13:07:40.225754+0000","last_clean":"2026-02-23T13:07:40.225754+0000","last_became_active":"2026-02-23T13:07:31.233796+0000","last_became_peered":"2026-02-23T13:07:31.233796+0000","last_unstale":"2026-02-23T13:07:40.225754+0000","last_undegraded":"2026-02-23T13:07:40.225754+0000","last_fullsized":"2026-02-23T13:07:40.225754+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_clean_scrub_stamp":"2026-02-23T13:07:30.224555+0000","objects_scrubbed":0,"log_size":4288,"log_dups_size":0,"ondisk_log_size":4288,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T13:08:52.748856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35127296,"num_objects":536,"num_object_clones":0,"num_object_copies":1608,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":536,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4288,"num_write_kb":34304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"31'3952","reported_seq":3966,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-23T13:07:40.213772+0000","last_change":"2026-02-23T13:07:31.234133+0000","last_active":"2026-02-23T13:07:40.213772+0000","last_peered":"2026-02-23T13:07:40.213772+0000","last_clean":"2026-02-23T13:07:40.213772+0000","last_became_active":"2026-02-23T13:07:31.233956+0000","last_became_peered":"2026-02-23T13:07:31.233956+0000","last_unstale":"2026-02-23T13:07:40.213772+0000","last_undegraded":"2026-02-23T13:07:40.213772+0000","last_fullsized":"2026-02-23T13:07:40.213772+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_clean_scrub_stamp":"2026-02-23T13:07:30.224555+0000","objects_scrubbed":0,"log_size":3952,"log_dups_size":0,"ondisk_log_size":3952,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T13:08:47.743470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32374784,"num_objects":494,"num_object_clones":0,"num_object_copies":1482,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":494,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3952,"num_write_kb":31616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"31'4304","reported_seq":4318,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-23T13:07:40.227228+0000","last_change":"2026-02-23T13:07:31.232991+0000","last_active":"2026-02-23T13:07:40.227228+0000","last_peered":"2026-02-23T13:07:40.227228+0000","last_clean":"2026-02-23T13:07:40.227228+0000","last_became_active":"2026-02-23T13:07:31.232761+0000","last_became_peered":"2026-02-23T13:07:31.232761+0000","last_unstale":"2026-02-23T13:07:40.227228+0000","last_undegraded":"2026-02-23T13:07:40.227228+0000","last_fullsized":"2026-02-23T13:07:40.227228+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_clean_scrub_stamp":"2026-02-23T13:07:30.224555+0000","objects_scrubbed":0,"log_size":4304,"log_dups_size":0,"ondisk_log_size":4304,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T13:08:39.392260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35258368,"num_objects":538,"num_object_clones":0,"num_object_copies":1614,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":538,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4304,"num_write_kb":34432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"31'4024","reported_seq":4033,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-23T13:07:40.225458+0000","last_change":"2026-02-23T13:07:31.233397+0000","last_active":"2026-02-23T13:07:40.225458+0000","last_peered":"2026-02-23T13:07:40.225458+0000","last_clean":"2026-02-23T13:07:40.225458+0000","last_became_active":"2026-02-23T13:07:31.232889+0000","last_became_peered":"2026-02-23T13:07:31.232889+0000","last_unstale":"2026-02-23T13:07:40.225458+0000","last_undegraded":"2026-02-23T13:07:40.225458+0000","last_fullsized":"2026-02-23T13:07:40.225458+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_clean_scrub_stamp":"2026-02-23T13:07:30.224555+0000","objects_scrubbed":0,"log_size":4024,"log_dups_size":0,"ondisk_log_size":4024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T13:08:53.706564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32964608,"num_objects":503,"num_object_clones":0,"num_object_copies":1509,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4024,"num_write_kb":32192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"31'3902","reported_seq":3911,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-23T13:07:40.231119+0000","last_change":"2026-02-23T13:07:31.233949+0000","last_active":"2026-02-23T13:07:40.231119+0000","last_peered":"2026-02-23T13:07:40.231119+0000","last_clean":"2026-02-23T13:07:40.231119+0000","last_became_active":"2026-02-23T13:07:31.233614+0000","last_became_peered":"2026-02-23T13:07:31.233614+0000","last_unstale":"2026-02-23T13:07:40.231119+0000","last_undegraded":"2026-02-23T13:07:40.231119+0000","last_fullsized":"2026-02-23T13:07:40.231119+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_clean_scrub_stamp":"2026-02-23T13:07:30.224555+0000","objects_scrubbed":0,"log_size":3902,"log_dups_size":0,"ondisk_log_size":3902,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T13:08:59.147129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31965184,"num_objects":488,"num_object_clones":0,"num_object_copies":1464,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":488,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3902,"num_write_kb":31216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.8","version":"31'4334","reported_seq":4347,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-23T13:07:40.231513+0000","last_change":"2026-02-23T13:07:31.232555+0000","last_active":"2026-02-23T13:07:40.231513+0000","last_peered":"2026-02-23T13:07:40.231513+0000","last_clean":"2026-02-23T13:07:40.231513+0000","last_became_active":"2026-02-23T13:07:31.232268+0000","last_became_peered":"2026-02-23T13:07:31.232268+0000","last_unstale":"2026-02-23T13:07:40.231513+0000","last_undegraded":"2026-02-23T13:07:40.231513+0000","last_fullsized":"2026-02-23T13:07:40.231513+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_clean_scrub_stamp":"2026-02-23T13:07:30.224555+0000","objects_scrubbed":0,"log_size":4334,"log_dups_size":0,"ondisk_log_size":4334,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T13:08:52.518681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35504128,"num_objects":542,"num_object_clones":0,"num_object_copies":1626,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":542,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4334,"num_write_kb":34672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"31'4272","reported_seq":4286,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-23T13:07:40.211072+0000","last_change":"2026-02-23T13:07:31.233167+0000","last_active":"2026-02-23T13:07:40.211072+0000","last_peered":"2026-02-23T13:07:40.211072+0000","last_clean":"2026-02-23T13:07:40.211072+0000","last_became_active":"2026-02-23T13:07:31.232674+0000","last_became_peered":"2026-02-23T13:07:31.232674+0000","last_unstale":"2026-02-23T13:07:40.211072+0000","last_undegraded":"2026-02-23T13:07:40.211072+0000","last_fullsized":"2026-02-23T13:07:40.211072+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_clean_scrub_stamp":"2026-02-23T13:07:30.224555+0000","objects_scrubbed":0,"log_size":4272,"log_dups_size":0,"ondisk_log_size":4272,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T13:08:51.864124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34996224,"num_objects":534,"num_object_clones":0,"num_object_copies":1602,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":534,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4272,"num_write_kb":34176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.a","version":"31'4120","reported_seq":4134,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-23T13:07:40.194236+0000","last_change":"2026-02-23T13:07:31.233403+0000","last_active":"2026-02-23T13:07:40.194236+0000","last_peered":"2026-02-23T13:07:40.194236+0000","last_clean":"2026-02-23T13:07:40.194236+0000","last_became_active":"2026-02-23T13:07:31.233076+0000","last_became_peered":"2026-02-23T13:07:31.233076+0000","last_unstale":"2026-02-23T13:07:40.194236+0000","last_undegraded":"2026-02-23T13:07:40.194236+0000","last_fullsized":"2026-02-23T13:07:40.194236+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_clean_scrub_stamp":"2026-02-23T13:07:30.224555+0000","objects_scrubbed":0,"log_size":4120,"log_dups_size":0,"ondisk_log_size":4120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T13:08:42.126151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33751040,"num_objects":515,"num_object_clones":0,"num_object_copies":1545,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":515,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4120,"num_write_kb":32960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.b","version":"31'4277","reported_seq":4291,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-23T13:07:40.230108+0000","last_change":"2026-02-23T13:07:31.232618+0000","last_active":"2026-02-23T13:07:40.230108+0000","last_peered":"2026-02-23T13:07:40.230108+0000","last_clean":"2026-02-23T13:07:40.230108+0000","last_became_active":"2026-02-23T13:07:31.232339+0000","last_became_peered":"2026-02-23T13:07:31.232339+0000","last_unstale":"2026-02-23T13:07:40.230108+0000","last_undegraded":"2026-02-23T13:07:40.230108+0000","last_fullsized":"2026-02-23T13:07:40.230108+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_clean_scrub_stamp":"2026-02-23T13:07:30.224555+0000","objects_scrubbed":0,"log_size":4277,"log_dups_size":0,"ondisk_log_size":4277,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T13:08:51.536080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35037184,"num_objects":535,"num_object_clones":0,"num_object_copies":1605,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":535,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4277,"num_write_kb":34216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":539369472,"num_objects":8231,"num_object_clones":0,"num_object_copies":24693,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8231,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":65841,"num_write_kb":526728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1618255872,"data_stored":1618255872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":65841,"ondisk_log_size":65841,"up":48,"acting":48,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":240,"num_write":135,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":25,"seq":107374182407,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":734003200,"kb_used":501364,"kb_used_data":420256,"kb_used_omap":7,"kb_used_meta":81080,"kb_avail":733501836,"statfs":{"total":751619276800,"available":751105880064,"internally_reserved":0,"allocated":430342144,"data_stored":430159884,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8123,"internal_metadata":83025989},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411338,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":734003200,"kb_used":408492,"kb_used_data":345432,"kb_used_omap":8,"kb_used_meta":63031,"kb_avail":733594708,"statfs":{"total":751619276800,"available":751200980992,"internally_reserved":0,"allocated":353722368,"data_stored":353540108,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8786,"internal_metadata":64544174},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574860,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":734003200,"kb_used":428640,"kb_used_data":365580,"kb_used_omap":11,"kb_used_meta":63028,"kb_avail":733574560,"statfs":{"total":751619276800,"available":751180349440,"internally_reserved":0,"allocated":374353920,"data_stored":374175212,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12041,"internal_metadata":64540919},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,0,10,20,4],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705677,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":734003200,"kb_used":406772,"kb_used_data":343712,"kb_used_omap":15,"kb_used_meta":63024,"kb_avail":733596428,"statfs":{"total":751619276800,"available":751202742272,"internally_reserved":0,"allocated":351961088,"data_stored":351778828,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15947,"internal_metadata":64537013},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":371507200,"data_stored":371507200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":397615104,"data_stored":397615104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":374128640,"data_stored":374128640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":475004928,"data_stored":475004928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-23T13:07:46.197 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:07:46.323 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 14 127 102800 102673 57.2909 40.2344 0.0306545 0.0174254 2026-02-23T13:07:46.352 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:07:46.479 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:07:46.477+0000 7f43f0ab0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:07:46.480 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:07:46.478+0000 7f43f0ab0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:07:46.498 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:07:46.499 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:07:46.512 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:07:46.513 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:07:46.525 INFO:teuthology.orchestra.run.trial068.stdout:osd.2: {} 2026-02-23T13:07:46.525 INFO:teuthology.orchestra.run.trial068.stderr:osd.2: osd_enable_op_tracker = '' 2026-02-23T13:07:46.539 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:07:46.539 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:07:47.323 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 15 128 108347 108219 56.3598 43.3281 0.0618754 0.0177198 2026-02-23T13:07:47.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:47 trial068 ceph-mon[38805]: from='client.14363 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2026-02-23T13:07:47.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:47 trial068 ceph-mon[38805]: pgmap v17: 17 pgs: 17 active+clean; 758 MiB data, 2.5 GiB used, 2.7 TiB / 2.7 TiB avail; 63 MiB/s wr, 8.08k op/s 2026-02-23T13:07:47.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:47 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2346854990' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:07:48.323 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 16 127 113883 113756 55.5408 43.2578 0.0284198 0.0179815 2026-02-23T13:07:48.585 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:07:48.730 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:07:48.854 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:07:48.852+0000 7f139162e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:07:48.855 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:07:48.852+0000 7f139162e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:07:48.875 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:07:48.875 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:07:48.892 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:07:48.893 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:07:48.907 INFO:teuthology.orchestra.run.trial068.stdout:osd.2: {} 2026-02-23T13:07:48.907 INFO:teuthology.orchestra.run.trial068.stderr:osd.2: osd_enable_op_tracker = '' 2026-02-23T13:07:48.921 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:07:48.921 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:07:49.323 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 17 128 119147 119019 54.6921 41.1172 0.0439129 0.0182624 2026-02-23T13:07:49.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:49 trial068 ceph-mon[38805]: pgmap v18: 17 pgs: 17 active+clean; 758 MiB data, 2.5 GiB used, 2.7 TiB / 2.7 TiB avail; 47 MiB/s wr, 5.96k op/s 2026-02-23T13:07:49.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:49 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1032781268' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:07:50.323 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 18 128 124147 124019 53.8237 39.0625 0.00843718 0.0185612 2026-02-23T13:07:50.937 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2026-02-23T13:07:50.937 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2026-02-23T13:07:50.937 INFO:tasks.thrashosds.thrasher:Killing osd 3, live_osds are [0, 1, 2, 3] 2026-02-23T13:07:50.937 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2026-02-23T13:07:50.937 DEBUG:teuthology.orchestra.run.trial185:> sudo systemctl stop ceph-5f823084-10b8-11f1-be80-d404e6e7d460@osd.3 2026-02-23T13:07:50.969 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:07:51.121 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:07:51.239 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:07:51.238+0000 7f6b8a265640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:07:51.239 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:07:51.238+0000 7f6b8a265640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:07:51.256 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:07:51.257 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:07:51.273 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:07:51.273 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:07:51.284 INFO:teuthology.orchestra.run.trial068.stdout:osd.2: {} 2026-02-23T13:07:51.285 INFO:teuthology.orchestra.run.trial068.stderr:osd.2: osd_enable_op_tracker = '' 2026-02-23T13:07:51.323 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 19 128 128822 128694 52.913 36.5234 0.0566871 0.0188162 2026-02-23T13:07:51.347 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.3 2026-02-23T13:07:51.355 INFO:journalctl@ceph.osd.3.trial185.stdout:Feb 23 13:07:50 trial185 systemd[1]: Stopping Ceph osd.3 for 5f823084-10b8-11f1-be80-d404e6e7d460... 2026-02-23T13:07:51.355 INFO:journalctl@ceph.osd.3.trial185.stdout:Feb 23 13:07:51 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-3[54020]: 2026-02-23T13:07:51.060+0000 7fad6e6e5640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --osd-objectstore=bluestore (PID: 1) UID: 0 2026-02-23T13:07:51.355 INFO:journalctl@ceph.osd.3.trial185.stdout:Feb 23 13:07:51 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-3[54020]: 2026-02-23T13:07:51.060+0000 7fad6e6e5640 -1 osd.3 31 *** Got signal Terminated *** 2026-02-23T13:07:51.355 INFO:journalctl@ceph.osd.3.trial185.stdout:Feb 23 13:07:51 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-3[54020]: 2026-02-23T13:07:51.060+0000 7fad6e6e5640 -1 osd.3 31 *** Immediate shutdown (osd_fast_shutdown=true) *** 2026-02-23T13:07:51.378 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:07:51.378 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:07:51.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:51 trial068 ceph-mon[38805]: pgmap v19: 17 pgs: 17 active+clean; 758 MiB data, 2.5 GiB used, 2.7 TiB / 2.7 TiB avail; 47 MiB/s wr, 5.96k op/s 2026-02-23T13:07:51.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:51 trial068 ceph-mon[38805]: osd.3 marked itself down and dead 2026-02-23T13:07:51.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:51 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/782217289' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:07:51.606 INFO:journalctl@ceph.osd.3.trial185.stdout:Feb 23 13:07:51 trial185 podman[65938]: 2026-02-23 13:07:51.354002841 +0000 UTC m=+0.308575838 container died 5cbc736f6ef28e6d923a389d6b41f9e5492bb0e1b4383a634bc223c704d17923 (image=quay.ceph.io/ceph-ci/ceph@sha256:49621faaa845c39b2aae72e4fa24123ee9b2ac3fb8ae834753907068ee2972ea, name=ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-3, CEPH_GIT_REPO=https://github.com/ceph/ceph-ci.git, org.label-schema.license=GPLv2, org.opencontainers.image.documentation=https://docs.ceph.com/, org.label-schema.build-date=20260216, org.label-schema.vendor=CentOS, io.buildah.version=1.41.3, org.label-schema.schema-version=1.0, ceph=True, CEPH_REF=wip-shraddhaag-enable-cephadm-tests, CEPH_SHA1=e371433fcd1390a0b82cfb18effc422c1f5d8c1d, FROM_IMAGE=quay.io/centos/centos:stream9, OSD_FLAVOR=default, org.label-schema.name=CentOS Stream 9 Base Image, GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfsganesha-5/, org.opencontainers.image.authors=Ceph Release Team ) 2026-02-23T13:07:51.606 INFO:journalctl@ceph.osd.3.trial185.stdout:Feb 23 13:07:51 trial185 podman[65938]: 2026-02-23 13:07:51.363054476 +0000 UTC m=+0.317627473 container remove 5cbc736f6ef28e6d923a389d6b41f9e5492bb0e1b4383a634bc223c704d17923 (image=quay.ceph.io/ceph-ci/ceph@sha256:49621faaa845c39b2aae72e4fa24123ee9b2ac3fb8ae834753907068ee2972ea, name=ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-3, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfsganesha-5/, OSD_FLAVOR=default, org.label-schema.build-date=20260216, org.opencontainers.image.documentation=https://docs.ceph.com/, CEPH_GIT_REPO=https://github.com/ceph/ceph-ci.git, CEPH_REF=wip-shraddhaag-enable-cephadm-tests, CEPH_SHA1=e371433fcd1390a0b82cfb18effc422c1f5d8c1d, FROM_IMAGE=quay.io/centos/centos:stream9, org.opencontainers.image.authors=Ceph Release Team , ceph=True, io.buildah.version=1.41.3, org.label-schema.vendor=CentOS) 2026-02-23T13:07:51.606 INFO:journalctl@ceph.osd.3.trial185.stdout:Feb 23 13:07:51 trial185 bash[65938]: ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-3 2026-02-23T13:07:51.606 INFO:journalctl@ceph.osd.3.trial185.stdout:Feb 23 13:07:51 trial185 podman[66065]: 2026-02-23 13:07:51.489246063 +0000 UTC m=+0.016914190 container create e769c354b42568255f6eef5737bdc6e37f8b23a4c6594ba3b2ded1e904e29489 (image=quay.ceph.io/ceph-ci/ceph@sha256:49621faaa845c39b2aae72e4fa24123ee9b2ac3fb8ae834753907068ee2972ea, name=ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-3-deactivate, org.label-schema.vendor=CentOS, org.opencontainers.image.authors=Ceph Release Team , CEPH_REF=wip-shraddhaag-enable-cephadm-tests, org.label-schema.schema-version=1.0, FROM_IMAGE=quay.io/centos/centos:stream9, org.opencontainers.image.documentation=https://docs.ceph.com/, GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfsganesha-5/, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.41.3, CEPH_GIT_REPO=https://github.com/ceph/ceph-ci.git, ceph=True, OSD_FLAVOR=default, org.label-schema.build-date=20260216, org.label-schema.license=GPLv2, CEPH_SHA1=e371433fcd1390a0b82cfb18effc422c1f5d8c1d) 2026-02-23T13:07:51.606 INFO:journalctl@ceph.osd.3.trial185.stdout:Feb 23 13:07:51 trial185 podman[66065]: 2026-02-23 13:07:51.520527041 +0000 UTC m=+0.048195128 container init e769c354b42568255f6eef5737bdc6e37f8b23a4c6594ba3b2ded1e904e29489 (image=quay.ceph.io/ceph-ci/ceph@sha256:49621faaa845c39b2aae72e4fa24123ee9b2ac3fb8ae834753907068ee2972ea, name=ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-3-deactivate, CEPH_REF=wip-shraddhaag-enable-cephadm-tests, GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfsganesha-5/, FROM_IMAGE=quay.io/centos/centos:stream9, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_SHA1=e371433fcd1390a0b82cfb18effc422c1f5d8c1d, CEPH_GIT_REPO=https://github.com/ceph/ceph-ci.git, org.label-schema.schema-version=1.0, org.label-schema.build-date=20260216, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.opencontainers.image.authors=Ceph Release Team , ceph=True, OSD_FLAVOR=default, io.buildah.version=1.41.3, org.opencontainers.image.documentation=https://docs.ceph.com/) 2026-02-23T13:07:51.606 INFO:journalctl@ceph.osd.3.trial185.stdout:Feb 23 13:07:51 trial185 podman[66065]: 2026-02-23 13:07:51.522862305 +0000 UTC m=+0.050530372 container start e769c354b42568255f6eef5737bdc6e37f8b23a4c6594ba3b2ded1e904e29489 (image=quay.ceph.io/ceph-ci/ceph@sha256:49621faaa845c39b2aae72e4fa24123ee9b2ac3fb8ae834753907068ee2972ea, name=ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-3-deactivate, FROM_IMAGE=quay.io/centos/centos:stream9, ceph=True, CEPH_REF=wip-shraddhaag-enable-cephadm-tests, org.label-schema.license=GPLv2, io.buildah.version=1.41.3, org.label-schema.vendor=CentOS, org.opencontainers.image.documentation=https://docs.ceph.com/, CEPH_GIT_REPO=https://github.com/ceph/ceph-ci.git, org.label-schema.schema-version=1.0, GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfsganesha-5/, org.label-schema.build-date=20260216, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_SHA1=e371433fcd1390a0b82cfb18effc422c1f5d8c1d, org.opencontainers.image.authors=Ceph Release Team , OSD_FLAVOR=default) 2026-02-23T13:07:51.606 INFO:journalctl@ceph.osd.3.trial185.stdout:Feb 23 13:07:51 trial185 podman[66065]: 2026-02-23 13:07:51.523330279 +0000 UTC m=+0.050998356 container attach e769c354b42568255f6eef5737bdc6e37f8b23a4c6594ba3b2ded1e904e29489 (image=quay.ceph.io/ceph-ci/ceph@sha256:49621faaa845c39b2aae72e4fa24123ee9b2ac3fb8ae834753907068ee2972ea, name=ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-3-deactivate, CEPH_SHA1=e371433fcd1390a0b82cfb18effc422c1f5d8c1d, org.opencontainers.image.documentation=https://docs.ceph.com/, ceph=True, GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfsganesha-5/, OSD_FLAVOR=default, org.label-schema.build-date=20260216, org.label-schema.vendor=CentOS, CEPH_GIT_REPO=https://github.com/ceph/ceph-ci.git, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.opencontainers.image.authors=Ceph Release Team , CEPH_REF=wip-shraddhaag-enable-cephadm-tests, FROM_IMAGE=quay.io/centos/centos:stream9, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.41.3) 2026-02-23T13:07:51.606 INFO:journalctl@ceph.osd.3.trial185.stdout:Feb 23 13:07:51 trial185 podman[66065]: 2026-02-23 13:07:51.481625041 +0000 UTC m=+0.009293128 image pull 63df4b83db6f95a8ba965919b3671cb27173117c23b5c950aaadc8913f56213c quay.ceph.io/ceph-ci/ceph@sha256:49621faaa845c39b2aae72e4fa24123ee9b2ac3fb8ae834753907068ee2972ea 2026-02-23T13:07:51.686 DEBUG:teuthology.orchestra.run.trial185:> sudo pkill -f 'journalctl -f -n 0 -u ceph-5f823084-10b8-11f1-be80-d404e6e7d460@osd.3.service' 2026-02-23T13:07:51.734 DEBUG:teuthology.orchestra.run:got remote process result: None 2026-02-23T13:07:51.734 INFO:tasks.cephadm.osd.3:Stopped osd.3 2026-02-23T13:07:52.323 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T13:07:52.323138+0000 min lat: 0.000727387 max lat: 1.08556 avg lat: 0.0198298 2026-02-23T13:07:52.324 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T13:07:52.324 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 20 128 129124 128996 50.3853 2.35938 0.0153377 0.0198298 2026-02-23T13:07:52.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:52 trial068 ceph-mon[38805]: Health check failed: 1 osds down (OSD_DOWN) 2026-02-23T13:07:52.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:52 trial068 ceph-mon[38805]: osdmap e32: 4 total, 3 up, 4 in 2026-02-23T13:07:52.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:52 trial068 ceph-mon[38805]: osd.3 marked itself dead as of e32 2026-02-23T13:07:53.323 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 21 128 134361 134233 49.9341 40.9141 0.00802224 0.0199895 2026-02-23T13:07:53.379 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:07:53.527 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:07:53.553 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:53 trial068 ceph-mon[38805]: pgmap v21: 17 pgs: 8 stale+active+clean, 9 active+clean; 967 MiB data, 3.3 GiB used, 2.7 TiB / 2.7 TiB avail; 45 MiB/s wr, 5.78k op/s 2026-02-23T13:07:53.553 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:53 trial068 ceph-mon[38805]: osdmap e33: 4 total, 3 up, 4 in 2026-02-23T13:07:53.656 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:07:53.654+0000 7f223a97e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:07:53.656 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:07:53.655+0000 7f223a97e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:07:53.675 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:07:53.675 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:07:53.688 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:07:53.689 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:07:53.702 INFO:teuthology.orchestra.run.trial068.stdout:osd.2: {} 2026-02-23T13:07:53.702 INFO:teuthology.orchestra.run.trial068.stderr:osd.2: osd_enable_op_tracker = '' 2026-02-23T13:07:53.705 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.3 2026-02-23T13:07:53.752 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:07:53.752 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:07:54.253 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:07:53 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:07:53] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:07:54.323 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 22 128 139284 139156 49.4125 38.4609 0.0357339 0.0202225 2026-02-23T13:07:54.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:07:53 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:07:53.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:07:54.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:54 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2859639839' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:07:55.323 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 23 128 143926 143798 48.8407 36.2656 0.0220919 0.0204494 2026-02-23T13:07:55.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:55 trial068 ceph-mon[38805]: pgmap v23: 17 pgs: 8 stale+active+clean, 9 active+clean; 967 MiB data, 3.3 GiB used, 2.7 TiB / 2.7 TiB avail; 26 MiB/s wr, 3.33k op/s 2026-02-23T13:07:55.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:55 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T13:07:55.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:55 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T13:07:55.753 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:07:55.900 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:07:56.028 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:07:56.026+0000 7f1cd9392640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:07:56.028 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:07:56.027+0000 7f1cd9392640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:07:56.043 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:07:56.044 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:07:56.057 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:07:56.057 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:07:56.071 INFO:teuthology.orchestra.run.trial068.stdout:osd.2: {} 2026-02-23T13:07:56.071 INFO:teuthology.orchestra.run.trial068.stderr:osd.2: osd_enable_op_tracker = '' 2026-02-23T13:07:56.071 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.3 2026-02-23T13:07:56.323 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 24 127 148127 148000 48.1734 32.8281 0.0567636 0.0207373 2026-02-23T13:07:56.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:56 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2888985337' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:07:56.716 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:07:56.717 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:07:56.735 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [3] live_osds: [0, 1, 2] 2026-02-23T13:07:56.735 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2026-02-23T13:07:56.736 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph pg dump --format=json 2026-02-23T13:07:56.885 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:07:57.323 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 25 128 151290 151162 47.2346 24.7031 0.0114649 0.0211202 2026-02-23T13:07:57.379 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:07:57.329+0000 7fd4c2352640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:07:57.380 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:07:57.330+0000 7fd4c2352640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:07:57.483 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T13:07:57.484 INFO:teuthology.orchestra.run.trial068.stderr:dumped all 2026-02-23T13:07:57.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:57 trial068 ceph-mon[38805]: pgmap v24: 17 pgs: 15 active+undersized+degraded, 2 active+clean; 1.1 GiB data, 3.8 GiB used, 2.7 TiB / 2.7 TiB avail; 45 MiB/s wr, 5.81k op/s; 15776/53817 objects degraded (29.314%); 8.0 KiB/s, 0 objects/s recovering 2026-02-23T13:07:57.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:57 trial068 ceph-mon[38805]: Health check failed: Degraded data redundancy: 15776/53817 objects degraded (29.314%), 15 pgs degraded (PG_DEGRADED) 2026-02-23T13:07:57.957 INFO:teuthology.orchestra.run.trial068.stdout:{"pg_ready":true,"pg_map":{"version":24,"stamp":"2026-02-23T13:07:56.226668+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1175970336,"num_objects":17939,"num_object_clones":0,"num_object_copies":53817,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":15776,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17939,"num_whiteouts":0,"num_read":122,"num_read_kb":228,"num_write":143614,"num_write_kb":1149982,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":655904,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":143578,"ondisk_log_size":143578,"up":36,"acting":36,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":51,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":2936012800,"kb_used":3999020,"kb_used_data":3241436,"kb_used_omap":45,"kb_used_meta":757458,"kb_avail":2932013780,"statfs":{"total":3006477107200,"available":3002382110720,"internally_reserved":0,"allocated":3319230464,"data_stored":3318458995,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":46199,"internal_metadata":775637897},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[10,9,10,13,16,27,35,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":380968960,"num_objects":5814,"num_object_clones":0,"num_object_copies":17442,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":15776,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5814,"num_whiteouts":0,"num_read":-14,"num_read_kb":-12,"num_write":46505,"num_write_kb":372040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001037"},"pg_stats":[{"pgid":"2.f","version":"33'9400","reported_seq":9422,"reported_epoch":33,"state":"active+undersized+degraded","last_fresh":"2026-02-23T13:07:55.209563+0000","last_change":"2026-02-23T13:07:52.253191+0000","last_active":"2026-02-23T13:07:55.209563+0000","last_peered":"2026-02-23T13:07:55.209563+0000","last_clean":"2026-02-23T13:07:51.221395+0000","last_became_active":"2026-02-23T13:07:52.253191+0000","last_became_peered":"2026-02-23T13:07:52.253191+0000","last_unstale":"2026-02-23T13:07:55.209563+0000","last_undegraded":"2026-02-23T13:07:52.251048+0000","last_fullsized":"2026-02-23T13:07:52.250834+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_clean_scrub_stamp":"2026-02-23T13:07:30.224555+0000","objects_scrubbed":0,"log_size":9400,"log_dups_size":0,"ondisk_log_size":9400,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T13:08:30.945751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":77004800,"num_objects":1175,"num_object_clones":0,"num_object_copies":3525,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1175,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1175,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9400,"num_write_kb":75200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":1175}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.e","version":"33'8496","reported_seq":8514,"reported_epoch":33,"state":"active+clean","last_fresh":"2026-02-23T13:07:55.250033+0000","last_change":"2026-02-23T13:07:31.233069+0000","last_active":"2026-02-23T13:07:55.250033+0000","last_peered":"2026-02-23T13:07:55.250033+0000","last_clean":"2026-02-23T13:07:55.250033+0000","last_became_active":"2026-02-23T13:07:31.232454+0000","last_became_peered":"2026-02-23T13:07:31.232454+0000","last_unstale":"2026-02-23T13:07:55.250033+0000","last_undegraded":"2026-02-23T13:07:55.250033+0000","last_fullsized":"2026-02-23T13:07:55.250033+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_clean_scrub_stamp":"2026-02-23T13:07:30.224555+0000","objects_scrubbed":0,"log_size":8496,"log_dups_size":0,"ondisk_log_size":8496,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T13:08:31.879194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":69599232,"num_objects":1062,"num_object_clones":0,"num_object_copies":3186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1062,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8496,"num_write_kb":67968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.d","version":"33'8552","reported_seq":8574,"reported_epoch":33,"state":"active+undersized+degraded","last_fresh":"2026-02-23T13:07:55.188659+0000","last_change":"2026-02-23T13:07:52.253839+0000","last_active":"2026-02-23T13:07:55.188659+0000","last_peered":"2026-02-23T13:07:55.188659+0000","last_clean":"2026-02-23T13:07:51.244394+0000","last_became_active":"2026-02-23T13:07:52.253839+0000","last_became_peered":"2026-02-23T13:07:52.253839+0000","last_unstale":"2026-02-23T13:07:55.188659+0000","last_undegraded":"2026-02-23T13:07:52.252534+0000","last_fullsized":"2026-02-23T13:07:52.252427+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_clean_scrub_stamp":"2026-02-23T13:07:30.224555+0000","objects_scrubbed":0,"log_size":8552,"log_dups_size":0,"ondisk_log_size":8552,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T13:08:47.678054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70057984,"num_objects":1069,"num_object_clones":0,"num_object_copies":3207,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1069,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1069,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8552,"num_write_kb":68416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":["1","0"],"object_location_counts":[{"shards":"0,1","objects":1069}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"33'8810","reported_seq":8834,"reported_epoch":33,"state":"active+undersized+degraded","last_fresh":"2026-02-23T13:07:55.246812+0000","last_change":"2026-02-23T13:07:52.360832+0000","last_active":"2026-02-23T13:07:55.246812+0000","last_peered":"2026-02-23T13:07:55.246812+0000","last_clean":"2026-02-23T13:07:51.199255+0000","last_became_active":"2026-02-23T13:07:52.265942+0000","last_became_peered":"2026-02-23T13:07:52.265942+0000","last_unstale":"2026-02-23T13:07:55.246812+0000","last_undegraded":"2026-02-23T13:07:52.260096+0000","last_fullsized":"2026-02-23T13:07:52.252514+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_clean_scrub_stamp":"2026-02-23T13:07:30.224555+0000","objects_scrubbed":0,"log_size":8810,"log_dups_size":0,"ondisk_log_size":8810,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T13:08:45.268443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":72171520,"num_objects":1102,"num_object_clones":0,"num_object_copies":3306,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1102,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1102,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8810,"num_write_kb":70480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":16384,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":["1","0"],"object_location_counts":[{"shards":"0,1","objects":1102}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"28'99","reported_seq":52,"reported_epoch":33,"state":"active+undersized+degraded","last_fresh":"2026-02-23T13:07:52.253690+0000","last_change":"2026-02-23T13:07:52.253503+0000","last_active":"2026-02-23T13:07:52.253690+0000","last_peered":"2026-02-23T13:07:52.253690+0000","last_clean":"2026-02-23T13:07:26.430875+0000","last_became_active":"2026-02-23T13:07:52.253503+0000","last_became_peered":"2026-02-23T13:07:52.253503+0000","last_unstale":"2026-02-23T13:07:52.253690+0000","last_undegraded":"2026-02-23T13:07:52.252285+0000","last_fullsized":"2026-02-23T13:07:52.252181+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:07:05.619208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:07:05.619208+0000","last_clean_scrub_stamp":"2026-02-23T13:07:05.619208+0000","objects_scrubbed":0,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T13:08:21.130975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":122,"num_read_kb":228,"num_write":135,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,2","objects":2}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"33'9024","reported_seq":9044,"reported_epoch":33,"state":"active+undersized+degraded","last_fresh":"2026-02-23T13:07:55.209895+0000","last_change":"2026-02-23T13:07:52.255304+0000","last_active":"2026-02-23T13:07:55.209895+0000","last_peered":"2026-02-23T13:07:55.209895+0000","last_clean":"2026-02-23T13:07:51.221480+0000","last_became_active":"2026-02-23T13:07:52.255304+0000","last_became_peered":"2026-02-23T13:07:52.255304+0000","last_unstale":"2026-02-23T13:07:55.209895+0000","last_undegraded":"2026-02-23T13:07:52.250882+0000","last_fullsized":"2026-02-23T13:07:52.250686+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_clean_scrub_stamp":"2026-02-23T13:07:30.224555+0000","objects_scrubbed":0,"log_size":9024,"log_dups_size":0,"ondisk_log_size":9024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T13:08:58.555422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":73924608,"num_objects":1128,"num_object_clones":0,"num_object_copies":3384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1128,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1128,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9024,"num_write_kb":72192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1128}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"33'8800","reported_seq":8827,"reported_epoch":33,"state":"active+undersized+degraded","last_fresh":"2026-02-23T13:07:55.245524+0000","last_change":"2026-02-23T13:07:52.289976+0000","last_active":"2026-02-23T13:07:55.245524+0000","last_peered":"2026-02-23T13:07:55.245524+0000","last_clean":"2026-02-23T13:07:51.237052+0000","last_became_active":"2026-02-23T13:07:52.264947+0000","last_became_peered":"2026-02-23T13:07:52.264947+0000","last_unstale":"2026-02-23T13:07:55.245524+0000","last_undegraded":"2026-02-23T13:07:52.259791+0000","last_fullsized":"2026-02-23T13:07:52.252329+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_clean_scrub_stamp":"2026-02-23T13:07:30.224555+0000","objects_scrubbed":0,"log_size":8800,"log_dups_size":0,"ondisk_log_size":8800,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T13:08:59.602770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":72089600,"num_objects":1100,"num_object_clones":0,"num_object_copies":3300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1100,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1100,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8800,"num_write_kb":70400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":8192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":["1","0"],"object_location_counts":[{"shards":"0,1","objects":1100}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"33'8669","reported_seq":8690,"reported_epoch":33,"state":"active+undersized+degraded","last_fresh":"2026-02-23T13:07:55.255859+0000","last_change":"2026-02-23T13:07:52.260086+0000","last_active":"2026-02-23T13:07:55.255859+0000","last_peered":"2026-02-23T13:07:55.255859+0000","last_clean":"2026-02-23T13:07:51.227636+0000","last_became_active":"2026-02-23T13:07:52.260086+0000","last_became_peered":"2026-02-23T13:07:52.260086+0000","last_unstale":"2026-02-23T13:07:55.255859+0000","last_undegraded":"2026-02-23T13:07:52.252528+0000","last_fullsized":"2026-02-23T13:07:52.252391+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_clean_scrub_stamp":"2026-02-23T13:07:30.224555+0000","objects_scrubbed":0,"log_size":8669,"log_dups_size":0,"ondisk_log_size":8669,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T13:08:41.772214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71016448,"num_objects":1084,"num_object_clones":0,"num_object_copies":3252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1084,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1084,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8669,"num_write_kb":69352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":["1","0"],"object_location_counts":[{"shards":"0,1","objects":1084}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"33'8880","reported_seq":8898,"reported_epoch":33,"state":"active+undersized+degraded","last_fresh":"2026-02-23T13:07:55.230114+0000","last_change":"2026-02-23T13:07:52.308809+0000","last_active":"2026-02-23T13:07:55.230114+0000","last_peered":"2026-02-23T13:07:55.230114+0000","last_clean":"2026-02-23T13:07:51.242184+0000","last_became_active":"2026-02-23T13:07:52.255184+0000","last_became_peered":"2026-02-23T13:07:52.255184+0000","last_unstale":"2026-02-23T13:07:55.230114+0000","last_undegraded":"2026-02-23T13:07:52.253537+0000","last_fullsized":"2026-02-23T13:07:52.252254+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_clean_scrub_stamp":"2026-02-23T13:07:30.224555+0000","objects_scrubbed":0,"log_size":8880,"log_dups_size":0,"ondisk_log_size":8880,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T13:08:35.078806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":72744960,"num_objects":1110,"num_object_clones":0,"num_object_copies":3330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1110,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1110,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8880,"num_write_kb":71040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":16384,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[{"shards":"1,2","objects":1110}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"33'8808","reported_seq":8826,"reported_epoch":33,"state":"active+clean","last_fresh":"2026-02-23T13:07:55.254500+0000","last_change":"2026-02-23T13:07:31.234133+0000","last_active":"2026-02-23T13:07:55.254500+0000","last_peered":"2026-02-23T13:07:55.254500+0000","last_clean":"2026-02-23T13:07:55.254500+0000","last_became_active":"2026-02-23T13:07:31.233956+0000","last_became_peered":"2026-02-23T13:07:31.233956+0000","last_unstale":"2026-02-23T13:07:55.254500+0000","last_undegraded":"2026-02-23T13:07:55.254500+0000","last_fullsized":"2026-02-23T13:07:55.254500+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_clean_scrub_stamp":"2026-02-23T13:07:30.224555+0000","objects_scrubbed":0,"log_size":8808,"log_dups_size":0,"ondisk_log_size":8808,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T13:08:47.743470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":72155136,"num_objects":1101,"num_object_clones":0,"num_object_copies":3303,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1101,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8808,"num_write_kb":70464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"33'9088","reported_seq":9102,"reported_epoch":33,"state":"active+undersized+degraded","last_fresh":"2026-02-23T13:07:55.231140+0000","last_change":"2026-02-23T13:07:52.253652+0000","last_active":"2026-02-23T13:07:55.231140+0000","last_peered":"2026-02-23T13:07:55.231140+0000","last_clean":"2026-02-23T13:07:51.192350+0000","last_became_active":"2026-02-23T13:07:52.253652+0000","last_became_peered":"2026-02-23T13:07:52.253652+0000","last_unstale":"2026-02-23T13:07:55.231140+0000","last_undegraded":"2026-02-23T13:07:52.252283+0000","last_fullsized":"2026-02-23T13:07:52.252176+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_clean_scrub_stamp":"2026-02-23T13:07:30.224555+0000","objects_scrubbed":0,"log_size":9088,"log_dups_size":0,"ondisk_log_size":9088,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T13:08:46.067371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":74448896,"num_objects":1136,"num_object_clones":0,"num_object_copies":3408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1136,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9088,"num_write_kb":72704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,2","objects":1136}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"33'8846","reported_seq":8867,"reported_epoch":33,"state":"active+undersized+degraded","last_fresh":"2026-02-23T13:07:55.259492+0000","last_change":"2026-02-23T13:07:52.253985+0000","last_active":"2026-02-23T13:07:55.259492+0000","last_peered":"2026-02-23T13:07:55.259492+0000","last_clean":"2026-02-23T13:07:51.246400+0000","last_became_active":"2026-02-23T13:07:52.253985+0000","last_became_peered":"2026-02-23T13:07:52.253985+0000","last_unstale":"2026-02-23T13:07:55.259492+0000","last_undegraded":"2026-02-23T13:07:52.252432+0000","last_fullsized":"2026-02-23T13:07:52.252282+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_clean_scrub_stamp":"2026-02-23T13:07:30.224555+0000","objects_scrubbed":0,"log_size":8846,"log_dups_size":0,"ondisk_log_size":8846,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T13:08:57.222952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":72466432,"num_objects":1106,"num_object_clones":0,"num_object_copies":3318,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1106,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1106,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8846,"num_write_kb":70768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[{"shards":"1,2","objects":1106}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"33'8880","reported_seq":8899,"reported_epoch":33,"state":"active+undersized+degraded","last_fresh":"2026-02-23T13:07:55.142824+0000","last_change":"2026-02-23T13:07:52.253162+0000","last_active":"2026-02-23T13:07:55.142824+0000","last_peered":"2026-02-23T13:07:55.142824+0000","last_clean":"2026-02-23T13:07:51.204436+0000","last_became_active":"2026-02-23T13:07:52.253162+0000","last_became_peered":"2026-02-23T13:07:52.253162+0000","last_unstale":"2026-02-23T13:07:55.142824+0000","last_undegraded":"2026-02-23T13:07:52.250997+0000","last_fullsized":"2026-02-23T13:07:52.250814+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_clean_scrub_stamp":"2026-02-23T13:07:30.224555+0000","objects_scrubbed":0,"log_size":8880,"log_dups_size":0,"ondisk_log_size":8880,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T13:08:51.786706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":72744960,"num_objects":1110,"num_object_clones":0,"num_object_copies":3330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1110,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1110,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8880,"num_write_kb":71040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1110}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.8","version":"33'9016","reported_seq":9033,"reported_epoch":33,"state":"active+undersized+degraded","last_fresh":"2026-02-23T13:07:55.178412+0000","last_change":"2026-02-23T13:07:52.253159+0000","last_active":"2026-02-23T13:07:55.178412+0000","last_peered":"2026-02-23T13:07:55.178412+0000","last_clean":"2026-02-23T13:07:51.224200+0000","last_became_active":"2026-02-23T13:07:52.253159+0000","last_became_peered":"2026-02-23T13:07:52.253159+0000","last_unstale":"2026-02-23T13:07:55.178412+0000","last_undegraded":"2026-02-23T13:07:52.250861+0000","last_fullsized":"2026-02-23T13:07:52.250627+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_clean_scrub_stamp":"2026-02-23T13:07:30.224555+0000","objects_scrubbed":0,"log_size":9016,"log_dups_size":0,"ondisk_log_size":9016,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T13:08:57.489199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":73859072,"num_objects":1127,"num_object_clones":0,"num_object_copies":3381,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1127,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1127,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9016,"num_write_kb":72128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":1127}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.9","version":"33'9584","reported_seq":9609,"reported_epoch":33,"state":"active+undersized+degraded","last_fresh":"2026-02-23T13:07:55.247196+0000","last_change":"2026-02-23T13:07:52.352369+0000","last_active":"2026-02-23T13:07:55.247196+0000","last_peered":"2026-02-23T13:07:55.247196+0000","last_clean":"2026-02-23T13:07:51.245909+0000","last_became_active":"2026-02-23T13:07:52.265546+0000","last_became_peered":"2026-02-23T13:07:52.265546+0000","last_unstale":"2026-02-23T13:07:55.247196+0000","last_undegraded":"2026-02-23T13:07:52.259920+0000","last_fullsized":"2026-02-23T13:07:52.252451+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_clean_scrub_stamp":"2026-02-23T13:07:30.224555+0000","objects_scrubbed":0,"log_size":9584,"log_dups_size":0,"ondisk_log_size":9584,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T13:08:34.112462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":78512128,"num_objects":1198,"num_object_clones":0,"num_object_copies":3594,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1198,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1198,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9584,"num_write_kb":76672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":16384,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":["1","0"],"object_location_counts":[{"shards":"0,1","objects":1198}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.a","version":"33'9248","reported_seq":9270,"reported_epoch":33,"state":"active+undersized+degraded","last_fresh":"2026-02-23T13:07:55.246522+0000","last_change":"2026-02-23T13:07:52.254267+0000","last_active":"2026-02-23T13:07:55.246522+0000","last_peered":"2026-02-23T13:07:55.246522+0000","last_clean":"2026-02-23T13:07:51.233066+0000","last_became_active":"2026-02-23T13:07:52.254267+0000","last_became_peered":"2026-02-23T13:07:52.254267+0000","last_unstale":"2026-02-23T13:07:55.246522+0000","last_undegraded":"2026-02-23T13:07:52.252322+0000","last_fullsized":"2026-02-23T13:07:52.252236+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_clean_scrub_stamp":"2026-02-23T13:07:30.224555+0000","objects_scrubbed":0,"log_size":9248,"log_dups_size":0,"ondisk_log_size":9248,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T13:08:54.287035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":75759616,"num_objects":1156,"num_object_clones":0,"num_object_copies":3468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1156,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1156,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9248,"num_write_kb":73984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[{"shards":"1,2","objects":1156}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.b","version":"33'9378","reported_seq":9400,"reported_epoch":33,"state":"active+undersized+degraded","last_fresh":"2026-02-23T13:07:55.233251+0000","last_change":"2026-02-23T13:07:52.326877+0000","last_active":"2026-02-23T13:07:55.233251+0000","last_peered":"2026-02-23T13:07:55.233251+0000","last_clean":"2026-02-23T13:07:51.245808+0000","last_became_active":"2026-02-23T13:07:52.260618+0000","last_became_peered":"2026-02-23T13:07:52.260618+0000","last_unstale":"2026-02-23T13:07:55.233251+0000","last_undegraded":"2026-02-23T13:07:52.254941+0000","last_fullsized":"2026-02-23T13:07:52.250715+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_clean_scrub_stamp":"2026-02-23T13:07:30.224555+0000","objects_scrubbed":0,"log_size":9378,"log_dups_size":0,"ondisk_log_size":9378,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T13:08:35.138884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76824576,"num_objects":1173,"num_object_clones":0,"num_object_copies":3519,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1173,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1173,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9378,"num_write_kb":75024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":8192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":1173}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":1175379968,"num_objects":17937,"num_object_clones":0,"num_object_copies":53811,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":15774,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17937,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":143479,"num_write_kb":1147832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3381600256,"data_stored":3381600256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":143479,"ondisk_log_size":143479,"up":34,"acting":34,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":122,"num_read_kb":228,"num_write":135,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":25,"seq":107374182409,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":734003200,"kb_used":1029604,"kb_used_data":840208,"kb_used_omap":7,"kb_used_meta":189368,"kb_avail":732973596,"statfs":{"total":751619276800,"available":750564962304,"internally_reserved":0,"allocated":860372992,"data_stored":860190732,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8123,"internal_metadata":193912901},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411341,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":734003200,"kb_used":973444,"kb_used_data":784048,"kb_used_omap":8,"kb_used_meta":189367,"kb_avail":733029756,"statfs":{"total":751619276800,"available":750622470144,"internally_reserved":0,"allocated":802865152,"data_stored":802667565,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8786,"internal_metadata":193912238},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574863,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":734003200,"kb_used":1027832,"kb_used_data":838436,"kb_used_omap":11,"kb_used_meta":189364,"kb_avail":732975368,"statfs":{"total":751619276800,"available":750566776832,"internally_reserved":0,"allocated":858558464,"data_stored":858364429,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12041,"internal_metadata":193908983},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,7,6,10,14,27,35,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705680,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":734003200,"kb_used":968140,"kb_used_data":778744,"kb_used_omap":16,"kb_used_meta":189359,"kb_avail":733035060,"statfs":{"total":751619276800,"available":750627901440,"internally_reserved":0,"allocated":797433856,"data_stored":797236269,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17249,"internal_metadata":193903775},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[7,2,4,3,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":807264256,"data_stored":807264256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":873431040,"data_stored":873431040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":811278336,"data_stored":811278336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":889626624,"data_stored":889626624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-23T13:07:58.324 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 26 128 154852 154724 46.4881 27.8281 0.0142017 0.0214772 2026-02-23T13:07:58.718 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:07:58.859 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:07:58.985 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:07:58.983+0000 7f4c152e6640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:07:58.986 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:07:58.984+0000 7f4c152e6640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:07:59.001 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:07:59.001 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:07:59.016 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:07:59.017 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:07:59.030 INFO:teuthology.orchestra.run.trial068.stdout:osd.2: {} 2026-02-23T13:07:59.030 INFO:teuthology.orchestra.run.trial068.stderr:osd.2: osd_enable_op_tracker = '' 2026-02-23T13:07:59.030 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.3 2026-02-23T13:07:59.070 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:07:59.071 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:07:59.324 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 27 128 158841 158713 45.9204 31.1641 0.0343535 0.0217454 2026-02-23T13:07:59.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:59 trial068 ceph-mon[38805]: from='client.14375 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2026-02-23T13:07:59.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:59 trial068 ceph-mon[38805]: pgmap v25: 17 pgs: 15 active+undersized+degraded, 2 active+clean; 1.1 GiB data, 3.8 GiB used, 2.7 TiB / 2.7 TiB avail; 45 MiB/s wr, 5.81k op/s; 15776/53817 objects degraded (29.314%); 8.0 KiB/s, 0 objects/s recovering 2026-02-23T13:07:59.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:07:59 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/184180265' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:08:00.324 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 28 128 162744 162616 45.3694 30.4922 0.0101812 0.0220179 2026-02-23T13:08:01.072 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:08:01.219 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:08:01.324 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 29 128 166737 166609 44.8805 31.1953 0.0186235 0.0222613 2026-02-23T13:08:01.342 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:08:01.340+0000 7f3e1d4c1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:08:01.342 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:08:01.341+0000 7f3e1d4c1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:08:01.361 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:08:01 trial068 ceph-mon[38805]: pgmap v26: 17 pgs: 15 active+undersized+degraded, 2 active+clean; 1.1 GiB data, 3.8 GiB used, 2.7 TiB / 2.7 TiB avail; 17 MiB/s wr, 2.21k op/s; 15776/53817 objects degraded (29.314%); 7.1 KiB/s, 0 objects/s recovering 2026-02-23T13:08:01.361 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:08:01 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T13:08:01.361 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:08:01 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T13:08:01.362 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:08:01.362 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:08:01.377 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:08:01.377 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:08:01.391 INFO:teuthology.orchestra.run.trial068.stdout:osd.2: {} 2026-02-23T13:08:01.391 INFO:teuthology.orchestra.run.trial068.stderr:osd.2: osd_enable_op_tracker = '' 2026-02-23T13:08:01.391 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.3 2026-02-23T13:08:01.432 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:08:01.433 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:08:02.323 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 30 128 170907 170779 44.4703 32.5781 0.0309592 0.0224728 2026-02-23T13:08:02.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:08:02 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1676035385' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:08:02.960 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [3] live_osds: [0, 1, 2] 2026-02-23T13:08:02.960 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2026-02-23T13:08:02.960 INFO:tasks.thrashosds.thrasher:Reviving osd 3 2026-02-23T13:08:02.960 INFO:tasks.cephadm.osd.3:Restarting osd.3 (starting--it wasn't running)... 2026-02-23T13:08:02.961 DEBUG:teuthology.orchestra.run.trial185:osd.3> sudo journalctl -f -n 0 -u ceph-5f823084-10b8-11f1-be80-d404e6e7d460@osd.3.service 2026-02-23T13:08:02.963 DEBUG:teuthology.orchestra.run.trial185:> sudo systemctl start ceph-5f823084-10b8-11f1-be80-d404e6e7d460@osd.3 2026-02-23T13:08:03.264 INFO:journalctl@ceph.osd.3.trial185.stdout:Feb 23 13:08:03 trial185 systemd[1]: Starting Ceph osd.3 for 5f823084-10b8-11f1-be80-d404e6e7d460... 2026-02-23T13:08:03.265 INFO:journalctl@ceph.osd.3.trial185.stdout:Feb 23 13:08:03 trial185 podman[66250]: 2026-02-23 13:08:03.12042805 +0000 UTC m=+0.010409184 image pull 63df4b83db6f95a8ba965919b3671cb27173117c23b5c950aaadc8913f56213c quay.ceph.io/ceph-ci/ceph@sha256:49621faaa845c39b2aae72e4fa24123ee9b2ac3fb8ae834753907068ee2972ea 2026-02-23T13:08:03.265 INFO:journalctl@ceph.osd.3.trial185.stdout:Feb 23 13:08:03 trial185 podman[66250]: 2026-02-23 13:08:03.263478723 +0000 UTC m=+0.153459847 container create 7ef31bcdd83f4412a0b91453a9d8aea2fa045f28e508a69ca781067655edf510 (image=quay.ceph.io/ceph-ci/ceph@sha256:49621faaa845c39b2aae72e4fa24123ee9b2ac3fb8ae834753907068ee2972ea, name=ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-3-activate, ceph=True, org.label-schema.license=GPLv2, FROM_IMAGE=quay.io/centos/centos:stream9, CEPH_SHA1=e371433fcd1390a0b82cfb18effc422c1f5d8c1d, org.label-schema.schema-version=1.0, CEPH_GIT_REPO=https://github.com/ceph/ceph-ci.git, org.opencontainers.image.documentation=https://docs.ceph.com/, org.label-schema.name=CentOS Stream 9 Base Image, OSD_FLAVOR=default, org.label-schema.vendor=CentOS, io.buildah.version=1.41.3, GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfsganesha-5/, org.label-schema.build-date=20260216, org.opencontainers.image.authors=Ceph Release Team , CEPH_REF=wip-shraddhaag-enable-cephadm-tests) 2026-02-23T13:08:03.324 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 31 128 174854 174726 44.0305 30.8359 0.0341752 0.0226842 2026-02-23T13:08:03.434 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:08:03.591 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:08:03.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:08:03 trial068 ceph-mon[38805]: pgmap v27: 17 pgs: 15 active+undersized+degraded, 2 active+clean; 1.2 GiB data, 4.2 GiB used, 2.7 TiB / 2.7 TiB avail; 30 MiB/s wr, 3.88k op/s; 17820/60915 objects degraded (29.254%); 6.4 KiB/s, 0 objects/s recovering 2026-02-23T13:08:03.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:08:03 trial068 ceph-mon[38805]: Health check update: Degraded data redundancy: 17820/60915 objects degraded (29.254%), 15 pgs degraded (PG_DEGRADED) 2026-02-23T13:08:03.606 INFO:journalctl@ceph.osd.3.trial185.stdout:Feb 23 13:08:03 trial185 podman[66250]: 2026-02-23 13:08:03.357369586 +0000 UTC m=+0.247350710 container init 7ef31bcdd83f4412a0b91453a9d8aea2fa045f28e508a69ca781067655edf510 (image=quay.ceph.io/ceph-ci/ceph@sha256:49621faaa845c39b2aae72e4fa24123ee9b2ac3fb8ae834753907068ee2972ea, name=ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-3-activate, FROM_IMAGE=quay.io/centos/centos:stream9, GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfsganesha-5/, CEPH_REF=wip-shraddhaag-enable-cephadm-tests, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20260216, CEPH_SHA1=e371433fcd1390a0b82cfb18effc422c1f5d8c1d, CEPH_GIT_REPO=https://github.com/ceph/ceph-ci.git, org.opencontainers.image.documentation=https://docs.ceph.com/, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, org.opencontainers.image.authors=Ceph Release Team , io.buildah.version=1.41.3, OSD_FLAVOR=default) 2026-02-23T13:08:03.606 INFO:journalctl@ceph.osd.3.trial185.stdout:Feb 23 13:08:03 trial185 podman[66250]: 2026-02-23 13:08:03.359937424 +0000 UTC m=+0.249918558 container start 7ef31bcdd83f4412a0b91453a9d8aea2fa045f28e508a69ca781067655edf510 (image=quay.ceph.io/ceph-ci/ceph@sha256:49621faaa845c39b2aae72e4fa24123ee9b2ac3fb8ae834753907068ee2972ea, name=ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-3-activate, OSD_FLAVOR=default, org.opencontainers.image.authors=Ceph Release Team , org.label-schema.build-date=20260216, CEPH_REF=wip-shraddhaag-enable-cephadm-tests, org.label-schema.vendor=CentOS, ceph=True, FROM_IMAGE=quay.io/centos/centos:stream9, org.label-schema.schema-version=1.0, CEPH_GIT_REPO=https://github.com/ceph/ceph-ci.git, org.opencontainers.image.documentation=https://docs.ceph.com/, GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfsganesha-5/, CEPH_SHA1=e371433fcd1390a0b82cfb18effc422c1f5d8c1d, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.41.3, org.label-schema.license=GPLv2) 2026-02-23T13:08:03.606 INFO:journalctl@ceph.osd.3.trial185.stdout:Feb 23 13:08:03 trial185 podman[66250]: 2026-02-23 13:08:03.360416119 +0000 UTC m=+0.250397423 container attach 7ef31bcdd83f4412a0b91453a9d8aea2fa045f28e508a69ca781067655edf510 (image=quay.ceph.io/ceph-ci/ceph@sha256:49621faaa845c39b2aae72e4fa24123ee9b2ac3fb8ae834753907068ee2972ea, name=ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-3-activate, org.opencontainers.image.documentation=https://docs.ceph.com/, FROM_IMAGE=quay.io/centos/centos:stream9, CEPH_REF=wip-shraddhaag-enable-cephadm-tests, CEPH_GIT_REPO=https://github.com/ceph/ceph-ci.git, io.buildah.version=1.41.3, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20260216, OSD_FLAVOR=default, CEPH_SHA1=e371433fcd1390a0b82cfb18effc422c1f5d8c1d, GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfsganesha-5/, org.opencontainers.image.authors=Ceph Release Team ) 2026-02-23T13:08:03.606 INFO:journalctl@ceph.osd.3.trial185.stdout:Feb 23 13:08:03 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-3-activate[66275]: Running command: /usr/bin/ceph-authtool --gen-print-key 2026-02-23T13:08:03.606 INFO:journalctl@ceph.osd.3.trial185.stdout:Feb 23 13:08:03 trial185 bash[66250]: Running command: /usr/bin/ceph-authtool --gen-print-key 2026-02-23T13:08:03.606 INFO:journalctl@ceph.osd.3.trial185.stdout:Feb 23 13:08:03 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-3-activate[66275]: Running command: /usr/bin/ceph-authtool --gen-print-key 2026-02-23T13:08:03.606 INFO:journalctl@ceph.osd.3.trial185.stdout:Feb 23 13:08:03 trial185 bash[66250]: Running command: /usr/bin/ceph-authtool --gen-print-key 2026-02-23T13:08:03.727 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:08:03.725+0000 7fe624a70640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:08:03.728 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:08:03.726+0000 7fe624a70640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:08:03.742 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:08:03.743 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:08:03.755 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:08:03.755 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:08:03.767 INFO:teuthology.orchestra.run.trial068.stdout:osd.2: {} 2026-02-23T13:08:03.767 INFO:teuthology.orchestra.run.trial068.stderr:osd.2: osd_enable_op_tracker = '' 2026-02-23T13:08:03.767 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.3 2026-02-23T13:08:03.809 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:08:03.810 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:08:04.106 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:08:03 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:08:03.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:08:04.107 INFO:journalctl@ceph.osd.3.trial185.stdout:Feb 23 13:08:03 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-3-activate[66275]: --> Failed to activate via raw: did not find any matching OSD to activate 2026-02-23T13:08:04.107 INFO:journalctl@ceph.osd.3.trial185.stdout:Feb 23 13:08:03 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-3-activate[66275]: Running command: /usr/bin/ceph-authtool --gen-print-key 2026-02-23T13:08:04.107 INFO:journalctl@ceph.osd.3.trial185.stdout:Feb 23 13:08:03 trial185 bash[66250]: --> Failed to activate via raw: did not find any matching OSD to activate 2026-02-23T13:08:04.107 INFO:journalctl@ceph.osd.3.trial185.stdout:Feb 23 13:08:03 trial185 bash[66250]: Running command: /usr/bin/ceph-authtool --gen-print-key 2026-02-23T13:08:04.107 INFO:journalctl@ceph.osd.3.trial185.stdout:Feb 23 13:08:03 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-3-activate[66275]: Running command: /usr/bin/ceph-authtool --gen-print-key 2026-02-23T13:08:04.108 INFO:journalctl@ceph.osd.3.trial185.stdout:Feb 23 13:08:03 trial185 bash[66250]: Running command: /usr/bin/ceph-authtool --gen-print-key 2026-02-23T13:08:04.108 INFO:journalctl@ceph.osd.3.trial185.stdout:Feb 23 13:08:03 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-3-activate[66275]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2026-02-23T13:08:04.108 INFO:journalctl@ceph.osd.3.trial185.stdout:Feb 23 13:08:03 trial185 bash[66250]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2026-02-23T13:08:04.108 INFO:journalctl@ceph.osd.3.trial185.stdout:Feb 23 13:08:03 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-3-activate[66275]: Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/vg_nvme/lv_3 --path /var/lib/ceph/osd/ceph-3 --no-mon-config 2026-02-23T13:08:04.108 INFO:journalctl@ceph.osd.3.trial185.stdout:Feb 23 13:08:03 trial185 bash[66250]: Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/vg_nvme/lv_3 --path /var/lib/ceph/osd/ceph-3 --no-mon-config 2026-02-23T13:08:04.108 INFO:journalctl@ceph.osd.3.trial185.stdout:Feb 23 13:08:03 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-3-activate[66275]: Running command: /usr/bin/ln -snf /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-3/block 2026-02-23T13:08:04.108 INFO:journalctl@ceph.osd.3.trial185.stdout:Feb 23 13:08:03 trial185 bash[66250]: Running command: /usr/bin/ln -snf /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-3/block 2026-02-23T13:08:04.108 INFO:journalctl@ceph.osd.3.trial185.stdout:Feb 23 13:08:03 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-3-activate[66275]: Running command: /usr/bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-3/block 2026-02-23T13:08:04.108 INFO:journalctl@ceph.osd.3.trial185.stdout:Feb 23 13:08:03 trial185 bash[66250]: Running command: /usr/bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-3/block 2026-02-23T13:08:04.108 INFO:journalctl@ceph.osd.3.trial185.stdout:Feb 23 13:08:03 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-3-activate[66275]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2026-02-23T13:08:04.108 INFO:journalctl@ceph.osd.3.trial185.stdout:Feb 23 13:08:03 trial185 bash[66250]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2026-02-23T13:08:04.108 INFO:journalctl@ceph.osd.3.trial185.stdout:Feb 23 13:08:03 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-3-activate[66275]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2026-02-23T13:08:04.109 INFO:journalctl@ceph.osd.3.trial185.stdout:Feb 23 13:08:03 trial185 bash[66250]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2026-02-23T13:08:04.109 INFO:journalctl@ceph.osd.3.trial185.stdout:Feb 23 13:08:03 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-3-activate[66275]: --> ceph-volume lvm activate successful for osd ID: 3 2026-02-23T13:08:04.109 INFO:journalctl@ceph.osd.3.trial185.stdout:Feb 23 13:08:03 trial185 bash[66250]: --> ceph-volume lvm activate successful for osd ID: 3 2026-02-23T13:08:04.109 INFO:journalctl@ceph.osd.3.trial185.stdout:Feb 23 13:08:03 trial185 podman[66250]: 2026-02-23 13:08:03.991703949 +0000 UTC m=+0.881685073 container died 7ef31bcdd83f4412a0b91453a9d8aea2fa045f28e508a69ca781067655edf510 (image=quay.ceph.io/ceph-ci/ceph@sha256:49621faaa845c39b2aae72e4fa24123ee9b2ac3fb8ae834753907068ee2972ea, name=ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-3-activate, io.buildah.version=1.41.3, org.label-schema.license=GPLv2, CEPH_SHA1=e371433fcd1390a0b82cfb18effc422c1f5d8c1d, org.label-schema.schema-version=1.0, CEPH_REF=wip-shraddhaag-enable-cephadm-tests, CEPH_GIT_REPO=https://github.com/ceph/ceph-ci.git, org.opencontainers.image.authors=Ceph Release Team , org.label-schema.name=CentOS Stream 9 Base Image, FROM_IMAGE=quay.io/centos/centos:stream9, OSD_FLAVOR=default, GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfsganesha-5/, org.label-schema.vendor=CentOS, org.opencontainers.image.documentation=https://docs.ceph.com/, ceph=True, org.label-schema.build-date=20260216) 2026-02-23T13:08:04.109 INFO:journalctl@ceph.osd.3.trial185.stdout:Feb 23 13:08:04 trial185 podman[66250]: 2026-02-23 13:08:04.003049482 +0000 UTC m=+0.893030616 container remove 7ef31bcdd83f4412a0b91453a9d8aea2fa045f28e508a69ca781067655edf510 (image=quay.ceph.io/ceph-ci/ceph@sha256:49621faaa845c39b2aae72e4fa24123ee9b2ac3fb8ae834753907068ee2972ea, name=ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-3-activate, CEPH_REF=wip-shraddhaag-enable-cephadm-tests, CEPH_SHA1=e371433fcd1390a0b82cfb18effc422c1f5d8c1d, OSD_FLAVOR=default, org.opencontainers.image.documentation=https://docs.ceph.com/, org.label-schema.license=GPLv2, GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfsganesha-5/, org.label-schema.build-date=20260216, CEPH_GIT_REPO=https://github.com/ceph/ceph-ci.git, io.buildah.version=1.41.3, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, FROM_IMAGE=quay.io/centos/centos:stream9, org.opencontainers.image.authors=Ceph Release Team ) 2026-02-23T13:08:04.109 INFO:journalctl@ceph.osd.3.trial185.stdout:Feb 23 13:08:04 trial185 podman[66486]: 2026-02-23 13:08:04.086020388 +0000 UTC m=+0.016804545 container create b401ff5c1392a94c8c70aa1388589c699a0bc3093dc05c21eb0d8a391dcc36ca (image=quay.ceph.io/ceph-ci/ceph@sha256:49621faaa845c39b2aae72e4fa24123ee9b2ac3fb8ae834753907068ee2972ea, name=ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-3, OSD_FLAVOR=default, ceph=True, CEPH_SHA1=e371433fcd1390a0b82cfb18effc422c1f5d8c1d, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_REF=wip-shraddhaag-enable-cephadm-tests, FROM_IMAGE=quay.io/centos/centos:stream9, org.label-schema.build-date=20260216, org.label-schema.license=GPLv2, CEPH_GIT_REPO=https://github.com/ceph/ceph-ci.git, GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfsganesha-5/, org.label-schema.schema-version=1.0, org.opencontainers.image.authors=Ceph Release Team , org.opencontainers.image.documentation=https://docs.ceph.com/, org.label-schema.vendor=CentOS, io.buildah.version=1.41.3) 2026-02-23T13:08:04.134 DEBUG:teuthology.orchestra.run.trial185:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph daemon osd.3 dump_ops_in_flight 2026-02-23T13:08:04.260 INFO:teuthology.orchestra.run.trial185.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/config/ceph.conf 2026-02-23T13:08:04.263 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:08:03 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:08:03] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:08:04.324 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 32 128 178956 178828 43.6559 32.0469 0.0324575 0.0228838 2026-02-23T13:08:04.374 INFO:journalctl@ceph.osd.3.trial185.stdout:Feb 23 13:08:04 trial185 podman[66486]: 2026-02-23 13:08:04.122893125 +0000 UTC m=+0.053677232 container init b401ff5c1392a94c8c70aa1388589c699a0bc3093dc05c21eb0d8a391dcc36ca (image=quay.ceph.io/ceph-ci/ceph@sha256:49621faaa845c39b2aae72e4fa24123ee9b2ac3fb8ae834753907068ee2972ea, name=ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-3, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, CEPH_GIT_REPO=https://github.com/ceph/ceph-ci.git, OSD_FLAVOR=default, CEPH_REF=wip-shraddhaag-enable-cephadm-tests, ceph=True, FROM_IMAGE=quay.io/centos/centos:stream9, org.label-schema.license=GPLv2, org.opencontainers.image.authors=Ceph Release Team , GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfsganesha-5/, org.label-schema.build-date=20260216, CEPH_SHA1=e371433fcd1390a0b82cfb18effc422c1f5d8c1d, org.label-schema.vendor=CentOS, io.buildah.version=1.41.3, org.opencontainers.image.documentation=https://docs.ceph.com/) 2026-02-23T13:08:04.375 INFO:journalctl@ceph.osd.3.trial185.stdout:Feb 23 13:08:04 trial185 podman[66486]: 2026-02-23 13:08:04.126350437 +0000 UTC m=+0.057134534 container start b401ff5c1392a94c8c70aa1388589c699a0bc3093dc05c21eb0d8a391dcc36ca (image=quay.ceph.io/ceph-ci/ceph@sha256:49621faaa845c39b2aae72e4fa24123ee9b2ac3fb8ae834753907068ee2972ea, name=ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-3, GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfsganesha-5/, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, io.buildah.version=1.41.3, org.opencontainers.image.documentation=https://docs.ceph.com/, CEPH_SHA1=e371433fcd1390a0b82cfb18effc422c1f5d8c1d, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.build-date=20260216, org.opencontainers.image.authors=Ceph Release Team , OSD_FLAVOR=default, CEPH_REF=wip-shraddhaag-enable-cephadm-tests, FROM_IMAGE=quay.io/centos/centos:stream9, CEPH_GIT_REPO=https://github.com/ceph/ceph-ci.git) 2026-02-23T13:08:04.375 INFO:journalctl@ceph.osd.3.trial185.stdout:Feb 23 13:08:04 trial185 bash[66486]: b401ff5c1392a94c8c70aa1388589c699a0bc3093dc05c21eb0d8a391dcc36ca 2026-02-23T13:08:04.375 INFO:journalctl@ceph.osd.3.trial185.stdout:Feb 23 13:08:04 trial185 podman[66486]: 2026-02-23 13:08:04.079048238 +0000 UTC m=+0.009832345 image pull 63df4b83db6f95a8ba965919b3671cb27173117c23b5c950aaadc8913f56213c quay.ceph.io/ceph-ci/ceph@sha256:49621faaa845c39b2aae72e4fa24123ee9b2ac3fb8ae834753907068ee2972ea 2026-02-23T13:08:04.375 INFO:journalctl@ceph.osd.3.trial185.stdout:Feb 23 13:08:04 trial185 systemd[1]: Started Ceph osd.3 for 5f823084-10b8-11f1-be80-d404e6e7d460. 2026-02-23T13:08:04.375 INFO:journalctl@ceph.osd.3.trial185.stdout:Feb 23 13:08:04 trial185 ceph-osd[66510]: -- 10.20.193.185:0/459895029 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x40a7c00 msgr2=0x4ee4000 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).process 2026-02-23T13:08:04.375 INFO:journalctl@ceph.osd.3.trial185.stdout:Feb 23 13:08:04 trial185 ceph-osd[66510]: -- 10.20.193.185:0/459895029 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x40a7c00 msgr2=0x4ee4000 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read continue len=32 2026-02-23T13:08:04.375 INFO:journalctl@ceph.osd.3.trial185.stdout:Feb 23 13:08:04 trial185 ceph-osd[66510]: --2- 10.20.193.185:0/459895029 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x40a7c00 0x4ee4000 crc :-1 s=READY pgs=290 gs=1 cs=0 l=1 c_cookie=8779626044555146159 s_cookie=0 reconnecting=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main r=0 2026-02-23T13:08:04.375 INFO:journalctl@ceph.osd.3.trial185.stdout:Feb 23 13:08:04 trial185 ceph-osd[66510]: --2- 10.20.193.185:0/459895029 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x40a7c00 0x4ee4000 crc :-1 s=THROTTLE_MESSAGE pgs=290 gs=1 cs=0 l=1 c_cookie=8779626044555146159 s_cookie=0 reconnecting=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).throttle_message 2026-02-23T13:08:04.376 INFO:journalctl@ceph.osd.3.trial185.stdout:Feb 23 13:08:04 trial185 ceph-osd[66510]: --2- 10.20.193.185:0/459895029 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x40a7c00 0x4ee4000 crc :-1 s=THROTTLE_BYTES pgs=290 gs=1 cs=0 l=1 c_cookie=8779626044555146159 s_cookie=0 reconnecting=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).throttle_bytes 2026-02-23T13:08:04.376 INFO:journalctl@ceph.osd.3.trial185.stdout:Feb 23 13:08:04 trial185 ceph-osd[66510]: --2- 10.20.193.185:0/459895029 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x40a7c00 0x4ee4000 crc :-1 s=THROTTLE_DISPATCH_QUEUE pgs=290 gs=1 cs=0 l=1 c_cookie=8779626044555146159 s_cookie=0 reconnecting=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).throttle_dispatch_queue 2026-02-23T13:08:04.376 INFO:journalctl@ceph.osd.3.trial185.stdout:Feb 23 13:08:04 trial185 ceph-osd[66510]: --2- 10.20.193.185:0/459895029 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x40a7c00 0x4ee4000 crc :-1 s=THROTTLE_DONE pgs=290 gs=1 cs=0 l=1 c_cookie=8779626044555146159 s_cookie=0 reconnecting=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).read_frame_segment seg_idx=0 2026-02-23T13:08:04.376 INFO:journalctl@ceph.osd.3.trial185.stdout:Feb 23 13:08:04 trial185 ceph-osd[66510]: -- 10.20.193.185:0/459895029 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x40a7c00 msgr2=0x4ee4000 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read start len=45 2026-02-23T13:08:04.376 INFO:journalctl@ceph.osd.3.trial185.stdout:Feb 23 13:08:04 trial185 ceph-osd[66510]: --2- 10.20.193.185:0/459895029 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x40a7c00 0x4ee4000 crc :-1 s=THROTTLE_DONE pgs=290 gs=1 cs=0 l=1 c_cookie=8779626044555146159 s_cookie=0 reconnecting=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_segment r=0 2026-02-23T13:08:04.376 INFO:journalctl@ceph.osd.3.trial185.stdout:Feb 23 13:08:04 trial185 ceph-osd[66510]: --2- 10.20.193.185:0/459895029 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x40a7c00 0x4ee4000 crc :-1 s=THROTTLE_DONE pgs=290 gs=1 cs=0 l=1 c_cookie=8779626044555146159 s_cookie=0 reconnecting=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).read_frame_segment seg_idx=1 2026-02-23T13:08:04.376 INFO:journalctl@ceph.osd.3.trial185.stdout:Feb 23 13:08:04 trial185 ceph-osd[66510]: -- 10.20.193.185:0/459895029 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x40a7c00 msgr2=0x4ee4000 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read start len=194 2026-02-23T13:08:04.376 INFO:journalctl@ceph.osd.3.trial185.stdout:Feb 23 13:08:04 trial185 ceph-osd[66510]: --2- 10.20.193.185:0/459895029 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x40a7c00 0x4ee4000 crc :-1 s=THROTTLE_DONE pgs=290 gs=1 cs=0 l=1 c_cookie=8779626044555146159 s_cookie=0 reconnecting=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_segment r=0 2026-02-23T13:08:04.376 INFO:journalctl@ceph.osd.3.trial185.stdout:Feb 23 13:08:04 trial185 ceph-osd[66510]: -- 10.20.193.185:0/459895029 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x40a7c00 msgr2=0x4ee4000 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read start len=13 2026-02-23T13:08:04.376 INFO:journalctl@ceph.osd.3.trial185.stdout:Feb 23 13:08:04 trial185 ceph-osd[66510]: --2- 10.20.193.185:0/459895029 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x40a7c00 0x4ee4000 crc :-1 s=THROTTLE_DONE pgs=290 gs=1 cs=0 l=1 c_cookie=8779626044555146159 s_cookie=0 reconnecting=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_epilogue_main r=0 2026-02-23T13:08:04.376 INFO:journalctl@ceph.osd.3.trial185.stdout:Feb 23 13:08:04 trial185 ceph-osd[66510]: --2- 10.20.193.185:0/459895029 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x40a7c00 0x4ee4000 crc :-1 s=THROTTLE_DONE pgs=290 gs=1 cs=0 l=1 c_cookie=8779626044555146159 s_cookie=0 reconnecting=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_dispatch tag=17 2026-02-23T13:08:04.377 INFO:journalctl@ceph.osd.3.trial185.stdout:Feb 23 13:08:04 trial185 ceph-osd[66510]: --2- 10.20.193.185:0/459895029 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x40a7c00 0x4ee4000 crc :-1 s=THROTTLE_DONE pgs=290 gs=1 cs=0 l=1 c_cookie=8779626044555146159 s_cookie=0 reconnecting=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_message 2026-02-23T13:08:04.377 INFO:journalctl@ceph.osd.3.trial185.stdout:Feb 23 13:08:04 trial185 ceph-osd[66510]: --2- 10.20.193.185:0/459895029 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x40a7c00 0x4ee4000 crc :-1 s=THROTTLE_DONE pgs=290 gs=1 cs=0 l=1 c_cookie=8779626044555146159 s_cookie=0 reconnecting=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_message got 194 + 0 + 0 byte message. envelope type=18 src mon.0 off 0 2026-02-23T13:08:04.377 INFO:journalctl@ceph.osd.3.trial185.stdout:Feb 23 13:08:04 trial185 ceph-osd[66510]: --2- 10.20.193.185:0/459895029 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x40a7c00 0x4ee4000 crc :-1 s=READ_MESSAGE_COMPLETE pgs=290 gs=1 cs=0 l=1 c_cookie=8779626044555146159 s_cookie=0 reconnecting=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_message received message m=0x410b860 seq=4 from=mon.0 type=18 auth_reply(proto 2 0 (0) Success) 2026-02-23T13:08:04.377 INFO:journalctl@ceph.osd.3.trial185.stdout:Feb 23 13:08:04 trial185 ceph-osd[66510]: -- 10.20.193.185:0/459895029 queue 0x410b860 prio 196 2026-02-23T13:08:04.377 INFO:journalctl@ceph.osd.3.trial185.stdout:Feb 23 13:08:04 trial185 ceph-osd[66510]: --2- 10.20.193.185:0/459895029 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x40a7c00 0x4ee4000 crc :-1 s=READY pgs=290 gs=1 cs=0 l=1 c_cookie=8779626044555146159 s_cookie=0 reconnecting=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).read_frame 2026-02-23T13:08:04.377 INFO:journalctl@ceph.osd.3.trial185.stdout:Feb 23 13:08:04 trial185 ceph-osd[66510]: -- 10.20.193.185:0/459895029 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x40a7c00 msgr2=0x4ee4000 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read start len=32 2026-02-23T13:08:04.377 INFO:journalctl@ceph.osd.3.trial185.stdout:Feb 23 13:08:04 trial185 ceph-osd[66510]: -- 10.20.193.185:0/459895029 <== mon.0 v2:10.20.193.68:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (crc 0 0 0) 0x410b860 con 0x40a7c00 2026-02-23T13:08:04.377 INFO:journalctl@ceph.osd.3.trial185.stdout:Feb 23 13:08:04 trial185 ceph-osd[66510]: -- 10.20.193.185:0/459895029 dispatch_throttle_release 194 to dispatch throttler 194/104857600 2026-02-23T13:08:04.377 INFO:journalctl@ceph.osd.3.trial185.stdout:Feb 23 13:08:04 trial185 ceph-osd[66510]: -- 10.20.193.185:0/459895029 done calling dispatch on 0x410b860 2026-02-23T13:08:04.377 INFO:journalctl@ceph.osd.3.trial185.stdout:Feb 23 13:08:04 trial185 ceph-osd[66510]: WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:08:04.377 INFO:journalctl@ceph.osd.3.trial185.stdout:Feb 23 13:08:04 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-3[66506]: 2026-02-23T13:08:04.140+0000 7f99d8ba9780 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:08:04.377 INFO:journalctl@ceph.osd.3.trial185.stdout:Feb 23 13:08:04 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-3[66506]: 2026-02-23T13:08:04.140+0000 7f99d8ba9780 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:08:04.378 INFO:journalctl@ceph.osd.3.trial185.stdout:Feb 23 13:08:04 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-3[66506]: 2026-02-23T13:08:04.140+0000 7f99d8ba9780 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:08:04.386 INFO:teuthology.orchestra.run.trial185.stderr:no valid command found; 10 closest matches: 2026-02-23T13:08:04.386 INFO:teuthology.orchestra.run.trial185.stderr:0 2026-02-23T13:08:04.386 INFO:teuthology.orchestra.run.trial185.stderr:1 2026-02-23T13:08:04.386 INFO:teuthology.orchestra.run.trial185.stderr:2 2026-02-23T13:08:04.386 INFO:teuthology.orchestra.run.trial185.stderr:abort 2026-02-23T13:08:04.386 INFO:teuthology.orchestra.run.trial185.stderr:assert 2026-02-23T13:08:04.386 INFO:teuthology.orchestra.run.trial185.stderr:bluestore collections 2026-02-23T13:08:04.386 INFO:teuthology.orchestra.run.trial185.stderr:bluestore compression stats [] 2026-02-23T13:08:04.386 INFO:teuthology.orchestra.run.trial185.stderr:bluestore list [] [] 2026-02-23T13:08:04.386 INFO:teuthology.orchestra.run.trial185.stderr:bluestore onode metadata 2026-02-23T13:08:04.387 INFO:teuthology.orchestra.run.trial185.stderr:bluestore show sharding 2026-02-23T13:08:04.387 INFO:teuthology.orchestra.run.trial185.stderr:admin_socket: invalid command 2026-02-23T13:08:04.420 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-23T13:08:04.421 INFO:tasks.cephadm.ceph_manager.ceph:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2026-02-23T13:08:04.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:08:04 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2475549199' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:08:04.856 INFO:journalctl@ceph.osd.3.trial185.stdout:Feb 23 13:08:04 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-3[66506]: 2026-02-23T13:08:04.384+0000 7f99d8ba9780 -1 Falling back to public interface 2026-02-23T13:08:05.324 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 33 128 182913 182785 43.2697 30.9141 0.0432438 0.0230943 2026-02-23T13:08:05.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:08:05 trial068 ceph-mon[38805]: pgmap v28: 17 pgs: 15 active+undersized+degraded, 2 active+clean; 1.2 GiB data, 4.2 GiB used, 2.7 TiB / 2.7 TiB avail; 25 MiB/s wr, 3.24k op/s; 17820/60915 objects degraded (29.254%); 5.3 KiB/s, 0 objects/s recovering 2026-02-23T13:08:05.811 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:08:05.962 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:08:06.083 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:08:06.081+0000 7f220c966640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:08:06.083 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:08:06.082+0000 7f220c966640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:08:06.099 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:08:06.100 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:08:06.115 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:08:06.115 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:08:06.130 INFO:teuthology.orchestra.run.trial068.stdout:osd.2: {} 2026-02-23T13:08:06.131 INFO:teuthology.orchestra.run.trial068.stderr:osd.2: osd_enable_op_tracker = '' 2026-02-23T13:08:06.131 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.3 2026-02-23T13:08:06.172 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:08:06.172 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:08:06.324 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 34 128 186974 186846 42.9301 31.7266 0.0296723 0.0232778 2026-02-23T13:08:06.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:08:06 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3433726228' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:08:07.324 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 35 127 191098 190971 42.6242 32.2266 0.0251072 0.0234458 2026-02-23T13:08:07.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:08:07 trial068 ceph-mon[38805]: pgmap v29: 17 pgs: 15 active+undersized+degraded, 2 active+clean; 1.4 GiB data, 4.6 GiB used, 2.7 TiB / 2.7 TiB avail; 38 MiB/s wr, 4.91k op/s; 20032/68493 objects degraded (29.247%); 5.3 KiB/s, 0 objects/s recovering 2026-02-23T13:08:08.173 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:08:08.323 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:08:08.324 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 36 128 194901 194773 42.2652 29.7031 0.0120193 0.0236419 2026-02-23T13:08:08.452 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:08:08.449+0000 7f55a1a8f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:08:08.452 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:08:08.450+0000 7f55a1a8f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:08:08.467 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:08:08.467 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:08:08.480 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:08:08.480 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:08:08.493 INFO:teuthology.orchestra.run.trial068.stdout:osd.2: {} 2026-02-23T13:08:08.493 INFO:teuthology.orchestra.run.trial068.stderr:osd.2: osd_enable_op_tracker = '' 2026-02-23T13:08:08.494 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.3 2026-02-23T13:08:08.533 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:08:08.534 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:08:09.324 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 37 128 198812 198684 41.9487 30.5547 0.06348 0.0238257 2026-02-23T13:08:09.421 DEBUG:teuthology.orchestra.run.trial185:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph daemon osd.3 dump_ops_in_flight 2026-02-23T13:08:09.551 INFO:teuthology.orchestra.run.trial185.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/config/ceph.conf 2026-02-23T13:08:09.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:08:09 trial068 ceph-mon[38805]: pgmap v30: 17 pgs: 15 active+undersized+degraded, 2 active+clean; 1.4 GiB data, 4.6 GiB used, 2.7 TiB / 2.7 TiB avail; 25 MiB/s wr, 3.26k op/s; 20032/68493 objects degraded (29.247%) 2026-02-23T13:08:09.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:08:09 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1107819206' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:08:09.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:08:09 trial068 ceph-mon[38805]: Health check update: Degraded data redundancy: 20032/68493 objects degraded (29.247%), 15 pgs degraded (PG_DEGRADED) 2026-02-23T13:08:09.687 INFO:teuthology.orchestra.run.trial185.stderr:no valid command found; 10 closest matches: 2026-02-23T13:08:09.687 INFO:teuthology.orchestra.run.trial185.stderr:0 2026-02-23T13:08:09.687 INFO:teuthology.orchestra.run.trial185.stderr:1 2026-02-23T13:08:09.687 INFO:teuthology.orchestra.run.trial185.stderr:2 2026-02-23T13:08:09.687 INFO:teuthology.orchestra.run.trial185.stderr:abort 2026-02-23T13:08:09.687 INFO:teuthology.orchestra.run.trial185.stderr:assert 2026-02-23T13:08:09.687 INFO:teuthology.orchestra.run.trial185.stderr:bluefs debug_inject_read_zeros 2026-02-23T13:08:09.687 INFO:teuthology.orchestra.run.trial185.stderr:bluefs files list 2026-02-23T13:08:09.687 INFO:teuthology.orchestra.run.trial185.stderr:bluefs stats 2026-02-23T13:08:09.687 INFO:teuthology.orchestra.run.trial185.stderr:bluestore allocator dump block 2026-02-23T13:08:09.687 INFO:teuthology.orchestra.run.trial185.stderr:bluestore allocator fragmentation block 2026-02-23T13:08:09.687 INFO:teuthology.orchestra.run.trial185.stderr:admin_socket: invalid command 2026-02-23T13:08:09.719 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-23T13:08:09.719 INFO:tasks.cephadm.ceph_manager.ceph:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2026-02-23T13:08:10.274 INFO:journalctl@ceph.osd.3.trial185.stdout:Feb 23 13:08:09 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-3[66506]: 2026-02-23T13:08:09.954+0000 7f99d8ba9780 -1 osd.3 32 log_to_monitors true 2026-02-23T13:08:10.274 INFO:journalctl@ceph.osd.3.trial185.stdout:Feb 23 13:08:10 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-3[66506]: 2026-02-23T13:08:10.272+0000 7f99d09f2640 -1 osd.3 32 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2026-02-23T13:08:10.325 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 38 128 202723 202595 41.6488 30.5547 0.00422232 0.0239883 2026-02-23T13:08:10.536 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:08:10.560 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:08:10 trial068 ceph-mon[38805]: from='osd.3 [v2:10.20.193.185:6808/1590361891,v1:10.20.193.185:6809/1590361891]' entity='osd.3' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]} : dispatch 2026-02-23T13:08:10.692 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:08:10.815 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:08:10.814+0000 7f171e2b3640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:08:10.815 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:08:10.814+0000 7f171e2b3640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:08:10.831 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:08:10.831 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:08:10.846 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:08:10.846 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:08:10.860 INFO:teuthology.orchestra.run.trial068.stdout:osd.2: {} 2026-02-23T13:08:10.860 INFO:teuthology.orchestra.run.trial068.stderr:osd.2: osd_enable_op_tracker = '' 2026-02-23T13:08:10.860 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.3 2026-02-23T13:08:10.897 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:08:10.897 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:08:11.324 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 39 128 206255 206127 41.2883 27.5938 0.0140763 0.0241555 2026-02-23T13:08:11.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:08:11 trial068 ceph-mon[38805]: pgmap v31: 17 pgs: 15 active+undersized+degraded, 2 active+clean; 1.4 GiB data, 4.6 GiB used, 2.7 TiB / 2.7 TiB avail; 25 MiB/s wr, 3.26k op/s; 20032/68493 objects degraded (29.247%) 2026-02-23T13:08:11.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:08:11 trial068 ceph-mon[38805]: from='osd.3 [v2:10.20.193.185:6808/1590361891,v1:10.20.193.185:6809/1590361891]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2026-02-23T13:08:11.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:08:11 trial068 ceph-mon[38805]: osdmap e34: 4 total, 3 up, 4 in 2026-02-23T13:08:11.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:08:11 trial068 ceph-mon[38805]: from='osd.3 [v2:10.20.193.185:6808/1590361891,v1:10.20.193.185:6809/1590361891]' entity='osd.3' cmd={"prefix": "osd crush create-or-move", "id": 3, "weight":0.6836, "args": ["host=trial185", "root=default"]} : dispatch 2026-02-23T13:08:11.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:08:11 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/409414270' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:08:12.325 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T13:08:12.324682+0000 min lat: 0.000727387 max lat: 1.08556 avg lat: 0.0241555 2026-02-23T13:08:12.325 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T13:08:12.325 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 40 128 206255 206127 40.2561 0 - 0.0241555 2026-02-23T13:08:12.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:08:12 trial068 ceph-mon[38805]: Health check cleared: OSD_DOWN (was: 1 osds down) 2026-02-23T13:08:12.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:08:12 trial068 ceph-mon[38805]: osd.3 [v2:10.20.193.185:6808/1590361891,v1:10.20.193.185:6809/1590361891] boot 2026-02-23T13:08:12.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:08:12 trial068 ceph-mon[38805]: osdmap e35: 4 total, 4 up, 4 in 2026-02-23T13:08:12.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:08:12 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2026-02-23T13:08:12.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:08:12 trial068 ceph-mon[38805]: osdmap e36: 4 total, 4 up, 4 in 2026-02-23T13:08:12.898 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:08:13.013 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:08:13.177 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:08:13.176+0000 7fc18acac640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:08:13.177 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:08:13.176+0000 7fc18acac640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:08:13.189 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:08:13.190 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:08:13.199 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:08:13.200 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:08:13.210 INFO:teuthology.orchestra.run.trial068.stdout:osd.2: {} 2026-02-23T13:08:13.211 INFO:teuthology.orchestra.run.trial068.stderr:osd.2: osd_enable_op_tracker = '' 2026-02-23T13:08:13.221 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:08:13.221 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:08:13.325 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 41 128 206255 206127 39.2743 0 - 0.0241555 2026-02-23T13:08:13.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:08:13 trial068 ceph-mon[38805]: pgmap v34: 17 pgs: 15 active+undersized+degraded, 2 active+clean; 1.5 GiB data, 4.9 GiB used, 2.7 TiB / 2.7 TiB avail; 39 MiB/s wr, 5.01k op/s; 22212/75948 objects degraded (29.246%) 2026-02-23T13:08:13.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:08:13 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2796470438' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:08:13.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:08:13 trial068 ceph-mon[38805]: osdmap e37: 4 total, 4 up, 4 in 2026-02-23T13:08:14.278 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:08:13 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:08:13] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:08:14.324 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 42 128 209035 208907 38.8562 7.23958 0.008017 0.02572 2026-02-23T13:08:14.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:08:13 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:08:13.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:08:14.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:08:14 trial068 ceph-mon[38805]: Health check update: Degraded data redundancy: 22212/75948 objects degraded (29.246%), 15 pgs degraded (PG_DEGRADED) 2026-02-23T13:08:14.720 DEBUG:teuthology.orchestra.run.trial185:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph daemon osd.3 dump_ops_in_flight 2026-02-23T13:08:14.866 INFO:teuthology.orchestra.run.trial185.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/config/ceph.conf 2026-02-23T13:08:15.005 INFO:teuthology.orchestra.run.trial185.stdout:ERROR: (22) Invalid argument 2026-02-23T13:08:15.005 INFO:teuthology.orchestra.run.trial185.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2026-02-23T13:08:15.255 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:08:15.325 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 43 128 212276 212148 38.5414 25.3203 0.0629433 0.0259271 2026-02-23T13:08:15.407 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:08:15.538 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:08:15.536+0000 7f8f4101a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:08:15.538 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:08:15.536+0000 7f8f4101a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:08:15.539 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:08:15 trial068 ceph-mon[38805]: pgmap v37: 17 pgs: 15 active+undersized+degraded, 2 active+clean; 1.5 GiB data, 4.9 GiB used, 2.7 TiB / 2.7 TiB avail; 39 MiB/s wr, 4.97k op/s; 22212/75948 objects degraded (29.246%) 2026-02-23T13:08:15.553 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:08:15.553 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:08:15.567 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:08:15.568 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:08:15.581 INFO:teuthology.orchestra.run.trial068.stdout:osd.2: {} 2026-02-23T13:08:15.581 INFO:teuthology.orchestra.run.trial068.stderr:osd.2: osd_enable_op_tracker = '' 2026-02-23T13:08:15.593 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:08:15.593 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:08:16.325 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 44 128 215601 215473 38.2558 25.9766 0.0429904 0.0261215 2026-02-23T13:08:16.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:08:16 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3647244564' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:08:16.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:08:16 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T13:08:16.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:08:16 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T13:08:17.325 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 45 128 218887 218759 37.9761 25.6719 0.00778625 0.0263186 2026-02-23T13:08:17.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:08:17 trial068 ceph-mon[38805]: pgmap v38: 17 pgs: 11 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 3 active+clean; 1.6 GiB data, 5.2 GiB used, 2.7 TiB / 2.7 TiB avail; 29 MiB/s wr, 3.73k op/s; 7660/79572 objects degraded (9.627%); 8.6 MiB/s, 138 objects/s recovering 2026-02-23T13:08:17.637 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:08:17.763 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:08:17.870 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:08:17.869+0000 7fbf2f5f0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:08:17.871 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:08:17.869+0000 7fbf2f5f0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:08:17.884 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:08:17.884 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:08:17.895 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:08:17.895 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:08:17.906 INFO:teuthology.orchestra.run.trial068.stdout:osd.2: {} 2026-02-23T13:08:17.907 INFO:teuthology.orchestra.run.trial068.stderr:osd.2: osd_enable_op_tracker = '' 2026-02-23T13:08:17.917 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:08:17.918 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:08:18.325 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 46 128 220094 219966 37.3555 9.42969 0.0124134 0.0264257 2026-02-23T13:08:18.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:08:18 trial068 ceph-mon[38805]: osdmap e38: 4 total, 4 up, 4 in 2026-02-23T13:08:18.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:08:18 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2232471461' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:08:19.325 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 47 128 222785 222657 37.008 21.0234 0.0213843 0.0269776 2026-02-23T13:08:19.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:08:19 trial068 ceph-mon[38805]: pgmap v40: 17 pgs: 11 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 3 active+clean; 1.6 GiB data, 5.2 GiB used, 2.7 TiB / 2.7 TiB avail; 13 MiB/s wr, 1.61k op/s; 7660/79572 objects degraded (9.627%); 8.6 MiB/s, 137 objects/s recovering 2026-02-23T13:08:19.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:08:19 trial068 ceph-mon[38805]: osdmap e39: 4 total, 4 up, 4 in 2026-02-23T13:08:19.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:08:19 trial068 ceph-mon[38805]: Health check update: Degraded data redundancy: 7660/79572 objects degraded (9.627%), 14 pgs degraded (PG_DEGRADED) 2026-02-23T13:08:19.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:08:19 trial068 ceph-mon[38805]: osdmap e40: 4 total, 4 up, 4 in 2026-02-23T13:08:19.953 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:08:20.047 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2026-02-23T13:08:20.048 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd reweight 0 1 2026-02-23T13:08:20.082 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:08:20.325 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 48 128 224214 224086 36.4695 11.1641 0.015086 0.0273355 2026-02-23T13:08:20.359 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:08:20.555 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:08:20 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T13:08:20.555 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:08:20 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T13:08:20.555 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:08:20 trial068 ceph-mon[38805]: osdmap e41: 4 total, 4 up, 4 in 2026-02-23T13:08:20.602 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:08:20.517+0000 7fba19330640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:08:20.602 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:08:20.517+0000 7fba19330640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:08:20.602 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:08:20.603 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:08:20.603 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:08:20.603 INFO:teuthology.orchestra.run.trial068.stdout:osd.2: {} 2026-02-23T13:08:20.603 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:08:20.603 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:08:20.603 INFO:teuthology.orchestra.run.trial068.stderr:osd.2: osd_enable_op_tracker = '' 2026-02-23T13:08:20.603 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:08:21.115 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:08:21.113+0000 7f829f7bf640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:08:21.115 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:08:21.114+0000 7f829f7bf640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:08:21.325 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 49 128 225618 225490 35.9491 10.9688 0.0181386 0.0277475 2026-02-23T13:08:21.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:08:21 trial068 ceph-mon[38805]: pgmap v44: 17 pgs: 11 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 3 active+clean; 1.6 GiB data, 5.2 GiB used, 2.7 TiB / 2.7 TiB avail; 7660/79572 objects degraded (9.627%) 2026-02-23T13:08:21.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:08:21 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3995015801' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:08:21.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:08:21 trial068 ceph-mon[38805]: osdmap e42: 4 total, 4 up, 4 in 2026-02-23T13:08:21.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:08:21 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1221585394' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2026-02-23T13:08:21.965 INFO:teuthology.orchestra.run.trial068.stderr:reweighted osd.0 to 1 (10000) 2026-02-23T13:08:22.005 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd reweight 1 1 2026-02-23T13:08:22.140 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:08:22.266 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:08:22.265+0000 7f46a48d0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:08:22.266 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:08:22.265+0000 7f46a48d0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:08:22.325 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 50 128 226384 226256 35.3498 5.98438 0.00794367 0.0281881 2026-02-23T13:08:22.966 INFO:teuthology.orchestra.run.trial068.stderr:reweighted osd.1 to 1 (10000) 2026-02-23T13:08:23.011 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd reweight 2 1 2026-02-23T13:08:23.031 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:08:23.155 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:08:23.199 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:08:23.216 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:08:22 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1221585394' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2026-02-23T13:08:23.217 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:08:22 trial068 ceph-mon[38805]: osdmap e43: 4 total, 4 up, 4 in 2026-02-23T13:08:23.217 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:08:22 trial068 ceph-mon[38805]: pgmap v47: 17 pgs: 2 peering, 1 active+recovery_wait, 5 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 6 active+clean; 1.7 GiB data, 5.8 GiB used, 2.7 TiB / 2.7 TiB avail; 24 MiB/s wr, 3.03k op/s; 3615/84054 objects degraded (4.301%); 64 MiB/s, 1.02k objects/s recovering 2026-02-23T13:08:23.217 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:08:22 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/415434875' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2026-02-23T13:08:23.278 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:08:23.276+0000 7f7d0e9a7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:08:23.278 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:08:23.276+0000 7f7d0e9a7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:08:23.311 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:08:23.310+0000 7f914e010640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:08:23.311 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:08:23.310+0000 7f914e010640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:08:23.325 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:08:23.325 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:08:23.325 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 51 128 227326 227198 34.801 7.35938 0.0114272 0.0286516 2026-02-23T13:08:23.336 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:08:23.336 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:08:23.347 INFO:teuthology.orchestra.run.trial068.stdout:osd.2: {} 2026-02-23T13:08:23.347 INFO:teuthology.orchestra.run.trial068.stderr:osd.2: osd_enable_op_tracker = '' 2026-02-23T13:08:23.357 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:08:23.357 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:08:23.961 INFO:teuthology.orchestra.run.trial068.stderr:reweighted osd.2 to 1 (10000) 2026-02-23T13:08:24.007 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd reweight 3 1 2026-02-23T13:08:24.153 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:08:24.217 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:08:23 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:08:23] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:08:24.217 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:08:23 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/415434875' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2026-02-23T13:08:24.217 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:08:23 trial068 ceph-mon[38805]: osdmap e44: 4 total, 4 up, 4 in 2026-02-23T13:08:24.217 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:08:23 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3764540826' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:08:24.218 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:08:23 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2768338669' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2026-02-23T13:08:24.218 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:08:23 trial068 ceph-mon[38805]: Health check update: Degraded data redundancy: 3615/84054 objects degraded (4.301%), 8 pgs degraded (PG_DEGRADED) 2026-02-23T13:08:24.218 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:08:23 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2768338669' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2026-02-23T13:08:24.218 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:08:23 trial068 ceph-mon[38805]: osdmap e45: 4 total, 4 up, 4 in 2026-02-23T13:08:24.276 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:08:24.274+0000 7f4cd6363640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:08:24.276 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:08:24.274+0000 7f4cd6363640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:08:24.325 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 52 128 228346 228218 34.285 7.96875 0.00913466 0.0290708 2026-02-23T13:08:24.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:08:23 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:08:23.972Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:08:24.962 INFO:teuthology.orchestra.run.trial068.stderr:reweighted osd.3 to 1 (10000) 2026-02-23T13:08:25.007 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2026-02-23T13:08:25.008 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph pg dump --format=json 2026-02-23T13:08:25.146 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:08:25.214 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:08:24 trial068 ceph-mon[38805]: pgmap v50: 17 pgs: 2 peering, 1 active+recovery_wait, 5 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 6 active+clean; 1.7 GiB data, 5.8 GiB used, 2.7 TiB / 2.7 TiB avail; 23 MiB/s wr, 2.99k op/s; 3615/84054 objects degraded (4.301%); 63 MiB/s, 1.01k objects/s recovering 2026-02-23T13:08:25.215 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:08:24 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/4070378099' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2026-02-23T13:08:25.215 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:08:24 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/4070378099' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2026-02-23T13:08:25.215 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:08:24 trial068 ceph-mon[38805]: osdmap e46: 4 total, 4 up, 4 in 2026-02-23T13:08:25.267 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:08:25.266+0000 7f5594a54640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:08:25.268 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:08:25.266+0000 7f5594a54640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:08:25.326 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 53 128 229697 229569 33.8372 10.5547 0.070055 0.0295401 2026-02-23T13:08:25.397 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:08:25.418 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T13:08:25.419 INFO:teuthology.orchestra.run.trial068.stderr:dumped all 2026-02-23T13:08:25.468 INFO:teuthology.orchestra.run.trial068.stdout:{"pg_ready":true,"pg_map":{"version":51,"stamp":"2026-02-23T13:08:24.962277+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1836589600,"num_objects":28018,"num_object_clones":0,"num_object_copies":84054,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3615,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28018,"num_whiteouts":0,"num_read":122,"num_read_kb":228,"num_write":224256,"num_write_kb":1795118,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4881,"num_bytes_recovered":319676960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160820,"ondisk_log_size":160820,"up":51,"acting":43,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":51,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":2936012800,"kb_used":6056768,"kb_used_data":5107184,"kb_used_omap":64,"kb_used_meta":949439,"kb_avail":2929956032,"statfs":{"total":3006477107200,"available":3000274976768,"internally_reserved":0,"allocated":5229756416,"data_stored":5228756555,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":65916,"internal_metadata":972226180},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[6,8,8,2,27,28,24,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"3.997629"},"pg_stats":[{"pgid":"2.f","version":"41'14240","reported_seq":15050,"reported_epoch":41,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2026-02-23T13:08:20.258886+0000","last_change":"2026-02-23T13:08:19.354570+0000","last_active":"2026-02-23T13:08:20.258886+0000","last_peered":"2026-02-23T13:08:20.258886+0000","last_clean":"2026-02-23T13:07:51.221395+0000","last_became_active":"2026-02-23T13:08:13.377169+0000","last_became_peered":"2026-02-23T13:08:13.377169+0000","last_unstale":"2026-02-23T13:08:20.258886+0000","last_undegraded":"2026-02-23T13:08:13.288761+0000","last_fullsized":"2026-02-23T13:08:13.274687+0000","mapping_epoch":36,"log_start":"31'4200","ondisk_log_start":"31'4200","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_clean_scrub_stamp":"2026-02-23T13:07:30.224555+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":116654080,"num_objects":1780,"num_object_clones":0,"num_object_copies":5340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":225,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1780,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14240,"num_write_kb":113920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":382,"num_bytes_recovered":25034752,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":1555}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.e","version":"41'13696","reported_seq":13730,"reported_epoch":41,"state":"active+clean","last_fresh":"2026-02-23T13:08:20.350495+0000","last_change":"2026-02-23T13:07:31.233069+0000","last_active":"2026-02-23T13:08:20.350495+0000","last_peered":"2026-02-23T13:08:20.350495+0000","last_clean":"2026-02-23T13:08:20.350495+0000","last_became_active":"2026-02-23T13:07:31.232454+0000","last_became_peered":"2026-02-23T13:07:31.232454+0000","last_unstale":"2026-02-23T13:08:20.350495+0000","last_undegraded":"2026-02-23T13:08:20.350495+0000","last_fullsized":"2026-02-23T13:08:20.350495+0000","mapping_epoch":29,"log_start":"31'3600","ondisk_log_start":"31'3600","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_clean_scrub_stamp":"2026-02-23T13:07:30.224555+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T13:08:31.879194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112197632,"num_objects":1712,"num_object_clones":0,"num_object_copies":5136,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1712,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13696,"num_write_kb":109568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.d","version":"41'13791","reported_seq":14511,"reported_epoch":41,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2026-02-23T13:08:20.366757+0000","last_change":"2026-02-23T13:08:19.585496+0000","last_active":"2026-02-23T13:08:20.366757+0000","last_peered":"2026-02-23T13:08:20.366757+0000","last_clean":"2026-02-23T13:07:51.244394+0000","last_became_active":"2026-02-23T13:08:13.500901+0000","last_became_peered":"2026-02-23T13:08:13.500901+0000","last_unstale":"2026-02-23T13:08:20.366757+0000","last_undegraded":"2026-02-23T13:08:13.311356+0000","last_fullsized":"2026-02-23T13:08:13.274854+0000","mapping_epoch":36,"log_start":"31'3700","ondisk_log_start":"31'3700","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_clean_scrub_stamp":"2026-02-23T13:07:30.224555+0000","objects_scrubbed":0,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112975872,"num_objects":1724,"num_object_clones":0,"num_object_copies":5172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1724,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13791,"num_write_kb":110328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":340,"num_bytes_recovered":22175744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,0],"avail_no_missing":["1","0"],"object_location_counts":[{"shards":"0,1","objects":1448}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"41'14104","reported_seq":15399,"reported_epoch":41,"state":"active+clean","last_fresh":"2026-02-23T13:08:20.355345+0000","last_change":"2026-02-23T13:08:19.971521+0000","last_active":"2026-02-23T13:08:20.355345+0000","last_peered":"2026-02-23T13:08:20.355345+0000","last_clean":"2026-02-23T13:08:20.355345+0000","last_became_active":"2026-02-23T13:08:19.965066+0000","last_became_peered":"2026-02-23T13:08:19.965066+0000","last_unstale":"2026-02-23T13:08:20.355345+0000","last_undegraded":"2026-02-23T13:08:20.355345+0000","last_fullsized":"2026-02-23T13:08:20.355345+0000","mapping_epoch":40,"log_start":"31'4100","ondisk_log_start":"31'4100","created":29,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_clean_scrub_stamp":"2026-02-23T13:07:30.224555+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T13:08:34.075016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":115539968,"num_objects":1763,"num_object_clones":0,"num_object_copies":5289,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1763,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14104,"num_write_kb":112832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":625,"num_bytes_recovered":40861696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"28'99","reported_seq":68,"reported_epoch":41,"state":"active+clean","last_fresh":"2026-02-23T13:08:19.963135+0000","last_change":"2026-02-23T13:08:12.277636+0000","last_active":"2026-02-23T13:08:19.963135+0000","last_peered":"2026-02-23T13:08:19.963135+0000","last_clean":"2026-02-23T13:08:19.963135+0000","last_became_active":"2026-02-23T13:08:12.277258+0000","last_became_peered":"2026-02-23T13:08:12.277258+0000","last_unstale":"2026-02-23T13:08:19.963135+0000","last_undegraded":"2026-02-23T13:08:19.963135+0000","last_fullsized":"2026-02-23T13:08:19.963135+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:07:05.619208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:07:05.619208+0000","last_clean_scrub_stamp":"2026-02-23T13:07:05.619208+0000","objects_scrubbed":0,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T13:08:29.473610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":122,"num_read_kb":228,"num_write":135,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"41'13808","reported_seq":15003,"reported_epoch":41,"state":"active+recovery_wait","last_fresh":"2026-02-23T13:08:20.261566+0000","last_change":"2026-02-23T13:08:18.364572+0000","last_active":"2026-02-23T13:08:20.261566+0000","last_peered":"2026-02-23T13:08:20.261566+0000","last_clean":"2026-02-23T13:07:51.221480+0000","last_became_active":"2026-02-23T13:08:18.297689+0000","last_became_peered":"2026-02-23T13:08:18.297689+0000","last_unstale":"2026-02-23T13:08:20.261566+0000","last_undegraded":"2026-02-23T13:08:20.261566+0000","last_fullsized":"2026-02-23T13:08:20.261566+0000","mapping_epoch":38,"log_start":"31'3800","ondisk_log_start":"31'3800","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_clean_scrub_stamp":"2026-02-23T13:07:30.224555+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":113115136,"num_objects":1726,"num_object_clones":0,"num_object_copies":5178,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1726,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13808,"num_write_kb":110464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":580,"num_bytes_recovered":37953536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":["3","1","2"],"object_location_counts":[{"shards":"1,2,3","objects":1726}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"41'13752","reported_seq":15030,"reported_epoch":41,"state":"active+clean","last_fresh":"2026-02-23T13:08:20.233356+0000","last_change":"2026-02-23T13:08:18.292955+0000","last_active":"2026-02-23T13:08:20.233356+0000","last_peered":"2026-02-23T13:08:20.233356+0000","last_clean":"2026-02-23T13:08:20.233356+0000","last_became_active":"2026-02-23T13:08:18.289580+0000","last_became_peered":"2026-02-23T13:08:18.289580+0000","last_unstale":"2026-02-23T13:08:20.233356+0000","last_undegraded":"2026-02-23T13:08:20.233356+0000","last_fullsized":"2026-02-23T13:08:20.233356+0000","mapping_epoch":38,"log_start":"31'3700","ondisk_log_start":"31'3700","created":29,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_clean_scrub_stamp":"2026-02-23T13:07:30.224555+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T13:08:46.511057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112656384,"num_objects":1719,"num_object_clones":0,"num_object_copies":5157,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1719,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13752,"num_write_kb":110016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":616,"num_bytes_recovered":40247296,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"41'13730","reported_seq":13771,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2026-02-23T13:08:20.241889+0000","last_change":"2026-02-23T13:08:13.502275+0000","last_active":"2026-02-23T13:08:20.241889+0000","last_peered":"2026-02-23T13:08:20.241889+0000","last_clean":"2026-02-23T13:07:51.227636+0000","last_became_active":"2026-02-23T13:08:13.500184+0000","last_became_peered":"2026-02-23T13:08:13.500184+0000","last_unstale":"2026-02-23T13:08:20.241889+0000","last_undegraded":"2026-02-23T13:08:13.304194+0000","last_fullsized":"2026-02-23T13:08:13.274744+0000","mapping_epoch":36,"log_start":"31'3700","ondisk_log_start":"31'3700","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_clean_scrub_stamp":"2026-02-23T13:07:30.224555+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112476160,"num_objects":1717,"num_object_clones":0,"num_object_copies":5151,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":616,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1717,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13730,"num_write_kb":109840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0],"avail_no_missing":["1","0"],"object_location_counts":[{"shards":"0,1","objects":1101}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"41'13968","reported_seq":14004,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2026-02-23T13:08:20.305694+0000","last_change":"2026-02-23T13:08:13.496457+0000","last_active":"2026-02-23T13:08:20.305694+0000","last_peered":"2026-02-23T13:08:20.305694+0000","last_clean":"2026-02-23T13:07:51.242184+0000","last_became_active":"2026-02-23T13:08:13.492897+0000","last_became_peered":"2026-02-23T13:08:13.492897+0000","last_unstale":"2026-02-23T13:08:20.305694+0000","last_undegraded":"2026-02-23T13:08:13.300308+0000","last_fullsized":"2026-02-23T13:08:13.274784+0000","mapping_epoch":36,"log_start":"31'3900","ondisk_log_start":"31'3900","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_clean_scrub_stamp":"2026-02-23T13:07:30.224555+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":114425856,"num_objects":1746,"num_object_clones":0,"num_object_copies":5238,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":585,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1746,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13968,"num_write_kb":111744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":16384,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[{"shards":"1,2","objects":1161}],"blocked_by":[],"up_primary":3,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"41'14032","reported_seq":14066,"reported_epoch":41,"state":"active+clean","last_fresh":"2026-02-23T13:08:20.335005+0000","last_change":"2026-02-23T13:07:31.234133+0000","last_active":"2026-02-23T13:08:20.335005+0000","last_peered":"2026-02-23T13:08:20.335005+0000","last_clean":"2026-02-23T13:08:20.335005+0000","last_became_active":"2026-02-23T13:07:31.233956+0000","last_became_peered":"2026-02-23T13:07:31.233956+0000","last_unstale":"2026-02-23T13:08:20.335005+0000","last_undegraded":"2026-02-23T13:08:20.335005+0000","last_fullsized":"2026-02-23T13:08:20.335005+0000","mapping_epoch":29,"log_start":"31'4000","ondisk_log_start":"31'4000","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_clean_scrub_stamp":"2026-02-23T13:07:30.224555+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T13:08:47.743470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":114950144,"num_objects":1754,"num_object_clones":0,"num_object_copies":5262,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1754,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14032,"num_write_kb":112256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"40'14128","reported_seq":15414,"reported_epoch":41,"state":"peering","last_fresh":"2026-02-23T13:08:19.775339+0000","last_change":"2026-02-23T13:08:20.279538+0000","last_active":"2026-02-23T13:08:20.279538+0000","last_peered":"2026-02-23T13:08:19.775339+0000","last_clean":"2026-02-23T13:07:51.192350+0000","last_became_active":"2026-02-23T13:08:13.446785+0000","last_became_peered":"2026-02-23T13:08:13.446785+0000","last_unstale":"2026-02-23T13:08:19.775339+0000","last_undegraded":"2026-02-23T13:08:19.775339+0000","last_fullsized":"2026-02-23T13:08:13.276074+0000","mapping_epoch":41,"log_start":"31'4100","ondisk_log_start":"31'4100","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_clean_scrub_stamp":"2026-02-23T13:07:30.224555+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":115736576,"num_objects":1766,"num_object_clones":0,"num_object_copies":5298,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1766,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14128,"num_write_kb":113024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":624,"num_bytes_recovered":40894464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"41'14088","reported_seq":14130,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2026-02-23T13:08:20.294069+0000","last_change":"2026-02-23T13:08:13.495762+0000","last_active":"2026-02-23T13:08:20.294069+0000","last_peered":"2026-02-23T13:08:20.294069+0000","last_clean":"2026-02-23T13:07:51.246400+0000","last_became_active":"2026-02-23T13:08:13.493429+0000","last_became_peered":"2026-02-23T13:08:13.493429+0000","last_unstale":"2026-02-23T13:08:20.294069+0000","last_undegraded":"2026-02-23T13:08:13.304070+0000","last_fullsized":"2026-02-23T13:08:13.274974+0000","mapping_epoch":36,"log_start":"31'4000","ondisk_log_start":"31'4000","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_clean_scrub_stamp":"2026-02-23T13:07:30.224555+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":115408896,"num_objects":1761,"num_object_clones":0,"num_object_copies":5283,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":625,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1761,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14088,"num_write_kb":112704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[{"shards":"1,2","objects":1136}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"41'13616","reported_seq":14690,"reported_epoch":41,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2026-02-23T13:08:20.368222+0000","last_change":"2026-02-23T13:08:18.921823+0000","last_active":"2026-02-23T13:08:20.368222+0000","last_peered":"2026-02-23T13:08:20.368222+0000","last_clean":"2026-02-23T13:07:51.204436+0000","last_became_active":"2026-02-23T13:08:13.452184+0000","last_became_peered":"2026-02-23T13:08:13.452184+0000","last_unstale":"2026-02-23T13:08:20.368222+0000","last_undegraded":"2026-02-23T13:08:13.300567+0000","last_fullsized":"2026-02-23T13:08:13.274936+0000","mapping_epoch":36,"log_start":"31'3600","ondisk_log_start":"31'3600","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_clean_scrub_stamp":"2026-02-23T13:07:30.224555+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":111542272,"num_objects":1702,"num_object_clones":0,"num_object_copies":5106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":66,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1702,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13616,"num_write_kb":108928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":519,"num_bytes_recovered":34013184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[{"shards":"1,2","objects":1636}],"blocked_by":[],"up_primary":3,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"41'14016","reported_seq":14836,"reported_epoch":41,"state":"active+clean","last_fresh":"2026-02-23T13:08:20.250396+0000","last_change":"2026-02-23T13:08:18.965066+0000","last_active":"2026-02-23T13:08:20.250396+0000","last_peered":"2026-02-23T13:08:20.250396+0000","last_clean":"2026-02-23T13:08:20.250396+0000","last_became_active":"2026-02-23T13:08:18.961857+0000","last_became_peered":"2026-02-23T13:08:18.961857+0000","last_unstale":"2026-02-23T13:08:20.250396+0000","last_undegraded":"2026-02-23T13:08:20.250396+0000","last_fullsized":"2026-02-23T13:08:20.250396+0000","mapping_epoch":39,"log_start":"31'4000","ondisk_log_start":"31'4000","created":29,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_clean_scrub_stamp":"2026-02-23T13:07:30.224555+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T13:08:45.827389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":114819072,"num_objects":1752,"num_object_clones":0,"num_object_copies":5256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1752,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14016,"num_write_kb":112128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":578,"num_bytes_recovered":37879808,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"41'14416","reported_seq":14461,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2026-02-23T13:08:20.090584+0000","last_change":"2026-02-23T13:08:13.499890+0000","last_active":"2026-02-23T13:08:20.090584+0000","last_peered":"2026-02-23T13:08:20.090584+0000","last_clean":"2026-02-23T13:07:51.245909+0000","last_became_active":"2026-02-23T13:08:13.493775+0000","last_became_peered":"2026-02-23T13:08:13.493775+0000","last_unstale":"2026-02-23T13:08:20.090584+0000","last_undegraded":"2026-02-23T13:08:13.303540+0000","last_fullsized":"2026-02-23T13:08:13.274766+0000","mapping_epoch":36,"log_start":"31'4400","ondisk_log_start":"31'4400","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_clean_scrub_stamp":"2026-02-23T13:07:30.224555+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":118095872,"num_objects":1802,"num_object_clones":0,"num_object_copies":5406,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":608,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1802,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14416,"num_write_kb":115328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":16384,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,0],"avail_no_missing":["1","0"],"object_location_counts":[{"shards":"0,1","objects":1194}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.a","version":"40'14240","reported_seq":15478,"reported_epoch":40,"state":"peering","last_fresh":"2026-02-23T13:08:19.795204+0000","last_change":"2026-02-23T13:08:19.964412+0000","last_active":"2026-02-23T13:08:19.964412+0000","last_peered":"2026-02-23T13:08:19.795204+0000","last_clean":"2026-02-23T13:07:51.233066+0000","last_became_active":"2026-02-23T13:08:13.493495+0000","last_became_peered":"2026-02-23T13:08:13.493495+0000","last_unstale":"2026-02-23T13:08:19.795204+0000","last_undegraded":"2026-02-23T13:08:19.795204+0000","last_fullsized":"2026-02-23T13:08:13.274827+0000","mapping_epoch":41,"log_start":"31'4200","ondisk_log_start":"31'4200","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_clean_scrub_stamp":"2026-02-23T13:07:30.224555+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":116654080,"num_objects":1780,"num_object_clones":0,"num_object_copies":5340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1780,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14240,"num_write_kb":113920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":598,"num_bytes_recovered":39133184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2,3],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.b","version":"41'14496","reported_seq":14567,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2026-02-23T13:08:20.206072+0000","last_change":"2026-02-23T13:08:13.506166+0000","last_active":"2026-02-23T13:08:20.206072+0000","last_peered":"2026-02-23T13:08:20.206072+0000","last_clean":"2026-02-23T13:07:51.245808+0000","last_became_active":"2026-02-23T13:08:13.414518+0000","last_became_peered":"2026-02-23T13:08:13.414518+0000","last_unstale":"2026-02-23T13:08:20.206072+0000","last_undegraded":"2026-02-23T13:08:13.284483+0000","last_fullsized":"2026-02-23T13:08:13.276068+0000","mapping_epoch":36,"log_start":"31'4400","ondisk_log_start":"31'4400","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_clean_scrub_stamp":"2026-02-23T13:07:30.224555+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":118751232,"num_objects":1812,"num_object_clones":0,"num_object_copies":5436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":614,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1812,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14496,"num_write_kb":115968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":860160,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,2","objects":1198}],"blocked_by":[],"up_primary":3,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":1835999232,"num_objects":28016,"num_object_clones":0,"num_object_copies":84048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3615,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28016,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":224121,"num_write_kb":1792968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4879,"num_bytes_recovered":319086592,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":5267030016,"data_stored":5267030016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160721,"ondisk_log_size":160721,"up":48,"acting":40,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":122,"num_read_kb":228,"num_write":135,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":35,"seq":150323855363,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":734003200,"kb_used":1397640,"kb_used_data":1305012,"kb_used_omap":9,"kb_used_meta":92598,"kb_avail":732605560,"statfs":{"total":751619276800,"available":750188093440,"internally_reserved":0,"allocated":1336332288,"data_stored":1336083334,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9612,"internal_metadata":94820980},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411346,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":734003200,"kb_used":1513552,"kb_used_data":1233916,"kb_used_omap":18,"kb_used_meta":279597,"kb_avail":732489648,"statfs":{"total":751619276800,"available":750069399552,"internally_reserved":0,"allocated":1263529984,"data_stored":1263281030,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19202,"internal_metadata":286307582},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574868,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":734003200,"kb_used":1632652,"kb_used_data":1334968,"kb_used_omap":14,"kb_used_meta":297649,"kb_avail":732370548,"statfs":{"total":751619276800,"available":749947441152,"internally_reserved":0,"allocated":1367007232,"data_stored":1366761830,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14645,"internal_metadata":304793291},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[6,8,8,2,27,28,24,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705685,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":734003200,"kb_used":1512924,"kb_used_data":1233288,"kb_used_omap":21,"kb_used_meta":279594,"kb_avail":732490276,"statfs":{"total":751619276800,"available":750070042624,"internally_reserved":0,"allocated":1262886912,"data_stored":1262630361,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":22457,"internal_metadata":286304327},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1264238592,"data_stored":1264238592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1370267648,"data_stored":1370267648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1263788032,"data_stored":1263788032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1368735744,"data_stored":1368735744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-23T13:08:25.470 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph pg dump --format=json 2026-02-23T13:08:25.542 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:08:25.659 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:08:25.671 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:08:25.668+0000 7f43505d7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:08:25.671 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:08:25.669+0000 7f43505d7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:08:25.685 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:08:25.685 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:08:25.702 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:08:25.702 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:08:25.716 INFO:teuthology.orchestra.run.trial068.stdout:osd.2: {} 2026-02-23T13:08:25.716 INFO:teuthology.orchestra.run.trial068.stderr:osd.2: osd_enable_op_tracker = '' 2026-02-23T13:08:25.730 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:08:25.731 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:08:25.793 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:08:25.792+0000 7f2362bbf640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:08:25.794 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:08:25.792+0000 7f2362bbf640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:08:25.947 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T13:08:25.947 INFO:teuthology.orchestra.run.trial068.stderr:dumped all 2026-02-23T13:08:25.995 INFO:teuthology.orchestra.run.trial068.stdout:{"pg_ready":true,"pg_map":{"version":51,"stamp":"2026-02-23T13:08:24.962277+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1836589600,"num_objects":28018,"num_object_clones":0,"num_object_copies":84054,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3615,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28018,"num_whiteouts":0,"num_read":122,"num_read_kb":228,"num_write":224256,"num_write_kb":1795118,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4881,"num_bytes_recovered":319676960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160820,"ondisk_log_size":160820,"up":51,"acting":43,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":51,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":2936012800,"kb_used":6056768,"kb_used_data":5107184,"kb_used_omap":64,"kb_used_meta":949439,"kb_avail":2929956032,"statfs":{"total":3006477107200,"available":3000274976768,"internally_reserved":0,"allocated":5229756416,"data_stored":5228756555,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":65916,"internal_metadata":972226180},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[6,8,8,2,27,28,24,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"3.997629"},"pg_stats":[{"pgid":"2.f","version":"41'14240","reported_seq":15050,"reported_epoch":41,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2026-02-23T13:08:20.258886+0000","last_change":"2026-02-23T13:08:19.354570+0000","last_active":"2026-02-23T13:08:20.258886+0000","last_peered":"2026-02-23T13:08:20.258886+0000","last_clean":"2026-02-23T13:07:51.221395+0000","last_became_active":"2026-02-23T13:08:13.377169+0000","last_became_peered":"2026-02-23T13:08:13.377169+0000","last_unstale":"2026-02-23T13:08:20.258886+0000","last_undegraded":"2026-02-23T13:08:13.288761+0000","last_fullsized":"2026-02-23T13:08:13.274687+0000","mapping_epoch":36,"log_start":"31'4200","ondisk_log_start":"31'4200","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_clean_scrub_stamp":"2026-02-23T13:07:30.224555+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":116654080,"num_objects":1780,"num_object_clones":0,"num_object_copies":5340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":225,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1780,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14240,"num_write_kb":113920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":382,"num_bytes_recovered":25034752,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":1555}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.e","version":"41'13696","reported_seq":13730,"reported_epoch":41,"state":"active+clean","last_fresh":"2026-02-23T13:08:20.350495+0000","last_change":"2026-02-23T13:07:31.233069+0000","last_active":"2026-02-23T13:08:20.350495+0000","last_peered":"2026-02-23T13:08:20.350495+0000","last_clean":"2026-02-23T13:08:20.350495+0000","last_became_active":"2026-02-23T13:07:31.232454+0000","last_became_peered":"2026-02-23T13:07:31.232454+0000","last_unstale":"2026-02-23T13:08:20.350495+0000","last_undegraded":"2026-02-23T13:08:20.350495+0000","last_fullsized":"2026-02-23T13:08:20.350495+0000","mapping_epoch":29,"log_start":"31'3600","ondisk_log_start":"31'3600","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_clean_scrub_stamp":"2026-02-23T13:07:30.224555+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T13:08:31.879194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112197632,"num_objects":1712,"num_object_clones":0,"num_object_copies":5136,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1712,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13696,"num_write_kb":109568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.d","version":"41'13791","reported_seq":14511,"reported_epoch":41,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2026-02-23T13:08:20.366757+0000","last_change":"2026-02-23T13:08:19.585496+0000","last_active":"2026-02-23T13:08:20.366757+0000","last_peered":"2026-02-23T13:08:20.366757+0000","last_clean":"2026-02-23T13:07:51.244394+0000","last_became_active":"2026-02-23T13:08:13.500901+0000","last_became_peered":"2026-02-23T13:08:13.500901+0000","last_unstale":"2026-02-23T13:08:20.366757+0000","last_undegraded":"2026-02-23T13:08:13.311356+0000","last_fullsized":"2026-02-23T13:08:13.274854+0000","mapping_epoch":36,"log_start":"31'3700","ondisk_log_start":"31'3700","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_clean_scrub_stamp":"2026-02-23T13:07:30.224555+0000","objects_scrubbed":0,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112975872,"num_objects":1724,"num_object_clones":0,"num_object_copies":5172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1724,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13791,"num_write_kb":110328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":340,"num_bytes_recovered":22175744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,0],"avail_no_missing":["1","0"],"object_location_counts":[{"shards":"0,1","objects":1448}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"41'14104","reported_seq":15399,"reported_epoch":41,"state":"active+clean","last_fresh":"2026-02-23T13:08:20.355345+0000","last_change":"2026-02-23T13:08:19.971521+0000","last_active":"2026-02-23T13:08:20.355345+0000","last_peered":"2026-02-23T13:08:20.355345+0000","last_clean":"2026-02-23T13:08:20.355345+0000","last_became_active":"2026-02-23T13:08:19.965066+0000","last_became_peered":"2026-02-23T13:08:19.965066+0000","last_unstale":"2026-02-23T13:08:20.355345+0000","last_undegraded":"2026-02-23T13:08:20.355345+0000","last_fullsized":"2026-02-23T13:08:20.355345+0000","mapping_epoch":40,"log_start":"31'4100","ondisk_log_start":"31'4100","created":29,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_clean_scrub_stamp":"2026-02-23T13:07:30.224555+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T13:08:34.075016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":115539968,"num_objects":1763,"num_object_clones":0,"num_object_copies":5289,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1763,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14104,"num_write_kb":112832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":625,"num_bytes_recovered":40861696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"28'99","reported_seq":68,"reported_epoch":41,"state":"active+clean","last_fresh":"2026-02-23T13:08:19.963135+0000","last_change":"2026-02-23T13:08:12.277636+0000","last_active":"2026-02-23T13:08:19.963135+0000","last_peered":"2026-02-23T13:08:19.963135+0000","last_clean":"2026-02-23T13:08:19.963135+0000","last_became_active":"2026-02-23T13:08:12.277258+0000","last_became_peered":"2026-02-23T13:08:12.277258+0000","last_unstale":"2026-02-23T13:08:19.963135+0000","last_undegraded":"2026-02-23T13:08:19.963135+0000","last_fullsized":"2026-02-23T13:08:19.963135+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:07:05.619208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:07:05.619208+0000","last_clean_scrub_stamp":"2026-02-23T13:07:05.619208+0000","objects_scrubbed":0,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T13:08:29.473610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":122,"num_read_kb":228,"num_write":135,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"41'13808","reported_seq":15003,"reported_epoch":41,"state":"active+recovery_wait","last_fresh":"2026-02-23T13:08:20.261566+0000","last_change":"2026-02-23T13:08:18.364572+0000","last_active":"2026-02-23T13:08:20.261566+0000","last_peered":"2026-02-23T13:08:20.261566+0000","last_clean":"2026-02-23T13:07:51.221480+0000","last_became_active":"2026-02-23T13:08:18.297689+0000","last_became_peered":"2026-02-23T13:08:18.297689+0000","last_unstale":"2026-02-23T13:08:20.261566+0000","last_undegraded":"2026-02-23T13:08:20.261566+0000","last_fullsized":"2026-02-23T13:08:20.261566+0000","mapping_epoch":38,"log_start":"31'3800","ondisk_log_start":"31'3800","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_clean_scrub_stamp":"2026-02-23T13:07:30.224555+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":113115136,"num_objects":1726,"num_object_clones":0,"num_object_copies":5178,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1726,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13808,"num_write_kb":110464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":580,"num_bytes_recovered":37953536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":["3","1","2"],"object_location_counts":[{"shards":"1,2,3","objects":1726}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"41'13752","reported_seq":15030,"reported_epoch":41,"state":"active+clean","last_fresh":"2026-02-23T13:08:20.233356+0000","last_change":"2026-02-23T13:08:18.292955+0000","last_active":"2026-02-23T13:08:20.233356+0000","last_peered":"2026-02-23T13:08:20.233356+0000","last_clean":"2026-02-23T13:08:20.233356+0000","last_became_active":"2026-02-23T13:08:18.289580+0000","last_became_peered":"2026-02-23T13:08:18.289580+0000","last_unstale":"2026-02-23T13:08:20.233356+0000","last_undegraded":"2026-02-23T13:08:20.233356+0000","last_fullsized":"2026-02-23T13:08:20.233356+0000","mapping_epoch":38,"log_start":"31'3700","ondisk_log_start":"31'3700","created":29,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_clean_scrub_stamp":"2026-02-23T13:07:30.224555+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T13:08:46.511057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112656384,"num_objects":1719,"num_object_clones":0,"num_object_copies":5157,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1719,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13752,"num_write_kb":110016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":616,"num_bytes_recovered":40247296,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"41'13730","reported_seq":13771,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2026-02-23T13:08:20.241889+0000","last_change":"2026-02-23T13:08:13.502275+0000","last_active":"2026-02-23T13:08:20.241889+0000","last_peered":"2026-02-23T13:08:20.241889+0000","last_clean":"2026-02-23T13:07:51.227636+0000","last_became_active":"2026-02-23T13:08:13.500184+0000","last_became_peered":"2026-02-23T13:08:13.500184+0000","last_unstale":"2026-02-23T13:08:20.241889+0000","last_undegraded":"2026-02-23T13:08:13.304194+0000","last_fullsized":"2026-02-23T13:08:13.274744+0000","mapping_epoch":36,"log_start":"31'3700","ondisk_log_start":"31'3700","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_clean_scrub_stamp":"2026-02-23T13:07:30.224555+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112476160,"num_objects":1717,"num_object_clones":0,"num_object_copies":5151,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":616,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1717,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13730,"num_write_kb":109840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0],"avail_no_missing":["1","0"],"object_location_counts":[{"shards":"0,1","objects":1101}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"41'13968","reported_seq":14004,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2026-02-23T13:08:20.305694+0000","last_change":"2026-02-23T13:08:13.496457+0000","last_active":"2026-02-23T13:08:20.305694+0000","last_peered":"2026-02-23T13:08:20.305694+0000","last_clean":"2026-02-23T13:07:51.242184+0000","last_became_active":"2026-02-23T13:08:13.492897+0000","last_became_peered":"2026-02-23T13:08:13.492897+0000","last_unstale":"2026-02-23T13:08:20.305694+0000","last_undegraded":"2026-02-23T13:08:13.300308+0000","last_fullsized":"2026-02-23T13:08:13.274784+0000","mapping_epoch":36,"log_start":"31'3900","ondisk_log_start":"31'3900","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_clean_scrub_stamp":"2026-02-23T13:07:30.224555+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":114425856,"num_objects":1746,"num_object_clones":0,"num_object_copies":5238,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":585,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1746,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13968,"num_write_kb":111744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":16384,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[{"shards":"1,2","objects":1161}],"blocked_by":[],"up_primary":3,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"41'14032","reported_seq":14066,"reported_epoch":41,"state":"active+clean","last_fresh":"2026-02-23T13:08:20.335005+0000","last_change":"2026-02-23T13:07:31.234133+0000","last_active":"2026-02-23T13:08:20.335005+0000","last_peered":"2026-02-23T13:08:20.335005+0000","last_clean":"2026-02-23T13:08:20.335005+0000","last_became_active":"2026-02-23T13:07:31.233956+0000","last_became_peered":"2026-02-23T13:07:31.233956+0000","last_unstale":"2026-02-23T13:08:20.335005+0000","last_undegraded":"2026-02-23T13:08:20.335005+0000","last_fullsized":"2026-02-23T13:08:20.335005+0000","mapping_epoch":29,"log_start":"31'4000","ondisk_log_start":"31'4000","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_clean_scrub_stamp":"2026-02-23T13:07:30.224555+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T13:08:47.743470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":114950144,"num_objects":1754,"num_object_clones":0,"num_object_copies":5262,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1754,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14032,"num_write_kb":112256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"40'14128","reported_seq":15414,"reported_epoch":41,"state":"peering","last_fresh":"2026-02-23T13:08:19.775339+0000","last_change":"2026-02-23T13:08:20.279538+0000","last_active":"2026-02-23T13:08:20.279538+0000","last_peered":"2026-02-23T13:08:19.775339+0000","last_clean":"2026-02-23T13:07:51.192350+0000","last_became_active":"2026-02-23T13:08:13.446785+0000","last_became_peered":"2026-02-23T13:08:13.446785+0000","last_unstale":"2026-02-23T13:08:19.775339+0000","last_undegraded":"2026-02-23T13:08:19.775339+0000","last_fullsized":"2026-02-23T13:08:13.276074+0000","mapping_epoch":41,"log_start":"31'4100","ondisk_log_start":"31'4100","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_clean_scrub_stamp":"2026-02-23T13:07:30.224555+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":115736576,"num_objects":1766,"num_object_clones":0,"num_object_copies":5298,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1766,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14128,"num_write_kb":113024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":624,"num_bytes_recovered":40894464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"41'14088","reported_seq":14130,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2026-02-23T13:08:20.294069+0000","last_change":"2026-02-23T13:08:13.495762+0000","last_active":"2026-02-23T13:08:20.294069+0000","last_peered":"2026-02-23T13:08:20.294069+0000","last_clean":"2026-02-23T13:07:51.246400+0000","last_became_active":"2026-02-23T13:08:13.493429+0000","last_became_peered":"2026-02-23T13:08:13.493429+0000","last_unstale":"2026-02-23T13:08:20.294069+0000","last_undegraded":"2026-02-23T13:08:13.304070+0000","last_fullsized":"2026-02-23T13:08:13.274974+0000","mapping_epoch":36,"log_start":"31'4000","ondisk_log_start":"31'4000","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_clean_scrub_stamp":"2026-02-23T13:07:30.224555+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":115408896,"num_objects":1761,"num_object_clones":0,"num_object_copies":5283,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":625,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1761,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14088,"num_write_kb":112704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[{"shards":"1,2","objects":1136}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"41'13616","reported_seq":14690,"reported_epoch":41,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2026-02-23T13:08:20.368222+0000","last_change":"2026-02-23T13:08:18.921823+0000","last_active":"2026-02-23T13:08:20.368222+0000","last_peered":"2026-02-23T13:08:20.368222+0000","last_clean":"2026-02-23T13:07:51.204436+0000","last_became_active":"2026-02-23T13:08:13.452184+0000","last_became_peered":"2026-02-23T13:08:13.452184+0000","last_unstale":"2026-02-23T13:08:20.368222+0000","last_undegraded":"2026-02-23T13:08:13.300567+0000","last_fullsized":"2026-02-23T13:08:13.274936+0000","mapping_epoch":36,"log_start":"31'3600","ondisk_log_start":"31'3600","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_clean_scrub_stamp":"2026-02-23T13:07:30.224555+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":111542272,"num_objects":1702,"num_object_clones":0,"num_object_copies":5106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":66,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1702,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13616,"num_write_kb":108928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":519,"num_bytes_recovered":34013184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[{"shards":"1,2","objects":1636}],"blocked_by":[],"up_primary":3,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"41'14016","reported_seq":14836,"reported_epoch":41,"state":"active+clean","last_fresh":"2026-02-23T13:08:20.250396+0000","last_change":"2026-02-23T13:08:18.965066+0000","last_active":"2026-02-23T13:08:20.250396+0000","last_peered":"2026-02-23T13:08:20.250396+0000","last_clean":"2026-02-23T13:08:20.250396+0000","last_became_active":"2026-02-23T13:08:18.961857+0000","last_became_peered":"2026-02-23T13:08:18.961857+0000","last_unstale":"2026-02-23T13:08:20.250396+0000","last_undegraded":"2026-02-23T13:08:20.250396+0000","last_fullsized":"2026-02-23T13:08:20.250396+0000","mapping_epoch":39,"log_start":"31'4000","ondisk_log_start":"31'4000","created":29,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_clean_scrub_stamp":"2026-02-23T13:07:30.224555+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T13:08:45.827389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":114819072,"num_objects":1752,"num_object_clones":0,"num_object_copies":5256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1752,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14016,"num_write_kb":112128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":578,"num_bytes_recovered":37879808,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"41'14416","reported_seq":14461,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2026-02-23T13:08:20.090584+0000","last_change":"2026-02-23T13:08:13.499890+0000","last_active":"2026-02-23T13:08:20.090584+0000","last_peered":"2026-02-23T13:08:20.090584+0000","last_clean":"2026-02-23T13:07:51.245909+0000","last_became_active":"2026-02-23T13:08:13.493775+0000","last_became_peered":"2026-02-23T13:08:13.493775+0000","last_unstale":"2026-02-23T13:08:20.090584+0000","last_undegraded":"2026-02-23T13:08:13.303540+0000","last_fullsized":"2026-02-23T13:08:13.274766+0000","mapping_epoch":36,"log_start":"31'4400","ondisk_log_start":"31'4400","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_clean_scrub_stamp":"2026-02-23T13:07:30.224555+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":118095872,"num_objects":1802,"num_object_clones":0,"num_object_copies":5406,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":608,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1802,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14416,"num_write_kb":115328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":16384,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,0],"avail_no_missing":["1","0"],"object_location_counts":[{"shards":"0,1","objects":1194}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.a","version":"40'14240","reported_seq":15478,"reported_epoch":40,"state":"peering","last_fresh":"2026-02-23T13:08:19.795204+0000","last_change":"2026-02-23T13:08:19.964412+0000","last_active":"2026-02-23T13:08:19.964412+0000","last_peered":"2026-02-23T13:08:19.795204+0000","last_clean":"2026-02-23T13:07:51.233066+0000","last_became_active":"2026-02-23T13:08:13.493495+0000","last_became_peered":"2026-02-23T13:08:13.493495+0000","last_unstale":"2026-02-23T13:08:19.795204+0000","last_undegraded":"2026-02-23T13:08:19.795204+0000","last_fullsized":"2026-02-23T13:08:13.274827+0000","mapping_epoch":41,"log_start":"31'4200","ondisk_log_start":"31'4200","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_clean_scrub_stamp":"2026-02-23T13:07:30.224555+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":116654080,"num_objects":1780,"num_object_clones":0,"num_object_copies":5340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1780,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14240,"num_write_kb":113920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":598,"num_bytes_recovered":39133184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2,3],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.b","version":"41'14496","reported_seq":14567,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2026-02-23T13:08:20.206072+0000","last_change":"2026-02-23T13:08:13.506166+0000","last_active":"2026-02-23T13:08:20.206072+0000","last_peered":"2026-02-23T13:08:20.206072+0000","last_clean":"2026-02-23T13:07:51.245808+0000","last_became_active":"2026-02-23T13:08:13.414518+0000","last_became_peered":"2026-02-23T13:08:13.414518+0000","last_unstale":"2026-02-23T13:08:20.206072+0000","last_undegraded":"2026-02-23T13:08:13.284483+0000","last_fullsized":"2026-02-23T13:08:13.276068+0000","mapping_epoch":36,"log_start":"31'4400","ondisk_log_start":"31'4400","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_clean_scrub_stamp":"2026-02-23T13:07:30.224555+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":118751232,"num_objects":1812,"num_object_clones":0,"num_object_copies":5436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":614,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1812,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14496,"num_write_kb":115968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":860160,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,2","objects":1198}],"blocked_by":[],"up_primary":3,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":1835999232,"num_objects":28016,"num_object_clones":0,"num_object_copies":84048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3615,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28016,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":224121,"num_write_kb":1792968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4879,"num_bytes_recovered":319086592,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":5267030016,"data_stored":5267030016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160721,"ondisk_log_size":160721,"up":48,"acting":40,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":122,"num_read_kb":228,"num_write":135,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":35,"seq":150323855363,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":734003200,"kb_used":1397640,"kb_used_data":1305012,"kb_used_omap":9,"kb_used_meta":92598,"kb_avail":732605560,"statfs":{"total":751619276800,"available":750188093440,"internally_reserved":0,"allocated":1336332288,"data_stored":1336083334,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9612,"internal_metadata":94820980},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411346,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":734003200,"kb_used":1513552,"kb_used_data":1233916,"kb_used_omap":18,"kb_used_meta":279597,"kb_avail":732489648,"statfs":{"total":751619276800,"available":750069399552,"internally_reserved":0,"allocated":1263529984,"data_stored":1263281030,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19202,"internal_metadata":286307582},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574868,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":734003200,"kb_used":1632652,"kb_used_data":1334968,"kb_used_omap":14,"kb_used_meta":297649,"kb_avail":732370548,"statfs":{"total":751619276800,"available":749947441152,"internally_reserved":0,"allocated":1367007232,"data_stored":1366761830,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14645,"internal_metadata":304793291},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[6,8,8,2,27,28,24,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705685,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":734003200,"kb_used":1512924,"kb_used_data":1233288,"kb_used_omap":21,"kb_used_meta":279594,"kb_avail":732490276,"statfs":{"total":751619276800,"available":750070042624,"internally_reserved":0,"allocated":1262886912,"data_stored":1262630361,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":22457,"internal_metadata":286304327},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1264238592,"data_stored":1264238592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1370267648,"data_stored":1370267648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1263788032,"data_stored":1263788032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1368735744,"data_stored":1368735744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-23T13:08:25.997 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph status --format=json 2026-02-23T13:08:26.182 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:08:26.231 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:08:25 trial068 ceph-mon[38805]: from='client.14409 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2026-02-23T13:08:26.231 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:08:25 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2096360591' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:08:26.308 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:08:26.306+0000 7f689537a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:08:26.308 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:08:26.307+0000 7f689537a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:08:26.325 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 54 128 233359 233231 33.7403 28.6094 0.0237184 0.029623 2026-02-23T13:08:26.536 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T13:08:26.536 INFO:teuthology.orchestra.run.trial068.stdout:{"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 3615/84054 objects degraded (4.301%), 8 pgs degraded","count":8},"muted":false}},"mutes":[]},"election_epoch":5,"quorum":[0],"quorum_names":["a"],"quorum_age":142,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":1},"osdmap":{"epoch":46,"num_osds":4,"num_up_osds":4,"osd_up_since":1771852091,"num_in_osds":4,"osd_in_since":1771852023,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":5},{"state_name":"active+recovering+undersized+degraded+remapped","count":3},{"state_name":"peering","count":2},{"state_name":"active+recovery_wait","count":1}],"num_pgs":17,"num_pools":2,"num_objects":28018,"data_bytes":1836589600,"bytes_used":6202130432,"bytes_avail":3000274976768,"bytes_total":3006477107200,"inactive_pgs_ratio":0.11764705926179886,"degraded_objects":3615,"degraded_total":84054,"degraded_ratio":0.043008066243129417,"recovering_objects_per_sec":1011,"recovering_bytes_per_sec":66266810,"recovering_keys_per_sec":0,"num_objects_recovered":4046,"num_bytes_recovered":265101312,"num_keys_recovered":0,"write_bytes_sec":24507314,"read_op_per_sec":0,"write_op_per_sec":2991},"fsmap":{"epoch":1,"btime":"2026-02-23T13:06:03:098534+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["cephadm","dashboard","iostat","nfs","prometheus"],"services":{"dashboard":"https://10.20.193.68:8443/","prometheus":"http://10.20.193.68:9283/"}},"servicemap":{"epoch":1,"modified":"2026-02-23T13:06:03.098894+0000","services":{}},"progress_events":{}} 2026-02-23T13:08:26.586 INFO:tasks.cephadm.ceph_manager.ceph:making progress, resetting timeout 2026-02-23T13:08:26.587 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph pg dump --format=json 2026-02-23T13:08:26.735 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:08:26.870 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:08:26.868+0000 7fdd69bab640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:08:26.870 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:08:26.869+0000 7fdd69bab640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:08:27.024 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T13:08:27.024 INFO:teuthology.orchestra.run.trial068.stderr:dumped all 2026-02-23T13:08:27.048 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:08:26 trial068 ceph-mon[38805]: from='client.14413 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2026-02-23T13:08:27.048 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:08:26 trial068 ceph-mon[38805]: pgmap v52: 17 pgs: 17 active+clean; 1.8 GiB data, 6.2 GiB used, 2.7 TiB / 2.7 TiB avail; 10 MiB/s wr, 1.31k op/s; 53 MiB/s, 847 objects/s recovering 2026-02-23T13:08:27.049 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:08:26 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2469211027' entity='client.admin' cmd={"prefix": "status", "format": "json"} : dispatch 2026-02-23T13:08:27.070 INFO:teuthology.orchestra.run.trial068.stdout:{"pg_ready":true,"pg_map":{"version":52,"stamp":"2026-02-23T13:08:26.231329+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1882284576,"num_objects":28716,"num_object_clones":0,"num_object_copies":86148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28716,"num_whiteouts":0,"num_read":122,"num_read_kb":228,"num_write":229834,"num_write_kb":1839742,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8496,"num_bytes_recovered":556532256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160798,"ondisk_log_size":160798,"up":51,"acting":51,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":51,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":2936012800,"kb_used":6472492,"kb_used_data":5486812,"kb_used_omap":73,"kb_used_meta":985526,"kb_avail":2929540308,"statfs":{"total":3006477107200,"available":2999849275392,"internally_reserved":0,"allocated":5618495488,"data_stored":5617349124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":75686,"internal_metadata":1009178714},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[6,2,10,13,15,8,8,8,0,0,66],"upper_bound":2048},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":45694976,"num_objects":698,"num_object_clones":0,"num_object_copies":2094,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-3615,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":698,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5578,"num_write_kb":44624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3615,"num_bytes_recovered":236855296,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.266037"},"pg_stats":[{"pgid":"2.f","version":"46'14648","reported_seq":15926,"reported_epoch":46,"state":"active+clean","last_fresh":"2026-02-23T13:08:25.228499+0000","last_change":"2026-02-23T13:08:21.973471+0000","last_active":"2026-02-23T13:08:25.228499+0000","last_peered":"2026-02-23T13:08:25.228499+0000","last_clean":"2026-02-23T13:08:25.228499+0000","last_became_active":"2026-02-23T13:08:21.967829+0000","last_became_peered":"2026-02-23T13:08:21.967829+0000","last_unstale":"2026-02-23T13:08:25.228499+0000","last_undegraded":"2026-02-23T13:08:25.228499+0000","last_fullsized":"2026-02-23T13:08:25.228499+0000","mapping_epoch":42,"log_start":"31'4600","ondisk_log_start":"31'4600","created":29,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_clean_scrub_stamp":"2026-02-23T13:07:30.224555+0000","objects_scrubbed":0,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T13:08:36.049688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":119996416,"num_objects":1831,"num_object_clones":0,"num_object_copies":5493,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1831,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14648,"num_write_kb":117184,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":607,"num_bytes_recovered":39780352,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.e","version":"46'14048","reported_seq":14092,"reported_epoch":46,"state":"active+clean","last_fresh":"2026-02-23T13:08:25.391325+0000","last_change":"2026-02-23T13:07:31.233069+0000","last_active":"2026-02-23T13:08:25.391325+0000","last_peered":"2026-02-23T13:08:25.391325+0000","last_clean":"2026-02-23T13:08:25.391325+0000","last_became_active":"2026-02-23T13:07:31.232454+0000","last_became_peered":"2026-02-23T13:07:31.232454+0000","last_unstale":"2026-02-23T13:08:25.391325+0000","last_undegraded":"2026-02-23T13:08:25.391325+0000","last_fullsized":"2026-02-23T13:08:25.391325+0000","mapping_epoch":29,"log_start":"31'4000","ondisk_log_start":"31'4000","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_clean_scrub_stamp":"2026-02-23T13:07:30.224555+0000","objects_scrubbed":0,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T13:08:31.879194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":115081216,"num_objects":1756,"num_object_clones":0,"num_object_copies":5268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1756,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14048,"num_write_kb":112384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.d","version":"46'14232","reported_seq":15514,"reported_epoch":46,"state":"active+clean","last_fresh":"2026-02-23T13:08:25.383768+0000","last_change":"2026-02-23T13:08:22.975126+0000","last_active":"2026-02-23T13:08:25.383768+0000","last_peered":"2026-02-23T13:08:25.383768+0000","last_clean":"2026-02-23T13:08:25.383768+0000","last_became_active":"2026-02-23T13:08:22.968695+0000","last_became_peered":"2026-02-23T13:08:22.968695+0000","last_unstale":"2026-02-23T13:08:25.383768+0000","last_undegraded":"2026-02-23T13:08:25.383768+0000","last_fullsized":"2026-02-23T13:08:25.383768+0000","mapping_epoch":43,"log_start":"31'4200","ondisk_log_start":"31'4200","created":29,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_clean_scrub_stamp":"2026-02-23T13:07:30.224555+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T13:08:51.357883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":116588544,"num_objects":1779,"num_object_clones":0,"num_object_copies":5337,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1779,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14232,"num_write_kb":113856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":616,"num_bytes_recovered":40263680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"46'14416","reported_seq":15722,"reported_epoch":46,"state":"active+clean","last_fresh":"2026-02-23T13:08:25.369432+0000","last_change":"2026-02-23T13:08:19.971521+0000","last_active":"2026-02-23T13:08:25.369432+0000","last_peered":"2026-02-23T13:08:25.369432+0000","last_clean":"2026-02-23T13:08:25.369432+0000","last_became_active":"2026-02-23T13:08:19.965066+0000","last_became_peered":"2026-02-23T13:08:19.965066+0000","last_unstale":"2026-02-23T13:08:25.369432+0000","last_undegraded":"2026-02-23T13:08:25.369432+0000","last_fullsized":"2026-02-23T13:08:25.369432+0000","mapping_epoch":40,"log_start":"31'4400","ondisk_log_start":"31'4400","created":29,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_clean_scrub_stamp":"2026-02-23T13:07:30.224555+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T13:08:34.075016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":118095872,"num_objects":1802,"num_object_clones":0,"num_object_copies":5406,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1802,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14416,"num_write_kb":115328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":625,"num_bytes_recovered":40861696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"28'99","reported_seq":78,"reported_epoch":46,"state":"active+clean","last_fresh":"2026-02-23T13:08:24.964284+0000","last_change":"2026-02-23T13:08:12.277636+0000","last_active":"2026-02-23T13:08:24.964284+0000","last_peered":"2026-02-23T13:08:24.964284+0000","last_clean":"2026-02-23T13:08:24.964284+0000","last_became_active":"2026-02-23T13:08:12.277258+0000","last_became_peered":"2026-02-23T13:08:12.277258+0000","last_unstale":"2026-02-23T13:08:24.964284+0000","last_undegraded":"2026-02-23T13:08:24.964284+0000","last_fullsized":"2026-02-23T13:08:24.964284+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:07:05.619208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:07:05.619208+0000","last_clean_scrub_stamp":"2026-02-23T13:07:05.619208+0000","objects_scrubbed":0,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T13:08:29.473610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":122,"num_read_kb":228,"num_write":135,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"46'14176","reported_seq":15392,"reported_epoch":46,"state":"active+clean","last_fresh":"2026-02-23T13:08:25.269076+0000","last_change":"2026-02-23T13:08:20.590828+0000","last_active":"2026-02-23T13:08:25.269076+0000","last_peered":"2026-02-23T13:08:25.269076+0000","last_clean":"2026-02-23T13:08:25.269076+0000","last_became_active":"2026-02-23T13:08:18.297689+0000","last_became_peered":"2026-02-23T13:08:18.297689+0000","last_unstale":"2026-02-23T13:08:25.269076+0000","last_undegraded":"2026-02-23T13:08:25.269076+0000","last_fullsized":"2026-02-23T13:08:25.269076+0000","mapping_epoch":38,"log_start":"31'4100","ondisk_log_start":"31'4100","created":29,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_clean_scrub_stamp":"2026-02-23T13:07:30.224555+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T13:08:58.048174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":116129792,"num_objects":1772,"num_object_clones":0,"num_object_copies":5316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1772,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14176,"num_write_kb":113408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":580,"num_bytes_recovered":37953536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"46'14078","reported_seq":15356,"reported_epoch":46,"state":"active+clean","last_fresh":"2026-02-23T13:08:25.281503+0000","last_change":"2026-02-23T13:08:18.292955+0000","last_active":"2026-02-23T13:08:25.281503+0000","last_peered":"2026-02-23T13:08:25.281503+0000","last_clean":"2026-02-23T13:08:25.281503+0000","last_became_active":"2026-02-23T13:08:18.289580+0000","last_became_peered":"2026-02-23T13:08:18.289580+0000","last_unstale":"2026-02-23T13:08:25.281503+0000","last_undegraded":"2026-02-23T13:08:25.281503+0000","last_fullsized":"2026-02-23T13:08:25.281503+0000","mapping_epoch":38,"log_start":"31'4000","ondisk_log_start":"31'4000","created":29,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_clean_scrub_stamp":"2026-02-23T13:07:30.224555+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T13:08:46.511057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":115326976,"num_objects":1760,"num_object_clones":0,"num_object_copies":5280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1760,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14078,"num_write_kb":112624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":616,"num_bytes_recovered":40247296,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"46'14104","reported_seq":15397,"reported_epoch":46,"state":"active+clean","last_fresh":"2026-02-23T13:08:25.352365+0000","last_change":"2026-02-23T13:08:24.971084+0000","last_active":"2026-02-23T13:08:25.352365+0000","last_peered":"2026-02-23T13:08:25.352365+0000","last_clean":"2026-02-23T13:08:25.352365+0000","last_became_active":"2026-02-23T13:08:24.965478+0000","last_became_peered":"2026-02-23T13:08:24.965478+0000","last_unstale":"2026-02-23T13:08:25.352365+0000","last_undegraded":"2026-02-23T13:08:25.352365+0000","last_fullsized":"2026-02-23T13:08:25.352365+0000","mapping_epoch":45,"log_start":"31'4100","ondisk_log_start":"31'4100","created":29,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_clean_scrub_stamp":"2026-02-23T13:07:30.224555+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T13:08:39.312642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":115539968,"num_objects":1763,"num_object_clones":0,"num_object_copies":5289,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1763,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14104,"num_write_kb":112832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":616,"num_bytes_recovered":40370176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"46'14248","reported_seq":15472,"reported_epoch":46,"state":"active+clean","last_fresh":"2026-02-23T13:08:25.235154+0000","last_change":"2026-02-23T13:08:24.970596+0000","last_active":"2026-02-23T13:08:25.235154+0000","last_peered":"2026-02-23T13:08:25.235154+0000","last_clean":"2026-02-23T13:08:25.235154+0000","last_became_active":"2026-02-23T13:08:24.965589+0000","last_became_peered":"2026-02-23T13:08:24.965589+0000","last_unstale":"2026-02-23T13:08:25.235154+0000","last_undegraded":"2026-02-23T13:08:25.235154+0000","last_fullsized":"2026-02-23T13:08:25.235154+0000","mapping_epoch":45,"log_start":"31'4200","ondisk_log_start":"31'4200","created":29,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_clean_scrub_stamp":"2026-02-23T13:07:30.224555+0000","objects_scrubbed":0,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T13:08:46.466443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":116719616,"num_objects":1781,"num_object_clones":0,"num_object_copies":5343,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1781,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14248,"num_write_kb":113984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":586,"num_bytes_recovered":38354944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"46'14456","reported_seq":14500,"reported_epoch":46,"state":"active+clean","last_fresh":"2026-02-23T13:08:25.384385+0000","last_change":"2026-02-23T13:07:31.234133+0000","last_active":"2026-02-23T13:08:25.384385+0000","last_peered":"2026-02-23T13:08:25.384385+0000","last_clean":"2026-02-23T13:08:25.384385+0000","last_became_active":"2026-02-23T13:07:31.233956+0000","last_became_peered":"2026-02-23T13:07:31.233956+0000","last_unstale":"2026-02-23T13:08:25.384385+0000","last_undegraded":"2026-02-23T13:08:25.384385+0000","last_fullsized":"2026-02-23T13:08:25.384385+0000","mapping_epoch":29,"log_start":"31'4400","ondisk_log_start":"31'4400","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_clean_scrub_stamp":"2026-02-23T13:07:30.224555+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T13:08:47.743470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":118423552,"num_objects":1807,"num_object_clones":0,"num_object_copies":5421,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1807,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14456,"num_write_kb":115648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"46'14488","reported_seq":15788,"reported_epoch":46,"state":"active+clean","last_fresh":"2026-02-23T13:08:25.281888+0000","last_change":"2026-02-23T13:08:20.969777+0000","last_active":"2026-02-23T13:08:25.281888+0000","last_peered":"2026-02-23T13:08:25.281888+0000","last_clean":"2026-02-23T13:08:25.281888+0000","last_became_active":"2026-02-23T13:08:20.966430+0000","last_became_peered":"2026-02-23T13:08:20.966430+0000","last_unstale":"2026-02-23T13:08:25.281888+0000","last_undegraded":"2026-02-23T13:08:25.281888+0000","last_fullsized":"2026-02-23T13:08:25.281888+0000","mapping_epoch":41,"log_start":"31'4400","ondisk_log_start":"31'4400","created":29,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_clean_scrub_stamp":"2026-02-23T13:07:30.224555+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T13:08:53.668706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":118685696,"num_objects":1811,"num_object_clones":0,"num_object_copies":5433,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1811,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14488,"num_write_kb":115904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":624,"num_bytes_recovered":40894464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"46'14378","reported_seq":15689,"reported_epoch":46,"state":"active+clean","last_fresh":"2026-02-23T13:08:25.391168+0000","last_change":"2026-02-23T13:08:22.974935+0000","last_active":"2026-02-23T13:08:25.391168+0000","last_peered":"2026-02-23T13:08:25.391168+0000","last_clean":"2026-02-23T13:08:25.391168+0000","last_became_active":"2026-02-23T13:08:22.968536+0000","last_became_peered":"2026-02-23T13:08:22.968536+0000","last_unstale":"2026-02-23T13:08:25.391168+0000","last_undegraded":"2026-02-23T13:08:25.391168+0000","last_fullsized":"2026-02-23T13:08:25.391168+0000","mapping_epoch":43,"log_start":"31'4300","ondisk_log_start":"31'4300","created":29,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_clean_scrub_stamp":"2026-02-23T13:07:30.224555+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T13:08:53.561645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117784576,"num_objects":1798,"num_object_clones":0,"num_object_copies":5394,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1798,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14378,"num_write_kb":115024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":625,"num_bytes_recovered":40960000,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"46'13920","reported_seq":15145,"reported_epoch":46,"state":"active+clean","last_fresh":"2026-02-23T13:08:25.234183+0000","last_change":"2026-02-23T13:08:21.971120+0000","last_active":"2026-02-23T13:08:25.234183+0000","last_peered":"2026-02-23T13:08:25.234183+0000","last_clean":"2026-02-23T13:08:25.234183+0000","last_became_active":"2026-02-23T13:08:21.967481+0000","last_became_peered":"2026-02-23T13:08:21.967481+0000","last_unstale":"2026-02-23T13:08:25.234183+0000","last_undegraded":"2026-02-23T13:08:25.234183+0000","last_fullsized":"2026-02-23T13:08:25.234183+0000","mapping_epoch":42,"log_start":"31'3900","ondisk_log_start":"31'3900","created":29,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_clean_scrub_stamp":"2026-02-23T13:07:30.224555+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T13:08:31.921830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":114032640,"num_objects":1740,"num_object_clones":0,"num_object_copies":5220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1740,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13920,"num_write_kb":111360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":585,"num_bytes_recovered":38338560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.8","version":"46'14336","reported_seq":15158,"reported_epoch":46,"state":"active+clean","last_fresh":"2026-02-23T13:08:25.268279+0000","last_change":"2026-02-23T13:08:18.965066+0000","last_active":"2026-02-23T13:08:25.268279+0000","last_peered":"2026-02-23T13:08:25.268279+0000","last_clean":"2026-02-23T13:08:25.268279+0000","last_became_active":"2026-02-23T13:08:18.961857+0000","last_became_peered":"2026-02-23T13:08:18.961857+0000","last_unstale":"2026-02-23T13:08:25.268279+0000","last_undegraded":"2026-02-23T13:08:25.268279+0000","last_fullsized":"2026-02-23T13:08:25.268279+0000","mapping_epoch":39,"log_start":"31'4300","ondisk_log_start":"31'4300","created":29,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_clean_scrub_stamp":"2026-02-23T13:07:30.224555+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T13:08:45.827389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117440512,"num_objects":1792,"num_object_clones":0,"num_object_copies":5376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1792,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14336,"num_write_kb":114688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":578,"num_bytes_recovered":37879808,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"46'14731","reported_seq":16006,"reported_epoch":46,"state":"active+clean","last_fresh":"2026-02-23T13:08:25.384734+0000","last_change":"2026-02-23T13:08:23.972718+0000","last_active":"2026-02-23T13:08:25.384734+0000","last_peered":"2026-02-23T13:08:25.384734+0000","last_clean":"2026-02-23T13:08:25.384734+0000","last_became_active":"2026-02-23T13:08:23.964827+0000","last_became_peered":"2026-02-23T13:08:23.964827+0000","last_unstale":"2026-02-23T13:08:25.384734+0000","last_undegraded":"2026-02-23T13:08:25.384734+0000","last_fullsized":"2026-02-23T13:08:25.384734+0000","mapping_epoch":44,"log_start":"31'4700","ondisk_log_start":"31'4700","created":29,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_clean_scrub_stamp":"2026-02-23T13:07:30.224555+0000","objects_scrubbed":0,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T13:08:54.170047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":120676352,"num_objects":1842,"num_object_clones":0,"num_object_copies":5526,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1842,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14731,"num_write_kb":117848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":610,"num_bytes_recovered":39804928,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.a","version":"46'14616","reported_seq":15872,"reported_epoch":46,"state":"active+clean","last_fresh":"2026-02-23T13:08:25.382873+0000","last_change":"2026-02-23T13:08:20.999336+0000","last_active":"2026-02-23T13:08:25.382873+0000","last_peered":"2026-02-23T13:08:25.382873+0000","last_clean":"2026-02-23T13:08:25.382873+0000","last_became_active":"2026-02-23T13:08:20.992985+0000","last_became_peered":"2026-02-23T13:08:20.992985+0000","last_unstale":"2026-02-23T13:08:25.382873+0000","last_undegraded":"2026-02-23T13:08:25.382873+0000","last_fullsized":"2026-02-23T13:08:25.382873+0000","mapping_epoch":41,"log_start":"31'4600","ondisk_log_start":"31'4600","created":29,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_clean_scrub_stamp":"2026-02-23T13:07:30.224555+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T13:08:47.634555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":119734272,"num_objects":1827,"num_object_clones":0,"num_object_copies":5481,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1827,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14616,"num_write_kb":116928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":598,"num_bytes_recovered":39133184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.b","version":"46'14824","reported_seq":15879,"reported_epoch":46,"state":"active+clean","last_fresh":"2026-02-23T13:08:25.264478+0000","last_change":"2026-02-23T13:08:23.971015+0000","last_active":"2026-02-23T13:08:25.264478+0000","last_peered":"2026-02-23T13:08:25.264478+0000","last_clean":"2026-02-23T13:08:25.264478+0000","last_became_active":"2026-02-23T13:08:23.965643+0000","last_became_peered":"2026-02-23T13:08:23.965643+0000","last_unstale":"2026-02-23T13:08:25.264478+0000","last_undegraded":"2026-02-23T13:08:25.264478+0000","last_fullsized":"2026-02-23T13:08:25.264478+0000","mapping_epoch":44,"log_start":"31'4800","ondisk_log_start":"31'4800","created":29,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_clean_scrub_stamp":"2026-02-23T13:07:30.224555+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T13:08:33.664083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":121438208,"num_objects":1853,"num_object_clones":0,"num_object_copies":5559,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1853,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14824,"num_write_kb":118592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":628,"num_bytes_recovered":41099264,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":1881694208,"num_objects":28714,"num_object_clones":0,"num_object_copies":86142,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28714,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":229699,"num_write_kb":1837592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8494,"num_bytes_recovered":555941888,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":5641314304,"data_stored":5641314304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160699,"ondisk_log_size":160699,"up":48,"acting":48,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":122,"num_read_kb":228,"num_write":135,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":35,"seq":150323855364,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":734003200,"kb_used":1730956,"kb_used_data":1602232,"kb_used_omap":12,"kb_used_meta":128691,"kb_avail":732272244,"statfs":{"total":751619276800,"available":749846777856,"internally_reserved":0,"allocated":1640685568,"data_stored":1640398089,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12872,"internal_metadata":131780024},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,56],"upper_bound":2048},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411347,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":734003200,"kb_used":1539468,"kb_used_data":1259832,"kb_used_omap":21,"kb_used_meta":279594,"kb_avail":732463732,"statfs":{"total":751619276800,"available":750042861568,"internally_reserved":0,"allocated":1290067968,"data_stored":1289780489,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":22457,"internal_metadata":286304327},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574869,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":734003200,"kb_used":1661576,"kb_used_data":1363892,"kb_used_omap":14,"kb_used_meta":297649,"kb_avail":732341624,"statfs":{"total":751619276800,"available":749917822976,"internally_reserved":0,"allocated":1396625408,"data_stored":1396341481,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14645,"internal_metadata":304793291},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[6,2,10,13,15,8,8,8,0,0,10],"upper_bound":2048},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705686,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":734003200,"kb_used":1540492,"kb_used_data":1260856,"kb_used_omap":25,"kb_used_meta":279590,"kb_avail":732462708,"statfs":{"total":751619276800,"available":750041812992,"internally_reserved":0,"allocated":1291116544,"data_stored":1290829065,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":25712,"internal_metadata":286301072},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1295458304,"data_stored":1295458304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1404813312,"data_stored":1404813312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1294245888,"data_stored":1294245888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1646796800,"data_stored":1646796800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-23T13:08:27.326 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 55 128 237411 237283 33.7024 31.6562 0.0134612 0.0296589 2026-02-23T13:08:27.784 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:08:27.932 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:08:28.053 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:08:28.052+0000 7f33c62b1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:08:28.054 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:08:28.052+0000 7f33c62b1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:08:28.069 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:08:28.069 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:08:28.070 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:08:27 trial068 ceph-mon[38805]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 3615/84054 objects degraded (4.301%), 8 pgs degraded) 2026-02-23T13:08:28.070 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:08:27 trial068 ceph-mon[38805]: Cluster is now healthy 2026-02-23T13:08:28.070 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:08:27 trial068 ceph-mon[38805]: from='client.14417 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2026-02-23T13:08:28.083 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:08:28.083 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:08:28.098 INFO:teuthology.orchestra.run.trial068.stdout:osd.2: {} 2026-02-23T13:08:28.098 INFO:teuthology.orchestra.run.trial068.stderr:osd.2: osd_enable_op_tracker = '' 2026-02-23T13:08:28.111 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:08:28.112 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:08:28.326 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 56 128 241188 241060 33.6275 29.5078 0.0227816 0.0297235 2026-02-23T13:08:29.326 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 57 128 245024 244896 33.5632 29.9688 0.029356 0.0297807 2026-02-23T13:08:29.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:08:28 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/4282214763' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:08:29.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:08:28 trial068 ceph-mon[38805]: pgmap v53: 17 pgs: 17 active+clean; 1.8 GiB data, 6.2 GiB used, 2.7 TiB / 2.7 TiB avail; 7.3 MiB/s wr, 929 op/s; 38 MiB/s, 602 objects/s recovering 2026-02-23T13:08:30.073 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph pg dump --format=json 2026-02-23T13:08:30.157 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:08:30.234 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:08:30.259 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:08:29 trial068 ceph-mon[38805]: 1.0 scrub starts 2026-02-23T13:08:30.259 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:08:29 trial068 ceph-mon[38805]: 1.0 scrub ok 2026-02-23T13:08:30.305 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:08:30.327 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 58 128 248699 248571 33.479 28.7109 0.0549594 0.029853 2026-02-23T13:08:30.363 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:08:30.361+0000 7f8d6381d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:08:30.363 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:08:30.362+0000 7f8d6381d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:08:30.439 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:08:30.438+0000 7fbe0a183640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:08:30.440 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:08:30.438+0000 7fbe0a183640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:08:30.455 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:08:30.455 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:08:30.473 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:08:30.473 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:08:30.487 INFO:teuthology.orchestra.run.trial068.stdout:osd.2: {} 2026-02-23T13:08:30.487 INFO:teuthology.orchestra.run.trial068.stderr:osd.2: osd_enable_op_tracker = '' 2026-02-23T13:08:30.501 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:08:30.501 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:08:30.513 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T13:08:30.513 INFO:teuthology.orchestra.run.trial068.stderr:dumped all 2026-02-23T13:08:30.559 INFO:teuthology.orchestra.run.trial068.stdout:{"pg_ready":true,"pg_map":{"version":54,"stamp":"2026-02-23T13:08:30.231877+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1882284576,"num_objects":28716,"num_object_clones":0,"num_object_copies":86148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28716,"num_whiteouts":0,"num_read":122,"num_read_kb":228,"num_write":229834,"num_write_kb":1839742,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8496,"num_bytes_recovered":556532256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160798,"ondisk_log_size":160798,"up":51,"acting":51,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":51,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":2936012800,"kb_used":6472492,"kb_used_data":5486812,"kb_used_omap":73,"kb_used_meta":985526,"kb_avail":2929540308,"statfs":{"total":3006477107200,"available":2999849275392,"internally_reserved":0,"allocated":5618495488,"data_stored":5617349124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":75686,"internal_metadata":1009178714},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[6,2,10,13,15,8,8,8,0,0,66],"upper_bound":2048},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":45694976,"num_objects":698,"num_object_clones":0,"num_object_copies":2094,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-3615,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":698,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5578,"num_write_kb":44624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3615,"num_bytes_recovered":236855296,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.266007"},"pg_stats":[{"pgid":"2.f","version":"46'14648","reported_seq":15926,"reported_epoch":46,"state":"active+clean","last_fresh":"2026-02-23T13:08:25.228499+0000","last_change":"2026-02-23T13:08:21.973471+0000","last_active":"2026-02-23T13:08:25.228499+0000","last_peered":"2026-02-23T13:08:25.228499+0000","last_clean":"2026-02-23T13:08:25.228499+0000","last_became_active":"2026-02-23T13:08:21.967829+0000","last_became_peered":"2026-02-23T13:08:21.967829+0000","last_unstale":"2026-02-23T13:08:25.228499+0000","last_undegraded":"2026-02-23T13:08:25.228499+0000","last_fullsized":"2026-02-23T13:08:25.228499+0000","mapping_epoch":42,"log_start":"31'4600","ondisk_log_start":"31'4600","created":29,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_clean_scrub_stamp":"2026-02-23T13:07:30.224555+0000","objects_scrubbed":0,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T13:08:36.049688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":119996416,"num_objects":1831,"num_object_clones":0,"num_object_copies":5493,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1831,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14648,"num_write_kb":117184,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":607,"num_bytes_recovered":39780352,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.e","version":"46'14048","reported_seq":14092,"reported_epoch":46,"state":"active+clean","last_fresh":"2026-02-23T13:08:25.391325+0000","last_change":"2026-02-23T13:07:31.233069+0000","last_active":"2026-02-23T13:08:25.391325+0000","last_peered":"2026-02-23T13:08:25.391325+0000","last_clean":"2026-02-23T13:08:25.391325+0000","last_became_active":"2026-02-23T13:07:31.232454+0000","last_became_peered":"2026-02-23T13:07:31.232454+0000","last_unstale":"2026-02-23T13:08:25.391325+0000","last_undegraded":"2026-02-23T13:08:25.391325+0000","last_fullsized":"2026-02-23T13:08:25.391325+0000","mapping_epoch":29,"log_start":"31'4000","ondisk_log_start":"31'4000","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_clean_scrub_stamp":"2026-02-23T13:07:30.224555+0000","objects_scrubbed":0,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T13:08:31.879194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":115081216,"num_objects":1756,"num_object_clones":0,"num_object_copies":5268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1756,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14048,"num_write_kb":112384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.d","version":"46'14232","reported_seq":15514,"reported_epoch":46,"state":"active+clean","last_fresh":"2026-02-23T13:08:25.383768+0000","last_change":"2026-02-23T13:08:22.975126+0000","last_active":"2026-02-23T13:08:25.383768+0000","last_peered":"2026-02-23T13:08:25.383768+0000","last_clean":"2026-02-23T13:08:25.383768+0000","last_became_active":"2026-02-23T13:08:22.968695+0000","last_became_peered":"2026-02-23T13:08:22.968695+0000","last_unstale":"2026-02-23T13:08:25.383768+0000","last_undegraded":"2026-02-23T13:08:25.383768+0000","last_fullsized":"2026-02-23T13:08:25.383768+0000","mapping_epoch":43,"log_start":"31'4200","ondisk_log_start":"31'4200","created":29,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_clean_scrub_stamp":"2026-02-23T13:07:30.224555+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T13:08:51.357883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":116588544,"num_objects":1779,"num_object_clones":0,"num_object_copies":5337,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1779,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14232,"num_write_kb":113856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":616,"num_bytes_recovered":40263680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"46'14416","reported_seq":15722,"reported_epoch":46,"state":"active+clean","last_fresh":"2026-02-23T13:08:25.369432+0000","last_change":"2026-02-23T13:08:19.971521+0000","last_active":"2026-02-23T13:08:25.369432+0000","last_peered":"2026-02-23T13:08:25.369432+0000","last_clean":"2026-02-23T13:08:25.369432+0000","last_became_active":"2026-02-23T13:08:19.965066+0000","last_became_peered":"2026-02-23T13:08:19.965066+0000","last_unstale":"2026-02-23T13:08:25.369432+0000","last_undegraded":"2026-02-23T13:08:25.369432+0000","last_fullsized":"2026-02-23T13:08:25.369432+0000","mapping_epoch":40,"log_start":"31'4400","ondisk_log_start":"31'4400","created":29,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_clean_scrub_stamp":"2026-02-23T13:07:30.224555+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T13:08:34.075016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":118095872,"num_objects":1802,"num_object_clones":0,"num_object_copies":5406,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1802,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14416,"num_write_kb":115328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":625,"num_bytes_recovered":40861696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"28'99","reported_seq":78,"reported_epoch":46,"state":"active+clean","last_fresh":"2026-02-23T13:08:24.964284+0000","last_change":"2026-02-23T13:08:12.277636+0000","last_active":"2026-02-23T13:08:24.964284+0000","last_peered":"2026-02-23T13:08:24.964284+0000","last_clean":"2026-02-23T13:08:24.964284+0000","last_became_active":"2026-02-23T13:08:12.277258+0000","last_became_peered":"2026-02-23T13:08:12.277258+0000","last_unstale":"2026-02-23T13:08:24.964284+0000","last_undegraded":"2026-02-23T13:08:24.964284+0000","last_fullsized":"2026-02-23T13:08:24.964284+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:07:05.619208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:07:05.619208+0000","last_clean_scrub_stamp":"2026-02-23T13:07:05.619208+0000","objects_scrubbed":0,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T13:08:29.473610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":122,"num_read_kb":228,"num_write":135,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"46'14176","reported_seq":15392,"reported_epoch":46,"state":"active+clean","last_fresh":"2026-02-23T13:08:25.269076+0000","last_change":"2026-02-23T13:08:20.590828+0000","last_active":"2026-02-23T13:08:25.269076+0000","last_peered":"2026-02-23T13:08:25.269076+0000","last_clean":"2026-02-23T13:08:25.269076+0000","last_became_active":"2026-02-23T13:08:18.297689+0000","last_became_peered":"2026-02-23T13:08:18.297689+0000","last_unstale":"2026-02-23T13:08:25.269076+0000","last_undegraded":"2026-02-23T13:08:25.269076+0000","last_fullsized":"2026-02-23T13:08:25.269076+0000","mapping_epoch":38,"log_start":"31'4100","ondisk_log_start":"31'4100","created":29,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_clean_scrub_stamp":"2026-02-23T13:07:30.224555+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T13:08:58.048174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":116129792,"num_objects":1772,"num_object_clones":0,"num_object_copies":5316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1772,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14176,"num_write_kb":113408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":580,"num_bytes_recovered":37953536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"46'14078","reported_seq":15356,"reported_epoch":46,"state":"active+clean","last_fresh":"2026-02-23T13:08:25.281503+0000","last_change":"2026-02-23T13:08:18.292955+0000","last_active":"2026-02-23T13:08:25.281503+0000","last_peered":"2026-02-23T13:08:25.281503+0000","last_clean":"2026-02-23T13:08:25.281503+0000","last_became_active":"2026-02-23T13:08:18.289580+0000","last_became_peered":"2026-02-23T13:08:18.289580+0000","last_unstale":"2026-02-23T13:08:25.281503+0000","last_undegraded":"2026-02-23T13:08:25.281503+0000","last_fullsized":"2026-02-23T13:08:25.281503+0000","mapping_epoch":38,"log_start":"31'4000","ondisk_log_start":"31'4000","created":29,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_clean_scrub_stamp":"2026-02-23T13:07:30.224555+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T13:08:46.511057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":115326976,"num_objects":1760,"num_object_clones":0,"num_object_copies":5280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1760,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14078,"num_write_kb":112624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":616,"num_bytes_recovered":40247296,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"46'14104","reported_seq":15397,"reported_epoch":46,"state":"active+clean","last_fresh":"2026-02-23T13:08:25.352365+0000","last_change":"2026-02-23T13:08:24.971084+0000","last_active":"2026-02-23T13:08:25.352365+0000","last_peered":"2026-02-23T13:08:25.352365+0000","last_clean":"2026-02-23T13:08:25.352365+0000","last_became_active":"2026-02-23T13:08:24.965478+0000","last_became_peered":"2026-02-23T13:08:24.965478+0000","last_unstale":"2026-02-23T13:08:25.352365+0000","last_undegraded":"2026-02-23T13:08:25.352365+0000","last_fullsized":"2026-02-23T13:08:25.352365+0000","mapping_epoch":45,"log_start":"31'4100","ondisk_log_start":"31'4100","created":29,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_clean_scrub_stamp":"2026-02-23T13:07:30.224555+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T13:08:39.312642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":115539968,"num_objects":1763,"num_object_clones":0,"num_object_copies":5289,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1763,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14104,"num_write_kb":112832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":616,"num_bytes_recovered":40370176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"46'14248","reported_seq":15472,"reported_epoch":46,"state":"active+clean","last_fresh":"2026-02-23T13:08:25.235154+0000","last_change":"2026-02-23T13:08:24.970596+0000","last_active":"2026-02-23T13:08:25.235154+0000","last_peered":"2026-02-23T13:08:25.235154+0000","last_clean":"2026-02-23T13:08:25.235154+0000","last_became_active":"2026-02-23T13:08:24.965589+0000","last_became_peered":"2026-02-23T13:08:24.965589+0000","last_unstale":"2026-02-23T13:08:25.235154+0000","last_undegraded":"2026-02-23T13:08:25.235154+0000","last_fullsized":"2026-02-23T13:08:25.235154+0000","mapping_epoch":45,"log_start":"31'4200","ondisk_log_start":"31'4200","created":29,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_clean_scrub_stamp":"2026-02-23T13:07:30.224555+0000","objects_scrubbed":0,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T13:08:46.466443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":116719616,"num_objects":1781,"num_object_clones":0,"num_object_copies":5343,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1781,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14248,"num_write_kb":113984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":586,"num_bytes_recovered":38354944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"46'14456","reported_seq":14500,"reported_epoch":46,"state":"active+clean","last_fresh":"2026-02-23T13:08:25.384385+0000","last_change":"2026-02-23T13:07:31.234133+0000","last_active":"2026-02-23T13:08:25.384385+0000","last_peered":"2026-02-23T13:08:25.384385+0000","last_clean":"2026-02-23T13:08:25.384385+0000","last_became_active":"2026-02-23T13:07:31.233956+0000","last_became_peered":"2026-02-23T13:07:31.233956+0000","last_unstale":"2026-02-23T13:08:25.384385+0000","last_undegraded":"2026-02-23T13:08:25.384385+0000","last_fullsized":"2026-02-23T13:08:25.384385+0000","mapping_epoch":29,"log_start":"31'4400","ondisk_log_start":"31'4400","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_clean_scrub_stamp":"2026-02-23T13:07:30.224555+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T13:08:47.743470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":118423552,"num_objects":1807,"num_object_clones":0,"num_object_copies":5421,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1807,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14456,"num_write_kb":115648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"46'14488","reported_seq":15788,"reported_epoch":46,"state":"active+clean","last_fresh":"2026-02-23T13:08:25.281888+0000","last_change":"2026-02-23T13:08:20.969777+0000","last_active":"2026-02-23T13:08:25.281888+0000","last_peered":"2026-02-23T13:08:25.281888+0000","last_clean":"2026-02-23T13:08:25.281888+0000","last_became_active":"2026-02-23T13:08:20.966430+0000","last_became_peered":"2026-02-23T13:08:20.966430+0000","last_unstale":"2026-02-23T13:08:25.281888+0000","last_undegraded":"2026-02-23T13:08:25.281888+0000","last_fullsized":"2026-02-23T13:08:25.281888+0000","mapping_epoch":41,"log_start":"31'4400","ondisk_log_start":"31'4400","created":29,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_clean_scrub_stamp":"2026-02-23T13:07:30.224555+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T13:08:53.668706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":118685696,"num_objects":1811,"num_object_clones":0,"num_object_copies":5433,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1811,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14488,"num_write_kb":115904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":624,"num_bytes_recovered":40894464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"46'14378","reported_seq":15689,"reported_epoch":46,"state":"active+clean","last_fresh":"2026-02-23T13:08:25.391168+0000","last_change":"2026-02-23T13:08:22.974935+0000","last_active":"2026-02-23T13:08:25.391168+0000","last_peered":"2026-02-23T13:08:25.391168+0000","last_clean":"2026-02-23T13:08:25.391168+0000","last_became_active":"2026-02-23T13:08:22.968536+0000","last_became_peered":"2026-02-23T13:08:22.968536+0000","last_unstale":"2026-02-23T13:08:25.391168+0000","last_undegraded":"2026-02-23T13:08:25.391168+0000","last_fullsized":"2026-02-23T13:08:25.391168+0000","mapping_epoch":43,"log_start":"31'4300","ondisk_log_start":"31'4300","created":29,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_clean_scrub_stamp":"2026-02-23T13:07:30.224555+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T13:08:53.561645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117784576,"num_objects":1798,"num_object_clones":0,"num_object_copies":5394,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1798,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14378,"num_write_kb":115024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":625,"num_bytes_recovered":40960000,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"46'13920","reported_seq":15145,"reported_epoch":46,"state":"active+clean","last_fresh":"2026-02-23T13:08:25.234183+0000","last_change":"2026-02-23T13:08:21.971120+0000","last_active":"2026-02-23T13:08:25.234183+0000","last_peered":"2026-02-23T13:08:25.234183+0000","last_clean":"2026-02-23T13:08:25.234183+0000","last_became_active":"2026-02-23T13:08:21.967481+0000","last_became_peered":"2026-02-23T13:08:21.967481+0000","last_unstale":"2026-02-23T13:08:25.234183+0000","last_undegraded":"2026-02-23T13:08:25.234183+0000","last_fullsized":"2026-02-23T13:08:25.234183+0000","mapping_epoch":42,"log_start":"31'3900","ondisk_log_start":"31'3900","created":29,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_clean_scrub_stamp":"2026-02-23T13:07:30.224555+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T13:08:31.921830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":114032640,"num_objects":1740,"num_object_clones":0,"num_object_copies":5220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1740,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13920,"num_write_kb":111360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":585,"num_bytes_recovered":38338560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.8","version":"46'14336","reported_seq":15158,"reported_epoch":46,"state":"active+clean","last_fresh":"2026-02-23T13:08:25.268279+0000","last_change":"2026-02-23T13:08:18.965066+0000","last_active":"2026-02-23T13:08:25.268279+0000","last_peered":"2026-02-23T13:08:25.268279+0000","last_clean":"2026-02-23T13:08:25.268279+0000","last_became_active":"2026-02-23T13:08:18.961857+0000","last_became_peered":"2026-02-23T13:08:18.961857+0000","last_unstale":"2026-02-23T13:08:25.268279+0000","last_undegraded":"2026-02-23T13:08:25.268279+0000","last_fullsized":"2026-02-23T13:08:25.268279+0000","mapping_epoch":39,"log_start":"31'4300","ondisk_log_start":"31'4300","created":29,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_clean_scrub_stamp":"2026-02-23T13:07:30.224555+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T13:08:45.827389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117440512,"num_objects":1792,"num_object_clones":0,"num_object_copies":5376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1792,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14336,"num_write_kb":114688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":578,"num_bytes_recovered":37879808,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"46'14731","reported_seq":16006,"reported_epoch":46,"state":"active+clean","last_fresh":"2026-02-23T13:08:25.384734+0000","last_change":"2026-02-23T13:08:23.972718+0000","last_active":"2026-02-23T13:08:25.384734+0000","last_peered":"2026-02-23T13:08:25.384734+0000","last_clean":"2026-02-23T13:08:25.384734+0000","last_became_active":"2026-02-23T13:08:23.964827+0000","last_became_peered":"2026-02-23T13:08:23.964827+0000","last_unstale":"2026-02-23T13:08:25.384734+0000","last_undegraded":"2026-02-23T13:08:25.384734+0000","last_fullsized":"2026-02-23T13:08:25.384734+0000","mapping_epoch":44,"log_start":"31'4700","ondisk_log_start":"31'4700","created":29,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_clean_scrub_stamp":"2026-02-23T13:07:30.224555+0000","objects_scrubbed":0,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T13:08:54.170047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":120676352,"num_objects":1842,"num_object_clones":0,"num_object_copies":5526,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1842,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14731,"num_write_kb":117848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":610,"num_bytes_recovered":39804928,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.a","version":"46'14616","reported_seq":15872,"reported_epoch":46,"state":"active+clean","last_fresh":"2026-02-23T13:08:25.382873+0000","last_change":"2026-02-23T13:08:20.999336+0000","last_active":"2026-02-23T13:08:25.382873+0000","last_peered":"2026-02-23T13:08:25.382873+0000","last_clean":"2026-02-23T13:08:25.382873+0000","last_became_active":"2026-02-23T13:08:20.992985+0000","last_became_peered":"2026-02-23T13:08:20.992985+0000","last_unstale":"2026-02-23T13:08:25.382873+0000","last_undegraded":"2026-02-23T13:08:25.382873+0000","last_fullsized":"2026-02-23T13:08:25.382873+0000","mapping_epoch":41,"log_start":"31'4600","ondisk_log_start":"31'4600","created":29,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_clean_scrub_stamp":"2026-02-23T13:07:30.224555+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T13:08:47.634555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":119734272,"num_objects":1827,"num_object_clones":0,"num_object_copies":5481,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1827,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14616,"num_write_kb":116928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":598,"num_bytes_recovered":39133184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.b","version":"46'14824","reported_seq":15879,"reported_epoch":46,"state":"active+clean","last_fresh":"2026-02-23T13:08:25.264478+0000","last_change":"2026-02-23T13:08:23.971015+0000","last_active":"2026-02-23T13:08:25.264478+0000","last_peered":"2026-02-23T13:08:25.264478+0000","last_clean":"2026-02-23T13:08:25.264478+0000","last_became_active":"2026-02-23T13:08:23.965643+0000","last_became_peered":"2026-02-23T13:08:23.965643+0000","last_unstale":"2026-02-23T13:08:25.264478+0000","last_undegraded":"2026-02-23T13:08:25.264478+0000","last_fullsized":"2026-02-23T13:08:25.264478+0000","mapping_epoch":44,"log_start":"31'4800","ondisk_log_start":"31'4800","created":29,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_clean_scrub_stamp":"2026-02-23T13:07:30.224555+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T13:08:33.664083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":121438208,"num_objects":1853,"num_object_clones":0,"num_object_copies":5559,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1853,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14824,"num_write_kb":118592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":628,"num_bytes_recovered":41099264,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":1881694208,"num_objects":28714,"num_object_clones":0,"num_object_copies":86142,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28714,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":229699,"num_write_kb":1837592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8494,"num_bytes_recovered":555941888,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":5641314304,"data_stored":5641314304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160699,"ondisk_log_size":160699,"up":48,"acting":48,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":122,"num_read_kb":228,"num_write":135,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":35,"seq":150323855364,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":734003200,"kb_used":1730956,"kb_used_data":1602232,"kb_used_omap":12,"kb_used_meta":128691,"kb_avail":732272244,"statfs":{"total":751619276800,"available":749846777856,"internally_reserved":0,"allocated":1640685568,"data_stored":1640398089,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12872,"internal_metadata":131780024},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,56],"upper_bound":2048},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411347,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":734003200,"kb_used":1539468,"kb_used_data":1259832,"kb_used_omap":21,"kb_used_meta":279594,"kb_avail":732463732,"statfs":{"total":751619276800,"available":750042861568,"internally_reserved":0,"allocated":1290067968,"data_stored":1289780489,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":22457,"internal_metadata":286304327},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574869,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":734003200,"kb_used":1661576,"kb_used_data":1363892,"kb_used_omap":14,"kb_used_meta":297649,"kb_avail":732341624,"statfs":{"total":751619276800,"available":749917822976,"internally_reserved":0,"allocated":1396625408,"data_stored":1396341481,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14645,"internal_metadata":304793291},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[6,2,10,13,15,8,8,8,0,0,10],"upper_bound":2048},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705686,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":734003200,"kb_used":1540492,"kb_used_data":1260856,"kb_used_omap":25,"kb_used_meta":279590,"kb_avail":732462708,"statfs":{"total":751619276800,"available":750041812992,"internally_reserved":0,"allocated":1291116544,"data_stored":1290829065,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":25712,"internal_metadata":286301072},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1295458304,"data_stored":1295458304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1404813312,"data_stored":1404813312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1294245888,"data_stored":1294245888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1646796800,"data_stored":1646796800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-23T13:08:30.560 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2026-02-23T13:08:30.560 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2026-02-23T13:08:30.560 INFO:tasks.thrashosds.thrasher:Killing osd 1, live_osds are [0, 1, 2, 3] 2026-02-23T13:08:30.560 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2026-02-23T13:08:30.560 DEBUG:teuthology.orchestra.run.trial068:> sudo systemctl stop ceph-5f823084-10b8-11f1-be80-d404e6e7d460@osd.1 2026-02-23T13:08:30.844 INFO:journalctl@ceph.osd.1.trial068.stdout:Feb 23 13:08:30 trial068 systemd[1]: Stopping Ceph osd.1 for 5f823084-10b8-11f1-be80-d404e6e7d460... 2026-02-23T13:08:30.844 INFO:journalctl@ceph.osd.1.trial068.stdout:Feb 23 13:08:30 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-1[62702]: 2026-02-23T13:08:30.721+0000 7f72dabcb640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --osd-objectstore=bluestore (PID: 1) UID: 0 2026-02-23T13:08:30.844 INFO:journalctl@ceph.osd.1.trial068.stdout:Feb 23 13:08:30 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-1[62702]: 2026-02-23T13:08:30.721+0000 7f72dabcb640 -1 osd.1 46 *** Got signal Terminated *** 2026-02-23T13:08:30.844 INFO:journalctl@ceph.osd.1.trial068.stdout:Feb 23 13:08:30 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-1[62702]: 2026-02-23T13:08:30.721+0000 7f72dabcb640 -1 osd.1 46 *** Immediate shutdown (osd_fast_shutdown=true) *** 2026-02-23T13:08:31.327 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 59 128 252119 251991 33.3644 26.7188 0.00217922 0.0299302 2026-02-23T13:08:31.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:08:30 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2026-02-23T13:08:31.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:08:30 trial068 ceph-mon[38805]: pgmap v54: 17 pgs: 17 active+clean; 1.8 GiB data, 6.2 GiB used, 2.7 TiB / 2.7 TiB avail; 6.0 MiB/s wr, 767 op/s; 31 MiB/s, 497 objects/s recovering 2026-02-23T13:08:31.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:08:30 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-23T13:08:31.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:08:30 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2026-02-23T13:08:31.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:08:30 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T13:08:31.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:08:30 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2026-02-23T13:08:31.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:08:30 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/203132509' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:08:31.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:08:30 trial068 ceph-mon[38805]: osd.1 marked itself down and dead 2026-02-23T13:08:31.662 INFO:journalctl@ceph.osd.1.trial068.stdout:Feb 23 13:08:31 trial068 podman[81624]: 2026-02-23 13:08:31.4113545 +0000 UTC m=+0.706057892 container died ab0fbdb05622feb410d57a948daf047e5e7127f4f86490e19ac6074ff70c0c07 (image=quay.ceph.io/ceph-ci/ceph@sha256:49621faaa845c39b2aae72e4fa24123ee9b2ac3fb8ae834753907068ee2972ea, name=ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-1, org.opencontainers.image.documentation=https://docs.ceph.com/, FROM_IMAGE=quay.io/centos/centos:stream9, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.41.3, OSD_FLAVOR=default, org.opencontainers.image.authors=Ceph Release Team , CEPH_REF=wip-shraddhaag-enable-cephadm-tests, CEPH_SHA1=e371433fcd1390a0b82cfb18effc422c1f5d8c1d, org.label-schema.build-date=20260216, org.label-schema.vendor=CentOS, GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfsganesha-5/, org.label-schema.license=GPLv2, CEPH_GIT_REPO=https://github.com/ceph/ceph-ci.git, org.label-schema.name=CentOS Stream 9 Base Image) 2026-02-23T13:08:32.245 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:08:31 trial068 ceph-mon[38805]: from='client.14421 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2026-02-23T13:08:32.245 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:08:31 trial068 ceph-mon[38805]: Health check failed: 1 osds down (OSD_DOWN) 2026-02-23T13:08:32.245 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:08:31 trial068 ceph-mon[38805]: osdmap e47: 4 total, 3 up, 4 in 2026-02-23T13:08:32.246 INFO:journalctl@ceph.osd.1.trial068.stdout:Feb 23 13:08:31 trial068 podman[81624]: 2026-02-23 13:08:31.975737829 +0000 UTC m=+1.270441221 container remove ab0fbdb05622feb410d57a948daf047e5e7127f4f86490e19ac6074ff70c0c07 (image=quay.ceph.io/ceph-ci/ceph@sha256:49621faaa845c39b2aae72e4fa24123ee9b2ac3fb8ae834753907068ee2972ea, name=ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-1, ceph=True, org.opencontainers.image.authors=Ceph Release Team , io.buildah.version=1.41.3, org.label-schema.vendor=CentOS, FROM_IMAGE=quay.io/centos/centos:stream9, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_GIT_REPO=https://github.com/ceph/ceph-ci.git, CEPH_SHA1=e371433fcd1390a0b82cfb18effc422c1f5d8c1d, OSD_FLAVOR=default, org.label-schema.build-date=20260216, CEPH_REF=wip-shraddhaag-enable-cephadm-tests, org.label-schema.name=CentOS Stream 9 Base Image, org.opencontainers.image.documentation=https://docs.ceph.com/, GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfsganesha-5/) 2026-02-23T13:08:32.246 INFO:journalctl@ceph.osd.1.trial068.stdout:Feb 23 13:08:31 trial068 bash[81624]: ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-1 2026-02-23T13:08:32.246 INFO:journalctl@ceph.osd.1.trial068.stdout:Feb 23 13:08:32 trial068 podman[81752]: 2026-02-23 13:08:32.095536529 +0000 UTC m=+0.009399050 image pull 63df4b83db6f95a8ba965919b3671cb27173117c23b5c950aaadc8913f56213c quay.ceph.io/ceph-ci/ceph@sha256:49621faaa845c39b2aae72e4fa24123ee9b2ac3fb8ae834753907068ee2972ea 2026-02-23T13:08:32.246 INFO:journalctl@ceph.osd.1.trial068.stdout:Feb 23 13:08:32 trial068 podman[81752]: 2026-02-23 13:08:32.244713125 +0000 UTC m=+0.158575645 container create 99beb1a275a504af87bbf05c563e7b773227a9ed8ed783c647cc7d1d13670c2d (image=quay.ceph.io/ceph-ci/ceph@sha256:49621faaa845c39b2aae72e4fa24123ee9b2ac3fb8ae834753907068ee2972ea, name=ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-1-deactivate, org.label-schema.vendor=CentOS, CEPH_GIT_REPO=https://github.com/ceph/ceph-ci.git, org.label-schema.build-date=20260216, org.label-schema.name=CentOS Stream 9 Base Image, FROM_IMAGE=quay.io/centos/centos:stream9, org.label-schema.license=GPLv2, OSD_FLAVOR=default, org.opencontainers.image.documentation=https://docs.ceph.com/, CEPH_SHA1=e371433fcd1390a0b82cfb18effc422c1f5d8c1d, ceph=True, GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfsganesha-5/, CEPH_REF=wip-shraddhaag-enable-cephadm-tests, org.opencontainers.image.authors=Ceph Release Team , io.buildah.version=1.41.3, org.label-schema.schema-version=1.0) 2026-02-23T13:08:32.327 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T13:08:32.327170+0000 min lat: 0.000727387 max lat: 2.47295 avg lat: 0.0303414 2026-02-23T13:08:32.327 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T13:08:32.328 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 60 128 252322 252194 32.8347 1.58594 0.0112047 0.0303414 2026-02-23T13:08:32.503 INFO:journalctl@ceph.osd.1.trial068.stdout:Feb 23 13:08:32 trial068 podman[81752]: 2026-02-23 13:08:32.323490642 +0000 UTC m=+0.237353152 container init 99beb1a275a504af87bbf05c563e7b773227a9ed8ed783c647cc7d1d13670c2d (image=quay.ceph.io/ceph-ci/ceph@sha256:49621faaa845c39b2aae72e4fa24123ee9b2ac3fb8ae834753907068ee2972ea, name=ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-1-deactivate, GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfsganesha-5/, org.opencontainers.image.authors=Ceph Release Team , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, CEPH_REF=wip-shraddhaag-enable-cephadm-tests, io.buildah.version=1.41.3, org.label-schema.vendor=CentOS, org.opencontainers.image.documentation=https://docs.ceph.com/, CEPH_SHA1=e371433fcd1390a0b82cfb18effc422c1f5d8c1d, FROM_IMAGE=quay.io/centos/centos:stream9, org.label-schema.license=GPLv2, CEPH_GIT_REPO=https://github.com/ceph/ceph-ci.git, org.label-schema.build-date=20260216, OSD_FLAVOR=default) 2026-02-23T13:08:32.503 INFO:journalctl@ceph.osd.1.trial068.stdout:Feb 23 13:08:32 trial068 podman[81752]: 2026-02-23 13:08:32.326345258 +0000 UTC m=+0.240207768 container start 99beb1a275a504af87bbf05c563e7b773227a9ed8ed783c647cc7d1d13670c2d (image=quay.ceph.io/ceph-ci/ceph@sha256:49621faaa845c39b2aae72e4fa24123ee9b2ac3fb8ae834753907068ee2972ea, name=ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-1-deactivate, org.opencontainers.image.documentation=https://docs.ceph.com/, CEPH_REF=wip-shraddhaag-enable-cephadm-tests, org.label-schema.vendor=CentOS, org.opencontainers.image.authors=Ceph Release Team , org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.license=GPLv2, io.buildah.version=1.41.3, OSD_FLAVOR=default, FROM_IMAGE=quay.io/centos/centos:stream9, CEPH_SHA1=e371433fcd1390a0b82cfb18effc422c1f5d8c1d, CEPH_GIT_REPO=https://github.com/ceph/ceph-ci.git, GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfsganesha-5/, org.label-schema.schema-version=1.0, org.label-schema.build-date=20260216) 2026-02-23T13:08:32.504 INFO:journalctl@ceph.osd.1.trial068.stdout:Feb 23 13:08:32 trial068 podman[81752]: 2026-02-23 13:08:32.326754317 +0000 UTC m=+0.240616837 container attach 99beb1a275a504af87bbf05c563e7b773227a9ed8ed783c647cc7d1d13670c2d (image=quay.ceph.io/ceph-ci/ceph@sha256:49621faaa845c39b2aae72e4fa24123ee9b2ac3fb8ae834753907068ee2972ea, name=ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-1-deactivate, org.label-schema.vendor=CentOS, CEPH_REF=wip-shraddhaag-enable-cephadm-tests, org.label-schema.license=GPLv2, CEPH_GIT_REPO=https://github.com/ceph/ceph-ci.git, GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfsganesha-5/, org.opencontainers.image.authors=Ceph Release Team , org.opencontainers.image.documentation=https://docs.ceph.com/, CEPH_SHA1=e371433fcd1390a0b82cfb18effc422c1f5d8c1d, FROM_IMAGE=quay.io/centos/centos:stream9, org.label-schema.schema-version=1.0, org.label-schema.build-date=20260216, OSD_FLAVOR=default, io.buildah.version=1.41.3, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True) 2026-02-23T13:08:32.504 INFO:journalctl@ceph.osd.1.trial068.stdout:Feb 23 13:08:32 trial068 podman[81752]: 2026-02-23 13:08:32.469697084 +0000 UTC m=+0.383559604 container died 99beb1a275a504af87bbf05c563e7b773227a9ed8ed783c647cc7d1d13670c2d (image=quay.ceph.io/ceph-ci/ceph@sha256:49621faaa845c39b2aae72e4fa24123ee9b2ac3fb8ae834753907068ee2972ea, name=ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-1-deactivate, CEPH_REF=wip-shraddhaag-enable-cephadm-tests, org.label-schema.build-date=20260216, org.opencontainers.image.documentation=https://docs.ceph.com/, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.license=GPLv2, CEPH_GIT_REPO=https://github.com/ceph/ceph-ci.git, GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfsganesha-5/, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.opencontainers.image.authors=Ceph Release Team , io.buildah.version=1.41.3, CEPH_SHA1=e371433fcd1390a0b82cfb18effc422c1f5d8c1d, FROM_IMAGE=quay.io/centos/centos:stream9, OSD_FLAVOR=default) 2026-02-23T13:08:32.504 INFO:journalctl@ceph.osd.1.trial068.stdout:Feb 23 13:08:32 trial068 podman[81752]: 2026-02-23 13:08:32.483094739 +0000 UTC m=+0.396957259 container remove 99beb1a275a504af87bbf05c563e7b773227a9ed8ed783c647cc7d1d13670c2d (image=quay.ceph.io/ceph-ci/ceph@sha256:49621faaa845c39b2aae72e4fa24123ee9b2ac3fb8ae834753907068ee2972ea, name=ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-1-deactivate, org.opencontainers.image.authors=Ceph Release Team , org.label-schema.build-date=20260216, org.opencontainers.image.documentation=https://docs.ceph.com/, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, OSD_FLAVOR=default, FROM_IMAGE=quay.io/centos/centos:stream9, ceph=True, CEPH_REF=wip-shraddhaag-enable-cephadm-tests, CEPH_GIT_REPO=https://github.com/ceph/ceph-ci.git, GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfsganesha-5/, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_SHA1=e371433fcd1390a0b82cfb18effc422c1f5d8c1d, org.label-schema.vendor=CentOS, io.buildah.version=1.41.3) 2026-02-23T13:08:32.526 DEBUG:teuthology.orchestra.run.trial068:> sudo pkill -f 'journalctl -f -n 0 -u ceph-5f823084-10b8-11f1-be80-d404e6e7d460@osd.1.service' 2026-02-23T13:08:32.546 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:08:32.588 DEBUG:teuthology.orchestra.run:got remote process result: None 2026-02-23T13:08:32.588 INFO:tasks.cephadm.osd.1:Stopped osd.1 2026-02-23T13:08:32.705 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:08:32.824 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:08:32.822+0000 7f088b748640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:08:32.824 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:08:32.822+0000 7f088b748640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:08:32.837 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:08:32.838 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:08:32.839 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.1 2026-02-23T13:08:32.850 INFO:teuthology.orchestra.run.trial068.stdout:osd.2: {} 2026-02-23T13:08:32.851 INFO:teuthology.orchestra.run.trial068.stderr:osd.2: osd_enable_op_tracker = '' 2026-02-23T13:08:32.863 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:08:32.864 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:08:32.900 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:08:32.900 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:08:33.327 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 61 128 256725 256597 32.8603 34.3984 0.0136924 0.0304219 2026-02-23T13:08:33.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:08:33 trial068 ceph-mon[38805]: pgmap v56: 17 pgs: 8 stale+active+clean, 9 active+clean; 1.9 GiB data, 6.7 GiB used, 2.7 TiB / 2.7 TiB avail; 24 MiB/s wr, 3.07k op/s; 28 MiB/s, 451 objects/s recovering 2026-02-23T13:08:33.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:08:33 trial068 ceph-mon[38805]: osdmap e48: 4 total, 3 up, 4 in 2026-02-23T13:08:33.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:08:33 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2514299025' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:08:34.266 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:08:33 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:08:33.972Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:08:34.327 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 62 128 261082 260954 32.8793 34.0391 0.0156342 0.0304021 2026-02-23T13:08:34.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:08:33 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:08:33] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:08:34.901 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:08:35.032 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:08:35.154 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:08:35.152+0000 7f77f282f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:08:35.154 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:08:35.152+0000 7f77f282f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:08:35.167 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:08:35.167 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:08:35.167 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.1 2026-02-23T13:08:35.180 INFO:teuthology.orchestra.run.trial068.stdout:osd.2: {} 2026-02-23T13:08:35.180 INFO:teuthology.orchestra.run.trial068.stderr:osd.2: osd_enable_op_tracker = '' 2026-02-23T13:08:35.180 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:08:34 trial068 ceph-mon[38805]: pgmap v58: 17 pgs: 8 stale+active+clean, 9 active+clean; 1.9 GiB data, 6.7 GiB used, 2.7 TiB / 2.7 TiB avail; 19 MiB/s wr, 2.37k op/s 2026-02-23T13:08:35.195 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:08:35.195 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:08:35.231 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:08:35.231 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:08:35.327 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 63 128 265496 265368 32.9047 34.4844 0.0158143 0.030381 2026-02-23T13:08:36.328 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 64 127 270056 269929 32.9473 35.6328 0.0214259 0.0303345 2026-02-23T13:08:36.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:08:35 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2937670437' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:08:37.232 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:08:37.254 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:08:36 trial068 ceph-mon[38805]: pgmap v59: 17 pgs: 12 active+undersized+degraded, 5 active+clean; 2.0 GiB data, 7.1 GiB used, 2.7 TiB / 2.7 TiB avail; 35 MiB/s wr, 4.45k op/s; 24777/99495 objects degraded (24.903%); 3.0 KiB/s, 0 objects/s recovering 2026-02-23T13:08:37.327 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 65 128 274272 274144 32.9469 32.9297 0.0119984 0.0303343 2026-02-23T13:08:37.358 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:08:37.481 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:08:37.479+0000 7f7ce36f6640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:08:37.481 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:08:37.479+0000 7f7ce36f6640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:08:37.494 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:08:37.495 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:08:37.495 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.1 2026-02-23T13:08:37.508 INFO:teuthology.orchestra.run.trial068.stdout:osd.2: {} 2026-02-23T13:08:37.508 INFO:teuthology.orchestra.run.trial068.stderr:osd.2: osd_enable_op_tracker = '' 2026-02-23T13:08:37.521 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:08:37.521 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:08:37.561 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:08:37.561 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:08:37.589 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [1] live_osds: [0, 2, 3] 2026-02-23T13:08:37.589 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2026-02-23T13:08:37.589 INFO:tasks.thrashosds.thrasher:Reviving osd 1 2026-02-23T13:08:37.589 INFO:tasks.cephadm.osd.1:Restarting osd.1 (starting--it wasn't running)... 2026-02-23T13:08:37.589 DEBUG:teuthology.orchestra.run.trial068:osd.1> sudo journalctl -f -n 0 -u ceph-5f823084-10b8-11f1-be80-d404e6e7d460@osd.1.service 2026-02-23T13:08:37.604 DEBUG:teuthology.orchestra.run.trial068:> sudo systemctl start ceph-5f823084-10b8-11f1-be80-d404e6e7d460@osd.1 2026-02-23T13:08:37.785 INFO:journalctl@ceph.osd.1.trial068.stdout:Feb 23 13:08:37 trial068 systemd[1]: Starting Ceph osd.1 for 5f823084-10b8-11f1-be80-d404e6e7d460... 2026-02-23T13:08:38.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:08:37 trial068 ceph-mon[38805]: Health check failed: Degraded data redundancy: 24777/99495 objects degraded (24.903%), 12 pgs degraded (PG_DEGRADED) 2026-02-23T13:08:38.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:08:37 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3797800129' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:08:38.094 INFO:journalctl@ceph.osd.1.trial068.stdout:Feb 23 13:08:37 trial068 podman[82261]: 2026-02-23 13:08:37.784156239 +0000 UTC m=+0.017567314 container create 4e964257e55e23840f204d2a9ccdb871fc51fdaedda7e3a1184e5d978500a04a (image=quay.ceph.io/ceph-ci/ceph@sha256:49621faaa845c39b2aae72e4fa24123ee9b2ac3fb8ae834753907068ee2972ea, name=ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-1-activate, CEPH_GIT_REPO=https://github.com/ceph/ceph-ci.git, ceph=True, org.label-schema.schema-version=1.0, org.opencontainers.image.authors=Ceph Release Team , CEPH_SHA1=e371433fcd1390a0b82cfb18effc422c1f5d8c1d, org.label-schema.license=GPLv2, io.buildah.version=1.41.3, FROM_IMAGE=quay.io/centos/centos:stream9, OSD_FLAVOR=default, org.label-schema.build-date=20260216, org.opencontainers.image.documentation=https://docs.ceph.com/, org.label-schema.vendor=CentOS, GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfsganesha-5/, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_REF=wip-shraddhaag-enable-cephadm-tests) 2026-02-23T13:08:38.094 INFO:journalctl@ceph.osd.1.trial068.stdout:Feb 23 13:08:37 trial068 podman[82261]: 2026-02-23 13:08:37.82353273 +0000 UTC m=+0.056943805 container init 4e964257e55e23840f204d2a9ccdb871fc51fdaedda7e3a1184e5d978500a04a (image=quay.ceph.io/ceph-ci/ceph@sha256:49621faaa845c39b2aae72e4fa24123ee9b2ac3fb8ae834753907068ee2972ea, name=ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-1-activate, CEPH_GIT_REPO=https://github.com/ceph/ceph-ci.git, OSD_FLAVOR=default, org.label-schema.build-date=20260216, org.opencontainers.image.authors=Ceph Release Team , CEPH_SHA1=e371433fcd1390a0b82cfb18effc422c1f5d8c1d, ceph=True, GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfsganesha-5/, org.label-schema.name=CentOS Stream 9 Base Image, org.opencontainers.image.documentation=https://docs.ceph.com/, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, FROM_IMAGE=quay.io/centos/centos:stream9, org.label-schema.schema-version=1.0, CEPH_REF=wip-shraddhaag-enable-cephadm-tests, io.buildah.version=1.41.3) 2026-02-23T13:08:38.094 INFO:journalctl@ceph.osd.1.trial068.stdout:Feb 23 13:08:37 trial068 podman[82261]: 2026-02-23 13:08:37.826275436 +0000 UTC m=+0.059686511 container start 4e964257e55e23840f204d2a9ccdb871fc51fdaedda7e3a1184e5d978500a04a (image=quay.ceph.io/ceph-ci/ceph@sha256:49621faaa845c39b2aae72e4fa24123ee9b2ac3fb8ae834753907068ee2972ea, name=ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-1-activate, org.label-schema.name=CentOS Stream 9 Base Image, org.opencontainers.image.documentation=https://docs.ceph.com/, CEPH_GIT_REPO=https://github.com/ceph/ceph-ci.git, org.label-schema.license=GPLv2, OSD_FLAVOR=default, ceph=True, FROM_IMAGE=quay.io/centos/centos:stream9, org.opencontainers.image.authors=Ceph Release Team , CEPH_SHA1=e371433fcd1390a0b82cfb18effc422c1f5d8c1d, org.label-schema.vendor=CentOS, org.label-schema.build-date=20260216, CEPH_REF=wip-shraddhaag-enable-cephadm-tests, io.buildah.version=1.41.3, GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfsganesha-5/, org.label-schema.schema-version=1.0) 2026-02-23T13:08:38.094 INFO:journalctl@ceph.osd.1.trial068.stdout:Feb 23 13:08:37 trial068 podman[82261]: 2026-02-23 13:08:37.82677754 +0000 UTC m=+0.060188615 container attach 4e964257e55e23840f204d2a9ccdb871fc51fdaedda7e3a1184e5d978500a04a (image=quay.ceph.io/ceph-ci/ceph@sha256:49621faaa845c39b2aae72e4fa24123ee9b2ac3fb8ae834753907068ee2972ea, name=ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-1-activate, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, FROM_IMAGE=quay.io/centos/centos:stream9, org.opencontainers.image.documentation=https://docs.ceph.com/, io.buildah.version=1.41.3, ceph=True, GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfsganesha-5/, org.label-schema.schema-version=1.0, org.label-schema.build-date=20260216, CEPH_SHA1=e371433fcd1390a0b82cfb18effc422c1f5d8c1d, CEPH_GIT_REPO=https://github.com/ceph/ceph-ci.git, OSD_FLAVOR=default, org.label-schema.name=CentOS Stream 9 Base Image, org.opencontainers.image.authors=Ceph Release Team , CEPH_REF=wip-shraddhaag-enable-cephadm-tests) 2026-02-23T13:08:38.094 INFO:journalctl@ceph.osd.1.trial068.stdout:Feb 23 13:08:37 trial068 podman[82261]: 2026-02-23 13:08:37.77733178 +0000 UTC m=+0.010742865 image pull 63df4b83db6f95a8ba965919b3671cb27173117c23b5c950aaadc8913f56213c quay.ceph.io/ceph-ci/ceph@sha256:49621faaa845c39b2aae72e4fa24123ee9b2ac3fb8ae834753907068ee2972ea 2026-02-23T13:08:38.094 INFO:journalctl@ceph.osd.1.trial068.stdout:Feb 23 13:08:37 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-1-activate[82286]: Running command: /usr/bin/ceph-authtool --gen-print-key 2026-02-23T13:08:38.094 INFO:journalctl@ceph.osd.1.trial068.stdout:Feb 23 13:08:37 trial068 bash[82261]: Running command: /usr/bin/ceph-authtool --gen-print-key 2026-02-23T13:08:38.094 INFO:journalctl@ceph.osd.1.trial068.stdout:Feb 23 13:08:37 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-1-activate[82286]: Running command: /usr/bin/ceph-authtool --gen-print-key 2026-02-23T13:08:38.094 INFO:journalctl@ceph.osd.1.trial068.stdout:Feb 23 13:08:37 trial068 bash[82261]: Running command: /usr/bin/ceph-authtool --gen-print-key 2026-02-23T13:08:38.328 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 66 128 278389 278261 32.935 32.1641 0.00868126 0.0303458 2026-02-23T13:08:38.930 INFO:journalctl@ceph.osd.1.trial068.stdout:Feb 23 13:08:38 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-1-activate[82286]: --> Failed to activate via raw: did not find any matching OSD to activate 2026-02-23T13:08:38.930 INFO:journalctl@ceph.osd.1.trial068.stdout:Feb 23 13:08:38 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-1-activate[82286]: Running command: /usr/bin/ceph-authtool --gen-print-key 2026-02-23T13:08:38.930 INFO:journalctl@ceph.osd.1.trial068.stdout:Feb 23 13:08:38 trial068 bash[82261]: --> Failed to activate via raw: did not find any matching OSD to activate 2026-02-23T13:08:38.930 INFO:journalctl@ceph.osd.1.trial068.stdout:Feb 23 13:08:38 trial068 bash[82261]: Running command: /usr/bin/ceph-authtool --gen-print-key 2026-02-23T13:08:38.930 INFO:journalctl@ceph.osd.1.trial068.stdout:Feb 23 13:08:38 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-1-activate[82286]: Running command: /usr/bin/ceph-authtool --gen-print-key 2026-02-23T13:08:38.930 INFO:journalctl@ceph.osd.1.trial068.stdout:Feb 23 13:08:38 trial068 bash[82261]: Running command: /usr/bin/ceph-authtool --gen-print-key 2026-02-23T13:08:38.931 INFO:journalctl@ceph.osd.1.trial068.stdout:Feb 23 13:08:38 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-1-activate[82286]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2026-02-23T13:08:38.931 INFO:journalctl@ceph.osd.1.trial068.stdout:Feb 23 13:08:38 trial068 bash[82261]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2026-02-23T13:08:38.931 INFO:journalctl@ceph.osd.1.trial068.stdout:Feb 23 13:08:38 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-1-activate[82286]: Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/vg_nvme/lv_3 --path /var/lib/ceph/osd/ceph-1 --no-mon-config 2026-02-23T13:08:38.931 INFO:journalctl@ceph.osd.1.trial068.stdout:Feb 23 13:08:38 trial068 bash[82261]: Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/vg_nvme/lv_3 --path /var/lib/ceph/osd/ceph-1 --no-mon-config 2026-02-23T13:08:38.931 INFO:journalctl@ceph.osd.1.trial068.stdout:Feb 23 13:08:38 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-1-activate[82286]: Running command: /usr/bin/ln -snf /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-1/block 2026-02-23T13:08:38.931 INFO:journalctl@ceph.osd.1.trial068.stdout:Feb 23 13:08:38 trial068 bash[82261]: Running command: /usr/bin/ln -snf /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-1/block 2026-02-23T13:08:38.931 INFO:journalctl@ceph.osd.1.trial068.stdout:Feb 23 13:08:38 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-1-activate[82286]: Running command: /usr/bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-1/block 2026-02-23T13:08:38.931 INFO:journalctl@ceph.osd.1.trial068.stdout:Feb 23 13:08:38 trial068 bash[82261]: Running command: /usr/bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-1/block 2026-02-23T13:08:38.931 INFO:journalctl@ceph.osd.1.trial068.stdout:Feb 23 13:08:38 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-1-activate[82286]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2026-02-23T13:08:38.931 INFO:journalctl@ceph.osd.1.trial068.stdout:Feb 23 13:08:38 trial068 bash[82261]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2026-02-23T13:08:38.931 INFO:journalctl@ceph.osd.1.trial068.stdout:Feb 23 13:08:38 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-1-activate[82286]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2026-02-23T13:08:38.932 INFO:journalctl@ceph.osd.1.trial068.stdout:Feb 23 13:08:38 trial068 bash[82261]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2026-02-23T13:08:38.932 INFO:journalctl@ceph.osd.1.trial068.stdout:Feb 23 13:08:38 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-1-activate[82286]: --> ceph-volume lvm activate successful for osd ID: 1 2026-02-23T13:08:38.932 INFO:journalctl@ceph.osd.1.trial068.stdout:Feb 23 13:08:38 trial068 bash[82261]: --> ceph-volume lvm activate successful for osd ID: 1 2026-02-23T13:08:38.932 INFO:journalctl@ceph.osd.1.trial068.stdout:Feb 23 13:08:38 trial068 podman[82261]: 2026-02-23 13:08:38.834630822 +0000 UTC m=+1.068041897 container died 4e964257e55e23840f204d2a9ccdb871fc51fdaedda7e3a1184e5d978500a04a (image=quay.ceph.io/ceph-ci/ceph@sha256:49621faaa845c39b2aae72e4fa24123ee9b2ac3fb8ae834753907068ee2972ea, name=ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-1-activate, org.label-schema.build-date=20260216, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.41.3, org.label-schema.license=GPLv2, GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfsganesha-5/, CEPH_REF=wip-shraddhaag-enable-cephadm-tests, CEPH_SHA1=e371433fcd1390a0b82cfb18effc422c1f5d8c1d, org.label-schema.schema-version=1.0, org.opencontainers.image.documentation=https://docs.ceph.com/, CEPH_GIT_REPO=https://github.com/ceph/ceph-ci.git, ceph=True, FROM_IMAGE=quay.io/centos/centos:stream9, OSD_FLAVOR=default, org.opencontainers.image.authors=Ceph Release Team , org.label-schema.vendor=CentOS) 2026-02-23T13:08:38.932 INFO:journalctl@ceph.osd.1.trial068.stdout:Feb 23 13:08:38 trial068 podman[82261]: 2026-02-23 13:08:38.845262127 +0000 UTC m=+1.078673192 container remove 4e964257e55e23840f204d2a9ccdb871fc51fdaedda7e3a1184e5d978500a04a (image=quay.ceph.io/ceph-ci/ceph@sha256:49621faaa845c39b2aae72e4fa24123ee9b2ac3fb8ae834753907068ee2972ea, name=ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-1-activate, CEPH_REF=wip-shraddhaag-enable-cephadm-tests, org.label-schema.vendor=CentOS, org.opencontainers.image.authors=Ceph Release Team , CEPH_SHA1=e371433fcd1390a0b82cfb18effc422c1f5d8c1d, io.buildah.version=1.41.3, org.opencontainers.image.documentation=https://docs.ceph.com/, org.label-schema.license=GPLv2, FROM_IMAGE=quay.io/centos/centos:stream9, org.label-schema.build-date=20260216, ceph=True, GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfsganesha-5/, org.label-schema.name=CentOS Stream 9 Base Image, OSD_FLAVOR=default, CEPH_GIT_REPO=https://github.com/ceph/ceph-ci.git, org.label-schema.schema-version=1.0) 2026-02-23T13:08:38.932 INFO:journalctl@ceph.osd.1.trial068.stdout:Feb 23 13:08:38 trial068 podman[82513]: 2026-02-23 13:08:38.929136115 +0000 UTC m=+0.016422422 container create a6c602d586023fdefcb285fca8b69f126bbdea24a82f5221490992b55302cbc9 (image=quay.ceph.io/ceph-ci/ceph@sha256:49621faaa845c39b2aae72e4fa24123ee9b2ac3fb8ae834753907068ee2972ea, name=ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-1, org.label-schema.build-date=20260216, OSD_FLAVOR=default, CEPH_GIT_REPO=https://github.com/ceph/ceph-ci.git, CEPH_REF=wip-shraddhaag-enable-cephadm-tests, CEPH_SHA1=e371433fcd1390a0b82cfb18effc422c1f5d8c1d, ceph=True, io.buildah.version=1.41.3, org.label-schema.vendor=CentOS, FROM_IMAGE=quay.io/centos/centos:stream9, org.opencontainers.image.authors=Ceph Release Team , org.label-schema.name=CentOS Stream 9 Base Image, org.opencontainers.image.documentation=https://docs.ceph.com/, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfsganesha-5/) 2026-02-23T13:08:38.962 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph daemon osd.1 dump_ops_in_flight 2026-02-23T13:08:39.096 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:08:39.212 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:08:38 trial068 ceph-mon[38805]: 2.f scrub starts 2026-02-23T13:08:39.212 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:08:38 trial068 ceph-mon[38805]: pgmap v60: 17 pgs: 12 active+undersized+degraded, 5 active+clean; 2.0 GiB data, 7.1 GiB used, 2.7 TiB / 2.7 TiB avail; 35 MiB/s wr, 4.45k op/s; 24777/99495 objects degraded (24.903%); 3.0 KiB/s, 0 objects/s recovering 2026-02-23T13:08:39.213 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:08:38 trial068 ceph-mon[38805]: 2.f scrub ok 2026-02-23T13:08:39.213 INFO:journalctl@ceph.osd.1.trial068.stdout:Feb 23 13:08:38 trial068 podman[82513]: 2026-02-23 13:08:38.953283097 +0000 UTC m=+0.040569404 container init a6c602d586023fdefcb285fca8b69f126bbdea24a82f5221490992b55302cbc9 (image=quay.ceph.io/ceph-ci/ceph@sha256:49621faaa845c39b2aae72e4fa24123ee9b2ac3fb8ae834753907068ee2972ea, name=ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-1, OSD_FLAVOR=default, io.buildah.version=1.41.3, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, FROM_IMAGE=quay.io/centos/centos:stream9, CEPH_REF=wip-shraddhaag-enable-cephadm-tests, org.label-schema.vendor=CentOS, org.label-schema.build-date=20260216, CEPH_GIT_REPO=https://github.com/ceph/ceph-ci.git, GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfsganesha-5/, CEPH_SHA1=e371433fcd1390a0b82cfb18effc422c1f5d8c1d, org.opencontainers.image.authors=Ceph Release Team , org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.opencontainers.image.documentation=https://docs.ceph.com/) 2026-02-23T13:08:39.213 INFO:journalctl@ceph.osd.1.trial068.stdout:Feb 23 13:08:38 trial068 podman[82513]: 2026-02-23 13:08:38.955653353 +0000 UTC m=+0.042939650 container start a6c602d586023fdefcb285fca8b69f126bbdea24a82f5221490992b55302cbc9 (image=quay.ceph.io/ceph-ci/ceph@sha256:49621faaa845c39b2aae72e4fa24123ee9b2ac3fb8ae834753907068ee2972ea, name=ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-1, org.label-schema.schema-version=1.0, CEPH_GIT_REPO=https://github.com/ceph/ceph-ci.git, FROM_IMAGE=quay.io/centos/centos:stream9, CEPH_REF=wip-shraddhaag-enable-cephadm-tests, org.label-schema.license=GPLv2, io.buildah.version=1.41.3, org.label-schema.vendor=CentOS, GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfsganesha-5/, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.opencontainers.image.documentation=https://docs.ceph.com/, OSD_FLAVOR=default, org.label-schema.build-date=20260216, CEPH_SHA1=e371433fcd1390a0b82cfb18effc422c1f5d8c1d, org.opencontainers.image.authors=Ceph Release Team ) 2026-02-23T13:08:39.214 INFO:journalctl@ceph.osd.1.trial068.stdout:Feb 23 13:08:38 trial068 bash[82513]: a6c602d586023fdefcb285fca8b69f126bbdea24a82f5221490992b55302cbc9 2026-02-23T13:08:39.214 INFO:journalctl@ceph.osd.1.trial068.stdout:Feb 23 13:08:38 trial068 podman[82513]: 2026-02-23 13:08:38.923038523 +0000 UTC m=+0.010324830 image pull 63df4b83db6f95a8ba965919b3671cb27173117c23b5c950aaadc8913f56213c quay.ceph.io/ceph-ci/ceph@sha256:49621faaa845c39b2aae72e4fa24123ee9b2ac3fb8ae834753907068ee2972ea 2026-02-23T13:08:39.214 INFO:journalctl@ceph.osd.1.trial068.stdout:Feb 23 13:08:38 trial068 systemd[1]: Started Ceph osd.1 for 5f823084-10b8-11f1-be80-d404e6e7d460. 2026-02-23T13:08:39.214 INFO:journalctl@ceph.osd.1.trial068.stdout:Feb 23 13:08:38 trial068 ceph-osd[82538]: WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:08:39.214 INFO:journalctl@ceph.osd.1.trial068.stdout:Feb 23 13:08:38 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-1[82534]: 2026-02-23T13:08:38.968+0000 7f571c555780 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:08:39.214 INFO:journalctl@ceph.osd.1.trial068.stdout:Feb 23 13:08:38 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-1[82534]: 2026-02-23T13:08:38.969+0000 7f571c555780 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:08:39.215 INFO:journalctl@ceph.osd.1.trial068.stdout:Feb 23 13:08:38 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-1[82534]: 2026-02-23T13:08:38.969+0000 7f571c555780 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:08:39.233 INFO:teuthology.orchestra.run.trial068.stderr:no valid command found; 10 closest matches: 2026-02-23T13:08:39.233 INFO:teuthology.orchestra.run.trial068.stderr:0 2026-02-23T13:08:39.233 INFO:teuthology.orchestra.run.trial068.stderr:1 2026-02-23T13:08:39.233 INFO:teuthology.orchestra.run.trial068.stderr:2 2026-02-23T13:08:39.234 INFO:teuthology.orchestra.run.trial068.stderr:abort 2026-02-23T13:08:39.234 INFO:teuthology.orchestra.run.trial068.stderr:assert 2026-02-23T13:08:39.234 INFO:teuthology.orchestra.run.trial068.stderr:bluestore collections 2026-02-23T13:08:39.234 INFO:teuthology.orchestra.run.trial068.stderr:bluestore compression stats [] 2026-02-23T13:08:39.234 INFO:teuthology.orchestra.run.trial068.stderr:bluestore list [] [] 2026-02-23T13:08:39.234 INFO:teuthology.orchestra.run.trial068.stderr:bluestore onode metadata 2026-02-23T13:08:39.234 INFO:teuthology.orchestra.run.trial068.stderr:bluestore show sharding 2026-02-23T13:08:39.235 INFO:teuthology.orchestra.run.trial068.stderr:admin_socket: invalid command 2026-02-23T13:08:39.268 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-23T13:08:39.269 INFO:tasks.cephadm.ceph_manager.ceph:waiting on admin_socket for osd-1, ['dump_ops_in_flight'] 2026-02-23T13:08:39.328 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 67 128 282725 282597 32.949 33.875 0.00738296 0.0303377 2026-02-23T13:08:39.563 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:08:39.585 INFO:journalctl@ceph.osd.1.trial068.stdout:Feb 23 13:08:39 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-1[82534]: 2026-02-23T13:08:39.215+0000 7f571c555780 -1 Falling back to public interface 2026-02-23T13:08:39.692 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:08:39.809 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:08:39.807+0000 7fbb101a3640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:08:39.809 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:08:39.807+0000 7fbb101a3640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:08:39.822 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:08:39.822 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:08:39.822 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.1 2026-02-23T13:08:39.836 INFO:teuthology.orchestra.run.trial068.stdout:osd.2: {} 2026-02-23T13:08:39.836 INFO:teuthology.orchestra.run.trial068.stderr:osd.2: osd_enable_op_tracker = '' 2026-02-23T13:08:39.851 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:08:39.851 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:08:39.884 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:08:39.885 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:08:40.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:08:39 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/43713211' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:08:40.328 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 68 128 286943 286815 32.949 32.9531 0.0259988 0.0303399 2026-02-23T13:08:41.328 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 69 128 291150 291022 32.9478 32.8672 0.0212693 0.0303378 2026-02-23T13:08:41.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:08:40 trial068 ceph-mon[38805]: pgmap v61: 17 pgs: 12 active+undersized+degraded, 5 active+clean; 2.0 GiB data, 7.1 GiB used, 2.7 TiB / 2.7 TiB avail; 14 MiB/s wr, 1.84k op/s; 24777/99495 objects degraded (24.903%); 2.7 KiB/s, 0 objects/s recovering 2026-02-23T13:08:41.886 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:08:42.020 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:08:42.133 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:08:42.132+0000 7f48f11f0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:08:42.134 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:08:42.132+0000 7f48f11f0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:08:42.146 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:08:42.146 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:08:42.146 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.1 2026-02-23T13:08:42.160 INFO:teuthology.orchestra.run.trial068.stdout:osd.2: {} 2026-02-23T13:08:42.160 INFO:teuthology.orchestra.run.trial068.stderr:osd.2: osd_enable_op_tracker = '' 2026-02-23T13:08:42.160 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:08:41 trial068 ceph-mon[38805]: 2.b scrub starts 2026-02-23T13:08:42.173 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:08:42.174 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:08:42.210 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:08:42.210 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:08:42.328 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 70 127 295246 295119 32.9343 32.0078 0.0143439 0.030352 2026-02-23T13:08:43.328 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 71 128 299206 299078 32.906 30.9297 0.0216798 0.0303777 2026-02-23T13:08:43.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:08:42 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/539973193' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:08:43.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:08:42 trial068 ceph-mon[38805]: pgmap v62: 17 pgs: 12 active+undersized+degraded, 5 active+clean; 2.2 GiB data, 7.6 GiB used, 2.7 TiB / 2.7 TiB avail; 30 MiB/s wr, 3.81k op/s; 26782/107553 objects degraded (24.901%); 2.4 KiB/s, 0 objects/s recovering 2026-02-23T13:08:43.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:08:42 trial068 ceph-mon[38805]: 2.b scrub ok 2026-02-23T13:08:44.211 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:08:44.234 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:08:43 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:08:43] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:08:44.234 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:08:43 trial068 ceph-mon[38805]: Health check update: Degraded data redundancy: 26782/107553 objects degraded (24.901%), 12 pgs degraded (PG_DEGRADED) 2026-02-23T13:08:44.270 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph daemon osd.1 dump_ops_in_flight 2026-02-23T13:08:44.328 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 72 128 303410 303282 32.9051 32.8438 0.0554041 0.0303792 2026-02-23T13:08:44.342 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:08:44.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:08:43 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:08:43.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:08:44.401 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:08:44.468 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:08:44.466+0000 7fc86917e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:08:44.468 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:08:44.467+0000 7fc86917e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:08:44.484 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:08:44.485 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:08:44.485 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.1 2026-02-23T13:08:44.497 INFO:teuthology.orchestra.run.trial068.stdout:osd.2: {} 2026-02-23T13:08:44.497 INFO:teuthology.orchestra.run.trial068.stderr:osd.2: osd_enable_op_tracker = '' 2026-02-23T13:08:44.512 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:08:44.513 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:08:44.530 INFO:teuthology.orchestra.run.trial068.stderr:no valid command found; 10 closest matches: 2026-02-23T13:08:44.530 INFO:teuthology.orchestra.run.trial068.stderr:0 2026-02-23T13:08:44.530 INFO:teuthology.orchestra.run.trial068.stderr:1 2026-02-23T13:08:44.530 INFO:teuthology.orchestra.run.trial068.stderr:2 2026-02-23T13:08:44.530 INFO:teuthology.orchestra.run.trial068.stderr:abort 2026-02-23T13:08:44.531 INFO:teuthology.orchestra.run.trial068.stderr:assert 2026-02-23T13:08:44.531 INFO:teuthology.orchestra.run.trial068.stderr:bluefs debug_inject_read_zeros 2026-02-23T13:08:44.531 INFO:teuthology.orchestra.run.trial068.stderr:bluefs files list 2026-02-23T13:08:44.531 INFO:teuthology.orchestra.run.trial068.stderr:bluefs stats 2026-02-23T13:08:44.531 INFO:teuthology.orchestra.run.trial068.stderr:bluestore bluefs device info [] 2026-02-23T13:08:44.531 INFO:teuthology.orchestra.run.trial068.stderr:bluestore collections 2026-02-23T13:08:44.531 INFO:teuthology.orchestra.run.trial068.stderr:admin_socket: invalid command 2026-02-23T13:08:44.549 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:08:44.550 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:08:44.567 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-23T13:08:44.568 INFO:tasks.cephadm.ceph_manager.ceph:waiting on admin_socket for osd-1, ['dump_ops_in_flight'] 2026-02-23T13:08:45.328 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 73 128 307488 307360 32.8908 31.8594 0.0237669 0.0303943 2026-02-23T13:08:45.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:08:44 trial068 ceph-mon[38805]: pgmap v63: 17 pgs: 12 active+undersized+degraded, 5 active+clean; 2.2 GiB data, 7.6 GiB used, 2.7 TiB / 2.7 TiB avail; 25 MiB/s wr, 3.18k op/s; 26782/107553 objects degraded (24.901%); 2.0 KiB/s, 0 objects/s recovering 2026-02-23T13:08:45.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:08:44 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1056492889' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:08:46.329 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 74 127 311682 311555 32.8892 32.7734 0.0260984 0.0303956 2026-02-23T13:08:46.550 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:08:46.689 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:08:46.801 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:08:46.799+0000 7f43536a1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:08:46.801 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:08:46.799+0000 7f43536a1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:08:46.815 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:08:46.815 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:08:46.815 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.1 2026-02-23T13:08:46.829 INFO:teuthology.orchestra.run.trial068.stdout:osd.2: {} 2026-02-23T13:08:46.829 INFO:teuthology.orchestra.run.trial068.stderr:osd.2: osd_enable_op_tracker = '' 2026-02-23T13:08:46.842 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:08:46.842 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:08:46.874 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:08:46.875 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:08:47.328 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 75 128 315740 315612 32.8732 31.6953 0.0139944 0.0304076 2026-02-23T13:08:47.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:08:47 trial068 ceph-mon[38805]: pgmap v64: 17 pgs: 12 active+undersized+degraded, 5 active+clean; 2.3 GiB data, 8.0 GiB used, 2.7 TiB / 2.7 TiB avail; 38 MiB/s wr, 4.89k op/s; 28713/115287 objects degraded (24.906%); 2.0 KiB/s, 0 objects/s recovering 2026-02-23T13:08:47.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:08:47 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/4089288601' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:08:48.329 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 76 128 319629 319501 32.8404 30.3828 0.0341383 0.0304411 2026-02-23T13:08:48.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:08:48 trial068 ceph-mon[38805]: from='osd.1 [v2:10.20.193.68:6810/219168100,v1:10.20.193.68:6811/219168100]' entity='osd.1' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]} : dispatch 2026-02-23T13:08:48.594 INFO:journalctl@ceph.osd.1.trial068.stdout:Feb 23 13:08:48 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-1[82534]: 2026-02-23T13:08:48.170+0000 7f571c555780 -1 osd.1 46 log_to_monitors true 2026-02-23T13:08:48.594 INFO:journalctl@ceph.osd.1.trial068.stdout:Feb 23 13:08:48 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-1[82534]: 2026-02-23T13:08:48.291+0000 7f571439e640 -1 osd.1 46 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2026-02-23T13:08:48.875 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:08:49.004 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:08:49.123 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:08:49.121+0000 7fa7c6020640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:08:49.123 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:08:49.121+0000 7fa7c6020640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:08:49.136 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:08:49.136 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:08:49.136 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.1 2026-02-23T13:08:49.149 INFO:teuthology.orchestra.run.trial068.stdout:osd.2: {} 2026-02-23T13:08:49.149 INFO:teuthology.orchestra.run.trial068.stderr:osd.2: osd_enable_op_tracker = '' 2026-02-23T13:08:49.160 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:08:49.160 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:08:49.193 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:08:49.194 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:08:49.329 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 77 128 323626 323498 32.8194 31.2266 0.032862 0.0304583 2026-02-23T13:08:49.569 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph daemon osd.1 dump_ops_in_flight 2026-02-23T13:08:49.592 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:08:49 trial068 ceph-mon[38805]: pgmap v65: 17 pgs: 12 active+undersized+degraded, 5 active+clean; 2.3 GiB data, 8.0 GiB used, 2.7 TiB / 2.7 TiB avail; 27 MiB/s wr, 3.51k op/s; 28713/115287 objects degraded (24.906%) 2026-02-23T13:08:49.592 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:08:49 trial068 ceph-mon[38805]: from='osd.1 [v2:10.20.193.68:6810/219168100,v1:10.20.193.68:6811/219168100]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2026-02-23T13:08:49.592 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:08:49 trial068 ceph-mon[38805]: osdmap e49: 4 total, 3 up, 4 in 2026-02-23T13:08:49.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:08:49 trial068 ceph-mon[38805]: from='osd.1 [v2:10.20.193.68:6810/219168100,v1:10.20.193.68:6811/219168100]' entity='osd.1' cmd={"prefix": "osd crush create-or-move", "id": 1, "weight":0.6836, "args": ["host=trial068", "root=default"]} : dispatch 2026-02-23T13:08:49.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:08:49 trial068 ceph-mon[38805]: Health check update: Degraded data redundancy: 28713/115287 objects degraded (24.906%), 12 pgs degraded (PG_DEGRADED) 2026-02-23T13:08:49.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:08:49 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2516619585' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:08:49.696 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:08:49.846 INFO:teuthology.orchestra.run.trial068.stdout:{ 2026-02-23T13:08:49.846 INFO:teuthology.orchestra.run.trial068.stdout: "ops": [], 2026-02-23T13:08:49.846 INFO:teuthology.orchestra.run.trial068.stdout: "num_ops": 0 2026-02-23T13:08:49.846 INFO:teuthology.orchestra.run.trial068.stdout:} 2026-02-23T13:08:50.328 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 78 128 325903 325775 32.6267 17.7891 0.0386776 0.0304788 2026-02-23T13:08:50.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:08:50 trial068 ceph-mon[38805]: 2.8 scrub starts 2026-02-23T13:08:50.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:08:50 trial068 ceph-mon[38805]: Health check cleared: OSD_DOWN (was: 1 osds down) 2026-02-23T13:08:50.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:08:50 trial068 ceph-mon[38805]: osd.1 [v2:10.20.193.68:6810/219168100,v1:10.20.193.68:6811/219168100] boot 2026-02-23T13:08:50.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:08:50 trial068 ceph-mon[38805]: osdmap e50: 4 total, 4 up, 4 in 2026-02-23T13:08:50.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:08:50 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2026-02-23T13:08:51.195 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:08:51.315 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:08:51.329 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 79 128 325903 325775 32.2137 0 - 0.0304788 2026-02-23T13:08:51.414 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:08:51.412+0000 7f7004b1f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:08:51.414 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:08:51.412+0000 7f7004b1f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:08:51.426 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:08:51.427 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:08:51.437 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:08:51.437 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:08:51.449 INFO:teuthology.orchestra.run.trial068.stdout:osd.2: {} 2026-02-23T13:08:51.449 INFO:teuthology.orchestra.run.trial068.stderr:osd.2: osd_enable_op_tracker = '' 2026-02-23T13:08:51.460 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:08:51.460 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:08:51.473 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:08:51 trial068 ceph-mon[38805]: 2.8 scrub ok 2026-02-23T13:08:51.473 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:08:51 trial068 ceph-mon[38805]: pgmap v68: 17 pgs: 12 active+undersized+degraded, 5 active+clean; 2.3 GiB data, 7.8 GiB used, 2.7 TiB / 2.7 TiB avail; 20 MiB/s wr, 2.58k op/s; 28713/115287 objects degraded (24.906%) 2026-02-23T13:08:51.473 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:08:51 trial068 ceph-mon[38805]: osdmap e51: 4 total, 4 up, 4 in 2026-02-23T13:08:52.329 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T13:08:52.328973+0000 min lat: 0.000727387 max lat: 2.47295 avg lat: 0.0312481 2026-02-23T13:08:52.329 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T13:08:52.329 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 80 127 327684 327557 31.9851 6.96094 0.0230087 0.0312481 2026-02-23T13:08:52.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:08:52 trial068 ceph-mon[38805]: osdmap e52: 4 total, 4 up, 4 in 2026-02-23T13:08:52.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:08:52 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/991618562' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:08:53.329 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 81 128 331084 330956 31.918 26.5547 0.0345307 0.0313132 2026-02-23T13:08:53.494 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:08:53.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:08:53 trial068 ceph-mon[38805]: pgmap v71: 17 pgs: 12 remapped+peering, 5 active+clean; 2.5 GiB data, 8.1 GiB used, 2.7 TiB / 2.7 TiB avail; 36 MiB/s wr, 4.61k op/s 2026-02-23T13:08:53.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:08:53 trial068 ceph-mon[38805]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 28713/115287 objects degraded (24.906%), 12 pgs degraded) 2026-02-23T13:08:53.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:08:53 trial068 ceph-mon[38805]: Cluster is now healthy 2026-02-23T13:08:53.642 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:08:53.768 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:08:53.767+0000 7fd71c62e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:08:53.769 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:08:53.767+0000 7fd71c62e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:08:53.783 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:08:53.784 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:08:53.797 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:08:53.797 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:08:53.811 INFO:teuthology.orchestra.run.trial068.stdout:osd.2: {} 2026-02-23T13:08:53.811 INFO:teuthology.orchestra.run.trial068.stderr:osd.2: osd_enable_op_tracker = '' 2026-02-23T13:08:53.826 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:08:53.826 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:08:54.303 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:08:53 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:08:53] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:08:54.329 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 82 128 334205 334077 31.8261 24.3828 0.0700084 0.0314072 2026-02-23T13:08:54.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:08:53 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:08:53.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:08:54.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:08:54 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3817545311' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:08:54.885 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 2, 3, 1] 2026-02-23T13:08:54.885 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2026-02-23T13:08:54.885 INFO:tasks.thrashosds.thrasher:Killing osd 3, live_osds are [0, 2, 3, 1] 2026-02-23T13:08:54.886 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2026-02-23T13:08:54.886 DEBUG:teuthology.orchestra.run.trial185:> sudo systemctl stop ceph-5f823084-10b8-11f1-be80-d404e6e7d460@osd.3 2026-02-23T13:08:55.329 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 83 128 337534 337406 31.7559 26.0078 0.0232192 0.0314735 2026-02-23T13:08:55.356 INFO:journalctl@ceph.osd.3.trial185.stdout:Feb 23 13:08:54 trial185 systemd[1]: Stopping Ceph osd.3 for 5f823084-10b8-11f1-be80-d404e6e7d460... 2026-02-23T13:08:55.356 INFO:journalctl@ceph.osd.3.trial185.stdout:Feb 23 13:08:55 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-3[66506]: 2026-02-23T13:08:55.018+0000 7f99d5b2d640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --osd-objectstore=bluestore (PID: 1) UID: 0 2026-02-23T13:08:55.356 INFO:journalctl@ceph.osd.3.trial185.stdout:Feb 23 13:08:55 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-3[66506]: 2026-02-23T13:08:55.018+0000 7f99d5b2d640 -1 osd.3 52 *** Got signal Terminated *** 2026-02-23T13:08:55.356 INFO:journalctl@ceph.osd.3.trial185.stdout:Feb 23 13:08:55 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-3[66506]: 2026-02-23T13:08:55.018+0000 7f99d5b2d640 -1 osd.3 52 *** Immediate shutdown (osd_fast_shutdown=true) *** 2026-02-23T13:08:55.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:08:55 trial068 ceph-mon[38805]: pgmap v72: 17 pgs: 12 remapped+peering, 5 active+clean; 2.5 GiB data, 8.2 GiB used, 2.7 TiB / 2.7 TiB avail; 24 MiB/s wr, 3.10k op/s 2026-02-23T13:08:55.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:08:55 trial068 ceph-mon[38805]: osd.3 marked itself down and dead 2026-02-23T13:08:55.723 INFO:journalctl@ceph.osd.3.trial185.stdout:Feb 23 13:08:55 trial185 podman[67409]: 2026-02-23 13:08:55.425207617 +0000 UTC m=+0.432410840 container died b401ff5c1392a94c8c70aa1388589c699a0bc3093dc05c21eb0d8a391dcc36ca (image=quay.ceph.io/ceph-ci/ceph@sha256:49621faaa845c39b2aae72e4fa24123ee9b2ac3fb8ae834753907068ee2972ea, name=ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-3, io.buildah.version=1.41.3, org.label-schema.schema-version=1.0, org.opencontainers.image.documentation=https://docs.ceph.com/, CEPH_SHA1=e371433fcd1390a0b82cfb18effc422c1f5d8c1d, GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfsganesha-5/, org.label-schema.build-date=20260216, CEPH_GIT_REPO=https://github.com/ceph/ceph-ci.git, org.label-schema.license=GPLv2, FROM_IMAGE=quay.io/centos/centos:stream9, org.opencontainers.image.authors=Ceph Release Team , OSD_FLAVOR=default, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, ceph=True, CEPH_REF=wip-shraddhaag-enable-cephadm-tests) 2026-02-23T13:08:55.723 INFO:journalctl@ceph.osd.3.trial185.stdout:Feb 23 13:08:55 trial185 podman[67409]: 2026-02-23 13:08:55.435519377 +0000 UTC m=+0.442722600 container remove b401ff5c1392a94c8c70aa1388589c699a0bc3093dc05c21eb0d8a391dcc36ca (image=quay.ceph.io/ceph-ci/ceph@sha256:49621faaa845c39b2aae72e4fa24123ee9b2ac3fb8ae834753907068ee2972ea, name=ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-3, OSD_FLAVOR=default, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.opencontainers.image.documentation=https://docs.ceph.com/, org.opencontainers.image.authors=Ceph Release Team , CEPH_REF=wip-shraddhaag-enable-cephadm-tests, FROM_IMAGE=quay.io/centos/centos:stream9, org.label-schema.license=GPLv2, io.buildah.version=1.41.3, org.label-schema.build-date=20260216, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfsganesha-5/, CEPH_SHA1=e371433fcd1390a0b82cfb18effc422c1f5d8c1d, CEPH_GIT_REPO=https://github.com/ceph/ceph-ci.git) 2026-02-23T13:08:55.723 INFO:journalctl@ceph.osd.3.trial185.stdout:Feb 23 13:08:55 trial185 bash[67409]: ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-3 2026-02-23T13:08:55.723 INFO:journalctl@ceph.osd.3.trial185.stdout:Feb 23 13:08:55 trial185 podman[67533]: 2026-02-23 13:08:55.563262353 +0000 UTC m=+0.015149172 container create 70586de34679dc2baac83542f2dcf237f74e4cf2a32407aa862bf4387257a7a2 (image=quay.ceph.io/ceph-ci/ceph@sha256:49621faaa845c39b2aae72e4fa24123ee9b2ac3fb8ae834753907068ee2972ea, name=ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-3-deactivate, org.label-schema.vendor=CentOS, org.opencontainers.image.documentation=https://docs.ceph.com/, OSD_FLAVOR=default, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_GIT_REPO=https://github.com/ceph/ceph-ci.git, org.label-schema.build-date=20260216, org.opencontainers.image.authors=Ceph Release Team , io.buildah.version=1.41.3, FROM_IMAGE=quay.io/centos/centos:stream9, CEPH_SHA1=e371433fcd1390a0b82cfb18effc422c1f5d8c1d, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_REF=wip-shraddhaag-enable-cephadm-tests, ceph=True, GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfsganesha-5/) 2026-02-23T13:08:55.723 INFO:journalctl@ceph.osd.3.trial185.stdout:Feb 23 13:08:55 trial185 podman[67533]: 2026-02-23 13:08:55.590298508 +0000 UTC m=+0.042185337 container init 70586de34679dc2baac83542f2dcf237f74e4cf2a32407aa862bf4387257a7a2 (image=quay.ceph.io/ceph-ci/ceph@sha256:49621faaa845c39b2aae72e4fa24123ee9b2ac3fb8ae834753907068ee2972ea, name=ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-3-deactivate, org.label-schema.schema-version=1.0, GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfsganesha-5/, org.label-schema.license=GPLv2, FROM_IMAGE=quay.io/centos/centos:stream9, OSD_FLAVOR=default, org.label-schema.build-date=20260216, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_REF=wip-shraddhaag-enable-cephadm-tests, org.label-schema.vendor=CentOS, CEPH_GIT_REPO=https://github.com/ceph/ceph-ci.git, org.opencontainers.image.authors=Ceph Release Team , ceph=True, org.opencontainers.image.documentation=https://docs.ceph.com/, io.buildah.version=1.41.3, CEPH_SHA1=e371433fcd1390a0b82cfb18effc422c1f5d8c1d) 2026-02-23T13:08:55.723 INFO:journalctl@ceph.osd.3.trial185.stdout:Feb 23 13:08:55 trial185 podman[67533]: 2026-02-23 13:08:55.593342617 +0000 UTC m=+0.045229436 container start 70586de34679dc2baac83542f2dcf237f74e4cf2a32407aa862bf4387257a7a2 (image=quay.ceph.io/ceph-ci/ceph@sha256:49621faaa845c39b2aae72e4fa24123ee9b2ac3fb8ae834753907068ee2972ea, name=ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-3-deactivate, CEPH_GIT_REPO=https://github.com/ceph/ceph-ci.git, org.label-schema.build-date=20260216, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.opencontainers.image.documentation=https://docs.ceph.com/, org.label-schema.license=GPLv2, FROM_IMAGE=quay.io/centos/centos:stream9, GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfsganesha-5/, OSD_FLAVOR=default, CEPH_SHA1=e371433fcd1390a0b82cfb18effc422c1f5d8c1d, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_REF=wip-shraddhaag-enable-cephadm-tests, org.opencontainers.image.authors=Ceph Release Team , io.buildah.version=1.41.3) 2026-02-23T13:08:55.723 INFO:journalctl@ceph.osd.3.trial185.stdout:Feb 23 13:08:55 trial185 podman[67533]: 2026-02-23 13:08:55.59386838 +0000 UTC m=+0.045755209 container attach 70586de34679dc2baac83542f2dcf237f74e4cf2a32407aa862bf4387257a7a2 (image=quay.ceph.io/ceph-ci/ceph@sha256:49621faaa845c39b2aae72e4fa24123ee9b2ac3fb8ae834753907068ee2972ea, name=ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-3-deactivate, org.opencontainers.image.authors=Ceph Release Team , org.label-schema.vendor=CentOS, org.opencontainers.image.documentation=https://docs.ceph.com/, io.buildah.version=1.41.3, org.label-schema.license=GPLv2, FROM_IMAGE=quay.io/centos/centos:stream9, CEPH_GIT_REPO=https://github.com/ceph/ceph-ci.git, CEPH_REF=wip-shraddhaag-enable-cephadm-tests, org.label-schema.schema-version=1.0, OSD_FLAVOR=default, CEPH_SHA1=e371433fcd1390a0b82cfb18effc422c1f5d8c1d, org.label-schema.build-date=20260216, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfsganesha-5/) 2026-02-23T13:08:55.723 INFO:journalctl@ceph.osd.3.trial185.stdout:Feb 23 13:08:55 trial185 podman[67533]: 2026-02-23 13:08:55.557509622 +0000 UTC m=+0.009396451 image pull 63df4b83db6f95a8ba965919b3671cb27173117c23b5c950aaadc8913f56213c quay.ceph.io/ceph-ci/ceph@sha256:49621faaa845c39b2aae72e4fa24123ee9b2ac3fb8ae834753907068ee2972ea 2026-02-23T13:08:55.723 INFO:journalctl@ceph.osd.3.trial185.stdout:Feb 23 13:08:55 trial185 podman[67533]: 2026-02-23 13:08:55.722062338 +0000 UTC m=+0.173949168 container died 70586de34679dc2baac83542f2dcf237f74e4cf2a32407aa862bf4387257a7a2 (image=quay.ceph.io/ceph-ci/ceph@sha256:49621faaa845c39b2aae72e4fa24123ee9b2ac3fb8ae834753907068ee2972ea, name=ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-3-deactivate, org.label-schema.schema-version=1.0, CEPH_REF=wip-shraddhaag-enable-cephadm-tests, FROM_IMAGE=quay.io/centos/centos:stream9, org.label-schema.build-date=20260216, org.label-schema.name=CentOS Stream 9 Base Image, org.opencontainers.image.authors=Ceph Release Team , org.label-schema.vendor=CentOS, org.opencontainers.image.documentation=https://docs.ceph.com/, GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfsganesha-5/, CEPH_GIT_REPO=https://github.com/ceph/ceph-ci.git, ceph=True, io.buildah.version=1.41.3, org.label-schema.license=GPLv2, OSD_FLAVOR=default, CEPH_SHA1=e371433fcd1390a0b82cfb18effc422c1f5d8c1d) 2026-02-23T13:08:55.758 DEBUG:teuthology.orchestra.run.trial185:> sudo pkill -f 'journalctl -f -n 0 -u ceph-5f823084-10b8-11f1-be80-d404e6e7d460@osd.3.service' 2026-02-23T13:08:55.805 DEBUG:teuthology.orchestra.run:got remote process result: None 2026-02-23T13:08:55.806 INFO:tasks.cephadm.osd.3:Stopped osd.3 2026-02-23T13:08:55.871 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:08:55.998 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:08:56.104 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:08:56.102+0000 7f606abda640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:08:56.104 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:08:56.102+0000 7f606abda640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:08:56.117 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:08:56.117 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:08:56.128 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:08:56.128 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:08:56.142 INFO:teuthology.orchestra.run.trial068.stdout:osd.2: {} 2026-02-23T13:08:56.142 INFO:teuthology.orchestra.run.trial068.stderr:osd.2: osd_enable_op_tracker = '' 2026-02-23T13:08:56.143 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.3 2026-02-23T13:08:56.173 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:08:56.173 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:08:56.329 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 84 128 337545 337417 31.3789 0.0859375 1.03133 0.0314887 2026-02-23T13:08:56.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:08:56 trial068 ceph-mon[38805]: Health check failed: 1 osds down (OSD_DOWN) 2026-02-23T13:08:56.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:08:56 trial068 ceph-mon[38805]: osdmap e53: 4 total, 3 up, 4 in 2026-02-23T13:08:56.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:08:56 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2424909978' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:08:57.329 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 85 128 337622 337494 31.0168 0.601562 2.05935 0.0317094 2026-02-23T13:08:57.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:08:57 trial068 ceph-mon[38805]: pgmap v74: 17 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 5 peering, 1 active+recovering+undersized+degraded+remapped, 3 remapped, 1 stale+active+clean, 6 remapped+peering; 2.6 GiB data, 8.4 GiB used, 2.7 TiB / 2.7 TiB avail; 39 MiB/s wr, 5.00k op/s; 881/126549 objects degraded (0.696%); 15 MiB/s, 246 objects/s recovering 2026-02-23T13:08:57.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:08:57 trial068 ceph-mon[38805]: Health check failed: Degraded data redundancy: 881/126549 objects degraded (0.696%), 2 pgs degraded (PG_DEGRADED) 2026-02-23T13:08:57.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:08:57 trial068 ceph-mon[38805]: osdmap e54: 4 total, 3 up, 4 in 2026-02-23T13:08:58.174 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:08:58.321 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:08:58.329 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 86 128 340831 340703 30.9477 25.0703 0.0265022 0.0322997 2026-02-23T13:08:58.448 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:08:58.446+0000 7f3091cba640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:08:58.448 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:08:58.446+0000 7f3091cba640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:08:58.448 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:08:58 trial068 ceph-mon[38805]: osdmap e55: 4 total, 3 up, 4 in 2026-02-23T13:08:58.464 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:08:58.464 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:08:58.478 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:08:58.479 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:08:58.492 INFO:teuthology.orchestra.run.trial068.stdout:osd.2: {} 2026-02-23T13:08:58.492 INFO:teuthology.orchestra.run.trial068.stderr:osd.2: osd_enable_op_tracker = '' 2026-02-23T13:08:58.492 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.3 2026-02-23T13:08:58.531 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:08:58.531 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:08:59.329 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 87 128 343512 343384 30.8327 20.9453 0.0169221 0.0324104 2026-02-23T13:08:59.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:08:59 trial068 ceph-mon[38805]: pgmap v77: 17 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 5 peering, 1 active+recovering+undersized+degraded+remapped, 3 remapped, 1 stale+active+clean, 6 remapped+peering; 2.6 GiB data, 8.4 GiB used, 2.7 TiB / 2.7 TiB avail; 15 MiB/s wr, 1.93k op/s; 881/126549 objects degraded (0.696%); 15 MiB/s, 246 objects/s recovering 2026-02-23T13:08:59.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:08:59 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2662112162' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:09:00.330 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 88 128 345852 345724 30.69 18.2812 0.0234051 0.032551 2026-02-23T13:09:00.532 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:09:00.673 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:09:00.807 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [3] live_osds: [0, 2, 1] 2026-02-23T13:09:00.807 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2026-02-23T13:09:00.807 INFO:tasks.thrashosds.thrasher:Killing osd 1, live_osds are [0, 2, 1] 2026-02-23T13:09:00.807 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2026-02-23T13:09:00.807 DEBUG:teuthology.orchestra.run.trial068:> sudo systemctl stop ceph-5f823084-10b8-11f1-be80-d404e6e7d460@osd.1 2026-02-23T13:09:01.079 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:09:01.077+0000 7f3bf5a5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:09:01.079 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:09:01.077+0000 7f3bf5a5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:09:01.080 INFO:journalctl@ceph.osd.1.trial068.stdout:Feb 23 13:09:00 trial068 systemd[1]: Stopping Ceph osd.1 for 5f823084-10b8-11f1-be80-d404e6e7d460... 2026-02-23T13:09:01.093 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:09:01.093 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:09:01.106 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:09:01.107 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:09:01.117 INFO:teuthology.orchestra.run.trial068.stdout:osd.2: {} 2026-02-23T13:09:01.118 INFO:teuthology.orchestra.run.trial068.stderr:osd.2: osd_enable_op_tracker = '' 2026-02-23T13:09:01.118 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.3 2026-02-23T13:09:01.180 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:09:01.181 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:09:01.329 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 89 128 348293 348165 30.5595 19.0703 0.0200827 0.0326955 2026-02-23T13:09:01.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:09:01 trial068 ceph-mon[38805]: pgmap v78: 17 pgs: 4 active+recovery_wait+undersized+degraded, 2 active+clean, 1 active+undersized+degraded, 3 active+recovering+undersized+degraded, 5 peering, 1 stale+active+clean, 1 remapped+peering; 2.6 GiB data, 8.6 GiB used, 2.7 TiB / 2.7 TiB avail; 18 MiB/s wr, 2.26k op/s; 25338/127290 objects degraded (19.906%); 25 MiB/s, 407 objects/s recovering 2026-02-23T13:09:01.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:09:01 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2496902818' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:09:01.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:09:01 trial068 ceph-mon[38805]: osd.1 marked itself down and dead 2026-02-23T13:09:01.344 INFO:journalctl@ceph.osd.1.trial068.stdout:Feb 23 13:09:01 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-1[82534]: 2026-02-23T13:09:01.093+0000 7f57194d9640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --osd-objectstore=bluestore (PID: 1) UID: 0 2026-02-23T13:09:01.344 INFO:journalctl@ceph.osd.1.trial068.stdout:Feb 23 13:09:01 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-1[82534]: 2026-02-23T13:09:01.093+0000 7f57194d9640 -1 osd.1 55 *** Got signal Terminated *** 2026-02-23T13:09:01.344 INFO:journalctl@ceph.osd.1.trial068.stdout:Feb 23 13:09:01 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-1[82534]: 2026-02-23T13:09:01.093+0000 7f57194d9640 -1 osd.1 55 *** Immediate shutdown (osd_fast_shutdown=true) *** 2026-02-23T13:09:01.716 INFO:journalctl@ceph.osd.1.trial068.stdout:Feb 23 13:09:01 trial068 podman[84393]: 2026-02-23 13:09:01.408628851 +0000 UTC m=+0.328878721 container died a6c602d586023fdefcb285fca8b69f126bbdea24a82f5221490992b55302cbc9 (image=quay.ceph.io/ceph-ci/ceph@sha256:49621faaa845c39b2aae72e4fa24123ee9b2ac3fb8ae834753907068ee2972ea, name=ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-1, org.label-schema.name=CentOS Stream 9 Base Image, org.opencontainers.image.documentation=https://docs.ceph.com/, GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfsganesha-5/, OSD_FLAVOR=default, org.opencontainers.image.authors=Ceph Release Team , ceph=True, FROM_IMAGE=quay.io/centos/centos:stream9, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_REF=wip-shraddhaag-enable-cephadm-tests, CEPH_GIT_REPO=https://github.com/ceph/ceph-ci.git, CEPH_SHA1=e371433fcd1390a0b82cfb18effc422c1f5d8c1d, io.buildah.version=1.41.3, org.label-schema.build-date=20260216, org.label-schema.license=GPLv2) 2026-02-23T13:09:01.716 INFO:journalctl@ceph.osd.1.trial068.stdout:Feb 23 13:09:01 trial068 podman[84393]: 2026-02-23 13:09:01.418453792 +0000 UTC m=+0.338703652 container remove a6c602d586023fdefcb285fca8b69f126bbdea24a82f5221490992b55302cbc9 (image=quay.ceph.io/ceph-ci/ceph@sha256:49621faaa845c39b2aae72e4fa24123ee9b2ac3fb8ae834753907068ee2972ea, name=ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-1, CEPH_GIT_REPO=https://github.com/ceph/ceph-ci.git, CEPH_SHA1=e371433fcd1390a0b82cfb18effc422c1f5d8c1d, OSD_FLAVOR=default, FROM_IMAGE=quay.io/centos/centos:stream9, GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfsganesha-5/, ceph=True, io.buildah.version=1.41.3, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20260216, org.label-schema.license=GPLv2, CEPH_REF=wip-shraddhaag-enable-cephadm-tests, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.opencontainers.image.authors=Ceph Release Team , org.opencontainers.image.documentation=https://docs.ceph.com/) 2026-02-23T13:09:01.716 INFO:journalctl@ceph.osd.1.trial068.stdout:Feb 23 13:09:01 trial068 bash[84393]: ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-1 2026-02-23T13:09:01.716 INFO:journalctl@ceph.osd.1.trial068.stdout:Feb 23 13:09:01 trial068 podman[84549]: 2026-02-23 13:09:01.554609003 +0000 UTC m=+0.016300263 container create d00e4b2f0e2bb461a4bfb0442cd253c4bf2bdf7e181b22ec2a58d342965b60e6 (image=quay.ceph.io/ceph-ci/ceph@sha256:49621faaa845c39b2aae72e4fa24123ee9b2ac3fb8ae834753907068ee2972ea, name=ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-1-deactivate, OSD_FLAVOR=default, GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfsganesha-5/, org.label-schema.build-date=20260216, FROM_IMAGE=quay.io/centos/centos:stream9, CEPH_REF=wip-shraddhaag-enable-cephadm-tests, org.opencontainers.image.authors=Ceph Release Team , org.opencontainers.image.documentation=https://docs.ceph.com/, ceph=True, org.label-schema.vendor=CentOS, CEPH_GIT_REPO=https://github.com/ceph/ceph-ci.git, io.buildah.version=1.41.3, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_SHA1=e371433fcd1390a0b82cfb18effc422c1f5d8c1d, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2) 2026-02-23T13:09:01.716 INFO:journalctl@ceph.osd.1.trial068.stdout:Feb 23 13:09:01 trial068 podman[84549]: 2026-02-23 13:09:01.585480731 +0000 UTC m=+0.047171931 container init d00e4b2f0e2bb461a4bfb0442cd253c4bf2bdf7e181b22ec2a58d342965b60e6 (image=quay.ceph.io/ceph-ci/ceph@sha256:49621faaa845c39b2aae72e4fa24123ee9b2ac3fb8ae834753907068ee2972ea, name=ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-1-deactivate, CEPH_GIT_REPO=https://github.com/ceph/ceph-ci.git, org.label-schema.build-date=20260216, FROM_IMAGE=quay.io/centos/centos:stream9, CEPH_REF=wip-shraddhaag-enable-cephadm-tests, org.label-schema.vendor=CentOS, GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfsganesha-5/, CEPH_SHA1=e371433fcd1390a0b82cfb18effc422c1f5d8c1d, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, OSD_FLAVOR=default, io.buildah.version=1.41.3, org.opencontainers.image.authors=Ceph Release Team , org.label-schema.name=CentOS Stream 9 Base Image, org.opencontainers.image.documentation=https://docs.ceph.com/, ceph=True) 2026-02-23T13:09:01.716 INFO:journalctl@ceph.osd.1.trial068.stdout:Feb 23 13:09:01 trial068 podman[84549]: 2026-02-23 13:09:01.588035819 +0000 UTC m=+0.049727029 container start d00e4b2f0e2bb461a4bfb0442cd253c4bf2bdf7e181b22ec2a58d342965b60e6 (image=quay.ceph.io/ceph-ci/ceph@sha256:49621faaa845c39b2aae72e4fa24123ee9b2ac3fb8ae834753907068ee2972ea, name=ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-1-deactivate, org.label-schema.vendor=CentOS, CEPH_REF=wip-shraddhaag-enable-cephadm-tests, org.opencontainers.image.documentation=https://docs.ceph.com/, GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfsganesha-5/, CEPH_SHA1=e371433fcd1390a0b82cfb18effc422c1f5d8c1d, org.label-schema.schema-version=1.0, CEPH_GIT_REPO=https://github.com/ceph/ceph-ci.git, OSD_FLAVOR=default, ceph=True, org.label-schema.build-date=20260216, FROM_IMAGE=quay.io/centos/centos:stream9, org.label-schema.license=GPLv2, io.buildah.version=1.41.3, org.opencontainers.image.authors=Ceph Release Team , org.label-schema.name=CentOS Stream 9 Base Image) 2026-02-23T13:09:01.716 INFO:journalctl@ceph.osd.1.trial068.stdout:Feb 23 13:09:01 trial068 podman[84549]: 2026-02-23 13:09:01.588544825 +0000 UTC m=+0.050236025 container attach d00e4b2f0e2bb461a4bfb0442cd253c4bf2bdf7e181b22ec2a58d342965b60e6 (image=quay.ceph.io/ceph-ci/ceph@sha256:49621faaa845c39b2aae72e4fa24123ee9b2ac3fb8ae834753907068ee2972ea, name=ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-1-deactivate, org.label-schema.build-date=20260216, CEPH_REF=wip-shraddhaag-enable-cephadm-tests, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, io.buildah.version=1.41.3, org.opencontainers.image.authors=Ceph Release Team , GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfsganesha-5/, CEPH_SHA1=e371433fcd1390a0b82cfb18effc422c1f5d8c1d, CEPH_GIT_REPO=https://github.com/ceph/ceph-ci.git, OSD_FLAVOR=default, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, FROM_IMAGE=quay.io/centos/centos:stream9, org.opencontainers.image.documentation=https://docs.ceph.com/, ceph=True) 2026-02-23T13:09:01.716 INFO:journalctl@ceph.osd.1.trial068.stdout:Feb 23 13:09:01 trial068 podman[84549]: 2026-02-23 13:09:01.547895085 +0000 UTC m=+0.009586275 image pull 63df4b83db6f95a8ba965919b3671cb27173117c23b5c950aaadc8913f56213c quay.ceph.io/ceph-ci/ceph@sha256:49621faaa845c39b2aae72e4fa24123ee9b2ac3fb8ae834753907068ee2972ea 2026-02-23T13:09:01.716 INFO:journalctl@ceph.osd.1.trial068.stdout:Feb 23 13:09:01 trial068 podman[84549]: 2026-02-23 13:09:01.715718329 +0000 UTC m=+0.177409529 container died d00e4b2f0e2bb461a4bfb0442cd253c4bf2bdf7e181b22ec2a58d342965b60e6 (image=quay.ceph.io/ceph-ci/ceph@sha256:49621faaa845c39b2aae72e4fa24123ee9b2ac3fb8ae834753907068ee2972ea, name=ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-1-deactivate, ceph=True, org.opencontainers.image.documentation=https://docs.ceph.com/, FROM_IMAGE=quay.io/centos/centos:stream9, CEPH_REF=wip-shraddhaag-enable-cephadm-tests, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, io.buildah.version=1.41.3, OSD_FLAVOR=default, org.opencontainers.image.authors=Ceph Release Team , GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfsganesha-5/, CEPH_SHA1=e371433fcd1390a0b82cfb18effc422c1f5d8c1d, org.label-schema.build-date=20260216, org.label-schema.schema-version=1.0, CEPH_GIT_REPO=https://github.com/ceph/ceph-ci.git, org.label-schema.name=CentOS Stream 9 Base Image) 2026-02-23T13:09:01.747 DEBUG:teuthology.orchestra.run.trial068:> sudo pkill -f 'journalctl -f -n 0 -u ceph-5f823084-10b8-11f1-be80-d404e6e7d460@osd.1.service' 2026-02-23T13:09:01.799 DEBUG:teuthology.orchestra.run:got remote process result: None 2026-02-23T13:09:01.799 INFO:tasks.cephadm.osd.1:Stopped osd.1 2026-02-23T13:09:02.330 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 90 127 348300 348173 30.2206 0.0625 1.05919 0.0326982 2026-02-23T13:09:02.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:09:02 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T13:09:02.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:09:02 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T13:09:02.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:09:02 trial068 ceph-mon[38805]: Health check update: 2 osds down (OSD_DOWN) 2026-02-23T13:09:02.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:09:02 trial068 ceph-mon[38805]: osdmap e56: 4 total, 2 up, 4 in 2026-02-23T13:09:03.182 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:09:03.304 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:09:03.330 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 91 120 348300 348180 29.8891 0.0546875 1.06262 0.0327189 2026-02-23T13:09:03.408 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:09:03.407+0000 7fe68d6f1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:09:03.409 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:09:03.407+0000 7fe68d6f1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:09:03.421 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:09:03.421 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:09:03.422 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.1 2026-02-23T13:09:03.433 INFO:teuthology.orchestra.run.trial068.stdout:osd.2: {} 2026-02-23T13:09:03.434 INFO:teuthology.orchestra.run.trial068.stderr:osd.2: osd_enable_op_tracker = '' 2026-02-23T13:09:03.434 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.3 2026-02-23T13:09:03.461 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:09:03 trial068 ceph-mon[38805]: pgmap v80: 17 pgs: 4 stale+active+recovery_wait+undersized+degraded, 2 stale+active+clean, 1 stale+active+undersized+degraded, 3 stale+active+recovering+undersized+degraded, 7 active+undersized+degraded; 2.6 GiB data, 8.7 GiB used, 2.7 TiB / 2.7 TiB avail; 6.7 MiB/s wr, 852 op/s; 41578/128463 objects degraded (32.366%); 15 MiB/s, 238 objects/s recovering 2026-02-23T13:09:03.461 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:09:03 trial068 ceph-mon[38805]: osdmap e57: 4 total, 2 up, 4 in 2026-02-23T13:09:03.461 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:09:03 trial068 ceph-mon[38805]: Health check update: Degraded data redundancy: 41578/128463 objects degraded (32.366%), 15 pgs degraded (PG_DEGRADED) 2026-02-23T13:09:03.462 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:09:03.462 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:09:04.323 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:09:03 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:09:03] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:09:04.330 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 92 120 348300 348180 29.5643 0 - 0.0327189 2026-02-23T13:09:04.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:09:03 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:09:03.972Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:09:04.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:09:04 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3125424660' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:09:05.330 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 93 120 348300 348180 29.2464 0 - 0.0327189 2026-02-23T13:09:05.463 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:09:05.583 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:09:05.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:09:05 trial068 ceph-mon[38805]: pgmap v82: 17 pgs: 4 stale+active+recovery_wait+undersized+degraded, 2 stale+active+clean, 1 stale+active+undersized+degraded, 3 stale+active+recovering+undersized+degraded, 7 active+undersized+degraded; 2.6 GiB data, 8.7 GiB used, 2.7 TiB / 2.7 TiB avail; 5.8 MiB/s wr, 738 op/s; 41578/128463 objects degraded (32.366%); 13 MiB/s, 207 objects/s recovering 2026-02-23T13:09:05.694 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:09:05.692+0000 7f37879ee640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:09:05.694 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:09:05.693+0000 7f37879ee640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:09:05.706 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:09:05.706 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:09:05.707 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.1 2026-02-23T13:09:05.718 INFO:teuthology.orchestra.run.trial068.stdout:osd.2: {} 2026-02-23T13:09:05.718 INFO:teuthology.orchestra.run.trial068.stderr:osd.2: osd_enable_op_tracker = '' 2026-02-23T13:09:05.718 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.3 2026-02-23T13:09:05.747 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:09:05.748 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:09:06.330 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 94 120 348300 348180 28.9352 0 - 0.0327189 2026-02-23T13:09:06.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:09:06 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2194673013' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:09:06.800 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [3, 1] live_osds: [0, 2] 2026-02-23T13:09:06.800 INFO:tasks.thrashosds.thrasher:Reviving osd 3 2026-02-23T13:09:06.800 INFO:tasks.cephadm.osd.3:Restarting osd.3 (starting--it wasn't running)... 2026-02-23T13:09:06.800 DEBUG:teuthology.orchestra.run.trial185:osd.3> sudo journalctl -f -n 0 -u ceph-5f823084-10b8-11f1-be80-d404e6e7d460@osd.3.service 2026-02-23T13:09:06.804 DEBUG:teuthology.orchestra.run.trial185:> sudo systemctl start ceph-5f823084-10b8-11f1-be80-d404e6e7d460@osd.3 2026-02-23T13:09:07.080 INFO:journalctl@ceph.osd.3.trial185.stdout:Feb 23 13:09:06 trial185 systemd[1]: Starting Ceph osd.3 for 5f823084-10b8-11f1-be80-d404e6e7d460... 2026-02-23T13:09:07.080 INFO:journalctl@ceph.osd.3.trial185.stdout:Feb 23 13:09:06 trial185 podman[67717]: 2026-02-23 13:09:06.968228589 +0000 UTC m=+0.015695663 container create 3d9414134f55c49b0d8840d9f7d717c87448051753a46a7394021d656545f983 (image=quay.ceph.io/ceph-ci/ceph@sha256:49621faaa845c39b2aae72e4fa24123ee9b2ac3fb8ae834753907068ee2972ea, name=ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-3-activate, org.opencontainers.image.documentation=https://docs.ceph.com/, org.opencontainers.image.authors=Ceph Release Team , org.label-schema.build-date=20260216, OSD_FLAVOR=default, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, FROM_IMAGE=quay.io/centos/centos:stream9, io.buildah.version=1.41.3, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_GIT_REPO=https://github.com/ceph/ceph-ci.git, org.label-schema.license=GPLv2, CEPH_SHA1=e371433fcd1390a0b82cfb18effc422c1f5d8c1d, GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfsganesha-5/, CEPH_REF=wip-shraddhaag-enable-cephadm-tests) 2026-02-23T13:09:07.080 INFO:journalctl@ceph.osd.3.trial185.stdout:Feb 23 13:09:06 trial185 podman[67717]: 2026-02-23 13:09:06.996482498 +0000 UTC m=+0.043949582 container init 3d9414134f55c49b0d8840d9f7d717c87448051753a46a7394021d656545f983 (image=quay.ceph.io/ceph-ci/ceph@sha256:49621faaa845c39b2aae72e4fa24123ee9b2ac3fb8ae834753907068ee2972ea, name=ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-3-activate, org.label-schema.build-date=20260216, OSD_FLAVOR=default, CEPH_SHA1=e371433fcd1390a0b82cfb18effc422c1f5d8c1d, FROM_IMAGE=quay.io/centos/centos:stream9, CEPH_REF=wip-shraddhaag-enable-cephadm-tests, org.opencontainers.image.documentation=https://docs.ceph.com/, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.41.3, ceph=True, GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfsganesha-5/, org.opencontainers.image.authors=Ceph Release Team , CEPH_GIT_REPO=https://github.com/ceph/ceph-ci.git) 2026-02-23T13:09:07.081 INFO:journalctl@ceph.osd.3.trial185.stdout:Feb 23 13:09:06 trial185 podman[67717]: 2026-02-23 13:09:06.999082989 +0000 UTC m=+0.046550063 container start 3d9414134f55c49b0d8840d9f7d717c87448051753a46a7394021d656545f983 (image=quay.ceph.io/ceph-ci/ceph@sha256:49621faaa845c39b2aae72e4fa24123ee9b2ac3fb8ae834753907068ee2972ea, name=ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-3-activate, FROM_IMAGE=quay.io/centos/centos:stream9, io.buildah.version=1.41.3, org.label-schema.build-date=20260216, org.opencontainers.image.documentation=https://docs.ceph.com/, org.label-schema.vendor=CentOS, CEPH_REF=wip-shraddhaag-enable-cephadm-tests, OSD_FLAVOR=default, ceph=True, org.label-schema.schema-version=1.0, org.opencontainers.image.authors=Ceph Release Team , CEPH_GIT_REPO=https://github.com/ceph/ceph-ci.git, org.label-schema.license=GPLv2, GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfsganesha-5/, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_SHA1=e371433fcd1390a0b82cfb18effc422c1f5d8c1d) 2026-02-23T13:09:07.081 INFO:journalctl@ceph.osd.3.trial185.stdout:Feb 23 13:09:06 trial185 podman[67717]: 2026-02-23 13:09:06.999655567 +0000 UTC m=+0.047122662 container attach 3d9414134f55c49b0d8840d9f7d717c87448051753a46a7394021d656545f983 (image=quay.ceph.io/ceph-ci/ceph@sha256:49621faaa845c39b2aae72e4fa24123ee9b2ac3fb8ae834753907068ee2972ea, name=ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-3-activate, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfsganesha-5/, org.opencontainers.image.authors=Ceph Release Team , FROM_IMAGE=quay.io/centos/centos:stream9, io.buildah.version=1.41.3, org.label-schema.build-date=20260216, CEPH_REF=wip-shraddhaag-enable-cephadm-tests, OSD_FLAVOR=default, CEPH_GIT_REPO=https://github.com/ceph/ceph-ci.git, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.opencontainers.image.documentation=https://docs.ceph.com/, CEPH_SHA1=e371433fcd1390a0b82cfb18effc422c1f5d8c1d, org.label-schema.schema-version=1.0) 2026-02-23T13:09:07.081 INFO:journalctl@ceph.osd.3.trial185.stdout:Feb 23 13:09:07 trial185 podman[67717]: 2026-02-23 13:09:06.961706571 +0000 UTC m=+0.009173665 image pull 63df4b83db6f95a8ba965919b3671cb27173117c23b5c950aaadc8913f56213c quay.ceph.io/ceph-ci/ceph@sha256:49621faaa845c39b2aae72e4fa24123ee9b2ac3fb8ae834753907068ee2972ea 2026-02-23T13:09:07.081 INFO:journalctl@ceph.osd.3.trial185.stdout:Feb 23 13:09:07 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-3-activate[67740]: Running command: /usr/bin/ceph-authtool --gen-print-key 2026-02-23T13:09:07.081 INFO:journalctl@ceph.osd.3.trial185.stdout:Feb 23 13:09:07 trial185 bash[67717]: Running command: /usr/bin/ceph-authtool --gen-print-key 2026-02-23T13:09:07.081 INFO:journalctl@ceph.osd.3.trial185.stdout:Feb 23 13:09:07 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-3-activate[67740]: Running command: /usr/bin/ceph-authtool --gen-print-key 2026-02-23T13:09:07.081 INFO:journalctl@ceph.osd.3.trial185.stdout:Feb 23 13:09:07 trial185 bash[67717]: Running command: /usr/bin/ceph-authtool --gen-print-key 2026-02-23T13:09:07.330 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 95 120 348300 348180 28.6307 0 - 0.0327189 2026-02-23T13:09:07.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:09:07 trial068 ceph-mon[38805]: pgmap v83: 17 pgs: 10 undersized+degraded+peered, 7 active+undersized+degraded; 2.7 GiB data, 8.7 GiB used, 2.7 TiB / 2.7 TiB avail; 11 MiB/s wr, 1.35k op/s; 70683/130581 objects degraded (54.130%); 15 MiB/s, 245 objects/s recovering 2026-02-23T13:09:07.607 INFO:journalctl@ceph.osd.3.trial185.stdout:Feb 23 13:09:07 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-3-activate[67740]: --> Failed to activate via raw: did not find any matching OSD to activate 2026-02-23T13:09:07.607 INFO:journalctl@ceph.osd.3.trial185.stdout:Feb 23 13:09:07 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-3-activate[67740]: Running command: /usr/bin/ceph-authtool --gen-print-key 2026-02-23T13:09:07.607 INFO:journalctl@ceph.osd.3.trial185.stdout:Feb 23 13:09:07 trial185 bash[67717]: --> Failed to activate via raw: did not find any matching OSD to activate 2026-02-23T13:09:07.607 INFO:journalctl@ceph.osd.3.trial185.stdout:Feb 23 13:09:07 trial185 bash[67717]: Running command: /usr/bin/ceph-authtool --gen-print-key 2026-02-23T13:09:07.607 INFO:journalctl@ceph.osd.3.trial185.stdout:Feb 23 13:09:07 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-3-activate[67740]: Running command: /usr/bin/ceph-authtool --gen-print-key 2026-02-23T13:09:07.607 INFO:journalctl@ceph.osd.3.trial185.stdout:Feb 23 13:09:07 trial185 bash[67717]: Running command: /usr/bin/ceph-authtool --gen-print-key 2026-02-23T13:09:07.607 INFO:journalctl@ceph.osd.3.trial185.stdout:Feb 23 13:09:07 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-3-activate[67740]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2026-02-23T13:09:07.607 INFO:journalctl@ceph.osd.3.trial185.stdout:Feb 23 13:09:07 trial185 bash[67717]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2026-02-23T13:09:07.607 INFO:journalctl@ceph.osd.3.trial185.stdout:Feb 23 13:09:07 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-3-activate[67740]: Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/vg_nvme/lv_3 --path /var/lib/ceph/osd/ceph-3 --no-mon-config 2026-02-23T13:09:07.607 INFO:journalctl@ceph.osd.3.trial185.stdout:Feb 23 13:09:07 trial185 bash[67717]: Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/vg_nvme/lv_3 --path /var/lib/ceph/osd/ceph-3 --no-mon-config 2026-02-23T13:09:07.607 INFO:journalctl@ceph.osd.3.trial185.stdout:Feb 23 13:09:07 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-3-activate[67740]: Running command: /usr/bin/ln -snf /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-3/block 2026-02-23T13:09:07.607 INFO:journalctl@ceph.osd.3.trial185.stdout:Feb 23 13:09:07 trial185 bash[67717]: Running command: /usr/bin/ln -snf /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-3/block 2026-02-23T13:09:07.607 INFO:journalctl@ceph.osd.3.trial185.stdout:Feb 23 13:09:07 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-3-activate[67740]: Running command: /usr/bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-3/block 2026-02-23T13:09:07.607 INFO:journalctl@ceph.osd.3.trial185.stdout:Feb 23 13:09:07 trial185 bash[67717]: Running command: /usr/bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-3/block 2026-02-23T13:09:07.607 INFO:journalctl@ceph.osd.3.trial185.stdout:Feb 23 13:09:07 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-3-activate[67740]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2026-02-23T13:09:07.608 INFO:journalctl@ceph.osd.3.trial185.stdout:Feb 23 13:09:07 trial185 bash[67717]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2026-02-23T13:09:07.608 INFO:journalctl@ceph.osd.3.trial185.stdout:Feb 23 13:09:07 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-3-activate[67740]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2026-02-23T13:09:07.608 INFO:journalctl@ceph.osd.3.trial185.stdout:Feb 23 13:09:07 trial185 bash[67717]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2026-02-23T13:09:07.608 INFO:journalctl@ceph.osd.3.trial185.stdout:Feb 23 13:09:07 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-3-activate[67740]: --> ceph-volume lvm activate successful for osd ID: 3 2026-02-23T13:09:07.608 INFO:journalctl@ceph.osd.3.trial185.stdout:Feb 23 13:09:07 trial185 bash[67717]: --> ceph-volume lvm activate successful for osd ID: 3 2026-02-23T13:09:07.608 INFO:journalctl@ceph.osd.3.trial185.stdout:Feb 23 13:09:07 trial185 podman[67717]: 2026-02-23 13:09:07.478637778 +0000 UTC m=+0.526104952 container died 3d9414134f55c49b0d8840d9f7d717c87448051753a46a7394021d656545f983 (image=quay.ceph.io/ceph-ci/ceph@sha256:49621faaa845c39b2aae72e4fa24123ee9b2ac3fb8ae834753907068ee2972ea, name=ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-3-activate, org.label-schema.vendor=CentOS, org.label-schema.build-date=20260216, OSD_FLAVOR=default, org.label-schema.schema-version=1.0, FROM_IMAGE=quay.io/centos/centos:stream9, CEPH_REF=wip-shraddhaag-enable-cephadm-tests, org.opencontainers.image.documentation=https://docs.ceph.com/, org.label-schema.license=GPLv2, GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfsganesha-5/, org.opencontainers.image.authors=Ceph Release Team , io.buildah.version=1.41.3, CEPH_GIT_REPO=https://github.com/ceph/ceph-ci.git, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_SHA1=e371433fcd1390a0b82cfb18effc422c1f5d8c1d, ceph=True) 2026-02-23T13:09:07.608 INFO:journalctl@ceph.osd.3.trial185.stdout:Feb 23 13:09:07 trial185 podman[67717]: 2026-02-23 13:09:07.489124151 +0000 UTC m=+0.536591225 container remove 3d9414134f55c49b0d8840d9f7d717c87448051753a46a7394021d656545f983 (image=quay.ceph.io/ceph-ci/ceph@sha256:49621faaa845c39b2aae72e4fa24123ee9b2ac3fb8ae834753907068ee2972ea, name=ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-3-activate, io.buildah.version=1.41.3, org.opencontainers.image.documentation=https://docs.ceph.com/, OSD_FLAVOR=default, CEPH_REF=wip-shraddhaag-enable-cephadm-tests, FROM_IMAGE=quay.io/centos/centos:stream9, org.label-schema.vendor=CentOS, org.label-schema.build-date=20260216, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_SHA1=e371433fcd1390a0b82cfb18effc422c1f5d8c1d, org.label-schema.schema-version=1.0, CEPH_GIT_REPO=https://github.com/ceph/ceph-ci.git, GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfsganesha-5/, ceph=True, org.opencontainers.image.authors=Ceph Release Team ) 2026-02-23T13:09:07.608 INFO:journalctl@ceph.osd.3.trial185.stdout:Feb 23 13:09:07 trial185 podman[67960]: 2026-02-23 13:09:07.568214419 +0000 UTC m=+0.015201070 container create abc20c2683afe7a2d79ae87bcfc64ec8ad3ffa113911de8123e77c586f268c8b (image=quay.ceph.io/ceph-ci/ceph@sha256:49621faaa845c39b2aae72e4fa24123ee9b2ac3fb8ae834753907068ee2972ea, name=ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-3, org.label-schema.license=GPLv2, CEPH_GIT_REPO=https://github.com/ceph/ceph-ci.git, CEPH_SHA1=e371433fcd1390a0b82cfb18effc422c1f5d8c1d, FROM_IMAGE=quay.io/centos/centos:stream9, io.buildah.version=1.41.3, GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfsganesha-5/, CEPH_REF=wip-shraddhaag-enable-cephadm-tests, org.opencontainers.image.authors=Ceph Release Team , ceph=True, OSD_FLAVOR=default, org.label-schema.build-date=20260216, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, org.opencontainers.image.documentation=https://docs.ceph.com/) 2026-02-23T13:09:07.608 INFO:journalctl@ceph.osd.3.trial185.stdout:Feb 23 13:09:07 trial185 podman[67960]: 2026-02-23 13:09:07.599774877 +0000 UTC m=+0.046761538 container init abc20c2683afe7a2d79ae87bcfc64ec8ad3ffa113911de8123e77c586f268c8b (image=quay.ceph.io/ceph-ci/ceph@sha256:49621faaa845c39b2aae72e4fa24123ee9b2ac3fb8ae834753907068ee2972ea, name=ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-3, org.opencontainers.image.authors=Ceph Release Team , io.buildah.version=1.41.3, ceph=True, org.label-schema.build-date=20260216, FROM_IMAGE=quay.io/centos/centos:stream9, GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfsganesha-5/, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_REF=wip-shraddhaag-enable-cephadm-tests, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, CEPH_GIT_REPO=https://github.com/ceph/ceph-ci.git, CEPH_SHA1=e371433fcd1390a0b82cfb18effc422c1f5d8c1d, OSD_FLAVOR=default, org.opencontainers.image.documentation=https://docs.ceph.com/) 2026-02-23T13:09:07.608 INFO:journalctl@ceph.osd.3.trial185.stdout:Feb 23 13:09:07 trial185 podman[67960]: 2026-02-23 13:09:07.602224827 +0000 UTC m=+0.049211468 container start abc20c2683afe7a2d79ae87bcfc64ec8ad3ffa113911de8123e77c586f268c8b (image=quay.ceph.io/ceph-ci/ceph@sha256:49621faaa845c39b2aae72e4fa24123ee9b2ac3fb8ae834753907068ee2972ea, name=ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-3, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.opencontainers.image.documentation=https://docs.ceph.com/, GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfsganesha-5/, org.opencontainers.image.authors=Ceph Release Team , CEPH_SHA1=e371433fcd1390a0b82cfb18effc422c1f5d8c1d, FROM_IMAGE=quay.io/centos/centos:stream9, org.label-schema.license=GPLv2, org.label-schema.build-date=20260216, io.buildah.version=1.41.3, ceph=True, CEPH_GIT_REPO=https://github.com/ceph/ceph-ci.git, OSD_FLAVOR=default, CEPH_REF=wip-shraddhaag-enable-cephadm-tests, org.label-schema.name=CentOS Stream 9 Base Image) 2026-02-23T13:09:07.608 INFO:journalctl@ceph.osd.3.trial185.stdout:Feb 23 13:09:07 trial185 bash[67960]: abc20c2683afe7a2d79ae87bcfc64ec8ad3ffa113911de8123e77c586f268c8b 2026-02-23T13:09:07.608 INFO:journalctl@ceph.osd.3.trial185.stdout:Feb 23 13:09:07 trial185 podman[67960]: 2026-02-23 13:09:07.561964218 +0000 UTC m=+0.008950880 image pull 63df4b83db6f95a8ba965919b3671cb27173117c23b5c950aaadc8913f56213c quay.ceph.io/ceph-ci/ceph@sha256:49621faaa845c39b2aae72e4fa24123ee9b2ac3fb8ae834753907068ee2972ea 2026-02-23T13:09:07.608 INFO:journalctl@ceph.osd.3.trial185.stdout:Feb 23 13:09:07 trial185 systemd[1]: Started Ceph osd.3 for 5f823084-10b8-11f1-be80-d404e6e7d460. 2026-02-23T13:09:07.608 DEBUG:teuthology.orchestra.run.trial185:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph daemon osd.3 dump_ops_in_flight 2026-02-23T13:09:07.732 INFO:teuthology.orchestra.run.trial185.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/config/ceph.conf 2026-02-23T13:09:07.749 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:09:07.851 INFO:teuthology.orchestra.run.trial185.stderr:no valid command found; 10 closest matches: 2026-02-23T13:09:07.851 INFO:teuthology.orchestra.run.trial185.stderr:0 2026-02-23T13:09:07.851 INFO:teuthology.orchestra.run.trial185.stderr:1 2026-02-23T13:09:07.851 INFO:teuthology.orchestra.run.trial185.stderr:2 2026-02-23T13:09:07.852 INFO:teuthology.orchestra.run.trial185.stderr:abort 2026-02-23T13:09:07.852 INFO:teuthology.orchestra.run.trial185.stderr:assert 2026-02-23T13:09:07.852 INFO:teuthology.orchestra.run.trial185.stderr:bluestore collections 2026-02-23T13:09:07.852 INFO:teuthology.orchestra.run.trial185.stderr:bluestore compression stats [] 2026-02-23T13:09:07.852 INFO:teuthology.orchestra.run.trial185.stderr:bluestore list [] [] 2026-02-23T13:09:07.852 INFO:teuthology.orchestra.run.trial185.stderr:bluestore onode metadata 2026-02-23T13:09:07.852 INFO:teuthology.orchestra.run.trial185.stderr:bluestore show sharding 2026-02-23T13:09:07.852 INFO:teuthology.orchestra.run.trial185.stderr:admin_socket: invalid command 2026-02-23T13:09:07.859 INFO:journalctl@ceph.osd.3.trial185.stdout:Feb 23 13:09:07 trial185 ceph-osd[67989]: -- 10.20.193.185:0/2132420565 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x2e6bc00 msgr2=0x3ca8000 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).process 2026-02-23T13:09:07.859 INFO:journalctl@ceph.osd.3.trial185.stdout:Feb 23 13:09:07 trial185 ceph-osd[67989]: -- 10.20.193.185:0/2132420565 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x2e6bc00 msgr2=0x3ca8000 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read continue len=32 2026-02-23T13:09:07.860 INFO:journalctl@ceph.osd.3.trial185.stdout:Feb 23 13:09:07 trial185 ceph-osd[67989]: --2- 10.20.193.185:0/2132420565 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x2e6bc00 0x3ca8000 crc :-1 s=READY pgs=364 gs=1 cs=0 l=1 c_cookie=1694338221104916030 s_cookie=0 reconnecting=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main r=0 2026-02-23T13:09:07.860 INFO:journalctl@ceph.osd.3.trial185.stdout:Feb 23 13:09:07 trial185 ceph-osd[67989]: --2- 10.20.193.185:0/2132420565 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x2e6bc00 0x3ca8000 crc :-1 s=THROTTLE_MESSAGE pgs=364 gs=1 cs=0 l=1 c_cookie=1694338221104916030 s_cookie=0 reconnecting=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).throttle_message 2026-02-23T13:09:07.860 INFO:journalctl@ceph.osd.3.trial185.stdout:Feb 23 13:09:07 trial185 ceph-osd[67989]: --2- 10.20.193.185:0/2132420565 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x2e6bc00 0x3ca8000 crc :-1 s=THROTTLE_BYTES pgs=364 gs=1 cs=0 l=1 c_cookie=1694338221104916030 s_cookie=0 reconnecting=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).throttle_bytes 2026-02-23T13:09:07.860 INFO:journalctl@ceph.osd.3.trial185.stdout:Feb 23 13:09:07 trial185 ceph-osd[67989]: --2- 10.20.193.185:0/2132420565 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x2e6bc00 0x3ca8000 crc :-1 s=THROTTLE_DISPATCH_QUEUE pgs=364 gs=1 cs=0 l=1 c_cookie=1694338221104916030 s_cookie=0 reconnecting=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).throttle_dispatch_queue 2026-02-23T13:09:07.860 INFO:journalctl@ceph.osd.3.trial185.stdout:Feb 23 13:09:07 trial185 ceph-osd[67989]: --2- 10.20.193.185:0/2132420565 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x2e6bc00 0x3ca8000 crc :-1 s=THROTTLE_DONE pgs=364 gs=1 cs=0 l=1 c_cookie=1694338221104916030 s_cookie=0 reconnecting=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).read_frame_segment seg_idx=0 2026-02-23T13:09:07.860 INFO:journalctl@ceph.osd.3.trial185.stdout:Feb 23 13:09:07 trial185 ceph-osd[67989]: -- 10.20.193.185:0/2132420565 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x2e6bc00 msgr2=0x3ca8000 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read start len=45 2026-02-23T13:09:07.860 INFO:journalctl@ceph.osd.3.trial185.stdout:Feb 23 13:09:07 trial185 ceph-osd[67989]: --2- 10.20.193.185:0/2132420565 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x2e6bc00 0x3ca8000 crc :-1 s=THROTTLE_DONE pgs=364 gs=1 cs=0 l=1 c_cookie=1694338221104916030 s_cookie=0 reconnecting=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_segment r=0 2026-02-23T13:09:07.860 INFO:journalctl@ceph.osd.3.trial185.stdout:Feb 23 13:09:07 trial185 ceph-osd[67989]: --2- 10.20.193.185:0/2132420565 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x2e6bc00 0x3ca8000 crc :-1 s=THROTTLE_DONE pgs=364 gs=1 cs=0 l=1 c_cookie=1694338221104916030 s_cookie=0 reconnecting=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).read_frame_segment seg_idx=1 2026-02-23T13:09:07.860 INFO:journalctl@ceph.osd.3.trial185.stdout:Feb 23 13:09:07 trial185 ceph-osd[67989]: -- 10.20.193.185:0/2132420565 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x2e6bc00 msgr2=0x3ca8000 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read start len=194 2026-02-23T13:09:07.860 INFO:journalctl@ceph.osd.3.trial185.stdout:Feb 23 13:09:07 trial185 ceph-osd[67989]: --2- 10.20.193.185:0/2132420565 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x2e6bc00 0x3ca8000 crc :-1 s=THROTTLE_DONE pgs=364 gs=1 cs=0 l=1 c_cookie=1694338221104916030 s_cookie=0 reconnecting=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_segment r=0 2026-02-23T13:09:07.861 INFO:journalctl@ceph.osd.3.trial185.stdout:Feb 23 13:09:07 trial185 ceph-osd[67989]: -- 10.20.193.185:0/2132420565 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x2e6bc00 msgr2=0x3ca8000 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read start len=13 2026-02-23T13:09:07.861 INFO:journalctl@ceph.osd.3.trial185.stdout:Feb 23 13:09:07 trial185 ceph-osd[67989]: --2- 10.20.193.185:0/2132420565 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x2e6bc00 0x3ca8000 crc :-1 s=THROTTLE_DONE pgs=364 gs=1 cs=0 l=1 c_cookie=1694338221104916030 s_cookie=0 reconnecting=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_epilogue_main r=0 2026-02-23T13:09:07.861 INFO:journalctl@ceph.osd.3.trial185.stdout:Feb 23 13:09:07 trial185 ceph-osd[67989]: --2- 10.20.193.185:0/2132420565 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x2e6bc00 0x3ca8000 crc :-1 s=THROTTLE_DONE pgs=364 gs=1 cs=0 l=1 c_cookie=1694338221104916030 s_cookie=0 reconnecting=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_dispatch tag=17 2026-02-23T13:09:07.861 INFO:journalctl@ceph.osd.3.trial185.stdout:Feb 23 13:09:07 trial185 ceph-osd[67989]: --2- 10.20.193.185:0/2132420565 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x2e6bc00 0x3ca8000 crc :-1 s=THROTTLE_DONE pgs=364 gs=1 cs=0 l=1 c_cookie=1694338221104916030 s_cookie=0 reconnecting=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_message 2026-02-23T13:09:07.861 INFO:journalctl@ceph.osd.3.trial185.stdout:Feb 23 13:09:07 trial185 ceph-osd[67989]: --2- 10.20.193.185:0/2132420565 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x2e6bc00 0x3ca8000 crc :-1 s=THROTTLE_DONE pgs=364 gs=1 cs=0 l=1 c_cookie=1694338221104916030 s_cookie=0 reconnecting=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_message got 194 + 0 + 0 byte message. envelope type=18 src mon.0 off 0 2026-02-23T13:09:07.861 INFO:journalctl@ceph.osd.3.trial185.stdout:Feb 23 13:09:07 trial185 ceph-osd[67989]: --2- 10.20.193.185:0/2132420565 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x2e6bc00 0x3ca8000 crc :-1 s=READ_MESSAGE_COMPLETE pgs=364 gs=1 cs=0 l=1 c_cookie=1694338221104916030 s_cookie=0 reconnecting=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_message received message m=0x2ecf860 seq=4 from=mon.0 type=18 auth_reply(proto 2 0 (0) Success) 2026-02-23T13:09:07.861 INFO:journalctl@ceph.osd.3.trial185.stdout:Feb 23 13:09:07 trial185 ceph-osd[67989]: -- 10.20.193.185:0/2132420565 queue 0x2ecf860 prio 196 2026-02-23T13:09:07.861 INFO:journalctl@ceph.osd.3.trial185.stdout:Feb 23 13:09:07 trial185 ceph-osd[67989]: --2- 10.20.193.185:0/2132420565 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x2e6bc00 0x3ca8000 crc :-1 s=READY pgs=364 gs=1 cs=0 l=1 c_cookie=1694338221104916030 s_cookie=0 reconnecting=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).read_frame 2026-02-23T13:09:07.861 INFO:journalctl@ceph.osd.3.trial185.stdout:Feb 23 13:09:07 trial185 ceph-osd[67989]: -- 10.20.193.185:0/2132420565 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x2e6bc00 msgr2=0x3ca8000 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read start len=32 2026-02-23T13:09:07.861 INFO:journalctl@ceph.osd.3.trial185.stdout:Feb 23 13:09:07 trial185 ceph-osd[67989]: -- 10.20.193.185:0/2132420565 <== mon.0 v2:10.20.193.68:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (crc 0 0 0) 0x2ecf860 con 0x2e6bc00 2026-02-23T13:09:07.861 INFO:journalctl@ceph.osd.3.trial185.stdout:Feb 23 13:09:07 trial185 ceph-osd[67989]: -- 10.20.193.185:0/2132420565 dispatch_throttle_release 194 to dispatch throttler 194/104857600 2026-02-23T13:09:07.861 INFO:journalctl@ceph.osd.3.trial185.stdout:Feb 23 13:09:07 trial185 ceph-osd[67989]: -- 10.20.193.185:0/2132420565 done calling dispatch on 0x2ecf860 2026-02-23T13:09:07.862 INFO:journalctl@ceph.osd.3.trial185.stdout:Feb 23 13:09:07 trial185 ceph-osd[67989]: WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:09:07.862 INFO:journalctl@ceph.osd.3.trial185.stdout:Feb 23 13:09:07 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-3[67985]: 2026-02-23T13:09:07.616+0000 7fe5d14ea780 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:09:07.862 INFO:journalctl@ceph.osd.3.trial185.stdout:Feb 23 13:09:07 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-3[67985]: 2026-02-23T13:09:07.616+0000 7fe5d14ea780 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:09:07.862 INFO:journalctl@ceph.osd.3.trial185.stdout:Feb 23 13:09:07 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-3[67985]: 2026-02-23T13:09:07.617+0000 7fe5d14ea780 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:09:07.862 INFO:journalctl@ceph.osd.3.trial185.stdout:Feb 23 13:09:07 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-3[67985]: 2026-02-23T13:09:07.846+0000 7fe5d14ea780 -1 Falling back to public interface 2026-02-23T13:09:07.867 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:09:07.880 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-23T13:09:07.881 INFO:tasks.cephadm.ceph_manager.ceph:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2026-02-23T13:09:07.963 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:09:07.962+0000 7f3a38b18640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:09:07.964 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:09:07.962+0000 7f3a38b18640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:09:07.975 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:09:07.976 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:09:07.976 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.1 2026-02-23T13:09:07.987 INFO:teuthology.orchestra.run.trial068.stdout:osd.2: {} 2026-02-23T13:09:07.988 INFO:teuthology.orchestra.run.trial068.stderr:osd.2: osd_enable_op_tracker = '' 2026-02-23T13:09:07.988 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.3 2026-02-23T13:09:08.018 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:09:08.019 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:09:08.330 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 96 120 348300 348180 28.3324 0 - 0.0327189 2026-02-23T13:09:08.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:09:08 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/4185392396' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:09:09.330 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 97 120 348300 348180 28.0403 0 - 0.0327189 2026-02-23T13:09:09.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:09:09 trial068 ceph-mon[38805]: pgmap v84: 17 pgs: 10 undersized+degraded+peered, 7 active+undersized+degraded; 2.7 GiB data, 8.7 GiB used, 2.7 TiB / 2.7 TiB avail; 8.6 MiB/s wr, 1.10k op/s; 70683/130581 objects degraded (54.130%); 7.8 MiB/s, 124 objects/s recovering 2026-02-23T13:09:09.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:09:09 trial068 ceph-mon[38805]: Health check update: Degraded data redundancy: 70683/130581 objects degraded (54.130%), 17 pgs degraded (PG_DEGRADED) 2026-02-23T13:09:10.020 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:09:10.139 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:09:10.239 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:09:10.237+0000 7f3e67490640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:09:10.239 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:09:10.237+0000 7f3e67490640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:09:10.251 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:09:10.251 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:09:10.251 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.1 2026-02-23T13:09:10.263 INFO:teuthology.orchestra.run.trial068.stdout:osd.2: {} 2026-02-23T13:09:10.264 INFO:teuthology.orchestra.run.trial068.stderr:osd.2: osd_enable_op_tracker = '' 2026-02-23T13:09:10.264 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.3 2026-02-23T13:09:10.294 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:09:10.295 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:09:10.330 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 98 120 348300 348180 27.7542 0 - 0.0327189 2026-02-23T13:09:10.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:09:10 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/987045585' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:09:11.330 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 99 120 348300 348180 27.4739 0 - 0.0327189 2026-02-23T13:09:11.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:09:11 trial068 ceph-mon[38805]: pgmap v85: 17 pgs: 10 undersized+degraded+peered, 7 active+undersized+degraded; 2.7 GiB data, 8.7 GiB used, 2.7 TiB / 2.7 TiB avail; 4.9 MiB/s wr, 633 op/s; 70683/130581 objects degraded (54.130%); 3.7 MiB/s, 59 objects/s recovering 2026-02-23T13:09:11.856 INFO:journalctl@ceph.osd.3.trial185.stdout:Feb 23 13:09:11 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-3[67985]: 2026-02-23T13:09:11.517+0000 7fe5d14ea780 -1 osd.3 53 log_to_monitors true 2026-02-23T13:09:12.296 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:09:12.330 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T13:09:12.330509+0000 min lat: 0.000727387 max lat: 2.47295 avg lat: 0.0327189 2026-02-23T13:09:12.330 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T13:09:12.331 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 100 120 348300 348180 27.1992 0 - 0.0327189 2026-02-23T13:09:12.414 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:09:12.518 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:09:12.516+0000 7ff6ddbea640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:09:12.518 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:09:12.517+0000 7ff6ddbea640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:09:12.530 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:09:12.531 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:09:12.531 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.1 2026-02-23T13:09:12.542 INFO:teuthology.orchestra.run.trial068.stdout:osd.2: {} 2026-02-23T13:09:12.542 INFO:teuthology.orchestra.run.trial068.stderr:osd.2: osd_enable_op_tracker = '' 2026-02-23T13:09:12.542 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.3 2026-02-23T13:09:12.570 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:09:12.571 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:09:12.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:09:12 trial068 ceph-mon[38805]: from='osd.3 [v2:10.20.193.185:6808/3352069840,v1:10.20.193.185:6809/3352069840]' entity='osd.3' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]} : dispatch 2026-02-23T13:09:12.606 INFO:journalctl@ceph.osd.3.trial185.stdout:Feb 23 13:09:12 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-3[67985]: 2026-02-23T13:09:12.339+0000 7fe5c532b640 -1 osd.3 53 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2026-02-23T13:09:12.882 DEBUG:teuthology.orchestra.run.trial185:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph daemon osd.3 dump_ops_in_flight 2026-02-23T13:09:13.002 INFO:teuthology.orchestra.run.trial185.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/config/ceph.conf 2026-02-23T13:09:13.114 INFO:teuthology.orchestra.run.trial185.stdout:{ 2026-02-23T13:09:13.114 INFO:teuthology.orchestra.run.trial185.stdout: "ops": [], 2026-02-23T13:09:13.114 INFO:teuthology.orchestra.run.trial185.stdout: "num_ops": 0 2026-02-23T13:09:13.114 INFO:teuthology.orchestra.run.trial185.stdout:} 2026-02-23T13:09:13.148 INFO:tasks.thrashosds.thrasher:Reviving osd 1 2026-02-23T13:09:13.148 INFO:tasks.cephadm.osd.1:Restarting osd.1 (starting--it wasn't running)... 2026-02-23T13:09:13.148 DEBUG:teuthology.orchestra.run.trial068:osd.1> sudo journalctl -f -n 0 -u ceph-5f823084-10b8-11f1-be80-d404e6e7d460@osd.1.service 2026-02-23T13:09:13.151 DEBUG:teuthology.orchestra.run.trial068:> sudo systemctl start ceph-5f823084-10b8-11f1-be80-d404e6e7d460@osd.1 2026-02-23T13:09:13.331 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 101 120 348300 348180 26.9299 0 - 0.0327189 2026-02-23T13:09:13.432 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:09:13 trial068 ceph-mon[38805]: pgmap v86: 17 pgs: 10 undersized+degraded+peered, 7 active+undersized+degraded; 2.7 GiB data, 8.7 GiB used, 2.7 TiB / 2.7 TiB avail; 4.4 MiB/s wr, 565 op/s; 70683/130581 objects degraded (54.130%); 3.3 MiB/s, 52 objects/s recovering 2026-02-23T13:09:13.432 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:09:13 trial068 ceph-mon[38805]: from='osd.3 [v2:10.20.193.185:6808/3352069840,v1:10.20.193.185:6809/3352069840]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2026-02-23T13:09:13.432 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:09:13 trial068 ceph-mon[38805]: osdmap e58: 4 total, 2 up, 4 in 2026-02-23T13:09:13.432 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:09:13 trial068 ceph-mon[38805]: from='osd.3 [v2:10.20.193.185:6808/3352069840,v1:10.20.193.185:6809/3352069840]' entity='osd.3' cmd={"prefix": "osd crush create-or-move", "id": 3, "weight":0.6836, "args": ["host=trial185", "root=default"]} : dispatch 2026-02-23T13:09:13.432 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:09:13 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2277842778' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:09:13.433 INFO:journalctl@ceph.osd.1.trial068.stdout:Feb 23 13:09:13 trial068 systemd[1]: Starting Ceph osd.1 for 5f823084-10b8-11f1-be80-d404e6e7d460... 2026-02-23T13:09:13.433 INFO:journalctl@ceph.osd.1.trial068.stdout:Feb 23 13:09:13 trial068 podman[85294]: 2026-02-23 13:09:13.314867131 +0000 UTC m=+0.016618791 container create d48f950fd6de038c511a3deedcb6f7a719e07784a40652f87f66a27a28c09c0a (image=quay.ceph.io/ceph-ci/ceph@sha256:49621faaa845c39b2aae72e4fa24123ee9b2ac3fb8ae834753907068ee2972ea, name=ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-1-activate, CEPH_SHA1=e371433fcd1390a0b82cfb18effc422c1f5d8c1d, org.label-schema.vendor=CentOS, org.opencontainers.image.authors=Ceph Release Team , org.label-schema.license=GPLv2, OSD_FLAVOR=default, org.label-schema.schema-version=1.0, CEPH_REF=wip-shraddhaag-enable-cephadm-tests, io.buildah.version=1.41.3, org.opencontainers.image.documentation=https://docs.ceph.com/, GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfsganesha-5/, CEPH_GIT_REPO=https://github.com/ceph/ceph-ci.git, ceph=True, org.label-schema.build-date=20260216, FROM_IMAGE=quay.io/centos/centos:stream9, org.label-schema.name=CentOS Stream 9 Base Image) 2026-02-23T13:09:13.433 INFO:journalctl@ceph.osd.1.trial068.stdout:Feb 23 13:09:13 trial068 podman[85294]: 2026-02-23 13:09:13.359464054 +0000 UTC m=+0.061215635 container init d48f950fd6de038c511a3deedcb6f7a719e07784a40652f87f66a27a28c09c0a (image=quay.ceph.io/ceph-ci/ceph@sha256:49621faaa845c39b2aae72e4fa24123ee9b2ac3fb8ae834753907068ee2972ea, name=ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-1-activate, GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfsganesha-5/, OSD_FLAVOR=default, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, CEPH_SHA1=e371433fcd1390a0b82cfb18effc422c1f5d8c1d, CEPH_GIT_REPO=https://github.com/ceph/ceph-ci.git, org.label-schema.build-date=20260216, CEPH_REF=wip-shraddhaag-enable-cephadm-tests, org.opencontainers.image.authors=Ceph Release Team , org.opencontainers.image.documentation=https://docs.ceph.com/, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, FROM_IMAGE=quay.io/centos/centos:stream9, io.buildah.version=1.41.3, org.label-schema.license=GPLv2) 2026-02-23T13:09:13.434 INFO:journalctl@ceph.osd.1.trial068.stdout:Feb 23 13:09:13 trial068 podman[85294]: 2026-02-23 13:09:13.362132987 +0000 UTC m=+0.063884557 container start d48f950fd6de038c511a3deedcb6f7a719e07784a40652f87f66a27a28c09c0a (image=quay.ceph.io/ceph-ci/ceph@sha256:49621faaa845c39b2aae72e4fa24123ee9b2ac3fb8ae834753907068ee2972ea, name=ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-1-activate, CEPH_SHA1=e371433fcd1390a0b82cfb18effc422c1f5d8c1d, org.opencontainers.image.documentation=https://docs.ceph.com/, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.41.3, org.label-schema.license=GPLv2, CEPH_GIT_REPO=https://github.com/ceph/ceph-ci.git, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, CEPH_REF=wip-shraddhaag-enable-cephadm-tests, org.opencontainers.image.authors=Ceph Release Team , GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfsganesha-5/, OSD_FLAVOR=default, org.label-schema.build-date=20260216, FROM_IMAGE=quay.io/centos/centos:stream9) 2026-02-23T13:09:13.434 INFO:journalctl@ceph.osd.1.trial068.stdout:Feb 23 13:09:13 trial068 podman[85294]: 2026-02-23 13:09:13.362567132 +0000 UTC m=+0.064318712 container attach d48f950fd6de038c511a3deedcb6f7a719e07784a40652f87f66a27a28c09c0a (image=quay.ceph.io/ceph-ci/ceph@sha256:49621faaa845c39b2aae72e4fa24123ee9b2ac3fb8ae834753907068ee2972ea, name=ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-1-activate, FROM_IMAGE=quay.io/centos/centos:stream9, CEPH_REF=wip-shraddhaag-enable-cephadm-tests, OSD_FLAVOR=default, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20260216, org.opencontainers.image.authors=Ceph Release Team , io.buildah.version=1.41.3, org.opencontainers.image.documentation=https://docs.ceph.com/, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfsganesha-5/, CEPH_SHA1=e371433fcd1390a0b82cfb18effc422c1f5d8c1d, CEPH_GIT_REPO=https://github.com/ceph/ceph-ci.git, ceph=True) 2026-02-23T13:09:13.434 INFO:journalctl@ceph.osd.1.trial068.stdout:Feb 23 13:09:13 trial068 podman[85294]: 2026-02-23 13:09:13.309131229 +0000 UTC m=+0.010882809 image pull 63df4b83db6f95a8ba965919b3671cb27173117c23b5c950aaadc8913f56213c quay.ceph.io/ceph-ci/ceph@sha256:49621faaa845c39b2aae72e4fa24123ee9b2ac3fb8ae834753907068ee2972ea 2026-02-23T13:09:13.434 INFO:journalctl@ceph.osd.1.trial068.stdout:Feb 23 13:09:13 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-1-activate[85318]: Running command: /usr/bin/ceph-authtool --gen-print-key 2026-02-23T13:09:13.434 INFO:journalctl@ceph.osd.1.trial068.stdout:Feb 23 13:09:13 trial068 bash[85294]: Running command: /usr/bin/ceph-authtool --gen-print-key 2026-02-23T13:09:13.843 INFO:journalctl@ceph.osd.1.trial068.stdout:Feb 23 13:09:13 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-1-activate[85318]: Running command: /usr/bin/ceph-authtool --gen-print-key 2026-02-23T13:09:13.844 INFO:journalctl@ceph.osd.1.trial068.stdout:Feb 23 13:09:13 trial068 bash[85294]: Running command: /usr/bin/ceph-authtool --gen-print-key 2026-02-23T13:09:14.236 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:09:13 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:09:13] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:09:14.237 INFO:journalctl@ceph.osd.1.trial068.stdout:Feb 23 13:09:13 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-1-activate[85318]: --> Failed to activate via raw: did not find any matching OSD to activate 2026-02-23T13:09:14.237 INFO:journalctl@ceph.osd.1.trial068.stdout:Feb 23 13:09:13 trial068 bash[85294]: --> Failed to activate via raw: did not find any matching OSD to activate 2026-02-23T13:09:14.238 INFO:journalctl@ceph.osd.1.trial068.stdout:Feb 23 13:09:13 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-1-activate[85318]: Running command: /usr/bin/ceph-authtool --gen-print-key 2026-02-23T13:09:14.238 INFO:journalctl@ceph.osd.1.trial068.stdout:Feb 23 13:09:13 trial068 bash[85294]: Running command: /usr/bin/ceph-authtool --gen-print-key 2026-02-23T13:09:14.238 INFO:journalctl@ceph.osd.1.trial068.stdout:Feb 23 13:09:13 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-1-activate[85318]: Running command: /usr/bin/ceph-authtool --gen-print-key 2026-02-23T13:09:14.238 INFO:journalctl@ceph.osd.1.trial068.stdout:Feb 23 13:09:13 trial068 bash[85294]: Running command: /usr/bin/ceph-authtool --gen-print-key 2026-02-23T13:09:14.238 INFO:journalctl@ceph.osd.1.trial068.stdout:Feb 23 13:09:14 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-1-activate[85318]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2026-02-23T13:09:14.238 INFO:journalctl@ceph.osd.1.trial068.stdout:Feb 23 13:09:14 trial068 bash[85294]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2026-02-23T13:09:14.238 INFO:journalctl@ceph.osd.1.trial068.stdout:Feb 23 13:09:14 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-1-activate[85318]: Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/vg_nvme/lv_3 --path /var/lib/ceph/osd/ceph-1 --no-mon-config 2026-02-23T13:09:14.239 INFO:journalctl@ceph.osd.1.trial068.stdout:Feb 23 13:09:14 trial068 bash[85294]: Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/vg_nvme/lv_3 --path /var/lib/ceph/osd/ceph-1 --no-mon-config 2026-02-23T13:09:14.239 INFO:journalctl@ceph.osd.1.trial068.stdout:Feb 23 13:09:14 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-1-activate[85318]: Running command: /usr/bin/ln -snf /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-1/block 2026-02-23T13:09:14.239 INFO:journalctl@ceph.osd.1.trial068.stdout:Feb 23 13:09:14 trial068 bash[85294]: Running command: /usr/bin/ln -snf /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-1/block 2026-02-23T13:09:14.239 INFO:journalctl@ceph.osd.1.trial068.stdout:Feb 23 13:09:14 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-1-activate[85318]: Running command: /usr/bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-1/block 2026-02-23T13:09:14.239 INFO:journalctl@ceph.osd.1.trial068.stdout:Feb 23 13:09:14 trial068 bash[85294]: Running command: /usr/bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-1/block 2026-02-23T13:09:14.239 INFO:journalctl@ceph.osd.1.trial068.stdout:Feb 23 13:09:14 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-1-activate[85318]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2026-02-23T13:09:14.239 INFO:journalctl@ceph.osd.1.trial068.stdout:Feb 23 13:09:14 trial068 bash[85294]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2026-02-23T13:09:14.239 INFO:journalctl@ceph.osd.1.trial068.stdout:Feb 23 13:09:14 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-1-activate[85318]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2026-02-23T13:09:14.240 INFO:journalctl@ceph.osd.1.trial068.stdout:Feb 23 13:09:14 trial068 bash[85294]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2026-02-23T13:09:14.240 INFO:journalctl@ceph.osd.1.trial068.stdout:Feb 23 13:09:14 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-1-activate[85318]: --> ceph-volume lvm activate successful for osd ID: 1 2026-02-23T13:09:14.240 INFO:journalctl@ceph.osd.1.trial068.stdout:Feb 23 13:09:14 trial068 bash[85294]: --> ceph-volume lvm activate successful for osd ID: 1 2026-02-23T13:09:14.240 INFO:journalctl@ceph.osd.1.trial068.stdout:Feb 23 13:09:14 trial068 podman[85294]: 2026-02-23 13:09:14.117342221 +0000 UTC m=+0.819093801 container died d48f950fd6de038c511a3deedcb6f7a719e07784a40652f87f66a27a28c09c0a (image=quay.ceph.io/ceph-ci/ceph@sha256:49621faaa845c39b2aae72e4fa24123ee9b2ac3fb8ae834753907068ee2972ea, name=ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-1-activate, CEPH_GIT_REPO=https://github.com/ceph/ceph-ci.git, org.label-schema.schema-version=1.0, io.buildah.version=1.41.3, org.label-schema.name=CentOS Stream 9 Base Image, FROM_IMAGE=quay.io/centos/centos:stream9, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, OSD_FLAVOR=default, CEPH_SHA1=e371433fcd1390a0b82cfb18effc422c1f5d8c1d, CEPH_REF=wip-shraddhaag-enable-cephadm-tests, GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfsganesha-5/, ceph=True, org.label-schema.build-date=20260216, org.opencontainers.image.authors=Ceph Release Team , org.opencontainers.image.documentation=https://docs.ceph.com/) 2026-02-23T13:09:14.240 INFO:journalctl@ceph.osd.1.trial068.stdout:Feb 23 13:09:14 trial068 podman[85294]: 2026-02-23 13:09:14.128786964 +0000 UTC m=+0.830538534 container remove d48f950fd6de038c511a3deedcb6f7a719e07784a40652f87f66a27a28c09c0a (image=quay.ceph.io/ceph-ci/ceph@sha256:49621faaa845c39b2aae72e4fa24123ee9b2ac3fb8ae834753907068ee2972ea, name=ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-1-activate, GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfsganesha-5/, CEPH_GIT_REPO=https://github.com/ceph/ceph-ci.git, CEPH_SHA1=e371433fcd1390a0b82cfb18effc422c1f5d8c1d, org.opencontainers.image.authors=Ceph Release Team , ceph=True, io.buildah.version=1.41.3, CEPH_REF=wip-shraddhaag-enable-cephadm-tests, FROM_IMAGE=quay.io/centos/centos:stream9, org.label-schema.license=GPLv2, org.label-schema.build-date=20260216, org.label-schema.name=CentOS Stream 9 Base Image, OSD_FLAVOR=default, org.opencontainers.image.documentation=https://docs.ceph.com/, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2026-02-23T13:09:14.240 INFO:journalctl@ceph.osd.1.trial068.stdout:Feb 23 13:09:14 trial068 podman[85535]: 2026-02-23 13:09:14.207851709 +0000 UTC m=+0.015189257 container create 1db5dbbd33423db9ff24044ed084029c9c425941c10daf030d13b0c68271c85c (image=quay.ceph.io/ceph-ci/ceph@sha256:49621faaa845c39b2aae72e4fa24123ee9b2ac3fb8ae834753907068ee2972ea, name=ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-1, io.buildah.version=1.41.3, org.label-schema.build-date=20260216, GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfsganesha-5/, CEPH_GIT_REPO=https://github.com/ceph/ceph-ci.git, FROM_IMAGE=quay.io/centos/centos:stream9, CEPH_REF=wip-shraddhaag-enable-cephadm-tests, org.opencontainers.image.documentation=https://docs.ceph.com/, OSD_FLAVOR=default, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, CEPH_SHA1=e371433fcd1390a0b82cfb18effc422c1f5d8c1d, org.opencontainers.image.authors=Ceph Release Team , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image) 2026-02-23T13:09:14.247 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph daemon osd.1 dump_ops_in_flight 2026-02-23T13:09:14.331 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 102 88 348300 348212 26.6683 0.0227273 12.7711 0.0338917 2026-02-23T13:09:14.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:09:13 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:09:13.972Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:09:14.371 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:09:14.489 INFO:teuthology.orchestra.run.trial068.stderr:no valid command found; 10 closest matches: 2026-02-23T13:09:14.489 INFO:teuthology.orchestra.run.trial068.stderr:0 2026-02-23T13:09:14.489 INFO:teuthology.orchestra.run.trial068.stderr:1 2026-02-23T13:09:14.489 INFO:teuthology.orchestra.run.trial068.stderr:2 2026-02-23T13:09:14.489 INFO:teuthology.orchestra.run.trial068.stderr:abort 2026-02-23T13:09:14.489 INFO:teuthology.orchestra.run.trial068.stderr:assert 2026-02-23T13:09:14.490 INFO:teuthology.orchestra.run.trial068.stderr:bluestore collections 2026-02-23T13:09:14.490 INFO:teuthology.orchestra.run.trial068.stderr:bluestore compression stats [] 2026-02-23T13:09:14.490 INFO:teuthology.orchestra.run.trial068.stderr:bluestore list [] [] 2026-02-23T13:09:14.490 INFO:teuthology.orchestra.run.trial068.stderr:bluestore onode metadata 2026-02-23T13:09:14.490 INFO:teuthology.orchestra.run.trial068.stderr:bluestore show sharding 2026-02-23T13:09:14.490 INFO:teuthology.orchestra.run.trial068.stderr:admin_socket: invalid command 2026-02-23T13:09:14.490 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:09:14 trial068 ceph-mon[38805]: Health check update: 1 osds down (OSD_DOWN) 2026-02-23T13:09:14.491 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:09:14 trial068 ceph-mon[38805]: osd.3 [v2:10.20.193.185:6808/3352069840,v1:10.20.193.185:6809/3352069840] boot 2026-02-23T13:09:14.491 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:09:14 trial068 ceph-mon[38805]: osdmap e59: 4 total, 3 up, 4 in 2026-02-23T13:09:14.491 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:09:14 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2026-02-23T13:09:14.491 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:09:14 trial068 ceph-mon[38805]: osdmap e60: 4 total, 3 up, 4 in 2026-02-23T13:09:14.491 INFO:journalctl@ceph.osd.1.trial068.stdout:Feb 23 13:09:14 trial068 podman[85535]: 2026-02-23 13:09:14.237961465 +0000 UTC m=+0.045299023 container init 1db5dbbd33423db9ff24044ed084029c9c425941c10daf030d13b0c68271c85c (image=quay.ceph.io/ceph-ci/ceph@sha256:49621faaa845c39b2aae72e4fa24123ee9b2ac3fb8ae834753907068ee2972ea, name=ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-1, OSD_FLAVOR=default, org.opencontainers.image.authors=Ceph Release Team , CEPH_SHA1=e371433fcd1390a0b82cfb18effc422c1f5d8c1d, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.41.3, org.label-schema.license=GPLv2, CEPH_GIT_REPO=https://github.com/ceph/ceph-ci.git, FROM_IMAGE=quay.io/centos/centos:stream9, org.label-schema.build-date=20260216, CEPH_REF=wip-shraddhaag-enable-cephadm-tests, org.opencontainers.image.documentation=https://docs.ceph.com/, GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfsganesha-5/, ceph=True, org.label-schema.vendor=CentOS) 2026-02-23T13:09:14.491 INFO:journalctl@ceph.osd.1.trial068.stdout:Feb 23 13:09:14 trial068 podman[85535]: 2026-02-23 13:09:14.240220177 +0000 UTC m=+0.047557745 container start 1db5dbbd33423db9ff24044ed084029c9c425941c10daf030d13b0c68271c85c (image=quay.ceph.io/ceph-ci/ceph@sha256:49621faaa845c39b2aae72e4fa24123ee9b2ac3fb8ae834753907068ee2972ea, name=ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-1, GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfsganesha-5/, org.label-schema.license=GPLv2, CEPH_GIT_REPO=https://github.com/ceph/ceph-ci.git, ceph=True, CEPH_SHA1=e371433fcd1390a0b82cfb18effc422c1f5d8c1d, org.label-schema.vendor=CentOS, OSD_FLAVOR=default, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.41.3, org.opencontainers.image.documentation=https://docs.ceph.com/, org.opencontainers.image.authors=Ceph Release Team , FROM_IMAGE=quay.io/centos/centos:stream9, org.label-schema.schema-version=1.0, CEPH_REF=wip-shraddhaag-enable-cephadm-tests, org.label-schema.build-date=20260216) 2026-02-23T13:09:14.491 INFO:journalctl@ceph.osd.1.trial068.stdout:Feb 23 13:09:14 trial068 bash[85535]: 1db5dbbd33423db9ff24044ed084029c9c425941c10daf030d13b0c68271c85c 2026-02-23T13:09:14.491 INFO:journalctl@ceph.osd.1.trial068.stdout:Feb 23 13:09:14 trial068 podman[85535]: 2026-02-23 13:09:14.201866278 +0000 UTC m=+0.009203836 image pull 63df4b83db6f95a8ba965919b3671cb27173117c23b5c950aaadc8913f56213c quay.ceph.io/ceph-ci/ceph@sha256:49621faaa845c39b2aae72e4fa24123ee9b2ac3fb8ae834753907068ee2972ea 2026-02-23T13:09:14.492 INFO:journalctl@ceph.osd.1.trial068.stdout:Feb 23 13:09:14 trial068 systemd[1]: Started Ceph osd.1 for 5f823084-10b8-11f1-be80-d404e6e7d460. 2026-02-23T13:09:14.492 INFO:journalctl@ceph.osd.1.trial068.stdout:Feb 23 13:09:14 trial068 ceph-osd[85562]: -- 10.20.193.68:0/2931975993 dispatch_throttle_release 194 to dispatch throttler 194/104857600 2026-02-23T13:09:14.492 INFO:journalctl@ceph.osd.1.trial068.stdout:Feb 23 13:09:14 trial068 ceph-osd[85562]: -- 10.20.193.68:0/2931975993 done calling dispatch on 0x4325860 2026-02-23T13:09:14.492 INFO:journalctl@ceph.osd.1.trial068.stdout:Feb 23 13:09:14 trial068 ceph-osd[85562]: WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:09:14.492 INFO:journalctl@ceph.osd.1.trial068.stdout:Feb 23 13:09:14 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-1[85558]: 2026-02-23T13:09:14.253+0000 7f0ceb840780 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:09:14.492 INFO:journalctl@ceph.osd.1.trial068.stdout:Feb 23 13:09:14 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-1[85558]: 2026-02-23T13:09:14.253+0000 7f0ceb840780 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:09:14.492 INFO:journalctl@ceph.osd.1.trial068.stdout:Feb 23 13:09:14 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-1[85558]: 2026-02-23T13:09:14.253+0000 7f0ceb840780 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:09:14.519 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-23T13:09:14.520 INFO:tasks.cephadm.ceph_manager.ceph:waiting on admin_socket for osd-1, ['dump_ops_in_flight'] 2026-02-23T13:09:14.572 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:09:14.691 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:09:14.743 INFO:journalctl@ceph.osd.1.trial068.stdout:Feb 23 13:09:14 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-1[85558]: 2026-02-23T13:09:14.497+0000 7f0ceb840780 -1 Falling back to public interface 2026-02-23T13:09:14.795 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:09:14.794+0000 7f7e91bde640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:09:14.795 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:09:14.794+0000 7f7e91bde640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:09:14.808 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:09:14.809 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:09:14.809 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.1 2026-02-23T13:09:14.822 INFO:teuthology.orchestra.run.trial068.stdout:osd.2: {} 2026-02-23T13:09:14.822 INFO:teuthology.orchestra.run.trial068.stderr:osd.2: osd_enable_op_tracker = '' 2026-02-23T13:09:14.833 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:09:14.834 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_delete_sleep = '' osd_delete_sleep_hdd = '' osd_delete_sleep_hybrid = '' osd_delete_sleep_ssd = '' osd_enable_op_tracker = '' osd_recovery_max_active = '' osd_recovery_max_active_hdd = '' osd_recovery_max_active_ssd = '' osd_recovery_sleep = '' osd_recovery_sleep_degraded = '' osd_recovery_sleep_degraded_hdd = '' osd_recovery_sleep_degraded_hybrid = '' osd_recovery_sleep_degraded_ssd = '' osd_recovery_sleep_hdd = '' osd_recovery_sleep_hybrid = '' osd_recovery_sleep_ssd = '' osd_scrub_sleep = '' osd_snap_trim_sleep = '' osd_snap_trim_sleep_hdd = '' osd_snap_trim_sleep_hybrid = '' osd_snap_trim_sleep_ssd = '' 2026-02-23T13:09:14.864 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:09:14.864 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:09:15.331 INFO:tasks.radosbench.radosbench.0.trial068.stdout:Total time run: 102.749 2026-02-23T13:09:15.331 INFO:tasks.radosbench.radosbench.0.trial068.stdout:Total writes made: 348300 2026-02-23T13:09:15.331 INFO:tasks.radosbench.radosbench.0.trial068.stdout:Write size: 8192 2026-02-23T13:09:15.331 INFO:tasks.radosbench.radosbench.0.trial068.stdout:Object size: 65536 2026-02-23T13:09:15.331 INFO:tasks.radosbench.radosbench.0.trial068.stdout:Bandwidth (MB/sec): 26.4829 2026-02-23T13:09:15.331 INFO:tasks.radosbench.radosbench.0.trial068.stdout:Stddev Bandwidth: 18.4349 2026-02-23T13:09:15.332 INFO:tasks.radosbench.radosbench.0.trial068.stdout:Max bandwidth (MB/sec): 70.375 2026-02-23T13:09:15.332 INFO:tasks.radosbench.radosbench.0.trial068.stdout:Min bandwidth (MB/sec): 0 2026-02-23T13:09:15.332 INFO:tasks.radosbench.radosbench.0.trial068.stdout:Average IOPS: 3389 2026-02-23T13:09:15.332 INFO:tasks.radosbench.radosbench.0.trial068.stdout:Stddev IOPS: 2359.69 2026-02-23T13:09:15.332 INFO:tasks.radosbench.radosbench.0.trial068.stdout:Max IOPS: 9008 2026-02-23T13:09:15.332 INFO:tasks.radosbench.radosbench.0.trial068.stdout:Min IOPS: 0 2026-02-23T13:09:15.332 INFO:tasks.radosbench.radosbench.0.trial068.stdout:Average Latency(s): 0.0373605 2026-02-23T13:09:15.332 INFO:tasks.radosbench.radosbench.0.trial068.stdout:Stddev Latency(s): 0.264972 2026-02-23T13:09:15.332 INFO:tasks.radosbench.radosbench.0.trial068.stdout:Max latency(s): 13.8833 2026-02-23T13:09:15.332 INFO:tasks.radosbench.radosbench.0.trial068.stdout:Min latency(s): 0.000727387 2026-02-23T13:09:15.336 INFO:tasks.radosbench.radosbench.0.trial068.stdout:Cleaning up (deleting benchmark objects) 2026-02-23T13:09:15.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:09:15 trial068 ceph-mon[38805]: pgmap v90: 17 pgs: 10 undersized+degraded+peered, 7 active+undersized+degraded; 2.7 GiB data, 8.6 GiB used, 2.7 TiB / 2.7 TiB avail; 70683/130581 objects degraded (54.130%) 2026-02-23T13:09:15.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:09:15 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3617655671' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:09:15.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:09:15 trial068 ceph-mon[38805]: osdmap e61: 4 total, 3 up, 4 in 2026-02-23T13:09:16.593 INFO:journalctl@ceph.osd.1.trial068.stdout:Feb 23 13:09:16 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-1[85558]: 2026-02-23T13:09:16.276+0000 7f0ceb840780 -1 osd.1 55 log_to_monitors true 2026-02-23T13:09:16.866 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:09:16.985 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:09:17.099 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:09:17.098+0000 7f2198cef640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:09:17.100 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:09:17.098+0000 7f2198cef640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:09:17.113 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:09:17.113 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:09:17.113 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.1 2026-02-23T13:09:17.125 INFO:teuthology.orchestra.run.trial068.stdout:osd.2: {} 2026-02-23T13:09:17.125 INFO:teuthology.orchestra.run.trial068.stderr:osd.2: osd_enable_op_tracker = '' 2026-02-23T13:09:17.136 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:09:17.137 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:09:17.146 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:09:16 trial068 ceph-mon[38805]: osdmap e62: 4 total, 3 up, 4 in 2026-02-23T13:09:17.146 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:09:16 trial068 ceph-mon[38805]: pgmap v93: 17 pgs: 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 5 active+undersized+degraded+remapped, 5 active+undersized+degraded; 2.7 GiB data, 8.6 GiB used, 2.7 TiB / 2.7 TiB avail; 223 KiB/s wr, 35 op/s; 32943/130527 objects degraded (25.238%); 12 MiB/s, 189 objects/s recovering 2026-02-23T13:09:17.146 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:09:16 trial068 ceph-mon[38805]: from='osd.1 [v2:10.20.193.68:6810/3895588833,v1:10.20.193.68:6811/3895588833]' entity='osd.1' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]} : dispatch 2026-02-23T13:09:17.146 INFO:journalctl@ceph.osd.1.trial068.stdout:Feb 23 13:09:16 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-1[85558]: 2026-02-23T13:09:16.975+0000 7f0ce3689640 -1 osd.1 55 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2026-02-23T13:09:17.166 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:09:17.166 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:09:18.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:09:17 trial068 ceph-mon[38805]: Health check update: Degraded data redundancy: 32943/130527 objects degraded (25.238%), 17 pgs degraded (PG_DEGRADED) 2026-02-23T13:09:18.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:09:17 trial068 ceph-mon[38805]: from='osd.1 [v2:10.20.193.68:6810/3895588833,v1:10.20.193.68:6811/3895588833]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2026-02-23T13:09:18.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:09:17 trial068 ceph-mon[38805]: osdmap e63: 4 total, 3 up, 4 in 2026-02-23T13:09:18.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:09:17 trial068 ceph-mon[38805]: from='osd.1 [v2:10.20.193.68:6810/3895588833,v1:10.20.193.68:6811/3895588833]' entity='osd.1' cmd={"prefix": "osd crush create-or-move", "id": 1, "weight":0.6836, "args": ["host=trial068", "root=default"]} : dispatch 2026-02-23T13:09:18.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:09:17 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3138893147' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:09:19.168 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:09:19.284 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:09:19.305 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:09:18 trial068 ceph-mon[38805]: Health check cleared: OSD_DOWN (was: 1 osds down) 2026-02-23T13:09:19.306 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:09:18 trial068 ceph-mon[38805]: osd.1 [v2:10.20.193.68:6810/3895588833,v1:10.20.193.68:6811/3895588833] boot 2026-02-23T13:09:19.306 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:09:18 trial068 ceph-mon[38805]: osdmap e64: 4 total, 4 up, 4 in 2026-02-23T13:09:19.306 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:09:18 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2026-02-23T13:09:19.306 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:09:18 trial068 ceph-mon[38805]: pgmap v96: 17 pgs: 4 peering, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+clean, 11 active+undersized+degraded; 2.5 GiB data, 8.6 GiB used, 2.7 TiB / 2.7 TiB avail; 255 B/s rd, 218 KiB/s wr, 532 op/s; 28635/124563 objects degraded (22.988%); 16 MiB/s, 261 objects/s recovering 2026-02-23T13:09:19.306 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:09:18 trial068 ceph-mon[38805]: osdmap e65: 4 total, 4 up, 4 in 2026-02-23T13:09:19.381 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:09:19.379+0000 7f9e2661e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:09:19.381 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:09:19.379+0000 7f9e2661e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:09:19.393 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:09:19.393 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:09:19.403 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:09:19.403 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:09:19.414 INFO:teuthology.orchestra.run.trial068.stdout:osd.2: {} 2026-02-23T13:09:19.414 INFO:teuthology.orchestra.run.trial068.stderr:osd.2: osd_enable_op_tracker = '' 2026-02-23T13:09:19.424 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:09:19.424 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:09:19.521 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph daemon osd.1 dump_ops_in_flight 2026-02-23T13:09:19.639 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:09:19.752 INFO:teuthology.orchestra.run.trial068.stdout:ERROR: (22) Invalid argument 2026-02-23T13:09:19.753 INFO:teuthology.orchestra.run.trial068.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2026-02-23T13:09:19.786 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd reweight 0 1 2026-02-23T13:09:19.899 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:09:20.006 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:09:20.004+0000 7f761db49640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:09:20.006 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:09:20.004+0000 7f761db49640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:09:20.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:09:19 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T13:09:20.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:09:19 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T13:09:20.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:09:19 trial068 ceph-mon[38805]: 2.5 scrub starts 2026-02-23T13:09:20.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:09:19 trial068 ceph-mon[38805]: 2.5 scrub ok 2026-02-23T13:09:20.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:09:19 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1773789205' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:09:20.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:09:19 trial068 ceph-mon[38805]: osdmap e66: 4 total, 4 up, 4 in 2026-02-23T13:09:20.979 INFO:teuthology.orchestra.run.trial068.stderr:reweighted osd.0 to 1 (10000) 2026-02-23T13:09:21.026 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd reweight 1 1 2026-02-23T13:09:21.173 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:09:21.234 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:09:20 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1871451374' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2026-02-23T13:09:21.234 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:09:20 trial068 ceph-mon[38805]: pgmap v99: 17 pgs: 4 peering, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+clean, 11 active+undersized+degraded; 2.5 GiB data, 8.6 GiB used, 2.7 TiB / 2.7 TiB avail; 255 B/s rd, 255 B/s wr, 497 op/s; 28635/124563 objects degraded (22.988%); 4.8 MiB/s, 76 objects/s recovering 2026-02-23T13:09:21.297 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:09:21.296+0000 7f77a5ad5640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:09:21.297 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:09:21.296+0000 7f77a5ad5640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:09:21.460 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:09:21.582 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:09:21.690 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:09:21.688+0000 7f0d01dad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:09:21.691 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:09:21.688+0000 7f0d01dad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:09:21.704 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:09:21.705 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:09:21.715 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:09:21.715 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:09:21.726 INFO:teuthology.orchestra.run.trial068.stdout:osd.2: {} 2026-02-23T13:09:21.726 INFO:teuthology.orchestra.run.trial068.stderr:osd.2: osd_enable_op_tracker = '' 2026-02-23T13:09:21.736 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:09:21.737 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:09:21.984 INFO:teuthology.orchestra.run.trial068.stderr:reweighted osd.1 to 1 (10000) 2026-02-23T13:09:22.028 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd reweight 2 1 2026-02-23T13:09:22.176 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:09:22.293 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:09:21 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1871451374' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2026-02-23T13:09:22.293 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:09:21 trial068 ceph-mon[38805]: osdmap e67: 4 total, 4 up, 4 in 2026-02-23T13:09:22.293 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:09:21 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/923231817' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2026-02-23T13:09:22.294 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:09:21 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2489569453' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:09:22.685 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:09:22.648+0000 7efe77d89640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:09:22.686 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:09:22.648+0000 7efe77d89640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:09:22.985 INFO:teuthology.orchestra.run.trial068.stderr:reweighted osd.2 to 1 (10000) 2026-02-23T13:09:23.037 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd reweight 3 1 2026-02-23T13:09:23.184 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:09:23.237 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:09:22 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/923231817' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2026-02-23T13:09:23.237 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:09:22 trial068 ceph-mon[38805]: osdmap e68: 4 total, 4 up, 4 in 2026-02-23T13:09:23.237 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:09:22 trial068 ceph-mon[38805]: pgmap v102: 17 pgs: 1 active+recovering+undersized+degraded+remapped, 4 peering, 7 active+recovery_wait+undersized+degraded+remapped, 2 active+clean, 3 active+undersized+degraded; 2.4 GiB data, 8.2 GiB used, 2.7 TiB / 2.7 TiB avail; 0 B/s wr, 351 op/s; 12183/120348 objects degraded (10.123%); 2.4 MiB/s, 37 objects/s recovering 2026-02-23T13:09:23.237 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:09:22 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3750110431' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2026-02-23T13:09:23.308 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:09:23.306+0000 7fc0452f9640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:09:23.308 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:09:23.306+0000 7fc0452f9640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:09:23.774 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:09:23.902 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:09:23.969 INFO:teuthology.orchestra.run.trial068.stderr:reweighted osd.3 to 1 (10000) 2026-02-23T13:09:24.021 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2026-02-23T13:09:24.022 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph pg dump --format=json 2026-02-23T13:09:24.026 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:09:24.023+0000 7f30b25a4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:09:24.026 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:09:24.024+0000 7f30b25a4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:09:24.041 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:09:24.041 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:09:24.056 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:09:24.056 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:09:24.057 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:09:23 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:09:23] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:09:24.057 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:09:23 trial068 ceph-mon[38805]: Health check update: Degraded data redundancy: 12183/120348 objects degraded (10.123%), 11 pgs degraded (PG_DEGRADED) 2026-02-23T13:09:24.057 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:09:23 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3750110431' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2026-02-23T13:09:24.057 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:09:23 trial068 ceph-mon[38805]: osdmap e69: 4 total, 4 up, 4 in 2026-02-23T13:09:24.057 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:09:23 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3629793369' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2026-02-23T13:09:24.058 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:09:23 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3629793369' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2026-02-23T13:09:24.058 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:09:23 trial068 ceph-mon[38805]: osdmap e70: 4 total, 4 up, 4 in 2026-02-23T13:09:24.070 INFO:teuthology.orchestra.run.trial068.stdout:osd.2: {} 2026-02-23T13:09:24.071 INFO:teuthology.orchestra.run.trial068.stderr:osd.2: osd_enable_op_tracker = '' 2026-02-23T13:09:24.084 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:09:24.084 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:09:24.160 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:09:24.277 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:09:24.275+0000 7fa7980bf640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:09:24.277 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:09:24.275+0000 7fa7980bf640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:09:24.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:09:23 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:09:23.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:09:24.389 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T13:09:24.390 INFO:teuthology.orchestra.run.trial068.stderr:dumped all 2026-02-23T13:09:24.432 INFO:teuthology.orchestra.run.trial068.stdout:{"pg_ready":true,"pg_map":{"version":105,"stamp":"2026-02-23T13:09:24.241907+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2481291832,"num_objects":37856,"num_object_clones":0,"num_object_copies":113568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5093,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37856,"num_whiteouts":0,"num_read":123,"num_read_kb":229,"num_write":354121,"num_write_kb":2788551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13505,"num_bytes_recovered":860045856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160885,"ondisk_log_size":160885,"up":51,"acting":41,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":51,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":2936012800,"kb_used":8324972,"kb_used_data":7312732,"kb_used_omap":91,"kb_used_meta":1012068,"kb_avail":2927687828,"statfs":{"total":3006477107200,"available":2997952335872,"internally_reserved":0,"allocated":7488237568,"data_stored":7486459300,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":93471,"internal_metadata":1036358369},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,3,52],"upper_bound":4096},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-240189440,"num_objects":-3665,"num_object_clones":0,"num_object_copies":-10995,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-23542,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3665,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":518,"num_bytes_recovered":9928704,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000657"},"pg_stats":[{"pgid":"2.f","version":"66'22517","reported_seq":24085,"reported_epoch":66,"state":"active+clean","last_fresh":"2026-02-23T13:09:20.371853+0000","last_change":"2026-02-23T13:09:17.988736+0000","last_active":"2026-02-23T13:09:20.371853+0000","last_peered":"2026-02-23T13:09:20.371853+0000","last_clean":"2026-02-23T13:09:20.371853+0000","last_became_active":"2026-02-23T13:09:17.980759+0000","last_became_peered":"2026-02-23T13:09:17.980759+0000","last_unstale":"2026-02-23T13:09:20.371853+0000","last_undegraded":"2026-02-23T13:09:20.371853+0000","last_fullsized":"2026-02-23T13:09:20.371853+0000","mapping_epoch":63,"log_start":"33'12500","ondisk_log_start":"33'12500","created":29,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"48'17576","last_scrub_stamp":"2026-02-23T13:08:38.307705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_clean_scrub_stamp":"2026-02-23T13:08:38.307705+0000","objects_scrubbed":2192,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T13:10:04.053022+0000","scrub_duration":368,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":165216256,"num_objects":2521,"num_object_clones":0,"num_object_copies":7563,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2521,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22517,"num_write_kb":178048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":710,"num_bytes_recovered":46530560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.e","version":"66'21681","reported_seq":21780,"reported_epoch":66,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2026-02-23T13:09:20.338985+0000","last_change":"2026-02-23T13:09:20.057612+0000","last_active":"2026-02-23T13:09:20.338985+0000","last_peered":"2026-02-23T13:09:20.338985+0000","last_clean":"2026-02-23T13:09:01.297181+0000","last_became_active":"2026-02-23T13:09:20.050781+0000","last_became_peered":"2026-02-23T13:09:20.050781+0000","last_unstale":"2026-02-23T13:09:20.338985+0000","last_undegraded":"2026-02-23T13:09:19.980426+0000","last_fullsized":"2026-02-23T13:09:19.970754+0000","mapping_epoch":65,"log_start":"33'11600","ondisk_log_start":"33'11600","created":29,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_clean_scrub_stamp":"2026-02-23T13:07:30.224555+0000","objects_scrubbed":0,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":159842304,"num_objects":2439,"num_object_clones":0,"num_object_copies":7317,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":168,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2439,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21681,"num_write_kb":171520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":577,"num_bytes_recovered":37814272,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,2","objects":2439}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"66'21690","reported_seq":23197,"reported_epoch":66,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2026-02-23T13:09:20.348302+0000","last_change":"2026-02-23T13:09:20.058876+0000","last_active":"2026-02-23T13:09:20.348302+0000","last_peered":"2026-02-23T13:09:20.348302+0000","last_clean":"2026-02-23T13:08:31.228846+0000","last_became_active":"2026-02-23T13:09:20.049449+0000","last_became_peered":"2026-02-23T13:09:20.049449+0000","last_unstale":"2026-02-23T13:09:20.348302+0000","last_undegraded":"2026-02-23T13:09:19.980906+0000","last_fullsized":"2026-02-23T13:09:19.970900+0000","mapping_epoch":65,"log_start":"33'11600","ondisk_log_start":"33'11600","created":29,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_clean_scrub_stamp":"2026-02-23T13:07:30.224555+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":162267160,"num_objects":2477,"num_object_clones":0,"num_object_copies":7431,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":721,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2477,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":21690,"num_write_kb":171841,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":700,"num_bytes_recovered":45711360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,3","objects":1913}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.c","version":"66'22355","reported_seq":23882,"reported_epoch":66,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2026-02-23T13:09:20.346840+0000","last_change":"2026-02-23T13:09:20.058916+0000","last_active":"2026-02-23T13:09:20.346840+0000","last_peered":"2026-02-23T13:09:20.346840+0000","last_clean":"2026-02-23T13:08:31.176488+0000","last_became_active":"2026-02-23T13:09:20.052166+0000","last_became_peered":"2026-02-23T13:09:20.052166+0000","last_unstale":"2026-02-23T13:09:20.346840+0000","last_undegraded":"2026-02-23T13:09:19.983883+0000","last_fullsized":"2026-02-23T13:09:19.980353+0000","mapping_epoch":65,"log_start":"33'12300","ondisk_log_start":"33'12300","created":29,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_clean_scrub_stamp":"2026-02-23T13:07:30.224555+0000","objects_scrubbed":0,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":165216256,"num_objects":2521,"num_object_clones":0,"num_object_copies":7563,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":800,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2521,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22355,"num_write_kb":176896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":710,"num_bytes_recovered":46317568,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,3","objects":1926}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"28'99","reported_seq":114,"reported_epoch":68,"state":"active+clean","last_fresh":"2026-02-23T13:09:21.984434+0000","last_change":"2026-02-23T13:09:13.973193+0000","last_active":"2026-02-23T13:09:21.984434+0000","last_peered":"2026-02-23T13:09:21.984434+0000","last_clean":"2026-02-23T13:09:21.984434+0000","last_became_active":"2026-02-23T13:09:13.972259+0000","last_became_peered":"2026-02-23T13:09:13.972259+0000","last_unstale":"2026-02-23T13:09:21.984434+0000","last_undegraded":"2026-02-23T13:09:21.984434+0000","last_fullsized":"2026-02-23T13:09:21.984434+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'99","last_scrub_stamp":"2026-02-23T13:08:29.915607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:07:05.619208+0000","last_clean_scrub_stamp":"2026-02-23T13:08:29.915607+0000","objects_scrubbed":0,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T13:09:45.841302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":122,"num_read_kb":228,"num_write":135,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"68'22026","reported_seq":24924,"reported_epoch":68,"state":"active+clean","last_fresh":"2026-02-23T13:09:22.343263+0000","last_change":"2026-02-23T13:09:21.988904+0000","last_active":"2026-02-23T13:09:22.343263+0000","last_peered":"2026-02-23T13:09:22.343263+0000","last_clean":"2026-02-23T13:09:22.343263+0000","last_became_active":"2026-02-23T13:09:21.985593+0000","last_became_peered":"2026-02-23T13:09:21.985593+0000","last_unstale":"2026-02-23T13:09:22.343263+0000","last_undegraded":"2026-02-23T13:09:22.343263+0000","last_fullsized":"2026-02-23T13:09:22.343263+0000","mapping_epoch":67,"log_start":"33'12000","ondisk_log_start":"33'12000","created":29,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_clean_scrub_stamp":"2026-02-23T13:07:30.224555+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":143130624,"num_objects":2184,"num_object_clones":0,"num_object_copies":6552,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2184,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22026,"num_write_kb":172160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1389,"num_bytes_recovered":79233024,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"68'22004","reported_seq":24628,"reported_epoch":68,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2026-02-23T13:09:22.327642+0000","last_change":"2026-02-23T13:09:20.054037+0000","last_active":"2026-02-23T13:09:22.327642+0000","last_peered":"2026-02-23T13:09:22.327642+0000","last_clean":"2026-02-23T13:08:31.237071+0000","last_became_active":"2026-02-23T13:09:20.046498+0000","last_became_peered":"2026-02-23T13:09:20.046498+0000","last_unstale":"2026-02-23T13:09:22.327642+0000","last_undegraded":"2026-02-23T13:09:19.980807+0000","last_fullsized":"2026-02-23T13:09:19.970180+0000","mapping_epoch":65,"log_start":"33'12000","ondisk_log_start":"33'12000","created":29,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_clean_scrub_stamp":"2026-02-23T13:07:30.224555+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":143982592,"num_objects":2197,"num_object_clones":0,"num_object_copies":6591,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":190,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2197,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22004,"num_write_kb":172096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1259,"num_bytes_recovered":82378752,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[{"shards":"0,3","objects":2195}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"66'21867","reported_seq":23411,"reported_epoch":66,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2026-02-23T13:09:20.345641+0000","last_change":"2026-02-23T13:09:20.059842+0000","last_active":"2026-02-23T13:09:20.345641+0000","last_peered":"2026-02-23T13:09:20.345641+0000","last_clean":"2026-02-23T13:08:31.247020+0000","last_became_active":"2026-02-23T13:09:20.051670+0000","last_became_peered":"2026-02-23T13:09:20.051670+0000","last_unstale":"2026-02-23T13:09:20.345641+0000","last_undegraded":"2026-02-23T13:09:19.981209+0000","last_fullsized":"2026-02-23T13:09:19.970907+0000","mapping_epoch":65,"log_start":"33'11800","ondisk_log_start":"33'11800","created":29,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_clean_scrub_stamp":"2026-02-23T13:07:30.224555+0000","objects_scrubbed":0,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":162398208,"num_objects":2478,"num_object_clones":0,"num_object_copies":7434,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":779,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2478,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21867,"num_write_kb":173120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":717,"num_bytes_recovered":46931968,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,3","objects":1892}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"68'22174","reported_seq":24796,"reported_epoch":68,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2026-02-23T13:09:22.345946+0000","last_change":"2026-02-23T13:09:20.055811+0000","last_active":"2026-02-23T13:09:22.345946+0000","last_peered":"2026-02-23T13:09:22.345946+0000","last_clean":"2026-02-23T13:08:31.252741+0000","last_became_active":"2026-02-23T13:09:20.050327+0000","last_became_peered":"2026-02-23T13:09:20.050327+0000","last_unstale":"2026-02-23T13:09:22.345946+0000","last_undegraded":"2026-02-23T13:09:19.980985+0000","last_fullsized":"2026-02-23T13:09:19.970054+0000","mapping_epoch":65,"log_start":"33'12100","ondisk_log_start":"33'12100","created":29,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_clean_scrub_stamp":"2026-02-23T13:07:30.224555+0000","objects_scrubbed":0,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":144048128,"num_objects":2198,"num_object_clones":0,"num_object_copies":6594,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":203,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2198,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22174,"num_write_kb":173312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1258,"num_bytes_recovered":82255872,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":2197}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"66'21810","reported_seq":22364,"reported_epoch":66,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2026-02-23T13:09:20.330262+0000","last_change":"2026-02-23T13:09:20.058899+0000","last_active":"2026-02-23T13:09:20.330262+0000","last_peered":"2026-02-23T13:09:20.330262+0000","last_clean":"2026-02-23T13:09:01.266173+0000","last_became_active":"2026-02-23T13:09:20.051618+0000","last_became_peered":"2026-02-23T13:09:20.051618+0000","last_unstale":"2026-02-23T13:09:20.330262+0000","last_undegraded":"2026-02-23T13:09:19.980199+0000","last_fullsized":"2026-02-23T13:09:19.970748+0000","mapping_epoch":65,"log_start":"33'11800","ondisk_log_start":"33'11800","created":29,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_clean_scrub_stamp":"2026-02-23T13:07:30.224555+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":162004992,"num_objects":2472,"num_object_clones":0,"num_object_copies":7416,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":167,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2472,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21810,"num_write_kb":172672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":543,"num_bytes_recovered":35471360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,2","objects":2471}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"68'22350","reported_seq":23890,"reported_epoch":68,"state":"active+clean","last_fresh":"2026-02-23T13:09:22.339386+0000","last_change":"2026-02-23T13:09:18.710906+0000","last_active":"2026-02-23T13:09:22.339386+0000","last_peered":"2026-02-23T13:09:22.339386+0000","last_clean":"2026-02-23T13:09:22.339386+0000","last_became_active":"2026-02-23T13:09:17.980954+0000","last_became_peered":"2026-02-23T13:09:17.980954+0000","last_unstale":"2026-02-23T13:09:22.339386+0000","last_undegraded":"2026-02-23T13:09:22.339386+0000","last_fullsized":"2026-02-23T13:09:22.339386+0000","mapping_epoch":63,"log_start":"33'12300","ondisk_log_start":"33'12300","created":29,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'22042","last_scrub_stamp":"2026-02-23T13:09:18.710867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_clean_scrub_stamp":"2026-02-23T13:09:18.710867+0000","objects_scrubbed":2519,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T13:10:31.775316+0000","scrub_duration":127,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":144900096,"num_objects":2211,"num_object_clones":0,"num_object_copies":6633,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2211,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22350,"num_write_kb":174656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":707,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"66'21867","reported_seq":23473,"reported_epoch":66,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2026-02-23T13:09:20.378847+0000","last_change":"2026-02-23T13:09:20.058603+0000","last_active":"2026-02-23T13:09:20.378847+0000","last_peered":"2026-02-23T13:09:20.378847+0000","last_clean":"2026-02-23T13:08:31.205210+0000","last_became_active":"2026-02-23T13:09:20.050775+0000","last_became_peered":"2026-02-23T13:09:20.050775+0000","last_unstale":"2026-02-23T13:09:20.378847+0000","last_undegraded":"2026-02-23T13:09:19.978113+0000","last_fullsized":"2026-02-23T13:09:19.969317+0000","mapping_epoch":65,"log_start":"33'11800","ondisk_log_start":"33'11800","created":29,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_clean_scrub_stamp":"2026-02-23T13:07:30.224555+0000","objects_scrubbed":0,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":160628736,"num_objects":2451,"num_object_clones":0,"num_object_copies":7353,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":689,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2451,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21867,"num_write_kb":172928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":744,"num_bytes_recovered":48734208,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[2,3],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":1946}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"68'21762","reported_seq":24718,"reported_epoch":68,"state":"active+clean","last_fresh":"2026-02-23T13:09:22.336115+0000","last_change":"2026-02-23T13:09:22.090938+0000","last_active":"2026-02-23T13:09:22.336115+0000","last_peered":"2026-02-23T13:09:22.336115+0000","last_clean":"2026-02-23T13:09:22.336115+0000","last_became_active":"2026-02-23T13:09:21.996628+0000","last_became_peered":"2026-02-23T13:09:21.996628+0000","last_unstale":"2026-02-23T13:09:22.336115+0000","last_undegraded":"2026-02-23T13:09:22.336115+0000","last_fullsized":"2026-02-23T13:09:22.336115+0000","mapping_epoch":67,"log_start":"33'11700","ondisk_log_start":"33'11700","created":29,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_clean_scrub_stamp":"2026-02-23T13:07:30.224555+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":143917056,"num_objects":2196,"num_object_clones":0,"num_object_copies":6588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2196,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21762,"num_write_kb":170368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1419,"num_bytes_recovered":80683008,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.8","version":"68'22361","reported_seq":23446,"reported_epoch":68,"state":"active+clean","last_fresh":"2026-02-23T13:09:22.339495+0000","last_change":"2026-02-23T13:09:17.986383+0000","last_active":"2026-02-23T13:09:22.339495+0000","last_peered":"2026-02-23T13:09:22.339495+0000","last_clean":"2026-02-23T13:09:22.339495+0000","last_became_active":"2026-02-23T13:09:17.980940+0000","last_became_peered":"2026-02-23T13:09:17.980940+0000","last_unstale":"2026-02-23T13:09:22.339495+0000","last_undegraded":"2026-02-23T13:09:22.339495+0000","last_fullsized":"2026-02-23T13:09:22.339495+0000","mapping_epoch":63,"log_start":"33'12300","ondisk_log_start":"33'12300","created":29,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"49'20400","last_scrub_stamp":"2026-02-23T13:08:49.984469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_clean_scrub_stamp":"2026-02-23T13:08:49.984469+0000","objects_scrubbed":2548,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T13:09:50.946231+0000","scrub_duration":928,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":144769024,"num_objects":2209,"num_object_clones":0,"num_object_copies":6627,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2209,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22361,"num_write_kb":174720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":664,"num_bytes_recovered":43491328,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"66'22915","reported_seq":24409,"reported_epoch":66,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2026-02-23T13:09:20.342277+0000","last_change":"2026-02-23T13:09:20.059873+0000","last_active":"2026-02-23T13:09:20.342277+0000","last_peered":"2026-02-23T13:09:20.342277+0000","last_clean":"2026-02-23T13:08:31.257695+0000","last_became_active":"2026-02-23T13:09:20.051907+0000","last_became_peered":"2026-02-23T13:09:20.051907+0000","last_unstale":"2026-02-23T13:09:20.342277+0000","last_undegraded":"2026-02-23T13:09:19.980849+0000","last_fullsized":"2026-02-23T13:09:19.970886+0000","mapping_epoch":65,"log_start":"33'12900","ondisk_log_start":"33'12900","created":29,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_clean_scrub_stamp":"2026-02-23T13:07:30.224555+0000","objects_scrubbed":0,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":170098688,"num_objects":2596,"num_object_clones":0,"num_object_copies":7788,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":826,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2596,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22915,"num_write_kb":181408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":691,"num_bytes_recovered":45056000,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,3","objects":1959}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"66'22095","reported_seq":23620,"reported_epoch":66,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2026-02-23T13:09:20.378266+0000","last_change":"2026-02-23T13:09:20.074080+0000","last_active":"2026-02-23T13:09:20.378266+0000","last_peered":"2026-02-23T13:09:20.378266+0000","last_clean":"2026-02-23T13:08:31.221592+0000","last_became_active":"2026-02-23T13:09:20.050909+0000","last_became_peered":"2026-02-23T13:09:20.050909+0000","last_unstale":"2026-02-23T13:09:20.378266+0000","last_undegraded":"2026-02-23T13:09:19.978399+0000","last_fullsized":"2026-02-23T13:09:19.968983+0000","mapping_epoch":65,"log_start":"33'12000","ondisk_log_start":"33'12000","created":29,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_clean_scrub_stamp":"2026-02-23T13:07:30.224555+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":161611776,"num_objects":2466,"num_object_clones":0,"num_object_copies":7398,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":550,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2466,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22095,"num_write_kb":174656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":705,"num_bytes_recovered":46039040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[2,3],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":1916}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"68'22512","reported_seq":23644,"reported_epoch":68,"state":"active+clean","last_fresh":"2026-02-23T13:09:22.325793+0000","last_change":"2026-02-23T13:09:17.986451+0000","last_active":"2026-02-23T13:09:22.325793+0000","last_peered":"2026-02-23T13:09:22.325793+0000","last_clean":"2026-02-23T13:09:22.325793+0000","last_became_active":"2026-02-23T13:09:17.981144+0000","last_became_peered":"2026-02-23T13:09:17.981144+0000","last_unstale":"2026-02-23T13:09:22.325793+0000","last_undegraded":"2026-02-23T13:09:22.325793+0000","last_fullsized":"2026-02-23T13:09:22.325793+0000","mapping_epoch":63,"log_start":"33'12500","ondisk_log_start":"33'12500","created":29,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"48'18824","last_scrub_stamp":"2026-02-23T13:08:42.411326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_clean_scrub_stamp":"2026-02-23T13:08:42.411326+0000","objects_scrubbed":2347,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T13:10:03.835120+0000","scrub_duration":1426,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":146669568,"num_objects":2238,"num_object_clones":0,"num_object_copies":6714,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2238,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22512,"num_write_kb":176000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":710,"num_bytes_recovered":46473216,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":2480701464,"num_objects":37854,"num_object_clones":0,"num_object_copies":113562,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5093,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37854,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":353986,"num_write_kb":2786401,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13503,"num_bytes_recovered":859455488,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":7380680704,"data_stored":7380672560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160786,"ondisk_log_size":160786,"up":48,"acting":38,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":122,"num_read_kb":228,"num_write":135,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":59,"seq":253403070467,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":734003200,"kb_used":2120568,"kb_used_data":2060196,"kb_used_omap":12,"kb_used_meta":60339,"kb_avail":731882632,"statfs":{"total":751619276800,"available":749447815168,"internally_reserved":0,"allocated":2109640704,"data_stored":2109185686,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12568,"internal_metadata":61787880},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,2],"upper_bound":2048},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411358,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":734003200,"kb_used":2238716,"kb_used_data":1778600,"kb_used_omap":34,"kb_used_meta":460061,"kb_avail":731764484,"statfs":{"total":751619276800,"available":749326831616,"internally_reserved":0,"allocated":1821286400,"data_stored":1820850508,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":35477,"internal_metadata":471102827},"hb_peers":[0,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,1],"upper_bound":2048},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":64,"seq":274877906946,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":734003200,"kb_used":1731928,"kb_used_data":1682436,"kb_used_omap":9,"kb_used_meta":49462,"kb_avail":732271272,"statfs":{"total":751619276800,"available":749845782528,"internally_reserved":0,"allocated":1722814464,"data_stored":1722367070,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9949,"internal_metadata":50649379},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705697,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":734003200,"kb_used":2233760,"kb_used_data":1791500,"kb_used_omap":34,"kb_used_meta":442205,"kb_avail":731769440,"statfs":{"total":751619276800,"available":749331906560,"internally_reserved":0,"allocated":1834496000,"data_stored":1834056036,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":35477,"internal_metadata":452818283},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,52],"upper_bound":4096},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1793363968,"data_stored":1793359896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1770397696,"data_stored":1770397696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1770323968,"data_stored":1770323968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2046595072,"data_stored":2046591000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-23T13:09:24.434 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph pg dump --format=json 2026-02-23T13:09:24.598 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:09:24.708 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:09:24.707+0000 7fe858b67640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:09:24.709 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:09:24.707+0000 7fe858b67640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:09:24.827 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T13:09:24.828 INFO:teuthology.orchestra.run.trial068.stderr:dumped all 2026-02-23T13:09:24.868 INFO:teuthology.orchestra.run.trial068.stdout:{"pg_ready":true,"pg_map":{"version":105,"stamp":"2026-02-23T13:09:24.241907+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2481291832,"num_objects":37856,"num_object_clones":0,"num_object_copies":113568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5093,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37856,"num_whiteouts":0,"num_read":123,"num_read_kb":229,"num_write":354121,"num_write_kb":2788551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13505,"num_bytes_recovered":860045856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160885,"ondisk_log_size":160885,"up":51,"acting":41,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":51,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":2936012800,"kb_used":8324972,"kb_used_data":7312732,"kb_used_omap":91,"kb_used_meta":1012068,"kb_avail":2927687828,"statfs":{"total":3006477107200,"available":2997952335872,"internally_reserved":0,"allocated":7488237568,"data_stored":7486459300,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":93471,"internal_metadata":1036358369},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,3,52],"upper_bound":4096},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-240189440,"num_objects":-3665,"num_object_clones":0,"num_object_copies":-10995,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-23542,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3665,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":518,"num_bytes_recovered":9928704,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000657"},"pg_stats":[{"pgid":"2.f","version":"66'22517","reported_seq":24085,"reported_epoch":66,"state":"active+clean","last_fresh":"2026-02-23T13:09:20.371853+0000","last_change":"2026-02-23T13:09:17.988736+0000","last_active":"2026-02-23T13:09:20.371853+0000","last_peered":"2026-02-23T13:09:20.371853+0000","last_clean":"2026-02-23T13:09:20.371853+0000","last_became_active":"2026-02-23T13:09:17.980759+0000","last_became_peered":"2026-02-23T13:09:17.980759+0000","last_unstale":"2026-02-23T13:09:20.371853+0000","last_undegraded":"2026-02-23T13:09:20.371853+0000","last_fullsized":"2026-02-23T13:09:20.371853+0000","mapping_epoch":63,"log_start":"33'12500","ondisk_log_start":"33'12500","created":29,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"48'17576","last_scrub_stamp":"2026-02-23T13:08:38.307705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_clean_scrub_stamp":"2026-02-23T13:08:38.307705+0000","objects_scrubbed":2192,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T13:10:04.053022+0000","scrub_duration":368,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":165216256,"num_objects":2521,"num_object_clones":0,"num_object_copies":7563,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2521,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22517,"num_write_kb":178048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":710,"num_bytes_recovered":46530560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.e","version":"66'21681","reported_seq":21780,"reported_epoch":66,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2026-02-23T13:09:20.338985+0000","last_change":"2026-02-23T13:09:20.057612+0000","last_active":"2026-02-23T13:09:20.338985+0000","last_peered":"2026-02-23T13:09:20.338985+0000","last_clean":"2026-02-23T13:09:01.297181+0000","last_became_active":"2026-02-23T13:09:20.050781+0000","last_became_peered":"2026-02-23T13:09:20.050781+0000","last_unstale":"2026-02-23T13:09:20.338985+0000","last_undegraded":"2026-02-23T13:09:19.980426+0000","last_fullsized":"2026-02-23T13:09:19.970754+0000","mapping_epoch":65,"log_start":"33'11600","ondisk_log_start":"33'11600","created":29,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_clean_scrub_stamp":"2026-02-23T13:07:30.224555+0000","objects_scrubbed":0,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":159842304,"num_objects":2439,"num_object_clones":0,"num_object_copies":7317,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":168,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2439,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21681,"num_write_kb":171520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":577,"num_bytes_recovered":37814272,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,2","objects":2439}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"66'21690","reported_seq":23197,"reported_epoch":66,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2026-02-23T13:09:20.348302+0000","last_change":"2026-02-23T13:09:20.058876+0000","last_active":"2026-02-23T13:09:20.348302+0000","last_peered":"2026-02-23T13:09:20.348302+0000","last_clean":"2026-02-23T13:08:31.228846+0000","last_became_active":"2026-02-23T13:09:20.049449+0000","last_became_peered":"2026-02-23T13:09:20.049449+0000","last_unstale":"2026-02-23T13:09:20.348302+0000","last_undegraded":"2026-02-23T13:09:19.980906+0000","last_fullsized":"2026-02-23T13:09:19.970900+0000","mapping_epoch":65,"log_start":"33'11600","ondisk_log_start":"33'11600","created":29,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_clean_scrub_stamp":"2026-02-23T13:07:30.224555+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":162267160,"num_objects":2477,"num_object_clones":0,"num_object_copies":7431,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":721,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2477,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":21690,"num_write_kb":171841,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":700,"num_bytes_recovered":45711360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,3","objects":1913}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.c","version":"66'22355","reported_seq":23882,"reported_epoch":66,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2026-02-23T13:09:20.346840+0000","last_change":"2026-02-23T13:09:20.058916+0000","last_active":"2026-02-23T13:09:20.346840+0000","last_peered":"2026-02-23T13:09:20.346840+0000","last_clean":"2026-02-23T13:08:31.176488+0000","last_became_active":"2026-02-23T13:09:20.052166+0000","last_became_peered":"2026-02-23T13:09:20.052166+0000","last_unstale":"2026-02-23T13:09:20.346840+0000","last_undegraded":"2026-02-23T13:09:19.983883+0000","last_fullsized":"2026-02-23T13:09:19.980353+0000","mapping_epoch":65,"log_start":"33'12300","ondisk_log_start":"33'12300","created":29,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_clean_scrub_stamp":"2026-02-23T13:07:30.224555+0000","objects_scrubbed":0,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":165216256,"num_objects":2521,"num_object_clones":0,"num_object_copies":7563,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":800,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2521,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22355,"num_write_kb":176896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":710,"num_bytes_recovered":46317568,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,3","objects":1926}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"28'99","reported_seq":114,"reported_epoch":68,"state":"active+clean","last_fresh":"2026-02-23T13:09:21.984434+0000","last_change":"2026-02-23T13:09:13.973193+0000","last_active":"2026-02-23T13:09:21.984434+0000","last_peered":"2026-02-23T13:09:21.984434+0000","last_clean":"2026-02-23T13:09:21.984434+0000","last_became_active":"2026-02-23T13:09:13.972259+0000","last_became_peered":"2026-02-23T13:09:13.972259+0000","last_unstale":"2026-02-23T13:09:21.984434+0000","last_undegraded":"2026-02-23T13:09:21.984434+0000","last_fullsized":"2026-02-23T13:09:21.984434+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'99","last_scrub_stamp":"2026-02-23T13:08:29.915607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:07:05.619208+0000","last_clean_scrub_stamp":"2026-02-23T13:08:29.915607+0000","objects_scrubbed":0,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T13:09:45.841302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":122,"num_read_kb":228,"num_write":135,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"68'22026","reported_seq":24924,"reported_epoch":68,"state":"active+clean","last_fresh":"2026-02-23T13:09:22.343263+0000","last_change":"2026-02-23T13:09:21.988904+0000","last_active":"2026-02-23T13:09:22.343263+0000","last_peered":"2026-02-23T13:09:22.343263+0000","last_clean":"2026-02-23T13:09:22.343263+0000","last_became_active":"2026-02-23T13:09:21.985593+0000","last_became_peered":"2026-02-23T13:09:21.985593+0000","last_unstale":"2026-02-23T13:09:22.343263+0000","last_undegraded":"2026-02-23T13:09:22.343263+0000","last_fullsized":"2026-02-23T13:09:22.343263+0000","mapping_epoch":67,"log_start":"33'12000","ondisk_log_start":"33'12000","created":29,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_clean_scrub_stamp":"2026-02-23T13:07:30.224555+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":143130624,"num_objects":2184,"num_object_clones":0,"num_object_copies":6552,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2184,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22026,"num_write_kb":172160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1389,"num_bytes_recovered":79233024,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"68'22004","reported_seq":24628,"reported_epoch":68,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2026-02-23T13:09:22.327642+0000","last_change":"2026-02-23T13:09:20.054037+0000","last_active":"2026-02-23T13:09:22.327642+0000","last_peered":"2026-02-23T13:09:22.327642+0000","last_clean":"2026-02-23T13:08:31.237071+0000","last_became_active":"2026-02-23T13:09:20.046498+0000","last_became_peered":"2026-02-23T13:09:20.046498+0000","last_unstale":"2026-02-23T13:09:22.327642+0000","last_undegraded":"2026-02-23T13:09:19.980807+0000","last_fullsized":"2026-02-23T13:09:19.970180+0000","mapping_epoch":65,"log_start":"33'12000","ondisk_log_start":"33'12000","created":29,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_clean_scrub_stamp":"2026-02-23T13:07:30.224555+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":143982592,"num_objects":2197,"num_object_clones":0,"num_object_copies":6591,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":190,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2197,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22004,"num_write_kb":172096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1259,"num_bytes_recovered":82378752,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[{"shards":"0,3","objects":2195}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"66'21867","reported_seq":23411,"reported_epoch":66,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2026-02-23T13:09:20.345641+0000","last_change":"2026-02-23T13:09:20.059842+0000","last_active":"2026-02-23T13:09:20.345641+0000","last_peered":"2026-02-23T13:09:20.345641+0000","last_clean":"2026-02-23T13:08:31.247020+0000","last_became_active":"2026-02-23T13:09:20.051670+0000","last_became_peered":"2026-02-23T13:09:20.051670+0000","last_unstale":"2026-02-23T13:09:20.345641+0000","last_undegraded":"2026-02-23T13:09:19.981209+0000","last_fullsized":"2026-02-23T13:09:19.970907+0000","mapping_epoch":65,"log_start":"33'11800","ondisk_log_start":"33'11800","created":29,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_clean_scrub_stamp":"2026-02-23T13:07:30.224555+0000","objects_scrubbed":0,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":162398208,"num_objects":2478,"num_object_clones":0,"num_object_copies":7434,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":779,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2478,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21867,"num_write_kb":173120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":717,"num_bytes_recovered":46931968,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,3","objects":1892}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"68'22174","reported_seq":24796,"reported_epoch":68,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2026-02-23T13:09:22.345946+0000","last_change":"2026-02-23T13:09:20.055811+0000","last_active":"2026-02-23T13:09:22.345946+0000","last_peered":"2026-02-23T13:09:22.345946+0000","last_clean":"2026-02-23T13:08:31.252741+0000","last_became_active":"2026-02-23T13:09:20.050327+0000","last_became_peered":"2026-02-23T13:09:20.050327+0000","last_unstale":"2026-02-23T13:09:22.345946+0000","last_undegraded":"2026-02-23T13:09:19.980985+0000","last_fullsized":"2026-02-23T13:09:19.970054+0000","mapping_epoch":65,"log_start":"33'12100","ondisk_log_start":"33'12100","created":29,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_clean_scrub_stamp":"2026-02-23T13:07:30.224555+0000","objects_scrubbed":0,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":144048128,"num_objects":2198,"num_object_clones":0,"num_object_copies":6594,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":203,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2198,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22174,"num_write_kb":173312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1258,"num_bytes_recovered":82255872,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":2197}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"66'21810","reported_seq":22364,"reported_epoch":66,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2026-02-23T13:09:20.330262+0000","last_change":"2026-02-23T13:09:20.058899+0000","last_active":"2026-02-23T13:09:20.330262+0000","last_peered":"2026-02-23T13:09:20.330262+0000","last_clean":"2026-02-23T13:09:01.266173+0000","last_became_active":"2026-02-23T13:09:20.051618+0000","last_became_peered":"2026-02-23T13:09:20.051618+0000","last_unstale":"2026-02-23T13:09:20.330262+0000","last_undegraded":"2026-02-23T13:09:19.980199+0000","last_fullsized":"2026-02-23T13:09:19.970748+0000","mapping_epoch":65,"log_start":"33'11800","ondisk_log_start":"33'11800","created":29,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_clean_scrub_stamp":"2026-02-23T13:07:30.224555+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":162004992,"num_objects":2472,"num_object_clones":0,"num_object_copies":7416,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":167,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2472,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21810,"num_write_kb":172672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":543,"num_bytes_recovered":35471360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,2","objects":2471}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"68'22350","reported_seq":23890,"reported_epoch":68,"state":"active+clean","last_fresh":"2026-02-23T13:09:22.339386+0000","last_change":"2026-02-23T13:09:18.710906+0000","last_active":"2026-02-23T13:09:22.339386+0000","last_peered":"2026-02-23T13:09:22.339386+0000","last_clean":"2026-02-23T13:09:22.339386+0000","last_became_active":"2026-02-23T13:09:17.980954+0000","last_became_peered":"2026-02-23T13:09:17.980954+0000","last_unstale":"2026-02-23T13:09:22.339386+0000","last_undegraded":"2026-02-23T13:09:22.339386+0000","last_fullsized":"2026-02-23T13:09:22.339386+0000","mapping_epoch":63,"log_start":"33'12300","ondisk_log_start":"33'12300","created":29,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'22042","last_scrub_stamp":"2026-02-23T13:09:18.710867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_clean_scrub_stamp":"2026-02-23T13:09:18.710867+0000","objects_scrubbed":2519,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T13:10:31.775316+0000","scrub_duration":127,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":144900096,"num_objects":2211,"num_object_clones":0,"num_object_copies":6633,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2211,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22350,"num_write_kb":174656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":707,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"66'21867","reported_seq":23473,"reported_epoch":66,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2026-02-23T13:09:20.378847+0000","last_change":"2026-02-23T13:09:20.058603+0000","last_active":"2026-02-23T13:09:20.378847+0000","last_peered":"2026-02-23T13:09:20.378847+0000","last_clean":"2026-02-23T13:08:31.205210+0000","last_became_active":"2026-02-23T13:09:20.050775+0000","last_became_peered":"2026-02-23T13:09:20.050775+0000","last_unstale":"2026-02-23T13:09:20.378847+0000","last_undegraded":"2026-02-23T13:09:19.978113+0000","last_fullsized":"2026-02-23T13:09:19.969317+0000","mapping_epoch":65,"log_start":"33'11800","ondisk_log_start":"33'11800","created":29,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_clean_scrub_stamp":"2026-02-23T13:07:30.224555+0000","objects_scrubbed":0,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":160628736,"num_objects":2451,"num_object_clones":0,"num_object_copies":7353,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":689,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2451,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21867,"num_write_kb":172928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":744,"num_bytes_recovered":48734208,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[2,3],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":1946}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"68'21762","reported_seq":24718,"reported_epoch":68,"state":"active+clean","last_fresh":"2026-02-23T13:09:22.336115+0000","last_change":"2026-02-23T13:09:22.090938+0000","last_active":"2026-02-23T13:09:22.336115+0000","last_peered":"2026-02-23T13:09:22.336115+0000","last_clean":"2026-02-23T13:09:22.336115+0000","last_became_active":"2026-02-23T13:09:21.996628+0000","last_became_peered":"2026-02-23T13:09:21.996628+0000","last_unstale":"2026-02-23T13:09:22.336115+0000","last_undegraded":"2026-02-23T13:09:22.336115+0000","last_fullsized":"2026-02-23T13:09:22.336115+0000","mapping_epoch":67,"log_start":"33'11700","ondisk_log_start":"33'11700","created":29,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_clean_scrub_stamp":"2026-02-23T13:07:30.224555+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":143917056,"num_objects":2196,"num_object_clones":0,"num_object_copies":6588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2196,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21762,"num_write_kb":170368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1419,"num_bytes_recovered":80683008,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.8","version":"68'22361","reported_seq":23446,"reported_epoch":68,"state":"active+clean","last_fresh":"2026-02-23T13:09:22.339495+0000","last_change":"2026-02-23T13:09:17.986383+0000","last_active":"2026-02-23T13:09:22.339495+0000","last_peered":"2026-02-23T13:09:22.339495+0000","last_clean":"2026-02-23T13:09:22.339495+0000","last_became_active":"2026-02-23T13:09:17.980940+0000","last_became_peered":"2026-02-23T13:09:17.980940+0000","last_unstale":"2026-02-23T13:09:22.339495+0000","last_undegraded":"2026-02-23T13:09:22.339495+0000","last_fullsized":"2026-02-23T13:09:22.339495+0000","mapping_epoch":63,"log_start":"33'12300","ondisk_log_start":"33'12300","created":29,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"49'20400","last_scrub_stamp":"2026-02-23T13:08:49.984469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_clean_scrub_stamp":"2026-02-23T13:08:49.984469+0000","objects_scrubbed":2548,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T13:09:50.946231+0000","scrub_duration":928,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":144769024,"num_objects":2209,"num_object_clones":0,"num_object_copies":6627,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2209,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22361,"num_write_kb":174720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":664,"num_bytes_recovered":43491328,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"66'22915","reported_seq":24409,"reported_epoch":66,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2026-02-23T13:09:20.342277+0000","last_change":"2026-02-23T13:09:20.059873+0000","last_active":"2026-02-23T13:09:20.342277+0000","last_peered":"2026-02-23T13:09:20.342277+0000","last_clean":"2026-02-23T13:08:31.257695+0000","last_became_active":"2026-02-23T13:09:20.051907+0000","last_became_peered":"2026-02-23T13:09:20.051907+0000","last_unstale":"2026-02-23T13:09:20.342277+0000","last_undegraded":"2026-02-23T13:09:19.980849+0000","last_fullsized":"2026-02-23T13:09:19.970886+0000","mapping_epoch":65,"log_start":"33'12900","ondisk_log_start":"33'12900","created":29,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_clean_scrub_stamp":"2026-02-23T13:07:30.224555+0000","objects_scrubbed":0,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":170098688,"num_objects":2596,"num_object_clones":0,"num_object_copies":7788,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":826,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2596,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22915,"num_write_kb":181408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":691,"num_bytes_recovered":45056000,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,3","objects":1959}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"66'22095","reported_seq":23620,"reported_epoch":66,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2026-02-23T13:09:20.378266+0000","last_change":"2026-02-23T13:09:20.074080+0000","last_active":"2026-02-23T13:09:20.378266+0000","last_peered":"2026-02-23T13:09:20.378266+0000","last_clean":"2026-02-23T13:08:31.221592+0000","last_became_active":"2026-02-23T13:09:20.050909+0000","last_became_peered":"2026-02-23T13:09:20.050909+0000","last_unstale":"2026-02-23T13:09:20.378266+0000","last_undegraded":"2026-02-23T13:09:19.978399+0000","last_fullsized":"2026-02-23T13:09:19.968983+0000","mapping_epoch":65,"log_start":"33'12000","ondisk_log_start":"33'12000","created":29,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_clean_scrub_stamp":"2026-02-23T13:07:30.224555+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":161611776,"num_objects":2466,"num_object_clones":0,"num_object_copies":7398,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":550,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2466,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22095,"num_write_kb":174656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":705,"num_bytes_recovered":46039040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[2,3],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":1916}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"68'22512","reported_seq":23644,"reported_epoch":68,"state":"active+clean","last_fresh":"2026-02-23T13:09:22.325793+0000","last_change":"2026-02-23T13:09:17.986451+0000","last_active":"2026-02-23T13:09:22.325793+0000","last_peered":"2026-02-23T13:09:22.325793+0000","last_clean":"2026-02-23T13:09:22.325793+0000","last_became_active":"2026-02-23T13:09:17.981144+0000","last_became_peered":"2026-02-23T13:09:17.981144+0000","last_unstale":"2026-02-23T13:09:22.325793+0000","last_undegraded":"2026-02-23T13:09:22.325793+0000","last_fullsized":"2026-02-23T13:09:22.325793+0000","mapping_epoch":63,"log_start":"33'12500","ondisk_log_start":"33'12500","created":29,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"48'18824","last_scrub_stamp":"2026-02-23T13:08:42.411326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_clean_scrub_stamp":"2026-02-23T13:08:42.411326+0000","objects_scrubbed":2347,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T13:10:03.835120+0000","scrub_duration":1426,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":146669568,"num_objects":2238,"num_object_clones":0,"num_object_copies":6714,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2238,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22512,"num_write_kb":176000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":710,"num_bytes_recovered":46473216,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":2480701464,"num_objects":37854,"num_object_clones":0,"num_object_copies":113562,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5093,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37854,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":353986,"num_write_kb":2786401,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13503,"num_bytes_recovered":859455488,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":7380680704,"data_stored":7380672560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160786,"ondisk_log_size":160786,"up":48,"acting":38,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":122,"num_read_kb":228,"num_write":135,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":59,"seq":253403070467,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":734003200,"kb_used":2120568,"kb_used_data":2060196,"kb_used_omap":12,"kb_used_meta":60339,"kb_avail":731882632,"statfs":{"total":751619276800,"available":749447815168,"internally_reserved":0,"allocated":2109640704,"data_stored":2109185686,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12568,"internal_metadata":61787880},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,2],"upper_bound":2048},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411358,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":734003200,"kb_used":2238716,"kb_used_data":1778600,"kb_used_omap":34,"kb_used_meta":460061,"kb_avail":731764484,"statfs":{"total":751619276800,"available":749326831616,"internally_reserved":0,"allocated":1821286400,"data_stored":1820850508,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":35477,"internal_metadata":471102827},"hb_peers":[0,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,1],"upper_bound":2048},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":64,"seq":274877906946,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":734003200,"kb_used":1731928,"kb_used_data":1682436,"kb_used_omap":9,"kb_used_meta":49462,"kb_avail":732271272,"statfs":{"total":751619276800,"available":749845782528,"internally_reserved":0,"allocated":1722814464,"data_stored":1722367070,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9949,"internal_metadata":50649379},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705697,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":734003200,"kb_used":2233760,"kb_used_data":1791500,"kb_used_omap":34,"kb_used_meta":442205,"kb_avail":731769440,"statfs":{"total":751619276800,"available":749331906560,"internally_reserved":0,"allocated":1834496000,"data_stored":1834056036,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":35477,"internal_metadata":452818283},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,52],"upper_bound":4096},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1793363968,"data_stored":1793359896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1770397696,"data_stored":1770397696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1770323968,"data_stored":1770323968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2046595072,"data_stored":2046591000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-23T13:09:24.870 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph status --format=json 2026-02-23T13:09:24.997 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:09:25.130 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:09:25.128+0000 7f3306810640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:09:25.132 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:09:25.130+0000 7f3306810640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:09:25.335 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T13:09:25.336 INFO:teuthology.orchestra.run.trial068.stdout:{"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 5093/113568 objects degraded (4.485%), 10 pgs degraded","count":10},"muted":false}},"mutes":[]},"election_epoch":5,"quorum":[0],"quorum_names":["a"],"quorum_age":201,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":1},"osdmap":{"epoch":71,"num_osds":4,"num_up_osds":4,"osd_up_since":1771852157,"num_in_osds":4,"osd_in_since":1771852023,"num_remapped_pgs":2},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":9},{"state_name":"active+clean","count":7},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":17,"num_pools":2,"num_objects":37856,"data_bytes":2481291832,"bytes_used":8524771328,"bytes_avail":2997952335872,"bytes_total":3006477107200,"degraded_objects":5093,"degraded_total":113568,"degraded_ratio":0.044845378979994362,"recovering_objects_per_sec":129,"recovering_bytes_per_sec":2481767,"recovering_keys_per_sec":0,"num_objects_recovered":518,"num_bytes_recovered":9928704,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":916},"fsmap":{"epoch":1,"btime":"2026-02-23T13:06:03:098534+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["cephadm","dashboard","iostat","nfs","prometheus"],"services":{"dashboard":"https://10.20.193.68:8443/","prometheus":"http://10.20.193.68:9283/"}},"servicemap":{"epoch":3,"modified":"2026-02-23T13:09:12.240067+0000","services":{"osd":{"daemons":{"summary":"","1":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2026-02-23T13:09:25.336 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:09:24 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/988609824' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:09:25.336 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:09:24 trial068 ceph-mon[38805]: pgmap v105: 17 pgs: 1 active+recovering+undersized+degraded+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 7 active+clean; 2.3 GiB data, 7.9 GiB used, 2.7 TiB / 2.7 TiB avail; 0 B/s wr, 916 op/s; 5093/113568 objects degraded (4.485%); 2.4 MiB/s, 129 objects/s recovering 2026-02-23T13:09:25.336 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:09:24 trial068 ceph-mon[38805]: from='client.14485 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2026-02-23T13:09:25.336 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:09:24 trial068 ceph-mon[38805]: osdmap e71: 4 total, 4 up, 4 in 2026-02-23T13:09:25.385 INFO:tasks.cephadm.ceph_manager.ceph:making progress, resetting timeout 2026-02-23T13:09:25.385 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph pg dump --format=json 2026-02-23T13:09:25.520 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:09:25.643 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:09:25.641+0000 7fb31870a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:09:25.643 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:09:25.641+0000 7fb31870a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:09:25.762 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T13:09:25.762 INFO:teuthology.orchestra.run.trial068.stderr:dumped all 2026-02-23T13:09:25.801 INFO:teuthology.orchestra.run.trial068.stdout:{"pg_ready":true,"pg_map":{"version":106,"stamp":"2026-02-23T13:09:24.971756+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2481291832,"num_objects":37856,"num_object_clones":0,"num_object_copies":113568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5093,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37856,"num_whiteouts":0,"num_read":123,"num_read_kb":229,"num_write":354121,"num_write_kb":2788551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13505,"num_bytes_recovered":860045856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160885,"ondisk_log_size":160885,"up":51,"acting":41,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":51,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":2936012800,"kb_used":8324972,"kb_used_data":7312732,"kb_used_omap":91,"kb_used_meta":1012068,"kb_avail":2927687828,"statfs":{"total":3006477107200,"available":2997952335872,"internally_reserved":0,"allocated":7488237568,"data_stored":7486459300,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":93471,"internal_metadata":1036358369},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,3,52],"upper_bound":4096},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-148111360,"num_objects":-2260,"num_object_clones":0,"num_object_copies":-6780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-7090,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2260,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2260,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":366,"num_bytes_recovered":24576,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"3.993564"},"pg_stats":[{"pgid":"2.f","version":"66'22517","reported_seq":24085,"reported_epoch":66,"state":"active+clean","last_fresh":"2026-02-23T13:09:20.371853+0000","last_change":"2026-02-23T13:09:17.988736+0000","last_active":"2026-02-23T13:09:20.371853+0000","last_peered":"2026-02-23T13:09:20.371853+0000","last_clean":"2026-02-23T13:09:20.371853+0000","last_became_active":"2026-02-23T13:09:17.980759+0000","last_became_peered":"2026-02-23T13:09:17.980759+0000","last_unstale":"2026-02-23T13:09:20.371853+0000","last_undegraded":"2026-02-23T13:09:20.371853+0000","last_fullsized":"2026-02-23T13:09:20.371853+0000","mapping_epoch":63,"log_start":"33'12500","ondisk_log_start":"33'12500","created":29,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"48'17576","last_scrub_stamp":"2026-02-23T13:08:38.307705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_clean_scrub_stamp":"2026-02-23T13:08:38.307705+0000","objects_scrubbed":2192,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T13:10:04.053022+0000","scrub_duration":368,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":165216256,"num_objects":2521,"num_object_clones":0,"num_object_copies":7563,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2521,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22517,"num_write_kb":178048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":710,"num_bytes_recovered":46530560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.e","version":"66'21681","reported_seq":21780,"reported_epoch":66,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2026-02-23T13:09:20.338985+0000","last_change":"2026-02-23T13:09:20.057612+0000","last_active":"2026-02-23T13:09:20.338985+0000","last_peered":"2026-02-23T13:09:20.338985+0000","last_clean":"2026-02-23T13:09:01.297181+0000","last_became_active":"2026-02-23T13:09:20.050781+0000","last_became_peered":"2026-02-23T13:09:20.050781+0000","last_unstale":"2026-02-23T13:09:20.338985+0000","last_undegraded":"2026-02-23T13:09:19.980426+0000","last_fullsized":"2026-02-23T13:09:19.970754+0000","mapping_epoch":65,"log_start":"33'11600","ondisk_log_start":"33'11600","created":29,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_clean_scrub_stamp":"2026-02-23T13:07:30.224555+0000","objects_scrubbed":0,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":159842304,"num_objects":2439,"num_object_clones":0,"num_object_copies":7317,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":168,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2439,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21681,"num_write_kb":171520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":577,"num_bytes_recovered":37814272,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,2","objects":2439}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"66'21690","reported_seq":23197,"reported_epoch":66,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2026-02-23T13:09:20.348302+0000","last_change":"2026-02-23T13:09:20.058876+0000","last_active":"2026-02-23T13:09:20.348302+0000","last_peered":"2026-02-23T13:09:20.348302+0000","last_clean":"2026-02-23T13:08:31.228846+0000","last_became_active":"2026-02-23T13:09:20.049449+0000","last_became_peered":"2026-02-23T13:09:20.049449+0000","last_unstale":"2026-02-23T13:09:20.348302+0000","last_undegraded":"2026-02-23T13:09:19.980906+0000","last_fullsized":"2026-02-23T13:09:19.970900+0000","mapping_epoch":65,"log_start":"33'11600","ondisk_log_start":"33'11600","created":29,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_clean_scrub_stamp":"2026-02-23T13:07:30.224555+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":162267160,"num_objects":2477,"num_object_clones":0,"num_object_copies":7431,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":721,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2477,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":21690,"num_write_kb":171841,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":700,"num_bytes_recovered":45711360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,3","objects":1913}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.c","version":"66'22355","reported_seq":23882,"reported_epoch":66,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2026-02-23T13:09:20.346840+0000","last_change":"2026-02-23T13:09:20.058916+0000","last_active":"2026-02-23T13:09:20.346840+0000","last_peered":"2026-02-23T13:09:20.346840+0000","last_clean":"2026-02-23T13:08:31.176488+0000","last_became_active":"2026-02-23T13:09:20.052166+0000","last_became_peered":"2026-02-23T13:09:20.052166+0000","last_unstale":"2026-02-23T13:09:20.346840+0000","last_undegraded":"2026-02-23T13:09:19.983883+0000","last_fullsized":"2026-02-23T13:09:19.980353+0000","mapping_epoch":65,"log_start":"33'12300","ondisk_log_start":"33'12300","created":29,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_clean_scrub_stamp":"2026-02-23T13:07:30.224555+0000","objects_scrubbed":0,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":165216256,"num_objects":2521,"num_object_clones":0,"num_object_copies":7563,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":800,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2521,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22355,"num_write_kb":176896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":710,"num_bytes_recovered":46317568,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,3","objects":1926}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"28'99","reported_seq":114,"reported_epoch":68,"state":"active+clean","last_fresh":"2026-02-23T13:09:21.984434+0000","last_change":"2026-02-23T13:09:13.973193+0000","last_active":"2026-02-23T13:09:21.984434+0000","last_peered":"2026-02-23T13:09:21.984434+0000","last_clean":"2026-02-23T13:09:21.984434+0000","last_became_active":"2026-02-23T13:09:13.972259+0000","last_became_peered":"2026-02-23T13:09:13.972259+0000","last_unstale":"2026-02-23T13:09:21.984434+0000","last_undegraded":"2026-02-23T13:09:21.984434+0000","last_fullsized":"2026-02-23T13:09:21.984434+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'99","last_scrub_stamp":"2026-02-23T13:08:29.915607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:07:05.619208+0000","last_clean_scrub_stamp":"2026-02-23T13:08:29.915607+0000","objects_scrubbed":0,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T13:09:45.841302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":122,"num_read_kb":228,"num_write":135,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"68'22026","reported_seq":24924,"reported_epoch":68,"state":"active+clean","last_fresh":"2026-02-23T13:09:22.343263+0000","last_change":"2026-02-23T13:09:21.988904+0000","last_active":"2026-02-23T13:09:22.343263+0000","last_peered":"2026-02-23T13:09:22.343263+0000","last_clean":"2026-02-23T13:09:22.343263+0000","last_became_active":"2026-02-23T13:09:21.985593+0000","last_became_peered":"2026-02-23T13:09:21.985593+0000","last_unstale":"2026-02-23T13:09:22.343263+0000","last_undegraded":"2026-02-23T13:09:22.343263+0000","last_fullsized":"2026-02-23T13:09:22.343263+0000","mapping_epoch":67,"log_start":"33'12000","ondisk_log_start":"33'12000","created":29,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_clean_scrub_stamp":"2026-02-23T13:07:30.224555+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":143130624,"num_objects":2184,"num_object_clones":0,"num_object_copies":6552,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2184,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22026,"num_write_kb":172160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1389,"num_bytes_recovered":79233024,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"68'22004","reported_seq":24628,"reported_epoch":68,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2026-02-23T13:09:22.327642+0000","last_change":"2026-02-23T13:09:20.054037+0000","last_active":"2026-02-23T13:09:22.327642+0000","last_peered":"2026-02-23T13:09:22.327642+0000","last_clean":"2026-02-23T13:08:31.237071+0000","last_became_active":"2026-02-23T13:09:20.046498+0000","last_became_peered":"2026-02-23T13:09:20.046498+0000","last_unstale":"2026-02-23T13:09:22.327642+0000","last_undegraded":"2026-02-23T13:09:19.980807+0000","last_fullsized":"2026-02-23T13:09:19.970180+0000","mapping_epoch":65,"log_start":"33'12000","ondisk_log_start":"33'12000","created":29,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_clean_scrub_stamp":"2026-02-23T13:07:30.224555+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":143982592,"num_objects":2197,"num_object_clones":0,"num_object_copies":6591,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":190,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2197,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22004,"num_write_kb":172096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1259,"num_bytes_recovered":82378752,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[{"shards":"0,3","objects":2195}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"66'21867","reported_seq":23411,"reported_epoch":66,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2026-02-23T13:09:20.345641+0000","last_change":"2026-02-23T13:09:20.059842+0000","last_active":"2026-02-23T13:09:20.345641+0000","last_peered":"2026-02-23T13:09:20.345641+0000","last_clean":"2026-02-23T13:08:31.247020+0000","last_became_active":"2026-02-23T13:09:20.051670+0000","last_became_peered":"2026-02-23T13:09:20.051670+0000","last_unstale":"2026-02-23T13:09:20.345641+0000","last_undegraded":"2026-02-23T13:09:19.981209+0000","last_fullsized":"2026-02-23T13:09:19.970907+0000","mapping_epoch":65,"log_start":"33'11800","ondisk_log_start":"33'11800","created":29,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_clean_scrub_stamp":"2026-02-23T13:07:30.224555+0000","objects_scrubbed":0,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":162398208,"num_objects":2478,"num_object_clones":0,"num_object_copies":7434,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":779,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2478,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21867,"num_write_kb":173120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":717,"num_bytes_recovered":46931968,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,3","objects":1892}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"68'22174","reported_seq":24796,"reported_epoch":68,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2026-02-23T13:09:22.345946+0000","last_change":"2026-02-23T13:09:20.055811+0000","last_active":"2026-02-23T13:09:22.345946+0000","last_peered":"2026-02-23T13:09:22.345946+0000","last_clean":"2026-02-23T13:08:31.252741+0000","last_became_active":"2026-02-23T13:09:20.050327+0000","last_became_peered":"2026-02-23T13:09:20.050327+0000","last_unstale":"2026-02-23T13:09:22.345946+0000","last_undegraded":"2026-02-23T13:09:19.980985+0000","last_fullsized":"2026-02-23T13:09:19.970054+0000","mapping_epoch":65,"log_start":"33'12100","ondisk_log_start":"33'12100","created":29,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_clean_scrub_stamp":"2026-02-23T13:07:30.224555+0000","objects_scrubbed":0,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":144048128,"num_objects":2198,"num_object_clones":0,"num_object_copies":6594,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":203,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2198,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22174,"num_write_kb":173312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1258,"num_bytes_recovered":82255872,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":2197}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"66'21810","reported_seq":22364,"reported_epoch":66,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2026-02-23T13:09:20.330262+0000","last_change":"2026-02-23T13:09:20.058899+0000","last_active":"2026-02-23T13:09:20.330262+0000","last_peered":"2026-02-23T13:09:20.330262+0000","last_clean":"2026-02-23T13:09:01.266173+0000","last_became_active":"2026-02-23T13:09:20.051618+0000","last_became_peered":"2026-02-23T13:09:20.051618+0000","last_unstale":"2026-02-23T13:09:20.330262+0000","last_undegraded":"2026-02-23T13:09:19.980199+0000","last_fullsized":"2026-02-23T13:09:19.970748+0000","mapping_epoch":65,"log_start":"33'11800","ondisk_log_start":"33'11800","created":29,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_clean_scrub_stamp":"2026-02-23T13:07:30.224555+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":162004992,"num_objects":2472,"num_object_clones":0,"num_object_copies":7416,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":167,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2472,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21810,"num_write_kb":172672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":543,"num_bytes_recovered":35471360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,2","objects":2471}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"68'22350","reported_seq":23890,"reported_epoch":68,"state":"active+clean","last_fresh":"2026-02-23T13:09:22.339386+0000","last_change":"2026-02-23T13:09:18.710906+0000","last_active":"2026-02-23T13:09:22.339386+0000","last_peered":"2026-02-23T13:09:22.339386+0000","last_clean":"2026-02-23T13:09:22.339386+0000","last_became_active":"2026-02-23T13:09:17.980954+0000","last_became_peered":"2026-02-23T13:09:17.980954+0000","last_unstale":"2026-02-23T13:09:22.339386+0000","last_undegraded":"2026-02-23T13:09:22.339386+0000","last_fullsized":"2026-02-23T13:09:22.339386+0000","mapping_epoch":63,"log_start":"33'12300","ondisk_log_start":"33'12300","created":29,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'22042","last_scrub_stamp":"2026-02-23T13:09:18.710867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_clean_scrub_stamp":"2026-02-23T13:09:18.710867+0000","objects_scrubbed":2519,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T13:10:31.775316+0000","scrub_duration":127,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":144900096,"num_objects":2211,"num_object_clones":0,"num_object_copies":6633,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2211,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22350,"num_write_kb":174656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":707,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"66'21867","reported_seq":23473,"reported_epoch":66,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2026-02-23T13:09:20.378847+0000","last_change":"2026-02-23T13:09:20.058603+0000","last_active":"2026-02-23T13:09:20.378847+0000","last_peered":"2026-02-23T13:09:20.378847+0000","last_clean":"2026-02-23T13:08:31.205210+0000","last_became_active":"2026-02-23T13:09:20.050775+0000","last_became_peered":"2026-02-23T13:09:20.050775+0000","last_unstale":"2026-02-23T13:09:20.378847+0000","last_undegraded":"2026-02-23T13:09:19.978113+0000","last_fullsized":"2026-02-23T13:09:19.969317+0000","mapping_epoch":65,"log_start":"33'11800","ondisk_log_start":"33'11800","created":29,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_clean_scrub_stamp":"2026-02-23T13:07:30.224555+0000","objects_scrubbed":0,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":160628736,"num_objects":2451,"num_object_clones":0,"num_object_copies":7353,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":689,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2451,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21867,"num_write_kb":172928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":744,"num_bytes_recovered":48734208,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[2,3],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":1946}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"68'21762","reported_seq":24718,"reported_epoch":68,"state":"active+clean","last_fresh":"2026-02-23T13:09:22.336115+0000","last_change":"2026-02-23T13:09:22.090938+0000","last_active":"2026-02-23T13:09:22.336115+0000","last_peered":"2026-02-23T13:09:22.336115+0000","last_clean":"2026-02-23T13:09:22.336115+0000","last_became_active":"2026-02-23T13:09:21.996628+0000","last_became_peered":"2026-02-23T13:09:21.996628+0000","last_unstale":"2026-02-23T13:09:22.336115+0000","last_undegraded":"2026-02-23T13:09:22.336115+0000","last_fullsized":"2026-02-23T13:09:22.336115+0000","mapping_epoch":67,"log_start":"33'11700","ondisk_log_start":"33'11700","created":29,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_clean_scrub_stamp":"2026-02-23T13:07:30.224555+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":143917056,"num_objects":2196,"num_object_clones":0,"num_object_copies":6588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2196,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21762,"num_write_kb":170368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1419,"num_bytes_recovered":80683008,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.8","version":"68'22361","reported_seq":23446,"reported_epoch":68,"state":"active+clean","last_fresh":"2026-02-23T13:09:22.339495+0000","last_change":"2026-02-23T13:09:17.986383+0000","last_active":"2026-02-23T13:09:22.339495+0000","last_peered":"2026-02-23T13:09:22.339495+0000","last_clean":"2026-02-23T13:09:22.339495+0000","last_became_active":"2026-02-23T13:09:17.980940+0000","last_became_peered":"2026-02-23T13:09:17.980940+0000","last_unstale":"2026-02-23T13:09:22.339495+0000","last_undegraded":"2026-02-23T13:09:22.339495+0000","last_fullsized":"2026-02-23T13:09:22.339495+0000","mapping_epoch":63,"log_start":"33'12300","ondisk_log_start":"33'12300","created":29,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"49'20400","last_scrub_stamp":"2026-02-23T13:08:49.984469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_clean_scrub_stamp":"2026-02-23T13:08:49.984469+0000","objects_scrubbed":2548,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T13:09:50.946231+0000","scrub_duration":928,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":144769024,"num_objects":2209,"num_object_clones":0,"num_object_copies":6627,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2209,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22361,"num_write_kb":174720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":664,"num_bytes_recovered":43491328,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"66'22915","reported_seq":24409,"reported_epoch":66,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2026-02-23T13:09:20.342277+0000","last_change":"2026-02-23T13:09:20.059873+0000","last_active":"2026-02-23T13:09:20.342277+0000","last_peered":"2026-02-23T13:09:20.342277+0000","last_clean":"2026-02-23T13:08:31.257695+0000","last_became_active":"2026-02-23T13:09:20.051907+0000","last_became_peered":"2026-02-23T13:09:20.051907+0000","last_unstale":"2026-02-23T13:09:20.342277+0000","last_undegraded":"2026-02-23T13:09:19.980849+0000","last_fullsized":"2026-02-23T13:09:19.970886+0000","mapping_epoch":65,"log_start":"33'12900","ondisk_log_start":"33'12900","created":29,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_clean_scrub_stamp":"2026-02-23T13:07:30.224555+0000","objects_scrubbed":0,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":170098688,"num_objects":2596,"num_object_clones":0,"num_object_copies":7788,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":826,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2596,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22915,"num_write_kb":181408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":691,"num_bytes_recovered":45056000,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,3","objects":1959}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"66'22095","reported_seq":23620,"reported_epoch":66,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2026-02-23T13:09:20.378266+0000","last_change":"2026-02-23T13:09:20.074080+0000","last_active":"2026-02-23T13:09:20.378266+0000","last_peered":"2026-02-23T13:09:20.378266+0000","last_clean":"2026-02-23T13:08:31.221592+0000","last_became_active":"2026-02-23T13:09:20.050909+0000","last_became_peered":"2026-02-23T13:09:20.050909+0000","last_unstale":"2026-02-23T13:09:20.378266+0000","last_undegraded":"2026-02-23T13:09:19.978399+0000","last_fullsized":"2026-02-23T13:09:19.968983+0000","mapping_epoch":65,"log_start":"33'12000","ondisk_log_start":"33'12000","created":29,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_clean_scrub_stamp":"2026-02-23T13:07:30.224555+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":161611776,"num_objects":2466,"num_object_clones":0,"num_object_copies":7398,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":550,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2466,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22095,"num_write_kb":174656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":705,"num_bytes_recovered":46039040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[2,3],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":1916}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"68'22512","reported_seq":23644,"reported_epoch":68,"state":"active+clean","last_fresh":"2026-02-23T13:09:22.325793+0000","last_change":"2026-02-23T13:09:17.986451+0000","last_active":"2026-02-23T13:09:22.325793+0000","last_peered":"2026-02-23T13:09:22.325793+0000","last_clean":"2026-02-23T13:09:22.325793+0000","last_became_active":"2026-02-23T13:09:17.981144+0000","last_became_peered":"2026-02-23T13:09:17.981144+0000","last_unstale":"2026-02-23T13:09:22.325793+0000","last_undegraded":"2026-02-23T13:09:22.325793+0000","last_fullsized":"2026-02-23T13:09:22.325793+0000","mapping_epoch":63,"log_start":"33'12500","ondisk_log_start":"33'12500","created":29,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"48'18824","last_scrub_stamp":"2026-02-23T13:08:42.411326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_clean_scrub_stamp":"2026-02-23T13:08:42.411326+0000","objects_scrubbed":2347,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T13:10:03.835120+0000","scrub_duration":1426,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":146669568,"num_objects":2238,"num_object_clones":0,"num_object_copies":6714,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2238,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22512,"num_write_kb":176000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":710,"num_bytes_recovered":46473216,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":2480701464,"num_objects":37854,"num_object_clones":0,"num_object_copies":113562,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5093,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37854,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":353986,"num_write_kb":2786401,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13503,"num_bytes_recovered":859455488,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":7380680704,"data_stored":7380672560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160786,"ondisk_log_size":160786,"up":48,"acting":38,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":122,"num_read_kb":228,"num_write":135,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":59,"seq":253403070467,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":734003200,"kb_used":2120568,"kb_used_data":2060196,"kb_used_omap":12,"kb_used_meta":60339,"kb_avail":731882632,"statfs":{"total":751619276800,"available":749447815168,"internally_reserved":0,"allocated":2109640704,"data_stored":2109185686,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12568,"internal_metadata":61787880},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,2],"upper_bound":2048},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411358,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":734003200,"kb_used":2238716,"kb_used_data":1778600,"kb_used_omap":34,"kb_used_meta":460061,"kb_avail":731764484,"statfs":{"total":751619276800,"available":749326831616,"internally_reserved":0,"allocated":1821286400,"data_stored":1820850508,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":35477,"internal_metadata":471102827},"hb_peers":[0,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,1],"upper_bound":2048},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":64,"seq":274877906946,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":734003200,"kb_used":1731928,"kb_used_data":1682436,"kb_used_omap":9,"kb_used_meta":49462,"kb_avail":732271272,"statfs":{"total":751619276800,"available":749845782528,"internally_reserved":0,"allocated":1722814464,"data_stored":1722367070,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9949,"internal_metadata":50649379},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705697,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":734003200,"kb_used":2233760,"kb_used_data":1791500,"kb_used_omap":34,"kb_used_meta":442205,"kb_avail":731769440,"statfs":{"total":751619276800,"available":749331906560,"internally_reserved":0,"allocated":1834496000,"data_stored":1834056036,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":35477,"internal_metadata":452818283},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,52],"upper_bound":4096},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1793363968,"data_stored":1793359896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1770397696,"data_stored":1770397696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1770323968,"data_stored":1770323968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2046595072,"data_stored":2046591000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-23T13:09:26.132 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:09:26.156 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:09:25 trial068 ceph-mon[38805]: from='client.14487 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2026-02-23T13:09:26.156 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:09:25 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/710300446' entity='client.admin' cmd={"prefix": "status", "format": "json"} : dispatch 2026-02-23T13:09:26.156 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:09:25 trial068 ceph-mon[38805]: osdmap e72: 4 total, 4 up, 4 in 2026-02-23T13:09:26.270 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:09:26.383 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:09:26.382+0000 7f8b5bc12640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:09:26.384 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:09:26.382+0000 7f8b5bc12640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:09:26.397 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:09:26.398 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:09:26.408 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:09:26.408 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:09:26.418 INFO:teuthology.orchestra.run.trial068.stdout:osd.2: {} 2026-02-23T13:09:26.418 INFO:teuthology.orchestra.run.trial068.stderr:osd.2: osd_enable_op_tracker = '' 2026-02-23T13:09:26.429 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:09:26.430 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:09:27.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:09:26 trial068 ceph-mon[38805]: from='client.14491 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2026-02-23T13:09:27.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:09:26 trial068 ceph-mon[38805]: pgmap v108: 17 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 7 active+clean; 2.2 GiB data, 7.3 GiB used, 2.7 TiB / 2.7 TiB avail; 0 B/s wr, 941 op/s; 3542/109050 objects degraded (3.248%); 18 MiB/s, 410 objects/s recovering 2026-02-23T13:09:27.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:09:26 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/395708902' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:09:27.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:09:26 trial068 ceph-mon[38805]: 2.7 scrub starts 2026-02-23T13:09:27.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:09:26 trial068 ceph-mon[38805]: 2.7 scrub ok 2026-02-23T13:09:27.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:09:26 trial068 ceph-mon[38805]: osdmap e73: 4 total, 4 up, 4 in 2026-02-23T13:09:28.466 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:09:28.618 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:09:28.647 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:09:28 trial068 ceph-mon[38805]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 3542/109050 objects degraded (3.248%), 9 pgs degraded) 2026-02-23T13:09:28.647 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:09:28 trial068 ceph-mon[38805]: Cluster is now healthy 2026-02-23T13:09:28.751 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:09:28.750+0000 7f2f9aaad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:09:28.752 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:09:28.750+0000 7f2f9aaad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:09:28.772 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:09:28.773 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:09:28.788 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:09:28.789 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:09:28.799 INFO:teuthology.orchestra.run.trial068.stdout:osd.2: {} 2026-02-23T13:09:28.799 INFO:teuthology.orchestra.run.trial068.stderr:osd.2: osd_enable_op_tracker = '' 2026-02-23T13:09:28.803 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph pg dump --format=json 2026-02-23T13:09:28.814 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:09:28.815 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:09:28.960 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:09:29.083 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:09:29.081+0000 7f285eea6640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:09:29.085 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:09:29.084+0000 7f285eea6640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:09:29.246 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T13:09:29.247 INFO:teuthology.orchestra.run.trial068.stderr:dumped all 2026-02-23T13:09:29.294 INFO:teuthology.orchestra.run.trial068.stdout:{"pg_ready":true,"pg_map":{"version":111,"stamp":"2026-02-23T13:09:28.969896+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1958642232,"num_objects":29881,"num_object_clones":0,"num_object_copies":89643,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":29881,"num_whiteouts":0,"num_read":123,"num_read_kb":229,"num_write":362096,"num_write_kb":2788551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18795,"num_bytes_recovered":1085448760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":161060,"ondisk_log_size":161060,"up":51,"acting":51,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":51,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":2936012800,"kb_used":7084664,"kb_used_data":6072424,"kb_used_omap":102,"kb_used_meta":1012057,"kb_avail":2928928136,"statfs":{"total":3006477107200,"available":2999222411264,"internally_reserved":0,"allocated":6218162176,"data_stored":6216227034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":105198,"internal_metadata":1036346642},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-522649600,"num_objects":-7975,"num_object_clones":0,"num_object_copies":-23925,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-5093,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-7975,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7975,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5290,"num_bytes_recovered":225402904,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.727989"},"pg_stats":[{"pgid":"2.f","version":"71'23031","reported_seq":24610,"reported_epoch":71,"state":"active+clean","last_fresh":"2026-02-23T13:09:25.369295+0000","last_change":"2026-02-23T13:09:17.988736+0000","last_active":"2026-02-23T13:09:25.369295+0000","last_peered":"2026-02-23T13:09:25.369295+0000","last_clean":"2026-02-23T13:09:25.369295+0000","last_became_active":"2026-02-23T13:09:17.980759+0000","last_became_peered":"2026-02-23T13:09:17.980759+0000","last_unstale":"2026-02-23T13:09:25.369295+0000","last_undegraded":"2026-02-23T13:09:25.369295+0000","last_fullsized":"2026-02-23T13:09:25.369295+0000","mapping_epoch":63,"log_start":"34'13000","ondisk_log_start":"34'13000","created":29,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"48'17576","last_scrub_stamp":"2026-02-23T13:08:38.307705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_clean_scrub_stamp":"2026-02-23T13:08:38.307705+0000","objects_scrubbed":2192,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T13:10:04.053022+0000","scrub_duration":368,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":131530752,"num_objects":2007,"num_object_clones":0,"num_object_copies":6021,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2007,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23031,"num_write_kb":178048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":710,"num_bytes_recovered":46530560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.e","version":"72'22258","reported_seq":22706,"reported_epoch":73,"state":"active+clean","last_fresh":"2026-02-23T13:09:26.975677+0000","last_change":"2026-02-23T13:09:23.978699+0000","last_active":"2026-02-23T13:09:26.975677+0000","last_peered":"2026-02-23T13:09:26.975677+0000","last_clean":"2026-02-23T13:09:26.975677+0000","last_became_active":"2026-02-23T13:09:23.972411+0000","last_became_peered":"2026-02-23T13:09:23.972411+0000","last_unstale":"2026-02-23T13:09:26.975677+0000","last_undegraded":"2026-02-23T13:09:26.975677+0000","last_fullsized":"2026-02-23T13:09:26.975677+0000","mapping_epoch":69,"log_start":"33'12200","ondisk_log_start":"33'12200","created":29,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_clean_scrub_stamp":"2026-02-23T13:07:30.224555+0000","objects_scrubbed":0,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":122028032,"num_objects":1862,"num_object_clones":0,"num_object_copies":5586,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1862,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22258,"num_write_kb":171520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":745,"num_bytes_recovered":37814272,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.d","version":"72'22273","reported_seq":24840,"reported_epoch":73,"state":"active+clean","last_fresh":"2026-02-23T13:09:26.975662+0000","last_change":"2026-02-23T13:09:24.982507+0000","last_active":"2026-02-23T13:09:26.975662+0000","last_peered":"2026-02-23T13:09:26.975662+0000","last_clean":"2026-02-23T13:09:26.975662+0000","last_became_active":"2026-02-23T13:09:24.975630+0000","last_became_peered":"2026-02-23T13:09:24.975630+0000","last_unstale":"2026-02-23T13:09:26.975662+0000","last_undegraded":"2026-02-23T13:09:26.975662+0000","last_fullsized":"2026-02-23T13:09:26.975662+0000","mapping_epoch":70,"log_start":"33'12200","ondisk_log_start":"33'12200","created":29,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_clean_scrub_stamp":"2026-02-23T13:07:30.224555+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":124059672,"num_objects":1894,"num_object_clones":0,"num_object_copies":5682,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1894,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":22273,"num_write_kb":171841,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1421,"num_bytes_recovered":82608152,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"72'22960","reported_seq":25198,"reported_epoch":73,"state":"active+clean","last_fresh":"2026-02-23T13:09:26.975499+0000","last_change":"2026-02-23T13:09:25.982940+0000","last_active":"2026-02-23T13:09:26.975499+0000","last_peered":"2026-02-23T13:09:26.975499+0000","last_clean":"2026-02-23T13:09:26.975499+0000","last_became_active":"2026-02-23T13:09:25.976866+0000","last_became_peered":"2026-02-23T13:09:25.976866+0000","last_unstale":"2026-02-23T13:09:26.975499+0000","last_undegraded":"2026-02-23T13:09:26.975499+0000","last_fullsized":"2026-02-23T13:09:26.975499+0000","mapping_epoch":71,"log_start":"34'12900","ondisk_log_start":"34'12900","created":29,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_clean_scrub_stamp":"2026-02-23T13:07:30.224555+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":125566976,"num_objects":1916,"num_object_clones":0,"num_object_copies":5748,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1916,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22960,"num_write_kb":176896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1510,"num_bytes_recovered":85303296,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"28'99","reported_seq":124,"reported_epoch":73,"state":"active+clean","last_fresh":"2026-02-23T13:09:26.976196+0000","last_change":"2026-02-23T13:09:13.973193+0000","last_active":"2026-02-23T13:09:26.976196+0000","last_peered":"2026-02-23T13:09:26.976196+0000","last_clean":"2026-02-23T13:09:26.976196+0000","last_became_active":"2026-02-23T13:09:13.972259+0000","last_became_peered":"2026-02-23T13:09:13.972259+0000","last_unstale":"2026-02-23T13:09:26.976196+0000","last_undegraded":"2026-02-23T13:09:26.976196+0000","last_fullsized":"2026-02-23T13:09:26.976196+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'99","last_scrub_stamp":"2026-02-23T13:08:29.915607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:07:05.619208+0000","last_clean_scrub_stamp":"2026-02-23T13:08:29.915607+0000","objects_scrubbed":0,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T13:09:45.841302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":122,"num_read_kb":228,"num_write":135,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"73'22440","reported_seq":25345,"reported_epoch":73,"state":"active+clean","last_fresh":"2026-02-23T13:09:27.346016+0000","last_change":"2026-02-23T13:09:21.988904+0000","last_active":"2026-02-23T13:09:27.346016+0000","last_peered":"2026-02-23T13:09:27.346016+0000","last_clean":"2026-02-23T13:09:27.346016+0000","last_became_active":"2026-02-23T13:09:21.985593+0000","last_became_peered":"2026-02-23T13:09:21.985593+0000","last_unstale":"2026-02-23T13:09:27.346016+0000","last_undegraded":"2026-02-23T13:09:27.346016+0000","last_fullsized":"2026-02-23T13:09:27.346016+0000","mapping_epoch":67,"log_start":"33'12400","ondisk_log_start":"33'12400","created":29,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_clean_scrub_stamp":"2026-02-23T13:07:30.224555+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":115998720,"num_objects":1770,"num_object_clones":0,"num_object_copies":5310,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1770,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22440,"num_write_kb":172160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1389,"num_bytes_recovered":79233024,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"73'22399","reported_seq":25422,"reported_epoch":73,"state":"active+clean","last_fresh":"2026-02-23T13:09:27.346250+0000","last_change":"2026-02-23T13:09:24.982455+0000","last_active":"2026-02-23T13:09:27.346250+0000","last_peered":"2026-02-23T13:09:27.346250+0000","last_clean":"2026-02-23T13:09:27.346250+0000","last_became_active":"2026-02-23T13:09:24.975473+0000","last_became_peered":"2026-02-23T13:09:24.975473+0000","last_unstale":"2026-02-23T13:09:27.346250+0000","last_undegraded":"2026-02-23T13:09:27.346250+0000","last_fullsized":"2026-02-23T13:09:27.346250+0000","mapping_epoch":70,"log_start":"33'12300","ondisk_log_start":"33'12300","created":29,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_clean_scrub_stamp":"2026-02-23T13:07:30.224555+0000","objects_scrubbed":0,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":118095872,"num_objects":1802,"num_object_clones":0,"num_object_copies":5406,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1802,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22399,"num_write_kb":172096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1449,"num_bytes_recovered":82509824,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"71'22374","reported_seq":25499,"reported_epoch":73,"state":"active+clean","last_fresh":"2026-02-23T13:09:26.983704+0000","last_change":"2026-02-23T13:09:26.983704+0000","last_active":"2026-02-23T13:09:26.983704+0000","last_peered":"2026-02-23T13:09:26.983704+0000","last_clean":"2026-02-23T13:09:26.983704+0000","last_became_active":"2026-02-23T13:09:26.977642+0000","last_became_peered":"2026-02-23T13:09:26.977642+0000","last_unstale":"2026-02-23T13:09:26.983704+0000","last_undegraded":"2026-02-23T13:09:26.983704+0000","last_fullsized":"2026-02-23T13:09:26.983704+0000","mapping_epoch":72,"log_start":"33'12300","ondisk_log_start":"33'12300","created":29,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_clean_scrub_stamp":"2026-02-23T13:07:30.224555+0000","objects_scrubbed":0,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":129171456,"num_objects":1971,"num_object_clones":0,"num_object_copies":5913,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1971,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22374,"num_write_kb":173120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1496,"num_bytes_recovered":85336064,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"73'22580","reported_seq":25626,"reported_epoch":73,"state":"active+clean","last_fresh":"2026-02-23T13:09:27.347195+0000","last_change":"2026-02-23T13:09:24.981707+0000","last_active":"2026-02-23T13:09:27.347195+0000","last_peered":"2026-02-23T13:09:27.347195+0000","last_clean":"2026-02-23T13:09:27.347195+0000","last_became_active":"2026-02-23T13:09:24.975201+0000","last_became_peered":"2026-02-23T13:09:24.975201+0000","last_unstale":"2026-02-23T13:09:27.347195+0000","last_undegraded":"2026-02-23T13:09:27.347195+0000","last_fullsized":"2026-02-23T13:09:27.347195+0000","mapping_epoch":70,"log_start":"34'12500","ondisk_log_start":"34'12500","created":29,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_clean_scrub_stamp":"2026-02-23T13:07:30.224555+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117440512,"num_objects":1792,"num_object_clones":0,"num_object_copies":5376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1792,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22580,"num_write_kb":173312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1461,"num_bytes_recovered":82321408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"72'22401","reported_seq":23070,"reported_epoch":73,"state":"active+clean","last_fresh":"2026-02-23T13:09:26.975617+0000","last_change":"2026-02-23T13:09:22.993742+0000","last_active":"2026-02-23T13:09:26.975617+0000","last_peered":"2026-02-23T13:09:26.975617+0000","last_clean":"2026-02-23T13:09:26.975617+0000","last_became_active":"2026-02-23T13:09:22.987204+0000","last_became_peered":"2026-02-23T13:09:22.987204+0000","last_unstale":"2026-02-23T13:09:26.975617+0000","last_undegraded":"2026-02-23T13:09:26.975617+0000","last_fullsized":"2026-02-23T13:09:26.975617+0000","mapping_epoch":68,"log_start":"33'12400","ondisk_log_start":"33'12400","created":29,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_clean_scrub_stamp":"2026-02-23T13:07:30.224555+0000","objects_scrubbed":0,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":123273216,"num_objects":1881,"num_object_clones":0,"num_object_copies":5643,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1881,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22401,"num_write_kb":172672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":710,"num_bytes_recovered":35536896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"73'22763","reported_seq":24320,"reported_epoch":73,"state":"active+clean","last_fresh":"2026-02-23T13:09:27.337423+0000","last_change":"2026-02-23T13:09:18.710906+0000","last_active":"2026-02-23T13:09:27.337423+0000","last_peered":"2026-02-23T13:09:27.337423+0000","last_clean":"2026-02-23T13:09:27.337423+0000","last_became_active":"2026-02-23T13:09:17.980954+0000","last_became_peered":"2026-02-23T13:09:17.980954+0000","last_unstale":"2026-02-23T13:09:27.337423+0000","last_undegraded":"2026-02-23T13:09:27.337423+0000","last_fullsized":"2026-02-23T13:09:27.337423+0000","mapping_epoch":63,"log_start":"33'12700","ondisk_log_start":"33'12700","created":29,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'22042","last_scrub_stamp":"2026-02-23T13:09:18.710867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_clean_scrub_stamp":"2026-02-23T13:09:18.710867+0000","objects_scrubbed":2519,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T13:10:31.775316+0000","scrub_duration":127,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117833728,"num_objects":1798,"num_object_clones":0,"num_object_copies":5394,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1798,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22763,"num_write_kb":174656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":707,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"72'22418","reported_seq":25236,"reported_epoch":73,"state":"active+clean","last_fresh":"2026-02-23T13:09:26.975678+0000","last_change":"2026-02-23T13:09:24.981552+0000","last_active":"2026-02-23T13:09:26.975678+0000","last_peered":"2026-02-23T13:09:26.975678+0000","last_clean":"2026-02-23T13:09:26.975678+0000","last_became_active":"2026-02-23T13:09:24.975730+0000","last_became_peered":"2026-02-23T13:09:24.975730+0000","last_unstale":"2026-02-23T13:09:26.975678+0000","last_undegraded":"2026-02-23T13:09:26.975678+0000","last_fullsized":"2026-02-23T13:09:26.975678+0000","mapping_epoch":70,"log_start":"33'12400","ondisk_log_start":"33'12400","created":29,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_clean_scrub_stamp":"2026-02-23T13:07:30.224555+0000","objects_scrubbed":0,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":124518400,"num_objects":1900,"num_object_clones":0,"num_object_copies":5700,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1900,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22418,"num_write_kb":172928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1433,"num_bytes_recovered":81829888,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"73'22180","reported_seq":25170,"reported_epoch":73,"state":"active+clean","last_fresh":"2026-02-23T13:09:27.355174+0000","last_change":"2026-02-23T13:09:26.478501+0000","last_active":"2026-02-23T13:09:27.355174+0000","last_peered":"2026-02-23T13:09:27.355174+0000","last_clean":"2026-02-23T13:09:27.355174+0000","last_became_active":"2026-02-23T13:09:21.996628+0000","last_became_peered":"2026-02-23T13:09:21.996628+0000","last_unstale":"2026-02-23T13:09:27.355174+0000","last_undegraded":"2026-02-23T13:09:27.355174+0000","last_fullsized":"2026-02-23T13:09:27.355174+0000","mapping_epoch":67,"log_start":"33'12100","ondisk_log_start":"33'12100","created":29,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'22113","last_scrub_stamp":"2026-02-23T13:09:26.478463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_clean_scrub_stamp":"2026-02-23T13:09:26.478463+0000","objects_scrubbed":1845,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T13:10:47.749749+0000","scrub_duration":92,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":116523008,"num_objects":1778,"num_object_clones":0,"num_object_copies":5334,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1778,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22180,"num_write_kb":170368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1419,"num_bytes_recovered":80683008,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.8","version":"73'22777","reported_seq":23872,"reported_epoch":73,"state":"active+clean","last_fresh":"2026-02-23T13:09:27.337917+0000","last_change":"2026-02-23T13:09:17.986383+0000","last_active":"2026-02-23T13:09:27.337917+0000","last_peered":"2026-02-23T13:09:27.337917+0000","last_clean":"2026-02-23T13:09:27.337917+0000","last_became_active":"2026-02-23T13:09:17.980940+0000","last_became_peered":"2026-02-23T13:09:17.980940+0000","last_unstale":"2026-02-23T13:09:27.337917+0000","last_undegraded":"2026-02-23T13:09:27.337917+0000","last_fullsized":"2026-02-23T13:09:27.337917+0000","mapping_epoch":63,"log_start":"34'12700","ondisk_log_start":"34'12700","created":29,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"49'20400","last_scrub_stamp":"2026-02-23T13:08:49.984469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_clean_scrub_stamp":"2026-02-23T13:08:49.984469+0000","objects_scrubbed":2548,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T13:09:50.946231+0000","scrub_duration":928,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117506048,"num_objects":1793,"num_object_clones":0,"num_object_copies":5379,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1793,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22777,"num_write_kb":174720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":664,"num_bytes_recovered":43491328,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"73'23482","reported_seq":26195,"reported_epoch":73,"state":"active+clean","last_fresh":"2026-02-23T13:09:26.983579+0000","last_change":"2026-02-23T13:09:26.983579+0000","last_active":"2026-02-23T13:09:26.983579+0000","last_peered":"2026-02-23T13:09:26.983579+0000","last_clean":"2026-02-23T13:09:26.983579+0000","last_became_active":"2026-02-23T13:09:26.977816+0000","last_became_peered":"2026-02-23T13:09:26.977816+0000","last_unstale":"2026-02-23T13:09:26.983579+0000","last_undegraded":"2026-02-23T13:09:26.983579+0000","last_fullsized":"2026-02-23T13:09:26.983579+0000","mapping_epoch":72,"log_start":"34'13400","ondisk_log_start":"34'13400","created":29,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_clean_scrub_stamp":"2026-02-23T13:07:30.224555+0000","objects_scrubbed":0,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":132939776,"num_objects":2029,"num_object_clones":0,"num_object_copies":6087,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2029,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23482,"num_write_kb":181408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1517,"num_bytes_recovered":86769664,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.a","version":"72'22679","reported_seq":25704,"reported_epoch":73,"state":"active+clean","last_fresh":"2026-02-23T13:09:26.975626+0000","last_change":"2026-02-23T13:09:23.979296+0000","last_active":"2026-02-23T13:09:26.975626+0000","last_peered":"2026-02-23T13:09:26.975626+0000","last_clean":"2026-02-23T13:09:26.975626+0000","last_became_active":"2026-02-23T13:09:23.973117+0000","last_became_peered":"2026-02-23T13:09:23.973117+0000","last_unstale":"2026-02-23T13:09:26.975626+0000","last_undegraded":"2026-02-23T13:09:26.975626+0000","last_fullsized":"2026-02-23T13:09:26.975626+0000","mapping_epoch":69,"log_start":"33'12600","ondisk_log_start":"33'12600","created":29,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_clean_scrub_stamp":"2026-02-23T13:07:30.224555+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":123338752,"num_objects":1882,"num_object_clones":0,"num_object_copies":5646,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1882,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22679,"num_write_kb":174656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1452,"num_bytes_recovered":82083840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.b","version":"73'22946","reported_seq":24085,"reported_epoch":73,"state":"active+clean","last_fresh":"2026-02-23T13:09:27.350908+0000","last_change":"2026-02-23T13:09:17.986451+0000","last_active":"2026-02-23T13:09:27.350908+0000","last_peered":"2026-02-23T13:09:27.350908+0000","last_clean":"2026-02-23T13:09:27.350908+0000","last_became_active":"2026-02-23T13:09:17.981144+0000","last_became_peered":"2026-02-23T13:09:17.981144+0000","last_unstale":"2026-02-23T13:09:27.350908+0000","last_undegraded":"2026-02-23T13:09:27.350908+0000","last_fullsized":"2026-02-23T13:09:27.350908+0000","mapping_epoch":63,"log_start":"33'12900","ondisk_log_start":"33'12900","created":29,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"48'18824","last_scrub_stamp":"2026-02-23T13:08:42.411326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_clean_scrub_stamp":"2026-02-23T13:08:42.411326+0000","objects_scrubbed":2347,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T13:10:03.835120+0000","scrub_duration":1426,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":118226944,"num_objects":1804,"num_object_clones":0,"num_object_copies":5412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1804,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22946,"num_write_kb":176000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":710,"num_bytes_recovered":46473216,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":1958051864,"num_objects":29879,"num_object_clones":0,"num_object_copies":89637,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":29879,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":361961,"num_write_kb":2786401,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18793,"num_bytes_recovered":1084858392,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":6010286080,"data_stored":6010273864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160961,"ondisk_log_size":160961,"up":48,"acting":48,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":122,"num_read_kb":228,"num_write":135,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":59,"seq":253403070468,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":734003200,"kb_used":1760948,"kb_used_data":1700576,"kb_used_omap":15,"kb_used_meta":60336,"kb_avail":732242252,"statfs":{"total":751619276800,"available":749816066048,"internally_reserved":0,"allocated":1741389824,"data_stored":1740896326,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15828,"internal_metadata":61784620},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411359,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":734003200,"kb_used":1908664,"kb_used_data":1448548,"kb_used_omap":37,"kb_used_meta":460058,"kb_avail":732094536,"statfs":{"total":751619276800,"available":749664804864,"internally_reserved":0,"allocated":1483313152,"data_stored":1482839339,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":38732,"internal_metadata":471099572},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":64,"seq":274877906947,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":734003200,"kb_used":1505904,"kb_used_data":1456412,"kb_used_omap":12,"kb_used_meta":49459,"kb_avail":732497296,"statfs":{"total":751619276800,"available":750077231104,"internally_reserved":0,"allocated":1491365888,"data_stored":1490875942,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12557,"internal_metadata":50646771},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705698,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":734003200,"kb_used":1909148,"kb_used_data":1466888,"kb_used_omap":37,"kb_used_meta":442202,"kb_avail":732094052,"statfs":{"total":751619276800,"available":749664309248,"internally_reserved":0,"allocated":1502093312,"data_stored":1501615427,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":38081,"internal_metadata":452815679},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1444057088,"data_stored":1444053016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1490456576,"data_stored":1490452504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1416298496,"data_stored":1416298496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1659473920,"data_stored":1659469848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-23T13:09:29.295 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2026-02-23T13:09:29.295 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2026-02-23T13:09:29.295 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph pg dump --format=json 2026-02-23T13:09:29.448 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:09:29.501 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:09:29 trial068 ceph-mon[38805]: pgmap v110: 17 pgs: 17 active+clean; 1.8 GiB data, 6.8 GiB used, 2.7 TiB / 2.7 TiB avail; 0 B/s wr, 1.87k op/s; 50 MiB/s, 1.24k objects/s recovering 2026-02-23T13:09:29.502 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:09:29 trial068 ceph-mon[38805]: 2.c scrub starts 2026-02-23T13:09:29.502 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:09:29 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1739044509' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:09:29.502 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:09:29 trial068 ceph-mon[38805]: osdmap e74: 4 total, 4 up, 4 in 2026-02-23T13:09:29.575 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:09:29.574+0000 7fb402a8b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:09:29.575 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:09:29.574+0000 7fb402a8b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:09:29.731 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T13:09:29.731 INFO:teuthology.orchestra.run.trial068.stderr:dumped all 2026-02-23T13:09:29.787 INFO:teuthology.orchestra.run.trial068.stdout:{"pg_ready":true,"pg_map":{"version":111,"stamp":"2026-02-23T13:09:28.969896+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1958642232,"num_objects":29881,"num_object_clones":0,"num_object_copies":89643,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":29881,"num_whiteouts":0,"num_read":123,"num_read_kb":229,"num_write":362096,"num_write_kb":2788551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18795,"num_bytes_recovered":1085448760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":161060,"ondisk_log_size":161060,"up":51,"acting":51,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":51,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":2936012800,"kb_used":7084664,"kb_used_data":6072424,"kb_used_omap":102,"kb_used_meta":1012057,"kb_avail":2928928136,"statfs":{"total":3006477107200,"available":2999222411264,"internally_reserved":0,"allocated":6218162176,"data_stored":6216227034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":105198,"internal_metadata":1036346642},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-522649600,"num_objects":-7975,"num_object_clones":0,"num_object_copies":-23925,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-5093,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-7975,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7975,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5290,"num_bytes_recovered":225402904,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.727989"},"pg_stats":[{"pgid":"2.f","version":"71'23031","reported_seq":24610,"reported_epoch":71,"state":"active+clean","last_fresh":"2026-02-23T13:09:25.369295+0000","last_change":"2026-02-23T13:09:17.988736+0000","last_active":"2026-02-23T13:09:25.369295+0000","last_peered":"2026-02-23T13:09:25.369295+0000","last_clean":"2026-02-23T13:09:25.369295+0000","last_became_active":"2026-02-23T13:09:17.980759+0000","last_became_peered":"2026-02-23T13:09:17.980759+0000","last_unstale":"2026-02-23T13:09:25.369295+0000","last_undegraded":"2026-02-23T13:09:25.369295+0000","last_fullsized":"2026-02-23T13:09:25.369295+0000","mapping_epoch":63,"log_start":"34'13000","ondisk_log_start":"34'13000","created":29,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"48'17576","last_scrub_stamp":"2026-02-23T13:08:38.307705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_clean_scrub_stamp":"2026-02-23T13:08:38.307705+0000","objects_scrubbed":2192,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T13:10:04.053022+0000","scrub_duration":368,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":131530752,"num_objects":2007,"num_object_clones":0,"num_object_copies":6021,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2007,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23031,"num_write_kb":178048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":710,"num_bytes_recovered":46530560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.e","version":"72'22258","reported_seq":22706,"reported_epoch":73,"state":"active+clean","last_fresh":"2026-02-23T13:09:26.975677+0000","last_change":"2026-02-23T13:09:23.978699+0000","last_active":"2026-02-23T13:09:26.975677+0000","last_peered":"2026-02-23T13:09:26.975677+0000","last_clean":"2026-02-23T13:09:26.975677+0000","last_became_active":"2026-02-23T13:09:23.972411+0000","last_became_peered":"2026-02-23T13:09:23.972411+0000","last_unstale":"2026-02-23T13:09:26.975677+0000","last_undegraded":"2026-02-23T13:09:26.975677+0000","last_fullsized":"2026-02-23T13:09:26.975677+0000","mapping_epoch":69,"log_start":"33'12200","ondisk_log_start":"33'12200","created":29,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_clean_scrub_stamp":"2026-02-23T13:07:30.224555+0000","objects_scrubbed":0,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":122028032,"num_objects":1862,"num_object_clones":0,"num_object_copies":5586,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1862,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22258,"num_write_kb":171520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":745,"num_bytes_recovered":37814272,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.d","version":"72'22273","reported_seq":24840,"reported_epoch":73,"state":"active+clean","last_fresh":"2026-02-23T13:09:26.975662+0000","last_change":"2026-02-23T13:09:24.982507+0000","last_active":"2026-02-23T13:09:26.975662+0000","last_peered":"2026-02-23T13:09:26.975662+0000","last_clean":"2026-02-23T13:09:26.975662+0000","last_became_active":"2026-02-23T13:09:24.975630+0000","last_became_peered":"2026-02-23T13:09:24.975630+0000","last_unstale":"2026-02-23T13:09:26.975662+0000","last_undegraded":"2026-02-23T13:09:26.975662+0000","last_fullsized":"2026-02-23T13:09:26.975662+0000","mapping_epoch":70,"log_start":"33'12200","ondisk_log_start":"33'12200","created":29,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_clean_scrub_stamp":"2026-02-23T13:07:30.224555+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":124059672,"num_objects":1894,"num_object_clones":0,"num_object_copies":5682,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1894,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":22273,"num_write_kb":171841,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1421,"num_bytes_recovered":82608152,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"72'22960","reported_seq":25198,"reported_epoch":73,"state":"active+clean","last_fresh":"2026-02-23T13:09:26.975499+0000","last_change":"2026-02-23T13:09:25.982940+0000","last_active":"2026-02-23T13:09:26.975499+0000","last_peered":"2026-02-23T13:09:26.975499+0000","last_clean":"2026-02-23T13:09:26.975499+0000","last_became_active":"2026-02-23T13:09:25.976866+0000","last_became_peered":"2026-02-23T13:09:25.976866+0000","last_unstale":"2026-02-23T13:09:26.975499+0000","last_undegraded":"2026-02-23T13:09:26.975499+0000","last_fullsized":"2026-02-23T13:09:26.975499+0000","mapping_epoch":71,"log_start":"34'12900","ondisk_log_start":"34'12900","created":29,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_clean_scrub_stamp":"2026-02-23T13:07:30.224555+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":125566976,"num_objects":1916,"num_object_clones":0,"num_object_copies":5748,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1916,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22960,"num_write_kb":176896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1510,"num_bytes_recovered":85303296,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"28'99","reported_seq":124,"reported_epoch":73,"state":"active+clean","last_fresh":"2026-02-23T13:09:26.976196+0000","last_change":"2026-02-23T13:09:13.973193+0000","last_active":"2026-02-23T13:09:26.976196+0000","last_peered":"2026-02-23T13:09:26.976196+0000","last_clean":"2026-02-23T13:09:26.976196+0000","last_became_active":"2026-02-23T13:09:13.972259+0000","last_became_peered":"2026-02-23T13:09:13.972259+0000","last_unstale":"2026-02-23T13:09:26.976196+0000","last_undegraded":"2026-02-23T13:09:26.976196+0000","last_fullsized":"2026-02-23T13:09:26.976196+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'99","last_scrub_stamp":"2026-02-23T13:08:29.915607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:07:05.619208+0000","last_clean_scrub_stamp":"2026-02-23T13:08:29.915607+0000","objects_scrubbed":0,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T13:09:45.841302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":122,"num_read_kb":228,"num_write":135,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"73'22440","reported_seq":25345,"reported_epoch":73,"state":"active+clean","last_fresh":"2026-02-23T13:09:27.346016+0000","last_change":"2026-02-23T13:09:21.988904+0000","last_active":"2026-02-23T13:09:27.346016+0000","last_peered":"2026-02-23T13:09:27.346016+0000","last_clean":"2026-02-23T13:09:27.346016+0000","last_became_active":"2026-02-23T13:09:21.985593+0000","last_became_peered":"2026-02-23T13:09:21.985593+0000","last_unstale":"2026-02-23T13:09:27.346016+0000","last_undegraded":"2026-02-23T13:09:27.346016+0000","last_fullsized":"2026-02-23T13:09:27.346016+0000","mapping_epoch":67,"log_start":"33'12400","ondisk_log_start":"33'12400","created":29,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_clean_scrub_stamp":"2026-02-23T13:07:30.224555+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":115998720,"num_objects":1770,"num_object_clones":0,"num_object_copies":5310,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1770,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22440,"num_write_kb":172160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1389,"num_bytes_recovered":79233024,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"73'22399","reported_seq":25422,"reported_epoch":73,"state":"active+clean","last_fresh":"2026-02-23T13:09:27.346250+0000","last_change":"2026-02-23T13:09:24.982455+0000","last_active":"2026-02-23T13:09:27.346250+0000","last_peered":"2026-02-23T13:09:27.346250+0000","last_clean":"2026-02-23T13:09:27.346250+0000","last_became_active":"2026-02-23T13:09:24.975473+0000","last_became_peered":"2026-02-23T13:09:24.975473+0000","last_unstale":"2026-02-23T13:09:27.346250+0000","last_undegraded":"2026-02-23T13:09:27.346250+0000","last_fullsized":"2026-02-23T13:09:27.346250+0000","mapping_epoch":70,"log_start":"33'12300","ondisk_log_start":"33'12300","created":29,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_clean_scrub_stamp":"2026-02-23T13:07:30.224555+0000","objects_scrubbed":0,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":118095872,"num_objects":1802,"num_object_clones":0,"num_object_copies":5406,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1802,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22399,"num_write_kb":172096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1449,"num_bytes_recovered":82509824,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"71'22374","reported_seq":25499,"reported_epoch":73,"state":"active+clean","last_fresh":"2026-02-23T13:09:26.983704+0000","last_change":"2026-02-23T13:09:26.983704+0000","last_active":"2026-02-23T13:09:26.983704+0000","last_peered":"2026-02-23T13:09:26.983704+0000","last_clean":"2026-02-23T13:09:26.983704+0000","last_became_active":"2026-02-23T13:09:26.977642+0000","last_became_peered":"2026-02-23T13:09:26.977642+0000","last_unstale":"2026-02-23T13:09:26.983704+0000","last_undegraded":"2026-02-23T13:09:26.983704+0000","last_fullsized":"2026-02-23T13:09:26.983704+0000","mapping_epoch":72,"log_start":"33'12300","ondisk_log_start":"33'12300","created":29,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_clean_scrub_stamp":"2026-02-23T13:07:30.224555+0000","objects_scrubbed":0,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":129171456,"num_objects":1971,"num_object_clones":0,"num_object_copies":5913,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1971,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22374,"num_write_kb":173120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1496,"num_bytes_recovered":85336064,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"73'22580","reported_seq":25626,"reported_epoch":73,"state":"active+clean","last_fresh":"2026-02-23T13:09:27.347195+0000","last_change":"2026-02-23T13:09:24.981707+0000","last_active":"2026-02-23T13:09:27.347195+0000","last_peered":"2026-02-23T13:09:27.347195+0000","last_clean":"2026-02-23T13:09:27.347195+0000","last_became_active":"2026-02-23T13:09:24.975201+0000","last_became_peered":"2026-02-23T13:09:24.975201+0000","last_unstale":"2026-02-23T13:09:27.347195+0000","last_undegraded":"2026-02-23T13:09:27.347195+0000","last_fullsized":"2026-02-23T13:09:27.347195+0000","mapping_epoch":70,"log_start":"34'12500","ondisk_log_start":"34'12500","created":29,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_clean_scrub_stamp":"2026-02-23T13:07:30.224555+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117440512,"num_objects":1792,"num_object_clones":0,"num_object_copies":5376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1792,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22580,"num_write_kb":173312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1461,"num_bytes_recovered":82321408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"72'22401","reported_seq":23070,"reported_epoch":73,"state":"active+clean","last_fresh":"2026-02-23T13:09:26.975617+0000","last_change":"2026-02-23T13:09:22.993742+0000","last_active":"2026-02-23T13:09:26.975617+0000","last_peered":"2026-02-23T13:09:26.975617+0000","last_clean":"2026-02-23T13:09:26.975617+0000","last_became_active":"2026-02-23T13:09:22.987204+0000","last_became_peered":"2026-02-23T13:09:22.987204+0000","last_unstale":"2026-02-23T13:09:26.975617+0000","last_undegraded":"2026-02-23T13:09:26.975617+0000","last_fullsized":"2026-02-23T13:09:26.975617+0000","mapping_epoch":68,"log_start":"33'12400","ondisk_log_start":"33'12400","created":29,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_clean_scrub_stamp":"2026-02-23T13:07:30.224555+0000","objects_scrubbed":0,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":123273216,"num_objects":1881,"num_object_clones":0,"num_object_copies":5643,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1881,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22401,"num_write_kb":172672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":710,"num_bytes_recovered":35536896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"73'22763","reported_seq":24320,"reported_epoch":73,"state":"active+clean","last_fresh":"2026-02-23T13:09:27.337423+0000","last_change":"2026-02-23T13:09:18.710906+0000","last_active":"2026-02-23T13:09:27.337423+0000","last_peered":"2026-02-23T13:09:27.337423+0000","last_clean":"2026-02-23T13:09:27.337423+0000","last_became_active":"2026-02-23T13:09:17.980954+0000","last_became_peered":"2026-02-23T13:09:17.980954+0000","last_unstale":"2026-02-23T13:09:27.337423+0000","last_undegraded":"2026-02-23T13:09:27.337423+0000","last_fullsized":"2026-02-23T13:09:27.337423+0000","mapping_epoch":63,"log_start":"33'12700","ondisk_log_start":"33'12700","created":29,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'22042","last_scrub_stamp":"2026-02-23T13:09:18.710867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_clean_scrub_stamp":"2026-02-23T13:09:18.710867+0000","objects_scrubbed":2519,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T13:10:31.775316+0000","scrub_duration":127,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117833728,"num_objects":1798,"num_object_clones":0,"num_object_copies":5394,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1798,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22763,"num_write_kb":174656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":707,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"72'22418","reported_seq":25236,"reported_epoch":73,"state":"active+clean","last_fresh":"2026-02-23T13:09:26.975678+0000","last_change":"2026-02-23T13:09:24.981552+0000","last_active":"2026-02-23T13:09:26.975678+0000","last_peered":"2026-02-23T13:09:26.975678+0000","last_clean":"2026-02-23T13:09:26.975678+0000","last_became_active":"2026-02-23T13:09:24.975730+0000","last_became_peered":"2026-02-23T13:09:24.975730+0000","last_unstale":"2026-02-23T13:09:26.975678+0000","last_undegraded":"2026-02-23T13:09:26.975678+0000","last_fullsized":"2026-02-23T13:09:26.975678+0000","mapping_epoch":70,"log_start":"33'12400","ondisk_log_start":"33'12400","created":29,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_clean_scrub_stamp":"2026-02-23T13:07:30.224555+0000","objects_scrubbed":0,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":124518400,"num_objects":1900,"num_object_clones":0,"num_object_copies":5700,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1900,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22418,"num_write_kb":172928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1433,"num_bytes_recovered":81829888,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"73'22180","reported_seq":25170,"reported_epoch":73,"state":"active+clean","last_fresh":"2026-02-23T13:09:27.355174+0000","last_change":"2026-02-23T13:09:26.478501+0000","last_active":"2026-02-23T13:09:27.355174+0000","last_peered":"2026-02-23T13:09:27.355174+0000","last_clean":"2026-02-23T13:09:27.355174+0000","last_became_active":"2026-02-23T13:09:21.996628+0000","last_became_peered":"2026-02-23T13:09:21.996628+0000","last_unstale":"2026-02-23T13:09:27.355174+0000","last_undegraded":"2026-02-23T13:09:27.355174+0000","last_fullsized":"2026-02-23T13:09:27.355174+0000","mapping_epoch":67,"log_start":"33'12100","ondisk_log_start":"33'12100","created":29,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'22113","last_scrub_stamp":"2026-02-23T13:09:26.478463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_clean_scrub_stamp":"2026-02-23T13:09:26.478463+0000","objects_scrubbed":1845,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T13:10:47.749749+0000","scrub_duration":92,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":116523008,"num_objects":1778,"num_object_clones":0,"num_object_copies":5334,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1778,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22180,"num_write_kb":170368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1419,"num_bytes_recovered":80683008,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.8","version":"73'22777","reported_seq":23872,"reported_epoch":73,"state":"active+clean","last_fresh":"2026-02-23T13:09:27.337917+0000","last_change":"2026-02-23T13:09:17.986383+0000","last_active":"2026-02-23T13:09:27.337917+0000","last_peered":"2026-02-23T13:09:27.337917+0000","last_clean":"2026-02-23T13:09:27.337917+0000","last_became_active":"2026-02-23T13:09:17.980940+0000","last_became_peered":"2026-02-23T13:09:17.980940+0000","last_unstale":"2026-02-23T13:09:27.337917+0000","last_undegraded":"2026-02-23T13:09:27.337917+0000","last_fullsized":"2026-02-23T13:09:27.337917+0000","mapping_epoch":63,"log_start":"34'12700","ondisk_log_start":"34'12700","created":29,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"49'20400","last_scrub_stamp":"2026-02-23T13:08:49.984469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_clean_scrub_stamp":"2026-02-23T13:08:49.984469+0000","objects_scrubbed":2548,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T13:09:50.946231+0000","scrub_duration":928,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117506048,"num_objects":1793,"num_object_clones":0,"num_object_copies":5379,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1793,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22777,"num_write_kb":174720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":664,"num_bytes_recovered":43491328,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"73'23482","reported_seq":26195,"reported_epoch":73,"state":"active+clean","last_fresh":"2026-02-23T13:09:26.983579+0000","last_change":"2026-02-23T13:09:26.983579+0000","last_active":"2026-02-23T13:09:26.983579+0000","last_peered":"2026-02-23T13:09:26.983579+0000","last_clean":"2026-02-23T13:09:26.983579+0000","last_became_active":"2026-02-23T13:09:26.977816+0000","last_became_peered":"2026-02-23T13:09:26.977816+0000","last_unstale":"2026-02-23T13:09:26.983579+0000","last_undegraded":"2026-02-23T13:09:26.983579+0000","last_fullsized":"2026-02-23T13:09:26.983579+0000","mapping_epoch":72,"log_start":"34'13400","ondisk_log_start":"34'13400","created":29,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_clean_scrub_stamp":"2026-02-23T13:07:30.224555+0000","objects_scrubbed":0,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":132939776,"num_objects":2029,"num_object_clones":0,"num_object_copies":6087,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2029,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23482,"num_write_kb":181408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1517,"num_bytes_recovered":86769664,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.a","version":"72'22679","reported_seq":25704,"reported_epoch":73,"state":"active+clean","last_fresh":"2026-02-23T13:09:26.975626+0000","last_change":"2026-02-23T13:09:23.979296+0000","last_active":"2026-02-23T13:09:26.975626+0000","last_peered":"2026-02-23T13:09:26.975626+0000","last_clean":"2026-02-23T13:09:26.975626+0000","last_became_active":"2026-02-23T13:09:23.973117+0000","last_became_peered":"2026-02-23T13:09:23.973117+0000","last_unstale":"2026-02-23T13:09:26.975626+0000","last_undegraded":"2026-02-23T13:09:26.975626+0000","last_fullsized":"2026-02-23T13:09:26.975626+0000","mapping_epoch":69,"log_start":"33'12600","ondisk_log_start":"33'12600","created":29,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_clean_scrub_stamp":"2026-02-23T13:07:30.224555+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":123338752,"num_objects":1882,"num_object_clones":0,"num_object_copies":5646,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1882,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22679,"num_write_kb":174656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1452,"num_bytes_recovered":82083840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.b","version":"73'22946","reported_seq":24085,"reported_epoch":73,"state":"active+clean","last_fresh":"2026-02-23T13:09:27.350908+0000","last_change":"2026-02-23T13:09:17.986451+0000","last_active":"2026-02-23T13:09:27.350908+0000","last_peered":"2026-02-23T13:09:27.350908+0000","last_clean":"2026-02-23T13:09:27.350908+0000","last_became_active":"2026-02-23T13:09:17.981144+0000","last_became_peered":"2026-02-23T13:09:17.981144+0000","last_unstale":"2026-02-23T13:09:27.350908+0000","last_undegraded":"2026-02-23T13:09:27.350908+0000","last_fullsized":"2026-02-23T13:09:27.350908+0000","mapping_epoch":63,"log_start":"33'12900","ondisk_log_start":"33'12900","created":29,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"48'18824","last_scrub_stamp":"2026-02-23T13:08:42.411326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_clean_scrub_stamp":"2026-02-23T13:08:42.411326+0000","objects_scrubbed":2347,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T13:10:03.835120+0000","scrub_duration":1426,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":118226944,"num_objects":1804,"num_object_clones":0,"num_object_copies":5412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1804,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22946,"num_write_kb":176000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":710,"num_bytes_recovered":46473216,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":1958051864,"num_objects":29879,"num_object_clones":0,"num_object_copies":89637,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":29879,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":361961,"num_write_kb":2786401,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18793,"num_bytes_recovered":1084858392,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":6010286080,"data_stored":6010273864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160961,"ondisk_log_size":160961,"up":48,"acting":48,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":122,"num_read_kb":228,"num_write":135,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":59,"seq":253403070468,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":734003200,"kb_used":1760948,"kb_used_data":1700576,"kb_used_omap":15,"kb_used_meta":60336,"kb_avail":732242252,"statfs":{"total":751619276800,"available":749816066048,"internally_reserved":0,"allocated":1741389824,"data_stored":1740896326,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15828,"internal_metadata":61784620},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411359,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":734003200,"kb_used":1908664,"kb_used_data":1448548,"kb_used_omap":37,"kb_used_meta":460058,"kb_avail":732094536,"statfs":{"total":751619276800,"available":749664804864,"internally_reserved":0,"allocated":1483313152,"data_stored":1482839339,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":38732,"internal_metadata":471099572},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":64,"seq":274877906947,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":734003200,"kb_used":1505904,"kb_used_data":1456412,"kb_used_omap":12,"kb_used_meta":49459,"kb_avail":732497296,"statfs":{"total":751619276800,"available":750077231104,"internally_reserved":0,"allocated":1491365888,"data_stored":1490875942,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12557,"internal_metadata":50646771},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705698,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":734003200,"kb_used":1909148,"kb_used_data":1466888,"kb_used_omap":37,"kb_used_meta":442202,"kb_avail":732094052,"statfs":{"total":751619276800,"available":749664309248,"internally_reserved":0,"allocated":1502093312,"data_stored":1501615427,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":38081,"internal_metadata":452815679},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1444057088,"data_stored":1444053016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1490456576,"data_stored":1490452504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1416298496,"data_stored":1416298496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1659473920,"data_stored":1659469848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-23T13:09:30.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:09:30 trial068 ceph-mon[38805]: from='client.14497 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2026-02-23T13:09:30.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:09:30 trial068 ceph-mon[38805]: 2.0 scrub starts 2026-02-23T13:09:30.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:09:30 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2026-02-23T13:09:30.873 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:09:31.024 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:09:31.162 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:09:31.160+0000 7f56b1462640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:09:31.162 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:09:31.160+0000 7f56b1462640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:09:31.184 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:09:31.184 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:09:31.202 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:09:31.203 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:09:31.220 INFO:teuthology.orchestra.run.trial068.stdout:osd.2: {} 2026-02-23T13:09:31.221 INFO:teuthology.orchestra.run.trial068.stderr:osd.2: osd_enable_op_tracker = '' 2026-02-23T13:09:31.234 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:09:31.235 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:09:31.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:09:31 trial068 ceph-mon[38805]: from='client.14499 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2026-02-23T13:09:31.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:09:31 trial068 ceph-mon[38805]: 2.0 scrub ok 2026-02-23T13:09:31.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:09:31 trial068 ceph-mon[38805]: pgmap v112: 17 pgs: 17 active+clean; 1.8 GiB data, 6.8 GiB used, 2.7 TiB / 2.7 TiB avail; 0 B/s wr, 1.51k op/s; 41 MiB/s, 1.00k objects/s recovering 2026-02-23T13:09:31.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:09:31 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-23T13:09:31.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:09:31 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2026-02-23T13:09:31.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:09:31 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T13:09:31.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:09:31 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2026-02-23T13:09:31.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:09:31 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3202012285' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:09:32.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:09:32 trial068 ceph-mon[38805]: 2.c scrub ok 2026-02-23T13:09:33.285 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:09:33.440 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:09:33.583 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:09:33.582+0000 7f07825a6640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:09:33.584 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:09:33 trial068 ceph-mon[38805]: pgmap v113: 17 pgs: 17 active+clean; 1.2 GiB data, 5.0 GiB used, 2.7 TiB / 2.7 TiB avail; 0 B/s wr, 2.65k op/s; 23 MiB/s, 640 objects/s recovering 2026-02-23T13:09:33.584 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:09:33 trial068 ceph-mon[38805]: 2.3 scrub starts 2026-02-23T13:09:33.585 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:09:33.584+0000 7f07825a6640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:09:33.602 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:09:33.602 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:09:33.616 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:09:33.617 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:09:33.630 INFO:teuthology.orchestra.run.trial068.stdout:osd.2: {} 2026-02-23T13:09:33.631 INFO:teuthology.orchestra.run.trial068.stderr:osd.2: osd_enable_op_tracker = '' 2026-02-23T13:09:33.647 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:09:33.648 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:09:34.263 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:09:33 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:09:33.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:09:34.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:09:33 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:09:33] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:09:34.788 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 2, 3, 1] 2026-02-23T13:09:34.788 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2026-02-23T13:09:34.789 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph pg dump --format=json 2026-02-23T13:09:34.814 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:09:34 trial068 ceph-mon[38805]: 2.3 scrub ok 2026-02-23T13:09:34.814 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:09:34 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3023443571' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:09:34.947 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:09:35.077 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:09:35.075+0000 7fa6ff671640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:09:35.077 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:09:35.075+0000 7fa6ff671640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:09:35.234 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T13:09:35.234 INFO:teuthology.orchestra.run.trial068.stderr:dumped all 2026-02-23T13:09:35.275 INFO:teuthology.orchestra.run.trial068.stdout:{"pg_ready":true,"pg_map":{"version":114,"stamp":"2026-02-23T13:09:34.243321+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":776176184,"num_objects":11838,"num_object_clones":0,"num_object_copies":35514,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11838,"num_whiteouts":0,"num_read":123,"num_read_kb":229,"num_write":380139,"num_write_kb":2788551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18795,"num_bytes_recovered":1085448760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":161003,"ondisk_log_size":161003,"up":51,"acting":51,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":51,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":2936012800,"kb_used":4164488,"kb_used_data":3043960,"kb_used_omap":108,"kb_used_meta":1120339,"kb_avail":2931848312,"statfs":{"total":3006477107200,"available":3002212671488,"internally_reserved":0,"allocated":3117015040,"data_stored":3114985608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":111062,"internal_metadata":1147227690},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[11,3,9,7,11,11,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-1606418432,"num_objects":-24512,"num_object_clones":0,"num_object_copies":-73536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-3542,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-24512,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":24512,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4014,"num_bytes_recovered":148791320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001125"},"pg_stats":[{"pgid":"2.f","version":"74'23894","reported_seq":25478,"reported_epoch":74,"state":"active+clean","last_fresh":"2026-02-23T13:09:30.410448+0000","last_change":"2026-02-23T13:09:17.988736+0000","last_active":"2026-02-23T13:09:30.410448+0000","last_peered":"2026-02-23T13:09:30.410448+0000","last_clean":"2026-02-23T13:09:30.410448+0000","last_became_active":"2026-02-23T13:09:17.980759+0000","last_became_peered":"2026-02-23T13:09:17.980759+0000","last_unstale":"2026-02-23T13:09:30.410448+0000","last_undegraded":"2026-02-23T13:09:30.410448+0000","last_fullsized":"2026-02-23T13:09:30.410448+0000","mapping_epoch":63,"log_start":"37'13800","ondisk_log_start":"37'13800","created":29,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"48'17576","last_scrub_stamp":"2026-02-23T13:08:38.307705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_clean_scrub_stamp":"2026-02-23T13:08:38.307705+0000","objects_scrubbed":2192,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T13:10:04.053022+0000","scrub_duration":368,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":74973184,"num_objects":1144,"num_object_clones":0,"num_object_copies":3432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1144,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23894,"num_write_kb":178048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":710,"num_bytes_recovered":46530560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.e","version":"74'23375","reported_seq":23810,"reported_epoch":74,"state":"active+clean","last_fresh":"2026-02-23T13:09:31.994270+0000","last_change":"2026-02-23T13:09:23.978699+0000","last_active":"2026-02-23T13:09:31.994270+0000","last_peered":"2026-02-23T13:09:31.994270+0000","last_clean":"2026-02-23T13:09:31.994270+0000","last_became_active":"2026-02-23T13:09:23.972411+0000","last_became_peered":"2026-02-23T13:09:23.972411+0000","last_unstale":"2026-02-23T13:09:31.994270+0000","last_undegraded":"2026-02-23T13:09:31.994270+0000","last_fullsized":"2026-02-23T13:09:31.994270+0000","mapping_epoch":69,"log_start":"37'13300","ondisk_log_start":"37'13300","created":29,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_clean_scrub_stamp":"2026-02-23T13:07:30.224555+0000","objects_scrubbed":0,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48824320,"num_objects":745,"num_object_clones":0,"num_object_copies":2235,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":745,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23375,"num_write_kb":171520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":745,"num_bytes_recovered":37814272,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.d","version":"74'23432","reported_seq":25998,"reported_epoch":74,"state":"active+clean","last_fresh":"2026-02-23T13:09:31.989181+0000","last_change":"2026-02-23T13:09:24.982507+0000","last_active":"2026-02-23T13:09:31.989181+0000","last_peered":"2026-02-23T13:09:31.989181+0000","last_clean":"2026-02-23T13:09:31.989181+0000","last_became_active":"2026-02-23T13:09:24.975630+0000","last_became_peered":"2026-02-23T13:09:24.975630+0000","last_unstale":"2026-02-23T13:09:31.989181+0000","last_undegraded":"2026-02-23T13:09:31.989181+0000","last_fullsized":"2026-02-23T13:09:31.989181+0000","mapping_epoch":70,"log_start":"38'13400","ondisk_log_start":"38'13400","created":29,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_clean_scrub_stamp":"2026-02-23T13:07:30.224555+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48103448,"num_objects":735,"num_object_clones":0,"num_object_copies":2205,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":735,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":23432,"num_write_kb":171841,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1421,"num_bytes_recovered":82608152,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"74'24114","reported_seq":26380,"reported_epoch":74,"state":"active+clean","last_fresh":"2026-02-23T13:09:31.989564+0000","last_change":"2026-02-23T13:09:31.744528+0000","last_active":"2026-02-23T13:09:31.989564+0000","last_peered":"2026-02-23T13:09:31.989564+0000","last_clean":"2026-02-23T13:09:31.989564+0000","last_became_active":"2026-02-23T13:09:25.976866+0000","last_became_peered":"2026-02-23T13:09:25.976866+0000","last_unstale":"2026-02-23T13:09:31.989564+0000","last_undegraded":"2026-02-23T13:09:31.989564+0000","last_fullsized":"2026-02-23T13:09:31.989564+0000","mapping_epoch":71,"log_start":"41'14100","ondisk_log_start":"41'14100","created":29,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'24046","last_scrub_stamp":"2026-02-23T13:09:31.744477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_clean_scrub_stamp":"2026-02-23T13:09:31.744477+0000","objects_scrubbed":1363,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T13:10:50.660450+0000","scrub_duration":3373,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49938432,"num_objects":762,"num_object_clones":0,"num_object_copies":2286,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":762,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":24114,"num_write_kb":176896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1510,"num_bytes_recovered":85303296,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"28'99","reported_seq":126,"reported_epoch":74,"state":"active+clean","last_fresh":"2026-02-23T13:09:28.972882+0000","last_change":"2026-02-23T13:09:13.973193+0000","last_active":"2026-02-23T13:09:28.972882+0000","last_peered":"2026-02-23T13:09:28.972882+0000","last_clean":"2026-02-23T13:09:28.972882+0000","last_became_active":"2026-02-23T13:09:13.972259+0000","last_became_peered":"2026-02-23T13:09:13.972259+0000","last_unstale":"2026-02-23T13:09:28.972882+0000","last_undegraded":"2026-02-23T13:09:28.972882+0000","last_fullsized":"2026-02-23T13:09:28.972882+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'99","last_scrub_stamp":"2026-02-23T13:08:29.915607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:07:05.619208+0000","last_clean_scrub_stamp":"2026-02-23T13:08:29.915607+0000","objects_scrubbed":0,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T13:09:45.841302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":122,"num_read_kb":228,"num_write":135,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"74'23537","reported_seq":26448,"reported_epoch":74,"state":"active+clean+scrubbing","last_fresh":"2026-02-23T13:09:32.372173+0000","last_change":"2026-02-23T13:09:32.342540+0000","last_active":"2026-02-23T13:09:32.372173+0000","last_peered":"2026-02-23T13:09:32.372173+0000","last_clean":"2026-02-23T13:09:32.372173+0000","last_became_active":"2026-02-23T13:09:21.985593+0000","last_became_peered":"2026-02-23T13:09:21.985593+0000","last_unstale":"2026-02-23T13:09:32.372173+0000","last_undegraded":"2026-02-23T13:09:32.372173+0000","last_fullsized":"2026-02-23T13:09:32.372173+0000","mapping_epoch":67,"log_start":"39'13500","ondisk_log_start":"39'13500","created":29,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_clean_scrub_stamp":"2026-02-23T13:07:30.224555+0000","objects_scrubbed":50,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"scrubbing for 1s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44105728,"num_objects":673,"num_object_clones":0,"num_object_copies":2019,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":673,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23537,"num_write_kb":172160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1389,"num_bytes_recovered":79233024,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"74'23542","reported_seq":26572,"reported_epoch":74,"state":"active+clean","last_fresh":"2026-02-23T13:09:32.357121+0000","last_change":"2026-02-23T13:09:30.080896+0000","last_active":"2026-02-23T13:09:32.357121+0000","last_peered":"2026-02-23T13:09:32.357121+0000","last_clean":"2026-02-23T13:09:32.357121+0000","last_became_active":"2026-02-23T13:09:24.975473+0000","last_became_peered":"2026-02-23T13:09:24.975473+0000","last_unstale":"2026-02-23T13:09:32.357121+0000","last_undegraded":"2026-02-23T13:09:32.357121+0000","last_fullsized":"2026-02-23T13:09:32.357121+0000","mapping_epoch":70,"log_start":"39'13500","ondisk_log_start":"39'13500","created":29,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'23022","last_scrub_stamp":"2026-02-23T13:09:30.080866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_clean_scrub_stamp":"2026-02-23T13:09:30.080866+0000","objects_scrubbed":1210,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T13:10:56.504609+0000","scrub_duration":662,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43188224,"num_objects":659,"num_object_clones":0,"num_object_copies":1977,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":659,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23542,"num_write_kb":172096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1449,"num_bytes_recovered":82509824,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"74'23570","reported_seq":26698,"reported_epoch":74,"state":"active+clean","last_fresh":"2026-02-23T13:09:31.986493+0000","last_change":"2026-02-23T13:09:26.983704+0000","last_active":"2026-02-23T13:09:31.986493+0000","last_peered":"2026-02-23T13:09:31.986493+0000","last_clean":"2026-02-23T13:09:31.986493+0000","last_became_active":"2026-02-23T13:09:26.977642+0000","last_became_peered":"2026-02-23T13:09:26.977642+0000","last_unstale":"2026-02-23T13:09:31.986493+0000","last_undegraded":"2026-02-23T13:09:31.986493+0000","last_fullsized":"2026-02-23T13:09:31.986493+0000","mapping_epoch":72,"log_start":"38'13500","ondisk_log_start":"38'13500","created":29,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_clean_scrub_stamp":"2026-02-23T13:07:30.224555+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50790400,"num_objects":775,"num_object_clones":0,"num_object_copies":2325,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":775,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23570,"num_write_kb":173120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1496,"num_bytes_recovered":85336064,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"74'23691","reported_seq":26743,"reported_epoch":74,"state":"active+clean","last_fresh":"2026-02-23T13:09:32.366899+0000","last_change":"2026-02-23T13:09:24.981707+0000","last_active":"2026-02-23T13:09:32.366899+0000","last_peered":"2026-02-23T13:09:32.366899+0000","last_clean":"2026-02-23T13:09:32.366899+0000","last_became_active":"2026-02-23T13:09:24.975201+0000","last_became_peered":"2026-02-23T13:09:24.975201+0000","last_unstale":"2026-02-23T13:09:32.366899+0000","last_undegraded":"2026-02-23T13:09:32.366899+0000","last_fullsized":"2026-02-23T13:09:32.366899+0000","mapping_epoch":70,"log_start":"38'13600","ondisk_log_start":"38'13600","created":29,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_clean_scrub_stamp":"2026-02-23T13:07:30.224555+0000","objects_scrubbed":0,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44630016,"num_objects":681,"num_object_clones":0,"num_object_copies":2043,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":681,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23691,"num_write_kb":173312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1461,"num_bytes_recovered":82321408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"74'23544","reported_seq":24215,"reported_epoch":74,"state":"active+clean","last_fresh":"2026-02-23T13:09:31.989143+0000","last_change":"2026-02-23T13:09:22.993742+0000","last_active":"2026-02-23T13:09:31.989143+0000","last_peered":"2026-02-23T13:09:31.989143+0000","last_clean":"2026-02-23T13:09:31.989143+0000","last_became_active":"2026-02-23T13:09:22.987204+0000","last_became_peered":"2026-02-23T13:09:22.987204+0000","last_unstale":"2026-02-23T13:09:31.989143+0000","last_undegraded":"2026-02-23T13:09:31.989143+0000","last_fullsized":"2026-02-23T13:09:31.989143+0000","mapping_epoch":68,"log_start":"37'13500","ondisk_log_start":"37'13500","created":29,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_clean_scrub_stamp":"2026-02-23T13:07:30.224555+0000","objects_scrubbed":0,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48365568,"num_objects":738,"num_object_clones":0,"num_object_copies":2214,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":738,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23544,"num_write_kb":172672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":710,"num_bytes_recovered":35536896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"74'23891","reported_seq":25450,"reported_epoch":74,"state":"active+clean","last_fresh":"2026-02-23T13:09:32.362385+0000","last_change":"2026-02-23T13:09:18.710906+0000","last_active":"2026-02-23T13:09:32.362385+0000","last_peered":"2026-02-23T13:09:32.362385+0000","last_clean":"2026-02-23T13:09:32.362385+0000","last_became_active":"2026-02-23T13:09:17.980954+0000","last_became_peered":"2026-02-23T13:09:17.980954+0000","last_unstale":"2026-02-23T13:09:32.362385+0000","last_undegraded":"2026-02-23T13:09:32.362385+0000","last_fullsized":"2026-02-23T13:09:32.362385+0000","mapping_epoch":63,"log_start":"37'13800","ondisk_log_start":"37'13800","created":29,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'22042","last_scrub_stamp":"2026-02-23T13:09:18.710867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_clean_scrub_stamp":"2026-02-23T13:09:18.710867+0000","objects_scrubbed":2519,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T13:10:31.775316+0000","scrub_duration":127,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43909120,"num_objects":670,"num_object_clones":0,"num_object_copies":2010,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":670,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23891,"num_write_kb":174656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":707,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"74'23584","reported_seq":26397,"reported_epoch":74,"state":"active+clean","last_fresh":"2026-02-23T13:09:31.995764+0000","last_change":"2026-02-23T13:09:24.981552+0000","last_active":"2026-02-23T13:09:31.995764+0000","last_peered":"2026-02-23T13:09:31.995764+0000","last_clean":"2026-02-23T13:09:31.995764+0000","last_became_active":"2026-02-23T13:09:24.975730+0000","last_became_peered":"2026-02-23T13:09:24.975730+0000","last_unstale":"2026-02-23T13:09:31.995764+0000","last_undegraded":"2026-02-23T13:09:31.995764+0000","last_fullsized":"2026-02-23T13:09:31.995764+0000","mapping_epoch":70,"log_start":"37'13500","ondisk_log_start":"37'13500","created":29,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_clean_scrub_stamp":"2026-02-23T13:07:30.224555+0000","objects_scrubbed":0,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48103424,"num_objects":734,"num_object_clones":0,"num_object_copies":2202,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":734,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23584,"num_write_kb":172928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1433,"num_bytes_recovered":81829888,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"74'23315","reported_seq":26304,"reported_epoch":74,"state":"active+clean","last_fresh":"2026-02-23T13:09:32.371266+0000","last_change":"2026-02-23T13:09:26.478501+0000","last_active":"2026-02-23T13:09:32.371266+0000","last_peered":"2026-02-23T13:09:32.371266+0000","last_clean":"2026-02-23T13:09:32.371266+0000","last_became_active":"2026-02-23T13:09:21.996628+0000","last_became_peered":"2026-02-23T13:09:21.996628+0000","last_unstale":"2026-02-23T13:09:32.371266+0000","last_undegraded":"2026-02-23T13:09:32.371266+0000","last_fullsized":"2026-02-23T13:09:32.371266+0000","mapping_epoch":67,"log_start":"37'13300","ondisk_log_start":"37'13300","created":29,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'22113","last_scrub_stamp":"2026-02-23T13:09:26.478463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_clean_scrub_stamp":"2026-02-23T13:09:26.478463+0000","objects_scrubbed":1845,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T13:10:47.749749+0000","scrub_duration":92,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42139648,"num_objects":643,"num_object_clones":0,"num_object_copies":1929,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":643,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23315,"num_write_kb":170368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1419,"num_bytes_recovered":80683008,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.8","version":"74'23893","reported_seq":24980,"reported_epoch":74,"state":"active+clean","last_fresh":"2026-02-23T13:09:32.363161+0000","last_change":"2026-02-23T13:09:17.986383+0000","last_active":"2026-02-23T13:09:32.363161+0000","last_peered":"2026-02-23T13:09:32.363161+0000","last_clean":"2026-02-23T13:09:32.363161+0000","last_became_active":"2026-02-23T13:09:17.980940+0000","last_became_peered":"2026-02-23T13:09:17.980940+0000","last_unstale":"2026-02-23T13:09:32.363161+0000","last_undegraded":"2026-02-23T13:09:32.363161+0000","last_fullsized":"2026-02-23T13:09:32.363161+0000","mapping_epoch":63,"log_start":"40'13800","ondisk_log_start":"40'13800","created":29,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"49'20400","last_scrub_stamp":"2026-02-23T13:08:49.984469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_clean_scrub_stamp":"2026-02-23T13:08:49.984469+0000","objects_scrubbed":2548,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T13:09:50.946231+0000","scrub_duration":928,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44367872,"num_objects":677,"num_object_clones":0,"num_object_copies":2031,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":677,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23893,"num_write_kb":174720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":664,"num_bytes_recovered":43491328,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"74'24693","reported_seq":27410,"reported_epoch":74,"state":"active+clean","last_fresh":"2026-02-23T13:09:31.978271+0000","last_change":"2026-02-23T13:09:26.983579+0000","last_active":"2026-02-23T13:09:31.978271+0000","last_peered":"2026-02-23T13:09:31.978271+0000","last_clean":"2026-02-23T13:09:31.978271+0000","last_became_active":"2026-02-23T13:09:26.977816+0000","last_became_peered":"2026-02-23T13:09:26.977816+0000","last_unstale":"2026-02-23T13:09:31.978271+0000","last_undegraded":"2026-02-23T13:09:31.978271+0000","last_fullsized":"2026-02-23T13:09:31.978271+0000","mapping_epoch":72,"log_start":"45'14600","ondisk_log_start":"45'14600","created":29,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_clean_scrub_stamp":"2026-02-23T13:07:30.224555+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53575680,"num_objects":818,"num_object_clones":0,"num_object_copies":2454,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":818,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":24693,"num_write_kb":181408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1517,"num_bytes_recovered":86769664,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.a","version":"74'23815","reported_seq":26840,"reported_epoch":74,"state":"active+clean","last_fresh":"2026-02-23T13:09:31.959030+0000","last_change":"2026-02-23T13:09:23.979296+0000","last_active":"2026-02-23T13:09:31.959030+0000","last_peered":"2026-02-23T13:09:31.959030+0000","last_clean":"2026-02-23T13:09:31.959030+0000","last_became_active":"2026-02-23T13:09:23.973117+0000","last_became_peered":"2026-02-23T13:09:23.973117+0000","last_unstale":"2026-02-23T13:09:31.959030+0000","last_undegraded":"2026-02-23T13:09:31.959030+0000","last_fullsized":"2026-02-23T13:09:31.959030+0000","mapping_epoch":69,"log_start":"37'13800","ondisk_log_start":"37'13800","created":29,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_clean_scrub_stamp":"2026-02-23T13:07:30.224555+0000","objects_scrubbed":0,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48889856,"num_objects":746,"num_object_clones":0,"num_object_copies":2238,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":746,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23815,"num_write_kb":174656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1452,"num_bytes_recovered":82083840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.b","version":"74'24114","reported_seq":25257,"reported_epoch":74,"state":"active+clean","last_fresh":"2026-02-23T13:09:32.358483+0000","last_change":"2026-02-23T13:09:17.986451+0000","last_active":"2026-02-23T13:09:32.358483+0000","last_peered":"2026-02-23T13:09:32.358483+0000","last_clean":"2026-02-23T13:09:32.358483+0000","last_became_active":"2026-02-23T13:09:17.981144+0000","last_became_peered":"2026-02-23T13:09:17.981144+0000","last_unstale":"2026-02-23T13:09:32.358483+0000","last_undegraded":"2026-02-23T13:09:32.358483+0000","last_fullsized":"2026-02-23T13:09:32.358483+0000","mapping_epoch":63,"log_start":"38'14100","ondisk_log_start":"38'14100","created":29,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"48'18824","last_scrub_stamp":"2026-02-23T13:08:42.411326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:07:30.224555+0000","last_clean_scrub_stamp":"2026-02-23T13:08:42.411326+0000","objects_scrubbed":2347,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T13:10:03.835120+0000","scrub_duration":1426,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41680896,"num_objects":636,"num_object_clones":0,"num_object_copies":1908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":24114,"num_write_kb":176000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":710,"num_bytes_recovered":46473216,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":775585816,"num_objects":11836,"num_object_clones":0,"num_object_copies":35508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11836,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":380004,"num_write_kb":2786401,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18793,"num_bytes_recovered":1084858392,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2827071488,"data_stored":2827059272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160904,"ondisk_log_size":160904,"up":48,"acting":48,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":122,"num_read_kb":228,"num_write":135,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":59,"seq":253403070469,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":734003200,"kb_used":687948,"kb_used_data":609528,"kb_used_omap":16,"kb_used_meta":78383,"kb_avail":733315252,"statfs":{"total":751619276800,"available":750914818048,"internally_reserved":0,"allocated":624156672,"data_stored":623647408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17132,"internal_metadata":80264468},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411360,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":734003200,"kb_used":1347688,"kb_used_data":869524,"kb_used_omap":40,"kb_used_meta":478103,"kb_avail":732655512,"statfs":{"total":751619276800,"available":750239244288,"internally_reserved":0,"allocated":890392576,"data_stored":889887384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":41336,"internal_metadata":489578120},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[9,0,4,1,5],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":64,"seq":274877906948,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":734003200,"kb_used":778056,"kb_used_data":692468,"kb_used_omap":14,"kb_used_meta":85553,"kb_avail":733225144,"statfs":{"total":751619276800,"available":750822547456,"internally_reserved":0,"allocated":709087232,"data_stored":708581520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14513,"internal_metadata":87607119},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705699,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":734003200,"kb_used":1350796,"kb_used_data":872440,"kb_used_omap":37,"kb_used_meta":478298,"kb_avail":732652404,"statfs":{"total":751619276800,"available":750236061696,"internally_reserved":0,"allocated":893378560,"data_stored":892869296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":38081,"internal_metadata":489777983},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,3,5,6,6,11,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":817008640,"data_stored":817004568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":591368192,"data_stored":591364120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":802291712,"data_stored":802291712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":616402944,"data_stored":616398872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-23T13:09:35.294 INFO:tasks.radosbench.radosbench.0.trial068.stdout:Removed 43538 objects 2026-02-23T13:09:35.294 INFO:tasks.radosbench.radosbench.0.trial068.stdout:Clean up completed and total clean up time :19.9577 2026-02-23T13:09:35.703 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:09:35.727 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:09:35 trial068 ceph-mon[38805]: pgmap v114: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 740 MiB data, 4.0 GiB used, 2.7 TiB / 2.7 TiB avail; 0 B/s wr, 3.06k op/s; 18 MiB/s, 501 objects/s recovering 2026-02-23T13:09:35.727 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:09:35 trial068 ceph-mon[38805]: 2.2 scrub starts 2026-02-23T13:09:35.820 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:09:35.916 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:09:35.914+0000 7f6e012b3640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:09:35.916 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:09:35.914+0000 7f6e012b3640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:09:35.928 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:09:35.928 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:09:35.938 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:09:35.938 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:09:35.949 INFO:teuthology.orchestra.run.trial068.stdout:osd.2: {} 2026-02-23T13:09:35.949 INFO:teuthology.orchestra.run.trial068.stderr:osd.2: osd_enable_op_tracker = '' 2026-02-23T13:09:35.959 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:09:35.959 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:09:36.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:09:36 trial068 ceph-mon[38805]: 2.2 scrub ok 2026-02-23T13:09:36.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:09:36 trial068 ceph-mon[38805]: from='client.14505 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2026-02-23T13:09:36.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:09:36 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1930423274' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:09:37.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:09:37 trial068 ceph-mon[38805]: pgmap v115: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 669 MiB data, 2.4 GiB used, 2.7 TiB / 2.7 TiB avail; 0 B/s wr, 2.77k op/s; 15 MiB/s, 432 objects/s recovering 2026-02-23T13:09:37.995 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:09:38.113 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:09:38.209 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:09:38.207+0000 7f1ef80c0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:09:38.209 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:09:38.208+0000 7f1ef80c0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:09:38.221 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:09:38.221 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:09:38.232 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:09:38.232 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:09:38.243 INFO:teuthology.orchestra.run.trial068.stdout:osd.2: {} 2026-02-23T13:09:38.243 INFO:teuthology.orchestra.run.trial068.stderr:osd.2: osd_enable_op_tracker = '' 2026-02-23T13:09:38.253 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:09:38.253 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:09:38.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:09:38 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T13:09:38.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:09:38 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T13:09:38.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:09:38 trial068 ceph-mon[38805]: 2.4 scrub starts 2026-02-23T13:09:38.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:09:38 trial068 ceph-mon[38805]: 2.4 scrub ok 2026-02-23T13:09:38.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:09:38 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1640519563' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:09:39.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:09:39 trial068 ceph-mon[38805]: pgmap v116: 17 pgs: 17 active+clean; 577 KiB data, 1.2 GiB used, 2.7 TiB / 2.7 TiB avail; 0 B/s wr, 2.99k op/s 2026-02-23T13:09:39.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:09:39 trial068 ceph-mon[38805]: 2.6 scrub starts 2026-02-23T13:09:39.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:09:39 trial068 ceph-mon[38805]: 2.6 scrub ok 2026-02-23T13:09:40.278 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 2, 3, 1] 2026-02-23T13:09:40.279 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2026-02-23T13:09:40.279 INFO:tasks.thrashosds.thrasher:Killing osd 3, live_osds are [0, 2, 3, 1] 2026-02-23T13:09:40.279 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2026-02-23T13:09:40.279 DEBUG:teuthology.orchestra.run.trial185:> sudo systemctl stop ceph-5f823084-10b8-11f1-be80-d404e6e7d460@osd.3 2026-02-23T13:09:40.290 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:09:40.399 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_0 2026-02-23T13:09:40.399 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd pool rm unique_pool_0 unique_pool_0 --yes-i-really-really-mean-it 2026-02-23T13:09:40.406 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:09:40.512 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:09:40.514 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:09:40.513+0000 7fe502ac7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:09:40.514 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:09:40.513+0000 7fe502ac7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:09:40.527 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:09:40.528 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:09:40.538 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:09:40.538 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:09:40.549 INFO:teuthology.orchestra.run.trial068.stdout:osd.2: {} 2026-02-23T13:09:40.550 INFO:teuthology.orchestra.run.trial068.stderr:osd.2: osd_enable_op_tracker = '' 2026-02-23T13:09:40.560 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:09:40.560 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:09:40.606 INFO:journalctl@ceph.osd.3.trial185.stdout:Feb 23 13:09:40 trial185 systemd[1]: Stopping Ceph osd.3 for 5f823084-10b8-11f1-be80-d404e6e7d460... 2026-02-23T13:09:40.607 INFO:journalctl@ceph.osd.3.trial185.stdout:Feb 23 13:09:40 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-3[67985]: 2026-02-23T13:09:40.384+0000 7fe5ce46e640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --osd-objectstore=bluestore (PID: 1) UID: 0 2026-02-23T13:09:40.607 INFO:journalctl@ceph.osd.3.trial185.stdout:Feb 23 13:09:40 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-3[67985]: 2026-02-23T13:09:40.384+0000 7fe5ce46e640 -1 osd.3 74 *** Got signal Terminated *** 2026-02-23T13:09:40.607 INFO:journalctl@ceph.osd.3.trial185.stdout:Feb 23 13:09:40 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-3[67985]: 2026-02-23T13:09:40.384+0000 7fe5ce46e640 -1 osd.3 74 *** Immediate shutdown (osd_fast_shutdown=true) *** 2026-02-23T13:09:40.615 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:09:40.614+0000 7f8c348bb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:09:40.616 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:09:40.614+0000 7f8c348bb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:09:41.363 INFO:teuthology.orchestra.run.trial068.stderr:pool 'unique_pool_0' removed 2026-02-23T13:09:41.405 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2026-02-23T13:09:41.406 INFO:tasks.radosbench:Beginning radosbench... 2026-02-23T13:09:41.406 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_1 2026-02-23T13:09:41.406 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd pool create unique_pool_1 16 2026-02-23T13:09:41.570 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:09:41.610 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:09:41 trial068 ceph-mon[38805]: pgmap v117: 17 pgs: 17 active+clean; 577 KiB data, 1.2 GiB used, 2.7 TiB / 2.7 TiB avail; 0 B/s wr, 2.65k op/s 2026-02-23T13:09:41.610 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:09:41 trial068 ceph-mon[38805]: osd.3 marked itself down and dead 2026-02-23T13:09:41.610 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:09:41 trial068 ceph-mon[38805]: 2.a scrub starts 2026-02-23T13:09:41.610 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:09:41 trial068 ceph-mon[38805]: 2.a scrub ok 2026-02-23T13:09:41.611 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:09:41 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1108137095' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:09:41.611 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:09:41 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/123753460' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true} : dispatch 2026-02-23T13:09:41.669 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:09:41.668+0000 7fa9d807d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:09:41.669 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:09:41.668+0000 7fa9d807d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:09:41.774 INFO:journalctl@ceph.osd.3.trial185.stdout:Feb 23 13:09:41 trial185 podman[68836]: 2026-02-23 13:09:41.48789607 +0000 UTC m=+1.116760214 container died abc20c2683afe7a2d79ae87bcfc64ec8ad3ffa113911de8123e77c586f268c8b (image=quay.ceph.io/ceph-ci/ceph@sha256:49621faaa845c39b2aae72e4fa24123ee9b2ac3fb8ae834753907068ee2972ea, name=ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-3, org.opencontainers.image.documentation=https://docs.ceph.com/, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, FROM_IMAGE=quay.io/centos/centos:stream9, CEPH_GIT_REPO=https://github.com/ceph/ceph-ci.git, org.label-schema.license=GPLv2, io.buildah.version=1.41.3, OSD_FLAVOR=default, ceph=True, org.label-schema.build-date=20260216, org.label-schema.vendor=CentOS, CEPH_SHA1=e371433fcd1390a0b82cfb18effc422c1f5d8c1d, CEPH_REF=wip-shraddhaag-enable-cephadm-tests, GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfsganesha-5/, org.opencontainers.image.authors=Ceph Release Team ) 2026-02-23T13:09:41.775 INFO:journalctl@ceph.osd.3.trial185.stdout:Feb 23 13:09:41 trial185 podman[68836]: 2026-02-23 13:09:41.496078354 +0000 UTC m=+1.124942509 container remove abc20c2683afe7a2d79ae87bcfc64ec8ad3ffa113911de8123e77c586f268c8b (image=quay.ceph.io/ceph-ci/ceph@sha256:49621faaa845c39b2aae72e4fa24123ee9b2ac3fb8ae834753907068ee2972ea, name=ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-3, org.label-schema.schema-version=1.0, CEPH_GIT_REPO=https://github.com/ceph/ceph-ci.git, org.label-schema.build-date=20260216, FROM_IMAGE=quay.io/centos/centos:stream9, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, org.opencontainers.image.authors=Ceph Release Team , io.buildah.version=1.41.3, CEPH_SHA1=e371433fcd1390a0b82cfb18effc422c1f5d8c1d, CEPH_REF=wip-shraddhaag-enable-cephadm-tests, org.opencontainers.image.documentation=https://docs.ceph.com/, OSD_FLAVOR=default, org.label-schema.name=CentOS Stream 9 Base Image, GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfsganesha-5/) 2026-02-23T13:09:41.775 INFO:journalctl@ceph.osd.3.trial185.stdout:Feb 23 13:09:41 trial185 bash[68836]: ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-3 2026-02-23T13:09:41.775 INFO:journalctl@ceph.osd.3.trial185.stdout:Feb 23 13:09:41 trial185 podman[68971]: 2026-02-23 13:09:41.62581675 +0000 UTC m=+0.015915002 container create 9862f33ab6c094ccd53085fe4b956760d66f52cea0ebefb5e36b29278d6d412e (image=quay.ceph.io/ceph-ci/ceph@sha256:49621faaa845c39b2aae72e4fa24123ee9b2ac3fb8ae834753907068ee2972ea, name=ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-3-deactivate, org.label-schema.schema-version=1.0, org.opencontainers.image.authors=Ceph Release Team , org.label-schema.name=CentOS Stream 9 Base Image, CEPH_REF=wip-shraddhaag-enable-cephadm-tests, org.label-schema.build-date=20260216, org.opencontainers.image.documentation=https://docs.ceph.com/, OSD_FLAVOR=default, io.buildah.version=1.41.3, ceph=True, org.label-schema.license=GPLv2, FROM_IMAGE=quay.io/centos/centos:stream9, GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfsganesha-5/, CEPH_GIT_REPO=https://github.com/ceph/ceph-ci.git, org.label-schema.vendor=CentOS, CEPH_SHA1=e371433fcd1390a0b82cfb18effc422c1f5d8c1d) 2026-02-23T13:09:41.775 INFO:journalctl@ceph.osd.3.trial185.stdout:Feb 23 13:09:41 trial185 podman[68971]: 2026-02-23 13:09:41.656446091 +0000 UTC m=+0.046544273 container init 9862f33ab6c094ccd53085fe4b956760d66f52cea0ebefb5e36b29278d6d412e (image=quay.ceph.io/ceph-ci/ceph@sha256:49621faaa845c39b2aae72e4fa24123ee9b2ac3fb8ae834753907068ee2972ea, name=ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-3-deactivate, ceph=True, FROM_IMAGE=quay.io/centos/centos:stream9, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_REF=wip-shraddhaag-enable-cephadm-tests, CEPH_GIT_REPO=https://github.com/ceph/ceph-ci.git, CEPH_SHA1=e371433fcd1390a0b82cfb18effc422c1f5d8c1d, org.opencontainers.image.documentation=https://docs.ceph.com/, io.buildah.version=1.41.3, GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfsganesha-5/, org.label-schema.build-date=20260216, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, OSD_FLAVOR=default, org.label-schema.license=GPLv2, org.opencontainers.image.authors=Ceph Release Team ) 2026-02-23T13:09:41.775 INFO:journalctl@ceph.osd.3.trial185.stdout:Feb 23 13:09:41 trial185 podman[68971]: 2026-02-23 13:09:41.658878698 +0000 UTC m=+0.048976880 container start 9862f33ab6c094ccd53085fe4b956760d66f52cea0ebefb5e36b29278d6d412e (image=quay.ceph.io/ceph-ci/ceph@sha256:49621faaa845c39b2aae72e4fa24123ee9b2ac3fb8ae834753907068ee2972ea, name=ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-3-deactivate, org.opencontainers.image.documentation=https://docs.ceph.com/, org.opencontainers.image.authors=Ceph Release Team , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20260216, CEPH_SHA1=e371433fcd1390a0b82cfb18effc422c1f5d8c1d, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, FROM_IMAGE=quay.io/centos/centos:stream9, CEPH_REF=wip-shraddhaag-enable-cephadm-tests, CEPH_GIT_REPO=https://github.com/ceph/ceph-ci.git, org.label-schema.vendor=CentOS, OSD_FLAVOR=default, GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfsganesha-5/, io.buildah.version=1.41.3, ceph=True) 2026-02-23T13:09:41.775 INFO:journalctl@ceph.osd.3.trial185.stdout:Feb 23 13:09:41 trial185 podman[68971]: 2026-02-23 13:09:41.659462247 +0000 UTC m=+0.049560429 container attach 9862f33ab6c094ccd53085fe4b956760d66f52cea0ebefb5e36b29278d6d412e (image=quay.ceph.io/ceph-ci/ceph@sha256:49621faaa845c39b2aae72e4fa24123ee9b2ac3fb8ae834753907068ee2972ea, name=ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-3-deactivate, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, org.opencontainers.image.documentation=https://docs.ceph.com/, io.buildah.version=1.41.3, CEPH_SHA1=e371433fcd1390a0b82cfb18effc422c1f5d8c1d, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.opencontainers.image.authors=Ceph Release Team , FROM_IMAGE=quay.io/centos/centos:stream9, GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfsganesha-5/, ceph=True, OSD_FLAVOR=default, CEPH_REF=wip-shraddhaag-enable-cephadm-tests, org.label-schema.build-date=20260216, CEPH_GIT_REPO=https://github.com/ceph/ceph-ci.git) 2026-02-23T13:09:41.775 INFO:journalctl@ceph.osd.3.trial185.stdout:Feb 23 13:09:41 trial185 podman[68971]: 2026-02-23 13:09:41.618915886 +0000 UTC m=+0.009014068 image pull 63df4b83db6f95a8ba965919b3671cb27173117c23b5c950aaadc8913f56213c quay.ceph.io/ceph-ci/ceph@sha256:49621faaa845c39b2aae72e4fa24123ee9b2ac3fb8ae834753907068ee2972ea 2026-02-23T13:09:41.775 INFO:journalctl@ceph.osd.3.trial185.stdout:Feb 23 13:09:41 trial185 podman[68971]: 2026-02-23 13:09:41.773195644 +0000 UTC m=+0.163293826 container died 9862f33ab6c094ccd53085fe4b956760d66f52cea0ebefb5e36b29278d6d412e (image=quay.ceph.io/ceph-ci/ceph@sha256:49621faaa845c39b2aae72e4fa24123ee9b2ac3fb8ae834753907068ee2972ea, name=ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-3-deactivate, org.label-schema.license=GPLv2, org.opencontainers.image.authors=Ceph Release Team , CEPH_GIT_REPO=https://github.com/ceph/ceph-ci.git, io.buildah.version=1.41.3, ceph=True, CEPH_SHA1=e371433fcd1390a0b82cfb18effc422c1f5d8c1d, org.label-schema.schema-version=1.0, OSD_FLAVOR=default, FROM_IMAGE=quay.io/centos/centos:stream9, org.label-schema.build-date=20260216, org.label-schema.vendor=CentOS, GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfsganesha-5/, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_REF=wip-shraddhaag-enable-cephadm-tests, org.opencontainers.image.documentation=https://docs.ceph.com/) 2026-02-23T13:09:41.814 DEBUG:teuthology.orchestra.run.trial185:> sudo pkill -f 'journalctl -f -n 0 -u ceph-5f823084-10b8-11f1-be80-d404e6e7d460@osd.3.service' 2026-02-23T13:09:41.856 DEBUG:teuthology.orchestra.run:got remote process result: None 2026-02-23T13:09:41.856 INFO:tasks.cephadm.osd.3:Stopped osd.3 2026-02-23T13:09:42.365 INFO:teuthology.orchestra.run.trial068.stderr:pool 'unique_pool_1' created 2026-02-23T13:09:42.404 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd pool application enable unique_pool_1 rados --yes-i-really-mean-it || true 2026-02-23T13:09:42.520 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:09:42.596 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:09:42.620 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:09:42.618+0000 7fbaf35fb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:09:42.620 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:09:42.618+0000 7fbaf35fb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:09:42.620 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:09:42 trial068 ceph-mon[38805]: Health check failed: 1 osds down (OSD_DOWN) 2026-02-23T13:09:42.621 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:09:42 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/123753460' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]': finished 2026-02-23T13:09:42.621 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:09:42 trial068 ceph-mon[38805]: osdmap e75: 4 total, 3 up, 4 in 2026-02-23T13:09:42.621 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:09:42 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1010247689' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16} : dispatch 2026-02-23T13:09:42.714 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:09:42.814 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:09:42.813+0000 7fc10437b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:09:42.815 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:09:42.813+0000 7fc10437b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:09:42.827 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:09:42.827 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:09:42.837 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:09:42.838 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:09:42.849 INFO:teuthology.orchestra.run.trial068.stdout:osd.2: {} 2026-02-23T13:09:42.850 INFO:teuthology.orchestra.run.trial068.stderr:osd.2: osd_enable_op_tracker = '' 2026-02-23T13:09:42.850 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.3 2026-02-23T13:09:42.880 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:09:42.881 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:09:43.367 INFO:teuthology.orchestra.run.trial068.stderr:enabled application 'rados' on pool 'unique_pool_1' 2026-02-23T13:09:43.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:09:43 trial068 ceph-mon[38805]: pgmap v119: 1 pgs: 1 stale+active+clean; 577 KiB data, 1.1 GiB used, 2.7 TiB / 2.7 TiB avail 2026-02-23T13:09:43.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:09:43 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1010247689' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]': finished 2026-02-23T13:09:43.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:09:43 trial068 ceph-mon[38805]: osdmap e76: 4 total, 3 up, 4 in 2026-02-23T13:09:43.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:09:43 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1880211740' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2026-02-23T13:09:43.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:09:43 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/215124422' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:09:44.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:09:43 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:09:43] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:09:44.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:09:43 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:09:43.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:09:44.408 DEBUG:teuthology.orchestra.run.trial068:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -t 128 -b 8192 --object-size 65536 -p unique_pool_1 bench 90 write' 2026-02-23T13:09:44.410 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2026-02-23T13:09:44.411 INFO:teuthology.orchestra.run:waiting for 3000 2026-02-23T13:09:44.457 INFO:tasks.radosbench.radosbench.0.trial068.stdout:hints = 1 2026-02-23T13:09:44.457 INFO:tasks.radosbench.radosbench.0.trial068.stdout:Maintaining 128 concurrent writes of 8192 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2026-02-23T13:09:44.457 INFO:tasks.radosbench.radosbench.0.trial068.stdout:Object prefix: benchmark_data_trial068_89165 2026-02-23T13:09:44.458 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T13:09:44.458 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 0 0 0 0 0 0 - 0 2026-02-23T13:09:44.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:09:44 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1880211740' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]': finished 2026-02-23T13:09:44.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:09:44 trial068 ceph-mon[38805]: osdmap e77: 4 total, 3 up, 4 in 2026-02-23T13:09:44.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:09:44 trial068 ceph-mon[38805]: osdmap e78: 4 total, 3 up, 4 in 2026-02-23T13:09:44.882 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:09:45.033 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:09:45.159 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:09:45.158+0000 7f771ceb0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:09:45.160 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:09:45.159+0000 7f771ceb0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:09:45.176 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:09:45.176 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:09:45.189 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:09:45.189 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:09:45.205 INFO:teuthology.orchestra.run.trial068.stdout:osd.2: {} 2026-02-23T13:09:45.205 INFO:teuthology.orchestra.run.trial068.stderr:osd.2: osd_enable_op_tracker = '' 2026-02-23T13:09:45.205 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.3 2026-02-23T13:09:45.241 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:09:45.241 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:09:45.457 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1 128 11150 11022 86.1022 86.1094 0.00977442 0.0115538 2026-02-23T13:09:45.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:09:45 trial068 ceph-mon[38805]: pgmap v123: 17 pgs: 16 unknown, 1 stale+active+clean; 577 KiB data, 1.1 GiB used, 2.7 TiB / 2.7 TiB avail 2026-02-23T13:09:45.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:09:45 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/908195476' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:09:46.458 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2 127 22530 22403 87.5046 88.9141 0.0063838 0.0113773 2026-02-23T13:09:46.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:09:46 trial068 ceph-mon[38805]: Health check failed: Degraded data redundancy: 641/2472 objects degraded (25.930%), 9 pgs degraded (PG_DEGRADED) 2026-02-23T13:09:46.856 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [3] live_osds: [0, 2, 1] 2026-02-23T13:09:46.856 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2026-02-23T13:09:46.857 INFO:tasks.thrashosds.thrasher:Reviving osd 3 2026-02-23T13:09:46.857 INFO:tasks.cephadm.osd.3:Restarting osd.3 (starting--it wasn't running)... 2026-02-23T13:09:46.857 DEBUG:teuthology.orchestra.run.trial185:osd.3> sudo journalctl -f -n 0 -u ceph-5f823084-10b8-11f1-be80-d404e6e7d460@osd.3.service 2026-02-23T13:09:46.860 DEBUG:teuthology.orchestra.run.trial185:> sudo systemctl start ceph-5f823084-10b8-11f1-be80-d404e6e7d460@osd.3 2026-02-23T13:09:47.148 INFO:journalctl@ceph.osd.3.trial185.stdout:Feb 23 13:09:46 trial185 systemd[1]: Starting Ceph osd.3 for 5f823084-10b8-11f1-be80-d404e6e7d460... 2026-02-23T13:09:47.148 INFO:journalctl@ceph.osd.3.trial185.stdout:Feb 23 13:09:47 trial185 podman[69151]: 2026-02-23 13:09:47.025214353 +0000 UTC m=+0.016362506 container create 7858295cde5c48585dfa52e62003addf1215618ee877bd5f4a28da436865475a (image=quay.ceph.io/ceph-ci/ceph@sha256:49621faaa845c39b2aae72e4fa24123ee9b2ac3fb8ae834753907068ee2972ea, name=ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-3-activate, CEPH_GIT_REPO=https://github.com/ceph/ceph-ci.git, OSD_FLAVOR=default, GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfsganesha-5/, org.opencontainers.image.authors=Ceph Release Team , CEPH_SHA1=e371433fcd1390a0b82cfb18effc422c1f5d8c1d, org.label-schema.name=CentOS Stream 9 Base Image, FROM_IMAGE=quay.io/centos/centos:stream9, CEPH_REF=wip-shraddhaag-enable-cephadm-tests, org.label-schema.vendor=CentOS, org.label-schema.build-date=20260216, org.label-schema.schema-version=1.0, io.buildah.version=1.41.3, ceph=True, org.opencontainers.image.documentation=https://docs.ceph.com/, org.label-schema.license=GPLv2) 2026-02-23T13:09:47.148 INFO:journalctl@ceph.osd.3.trial185.stdout:Feb 23 13:09:47 trial185 podman[69151]: 2026-02-23 13:09:47.064562089 +0000 UTC m=+0.055710242 container init 7858295cde5c48585dfa52e62003addf1215618ee877bd5f4a28da436865475a (image=quay.ceph.io/ceph-ci/ceph@sha256:49621faaa845c39b2aae72e4fa24123ee9b2ac3fb8ae834753907068ee2972ea, name=ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-3-activate, org.opencontainers.image.authors=Ceph Release Team , CEPH_GIT_REPO=https://github.com/ceph/ceph-ci.git, org.label-schema.schema-version=1.0, GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfsganesha-5/, org.opencontainers.image.documentation=https://docs.ceph.com/, FROM_IMAGE=quay.io/centos/centos:stream9, OSD_FLAVOR=default, CEPH_REF=wip-shraddhaag-enable-cephadm-tests, CEPH_SHA1=e371433fcd1390a0b82cfb18effc422c1f5d8c1d, io.buildah.version=1.41.3, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20260216, ceph=True) 2026-02-23T13:09:47.148 INFO:journalctl@ceph.osd.3.trial185.stdout:Feb 23 13:09:47 trial185 podman[69151]: 2026-02-23 13:09:47.067265291 +0000 UTC m=+0.058413444 container start 7858295cde5c48585dfa52e62003addf1215618ee877bd5f4a28da436865475a (image=quay.ceph.io/ceph-ci/ceph@sha256:49621faaa845c39b2aae72e4fa24123ee9b2ac3fb8ae834753907068ee2972ea, name=ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-3-activate, org.opencontainers.image.authors=Ceph Release Team , FROM_IMAGE=quay.io/centos/centos:stream9, CEPH_GIT_REPO=https://github.com/ceph/ceph-ci.git, CEPH_SHA1=e371433fcd1390a0b82cfb18effc422c1f5d8c1d, io.buildah.version=1.41.3, GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfsganesha-5/, ceph=True, CEPH_REF=wip-shraddhaag-enable-cephadm-tests, org.opencontainers.image.documentation=https://docs.ceph.com/, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, OSD_FLAVOR=default, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20260216, org.label-schema.schema-version=1.0) 2026-02-23T13:09:47.148 INFO:journalctl@ceph.osd.3.trial185.stdout:Feb 23 13:09:47 trial185 podman[69151]: 2026-02-23 13:09:47.067770454 +0000 UTC m=+0.058918607 container attach 7858295cde5c48585dfa52e62003addf1215618ee877bd5f4a28da436865475a (image=quay.ceph.io/ceph-ci/ceph@sha256:49621faaa845c39b2aae72e4fa24123ee9b2ac3fb8ae834753907068ee2972ea, name=ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-3-activate, org.label-schema.license=GPLv2, FROM_IMAGE=quay.io/centos/centos:stream9, CEPH_GIT_REPO=https://github.com/ceph/ceph-ci.git, CEPH_SHA1=e371433fcd1390a0b82cfb18effc422c1f5d8c1d, CEPH_REF=wip-shraddhaag-enable-cephadm-tests, org.opencontainers.image.authors=Ceph Release Team , org.label-schema.vendor=CentOS, org.label-schema.build-date=20260216, org.label-schema.schema-version=1.0, OSD_FLAVOR=default, GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfsganesha-5/, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.41.3, org.opencontainers.image.documentation=https://docs.ceph.com/) 2026-02-23T13:09:47.148 INFO:journalctl@ceph.osd.3.trial185.stdout:Feb 23 13:09:47 trial185 podman[69151]: 2026-02-23 13:09:47.018608038 +0000 UTC m=+0.009756181 image pull 63df4b83db6f95a8ba965919b3671cb27173117c23b5c950aaadc8913f56213c quay.ceph.io/ceph-ci/ceph@sha256:49621faaa845c39b2aae72e4fa24123ee9b2ac3fb8ae834753907068ee2972ea 2026-02-23T13:09:47.149 INFO:journalctl@ceph.osd.3.trial185.stdout:Feb 23 13:09:47 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-3-activate[69171]: Running command: /usr/bin/ceph-authtool --gen-print-key 2026-02-23T13:09:47.149 INFO:journalctl@ceph.osd.3.trial185.stdout:Feb 23 13:09:47 trial185 bash[69151]: Running command: /usr/bin/ceph-authtool --gen-print-key 2026-02-23T13:09:47.242 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:09:47.400 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:09:47.457 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3 128 33500 33372 86.8989 85.6953 0.0235079 0.0114745 2026-02-23T13:09:47.533 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:09:47 trial068 ceph-mon[38805]: pgmap v124: 17 pgs: 2 active+clean, 9 active+undersized+degraded, 6 unknown; 52 MiB data, 1.2 GiB used, 2.7 TiB / 2.7 TiB avail; 11 MiB/s wr, 1.34k op/s; 641/2472 objects degraded (25.930%) 2026-02-23T13:09:47.534 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:09:47.531+0000 7f66a691c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:09:47.534 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:09:47.532+0000 7f66a691c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:09:47.550 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:09:47.550 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:09:47.554 INFO:journalctl@ceph.osd.3.trial185.stdout:Feb 23 13:09:47 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-3-activate[69171]: Running command: /usr/bin/ceph-authtool --gen-print-key 2026-02-23T13:09:47.554 INFO:journalctl@ceph.osd.3.trial185.stdout:Feb 23 13:09:47 trial185 bash[69151]: Running command: /usr/bin/ceph-authtool --gen-print-key 2026-02-23T13:09:47.554 INFO:journalctl@ceph.osd.3.trial185.stdout:Feb 23 13:09:47 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-3-activate[69171]: --> Failed to activate via raw: did not find any matching OSD to activate 2026-02-23T13:09:47.566 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:09:47.566 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:09:47.584 INFO:teuthology.orchestra.run.trial068.stdout:osd.2: {} 2026-02-23T13:09:47.584 INFO:teuthology.orchestra.run.trial068.stderr:osd.2: osd_enable_op_tracker = '' 2026-02-23T13:09:47.584 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.3 2026-02-23T13:09:47.626 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:09:47.626 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:09:47.817 INFO:journalctl@ceph.osd.3.trial185.stdout:Feb 23 13:09:47 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-3-activate[69171]: Running command: /usr/bin/ceph-authtool --gen-print-key 2026-02-23T13:09:47.817 INFO:journalctl@ceph.osd.3.trial185.stdout:Feb 23 13:09:47 trial185 bash[69151]: --> Failed to activate via raw: did not find any matching OSD to activate 2026-02-23T13:09:47.817 INFO:journalctl@ceph.osd.3.trial185.stdout:Feb 23 13:09:47 trial185 bash[69151]: Running command: /usr/bin/ceph-authtool --gen-print-key 2026-02-23T13:09:47.818 INFO:journalctl@ceph.osd.3.trial185.stdout:Feb 23 13:09:47 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-3-activate[69171]: Running command: /usr/bin/ceph-authtool --gen-print-key 2026-02-23T13:09:47.818 INFO:journalctl@ceph.osd.3.trial185.stdout:Feb 23 13:09:47 trial185 bash[69151]: Running command: /usr/bin/ceph-authtool --gen-print-key 2026-02-23T13:09:47.818 INFO:journalctl@ceph.osd.3.trial185.stdout:Feb 23 13:09:47 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-3-activate[69171]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2026-02-23T13:09:47.818 INFO:journalctl@ceph.osd.3.trial185.stdout:Feb 23 13:09:47 trial185 bash[69151]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2026-02-23T13:09:47.818 INFO:journalctl@ceph.osd.3.trial185.stdout:Feb 23 13:09:47 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-3-activate[69171]: Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/vg_nvme/lv_3 --path /var/lib/ceph/osd/ceph-3 --no-mon-config 2026-02-23T13:09:47.818 INFO:journalctl@ceph.osd.3.trial185.stdout:Feb 23 13:09:47 trial185 bash[69151]: Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/vg_nvme/lv_3 --path /var/lib/ceph/osd/ceph-3 --no-mon-config 2026-02-23T13:09:47.818 INFO:journalctl@ceph.osd.3.trial185.stdout:Feb 23 13:09:47 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-3-activate[69171]: Running command: /usr/bin/ln -snf /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-3/block 2026-02-23T13:09:47.818 INFO:journalctl@ceph.osd.3.trial185.stdout:Feb 23 13:09:47 trial185 bash[69151]: Running command: /usr/bin/ln -snf /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-3/block 2026-02-23T13:09:47.818 INFO:journalctl@ceph.osd.3.trial185.stdout:Feb 23 13:09:47 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-3-activate[69171]: Running command: /usr/bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-3/block 2026-02-23T13:09:47.818 INFO:journalctl@ceph.osd.3.trial185.stdout:Feb 23 13:09:47 trial185 bash[69151]: Running command: /usr/bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-3/block 2026-02-23T13:09:47.818 INFO:journalctl@ceph.osd.3.trial185.stdout:Feb 23 13:09:47 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-3-activate[69171]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2026-02-23T13:09:47.818 INFO:journalctl@ceph.osd.3.trial185.stdout:Feb 23 13:09:47 trial185 bash[69151]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2026-02-23T13:09:47.818 INFO:journalctl@ceph.osd.3.trial185.stdout:Feb 23 13:09:47 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-3-activate[69171]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2026-02-23T13:09:47.818 INFO:journalctl@ceph.osd.3.trial185.stdout:Feb 23 13:09:47 trial185 bash[69151]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2026-02-23T13:09:47.818 INFO:journalctl@ceph.osd.3.trial185.stdout:Feb 23 13:09:47 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-3-activate[69171]: --> ceph-volume lvm activate successful for osd ID: 3 2026-02-23T13:09:47.818 INFO:journalctl@ceph.osd.3.trial185.stdout:Feb 23 13:09:47 trial185 bash[69151]: --> ceph-volume lvm activate successful for osd ID: 3 2026-02-23T13:09:47.818 INFO:journalctl@ceph.osd.3.trial185.stdout:Feb 23 13:09:47 trial185 podman[69151]: 2026-02-23 13:09:47.71825389 +0000 UTC m=+0.709402043 container died 7858295cde5c48585dfa52e62003addf1215618ee877bd5f4a28da436865475a (image=quay.ceph.io/ceph-ci/ceph@sha256:49621faaa845c39b2aae72e4fa24123ee9b2ac3fb8ae834753907068ee2972ea, name=ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-3-activate, CEPH_REF=wip-shraddhaag-enable-cephadm-tests, FROM_IMAGE=quay.io/centos/centos:stream9, org.opencontainers.image.documentation=https://docs.ceph.com/, CEPH_SHA1=e371433fcd1390a0b82cfb18effc422c1f5d8c1d, org.label-schema.schema-version=1.0, GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfsganesha-5/, org.opencontainers.image.authors=Ceph Release Team , org.label-schema.license=GPLv2, org.label-schema.build-date=20260216, io.buildah.version=1.41.3, OSD_FLAVOR=default, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, CEPH_GIT_REPO=https://github.com/ceph/ceph-ci.git) 2026-02-23T13:09:47.818 INFO:journalctl@ceph.osd.3.trial185.stdout:Feb 23 13:09:47 trial185 podman[69151]: 2026-02-23 13:09:47.73031962 +0000 UTC m=+0.721467763 container remove 7858295cde5c48585dfa52e62003addf1215618ee877bd5f4a28da436865475a (image=quay.ceph.io/ceph-ci/ceph@sha256:49621faaa845c39b2aae72e4fa24123ee9b2ac3fb8ae834753907068ee2972ea, name=ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-3-activate, io.buildah.version=1.41.3, org.opencontainers.image.authors=Ceph Release Team , org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfsganesha-5/, CEPH_GIT_REPO=https://github.com/ceph/ceph-ci.git, org.opencontainers.image.documentation=https://docs.ceph.com/, FROM_IMAGE=quay.io/centos/centos:stream9, org.label-schema.vendor=CentOS, CEPH_REF=wip-shraddhaag-enable-cephadm-tests, CEPH_SHA1=e371433fcd1390a0b82cfb18effc422c1f5d8c1d, org.label-schema.schema-version=1.0, org.label-schema.build-date=20260216, org.label-schema.license=GPLv2, OSD_FLAVOR=default) 2026-02-23T13:09:47.818 INFO:journalctl@ceph.osd.3.trial185.stdout:Feb 23 13:09:47 trial185 podman[69384]: 2026-02-23 13:09:47.815927436 +0000 UTC m=+0.017124738 container create c9b284b0def979be3a0427d23183d2e0b3fe3d41f36aca9da1b60b9c05925e9b (image=quay.ceph.io/ceph-ci/ceph@sha256:49621faaa845c39b2aae72e4fa24123ee9b2ac3fb8ae834753907068ee2972ea, name=ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-3, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, OSD_FLAVOR=default, org.label-schema.build-date=20260216, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_REF=wip-shraddhaag-enable-cephadm-tests, org.label-schema.vendor=CentOS, org.opencontainers.image.authors=Ceph Release Team , org.opencontainers.image.documentation=https://docs.ceph.com/, io.buildah.version=1.41.3, GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfsganesha-5/, CEPH_GIT_REPO=https://github.com/ceph/ceph-ci.git, FROM_IMAGE=quay.io/centos/centos:stream9, CEPH_SHA1=e371433fcd1390a0b82cfb18effc422c1f5d8c1d) 2026-02-23T13:09:47.858 DEBUG:teuthology.orchestra.run.trial185:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph daemon osd.3 dump_ops_in_flight 2026-02-23T13:09:47.990 INFO:teuthology.orchestra.run.trial185.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/config/ceph.conf 2026-02-23T13:09:48.107 INFO:journalctl@ceph.osd.3.trial185.stdout:Feb 23 13:09:47 trial185 podman[69384]: 2026-02-23 13:09:47.847894713 +0000 UTC m=+0.049092025 container init c9b284b0def979be3a0427d23183d2e0b3fe3d41f36aca9da1b60b9c05925e9b (image=quay.ceph.io/ceph-ci/ceph@sha256:49621faaa845c39b2aae72e4fa24123ee9b2ac3fb8ae834753907068ee2972ea, name=ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-3, org.opencontainers.image.authors=Ceph Release Team , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, io.buildah.version=1.41.3, CEPH_SHA1=e371433fcd1390a0b82cfb18effc422c1f5d8c1d, org.label-schema.schema-version=1.0, GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfsganesha-5/, org.label-schema.vendor=CentOS, org.opencontainers.image.documentation=https://docs.ceph.com/, FROM_IMAGE=quay.io/centos/centos:stream9, org.label-schema.build-date=20260216, CEPH_REF=wip-shraddhaag-enable-cephadm-tests, ceph=True, OSD_FLAVOR=default, CEPH_GIT_REPO=https://github.com/ceph/ceph-ci.git) 2026-02-23T13:09:48.107 INFO:journalctl@ceph.osd.3.trial185.stdout:Feb 23 13:09:47 trial185 podman[69384]: 2026-02-23 13:09:47.850586304 +0000 UTC m=+0.051783596 container start c9b284b0def979be3a0427d23183d2e0b3fe3d41f36aca9da1b60b9c05925e9b (image=quay.ceph.io/ceph-ci/ceph@sha256:49621faaa845c39b2aae72e4fa24123ee9b2ac3fb8ae834753907068ee2972ea, name=ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-3, ceph=True, org.label-schema.vendor=CentOS, FROM_IMAGE=quay.io/centos/centos:stream9, io.buildah.version=1.41.3, org.label-schema.name=CentOS Stream 9 Base Image, org.opencontainers.image.authors=Ceph Release Team , org.label-schema.build-date=20260216, CEPH_REF=wip-shraddhaag-enable-cephadm-tests, GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfsganesha-5/, CEPH_GIT_REPO=https://github.com/ceph/ceph-ci.git, org.label-schema.license=GPLv2, org.opencontainers.image.documentation=https://docs.ceph.com/, CEPH_SHA1=e371433fcd1390a0b82cfb18effc422c1f5d8c1d, org.label-schema.schema-version=1.0, OSD_FLAVOR=default) 2026-02-23T13:09:48.107 INFO:journalctl@ceph.osd.3.trial185.stdout:Feb 23 13:09:47 trial185 bash[69384]: c9b284b0def979be3a0427d23183d2e0b3fe3d41f36aca9da1b60b9c05925e9b 2026-02-23T13:09:48.107 INFO:journalctl@ceph.osd.3.trial185.stdout:Feb 23 13:09:47 trial185 podman[69384]: 2026-02-23 13:09:47.808775023 +0000 UTC m=+0.009972325 image pull 63df4b83db6f95a8ba965919b3671cb27173117c23b5c950aaadc8913f56213c quay.ceph.io/ceph-ci/ceph@sha256:49621faaa845c39b2aae72e4fa24123ee9b2ac3fb8ae834753907068ee2972ea 2026-02-23T13:09:48.107 INFO:journalctl@ceph.osd.3.trial185.stdout:Feb 23 13:09:47 trial185 systemd[1]: Started Ceph osd.3 for 5f823084-10b8-11f1-be80-d404e6e7d460. 2026-02-23T13:09:48.107 INFO:journalctl@ceph.osd.3.trial185.stdout:Feb 23 13:09:47 trial185 ceph-osd[69412]: -- 10.20.193.185:0/1215171229 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x3c59c00 msgr2=0x4a96000 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).process 2026-02-23T13:09:48.107 INFO:journalctl@ceph.osd.3.trial185.stdout:Feb 23 13:09:47 trial185 ceph-osd[69412]: -- 10.20.193.185:0/1215171229 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x3c59c00 msgr2=0x4a96000 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read continue len=32 2026-02-23T13:09:48.107 INFO:journalctl@ceph.osd.3.trial185.stdout:Feb 23 13:09:47 trial185 ceph-osd[69412]: --2- 10.20.193.185:0/1215171229 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x3c59c00 0x4a96000 crc :-1 s=READY pgs=435 gs=1 cs=0 l=1 c_cookie=17364509611876331545 s_cookie=0 reconnecting=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main r=0 2026-02-23T13:09:48.107 INFO:journalctl@ceph.osd.3.trial185.stdout:Feb 23 13:09:47 trial185 ceph-osd[69412]: --2- 10.20.193.185:0/1215171229 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x3c59c00 0x4a96000 crc :-1 s=THROTTLE_MESSAGE pgs=435 gs=1 cs=0 l=1 c_cookie=17364509611876331545 s_cookie=0 reconnecting=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).throttle_message 2026-02-23T13:09:48.107 INFO:journalctl@ceph.osd.3.trial185.stdout:Feb 23 13:09:47 trial185 ceph-osd[69412]: --2- 10.20.193.185:0/1215171229 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x3c59c00 0x4a96000 crc :-1 s=THROTTLE_BYTES pgs=435 gs=1 cs=0 l=1 c_cookie=17364509611876331545 s_cookie=0 reconnecting=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).throttle_bytes 2026-02-23T13:09:48.107 INFO:journalctl@ceph.osd.3.trial185.stdout:Feb 23 13:09:47 trial185 ceph-osd[69412]: --2- 10.20.193.185:0/1215171229 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x3c59c00 0x4a96000 crc :-1 s=THROTTLE_DISPATCH_QUEUE pgs=435 gs=1 cs=0 l=1 c_cookie=17364509611876331545 s_cookie=0 reconnecting=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).throttle_dispatch_queue 2026-02-23T13:09:48.107 INFO:journalctl@ceph.osd.3.trial185.stdout:Feb 23 13:09:47 trial185 ceph-osd[69412]: --2- 10.20.193.185:0/1215171229 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x3c59c00 0x4a96000 crc :-1 s=THROTTLE_DONE pgs=435 gs=1 cs=0 l=1 c_cookie=17364509611876331545 s_cookie=0 reconnecting=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).read_frame_segment seg_idx=0 2026-02-23T13:09:48.107 INFO:journalctl@ceph.osd.3.trial185.stdout:Feb 23 13:09:47 trial185 ceph-osd[69412]: -- 10.20.193.185:0/1215171229 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x3c59c00 msgr2=0x4a96000 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read start len=45 2026-02-23T13:09:48.107 INFO:journalctl@ceph.osd.3.trial185.stdout:Feb 23 13:09:47 trial185 ceph-osd[69412]: --2- 10.20.193.185:0/1215171229 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x3c59c00 0x4a96000 crc :-1 s=THROTTLE_DONE pgs=435 gs=1 cs=0 l=1 c_cookie=17364509611876331545 s_cookie=0 reconnecting=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_segment r=0 2026-02-23T13:09:48.107 INFO:journalctl@ceph.osd.3.trial185.stdout:Feb 23 13:09:47 trial185 ceph-osd[69412]: --2- 10.20.193.185:0/1215171229 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x3c59c00 0x4a96000 crc :-1 s=THROTTLE_DONE pgs=435 gs=1 cs=0 l=1 c_cookie=17364509611876331545 s_cookie=0 reconnecting=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).read_frame_segment seg_idx=1 2026-02-23T13:09:48.107 INFO:journalctl@ceph.osd.3.trial185.stdout:Feb 23 13:09:47 trial185 ceph-osd[69412]: -- 10.20.193.185:0/1215171229 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x3c59c00 msgr2=0x4a96000 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read start len=194 2026-02-23T13:09:48.107 INFO:journalctl@ceph.osd.3.trial185.stdout:Feb 23 13:09:47 trial185 ceph-osd[69412]: --2- 10.20.193.185:0/1215171229 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x3c59c00 0x4a96000 crc :-1 s=THROTTLE_DONE pgs=435 gs=1 cs=0 l=1 c_cookie=17364509611876331545 s_cookie=0 reconnecting=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_segment r=0 2026-02-23T13:09:48.107 INFO:journalctl@ceph.osd.3.trial185.stdout:Feb 23 13:09:47 trial185 ceph-osd[69412]: -- 10.20.193.185:0/1215171229 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x3c59c00 msgr2=0x4a96000 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read start len=13 2026-02-23T13:09:48.107 INFO:journalctl@ceph.osd.3.trial185.stdout:Feb 23 13:09:47 trial185 ceph-osd[69412]: --2- 10.20.193.185:0/1215171229 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x3c59c00 0x4a96000 crc :-1 s=THROTTLE_DONE pgs=435 gs=1 cs=0 l=1 c_cookie=17364509611876331545 s_cookie=0 reconnecting=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_epilogue_main r=0 2026-02-23T13:09:48.107 INFO:journalctl@ceph.osd.3.trial185.stdout:Feb 23 13:09:47 trial185 ceph-osd[69412]: --2- 10.20.193.185:0/1215171229 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x3c59c00 0x4a96000 crc :-1 s=THROTTLE_DONE pgs=435 gs=1 cs=0 l=1 c_cookie=17364509611876331545 s_cookie=0 reconnecting=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_dispatch tag=17 2026-02-23T13:09:48.107 INFO:journalctl@ceph.osd.3.trial185.stdout:Feb 23 13:09:47 trial185 ceph-osd[69412]: --2- 10.20.193.185:0/1215171229 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x3c59c00 0x4a96000 crc :-1 s=THROTTLE_DONE pgs=435 gs=1 cs=0 l=1 c_cookie=17364509611876331545 s_cookie=0 reconnecting=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_message 2026-02-23T13:09:48.107 INFO:journalctl@ceph.osd.3.trial185.stdout:Feb 23 13:09:47 trial185 ceph-osd[69412]: --2- 10.20.193.185:0/1215171229 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x3c59c00 0x4a96000 crc :-1 s=THROTTLE_DONE pgs=435 gs=1 cs=0 l=1 c_cookie=17364509611876331545 s_cookie=0 reconnecting=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_message got 194 + 0 + 0 byte message. envelope type=18 src mon.0 off 0 2026-02-23T13:09:48.107 INFO:journalctl@ceph.osd.3.trial185.stdout:Feb 23 13:09:47 trial185 ceph-osd[69412]: --2- 10.20.193.185:0/1215171229 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x3c59c00 0x4a96000 crc :-1 s=READ_MESSAGE_COMPLETE pgs=435 gs=1 cs=0 l=1 c_cookie=17364509611876331545 s_cookie=0 reconnecting=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_message received message m=0x3cbd860 seq=4 from=mon.0 type=18 auth_reply(proto 2 0 (0) Success) 2026-02-23T13:09:48.107 INFO:journalctl@ceph.osd.3.trial185.stdout:Feb 23 13:09:47 trial185 ceph-osd[69412]: -- 10.20.193.185:0/1215171229 queue 0x3cbd860 prio 196 2026-02-23T13:09:48.107 INFO:journalctl@ceph.osd.3.trial185.stdout:Feb 23 13:09:47 trial185 ceph-osd[69412]: --2- 10.20.193.185:0/1215171229 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x3c59c00 0x4a96000 crc :-1 s=READY pgs=435 gs=1 cs=0 l=1 c_cookie=17364509611876331545 s_cookie=0 reconnecting=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).read_frame 2026-02-23T13:09:48.107 INFO:journalctl@ceph.osd.3.trial185.stdout:Feb 23 13:09:47 trial185 ceph-osd[69412]: -- 10.20.193.185:0/1215171229 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x3c59c00 msgr2=0x4a96000 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read start len=32 2026-02-23T13:09:48.107 INFO:journalctl@ceph.osd.3.trial185.stdout:Feb 23 13:09:47 trial185 ceph-osd[69412]: -- 10.20.193.185:0/1215171229 <== mon.0 v2:10.20.193.68:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (crc 0 0 0) 0x3cbd860 con 0x3c59c00 2026-02-23T13:09:48.107 INFO:journalctl@ceph.osd.3.trial185.stdout:Feb 23 13:09:47 trial185 ceph-osd[69412]: -- 10.20.193.185:0/1215171229 dispatch_throttle_release 194 to dispatch throttler 194/104857600 2026-02-23T13:09:48.107 INFO:journalctl@ceph.osd.3.trial185.stdout:Feb 23 13:09:47 trial185 ceph-osd[69412]: -- 10.20.193.185:0/1215171229 done calling dispatch on 0x3cbd860 2026-02-23T13:09:48.107 INFO:journalctl@ceph.osd.3.trial185.stdout:Feb 23 13:09:47 trial185 ceph-osd[69412]: WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:09:48.107 INFO:journalctl@ceph.osd.3.trial185.stdout:Feb 23 13:09:47 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-3[69408]: 2026-02-23T13:09:47.864+0000 7fe5e6465780 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:09:48.107 INFO:journalctl@ceph.osd.3.trial185.stdout:Feb 23 13:09:47 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-3[69408]: 2026-02-23T13:09:47.864+0000 7fe5e6465780 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:09:48.107 INFO:journalctl@ceph.osd.3.trial185.stdout:Feb 23 13:09:47 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-3[69408]: 2026-02-23T13:09:47.864+0000 7fe5e6465780 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:09:48.120 INFO:teuthology.orchestra.run.trial185.stderr:no valid command found; 10 closest matches: 2026-02-23T13:09:48.120 INFO:teuthology.orchestra.run.trial185.stderr:0 2026-02-23T13:09:48.121 INFO:teuthology.orchestra.run.trial185.stderr:1 2026-02-23T13:09:48.121 INFO:teuthology.orchestra.run.trial185.stderr:2 2026-02-23T13:09:48.121 INFO:teuthology.orchestra.run.trial185.stderr:abort 2026-02-23T13:09:48.121 INFO:teuthology.orchestra.run.trial185.stderr:assert 2026-02-23T13:09:48.121 INFO:teuthology.orchestra.run.trial185.stderr:bluestore collections 2026-02-23T13:09:48.121 INFO:teuthology.orchestra.run.trial185.stderr:bluestore compression stats [] 2026-02-23T13:09:48.121 INFO:teuthology.orchestra.run.trial185.stderr:bluestore list [] [] 2026-02-23T13:09:48.121 INFO:teuthology.orchestra.run.trial185.stderr:bluestore onode metadata 2026-02-23T13:09:48.121 INFO:teuthology.orchestra.run.trial185.stderr:bluestore show sharding 2026-02-23T13:09:48.121 INFO:teuthology.orchestra.run.trial185.stderr:admin_socket: invalid command 2026-02-23T13:09:48.158 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-23T13:09:48.158 INFO:tasks.cephadm.ceph_manager.ceph:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2026-02-23T13:09:48.458 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4 127 43479 43352 84.6649 77.9688 0.0103338 0.0117904 2026-02-23T13:09:48.606 INFO:journalctl@ceph.osd.3.trial185.stdout:Feb 23 13:09:48 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-3[69408]: 2026-02-23T13:09:48.114+0000 7fe5e6465780 -1 Falling back to public interface 2026-02-23T13:09:48.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:09:48 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/379640153' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:09:49.458 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5 128 53201 53073 82.9198 75.9453 0.0241155 0.0120317 2026-02-23T13:09:49.627 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:09:49.777 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:09:49.806 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:09:49 trial068 ceph-mon[38805]: pgmap v125: 17 pgs: 6 active+clean, 11 active+undersized+degraded; 138 MiB data, 1.3 GiB used, 2.7 TiB / 2.7 TiB avail; 23 MiB/s wr, 2.93k op/s; 1103/6600 objects degraded (16.712%) 2026-02-23T13:09:49.901 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:09:49.899+0000 7ff6e85ea640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:09:49.901 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:09:49.900+0000 7ff6e85ea640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:09:49.918 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:09:49.919 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:09:49.934 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:09:49.934 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:09:49.949 INFO:teuthology.orchestra.run.trial068.stdout:osd.2: {} 2026-02-23T13:09:49.950 INFO:teuthology.orchestra.run.trial068.stderr:osd.2: osd_enable_op_tracker = '' 2026-02-23T13:09:49.950 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.3 2026-02-23T13:09:49.990 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:09:49.990 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:09:50.458 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6 128 61951 61823 80.4923 68.3594 0.021615 0.0124038 2026-02-23T13:09:50.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:09:50 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2551684024' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:09:51.107 INFO:journalctl@ceph.osd.3.trial185.stdout:Feb 23 13:09:50 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-3[69408]: 2026-02-23T13:09:50.840+0000 7fe5e6465780 -1 osd.3 74 log_to_monitors true 2026-02-23T13:09:51.458 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7 127 69945 69818 77.9157 62.4609 0.00626753 0.0128009 2026-02-23T13:09:51.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:09:51 trial068 ceph-mon[38805]: pgmap v126: 17 pgs: 6 active+clean, 11 active+undersized+degraded; 138 MiB data, 1.3 GiB used, 2.7 TiB / 2.7 TiB avail; 17 MiB/s wr, 2.23k op/s; 1103/6600 objects degraded (16.712%) 2026-02-23T13:09:51.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:09:51 trial068 ceph-mon[38805]: from='osd.3 [v2:10.20.193.185:6808/3696769086,v1:10.20.193.185:6809/3696769086]' entity='osd.3' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]} : dispatch 2026-02-23T13:09:51.856 INFO:journalctl@ceph.osd.3.trial185.stdout:Feb 23 13:09:51 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-3[69408]: 2026-02-23T13:09:51.430+0000 7fe5de2ae640 -1 osd.3 74 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2026-02-23T13:09:51.991 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:09:52.163 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:09:52.297 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:09:52.295+0000 7f0c88f72640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:09:52.298 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:09:52.296+0000 7f0c88f72640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:09:52.314 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:09:52.314 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:09:52.331 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:09:52.331 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:09:52.349 INFO:teuthology.orchestra.run.trial068.stdout:osd.2: {} 2026-02-23T13:09:52.350 INFO:teuthology.orchestra.run.trial068.stderr:osd.2: osd_enable_op_tracker = '' 2026-02-23T13:09:52.350 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.3 2026-02-23T13:09:52.398 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:09:52.399 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:09:52.458 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8 127 77382 77255 75.4384 58.1016 0.0294911 0.0132319 2026-02-23T13:09:52.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:09:52 trial068 ceph-mon[38805]: from='osd.3 [v2:10.20.193.185:6808/3696769086,v1:10.20.193.185:6809/3696769086]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2026-02-23T13:09:52.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:09:52 trial068 ceph-mon[38805]: osdmap e79: 4 total, 3 up, 4 in 2026-02-23T13:09:52.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:09:52 trial068 ceph-mon[38805]: from='osd.3 [v2:10.20.193.185:6808/3696769086,v1:10.20.193.185:6809/3696769086]' entity='osd.3' cmd={"prefix": "osd crush create-or-move", "id": 3, "weight":0.6836, "args": ["host=trial185", "root=default"]} : dispatch 2026-02-23T13:09:52.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:09:52 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1633568867' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:09:53.159 DEBUG:teuthology.orchestra.run.trial185:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph daemon osd.3 dump_ops_in_flight 2026-02-23T13:09:53.283 INFO:teuthology.orchestra.run.trial185.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/config/ceph.conf 2026-02-23T13:09:53.403 INFO:teuthology.orchestra.run.trial185.stdout:{ 2026-02-23T13:09:53.403 INFO:teuthology.orchestra.run.trial185.stdout: "ops": [ 2026-02-23T13:09:53.403 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.403 INFO:teuthology.orchestra.run.trial185.stdout: "description": "osd_op(client.14522.0:79278 3.c 3.8e4a5dec (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e80)", 2026-02-23T13:09:53.403 INFO:teuthology.orchestra.run.trial185.stdout: "initiated_at": "2026-02-23T13:09:52.718768+0000", 2026-02-23T13:09:53.403 INFO:teuthology.orchestra.run.trial185.stdout: "age": 0.68062890499999995, 2026-02-23T13:09:53.404 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 0.68064395300000002, 2026-02-23T13:09:53.404 INFO:teuthology.orchestra.run.trial185.stdout: "continuous": false, 2026-02-23T13:09:53.404 INFO:teuthology.orchestra.run.trial185.stdout: "type_data": { 2026-02-23T13:09:53.404 INFO:teuthology.orchestra.run.trial185.stdout: "flag_point": "waiting for peered", 2026-02-23T13:09:53.404 INFO:teuthology.orchestra.run.trial185.stdout: "client_info": { 2026-02-23T13:09:53.404 INFO:teuthology.orchestra.run.trial185.stdout: "client": "client.14522", 2026-02-23T13:09:53.404 INFO:teuthology.orchestra.run.trial185.stdout: "client_addr": "10.20.193.68:0/2327563370", 2026-02-23T13:09:53.404 INFO:teuthology.orchestra.run.trial185.stdout: "tid": 79278 2026-02-23T13:09:53.404 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.404 INFO:teuthology.orchestra.run.trial185.stdout: "events": [ 2026-02-23T13:09:53.404 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.405 INFO:teuthology.orchestra.run.trial185.stdout: "event": "initiated", 2026-02-23T13:09:53.405 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.718768+0000", 2026-02-23T13:09:53.405 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 0 2026-02-23T13:09:53.405 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.405 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.405 INFO:teuthology.orchestra.run.trial185.stdout: "event": "header_read", 2026-02-23T13:09:53.405 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.718768+0000", 2026-02-23T13:09:53.405 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 0 2026-02-23T13:09:53.405 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.405 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.405 INFO:teuthology.orchestra.run.trial185.stdout: "event": "throttled", 2026-02-23T13:09:53.405 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.718774+0000", 2026-02-23T13:09:53.406 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 6.5019999999999998e-06 2026-02-23T13:09:53.406 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.406 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.406 INFO:teuthology.orchestra.run.trial185.stdout: "event": "all_read", 2026-02-23T13:09:53.406 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.718799+0000", 2026-02-23T13:09:53.406 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 2.5137e-05 2026-02-23T13:09:53.406 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.406 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.406 INFO:teuthology.orchestra.run.trial185.stdout: "event": "dispatched", 2026-02-23T13:09:53.406 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.718803+0000", 2026-02-23T13:09:53.406 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 3.777e-06 2026-02-23T13:09:53.407 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.407 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.407 INFO:teuthology.orchestra.run.trial185.stdout: "event": "queued_for_pg", 2026-02-23T13:09:53.407 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.718804+0000", 2026-02-23T13:09:53.407 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 1.1119999999999999e-06 2026-02-23T13:09:53.407 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.407 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.407 INFO:teuthology.orchestra.run.trial185.stdout: "event": "reached_pg", 2026-02-23T13:09:53.407 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.718821+0000", 2026-02-23T13:09:53.407 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 1.6801999999999999e-05 2026-02-23T13:09:53.407 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.407 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.408 INFO:teuthology.orchestra.run.trial185.stdout: "event": "waiting for peered", 2026-02-23T13:09:53.408 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.718823+0000", 2026-02-23T13:09:53.408 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 2.334e-06 2026-02-23T13:09:53.408 INFO:teuthology.orchestra.run.trial185.stdout: } 2026-02-23T13:09:53.408 INFO:teuthology.orchestra.run.trial185.stdout: ] 2026-02-23T13:09:53.408 INFO:teuthology.orchestra.run.trial185.stdout: } 2026-02-23T13:09:53.408 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.408 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.408 INFO:teuthology.orchestra.run.trial185.stdout: "description": "osd_op(client.14522.0:79054 3.9 3.86388f89 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e80)", 2026-02-23T13:09:53.408 INFO:teuthology.orchestra.run.trial185.stdout: "initiated_at": "2026-02-23T13:09:52.717243+0000", 2026-02-23T13:09:53.409 INFO:teuthology.orchestra.run.trial185.stdout: "age": 0.68215402999999997, 2026-02-23T13:09:53.409 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 0.68219203100000003, 2026-02-23T13:09:53.409 INFO:teuthology.orchestra.run.trial185.stdout: "continuous": false, 2026-02-23T13:09:53.409 INFO:teuthology.orchestra.run.trial185.stdout: "type_data": { 2026-02-23T13:09:53.409 INFO:teuthology.orchestra.run.trial185.stdout: "flag_point": "waiting for peered", 2026-02-23T13:09:53.409 INFO:teuthology.orchestra.run.trial185.stdout: "client_info": { 2026-02-23T13:09:53.409 INFO:teuthology.orchestra.run.trial185.stdout: "client": "client.14522", 2026-02-23T13:09:53.409 INFO:teuthology.orchestra.run.trial185.stdout: "client_addr": "10.20.193.68:0/2327563370", 2026-02-23T13:09:53.409 INFO:teuthology.orchestra.run.trial185.stdout: "tid": 79054 2026-02-23T13:09:53.409 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.409 INFO:teuthology.orchestra.run.trial185.stdout: "events": [ 2026-02-23T13:09:53.410 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.410 INFO:teuthology.orchestra.run.trial185.stdout: "event": "initiated", 2026-02-23T13:09:53.410 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.717243+0000", 2026-02-23T13:09:53.410 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 0 2026-02-23T13:09:53.410 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.410 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.410 INFO:teuthology.orchestra.run.trial185.stdout: "event": "header_read", 2026-02-23T13:09:53.410 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.717243+0000", 2026-02-23T13:09:53.410 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 0 2026-02-23T13:09:53.410 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.410 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.410 INFO:teuthology.orchestra.run.trial185.stdout: "event": "throttled", 2026-02-23T13:09:53.411 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.717251+0000", 2026-02-23T13:09:53.411 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 8.4659999999999999e-06 2026-02-23T13:09:53.411 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.411 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.411 INFO:teuthology.orchestra.run.trial185.stdout: "event": "all_read", 2026-02-23T13:09:53.411 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.717277+0000", 2026-02-23T13:09:53.411 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 2.6129e-05 2026-02-23T13:09:53.411 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.411 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.411 INFO:teuthology.orchestra.run.trial185.stdout: "event": "dispatched", 2026-02-23T13:09:53.411 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.717284+0000", 2026-02-23T13:09:53.411 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 6.6619999999999999e-06 2026-02-23T13:09:53.412 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.412 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.412 INFO:teuthology.orchestra.run.trial185.stdout: "event": "queued_for_pg", 2026-02-23T13:09:53.412 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.717294+0000", 2026-02-23T13:09:53.412 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 1.061e-05 2026-02-23T13:09:53.412 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.412 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.412 INFO:teuthology.orchestra.run.trial185.stdout: "event": "reached_pg", 2026-02-23T13:09:53.412 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.717318+0000", 2026-02-23T13:09:53.412 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 2.3343e-05 2026-02-23T13:09:53.412 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.412 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.413 INFO:teuthology.orchestra.run.trial185.stdout: "event": "waiting for peered", 2026-02-23T13:09:53.413 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.717320+0000", 2026-02-23T13:09:53.413 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 2.295e-06 2026-02-23T13:09:53.413 INFO:teuthology.orchestra.run.trial185.stdout: } 2026-02-23T13:09:53.413 INFO:teuthology.orchestra.run.trial185.stdout: ] 2026-02-23T13:09:53.413 INFO:teuthology.orchestra.run.trial185.stdout: } 2026-02-23T13:09:53.413 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.413 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.413 INFO:teuthology.orchestra.run.trial185.stdout: "description": "osd_op(client.14522.0:79279 3.c 3.8e4a5dec (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e80)", 2026-02-23T13:09:53.413 INFO:teuthology.orchestra.run.trial185.stdout: "initiated_at": "2026-02-23T13:09:52.718812+0000", 2026-02-23T13:09:53.413 INFO:teuthology.orchestra.run.trial185.stdout: "age": 0.68058411100000005, 2026-02-23T13:09:53.413 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 0.68063685900000004, 2026-02-23T13:09:53.414 INFO:teuthology.orchestra.run.trial185.stdout: "continuous": false, 2026-02-23T13:09:53.414 INFO:teuthology.orchestra.run.trial185.stdout: "type_data": { 2026-02-23T13:09:53.414 INFO:teuthology.orchestra.run.trial185.stdout: "flag_point": "waiting for peered", 2026-02-23T13:09:53.414 INFO:teuthology.orchestra.run.trial185.stdout: "client_info": { 2026-02-23T13:09:53.414 INFO:teuthology.orchestra.run.trial185.stdout: "client": "client.14522", 2026-02-23T13:09:53.414 INFO:teuthology.orchestra.run.trial185.stdout: "client_addr": "10.20.193.68:0/2327563370", 2026-02-23T13:09:53.414 INFO:teuthology.orchestra.run.trial185.stdout: "tid": 79279 2026-02-23T13:09:53.414 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.414 INFO:teuthology.orchestra.run.trial185.stdout: "events": [ 2026-02-23T13:09:53.414 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.414 INFO:teuthology.orchestra.run.trial185.stdout: "event": "initiated", 2026-02-23T13:09:53.414 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.718812+0000", 2026-02-23T13:09:53.415 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 0 2026-02-23T13:09:53.415 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.415 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.415 INFO:teuthology.orchestra.run.trial185.stdout: "event": "header_read", 2026-02-23T13:09:53.415 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.718812+0000", 2026-02-23T13:09:53.415 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 0 2026-02-23T13:09:53.415 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.415 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.415 INFO:teuthology.orchestra.run.trial185.stdout: "event": "throttled", 2026-02-23T13:09:53.415 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.718821+0000", 2026-02-23T13:09:53.415 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 8.6759999999999996e-06 2026-02-23T13:09:53.415 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.415 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.416 INFO:teuthology.orchestra.run.trial185.stdout: "event": "all_read", 2026-02-23T13:09:53.416 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.718845+0000", 2026-02-23T13:09:53.416 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 2.3774e-05 2026-02-23T13:09:53.416 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.416 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.416 INFO:teuthology.orchestra.run.trial185.stdout: "event": "dispatched", 2026-02-23T13:09:53.416 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.718848+0000", 2026-02-23T13:09:53.416 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 3.2160000000000002e-06 2026-02-23T13:09:53.416 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.416 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.416 INFO:teuthology.orchestra.run.trial185.stdout: "event": "queued_for_pg", 2026-02-23T13:09:53.416 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.718849+0000", 2026-02-23T13:09:53.417 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 1.313e-06 2026-02-23T13:09:53.417 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.417 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.417 INFO:teuthology.orchestra.run.trial185.stdout: "event": "reached_pg", 2026-02-23T13:09:53.417 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.718869+0000", 2026-02-23T13:09:53.417 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 1.9666000000000001e-05 2026-02-23T13:09:53.417 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.417 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.417 INFO:teuthology.orchestra.run.trial185.stdout: "event": "waiting for peered", 2026-02-23T13:09:53.417 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.718870+0000", 2026-02-23T13:09:53.417 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 1.3430000000000001e-06 2026-02-23T13:09:53.417 INFO:teuthology.orchestra.run.trial185.stdout: } 2026-02-23T13:09:53.417 INFO:teuthology.orchestra.run.trial185.stdout: ] 2026-02-23T13:09:53.418 INFO:teuthology.orchestra.run.trial185.stdout: } 2026-02-23T13:09:53.418 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.418 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.418 INFO:teuthology.orchestra.run.trial185.stdout: "description": "osd_op(client.14522.0:79055 3.9 3.86388f89 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e80)", 2026-02-23T13:09:53.418 INFO:teuthology.orchestra.run.trial185.stdout: "initiated_at": "2026-02-23T13:09:52.717312+0000", 2026-02-23T13:09:53.418 INFO:teuthology.orchestra.run.trial185.stdout: "age": 0.68208410900000005, 2026-02-23T13:09:53.418 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 0.68215098399999996, 2026-02-23T13:09:53.418 INFO:teuthology.orchestra.run.trial185.stdout: "continuous": false, 2026-02-23T13:09:53.418 INFO:teuthology.orchestra.run.trial185.stdout: "type_data": { 2026-02-23T13:09:53.418 INFO:teuthology.orchestra.run.trial185.stdout: "flag_point": "waiting for peered", 2026-02-23T13:09:53.418 INFO:teuthology.orchestra.run.trial185.stdout: "client_info": { 2026-02-23T13:09:53.418 INFO:teuthology.orchestra.run.trial185.stdout: "client": "client.14522", 2026-02-23T13:09:53.419 INFO:teuthology.orchestra.run.trial185.stdout: "client_addr": "10.20.193.68:0/2327563370", 2026-02-23T13:09:53.419 INFO:teuthology.orchestra.run.trial185.stdout: "tid": 79055 2026-02-23T13:09:53.419 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.419 INFO:teuthology.orchestra.run.trial185.stdout: "events": [ 2026-02-23T13:09:53.419 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.419 INFO:teuthology.orchestra.run.trial185.stdout: "event": "initiated", 2026-02-23T13:09:53.419 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.717312+0000", 2026-02-23T13:09:53.419 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 0 2026-02-23T13:09:53.419 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.419 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.419 INFO:teuthology.orchestra.run.trial185.stdout: "event": "header_read", 2026-02-23T13:09:53.419 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.717312+0000", 2026-02-23T13:09:53.419 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 0 2026-02-23T13:09:53.420 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.420 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.420 INFO:teuthology.orchestra.run.trial185.stdout: "event": "throttled", 2026-02-23T13:09:53.421 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.717322+0000", 2026-02-23T13:09:53.422 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 9.5470000000000002e-06 2026-02-23T13:09:53.422 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.422 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.422 INFO:teuthology.orchestra.run.trial185.stdout: "event": "all_read", 2026-02-23T13:09:53.422 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.717364+0000", 2026-02-23T13:09:53.422 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 4.1647999999999998e-05 2026-02-23T13:09:53.422 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.422 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.422 INFO:teuthology.orchestra.run.trial185.stdout: "event": "dispatched", 2026-02-23T13:09:53.422 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.717368+0000", 2026-02-23T13:09:53.423 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 4.7489999999999998e-06 2026-02-23T13:09:53.423 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.423 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.423 INFO:teuthology.orchestra.run.trial185.stdout: "event": "queued_for_pg", 2026-02-23T13:09:53.423 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.717371+0000", 2026-02-23T13:09:53.423 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 2.2639999999999998e-06 2026-02-23T13:09:53.423 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.423 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.423 INFO:teuthology.orchestra.run.trial185.stdout: "event": "reached_pg", 2026-02-23T13:09:53.423 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.717390+0000", 2026-02-23T13:09:53.423 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 1.8935000000000001e-05 2026-02-23T13:09:53.423 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.424 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.424 INFO:teuthology.orchestra.run.trial185.stdout: "event": "waiting for peered", 2026-02-23T13:09:53.424 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.717392+0000", 2026-02-23T13:09:53.424 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 2.305e-06 2026-02-23T13:09:53.424 INFO:teuthology.orchestra.run.trial185.stdout: } 2026-02-23T13:09:53.424 INFO:teuthology.orchestra.run.trial185.stdout: ] 2026-02-23T13:09:53.424 INFO:teuthology.orchestra.run.trial185.stdout: } 2026-02-23T13:09:53.424 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.424 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.424 INFO:teuthology.orchestra.run.trial185.stdout: "description": "osd_op(client.14522.0:79280 3.c 3.8e4a5dec (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e80)", 2026-02-23T13:09:53.424 INFO:teuthology.orchestra.run.trial185.stdout: "initiated_at": "2026-02-23T13:09:52.718858+0000", 2026-02-23T13:09:53.424 INFO:teuthology.orchestra.run.trial185.stdout: "age": 0.68053840600000004, 2026-02-23T13:09:53.425 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 0.68061956700000004, 2026-02-23T13:09:53.425 INFO:teuthology.orchestra.run.trial185.stdout: "continuous": false, 2026-02-23T13:09:53.425 INFO:teuthology.orchestra.run.trial185.stdout: "type_data": { 2026-02-23T13:09:53.425 INFO:teuthology.orchestra.run.trial185.stdout: "flag_point": "waiting for peered", 2026-02-23T13:09:53.425 INFO:teuthology.orchestra.run.trial185.stdout: "client_info": { 2026-02-23T13:09:53.425 INFO:teuthology.orchestra.run.trial185.stdout: "client": "client.14522", 2026-02-23T13:09:53.425 INFO:teuthology.orchestra.run.trial185.stdout: "client_addr": "10.20.193.68:0/2327563370", 2026-02-23T13:09:53.425 INFO:teuthology.orchestra.run.trial185.stdout: "tid": 79280 2026-02-23T13:09:53.425 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.425 INFO:teuthology.orchestra.run.trial185.stdout: "events": [ 2026-02-23T13:09:53.425 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.425 INFO:teuthology.orchestra.run.trial185.stdout: "event": "initiated", 2026-02-23T13:09:53.426 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.718858+0000", 2026-02-23T13:09:53.426 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 0 2026-02-23T13:09:53.426 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.426 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.426 INFO:teuthology.orchestra.run.trial185.stdout: "event": "header_read", 2026-02-23T13:09:53.426 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.718858+0000", 2026-02-23T13:09:53.426 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 0 2026-02-23T13:09:53.426 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.426 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.426 INFO:teuthology.orchestra.run.trial185.stdout: "event": "throttled", 2026-02-23T13:09:53.426 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.718866+0000", 2026-02-23T13:09:53.426 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 7.9549999999999992e-06 2026-02-23T13:09:53.426 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.427 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.427 INFO:teuthology.orchestra.run.trial185.stdout: "event": "all_read", 2026-02-23T13:09:53.427 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.718888+0000", 2026-02-23T13:09:53.427 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 2.1659999999999999e-05 2026-02-23T13:09:53.427 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.427 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.427 INFO:teuthology.orchestra.run.trial185.stdout: "event": "dispatched", 2026-02-23T13:09:53.427 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.718891+0000", 2026-02-23T13:09:53.427 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 3.2660000000000002e-06 2026-02-23T13:09:53.427 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.427 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.427 INFO:teuthology.orchestra.run.trial185.stdout: "event": "queued_for_pg", 2026-02-23T13:09:53.428 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.718893+0000", 2026-02-23T13:09:53.428 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 2.3549999999999999e-06 2026-02-23T13:09:53.428 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.428 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.428 INFO:teuthology.orchestra.run.trial185.stdout: "event": "reached_pg", 2026-02-23T13:09:53.428 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.718909+0000", 2026-02-23T13:09:53.428 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 1.5447999999999998e-05 2026-02-23T13:09:53.428 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.428 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.428 INFO:teuthology.orchestra.run.trial185.stdout: "event": "waiting for peered", 2026-02-23T13:09:53.428 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.718911+0000", 2026-02-23T13:09:53.428 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 2.255e-06 2026-02-23T13:09:53.429 INFO:teuthology.orchestra.run.trial185.stdout: } 2026-02-23T13:09:53.429 INFO:teuthology.orchestra.run.trial185.stdout: ] 2026-02-23T13:09:53.429 INFO:teuthology.orchestra.run.trial185.stdout: } 2026-02-23T13:09:53.429 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.429 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.429 INFO:teuthology.orchestra.run.trial185.stdout: "description": "osd_op(client.14522.0:79056 3.9 3.86388f89 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e80)", 2026-02-23T13:09:53.429 INFO:teuthology.orchestra.run.trial185.stdout: "initiated_at": "2026-02-23T13:09:52.717386+0000", 2026-02-23T13:09:53.429 INFO:teuthology.orchestra.run.trial185.stdout: "age": 0.68201044200000005, 2026-02-23T13:09:53.429 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 0.682104127, 2026-02-23T13:09:53.429 INFO:teuthology.orchestra.run.trial185.stdout: "continuous": false, 2026-02-23T13:09:53.429 INFO:teuthology.orchestra.run.trial185.stdout: "type_data": { 2026-02-23T13:09:53.429 INFO:teuthology.orchestra.run.trial185.stdout: "flag_point": "waiting for peered", 2026-02-23T13:09:53.429 INFO:teuthology.orchestra.run.trial185.stdout: "client_info": { 2026-02-23T13:09:53.430 INFO:teuthology.orchestra.run.trial185.stdout: "client": "client.14522", 2026-02-23T13:09:53.430 INFO:teuthology.orchestra.run.trial185.stdout: "client_addr": "10.20.193.68:0/2327563370", 2026-02-23T13:09:53.430 INFO:teuthology.orchestra.run.trial185.stdout: "tid": 79056 2026-02-23T13:09:53.430 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.430 INFO:teuthology.orchestra.run.trial185.stdout: "events": [ 2026-02-23T13:09:53.430 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.430 INFO:teuthology.orchestra.run.trial185.stdout: "event": "initiated", 2026-02-23T13:09:53.430 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.717386+0000", 2026-02-23T13:09:53.430 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 0 2026-02-23T13:09:53.430 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.430 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.430 INFO:teuthology.orchestra.run.trial185.stdout: "event": "header_read", 2026-02-23T13:09:53.431 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.717386+0000", 2026-02-23T13:09:53.431 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 0 2026-02-23T13:09:53.431 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.431 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.431 INFO:teuthology.orchestra.run.trial185.stdout: "event": "throttled", 2026-02-23T13:09:53.431 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.717396+0000", 2026-02-23T13:09:53.431 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 9.6180000000000001e-06 2026-02-23T13:09:53.431 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.431 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.431 INFO:teuthology.orchestra.run.trial185.stdout: "event": "all_read", 2026-02-23T13:09:53.431 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.717417+0000", 2026-02-23T13:09:53.431 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 2.145e-05 2026-02-23T13:09:53.432 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.432 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.432 INFO:teuthology.orchestra.run.trial185.stdout: "event": "dispatched", 2026-02-23T13:09:53.432 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.717420+0000", 2026-02-23T13:09:53.432 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 2.9960000000000002e-06 2026-02-23T13:09:53.432 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.432 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.432 INFO:teuthology.orchestra.run.trial185.stdout: "event": "queued_for_pg", 2026-02-23T13:09:53.432 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.717422+0000", 2026-02-23T13:09:53.432 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 1.6729999999999999e-06 2026-02-23T13:09:53.432 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.432 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.432 INFO:teuthology.orchestra.run.trial185.stdout: "event": "reached_pg", 2026-02-23T13:09:53.433 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.717438+0000", 2026-02-23T13:09:53.433 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 1.5879000000000002e-05 2026-02-23T13:09:53.433 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.433 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.433 INFO:teuthology.orchestra.run.trial185.stdout: "event": "waiting for peered", 2026-02-23T13:09:53.433 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.717440+0000", 2026-02-23T13:09:53.433 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 1.784e-06 2026-02-23T13:09:53.433 INFO:teuthology.orchestra.run.trial185.stdout: } 2026-02-23T13:09:53.433 INFO:teuthology.orchestra.run.trial185.stdout: ] 2026-02-23T13:09:53.433 INFO:teuthology.orchestra.run.trial185.stdout: } 2026-02-23T13:09:53.433 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.433 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.433 INFO:teuthology.orchestra.run.trial185.stdout: "description": "osd_op(client.14522.0:79393 3.d 3.981104cd (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e80)", 2026-02-23T13:09:53.434 INFO:teuthology.orchestra.run.trial185.stdout: "initiated_at": "2026-02-23T13:09:52.721376+0000", 2026-02-23T13:09:53.434 INFO:teuthology.orchestra.run.trial185.stdout: "age": 0.67802018799999997, 2026-02-23T13:09:53.434 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 0.678126797, 2026-02-23T13:09:53.434 INFO:teuthology.orchestra.run.trial185.stdout: "continuous": false, 2026-02-23T13:09:53.434 INFO:teuthology.orchestra.run.trial185.stdout: "type_data": { 2026-02-23T13:09:53.434 INFO:teuthology.orchestra.run.trial185.stdout: "flag_point": "waiting for peered", 2026-02-23T13:09:53.434 INFO:teuthology.orchestra.run.trial185.stdout: "client_info": { 2026-02-23T13:09:53.434 INFO:teuthology.orchestra.run.trial185.stdout: "client": "client.14522", 2026-02-23T13:09:53.434 INFO:teuthology.orchestra.run.trial185.stdout: "client_addr": "10.20.193.68:0/2327563370", 2026-02-23T13:09:53.434 INFO:teuthology.orchestra.run.trial185.stdout: "tid": 79393 2026-02-23T13:09:53.434 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.434 INFO:teuthology.orchestra.run.trial185.stdout: "events": [ 2026-02-23T13:09:53.435 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.435 INFO:teuthology.orchestra.run.trial185.stdout: "event": "initiated", 2026-02-23T13:09:53.435 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.721376+0000", 2026-02-23T13:09:53.435 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 0 2026-02-23T13:09:53.435 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.435 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.435 INFO:teuthology.orchestra.run.trial185.stdout: "event": "header_read", 2026-02-23T13:09:53.435 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.721376+0000", 2026-02-23T13:09:53.435 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 0 2026-02-23T13:09:53.435 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.435 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.435 INFO:teuthology.orchestra.run.trial185.stdout: "event": "throttled", 2026-02-23T13:09:53.435 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.721382+0000", 2026-02-23T13:09:53.436 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 5.3000000000000001e-06 2026-02-23T13:09:53.436 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.436 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.436 INFO:teuthology.orchestra.run.trial185.stdout: "event": "all_read", 2026-02-23T13:09:53.436 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.721406+0000", 2026-02-23T13:09:53.436 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 2.4295e-05 2026-02-23T13:09:53.436 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.436 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.436 INFO:teuthology.orchestra.run.trial185.stdout: "event": "dispatched", 2026-02-23T13:09:53.436 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.721410+0000", 2026-02-23T13:09:53.436 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 3.9979999999999997e-06 2026-02-23T13:09:53.436 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.437 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.437 INFO:teuthology.orchestra.run.trial185.stdout: "event": "queued_for_pg", 2026-02-23T13:09:53.437 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.721412+0000", 2026-02-23T13:09:53.437 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 2.3740000000000001e-06 2026-02-23T13:09:53.437 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.437 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.437 INFO:teuthology.orchestra.run.trial185.stdout: "event": "reached_pg", 2026-02-23T13:09:53.437 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.721430+0000", 2026-02-23T13:09:53.437 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 1.7833e-05 2026-02-23T13:09:53.437 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.437 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.437 INFO:teuthology.orchestra.run.trial185.stdout: "event": "waiting for peered", 2026-02-23T13:09:53.437 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.721432+0000", 2026-02-23T13:09:53.438 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 1.9939999999999999e-06 2026-02-23T13:09:53.438 INFO:teuthology.orchestra.run.trial185.stdout: } 2026-02-23T13:09:53.438 INFO:teuthology.orchestra.run.trial185.stdout: ] 2026-02-23T13:09:53.438 INFO:teuthology.orchestra.run.trial185.stdout: } 2026-02-23T13:09:53.438 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.438 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.438 INFO:teuthology.orchestra.run.trial185.stdout: "description": "osd_op(client.14522.0:79184 3.c 3.531797fc (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e80)", 2026-02-23T13:09:53.438 INFO:teuthology.orchestra.run.trial185.stdout: "initiated_at": "2026-02-23T13:09:52.717430+0000", 2026-02-23T13:09:53.438 INFO:teuthology.orchestra.run.trial185.stdout: "age": 0.68196613900000003, 2026-02-23T13:09:53.438 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 0.68208670400000004, 2026-02-23T13:09:53.438 INFO:teuthology.orchestra.run.trial185.stdout: "continuous": false, 2026-02-23T13:09:53.438 INFO:teuthology.orchestra.run.trial185.stdout: "type_data": { 2026-02-23T13:09:53.439 INFO:teuthology.orchestra.run.trial185.stdout: "flag_point": "waiting for peered", 2026-02-23T13:09:53.439 INFO:teuthology.orchestra.run.trial185.stdout: "client_info": { 2026-02-23T13:09:53.439 INFO:teuthology.orchestra.run.trial185.stdout: "client": "client.14522", 2026-02-23T13:09:53.439 INFO:teuthology.orchestra.run.trial185.stdout: "client_addr": "10.20.193.68:0/2327563370", 2026-02-23T13:09:53.439 INFO:teuthology.orchestra.run.trial185.stdout: "tid": 79184 2026-02-23T13:09:53.439 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.439 INFO:teuthology.orchestra.run.trial185.stdout: "events": [ 2026-02-23T13:09:53.439 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.439 INFO:teuthology.orchestra.run.trial185.stdout: "event": "initiated", 2026-02-23T13:09:53.439 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.717430+0000", 2026-02-23T13:09:53.439 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 0 2026-02-23T13:09:53.439 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.439 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.440 INFO:teuthology.orchestra.run.trial185.stdout: "event": "header_read", 2026-02-23T13:09:53.440 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.717430+0000", 2026-02-23T13:09:53.440 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 0 2026-02-23T13:09:53.440 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.440 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.440 INFO:teuthology.orchestra.run.trial185.stdout: "event": "throttled", 2026-02-23T13:09:53.440 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.717439+0000", 2026-02-23T13:09:53.440 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 8.2749999999999995e-06 2026-02-23T13:09:53.440 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.440 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.440 INFO:teuthology.orchestra.run.trial185.stdout: "event": "all_read", 2026-02-23T13:09:53.440 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.717464+0000", 2026-02-23T13:09:53.440 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 2.5097e-05 2026-02-23T13:09:53.441 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.441 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.441 INFO:teuthology.orchestra.run.trial185.stdout: "event": "dispatched", 2026-02-23T13:09:53.441 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.717467+0000", 2026-02-23T13:09:53.441 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 3.2260000000000001e-06 2026-02-23T13:09:53.441 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.441 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.441 INFO:teuthology.orchestra.run.trial185.stdout: "event": "queued_for_pg", 2026-02-23T13:09:53.441 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.717469+0000", 2026-02-23T13:09:53.441 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 1.6530000000000001e-06 2026-02-23T13:09:53.441 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.441 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.441 INFO:teuthology.orchestra.run.trial185.stdout: "event": "reached_pg", 2026-02-23T13:09:53.441 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.717494+0000", 2026-02-23T13:09:53.442 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 2.5638000000000001e-05 2026-02-23T13:09:53.442 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.442 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.442 INFO:teuthology.orchestra.run.trial185.stdout: "event": "waiting for peered", 2026-02-23T13:09:53.442 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.717499+0000", 2026-02-23T13:09:53.442 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 4.5680000000000001e-06 2026-02-23T13:09:53.442 INFO:teuthology.orchestra.run.trial185.stdout: } 2026-02-23T13:09:53.442 INFO:teuthology.orchestra.run.trial185.stdout: ] 2026-02-23T13:09:53.442 INFO:teuthology.orchestra.run.trial185.stdout: } 2026-02-23T13:09:53.442 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.442 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.442 INFO:teuthology.orchestra.run.trial185.stdout: "description": "osd_op(client.14522.0:79394 3.d 3.981104cd (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e80)", 2026-02-23T13:09:53.442 INFO:teuthology.orchestra.run.trial185.stdout: "initiated_at": "2026-02-23T13:09:52.722230+0000", 2026-02-23T13:09:53.442 INFO:teuthology.orchestra.run.trial185.stdout: "age": 0.677166135, 2026-02-23T13:09:53.442 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 0.67729895299999998, 2026-02-23T13:09:53.442 INFO:teuthology.orchestra.run.trial185.stdout: "continuous": false, 2026-02-23T13:09:53.443 INFO:teuthology.orchestra.run.trial185.stdout: "type_data": { 2026-02-23T13:09:53.443 INFO:teuthology.orchestra.run.trial185.stdout: "flag_point": "waiting for peered", 2026-02-23T13:09:53.443 INFO:teuthology.orchestra.run.trial185.stdout: "client_info": { 2026-02-23T13:09:53.443 INFO:teuthology.orchestra.run.trial185.stdout: "client": "client.14522", 2026-02-23T13:09:53.443 INFO:teuthology.orchestra.run.trial185.stdout: "client_addr": "10.20.193.68:0/2327563370", 2026-02-23T13:09:53.443 INFO:teuthology.orchestra.run.trial185.stdout: "tid": 79394 2026-02-23T13:09:53.443 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.443 INFO:teuthology.orchestra.run.trial185.stdout: "events": [ 2026-02-23T13:09:53.443 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.443 INFO:teuthology.orchestra.run.trial185.stdout: "event": "initiated", 2026-02-23T13:09:53.443 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.722230+0000", 2026-02-23T13:09:53.443 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 0 2026-02-23T13:09:53.443 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.443 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.443 INFO:teuthology.orchestra.run.trial185.stdout: "event": "header_read", 2026-02-23T13:09:53.444 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.722230+0000", 2026-02-23T13:09:53.444 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 0 2026-02-23T13:09:53.444 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.444 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.444 INFO:teuthology.orchestra.run.trial185.stdout: "event": "throttled", 2026-02-23T13:09:53.444 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.722239+0000", 2026-02-23T13:09:53.444 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 8.0350000000000001e-06 2026-02-23T13:09:53.444 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.444 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.444 INFO:teuthology.orchestra.run.trial185.stdout: "event": "all_read", 2026-02-23T13:09:53.444 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.722268+0000", 2026-02-23T13:09:53.444 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 2.9374999999999999e-05 2026-02-23T13:09:53.444 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.444 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.444 INFO:teuthology.orchestra.run.trial185.stdout: "event": "dispatched", 2026-02-23T13:09:53.445 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.722273+0000", 2026-02-23T13:09:53.445 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 5.1399999999999999e-06 2026-02-23T13:09:53.445 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.445 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.445 INFO:teuthology.orchestra.run.trial185.stdout: "event": "queued_for_pg", 2026-02-23T13:09:53.445 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.722275+0000", 2026-02-23T13:09:53.445 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 1.502e-06 2026-02-23T13:09:53.445 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.445 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.445 INFO:teuthology.orchestra.run.trial185.stdout: "event": "reached_pg", 2026-02-23T13:09:53.445 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.722298+0000", 2026-02-23T13:09:53.445 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 2.3815000000000002e-05 2026-02-23T13:09:53.445 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.445 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.445 INFO:teuthology.orchestra.run.trial185.stdout: "event": "waiting for peered", 2026-02-23T13:09:53.445 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.722300+0000", 2026-02-23T13:09:53.446 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 2.114e-06 2026-02-23T13:09:53.446 INFO:teuthology.orchestra.run.trial185.stdout: } 2026-02-23T13:09:53.446 INFO:teuthology.orchestra.run.trial185.stdout: ] 2026-02-23T13:09:53.446 INFO:teuthology.orchestra.run.trial185.stdout: } 2026-02-23T13:09:53.446 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.446 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.446 INFO:teuthology.orchestra.run.trial185.stdout: "description": "osd_op(client.14522.0:79209 3.9 3.cb539f29 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e80)", 2026-02-23T13:09:53.446 INFO:teuthology.orchestra.run.trial185.stdout: "initiated_at": "2026-02-23T13:09:52.717476+0000", 2026-02-23T13:09:53.446 INFO:teuthology.orchestra.run.trial185.stdout: "age": 0.68192066500000004, 2026-02-23T13:09:53.446 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 0.68206648700000005, 2026-02-23T13:09:53.446 INFO:teuthology.orchestra.run.trial185.stdout: "continuous": false, 2026-02-23T13:09:53.446 INFO:teuthology.orchestra.run.trial185.stdout: "type_data": { 2026-02-23T13:09:53.446 INFO:teuthology.orchestra.run.trial185.stdout: "flag_point": "waiting for peered", 2026-02-23T13:09:53.446 INFO:teuthology.orchestra.run.trial185.stdout: "client_info": { 2026-02-23T13:09:53.446 INFO:teuthology.orchestra.run.trial185.stdout: "client": "client.14522", 2026-02-23T13:09:53.446 INFO:teuthology.orchestra.run.trial185.stdout: "client_addr": "10.20.193.68:0/2327563370", 2026-02-23T13:09:53.446 INFO:teuthology.orchestra.run.trial185.stdout: "tid": 79209 2026-02-23T13:09:53.447 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.447 INFO:teuthology.orchestra.run.trial185.stdout: "events": [ 2026-02-23T13:09:53.447 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.447 INFO:teuthology.orchestra.run.trial185.stdout: "event": "initiated", 2026-02-23T13:09:53.447 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.717476+0000", 2026-02-23T13:09:53.447 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 0 2026-02-23T13:09:53.447 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.447 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.447 INFO:teuthology.orchestra.run.trial185.stdout: "event": "header_read", 2026-02-23T13:09:53.447 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.717476+0000", 2026-02-23T13:09:53.447 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 0 2026-02-23T13:09:53.447 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.447 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.447 INFO:teuthology.orchestra.run.trial185.stdout: "event": "throttled", 2026-02-23T13:09:53.447 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.717489+0000", 2026-02-23T13:09:53.447 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 1.2904999999999999e-05 2026-02-23T13:09:53.447 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.448 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.448 INFO:teuthology.orchestra.run.trial185.stdout: "event": "all_read", 2026-02-23T13:09:53.448 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.717515+0000", 2026-02-23T13:09:53.448 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 2.6378999999999999e-05 2026-02-23T13:09:53.448 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.448 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.448 INFO:teuthology.orchestra.run.trial185.stdout: "event": "dispatched", 2026-02-23T13:09:53.448 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.717519+0000", 2026-02-23T13:09:53.448 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 3.3859999999999999e-06 2026-02-23T13:09:53.448 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.448 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.448 INFO:teuthology.orchestra.run.trial185.stdout: "event": "queued_for_pg", 2026-02-23T13:09:53.448 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.717520+0000", 2026-02-23T13:09:53.448 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 1.4330000000000001e-06 2026-02-23T13:09:53.448 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.448 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.448 INFO:teuthology.orchestra.run.trial185.stdout: "event": "reached_pg", 2026-02-23T13:09:53.449 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.717537+0000", 2026-02-23T13:09:53.449 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 1.6731e-05 2026-02-23T13:09:53.449 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.449 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.449 INFO:teuthology.orchestra.run.trial185.stdout: "event": "waiting for peered", 2026-02-23T13:09:53.449 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.717540+0000", 2026-02-23T13:09:53.449 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 3.6069999999999999e-06 2026-02-23T13:09:53.449 INFO:teuthology.orchestra.run.trial185.stdout: } 2026-02-23T13:09:53.449 INFO:teuthology.orchestra.run.trial185.stdout: ] 2026-02-23T13:09:53.449 INFO:teuthology.orchestra.run.trial185.stdout: } 2026-02-23T13:09:53.449 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.449 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.449 INFO:teuthology.orchestra.run.trial185.stdout: "description": "osd_op(client.14522.0:79395 3.d 3.981104cd (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e80)", 2026-02-23T13:09:53.449 INFO:teuthology.orchestra.run.trial185.stdout: "initiated_at": "2026-02-23T13:09:52.722288+0000", 2026-02-23T13:09:53.449 INFO:teuthology.orchestra.run.trial185.stdout: "age": 0.677108347, 2026-02-23T13:09:53.449 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 0.67726704299999996, 2026-02-23T13:09:53.449 INFO:teuthology.orchestra.run.trial185.stdout: "continuous": false, 2026-02-23T13:09:53.450 INFO:teuthology.orchestra.run.trial185.stdout: "type_data": { 2026-02-23T13:09:53.450 INFO:teuthology.orchestra.run.trial185.stdout: "flag_point": "waiting for peered", 2026-02-23T13:09:53.450 INFO:teuthology.orchestra.run.trial185.stdout: "client_info": { 2026-02-23T13:09:53.450 INFO:teuthology.orchestra.run.trial185.stdout: "client": "client.14522", 2026-02-23T13:09:53.450 INFO:teuthology.orchestra.run.trial185.stdout: "client_addr": "10.20.193.68:0/2327563370", 2026-02-23T13:09:53.450 INFO:teuthology.orchestra.run.trial185.stdout: "tid": 79395 2026-02-23T13:09:53.450 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.450 INFO:teuthology.orchestra.run.trial185.stdout: "events": [ 2026-02-23T13:09:53.450 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.450 INFO:teuthology.orchestra.run.trial185.stdout: "event": "initiated", 2026-02-23T13:09:53.450 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.722288+0000", 2026-02-23T13:09:53.450 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 0 2026-02-23T13:09:53.450 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.450 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.450 INFO:teuthology.orchestra.run.trial185.stdout: "event": "header_read", 2026-02-23T13:09:53.450 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.722288+0000", 2026-02-23T13:09:53.450 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 0 2026-02-23T13:09:53.451 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.451 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.451 INFO:teuthology.orchestra.run.trial185.stdout: "event": "throttled", 2026-02-23T13:09:53.451 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.722305+0000", 2026-02-23T13:09:53.451 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 1.7051999999999999e-05 2026-02-23T13:09:53.451 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.451 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.451 INFO:teuthology.orchestra.run.trial185.stdout: "event": "all_read", 2026-02-23T13:09:53.451 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.722337+0000", 2026-02-23T13:09:53.451 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 3.201e-05 2026-02-23T13:09:53.451 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.451 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.451 INFO:teuthology.orchestra.run.trial185.stdout: "event": "dispatched", 2026-02-23T13:09:53.451 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.722343+0000", 2026-02-23T13:09:53.451 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 5.49e-06 2026-02-23T13:09:53.451 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.451 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.452 INFO:teuthology.orchestra.run.trial185.stdout: "event": "queued_for_pg", 2026-02-23T13:09:53.452 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.722345+0000", 2026-02-23T13:09:53.452 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 1.823e-06 2026-02-23T13:09:53.452 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.452 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.452 INFO:teuthology.orchestra.run.trial185.stdout: "event": "reached_pg", 2026-02-23T13:09:53.452 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.722362+0000", 2026-02-23T13:09:53.452 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 1.7172000000000001e-05 2026-02-23T13:09:53.452 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.452 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.452 INFO:teuthology.orchestra.run.trial185.stdout: "event": "waiting for peered", 2026-02-23T13:09:53.452 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.722364+0000", 2026-02-23T13:09:53.452 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 2.0140000000000001e-06 2026-02-23T13:09:53.452 INFO:teuthology.orchestra.run.trial185.stdout: } 2026-02-23T13:09:53.452 INFO:teuthology.orchestra.run.trial185.stdout: ] 2026-02-23T13:09:53.452 INFO:teuthology.orchestra.run.trial185.stdout: } 2026-02-23T13:09:53.452 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.452 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.453 INFO:teuthology.orchestra.run.trial185.stdout: "description": "osd_op(client.14522.0:79210 3.9 3.cb539f29 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e80)", 2026-02-23T13:09:53.453 INFO:teuthology.orchestra.run.trial185.stdout: "initiated_at": "2026-02-23T13:09:52.717529+0000", 2026-02-23T13:09:53.453 INFO:teuthology.orchestra.run.trial185.stdout: "age": 0.68186777600000004, 2026-02-23T13:09:53.453 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 0.68203957599999998, 2026-02-23T13:09:53.453 INFO:teuthology.orchestra.run.trial185.stdout: "continuous": false, 2026-02-23T13:09:53.453 INFO:teuthology.orchestra.run.trial185.stdout: "type_data": { 2026-02-23T13:09:53.453 INFO:teuthology.orchestra.run.trial185.stdout: "flag_point": "waiting for peered", 2026-02-23T13:09:53.453 INFO:teuthology.orchestra.run.trial185.stdout: "client_info": { 2026-02-23T13:09:53.453 INFO:teuthology.orchestra.run.trial185.stdout: "client": "client.14522", 2026-02-23T13:09:53.453 INFO:teuthology.orchestra.run.trial185.stdout: "client_addr": "10.20.193.68:0/2327563370", 2026-02-23T13:09:53.453 INFO:teuthology.orchestra.run.trial185.stdout: "tid": 79210 2026-02-23T13:09:53.453 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.453 INFO:teuthology.orchestra.run.trial185.stdout: "events": [ 2026-02-23T13:09:53.453 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.453 INFO:teuthology.orchestra.run.trial185.stdout: "event": "initiated", 2026-02-23T13:09:53.453 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.717529+0000", 2026-02-23T13:09:53.453 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 0 2026-02-23T13:09:53.453 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.454 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.454 INFO:teuthology.orchestra.run.trial185.stdout: "event": "header_read", 2026-02-23T13:09:53.454 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.717529+0000", 2026-02-23T13:09:53.454 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 0 2026-02-23T13:09:53.454 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.454 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.454 INFO:teuthology.orchestra.run.trial185.stdout: "event": "throttled", 2026-02-23T13:09:53.454 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.717536+0000", 2026-02-23T13:09:53.454 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 6.793e-06 2026-02-23T13:09:53.454 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.454 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.454 INFO:teuthology.orchestra.run.trial185.stdout: "event": "all_read", 2026-02-23T13:09:53.454 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.717557+0000", 2026-02-23T13:09:53.454 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 2.1158999999999999e-05 2026-02-23T13:09:53.454 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.454 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.454 INFO:teuthology.orchestra.run.trial185.stdout: "event": "dispatched", 2026-02-23T13:09:53.454 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.717561+0000", 2026-02-23T13:09:53.454 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 4.3279999999999999e-06 2026-02-23T13:09:53.454 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.455 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.455 INFO:teuthology.orchestra.run.trial185.stdout: "event": "queued_for_pg", 2026-02-23T13:09:53.455 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.717562+0000", 2026-02-23T13:09:53.455 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 1.3629999999999999e-06 2026-02-23T13:09:53.455 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.455 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.455 INFO:teuthology.orchestra.run.trial185.stdout: "event": "reached_pg", 2026-02-23T13:09:53.455 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.717581+0000", 2026-02-23T13:09:53.455 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 1.8513999999999999e-05 2026-02-23T13:09:53.455 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.455 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.455 INFO:teuthology.orchestra.run.trial185.stdout: "event": "waiting for peered", 2026-02-23T13:09:53.455 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.717584+0000", 2026-02-23T13:09:53.455 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 3.2559999999999998e-06 2026-02-23T13:09:53.455 INFO:teuthology.orchestra.run.trial185.stdout: } 2026-02-23T13:09:53.455 INFO:teuthology.orchestra.run.trial185.stdout: ] 2026-02-23T13:09:53.455 INFO:teuthology.orchestra.run.trial185.stdout: } 2026-02-23T13:09:53.455 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.455 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.455 INFO:teuthology.orchestra.run.trial185.stdout: "description": "osd_op(client.14522.0:79396 3.d 3.981104cd (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e80)", 2026-02-23T13:09:53.456 INFO:teuthology.orchestra.run.trial185.stdout: "initiated_at": "2026-02-23T13:09:52.722358+0000", 2026-02-23T13:09:53.456 INFO:teuthology.orchestra.run.trial185.stdout: "age": 0.67703893699999995, 2026-02-23T13:09:53.456 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 0.67722333099999998, 2026-02-23T13:09:53.456 INFO:teuthology.orchestra.run.trial185.stdout: "continuous": false, 2026-02-23T13:09:53.456 INFO:teuthology.orchestra.run.trial185.stdout: "type_data": { 2026-02-23T13:09:53.456 INFO:teuthology.orchestra.run.trial185.stdout: "flag_point": "waiting for peered", 2026-02-23T13:09:53.456 INFO:teuthology.orchestra.run.trial185.stdout: "client_info": { 2026-02-23T13:09:53.456 INFO:teuthology.orchestra.run.trial185.stdout: "client": "client.14522", 2026-02-23T13:09:53.456 INFO:teuthology.orchestra.run.trial185.stdout: "client_addr": "10.20.193.68:0/2327563370", 2026-02-23T13:09:53.456 INFO:teuthology.orchestra.run.trial185.stdout: "tid": 79396 2026-02-23T13:09:53.456 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.456 INFO:teuthology.orchestra.run.trial185.stdout: "events": [ 2026-02-23T13:09:53.456 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.456 INFO:teuthology.orchestra.run.trial185.stdout: "event": "initiated", 2026-02-23T13:09:53.456 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.722358+0000", 2026-02-23T13:09:53.456 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 0 2026-02-23T13:09:53.456 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.456 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.456 INFO:teuthology.orchestra.run.trial185.stdout: "event": "header_read", 2026-02-23T13:09:53.456 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.722358+0000", 2026-02-23T13:09:53.457 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 0 2026-02-23T13:09:53.457 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.457 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.457 INFO:teuthology.orchestra.run.trial185.stdout: "event": "throttled", 2026-02-23T13:09:53.457 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.722369+0000", 2026-02-23T13:09:53.457 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 1.1681e-05 2026-02-23T13:09:53.457 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.457 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.457 INFO:teuthology.orchestra.run.trial185.stdout: "event": "all_read", 2026-02-23T13:09:53.457 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.722396+0000", 2026-02-23T13:09:53.457 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 2.6380000000000002e-05 2026-02-23T13:09:53.457 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.457 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.457 INFO:teuthology.orchestra.run.trial185.stdout: "event": "dispatched", 2026-02-23T13:09:53.457 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.722399+0000", 2026-02-23T13:09:53.457 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 3.456e-06 2026-02-23T13:09:53.457 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.457 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.457 INFO:teuthology.orchestra.run.trial185.stdout: "event": "queued_for_pg", 2026-02-23T13:09:53.457 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.722401+0000", 2026-02-23T13:09:53.457 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 1.3120000000000001e-06 2026-02-23T13:09:53.458 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.458 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.458 INFO:teuthology.orchestra.run.trial185.stdout: "event": "reached_pg", 2026-02-23T13:09:53.458 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.722415+0000", 2026-02-23T13:09:53.458 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 1.4848000000000001e-05 2026-02-23T13:09:53.458 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.458 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.458 INFO:teuthology.orchestra.run.trial185.stdout: "event": "waiting for peered", 2026-02-23T13:09:53.458 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.722418+0000", 2026-02-23T13:09:53.458 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 3.146e-06 2026-02-23T13:09:53.458 INFO:teuthology.orchestra.run.trial185.stdout: } 2026-02-23T13:09:53.458 INFO:teuthology.orchestra.run.trial185.stdout: ] 2026-02-23T13:09:53.458 INFO:teuthology.orchestra.run.trial185.stdout: } 2026-02-23T13:09:53.458 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.458 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.458 INFO:teuthology.orchestra.run.trial185.stdout: "description": "osd_op(client.14522.0:79211 3.9 3.cb539f29 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e80)", 2026-02-23T13:09:53.458 INFO:teuthology.orchestra.run.trial185.stdout: "initiated_at": "2026-02-23T13:09:52.717572+0000", 2026-02-23T13:09:53.458 INFO:teuthology.orchestra.run.trial185.stdout: "age": 0.68182468500000004, 2026-02-23T13:09:53.458 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 0.68202136199999996, 2026-02-23T13:09:53.458 INFO:teuthology.orchestra.run.trial185.stdout: "continuous": false, 2026-02-23T13:09:53.458 INFO:teuthology.orchestra.run.trial185.stdout: "type_data": { 2026-02-23T13:09:53.458 INFO:teuthology.orchestra.run.trial185.stdout: "flag_point": "waiting for peered", 2026-02-23T13:09:53.459 INFO:teuthology.orchestra.run.trial185.stdout: "client_info": { 2026-02-23T13:09:53.459 INFO:teuthology.orchestra.run.trial185.stdout: "client": "client.14522", 2026-02-23T13:09:53.459 INFO:teuthology.orchestra.run.trial185.stdout: "client_addr": "10.20.193.68:0/2327563370", 2026-02-23T13:09:53.459 INFO:teuthology.orchestra.run.trial185.stdout: "tid": 79211 2026-02-23T13:09:53.459 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.459 INFO:teuthology.orchestra.run.trial185.stdout: "events": [ 2026-02-23T13:09:53.459 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.459 INFO:teuthology.orchestra.run.trial185.stdout: "event": "initiated", 2026-02-23T13:09:53.459 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.717572+0000", 2026-02-23T13:09:53.459 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 0 2026-02-23T13:09:53.459 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.459 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.459 INFO:teuthology.orchestra.run.trial185.stdout: "event": "header_read", 2026-02-23T13:09:53.459 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.717572+0000", 2026-02-23T13:09:53.459 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 0 2026-02-23T13:09:53.459 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.459 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.459 INFO:teuthology.orchestra.run.trial185.stdout: "event": "throttled", 2026-02-23T13:09:53.459 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.717579+0000", 2026-02-23T13:09:53.459 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 7.1829999999999997e-06 2026-02-23T13:09:53.459 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.459 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.460 INFO:teuthology.orchestra.run.trial185.stdout: "event": "all_read", 2026-02-23T13:09:53.460 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.717623+0000", 2026-02-23T13:09:53.460 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 4.4061999999999999e-05 2026-02-23T13:09:53.460 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.460 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.460 INFO:teuthology.orchestra.run.trial185.stdout: "event": "dispatched", 2026-02-23T13:09:53.460 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.717628+0000", 2026-02-23T13:09:53.460 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 4.7790000000000003e-06 2026-02-23T13:09:53.460 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.460 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.460 INFO:teuthology.orchestra.run.trial185.stdout: "event": "queued_for_pg", 2026-02-23T13:09:53.460 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.717631+0000", 2026-02-23T13:09:53.460 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 2.5749999999999999e-06 2026-02-23T13:09:53.460 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.460 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.460 INFO:teuthology.orchestra.run.trial185.stdout: "event": "reached_pg", 2026-02-23T13:09:53.460 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.717648+0000", 2026-02-23T13:09:53.460 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 1.7552999999999999e-05 2026-02-23T13:09:53.460 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.460 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.460 INFO:teuthology.orchestra.run.trial185.stdout: "event": "waiting for peered", 2026-02-23T13:09:53.461 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.717650+0000", 2026-02-23T13:09:53.461 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 2.3640000000000002e-06 2026-02-23T13:09:53.461 INFO:teuthology.orchestra.run.trial185.stdout: } 2026-02-23T13:09:53.461 INFO:teuthology.orchestra.run.trial185.stdout: ] 2026-02-23T13:09:53.461 INFO:teuthology.orchestra.run.trial185.stdout: } 2026-02-23T13:09:53.461 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.461 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.461 INFO:teuthology.orchestra.run.trial185.stdout: "description": "osd_op(client.14522.0:79397 3.d 3.981104cd (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e80)", 2026-02-23T13:09:53.461 INFO:teuthology.orchestra.run.trial185.stdout: "initiated_at": "2026-02-23T13:09:52.722416+0000", 2026-02-23T13:09:53.461 INFO:teuthology.orchestra.run.trial185.stdout: "age": 0.67698108000000001, 2026-02-23T13:09:53.461 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 0.67719058099999996, 2026-02-23T13:09:53.461 INFO:teuthology.orchestra.run.trial185.stdout: "continuous": false, 2026-02-23T13:09:53.461 INFO:teuthology.orchestra.run.trial185.stdout: "type_data": { 2026-02-23T13:09:53.461 INFO:teuthology.orchestra.run.trial185.stdout: "flag_point": "waiting for peered", 2026-02-23T13:09:53.461 INFO:teuthology.orchestra.run.trial185.stdout: "client_info": { 2026-02-23T13:09:53.461 INFO:teuthology.orchestra.run.trial185.stdout: "client": "client.14522", 2026-02-23T13:09:53.461 INFO:teuthology.orchestra.run.trial185.stdout: "client_addr": "10.20.193.68:0/2327563370", 2026-02-23T13:09:53.461 INFO:teuthology.orchestra.run.trial185.stdout: "tid": 79397 2026-02-23T13:09:53.461 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.461 INFO:teuthology.orchestra.run.trial185.stdout: "events": [ 2026-02-23T13:09:53.462 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.462 INFO:teuthology.orchestra.run.trial185.stdout: "event": "initiated", 2026-02-23T13:09:53.462 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.722416+0000", 2026-02-23T13:09:53.462 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 0 2026-02-23T13:09:53.462 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.462 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.462 INFO:teuthology.orchestra.run.trial185.stdout: "event": "header_read", 2026-02-23T13:09:53.462 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.722416+0000", 2026-02-23T13:09:53.462 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 0 2026-02-23T13:09:53.462 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.462 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.462 INFO:teuthology.orchestra.run.trial185.stdout: "event": "throttled", 2026-02-23T13:09:53.462 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.722425+0000", 2026-02-23T13:09:53.462 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 9.3680000000000008e-06 2026-02-23T13:09:53.462 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.462 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.462 INFO:teuthology.orchestra.run.trial185.stdout: "event": "all_read", 2026-02-23T13:09:53.462 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.722453+0000", 2026-02-23T13:09:53.462 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 2.8161999999999999e-05 2026-02-23T13:09:53.462 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.462 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.463 INFO:teuthology.orchestra.run.trial185.stdout: "event": "dispatched", 2026-02-23T13:09:53.463 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.722457+0000", 2026-02-23T13:09:53.463 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 3.647e-06 2026-02-23T13:09:53.463 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.463 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.463 INFO:teuthology.orchestra.run.trial185.stdout: "event": "queued_for_pg", 2026-02-23T13:09:53.463 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.722458+0000", 2026-02-23T13:09:53.463 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 1.3629999999999999e-06 2026-02-23T13:09:53.463 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.463 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.463 INFO:teuthology.orchestra.run.trial185.stdout: "event": "reached_pg", 2026-02-23T13:09:53.463 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.722478+0000", 2026-02-23T13:09:53.463 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 2.0126999999999999e-05 2026-02-23T13:09:53.463 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.463 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.463 INFO:teuthology.orchestra.run.trial185.stdout: "event": "waiting for peered", 2026-02-23T13:09:53.463 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.722481+0000", 2026-02-23T13:09:53.463 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 2.3250000000000002e-06 2026-02-23T13:09:53.463 INFO:teuthology.orchestra.run.trial185.stdout: } 2026-02-23T13:09:53.463 INFO:teuthology.orchestra.run.trial185.stdout: ] 2026-02-23T13:09:53.463 INFO:teuthology.orchestra.run.trial185.stdout: } 2026-02-23T13:09:53.463 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.463 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.464 INFO:teuthology.orchestra.run.trial185.stdout: "description": "osd_op(client.14522.0:79212 3.9 3.cb539f29 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e80)", 2026-02-23T13:09:53.464 INFO:teuthology.orchestra.run.trial185.stdout: "initiated_at": "2026-02-23T13:09:52.717640+0000", 2026-02-23T13:09:53.464 INFO:teuthology.orchestra.run.trial185.stdout: "age": 0.68175625799999995, 2026-02-23T13:09:53.464 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 0.68197816200000005, 2026-02-23T13:09:53.464 INFO:teuthology.orchestra.run.trial185.stdout: "continuous": false, 2026-02-23T13:09:53.464 INFO:teuthology.orchestra.run.trial185.stdout: "type_data": { 2026-02-23T13:09:53.464 INFO:teuthology.orchestra.run.trial185.stdout: "flag_point": "waiting for peered", 2026-02-23T13:09:53.464 INFO:teuthology.orchestra.run.trial185.stdout: "client_info": { 2026-02-23T13:09:53.464 INFO:teuthology.orchestra.run.trial185.stdout: "client": "client.14522", 2026-02-23T13:09:53.464 INFO:teuthology.orchestra.run.trial185.stdout: "client_addr": "10.20.193.68:0/2327563370", 2026-02-23T13:09:53.464 INFO:teuthology.orchestra.run.trial185.stdout: "tid": 79212 2026-02-23T13:09:53.464 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.464 INFO:teuthology.orchestra.run.trial185.stdout: "events": [ 2026-02-23T13:09:53.464 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.464 INFO:teuthology.orchestra.run.trial185.stdout: "event": "initiated", 2026-02-23T13:09:53.464 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.717640+0000", 2026-02-23T13:09:53.464 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 0 2026-02-23T13:09:53.464 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.464 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.464 INFO:teuthology.orchestra.run.trial185.stdout: "event": "header_read", 2026-02-23T13:09:53.464 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.717640+0000", 2026-02-23T13:09:53.464 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 0 2026-02-23T13:09:53.464 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.465 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.465 INFO:teuthology.orchestra.run.trial185.stdout: "event": "throttled", 2026-02-23T13:09:53.465 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.717649+0000", 2026-02-23T13:09:53.465 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 8.5660000000000007e-06 2026-02-23T13:09:53.465 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.465 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.465 INFO:teuthology.orchestra.run.trial185.stdout: "event": "all_read", 2026-02-23T13:09:53.465 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.717670+0000", 2026-02-23T13:09:53.465 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 2.1120000000000001e-05 2026-02-23T13:09:53.465 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.465 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.465 INFO:teuthology.orchestra.run.trial185.stdout: "event": "dispatched", 2026-02-23T13:09:53.465 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.717673+0000", 2026-02-23T13:09:53.465 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 3.2459999999999999e-06 2026-02-23T13:09:53.465 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.465 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.465 INFO:teuthology.orchestra.run.trial185.stdout: "event": "queued_for_pg", 2026-02-23T13:09:53.465 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.717675+0000", 2026-02-23T13:09:53.465 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 1.3120000000000001e-06 2026-02-23T13:09:53.465 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.465 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.465 INFO:teuthology.orchestra.run.trial185.stdout: "event": "reached_pg", 2026-02-23T13:09:53.465 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.717690+0000", 2026-02-23T13:09:53.465 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 1.5319e-05 2026-02-23T13:09:53.466 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.466 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.466 INFO:teuthology.orchestra.run.trial185.stdout: "event": "waiting for peered", 2026-02-23T13:09:53.466 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.717692+0000", 2026-02-23T13:09:53.466 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 2.244e-06 2026-02-23T13:09:53.466 INFO:teuthology.orchestra.run.trial185.stdout: } 2026-02-23T13:09:53.466 INFO:teuthology.orchestra.run.trial185.stdout: ] 2026-02-23T13:09:53.466 INFO:teuthology.orchestra.run.trial185.stdout: } 2026-02-23T13:09:53.466 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.466 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.466 INFO:teuthology.orchestra.run.trial185.stdout: "description": "osd_op(client.14522.0:79398 3.d 3.981104cd (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e80)", 2026-02-23T13:09:53.466 INFO:teuthology.orchestra.run.trial185.stdout: "initiated_at": "2026-02-23T13:09:52.722470+0000", 2026-02-23T13:09:53.466 INFO:teuthology.orchestra.run.trial185.stdout: "age": 0.67692618699999996, 2026-02-23T13:09:53.466 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 0.67716279800000001, 2026-02-23T13:09:53.466 INFO:teuthology.orchestra.run.trial185.stdout: "continuous": false, 2026-02-23T13:09:53.466 INFO:teuthology.orchestra.run.trial185.stdout: "type_data": { 2026-02-23T13:09:53.466 INFO:teuthology.orchestra.run.trial185.stdout: "flag_point": "waiting for peered", 2026-02-23T13:09:53.466 INFO:teuthology.orchestra.run.trial185.stdout: "client_info": { 2026-02-23T13:09:53.466 INFO:teuthology.orchestra.run.trial185.stdout: "client": "client.14522", 2026-02-23T13:09:53.466 INFO:teuthology.orchestra.run.trial185.stdout: "client_addr": "10.20.193.68:0/2327563370", 2026-02-23T13:09:53.466 INFO:teuthology.orchestra.run.trial185.stdout: "tid": 79398 2026-02-23T13:09:53.466 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.466 INFO:teuthology.orchestra.run.trial185.stdout: "events": [ 2026-02-23T13:09:53.467 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.467 INFO:teuthology.orchestra.run.trial185.stdout: "event": "initiated", 2026-02-23T13:09:53.467 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.722470+0000", 2026-02-23T13:09:53.467 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 0 2026-02-23T13:09:53.467 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.467 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.467 INFO:teuthology.orchestra.run.trial185.stdout: "event": "header_read", 2026-02-23T13:09:53.467 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.722470+0000", 2026-02-23T13:09:53.467 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 0 2026-02-23T13:09:53.467 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.467 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.467 INFO:teuthology.orchestra.run.trial185.stdout: "event": "throttled", 2026-02-23T13:09:53.467 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.722479+0000", 2026-02-23T13:09:53.467 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 8.4160000000000004e-06 2026-02-23T13:09:53.467 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.467 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.467 INFO:teuthology.orchestra.run.trial185.stdout: "event": "all_read", 2026-02-23T13:09:53.467 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.722504+0000", 2026-02-23T13:09:53.467 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 2.5636999999999999e-05 2026-02-23T13:09:53.467 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.467 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.467 INFO:teuthology.orchestra.run.trial185.stdout: "event": "dispatched", 2026-02-23T13:09:53.467 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.722510+0000", 2026-02-23T13:09:53.467 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 5.6110000000000001e-06 2026-02-23T13:09:53.468 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.468 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.468 INFO:teuthology.orchestra.run.trial185.stdout: "event": "queued_for_pg", 2026-02-23T13:09:53.468 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.722511+0000", 2026-02-23T13:09:53.468 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 1.4020000000000001e-06 2026-02-23T13:09:53.468 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.468 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.468 INFO:teuthology.orchestra.run.trial185.stdout: "event": "reached_pg", 2026-02-23T13:09:53.468 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.722528+0000", 2026-02-23T13:09:53.468 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 1.6591000000000001e-05 2026-02-23T13:09:53.468 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.468 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.468 INFO:teuthology.orchestra.run.trial185.stdout: "event": "waiting for peered", 2026-02-23T13:09:53.468 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.722530+0000", 2026-02-23T13:09:53.468 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 1.7540000000000001e-06 2026-02-23T13:09:53.468 INFO:teuthology.orchestra.run.trial185.stdout: } 2026-02-23T13:09:53.468 INFO:teuthology.orchestra.run.trial185.stdout: ] 2026-02-23T13:09:53.468 INFO:teuthology.orchestra.run.trial185.stdout: } 2026-02-23T13:09:53.468 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.468 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.468 INFO:teuthology.orchestra.run.trial185.stdout: "description": "osd_op(client.14522.0:79213 3.9 3.cb539f29 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e80)", 2026-02-23T13:09:53.468 INFO:teuthology.orchestra.run.trial185.stdout: "initiated_at": "2026-02-23T13:09:52.717683+0000", 2026-02-23T13:09:53.468 INFO:teuthology.orchestra.run.trial185.stdout: "age": 0.68171367800000005, 2026-02-23T13:09:53.469 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 0.68196320300000002, 2026-02-23T13:09:53.469 INFO:teuthology.orchestra.run.trial185.stdout: "continuous": false, 2026-02-23T13:09:53.469 INFO:teuthology.orchestra.run.trial185.stdout: "type_data": { 2026-02-23T13:09:53.469 INFO:teuthology.orchestra.run.trial185.stdout: "flag_point": "waiting for peered", 2026-02-23T13:09:53.469 INFO:teuthology.orchestra.run.trial185.stdout: "client_info": { 2026-02-23T13:09:53.469 INFO:teuthology.orchestra.run.trial185.stdout: "client": "client.14522", 2026-02-23T13:09:53.469 INFO:teuthology.orchestra.run.trial185.stdout: "client_addr": "10.20.193.68:0/2327563370", 2026-02-23T13:09:53.469 INFO:teuthology.orchestra.run.trial185.stdout: "tid": 79213 2026-02-23T13:09:53.469 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.469 INFO:teuthology.orchestra.run.trial185.stdout: "events": [ 2026-02-23T13:09:53.469 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.469 INFO:teuthology.orchestra.run.trial185.stdout: "event": "initiated", 2026-02-23T13:09:53.469 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.717683+0000", 2026-02-23T13:09:53.469 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 0 2026-02-23T13:09:53.469 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.469 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.469 INFO:teuthology.orchestra.run.trial185.stdout: "event": "header_read", 2026-02-23T13:09:53.469 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.717683+0000", 2026-02-23T13:09:53.469 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 0 2026-02-23T13:09:53.469 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.469 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.469 INFO:teuthology.orchestra.run.trial185.stdout: "event": "throttled", 2026-02-23T13:09:53.470 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.717690+0000", 2026-02-23T13:09:53.470 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 7.464e-06 2026-02-23T13:09:53.470 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.470 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.470 INFO:teuthology.orchestra.run.trial185.stdout: "event": "all_read", 2026-02-23T13:09:53.470 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.717714+0000", 2026-02-23T13:09:53.470 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 2.3502999999999998e-05 2026-02-23T13:09:53.470 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.470 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.470 INFO:teuthology.orchestra.run.trial185.stdout: "event": "dispatched", 2026-02-23T13:09:53.470 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.717717+0000", 2026-02-23T13:09:53.470 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 3.287e-06 2026-02-23T13:09:53.470 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.470 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.470 INFO:teuthology.orchestra.run.trial185.stdout: "event": "queued_for_pg", 2026-02-23T13:09:53.470 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.717719+0000", 2026-02-23T13:09:53.470 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 1.4419999999999999e-06 2026-02-23T13:09:53.470 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.470 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.470 INFO:teuthology.orchestra.run.trial185.stdout: "event": "reached_pg", 2026-02-23T13:09:53.470 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.717740+0000", 2026-02-23T13:09:53.470 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 2.109e-05 2026-02-23T13:09:53.470 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.470 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.471 INFO:teuthology.orchestra.run.trial185.stdout: "event": "waiting for peered", 2026-02-23T13:09:53.471 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.717741+0000", 2026-02-23T13:09:53.471 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 1.502e-06 2026-02-23T13:09:53.471 INFO:teuthology.orchestra.run.trial185.stdout: } 2026-02-23T13:09:53.471 INFO:teuthology.orchestra.run.trial185.stdout: ] 2026-02-23T13:09:53.471 INFO:teuthology.orchestra.run.trial185.stdout: } 2026-02-23T13:09:53.471 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.471 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.471 INFO:teuthology.orchestra.run.trial185.stdout: "description": "osd_op(client.14522.0:79399 3.d 3.981104cd (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e80)", 2026-02-23T13:09:53.471 INFO:teuthology.orchestra.run.trial185.stdout: "initiated_at": "2026-02-23T13:09:52.722724+0000", 2026-02-23T13:09:53.471 INFO:teuthology.orchestra.run.trial185.stdout: "age": 0.67667233400000004, 2026-02-23T13:09:53.471 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 0.67693630599999999, 2026-02-23T13:09:53.471 INFO:teuthology.orchestra.run.trial185.stdout: "continuous": false, 2026-02-23T13:09:53.471 INFO:teuthology.orchestra.run.trial185.stdout: "type_data": { 2026-02-23T13:09:53.471 INFO:teuthology.orchestra.run.trial185.stdout: "flag_point": "waiting for peered", 2026-02-23T13:09:53.471 INFO:teuthology.orchestra.run.trial185.stdout: "client_info": { 2026-02-23T13:09:53.471 INFO:teuthology.orchestra.run.trial185.stdout: "client": "client.14522", 2026-02-23T13:09:53.471 INFO:teuthology.orchestra.run.trial185.stdout: "client_addr": "10.20.193.68:0/2327563370", 2026-02-23T13:09:53.471 INFO:teuthology.orchestra.run.trial185.stdout: "tid": 79399 2026-02-23T13:09:53.471 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.471 INFO:teuthology.orchestra.run.trial185.stdout: "events": [ 2026-02-23T13:09:53.471 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.471 INFO:teuthology.orchestra.run.trial185.stdout: "event": "initiated", 2026-02-23T13:09:53.472 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.722724+0000", 2026-02-23T13:09:53.472 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 0 2026-02-23T13:09:53.472 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.472 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.472 INFO:teuthology.orchestra.run.trial185.stdout: "event": "header_read", 2026-02-23T13:09:53.472 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.722724+0000", 2026-02-23T13:09:53.472 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 0 2026-02-23T13:09:53.472 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.472 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.472 INFO:teuthology.orchestra.run.trial185.stdout: "event": "throttled", 2026-02-23T13:09:53.472 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.722732+0000", 2026-02-23T13:09:53.472 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 7.7749999999999993e-06 2026-02-23T13:09:53.472 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.472 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.472 INFO:teuthology.orchestra.run.trial185.stdout: "event": "all_read", 2026-02-23T13:09:53.472 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.722761+0000", 2026-02-23T13:09:53.472 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 2.8864e-05 2026-02-23T13:09:53.472 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.472 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.472 INFO:teuthology.orchestra.run.trial185.stdout: "event": "dispatched", 2026-02-23T13:09:53.472 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.722766+0000", 2026-02-23T13:09:53.472 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 4.9289999999999997e-06 2026-02-23T13:09:53.472 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.473 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.473 INFO:teuthology.orchestra.run.trial185.stdout: "event": "queued_for_pg", 2026-02-23T13:09:53.473 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.722767+0000", 2026-02-23T13:09:53.473 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 1.322e-06 2026-02-23T13:09:53.473 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.473 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.473 INFO:teuthology.orchestra.run.trial185.stdout: "event": "reached_pg", 2026-02-23T13:09:53.473 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.722784+0000", 2026-02-23T13:09:53.473 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 1.7031999999999999e-05 2026-02-23T13:09:53.473 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.473 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.473 INFO:teuthology.orchestra.run.trial185.stdout: "event": "waiting for peered", 2026-02-23T13:09:53.473 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.722788+0000", 2026-02-23T13:09:53.473 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 3.326e-06 2026-02-23T13:09:53.473 INFO:teuthology.orchestra.run.trial185.stdout: } 2026-02-23T13:09:53.473 INFO:teuthology.orchestra.run.trial185.stdout: ] 2026-02-23T13:09:53.473 INFO:teuthology.orchestra.run.trial185.stdout: } 2026-02-23T13:09:53.473 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.473 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.473 INFO:teuthology.orchestra.run.trial185.stdout: "description": "osd_op(client.14522.0:79214 3.9 3.cb539f29 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e80)", 2026-02-23T13:09:53.473 INFO:teuthology.orchestra.run.trial185.stdout: "initiated_at": "2026-02-23T13:09:52.717730+0000", 2026-02-23T13:09:53.473 INFO:teuthology.orchestra.run.trial185.stdout: "age": 0.68166671099999998, 2026-02-23T13:09:53.473 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 0.68194452900000002, 2026-02-23T13:09:53.474 INFO:teuthology.orchestra.run.trial185.stdout: "continuous": false, 2026-02-23T13:09:53.474 INFO:teuthology.orchestra.run.trial185.stdout: "type_data": { 2026-02-23T13:09:53.474 INFO:teuthology.orchestra.run.trial185.stdout: "flag_point": "waiting for peered", 2026-02-23T13:09:53.474 INFO:teuthology.orchestra.run.trial185.stdout: "client_info": { 2026-02-23T13:09:53.474 INFO:teuthology.orchestra.run.trial185.stdout: "client": "client.14522", 2026-02-23T13:09:53.474 INFO:teuthology.orchestra.run.trial185.stdout: "client_addr": "10.20.193.68:0/2327563370", 2026-02-23T13:09:53.474 INFO:teuthology.orchestra.run.trial185.stdout: "tid": 79214 2026-02-23T13:09:53.474 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.474 INFO:teuthology.orchestra.run.trial185.stdout: "events": [ 2026-02-23T13:09:53.474 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.474 INFO:teuthology.orchestra.run.trial185.stdout: "event": "initiated", 2026-02-23T13:09:53.474 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.717730+0000", 2026-02-23T13:09:53.474 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 0 2026-02-23T13:09:53.474 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.474 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.474 INFO:teuthology.orchestra.run.trial185.stdout: "event": "header_read", 2026-02-23T13:09:53.474 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.717730+0000", 2026-02-23T13:09:53.474 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 0 2026-02-23T13:09:53.474 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.474 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.474 INFO:teuthology.orchestra.run.trial185.stdout: "event": "throttled", 2026-02-23T13:09:53.474 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.717737+0000", 2026-02-23T13:09:53.474 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 7.3239999999999997e-06 2026-02-23T13:09:53.475 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.475 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.475 INFO:teuthology.orchestra.run.trial185.stdout: "event": "all_read", 2026-02-23T13:09:53.475 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.717760+0000", 2026-02-23T13:09:53.475 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 2.3003e-05 2026-02-23T13:09:53.475 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.475 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.475 INFO:teuthology.orchestra.run.trial185.stdout: "event": "dispatched", 2026-02-23T13:09:53.475 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.717763+0000", 2026-02-23T13:09:53.475 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 2.8849999999999999e-06 2026-02-23T13:09:53.475 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.475 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.475 INFO:teuthology.orchestra.run.trial185.stdout: "event": "queued_for_pg", 2026-02-23T13:09:53.475 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.717764+0000", 2026-02-23T13:09:53.475 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 1.212e-06 2026-02-23T13:09:53.475 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.475 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.475 INFO:teuthology.orchestra.run.trial185.stdout: "event": "reached_pg", 2026-02-23T13:09:53.475 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.717779+0000", 2026-02-23T13:09:53.475 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 1.5169e-05 2026-02-23T13:09:53.475 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.475 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.475 INFO:teuthology.orchestra.run.trial185.stdout: "event": "waiting for peered", 2026-02-23T13:09:53.476 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.717781+0000", 2026-02-23T13:09:53.476 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 1.452e-06 2026-02-23T13:09:53.476 INFO:teuthology.orchestra.run.trial185.stdout: } 2026-02-23T13:09:53.476 INFO:teuthology.orchestra.run.trial185.stdout: ] 2026-02-23T13:09:53.476 INFO:teuthology.orchestra.run.trial185.stdout: } 2026-02-23T13:09:53.476 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.476 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.476 INFO:teuthology.orchestra.run.trial185.stdout: "description": "osd_op(client.14522.0:79400 3.d 3.981104cd (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e80)", 2026-02-23T13:09:53.476 INFO:teuthology.orchestra.run.trial185.stdout: "initiated_at": "2026-02-23T13:09:52.722778+0000", 2026-02-23T13:09:53.476 INFO:teuthology.orchestra.run.trial185.stdout: "age": 0.67661828300000004, 2026-02-23T13:09:53.476 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 0.67690865499999997, 2026-02-23T13:09:53.476 INFO:teuthology.orchestra.run.trial185.stdout: "continuous": false, 2026-02-23T13:09:53.476 INFO:teuthology.orchestra.run.trial185.stdout: "type_data": { 2026-02-23T13:09:53.476 INFO:teuthology.orchestra.run.trial185.stdout: "flag_point": "waiting for peered", 2026-02-23T13:09:53.476 INFO:teuthology.orchestra.run.trial185.stdout: "client_info": { 2026-02-23T13:09:53.476 INFO:teuthology.orchestra.run.trial185.stdout: "client": "client.14522", 2026-02-23T13:09:53.476 INFO:teuthology.orchestra.run.trial185.stdout: "client_addr": "10.20.193.68:0/2327563370", 2026-02-23T13:09:53.476 INFO:teuthology.orchestra.run.trial185.stdout: "tid": 79400 2026-02-23T13:09:53.476 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.476 INFO:teuthology.orchestra.run.trial185.stdout: "events": [ 2026-02-23T13:09:53.476 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.476 INFO:teuthology.orchestra.run.trial185.stdout: "event": "initiated", 2026-02-23T13:09:53.476 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.722778+0000", 2026-02-23T13:09:53.476 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 0 2026-02-23T13:09:53.477 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.477 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.477 INFO:teuthology.orchestra.run.trial185.stdout: "event": "header_read", 2026-02-23T13:09:53.477 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.722778+0000", 2026-02-23T13:09:53.477 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 0 2026-02-23T13:09:53.477 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.477 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.477 INFO:teuthology.orchestra.run.trial185.stdout: "event": "throttled", 2026-02-23T13:09:53.477 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.722786+0000", 2026-02-23T13:09:53.477 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 8.1149999999999994e-06 2026-02-23T13:09:53.477 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.477 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.477 INFO:teuthology.orchestra.run.trial185.stdout: "event": "all_read", 2026-02-23T13:09:53.477 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.722810+0000", 2026-02-23T13:09:53.477 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 2.3995e-05 2026-02-23T13:09:53.477 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.477 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.477 INFO:teuthology.orchestra.run.trial185.stdout: "event": "dispatched", 2026-02-23T13:09:53.477 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.722814+0000", 2026-02-23T13:09:53.477 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 3.2760000000000001e-06 2026-02-23T13:09:53.477 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.477 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.477 INFO:teuthology.orchestra.run.trial185.stdout: "event": "queued_for_pg", 2026-02-23T13:09:53.478 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.722815+0000", 2026-02-23T13:09:53.478 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 1.3629999999999999e-06 2026-02-23T13:09:53.478 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.478 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.478 INFO:teuthology.orchestra.run.trial185.stdout: "event": "reached_pg", 2026-02-23T13:09:53.478 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.722832+0000", 2026-02-23T13:09:53.478 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 1.6790999999999999e-05 2026-02-23T13:09:53.478 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.478 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.478 INFO:teuthology.orchestra.run.trial185.stdout: "event": "waiting for peered", 2026-02-23T13:09:53.478 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.722833+0000", 2026-02-23T13:09:53.478 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 1.6330000000000001e-06 2026-02-23T13:09:53.478 INFO:teuthology.orchestra.run.trial185.stdout: } 2026-02-23T13:09:53.478 INFO:teuthology.orchestra.run.trial185.stdout: ] 2026-02-23T13:09:53.478 INFO:teuthology.orchestra.run.trial185.stdout: } 2026-02-23T13:09:53.478 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.478 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.478 INFO:teuthology.orchestra.run.trial185.stdout: "description": "osd_op(client.14522.0:79215 3.9 3.cb539f29 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e80)", 2026-02-23T13:09:53.478 INFO:teuthology.orchestra.run.trial185.stdout: "initiated_at": "2026-02-23T13:09:52.717772+0000", 2026-02-23T13:09:53.478 INFO:teuthology.orchestra.run.trial185.stdout: "age": 0.68162476299999997, 2026-02-23T13:09:53.478 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 0.681934551, 2026-02-23T13:09:53.478 INFO:teuthology.orchestra.run.trial185.stdout: "continuous": false, 2026-02-23T13:09:53.479 INFO:teuthology.orchestra.run.trial185.stdout: "type_data": { 2026-02-23T13:09:53.479 INFO:teuthology.orchestra.run.trial185.stdout: "flag_point": "waiting for peered", 2026-02-23T13:09:53.479 INFO:teuthology.orchestra.run.trial185.stdout: "client_info": { 2026-02-23T13:09:53.479 INFO:teuthology.orchestra.run.trial185.stdout: "client": "client.14522", 2026-02-23T13:09:53.479 INFO:teuthology.orchestra.run.trial185.stdout: "client_addr": "10.20.193.68:0/2327563370", 2026-02-23T13:09:53.479 INFO:teuthology.orchestra.run.trial185.stdout: "tid": 79215 2026-02-23T13:09:53.479 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.479 INFO:teuthology.orchestra.run.trial185.stdout: "events": [ 2026-02-23T13:09:53.479 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.479 INFO:teuthology.orchestra.run.trial185.stdout: "event": "initiated", 2026-02-23T13:09:53.479 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.717772+0000", 2026-02-23T13:09:53.479 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 0 2026-02-23T13:09:53.479 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.479 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.479 INFO:teuthology.orchestra.run.trial185.stdout: "event": "header_read", 2026-02-23T13:09:53.479 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.717772+0000", 2026-02-23T13:09:53.479 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 0 2026-02-23T13:09:53.479 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.479 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.479 INFO:teuthology.orchestra.run.trial185.stdout: "event": "throttled", 2026-02-23T13:09:53.479 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.717780+0000", 2026-02-23T13:09:53.479 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 8.4160000000000004e-06 2026-02-23T13:09:53.479 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.480 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.480 INFO:teuthology.orchestra.run.trial185.stdout: "event": "all_read", 2026-02-23T13:09:53.480 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.717800+0000", 2026-02-23T13:09:53.480 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 1.9536999999999999e-05 2026-02-23T13:09:53.480 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.480 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.480 INFO:teuthology.orchestra.run.trial185.stdout: "event": "dispatched", 2026-02-23T13:09:53.480 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.717803+0000", 2026-02-23T13:09:53.480 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 3.0149999999999999e-06 2026-02-23T13:09:53.480 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.480 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.480 INFO:teuthology.orchestra.run.trial185.stdout: "event": "queued_for_pg", 2026-02-23T13:09:53.480 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.717805+0000", 2026-02-23T13:09:53.480 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 1.9740000000000001e-06 2026-02-23T13:09:53.480 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.480 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.480 INFO:teuthology.orchestra.run.trial185.stdout: "event": "reached_pg", 2026-02-23T13:09:53.480 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.717821+0000", 2026-02-23T13:09:53.480 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 1.5869999999999999e-05 2026-02-23T13:09:53.480 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.480 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.480 INFO:teuthology.orchestra.run.trial185.stdout: "event": "waiting for peered", 2026-02-23T13:09:53.480 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.717826+0000", 2026-02-23T13:09:53.481 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 5.6699999999999999e-06 2026-02-23T13:09:53.481 INFO:teuthology.orchestra.run.trial185.stdout: } 2026-02-23T13:09:53.481 INFO:teuthology.orchestra.run.trial185.stdout: ] 2026-02-23T13:09:53.481 INFO:teuthology.orchestra.run.trial185.stdout: } 2026-02-23T13:09:53.481 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.481 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.481 INFO:teuthology.orchestra.run.trial185.stdout: "description": "osd_op(client.14522.0:79409 3.d 3.6d32bbed (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e80)", 2026-02-23T13:09:53.481 INFO:teuthology.orchestra.run.trial185.stdout: "initiated_at": "2026-02-23T13:09:52.729148+0000", 2026-02-23T13:09:53.481 INFO:teuthology.orchestra.run.trial185.stdout: "age": 0.67024835900000002, 2026-02-23T13:09:53.481 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 0.67057351600000004, 2026-02-23T13:09:53.481 INFO:teuthology.orchestra.run.trial185.stdout: "continuous": false, 2026-02-23T13:09:53.481 INFO:teuthology.orchestra.run.trial185.stdout: "type_data": { 2026-02-23T13:09:53.481 INFO:teuthology.orchestra.run.trial185.stdout: "flag_point": "waiting for peered", 2026-02-23T13:09:53.481 INFO:teuthology.orchestra.run.trial185.stdout: "client_info": { 2026-02-23T13:09:53.481 INFO:teuthology.orchestra.run.trial185.stdout: "client": "client.14522", 2026-02-23T13:09:53.481 INFO:teuthology.orchestra.run.trial185.stdout: "client_addr": "10.20.193.68:0/2327563370", 2026-02-23T13:09:53.481 INFO:teuthology.orchestra.run.trial185.stdout: "tid": 79409 2026-02-23T13:09:53.481 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.481 INFO:teuthology.orchestra.run.trial185.stdout: "events": [ 2026-02-23T13:09:53.481 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.481 INFO:teuthology.orchestra.run.trial185.stdout: "event": "initiated", 2026-02-23T13:09:53.481 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.729148+0000", 2026-02-23T13:09:53.481 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 0 2026-02-23T13:09:53.482 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.482 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.482 INFO:teuthology.orchestra.run.trial185.stdout: "event": "header_read", 2026-02-23T13:09:53.482 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.729148+0000", 2026-02-23T13:09:53.482 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 0 2026-02-23T13:09:53.482 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.482 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.482 INFO:teuthology.orchestra.run.trial185.stdout: "event": "throttled", 2026-02-23T13:09:53.482 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.729156+0000", 2026-02-23T13:09:53.482 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 7.2529999999999998e-06 2026-02-23T13:09:53.482 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.482 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.482 INFO:teuthology.orchestra.run.trial185.stdout: "event": "all_read", 2026-02-23T13:09:53.482 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.729181+0000", 2026-02-23T13:09:53.482 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 2.5137e-05 2026-02-23T13:09:53.482 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.482 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.482 INFO:teuthology.orchestra.run.trial185.stdout: "event": "dispatched", 2026-02-23T13:09:53.482 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.729185+0000", 2026-02-23T13:09:53.482 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 4.1779999999999996e-06 2026-02-23T13:09:53.482 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.482 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.483 INFO:teuthology.orchestra.run.trial185.stdout: "event": "queued_for_pg", 2026-02-23T13:09:53.483 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.729187+0000", 2026-02-23T13:09:53.483 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 2.435e-06 2026-02-23T13:09:53.483 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.483 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.483 INFO:teuthology.orchestra.run.trial185.stdout: "event": "reached_pg", 2026-02-23T13:09:53.483 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.729206+0000", 2026-02-23T13:09:53.483 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 1.8253999999999998e-05 2026-02-23T13:09:53.483 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.483 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.483 INFO:teuthology.orchestra.run.trial185.stdout: "event": "waiting for peered", 2026-02-23T13:09:53.483 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.729207+0000", 2026-02-23T13:09:53.483 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 1.683e-06 2026-02-23T13:09:53.483 INFO:teuthology.orchestra.run.trial185.stdout: } 2026-02-23T13:09:53.483 INFO:teuthology.orchestra.run.trial185.stdout: ] 2026-02-23T13:09:53.483 INFO:teuthology.orchestra.run.trial185.stdout: } 2026-02-23T13:09:53.483 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.483 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.483 INFO:teuthology.orchestra.run.trial185.stdout: "description": "osd_op(client.14522.0:79216 3.9 3.cb539f29 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e80)", 2026-02-23T13:09:53.483 INFO:teuthology.orchestra.run.trial185.stdout: "initiated_at": "2026-02-23T13:09:52.717812+0000", 2026-02-23T13:09:53.483 INFO:teuthology.orchestra.run.trial185.stdout: "age": 0.68158476800000001, 2026-02-23T13:09:53.483 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 0.68192683600000004, 2026-02-23T13:09:53.483 INFO:teuthology.orchestra.run.trial185.stdout: "continuous": false, 2026-02-23T13:09:53.483 INFO:teuthology.orchestra.run.trial185.stdout: "type_data": { 2026-02-23T13:09:53.484 INFO:teuthology.orchestra.run.trial185.stdout: "flag_point": "waiting for peered", 2026-02-23T13:09:53.484 INFO:teuthology.orchestra.run.trial185.stdout: "client_info": { 2026-02-23T13:09:53.484 INFO:teuthology.orchestra.run.trial185.stdout: "client": "client.14522", 2026-02-23T13:09:53.484 INFO:teuthology.orchestra.run.trial185.stdout: "client_addr": "10.20.193.68:0/2327563370", 2026-02-23T13:09:53.484 INFO:teuthology.orchestra.run.trial185.stdout: "tid": 79216 2026-02-23T13:09:53.484 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.484 INFO:teuthology.orchestra.run.trial185.stdout: "events": [ 2026-02-23T13:09:53.484 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.484 INFO:teuthology.orchestra.run.trial185.stdout: "event": "initiated", 2026-02-23T13:09:53.484 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.717812+0000", 2026-02-23T13:09:53.484 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 0 2026-02-23T13:09:53.484 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.484 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.484 INFO:teuthology.orchestra.run.trial185.stdout: "event": "header_read", 2026-02-23T13:09:53.484 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.717812+0000", 2026-02-23T13:09:53.484 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 0 2026-02-23T13:09:53.484 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.484 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.484 INFO:teuthology.orchestra.run.trial185.stdout: "event": "throttled", 2026-02-23T13:09:53.484 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.717818+0000", 2026-02-23T13:09:53.484 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 6.212e-06 2026-02-23T13:09:53.484 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.485 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.485 INFO:teuthology.orchestra.run.trial185.stdout: "event": "all_read", 2026-02-23T13:09:53.485 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.717839+0000", 2026-02-23T13:09:53.485 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 2.0829e-05 2026-02-23T13:09:53.485 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.485 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.485 INFO:teuthology.orchestra.run.trial185.stdout: "event": "dispatched", 2026-02-23T13:09:53.485 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.717842+0000", 2026-02-23T13:09:53.485 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 3.4659999999999999e-06 2026-02-23T13:09:53.485 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.485 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.485 INFO:teuthology.orchestra.run.trial185.stdout: "event": "queued_for_pg", 2026-02-23T13:09:53.485 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.717844+0000", 2026-02-23T13:09:53.485 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 1.443e-06 2026-02-23T13:09:53.485 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.485 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.485 INFO:teuthology.orchestra.run.trial185.stdout: "event": "reached_pg", 2026-02-23T13:09:53.485 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.717860+0000", 2026-02-23T13:09:53.485 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 1.5758999999999999e-05 2026-02-23T13:09:53.485 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.485 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.485 INFO:teuthology.orchestra.run.trial185.stdout: "event": "waiting for peered", 2026-02-23T13:09:53.485 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.717861+0000", 2026-02-23T13:09:53.485 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 1.5829999999999999e-06 2026-02-23T13:09:53.485 INFO:teuthology.orchestra.run.trial185.stdout: } 2026-02-23T13:09:53.486 INFO:teuthology.orchestra.run.trial185.stdout: ] 2026-02-23T13:09:53.486 INFO:teuthology.orchestra.run.trial185.stdout: } 2026-02-23T13:09:53.486 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.486 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.486 INFO:teuthology.orchestra.run.trial185.stdout: "description": "osd_op(client.14522.0:79410 3.d 3.6d32bbed (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e80)", 2026-02-23T13:09:53.486 INFO:teuthology.orchestra.run.trial185.stdout: "initiated_at": "2026-02-23T13:09:52.729200+0000", 2026-02-23T13:09:53.486 INFO:teuthology.orchestra.run.trial185.stdout: "age": 0.67019677300000002, 2026-02-23T13:09:53.486 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 0.67055552200000001, 2026-02-23T13:09:53.486 INFO:teuthology.orchestra.run.trial185.stdout: "continuous": false, 2026-02-23T13:09:53.486 INFO:teuthology.orchestra.run.trial185.stdout: "type_data": { 2026-02-23T13:09:53.486 INFO:teuthology.orchestra.run.trial185.stdout: "flag_point": "waiting for peered", 2026-02-23T13:09:53.486 INFO:teuthology.orchestra.run.trial185.stdout: "client_info": { 2026-02-23T13:09:53.486 INFO:teuthology.orchestra.run.trial185.stdout: "client": "client.14522", 2026-02-23T13:09:53.486 INFO:teuthology.orchestra.run.trial185.stdout: "client_addr": "10.20.193.68:0/2327563370", 2026-02-23T13:09:53.486 INFO:teuthology.orchestra.run.trial185.stdout: "tid": 79410 2026-02-23T13:09:53.486 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.486 INFO:teuthology.orchestra.run.trial185.stdout: "events": [ 2026-02-23T13:09:53.486 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.486 INFO:teuthology.orchestra.run.trial185.stdout: "event": "initiated", 2026-02-23T13:09:53.486 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.729200+0000", 2026-02-23T13:09:53.486 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 0 2026-02-23T13:09:53.486 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.486 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.486 INFO:teuthology.orchestra.run.trial185.stdout: "event": "header_read", 2026-02-23T13:09:53.487 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.729200+0000", 2026-02-23T13:09:53.487 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 0 2026-02-23T13:09:53.487 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.487 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.487 INFO:teuthology.orchestra.run.trial185.stdout: "event": "throttled", 2026-02-23T13:09:53.487 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.729205+0000", 2026-02-23T13:09:53.487 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 4.899e-06 2026-02-23T13:09:53.487 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.487 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.487 INFO:teuthology.orchestra.run.trial185.stdout: "event": "all_read", 2026-02-23T13:09:53.487 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.729221+0000", 2026-02-23T13:09:53.487 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 1.607e-05 2026-02-23T13:09:53.487 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.487 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.487 INFO:teuthology.orchestra.run.trial185.stdout: "event": "dispatched", 2026-02-23T13:09:53.487 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.729224+0000", 2026-02-23T13:09:53.487 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 2.7750000000000001e-06 2026-02-23T13:09:53.487 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.487 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.487 INFO:teuthology.orchestra.run.trial185.stdout: "event": "queued_for_pg", 2026-02-23T13:09:53.487 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.729225+0000", 2026-02-23T13:09:53.487 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 1.7540000000000001e-06 2026-02-23T13:09:53.487 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.487 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.487 INFO:teuthology.orchestra.run.trial185.stdout: "event": "reached_pg", 2026-02-23T13:09:53.487 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.729240+0000", 2026-02-23T13:09:53.488 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 1.5048000000000001e-05 2026-02-23T13:09:53.488 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.488 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.488 INFO:teuthology.orchestra.run.trial185.stdout: "event": "waiting for peered", 2026-02-23T13:09:53.488 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.729243+0000", 2026-02-23T13:09:53.488 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 2.3039999999999999e-06 2026-02-23T13:09:53.488 INFO:teuthology.orchestra.run.trial185.stdout: } 2026-02-23T13:09:53.488 INFO:teuthology.orchestra.run.trial185.stdout: ] 2026-02-23T13:09:53.488 INFO:teuthology.orchestra.run.trial185.stdout: } 2026-02-23T13:09:53.488 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.488 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.488 INFO:teuthology.orchestra.run.trial185.stdout: "description": "osd_op(client.14522.0:79219 3.d 3.38b3af2d (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e80)", 2026-02-23T13:09:53.488 INFO:teuthology.orchestra.run.trial185.stdout: "initiated_at": "2026-02-23T13:09:52.717853+0000", 2026-02-23T13:09:53.488 INFO:teuthology.orchestra.run.trial185.stdout: "age": 0.68154379099999995, 2026-02-23T13:09:53.488 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 0.68191999299999995, 2026-02-23T13:09:53.488 INFO:teuthology.orchestra.run.trial185.stdout: "continuous": false, 2026-02-23T13:09:53.488 INFO:teuthology.orchestra.run.trial185.stdout: "type_data": { 2026-02-23T13:09:53.488 INFO:teuthology.orchestra.run.trial185.stdout: "flag_point": "waiting for peered", 2026-02-23T13:09:53.488 INFO:teuthology.orchestra.run.trial185.stdout: "client_info": { 2026-02-23T13:09:53.488 INFO:teuthology.orchestra.run.trial185.stdout: "client": "client.14522", 2026-02-23T13:09:53.488 INFO:teuthology.orchestra.run.trial185.stdout: "client_addr": "10.20.193.68:0/2327563370", 2026-02-23T13:09:53.488 INFO:teuthology.orchestra.run.trial185.stdout: "tid": 79219 2026-02-23T13:09:53.488 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.488 INFO:teuthology.orchestra.run.trial185.stdout: "events": [ 2026-02-23T13:09:53.488 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.488 INFO:teuthology.orchestra.run.trial185.stdout: "event": "initiated", 2026-02-23T13:09:53.489 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.717853+0000", 2026-02-23T13:09:53.489 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 0 2026-02-23T13:09:53.489 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.489 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.489 INFO:teuthology.orchestra.run.trial185.stdout: "event": "header_read", 2026-02-23T13:09:53.489 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.717853+0000", 2026-02-23T13:09:53.489 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 0 2026-02-23T13:09:53.489 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.489 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.489 INFO:teuthology.orchestra.run.trial185.stdout: "event": "throttled", 2026-02-23T13:09:53.489 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.717863+0000", 2026-02-23T13:09:53.489 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 9.8279999999999998e-06 2026-02-23T13:09:53.489 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.489 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.489 INFO:teuthology.orchestra.run.trial185.stdout: "event": "all_read", 2026-02-23T13:09:53.489 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.717887+0000", 2026-02-23T13:09:53.489 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 2.4456000000000001e-05 2026-02-23T13:09:53.489 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.489 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.489 INFO:teuthology.orchestra.run.trial185.stdout: "event": "dispatched", 2026-02-23T13:09:53.489 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.717890+0000", 2026-02-23T13:09:53.489 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 3.2559999999999998e-06 2026-02-23T13:09:53.489 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.489 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.489 INFO:teuthology.orchestra.run.trial185.stdout: "event": "queued_for_pg", 2026-02-23T13:09:53.490 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.717892+0000", 2026-02-23T13:09:53.490 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 1.482e-06 2026-02-23T13:09:53.490 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.490 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.490 INFO:teuthology.orchestra.run.trial185.stdout: "event": "reached_pg", 2026-02-23T13:09:53.490 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.717918+0000", 2026-02-23T13:09:53.490 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 2.6380000000000002e-05 2026-02-23T13:09:53.490 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.490 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.490 INFO:teuthology.orchestra.run.trial185.stdout: "event": "waiting for peered", 2026-02-23T13:09:53.490 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.717921+0000", 2026-02-23T13:09:53.490 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 2.8849999999999999e-06 2026-02-23T13:09:53.490 INFO:teuthology.orchestra.run.trial185.stdout: } 2026-02-23T13:09:53.490 INFO:teuthology.orchestra.run.trial185.stdout: ] 2026-02-23T13:09:53.490 INFO:teuthology.orchestra.run.trial185.stdout: } 2026-02-23T13:09:53.490 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.490 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.490 INFO:teuthology.orchestra.run.trial185.stdout: "description": "osd_op(client.14522.0:79411 3.d 3.6d32bbed (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e80)", 2026-02-23T13:09:53.490 INFO:teuthology.orchestra.run.trial185.stdout: "initiated_at": "2026-02-23T13:09:52.730400+0000", 2026-02-23T13:09:53.490 INFO:teuthology.orchestra.run.trial185.stdout: "age": 0.66899697499999999, 2026-02-23T13:09:53.490 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 0.66939066899999999, 2026-02-23T13:09:53.490 INFO:teuthology.orchestra.run.trial185.stdout: "continuous": false, 2026-02-23T13:09:53.490 INFO:teuthology.orchestra.run.trial185.stdout: "type_data": { 2026-02-23T13:09:53.490 INFO:teuthology.orchestra.run.trial185.stdout: "flag_point": "waiting for peered", 2026-02-23T13:09:53.490 INFO:teuthology.orchestra.run.trial185.stdout: "client_info": { 2026-02-23T13:09:53.490 INFO:teuthology.orchestra.run.trial185.stdout: "client": "client.14522", 2026-02-23T13:09:53.491 INFO:teuthology.orchestra.run.trial185.stdout: "client_addr": "10.20.193.68:0/2327563370", 2026-02-23T13:09:53.491 INFO:teuthology.orchestra.run.trial185.stdout: "tid": 79411 2026-02-23T13:09:53.491 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.491 INFO:teuthology.orchestra.run.trial185.stdout: "events": [ 2026-02-23T13:09:53.491 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.491 INFO:teuthology.orchestra.run.trial185.stdout: "event": "initiated", 2026-02-23T13:09:53.491 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.730400+0000", 2026-02-23T13:09:53.491 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 0 2026-02-23T13:09:53.491 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.491 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.491 INFO:teuthology.orchestra.run.trial185.stdout: "event": "header_read", 2026-02-23T13:09:53.491 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.730400+0000", 2026-02-23T13:09:53.491 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 0 2026-02-23T13:09:53.491 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.491 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.491 INFO:teuthology.orchestra.run.trial185.stdout: "event": "throttled", 2026-02-23T13:09:53.491 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.730404+0000", 2026-02-23T13:09:53.491 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 4.8389999999999997e-06 2026-02-23T13:09:53.491 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.491 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.491 INFO:teuthology.orchestra.run.trial185.stdout: "event": "all_read", 2026-02-23T13:09:53.491 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.730426+0000", 2026-02-23T13:09:53.491 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 2.139e-05 2026-02-23T13:09:53.491 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.491 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.491 INFO:teuthology.orchestra.run.trial185.stdout: "event": "dispatched", 2026-02-23T13:09:53.491 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.730429+0000", 2026-02-23T13:09:53.492 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 3.3759999999999999e-06 2026-02-23T13:09:53.492 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.492 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.492 INFO:teuthology.orchestra.run.trial185.stdout: "event": "queued_for_pg", 2026-02-23T13:09:53.492 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.730431+0000", 2026-02-23T13:09:53.492 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 1.4729999999999999e-06 2026-02-23T13:09:53.492 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.492 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.492 INFO:teuthology.orchestra.run.trial185.stdout: "event": "reached_pg", 2026-02-23T13:09:53.492 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.730446+0000", 2026-02-23T13:09:53.492 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 1.4868000000000001e-05 2026-02-23T13:09:53.492 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.492 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.492 INFO:teuthology.orchestra.run.trial185.stdout: "event": "waiting for peered", 2026-02-23T13:09:53.492 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.730448+0000", 2026-02-23T13:09:53.492 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 2.0140000000000001e-06 2026-02-23T13:09:53.492 INFO:teuthology.orchestra.run.trial185.stdout: } 2026-02-23T13:09:53.492 INFO:teuthology.orchestra.run.trial185.stdout: ] 2026-02-23T13:09:53.492 INFO:teuthology.orchestra.run.trial185.stdout: } 2026-02-23T13:09:53.492 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.492 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.492 INFO:teuthology.orchestra.run.trial185.stdout: "description": "osd_op(client.14522.0:79220 3.d 3.38b3af2d (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e80)", 2026-02-23T13:09:53.492 INFO:teuthology.orchestra.run.trial185.stdout: "initiated_at": "2026-02-23T13:09:52.717901+0000", 2026-02-23T13:09:53.492 INFO:teuthology.orchestra.run.trial185.stdout: "age": 0.68149567200000005, 2026-02-23T13:09:53.492 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 0.68190486500000003, 2026-02-23T13:09:53.492 INFO:teuthology.orchestra.run.trial185.stdout: "continuous": false, 2026-02-23T13:09:53.492 INFO:teuthology.orchestra.run.trial185.stdout: "type_data": { 2026-02-23T13:09:53.492 INFO:teuthology.orchestra.run.trial185.stdout: "flag_point": "waiting for peered", 2026-02-23T13:09:53.493 INFO:teuthology.orchestra.run.trial185.stdout: "client_info": { 2026-02-23T13:09:53.493 INFO:teuthology.orchestra.run.trial185.stdout: "client": "client.14522", 2026-02-23T13:09:53.493 INFO:teuthology.orchestra.run.trial185.stdout: "client_addr": "10.20.193.68:0/2327563370", 2026-02-23T13:09:53.493 INFO:teuthology.orchestra.run.trial185.stdout: "tid": 79220 2026-02-23T13:09:53.493 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.493 INFO:teuthology.orchestra.run.trial185.stdout: "events": [ 2026-02-23T13:09:53.493 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.493 INFO:teuthology.orchestra.run.trial185.stdout: "event": "initiated", 2026-02-23T13:09:53.493 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.717901+0000", 2026-02-23T13:09:53.493 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 0 2026-02-23T13:09:53.493 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.493 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.493 INFO:teuthology.orchestra.run.trial185.stdout: "event": "header_read", 2026-02-23T13:09:53.493 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.717901+0000", 2026-02-23T13:09:53.493 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 0 2026-02-23T13:09:53.493 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.493 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.493 INFO:teuthology.orchestra.run.trial185.stdout: "event": "throttled", 2026-02-23T13:09:53.493 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.717911+0000", 2026-02-23T13:09:53.493 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 1.041e-05 2026-02-23T13:09:53.493 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.493 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.493 INFO:teuthology.orchestra.run.trial185.stdout: "event": "all_read", 2026-02-23T13:09:53.493 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.717932+0000", 2026-02-23T13:09:53.493 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 2.0908999999999999e-05 2026-02-23T13:09:53.493 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.493 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.493 INFO:teuthology.orchestra.run.trial185.stdout: "event": "dispatched", 2026-02-23T13:09:53.493 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.717935+0000", 2026-02-23T13:09:53.494 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 2.9950000000000001e-06 2026-02-23T13:09:53.494 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.494 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.494 INFO:teuthology.orchestra.run.trial185.stdout: "event": "queued_for_pg", 2026-02-23T13:09:53.494 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.717936+0000", 2026-02-23T13:09:53.494 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 1.1930000000000001e-06 2026-02-23T13:09:53.494 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.494 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.494 INFO:teuthology.orchestra.run.trial185.stdout: "event": "reached_pg", 2026-02-23T13:09:53.494 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.717956+0000", 2026-02-23T13:09:53.494 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 1.9205e-05 2026-02-23T13:09:53.494 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.494 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.494 INFO:teuthology.orchestra.run.trial185.stdout: "event": "waiting for peered", 2026-02-23T13:09:53.494 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.717960+0000", 2026-02-23T13:09:53.494 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 3.9480000000000001e-06 2026-02-23T13:09:53.494 INFO:teuthology.orchestra.run.trial185.stdout: } 2026-02-23T13:09:53.494 INFO:teuthology.orchestra.run.trial185.stdout: ] 2026-02-23T13:09:53.494 INFO:teuthology.orchestra.run.trial185.stdout: } 2026-02-23T13:09:53.494 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.494 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.494 INFO:teuthology.orchestra.run.trial185.stdout: "description": "osd_op(client.14522.0:79412 3.d 3.6d32bbed (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e80)", 2026-02-23T13:09:53.494 INFO:teuthology.orchestra.run.trial185.stdout: "initiated_at": "2026-02-23T13:09:52.730441+0000", 2026-02-23T13:09:53.494 INFO:teuthology.orchestra.run.trial185.stdout: "age": 0.66895537699999996, 2026-02-23T13:09:53.494 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 0.66940757100000003, 2026-02-23T13:09:53.494 INFO:teuthology.orchestra.run.trial185.stdout: "continuous": false, 2026-02-23T13:09:53.494 INFO:teuthology.orchestra.run.trial185.stdout: "type_data": { 2026-02-23T13:09:53.494 INFO:teuthology.orchestra.run.trial185.stdout: "flag_point": "waiting for peered", 2026-02-23T13:09:53.495 INFO:teuthology.orchestra.run.trial185.stdout: "client_info": { 2026-02-23T13:09:53.495 INFO:teuthology.orchestra.run.trial185.stdout: "client": "client.14522", 2026-02-23T13:09:53.495 INFO:teuthology.orchestra.run.trial185.stdout: "client_addr": "10.20.193.68:0/2327563370", 2026-02-23T13:09:53.495 INFO:teuthology.orchestra.run.trial185.stdout: "tid": 79412 2026-02-23T13:09:53.495 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.495 INFO:teuthology.orchestra.run.trial185.stdout: "events": [ 2026-02-23T13:09:53.495 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.495 INFO:teuthology.orchestra.run.trial185.stdout: "event": "initiated", 2026-02-23T13:09:53.495 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.730441+0000", 2026-02-23T13:09:53.495 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 0 2026-02-23T13:09:53.495 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.495 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.495 INFO:teuthology.orchestra.run.trial185.stdout: "event": "header_read", 2026-02-23T13:09:53.495 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.730441+0000", 2026-02-23T13:09:53.495 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 0 2026-02-23T13:09:53.495 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.495 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.495 INFO:teuthology.orchestra.run.trial185.stdout: "event": "throttled", 2026-02-23T13:09:53.495 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.730446+0000", 2026-02-23T13:09:53.495 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 4.6689999999999997e-06 2026-02-23T13:09:53.495 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.495 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.495 INFO:teuthology.orchestra.run.trial185.stdout: "event": "all_read", 2026-02-23T13:09:53.495 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.730462+0000", 2026-02-23T13:09:53.495 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 1.5919000000000001e-05 2026-02-23T13:09:53.495 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.495 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.495 INFO:teuthology.orchestra.run.trial185.stdout: "event": "dispatched", 2026-02-23T13:09:53.495 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.730464+0000", 2026-02-23T13:09:53.495 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 2.5749999999999999e-06 2026-02-23T13:09:53.496 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.496 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.496 INFO:teuthology.orchestra.run.trial185.stdout: "event": "queued_for_pg", 2026-02-23T13:09:53.496 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.730465+0000", 2026-02-23T13:09:53.496 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 1.0419999999999999e-06 2026-02-23T13:09:53.496 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.496 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.496 INFO:teuthology.orchestra.run.trial185.stdout: "event": "reached_pg", 2026-02-23T13:09:53.496 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.730478+0000", 2026-02-23T13:09:53.496 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 1.2422999999999999e-05 2026-02-23T13:09:53.496 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.496 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.496 INFO:teuthology.orchestra.run.trial185.stdout: "event": "waiting for peered", 2026-02-23T13:09:53.496 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.730479+0000", 2026-02-23T13:09:53.496 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 1.5829999999999999e-06 2026-02-23T13:09:53.496 INFO:teuthology.orchestra.run.trial185.stdout: } 2026-02-23T13:09:53.496 INFO:teuthology.orchestra.run.trial185.stdout: ] 2026-02-23T13:09:53.496 INFO:teuthology.orchestra.run.trial185.stdout: } 2026-02-23T13:09:53.496 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.496 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.496 INFO:teuthology.orchestra.run.trial185.stdout: "description": "osd_op(client.14522.0:79221 3.d 3.38b3af2d (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e80)", 2026-02-23T13:09:53.496 INFO:teuthology.orchestra.run.trial185.stdout: "initiated_at": "2026-02-23T13:09:52.717946+0000", 2026-02-23T13:09:53.496 INFO:teuthology.orchestra.run.trial185.stdout: "age": 0.68145025699999995, 2026-02-23T13:09:53.496 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 0.68191781900000004, 2026-02-23T13:09:53.496 INFO:teuthology.orchestra.run.trial185.stdout: "continuous": false, 2026-02-23T13:09:53.496 INFO:teuthology.orchestra.run.trial185.stdout: "type_data": { 2026-02-23T13:09:53.496 INFO:teuthology.orchestra.run.trial185.stdout: "flag_point": "waiting for peered", 2026-02-23T13:09:53.496 INFO:teuthology.orchestra.run.trial185.stdout: "client_info": { 2026-02-23T13:09:53.496 INFO:teuthology.orchestra.run.trial185.stdout: "client": "client.14522", 2026-02-23T13:09:53.496 INFO:teuthology.orchestra.run.trial185.stdout: "client_addr": "10.20.193.68:0/2327563370", 2026-02-23T13:09:53.496 INFO:teuthology.orchestra.run.trial185.stdout: "tid": 79221 2026-02-23T13:09:53.497 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.497 INFO:teuthology.orchestra.run.trial185.stdout: "events": [ 2026-02-23T13:09:53.497 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.497 INFO:teuthology.orchestra.run.trial185.stdout: "event": "initiated", 2026-02-23T13:09:53.497 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.717946+0000", 2026-02-23T13:09:53.497 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 0 2026-02-23T13:09:53.497 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.497 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.497 INFO:teuthology.orchestra.run.trial185.stdout: "event": "header_read", 2026-02-23T13:09:53.497 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.717946+0000", 2026-02-23T13:09:53.497 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 0 2026-02-23T13:09:53.497 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.497 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.497 INFO:teuthology.orchestra.run.trial185.stdout: "event": "throttled", 2026-02-23T13:09:53.497 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.717953+0000", 2026-02-23T13:09:53.497 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 6.652e-06 2026-02-23T13:09:53.497 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.497 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.497 INFO:teuthology.orchestra.run.trial185.stdout: "event": "all_read", 2026-02-23T13:09:53.497 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.717997+0000", 2026-02-23T13:09:53.497 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 4.3602e-05 2026-02-23T13:09:53.497 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.497 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.497 INFO:teuthology.orchestra.run.trial185.stdout: "event": "dispatched", 2026-02-23T13:09:53.497 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.718000+0000", 2026-02-23T13:09:53.497 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 3.0450000000000001e-06 2026-02-23T13:09:53.497 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.497 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.497 INFO:teuthology.orchestra.run.trial185.stdout: "event": "queued_for_pg", 2026-02-23T13:09:53.497 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.718001+0000", 2026-02-23T13:09:53.497 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 1.1629999999999999e-06 2026-02-23T13:09:53.498 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.498 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.498 INFO:teuthology.orchestra.run.trial185.stdout: "event": "reached_pg", 2026-02-23T13:09:53.498 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.718017+0000", 2026-02-23T13:09:53.498 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 1.6169999999999999e-05 2026-02-23T13:09:53.498 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.498 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.498 INFO:teuthology.orchestra.run.trial185.stdout: "event": "waiting for peered", 2026-02-23T13:09:53.498 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.718020+0000", 2026-02-23T13:09:53.498 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 2.8849999999999999e-06 2026-02-23T13:09:53.498 INFO:teuthology.orchestra.run.trial185.stdout: } 2026-02-23T13:09:53.498 INFO:teuthology.orchestra.run.trial185.stdout: ] 2026-02-23T13:09:53.498 INFO:teuthology.orchestra.run.trial185.stdout: } 2026-02-23T13:09:53.498 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.498 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.498 INFO:teuthology.orchestra.run.trial185.stdout: "description": "osd_op(client.14522.0:79413 3.d 3.6d32bbed (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e80)", 2026-02-23T13:09:53.498 INFO:teuthology.orchestra.run.trial185.stdout: "initiated_at": "2026-02-23T13:09:52.731726+0000", 2026-02-23T13:09:53.498 INFO:teuthology.orchestra.run.trial185.stdout: "age": 0.66767058000000001, 2026-02-23T13:09:53.498 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 0.66815539400000001, 2026-02-23T13:09:53.498 INFO:teuthology.orchestra.run.trial185.stdout: "continuous": false, 2026-02-23T13:09:53.498 INFO:teuthology.orchestra.run.trial185.stdout: "type_data": { 2026-02-23T13:09:53.498 INFO:teuthology.orchestra.run.trial185.stdout: "flag_point": "waiting for peered", 2026-02-23T13:09:53.498 INFO:teuthology.orchestra.run.trial185.stdout: "client_info": { 2026-02-23T13:09:53.498 INFO:teuthology.orchestra.run.trial185.stdout: "client": "client.14522", 2026-02-23T13:09:53.498 INFO:teuthology.orchestra.run.trial185.stdout: "client_addr": "10.20.193.68:0/2327563370", 2026-02-23T13:09:53.498 INFO:teuthology.orchestra.run.trial185.stdout: "tid": 79413 2026-02-23T13:09:53.498 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.498 INFO:teuthology.orchestra.run.trial185.stdout: "events": [ 2026-02-23T13:09:53.498 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.498 INFO:teuthology.orchestra.run.trial185.stdout: "event": "initiated", 2026-02-23T13:09:53.498 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.731726+0000", 2026-02-23T13:09:53.498 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 0 2026-02-23T13:09:53.499 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.499 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.499 INFO:teuthology.orchestra.run.trial185.stdout: "event": "header_read", 2026-02-23T13:09:53.499 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.731726+0000", 2026-02-23T13:09:53.499 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 0 2026-02-23T13:09:53.499 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.499 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.499 INFO:teuthology.orchestra.run.trial185.stdout: "event": "throttled", 2026-02-23T13:09:53.499 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.731731+0000", 2026-02-23T13:09:53.499 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 4.5979999999999999e-06 2026-02-23T13:09:53.499 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.499 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.499 INFO:teuthology.orchestra.run.trial185.stdout: "event": "all_read", 2026-02-23T13:09:53.499 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.731750+0000", 2026-02-23T13:09:53.499 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 1.9006000000000001e-05 2026-02-23T13:09:53.499 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.499 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.499 INFO:teuthology.orchestra.run.trial185.stdout: "event": "dispatched", 2026-02-23T13:09:53.499 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.731752+0000", 2026-02-23T13:09:53.499 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 2.554e-06 2026-02-23T13:09:53.499 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.499 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.499 INFO:teuthology.orchestra.run.trial185.stdout: "event": "queued_for_pg", 2026-02-23T13:09:53.499 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.731753+0000", 2026-02-23T13:09:53.499 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 8.8199999999999998e-07 2026-02-23T13:09:53.499 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.499 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.499 INFO:teuthology.orchestra.run.trial185.stdout: "event": "reached_pg", 2026-02-23T13:09:53.499 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.731764+0000", 2026-02-23T13:09:53.499 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 1.1221e-05 2026-02-23T13:09:53.499 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.499 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.499 INFO:teuthology.orchestra.run.trial185.stdout: "event": "waiting for peered", 2026-02-23T13:09:53.500 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.731766+0000", 2026-02-23T13:09:53.500 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 1.403e-06 2026-02-23T13:09:53.500 INFO:teuthology.orchestra.run.trial185.stdout: } 2026-02-23T13:09:53.500 INFO:teuthology.orchestra.run.trial185.stdout: ] 2026-02-23T13:09:53.500 INFO:teuthology.orchestra.run.trial185.stdout: } 2026-02-23T13:09:53.500 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.500 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.500 INFO:teuthology.orchestra.run.trial185.stdout: "description": "osd_op(client.14522.0:79222 3.d 3.38b3af2d (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e80)", 2026-02-23T13:09:53.500 INFO:teuthology.orchestra.run.trial185.stdout: "initiated_at": "2026-02-23T13:09:52.718009+0000", 2026-02-23T13:09:53.500 INFO:teuthology.orchestra.run.trial185.stdout: "age": 0.68138781999999998, 2026-02-23T13:09:53.500 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 0.68188823399999998, 2026-02-23T13:09:53.500 INFO:teuthology.orchestra.run.trial185.stdout: "continuous": false, 2026-02-23T13:09:53.500 INFO:teuthology.orchestra.run.trial185.stdout: "type_data": { 2026-02-23T13:09:53.500 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9 128 79432 79304 68.8349 16.0078 0.0206672 0.0133219 2026-02-23T13:09:53.501 INFO:teuthology.orchestra.run.trial185.stdout: "flag_point": "waiting for peered", 2026-02-23T13:09:53.501 INFO:teuthology.orchestra.run.trial185.stdout: "client_info": { 2026-02-23T13:09:53.501 INFO:teuthology.orchestra.run.trial185.stdout: "client": "client.14522", 2026-02-23T13:09:53.501 INFO:teuthology.orchestra.run.trial185.stdout: "client_addr": "10.20.193.68:0/2327563370", 2026-02-23T13:09:53.501 INFO:teuthology.orchestra.run.trial185.stdout: "tid": 79222 2026-02-23T13:09:53.501 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.501 INFO:teuthology.orchestra.run.trial185.stdout: "events": [ 2026-02-23T13:09:53.501 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.501 INFO:teuthology.orchestra.run.trial185.stdout: "event": "initiated", 2026-02-23T13:09:53.501 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.718009+0000", 2026-02-23T13:09:53.501 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 0 2026-02-23T13:09:53.501 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.501 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.502 INFO:teuthology.orchestra.run.trial185.stdout: "event": "header_read", 2026-02-23T13:09:53.502 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.718009+0000", 2026-02-23T13:09:53.502 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 0 2026-02-23T13:09:53.502 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.502 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.502 INFO:teuthology.orchestra.run.trial185.stdout: "event": "throttled", 2026-02-23T13:09:53.502 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.718017+0000", 2026-02-23T13:09:53.502 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 8.5150000000000006e-06 2026-02-23T13:09:53.502 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.502 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.502 INFO:teuthology.orchestra.run.trial185.stdout: "event": "all_read", 2026-02-23T13:09:53.502 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.718039+0000", 2026-02-23T13:09:53.502 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 2.2132000000000001e-05 2026-02-23T13:09:53.502 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.502 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.502 INFO:teuthology.orchestra.run.trial185.stdout: "event": "dispatched", 2026-02-23T13:09:53.502 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.718043+0000", 2026-02-23T13:09:53.502 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 3.0649999999999999e-06 2026-02-23T13:09:53.502 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.502 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.502 INFO:teuthology.orchestra.run.trial185.stdout: "event": "queued_for_pg", 2026-02-23T13:09:53.502 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.718044+0000", 2026-02-23T13:09:53.502 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 1.874e-06 2026-02-23T13:09:53.502 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.502 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.502 INFO:teuthology.orchestra.run.trial185.stdout: "event": "reached_pg", 2026-02-23T13:09:53.502 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.718058+0000", 2026-02-23T13:09:53.502 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 1.3405e-05 2026-02-23T13:09:53.502 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.502 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.502 INFO:teuthology.orchestra.run.trial185.stdout: "event": "waiting for peered", 2026-02-23T13:09:53.502 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.718061+0000", 2026-02-23T13:09:53.502 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 2.7350000000000001e-06 2026-02-23T13:09:53.503 INFO:teuthology.orchestra.run.trial185.stdout: } 2026-02-23T13:09:53.503 INFO:teuthology.orchestra.run.trial185.stdout: ] 2026-02-23T13:09:53.503 INFO:teuthology.orchestra.run.trial185.stdout: } 2026-02-23T13:09:53.503 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.503 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.503 INFO:teuthology.orchestra.run.trial185.stdout: "description": "osd_op(client.14522.0:79414 3.d 3.6d32bbed (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e80)", 2026-02-23T13:09:53.503 INFO:teuthology.orchestra.run.trial185.stdout: "initiated_at": "2026-02-23T13:09:52.731761+0000", 2026-02-23T13:09:53.503 INFO:teuthology.orchestra.run.trial185.stdout: "age": 0.66763562499999995, 2026-02-23T13:09:53.503 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 0.66815366099999995, 2026-02-23T13:09:53.503 INFO:teuthology.orchestra.run.trial185.stdout: "continuous": false, 2026-02-23T13:09:53.503 INFO:teuthology.orchestra.run.trial185.stdout: "type_data": { 2026-02-23T13:09:53.503 INFO:teuthology.orchestra.run.trial185.stdout: "flag_point": "waiting for peered", 2026-02-23T13:09:53.503 INFO:teuthology.orchestra.run.trial185.stdout: "client_info": { 2026-02-23T13:09:53.503 INFO:teuthology.orchestra.run.trial185.stdout: "client": "client.14522", 2026-02-23T13:09:53.503 INFO:teuthology.orchestra.run.trial185.stdout: "client_addr": "10.20.193.68:0/2327563370", 2026-02-23T13:09:53.503 INFO:teuthology.orchestra.run.trial185.stdout: "tid": 79414 2026-02-23T13:09:53.503 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.503 INFO:teuthology.orchestra.run.trial185.stdout: "events": [ 2026-02-23T13:09:53.503 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.503 INFO:teuthology.orchestra.run.trial185.stdout: "event": "initiated", 2026-02-23T13:09:53.503 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.731761+0000", 2026-02-23T13:09:53.503 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 0 2026-02-23T13:09:53.503 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.503 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.503 INFO:teuthology.orchestra.run.trial185.stdout: "event": "header_read", 2026-02-23T13:09:53.503 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.731761+0000", 2026-02-23T13:09:53.503 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 0 2026-02-23T13:09:53.503 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.503 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.503 INFO:teuthology.orchestra.run.trial185.stdout: "event": "throttled", 2026-02-23T13:09:53.503 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.731767+0000", 2026-02-23T13:09:53.503 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 5.7100000000000004e-06 2026-02-23T13:09:53.503 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.503 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.504 INFO:teuthology.orchestra.run.trial185.stdout: "event": "all_read", 2026-02-23T13:09:53.504 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.731782+0000", 2026-02-23T13:09:53.504 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 1.5079000000000001e-05 2026-02-23T13:09:53.504 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.504 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.504 INFO:teuthology.orchestra.run.trial185.stdout: "event": "dispatched", 2026-02-23T13:09:53.504 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.731784+0000", 2026-02-23T13:09:53.504 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 2.4739999999999999e-06 2026-02-23T13:09:53.504 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.504 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.504 INFO:teuthology.orchestra.run.trial185.stdout: "event": "queued_for_pg", 2026-02-23T13:09:53.504 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.731785+0000", 2026-02-23T13:09:53.504 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 9.2200000000000002e-07 2026-02-23T13:09:53.504 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.504 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.504 INFO:teuthology.orchestra.run.trial185.stdout: "event": "reached_pg", 2026-02-23T13:09:53.504 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.731797+0000", 2026-02-23T13:09:53.504 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 1.1671999999999999e-05 2026-02-23T13:09:53.504 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.504 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.504 INFO:teuthology.orchestra.run.trial185.stdout: "event": "waiting for peered", 2026-02-23T13:09:53.504 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.731798+0000", 2026-02-23T13:09:53.504 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 1.4720000000000001e-06 2026-02-23T13:09:53.504 INFO:teuthology.orchestra.run.trial185.stdout: } 2026-02-23T13:09:53.504 INFO:teuthology.orchestra.run.trial185.stdout: ] 2026-02-23T13:09:53.504 INFO:teuthology.orchestra.run.trial185.stdout: } 2026-02-23T13:09:53.504 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.504 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.504 INFO:teuthology.orchestra.run.trial185.stdout: "description": "osd_op(client.14522.0:79223 3.d 3.38b3af2d (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e80)", 2026-02-23T13:09:53.504 INFO:teuthology.orchestra.run.trial185.stdout: "initiated_at": "2026-02-23T13:09:52.718053+0000", 2026-02-23T13:09:53.504 INFO:teuthology.orchestra.run.trial185.stdout: "age": 0.68134398900000004, 2026-02-23T13:09:53.504 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 0.68187912699999997, 2026-02-23T13:09:53.504 INFO:teuthology.orchestra.run.trial185.stdout: "continuous": false, 2026-02-23T13:09:53.504 INFO:teuthology.orchestra.run.trial185.stdout: "type_data": { 2026-02-23T13:09:53.504 INFO:teuthology.orchestra.run.trial185.stdout: "flag_point": "waiting for peered", 2026-02-23T13:09:53.505 INFO:teuthology.orchestra.run.trial185.stdout: "client_info": { 2026-02-23T13:09:53.505 INFO:teuthology.orchestra.run.trial185.stdout: "client": "client.14522", 2026-02-23T13:09:53.505 INFO:teuthology.orchestra.run.trial185.stdout: "client_addr": "10.20.193.68:0/2327563370", 2026-02-23T13:09:53.505 INFO:teuthology.orchestra.run.trial185.stdout: "tid": 79223 2026-02-23T13:09:53.505 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.505 INFO:teuthology.orchestra.run.trial185.stdout: "events": [ 2026-02-23T13:09:53.505 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.505 INFO:teuthology.orchestra.run.trial185.stdout: "event": "initiated", 2026-02-23T13:09:53.505 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.718053+0000", 2026-02-23T13:09:53.505 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 0 2026-02-23T13:09:53.505 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.505 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.505 INFO:teuthology.orchestra.run.trial185.stdout: "event": "header_read", 2026-02-23T13:09:53.505 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.718053+0000", 2026-02-23T13:09:53.505 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 0 2026-02-23T13:09:53.505 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.505 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.505 INFO:teuthology.orchestra.run.trial185.stdout: "event": "throttled", 2026-02-23T13:09:53.505 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.718058+0000", 2026-02-23T13:09:53.505 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 5.6509999999999998e-06 2026-02-23T13:09:53.505 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.505 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.505 INFO:teuthology.orchestra.run.trial185.stdout: "event": "all_read", 2026-02-23T13:09:53.505 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.718085+0000", 2026-02-23T13:09:53.505 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 2.6529e-05 2026-02-23T13:09:53.505 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.505 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.505 INFO:teuthology.orchestra.run.trial185.stdout: "event": "dispatched", 2026-02-23T13:09:53.505 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.718088+0000", 2026-02-23T13:09:53.505 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 3.427e-06 2026-02-23T13:09:53.505 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.505 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.505 INFO:teuthology.orchestra.run.trial185.stdout: "event": "queued_for_pg", 2026-02-23T13:09:53.505 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.718090+0000", 2026-02-23T13:09:53.505 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 1.3319999999999999e-06 2026-02-23T13:09:53.505 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.506 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.506 INFO:teuthology.orchestra.run.trial185.stdout: "event": "reached_pg", 2026-02-23T13:09:53.506 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.718105+0000", 2026-02-23T13:09:53.506 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 1.5489e-05 2026-02-23T13:09:53.506 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.506 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.506 INFO:teuthology.orchestra.run.trial185.stdout: "event": "waiting for peered", 2026-02-23T13:09:53.506 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.718109+0000", 2026-02-23T13:09:53.506 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 3.687e-06 2026-02-23T13:09:53.506 INFO:teuthology.orchestra.run.trial185.stdout: } 2026-02-23T13:09:53.506 INFO:teuthology.orchestra.run.trial185.stdout: ] 2026-02-23T13:09:53.506 INFO:teuthology.orchestra.run.trial185.stdout: } 2026-02-23T13:09:53.506 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.506 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.506 INFO:teuthology.orchestra.run.trial185.stdout: "description": "osd_op(client.14522.0:79415 3.d 3.6d32bbed (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e80)", 2026-02-23T13:09:53.506 INFO:teuthology.orchestra.run.trial185.stdout: "initiated_at": "2026-02-23T13:09:52.733054+0000", 2026-02-23T13:09:53.506 INFO:teuthology.orchestra.run.trial185.stdout: "age": 0.66634256300000005, 2026-02-23T13:09:53.506 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 0.66689317000000004, 2026-02-23T13:09:53.506 INFO:teuthology.orchestra.run.trial185.stdout: "continuous": false, 2026-02-23T13:09:53.506 INFO:teuthology.orchestra.run.trial185.stdout: "type_data": { 2026-02-23T13:09:53.506 INFO:teuthology.orchestra.run.trial185.stdout: "flag_point": "waiting for peered", 2026-02-23T13:09:53.506 INFO:teuthology.orchestra.run.trial185.stdout: "client_info": { 2026-02-23T13:09:53.506 INFO:teuthology.orchestra.run.trial185.stdout: "client": "client.14522", 2026-02-23T13:09:53.506 INFO:teuthology.orchestra.run.trial185.stdout: "client_addr": "10.20.193.68:0/2327563370", 2026-02-23T13:09:53.506 INFO:teuthology.orchestra.run.trial185.stdout: "tid": 79415 2026-02-23T13:09:53.506 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.506 INFO:teuthology.orchestra.run.trial185.stdout: "events": [ 2026-02-23T13:09:53.506 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.506 INFO:teuthology.orchestra.run.trial185.stdout: "event": "initiated", 2026-02-23T13:09:53.506 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.733054+0000", 2026-02-23T13:09:53.506 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 0 2026-02-23T13:09:53.506 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.506 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.506 INFO:teuthology.orchestra.run.trial185.stdout: "event": "header_read", 2026-02-23T13:09:53.506 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.733054+0000", 2026-02-23T13:09:53.507 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 0 2026-02-23T13:09:53.507 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.507 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.507 INFO:teuthology.orchestra.run.trial185.stdout: "event": "throttled", 2026-02-23T13:09:53.507 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.733059+0000", 2026-02-23T13:09:53.507 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 5.2700000000000004e-06 2026-02-23T13:09:53.507 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.507 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.507 INFO:teuthology.orchestra.run.trial185.stdout: "event": "all_read", 2026-02-23T13:09:53.507 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.733081+0000", 2026-02-23T13:09:53.507 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 2.1489999999999999e-05 2026-02-23T13:09:53.507 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.507 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.507 INFO:teuthology.orchestra.run.trial185.stdout: "event": "dispatched", 2026-02-23T13:09:53.507 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.733084+0000", 2026-02-23T13:09:53.507 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 2.926e-06 2026-02-23T13:09:53.507 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.507 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.507 INFO:teuthology.orchestra.run.trial185.stdout: "event": "queued_for_pg", 2026-02-23T13:09:53.507 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.733085+0000", 2026-02-23T13:09:53.507 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 1.1620000000000001e-06 2026-02-23T13:09:53.507 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.507 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.507 INFO:teuthology.orchestra.run.trial185.stdout: "event": "reached_pg", 2026-02-23T13:09:53.507 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.733097+0000", 2026-02-23T13:09:53.507 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 1.1841999999999999e-05 2026-02-23T13:09:53.507 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.507 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.507 INFO:teuthology.orchestra.run.trial185.stdout: "event": "waiting for peered", 2026-02-23T13:09:53.507 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.733098+0000", 2026-02-23T13:09:53.507 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 1.2219999999999999e-06 2026-02-23T13:09:53.507 INFO:teuthology.orchestra.run.trial185.stdout: } 2026-02-23T13:09:53.507 INFO:teuthology.orchestra.run.trial185.stdout: ] 2026-02-23T13:09:53.507 INFO:teuthology.orchestra.run.trial185.stdout: } 2026-02-23T13:09:53.507 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.507 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.508 INFO:teuthology.orchestra.run.trial185.stdout: "description": "osd_op(client.14522.0:79224 3.d 3.38b3af2d (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e80)", 2026-02-23T13:09:53.508 INFO:teuthology.orchestra.run.trial185.stdout: "initiated_at": "2026-02-23T13:09:52.718100+0000", 2026-02-23T13:09:53.508 INFO:teuthology.orchestra.run.trial185.stdout: "age": 0.68129681099999995, 2026-02-23T13:09:53.508 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 0.68186556200000004, 2026-02-23T13:09:53.508 INFO:teuthology.orchestra.run.trial185.stdout: "continuous": false, 2026-02-23T13:09:53.508 INFO:teuthology.orchestra.run.trial185.stdout: "type_data": { 2026-02-23T13:09:53.508 INFO:teuthology.orchestra.run.trial185.stdout: "flag_point": "waiting for peered", 2026-02-23T13:09:53.508 INFO:teuthology.orchestra.run.trial185.stdout: "client_info": { 2026-02-23T13:09:53.508 INFO:teuthology.orchestra.run.trial185.stdout: "client": "client.14522", 2026-02-23T13:09:53.508 INFO:teuthology.orchestra.run.trial185.stdout: "client_addr": "10.20.193.68:0/2327563370", 2026-02-23T13:09:53.508 INFO:teuthology.orchestra.run.trial185.stdout: "tid": 79224 2026-02-23T13:09:53.508 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.508 INFO:teuthology.orchestra.run.trial185.stdout: "events": [ 2026-02-23T13:09:53.508 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.508 INFO:teuthology.orchestra.run.trial185.stdout: "event": "initiated", 2026-02-23T13:09:53.508 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.718100+0000", 2026-02-23T13:09:53.508 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 0 2026-02-23T13:09:53.508 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.508 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.508 INFO:teuthology.orchestra.run.trial185.stdout: "event": "header_read", 2026-02-23T13:09:53.508 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.718100+0000", 2026-02-23T13:09:53.508 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 0 2026-02-23T13:09:53.508 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.508 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.508 INFO:teuthology.orchestra.run.trial185.stdout: "event": "throttled", 2026-02-23T13:09:53.508 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.718111+0000", 2026-02-23T13:09:53.508 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 1.0910999999999999e-05 2026-02-23T13:09:53.508 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.508 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.508 INFO:teuthology.orchestra.run.trial185.stdout: "event": "all_read", 2026-02-23T13:09:53.508 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.718131+0000", 2026-02-23T13:09:53.508 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 2.0497999999999999e-05 2026-02-23T13:09:53.508 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.508 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.508 INFO:teuthology.orchestra.run.trial185.stdout: "event": "dispatched", 2026-02-23T13:09:53.508 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.718134+0000", 2026-02-23T13:09:53.508 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 2.8550000000000002e-06 2026-02-23T13:09:53.509 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.509 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.509 INFO:teuthology.orchestra.run.trial185.stdout: "event": "queued_for_pg", 2026-02-23T13:09:53.509 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.718135+0000", 2026-02-23T13:09:53.509 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 1.1519999999999999e-06 2026-02-23T13:09:53.509 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.509 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.509 INFO:teuthology.orchestra.run.trial185.stdout: "event": "reached_pg", 2026-02-23T13:09:53.509 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.718154+0000", 2026-02-23T13:09:53.509 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 1.8734999999999999e-05 2026-02-23T13:09:53.509 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.509 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.509 INFO:teuthology.orchestra.run.trial185.stdout: "event": "waiting for peered", 2026-02-23T13:09:53.509 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.718156+0000", 2026-02-23T13:09:53.509 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 1.894e-06 2026-02-23T13:09:53.509 INFO:teuthology.orchestra.run.trial185.stdout: } 2026-02-23T13:09:53.509 INFO:teuthology.orchestra.run.trial185.stdout: ] 2026-02-23T13:09:53.509 INFO:teuthology.orchestra.run.trial185.stdout: } 2026-02-23T13:09:53.509 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.509 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.509 INFO:teuthology.orchestra.run.trial185.stdout: "description": "osd_op(client.14522.0:79416 3.d 3.6d32bbed (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e80)", 2026-02-23T13:09:53.509 INFO:teuthology.orchestra.run.trial185.stdout: "initiated_at": "2026-02-23T13:09:52.733097+0000", 2026-02-23T13:09:53.509 INFO:teuthology.orchestra.run.trial185.stdout: "age": 0.66629979299999997, 2026-02-23T13:09:53.509 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 0.66688418299999996, 2026-02-23T13:09:53.509 INFO:teuthology.orchestra.run.trial185.stdout: "continuous": false, 2026-02-23T13:09:53.509 INFO:teuthology.orchestra.run.trial185.stdout: "type_data": { 2026-02-23T13:09:53.509 INFO:teuthology.orchestra.run.trial185.stdout: "flag_point": "waiting for peered", 2026-02-23T13:09:53.509 INFO:teuthology.orchestra.run.trial185.stdout: "client_info": { 2026-02-23T13:09:53.509 INFO:teuthology.orchestra.run.trial185.stdout: "client": "client.14522", 2026-02-23T13:09:53.509 INFO:teuthology.orchestra.run.trial185.stdout: "client_addr": "10.20.193.68:0/2327563370", 2026-02-23T13:09:53.509 INFO:teuthology.orchestra.run.trial185.stdout: "tid": 79416 2026-02-23T13:09:53.509 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.509 INFO:teuthology.orchestra.run.trial185.stdout: "events": [ 2026-02-23T13:09:53.509 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.509 INFO:teuthology.orchestra.run.trial185.stdout: "event": "initiated", 2026-02-23T13:09:53.509 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.733097+0000", 2026-02-23T13:09:53.509 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 0 2026-02-23T13:09:53.510 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.510 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.510 INFO:teuthology.orchestra.run.trial185.stdout: "event": "header_read", 2026-02-23T13:09:53.510 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.733097+0000", 2026-02-23T13:09:53.510 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 0 2026-02-23T13:09:53.510 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.510 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.510 INFO:teuthology.orchestra.run.trial185.stdout: "event": "throttled", 2026-02-23T13:09:53.510 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.733108+0000", 2026-02-23T13:09:53.510 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 1.078e-05 2026-02-23T13:09:53.510 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.510 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.510 INFO:teuthology.orchestra.run.trial185.stdout: "event": "all_read", 2026-02-23T13:09:53.510 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.733124+0000", 2026-02-23T13:09:53.510 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 1.626e-05 2026-02-23T13:09:53.510 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.510 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.510 INFO:teuthology.orchestra.run.trial185.stdout: "event": "dispatched", 2026-02-23T13:09:53.510 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.733126+0000", 2026-02-23T13:09:53.510 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 2.5349999999999999e-06 2026-02-23T13:09:53.510 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.510 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.510 INFO:teuthology.orchestra.run.trial185.stdout: "event": "queued_for_pg", 2026-02-23T13:09:53.510 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.733127+0000", 2026-02-23T13:09:53.510 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 8.8199999999999998e-07 2026-02-23T13:09:53.510 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.510 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.510 INFO:teuthology.orchestra.run.trial185.stdout: "event": "reached_pg", 2026-02-23T13:09:53.510 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.733137+0000", 2026-02-23T13:09:53.510 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 9.7380000000000006e-06 2026-02-23T13:09:53.510 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.510 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.510 INFO:teuthology.orchestra.run.trial185.stdout: "event": "waiting for peered", 2026-02-23T13:09:53.510 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.733139+0000", 2026-02-23T13:09:53.510 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 1.663e-06 2026-02-23T13:09:53.510 INFO:teuthology.orchestra.run.trial185.stdout: } 2026-02-23T13:09:53.510 INFO:teuthology.orchestra.run.trial185.stdout: ] 2026-02-23T13:09:53.510 INFO:teuthology.orchestra.run.trial185.stdout: } 2026-02-23T13:09:53.511 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.511 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.511 INFO:teuthology.orchestra.run.trial185.stdout: "description": "osd_op(client.14522.0:79265 3.d 3.a4042bad (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e80)", 2026-02-23T13:09:53.511 INFO:teuthology.orchestra.run.trial185.stdout: "initiated_at": "2026-02-23T13:09:52.718143+0000", 2026-02-23T13:09:53.511 INFO:teuthology.orchestra.run.trial185.stdout: "age": 0.68125315900000005, 2026-02-23T13:09:53.511 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 0.68185523199999998, 2026-02-23T13:09:53.511 INFO:teuthology.orchestra.run.trial185.stdout: "continuous": false, 2026-02-23T13:09:53.511 INFO:teuthology.orchestra.run.trial185.stdout: "type_data": { 2026-02-23T13:09:53.511 INFO:teuthology.orchestra.run.trial185.stdout: "flag_point": "waiting for peered", 2026-02-23T13:09:53.511 INFO:teuthology.orchestra.run.trial185.stdout: "client_info": { 2026-02-23T13:09:53.511 INFO:teuthology.orchestra.run.trial185.stdout: "client": "client.14522", 2026-02-23T13:09:53.511 INFO:teuthology.orchestra.run.trial185.stdout: "client_addr": "10.20.193.68:0/2327563370", 2026-02-23T13:09:53.511 INFO:teuthology.orchestra.run.trial185.stdout: "tid": 79265 2026-02-23T13:09:53.511 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.511 INFO:teuthology.orchestra.run.trial185.stdout: "events": [ 2026-02-23T13:09:53.511 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.511 INFO:teuthology.orchestra.run.trial185.stdout: "event": "initiated", 2026-02-23T13:09:53.511 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.718143+0000", 2026-02-23T13:09:53.511 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 0 2026-02-23T13:09:53.511 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.511 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.511 INFO:teuthology.orchestra.run.trial185.stdout: "event": "header_read", 2026-02-23T13:09:53.511 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.718143+0000", 2026-02-23T13:09:53.511 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 0 2026-02-23T13:09:53.511 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.511 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.511 INFO:teuthology.orchestra.run.trial185.stdout: "event": "throttled", 2026-02-23T13:09:53.511 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.718156+0000", 2026-02-23T13:09:53.511 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 1.2904e-05 2026-02-23T13:09:53.511 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.511 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.511 INFO:teuthology.orchestra.run.trial185.stdout: "event": "all_read", 2026-02-23T13:09:53.511 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.718178+0000", 2026-02-23T13:09:53.511 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 2.1719999999999999e-05 2026-02-23T13:09:53.511 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.511 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.511 INFO:teuthology.orchestra.run.trial185.stdout: "event": "dispatched", 2026-02-23T13:09:53.511 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.718182+0000", 2026-02-23T13:09:53.511 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 4.1779999999999996e-06 2026-02-23T13:09:53.512 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.512 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.512 INFO:teuthology.orchestra.run.trial185.stdout: "event": "queued_for_pg", 2026-02-23T13:09:53.512 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.718186+0000", 2026-02-23T13:09:53.512 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 3.416e-06 2026-02-23T13:09:53.512 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.512 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.512 INFO:teuthology.orchestra.run.trial185.stdout: "event": "reached_pg", 2026-02-23T13:09:53.512 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.718205+0000", 2026-02-23T13:09:53.512 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 1.8896e-05 2026-02-23T13:09:53.512 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.512 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.512 INFO:teuthology.orchestra.run.trial185.stdout: "event": "waiting for peered", 2026-02-23T13:09:53.512 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.718207+0000", 2026-02-23T13:09:53.512 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 2.835e-06 2026-02-23T13:09:53.512 INFO:teuthology.orchestra.run.trial185.stdout: } 2026-02-23T13:09:53.512 INFO:teuthology.orchestra.run.trial185.stdout: ] 2026-02-23T13:09:53.512 INFO:teuthology.orchestra.run.trial185.stdout: } 2026-02-23T13:09:53.512 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.512 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.512 INFO:teuthology.orchestra.run.trial185.stdout: "description": "osd_op(client.14522.0:79266 3.d 3.a4042bad (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e80)", 2026-02-23T13:09:53.512 INFO:teuthology.orchestra.run.trial185.stdout: "initiated_at": "2026-02-23T13:09:52.718195+0000", 2026-02-23T13:09:53.512 INFO:teuthology.orchestra.run.trial185.stdout: "age": 0.68120143300000002, 2026-02-23T13:09:53.512 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 0.68181889500000004, 2026-02-23T13:09:53.512 INFO:teuthology.orchestra.run.trial185.stdout: "continuous": false, 2026-02-23T13:09:53.512 INFO:teuthology.orchestra.run.trial185.stdout: "type_data": { 2026-02-23T13:09:53.512 INFO:teuthology.orchestra.run.trial185.stdout: "flag_point": "waiting for peered", 2026-02-23T13:09:53.512 INFO:teuthology.orchestra.run.trial185.stdout: "client_info": { 2026-02-23T13:09:53.512 INFO:teuthology.orchestra.run.trial185.stdout: "client": "client.14522", 2026-02-23T13:09:53.512 INFO:teuthology.orchestra.run.trial185.stdout: "client_addr": "10.20.193.68:0/2327563370", 2026-02-23T13:09:53.512 INFO:teuthology.orchestra.run.trial185.stdout: "tid": 79266 2026-02-23T13:09:53.512 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.512 INFO:teuthology.orchestra.run.trial185.stdout: "events": [ 2026-02-23T13:09:53.512 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.512 INFO:teuthology.orchestra.run.trial185.stdout: "event": "initiated", 2026-02-23T13:09:53.512 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.718195+0000", 2026-02-23T13:09:53.512 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 0 2026-02-23T13:09:53.512 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.512 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.512 INFO:teuthology.orchestra.run.trial185.stdout: "event": "header_read", 2026-02-23T13:09:53.513 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.718195+0000", 2026-02-23T13:09:53.513 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 0 2026-02-23T13:09:53.513 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.513 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.513 INFO:teuthology.orchestra.run.trial185.stdout: "event": "throttled", 2026-02-23T13:09:53.513 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.718205+0000", 2026-02-23T13:09:53.513 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 9.7780000000000002e-06 2026-02-23T13:09:53.513 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.513 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.513 INFO:teuthology.orchestra.run.trial185.stdout: "event": "all_read", 2026-02-23T13:09:53.513 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.718224+0000", 2026-02-23T13:09:53.513 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 1.9366999999999999e-05 2026-02-23T13:09:53.513 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.513 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.513 INFO:teuthology.orchestra.run.trial185.stdout: "event": "dispatched", 2026-02-23T13:09:53.513 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.718228+0000", 2026-02-23T13:09:53.513 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 3.2959999999999999e-06 2026-02-23T13:09:53.513 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.513 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.513 INFO:teuthology.orchestra.run.trial185.stdout: "event": "queued_for_pg", 2026-02-23T13:09:53.513 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.718229+0000", 2026-02-23T13:09:53.513 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 9.6200000000000006e-07 2026-02-23T13:09:53.513 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.513 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.513 INFO:teuthology.orchestra.run.trial185.stdout: "event": "reached_pg", 2026-02-23T13:09:53.513 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.718245+0000", 2026-02-23T13:09:53.513 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 1.6861e-05 2026-02-23T13:09:53.513 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.513 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.513 INFO:teuthology.orchestra.run.trial185.stdout: "event": "waiting for peered", 2026-02-23T13:09:53.513 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.718247+0000", 2026-02-23T13:09:53.513 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 1.7829999999999999e-06 2026-02-23T13:09:53.513 INFO:teuthology.orchestra.run.trial185.stdout: } 2026-02-23T13:09:53.513 INFO:teuthology.orchestra.run.trial185.stdout: ] 2026-02-23T13:09:53.513 INFO:teuthology.orchestra.run.trial185.stdout: } 2026-02-23T13:09:53.513 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.513 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.513 INFO:teuthology.orchestra.run.trial185.stdout: "description": "osd_op(client.14522.0:79267 3.d 3.a4042bad (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e80)", 2026-02-23T13:09:53.513 INFO:teuthology.orchestra.run.trial185.stdout: "initiated_at": "2026-02-23T13:09:52.718237+0000", 2026-02-23T13:09:53.513 INFO:teuthology.orchestra.run.trial185.stdout: "age": 0.68115967499999996, 2026-02-23T13:09:53.513 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 0.68179432900000003, 2026-02-23T13:09:53.514 INFO:teuthology.orchestra.run.trial185.stdout: "continuous": false, 2026-02-23T13:09:53.514 INFO:teuthology.orchestra.run.trial185.stdout: "type_data": { 2026-02-23T13:09:53.514 INFO:teuthology.orchestra.run.trial185.stdout: "flag_point": "waiting for peered", 2026-02-23T13:09:53.514 INFO:teuthology.orchestra.run.trial185.stdout: "client_info": { 2026-02-23T13:09:53.514 INFO:teuthology.orchestra.run.trial185.stdout: "client": "client.14522", 2026-02-23T13:09:53.514 INFO:teuthology.orchestra.run.trial185.stdout: "client_addr": "10.20.193.68:0/2327563370", 2026-02-23T13:09:53.514 INFO:teuthology.orchestra.run.trial185.stdout: "tid": 79267 2026-02-23T13:09:53.514 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.514 INFO:teuthology.orchestra.run.trial185.stdout: "events": [ 2026-02-23T13:09:53.514 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.514 INFO:teuthology.orchestra.run.trial185.stdout: "event": "initiated", 2026-02-23T13:09:53.514 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.718237+0000", 2026-02-23T13:09:53.514 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 0 2026-02-23T13:09:53.514 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.514 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.514 INFO:teuthology.orchestra.run.trial185.stdout: "event": "header_read", 2026-02-23T13:09:53.514 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.718237+0000", 2026-02-23T13:09:53.514 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 0 2026-02-23T13:09:53.514 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.514 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.514 INFO:teuthology.orchestra.run.trial185.stdout: "event": "throttled", 2026-02-23T13:09:53.514 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.718245+0000", 2026-02-23T13:09:53.514 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 7.7249999999999997e-06 2026-02-23T13:09:53.514 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.514 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.514 INFO:teuthology.orchestra.run.trial185.stdout: "event": "all_read", 2026-02-23T13:09:53.514 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.718270+0000", 2026-02-23T13:09:53.514 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 2.5256999999999999e-05 2026-02-23T13:09:53.514 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.514 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.514 INFO:teuthology.orchestra.run.trial185.stdout: "event": "dispatched", 2026-02-23T13:09:53.514 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.718273+0000", 2026-02-23T13:09:53.514 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 2.9050000000000001e-06 2026-02-23T13:09:53.514 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.514 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.514 INFO:teuthology.orchestra.run.trial185.stdout: "event": "queued_for_pg", 2026-02-23T13:09:53.514 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.718274+0000", 2026-02-23T13:09:53.514 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 9.7199999999999997e-07 2026-02-23T13:09:53.514 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.514 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.514 INFO:teuthology.orchestra.run.trial185.stdout: "event": "reached_pg", 2026-02-23T13:09:53.514 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.718289+0000", 2026-02-23T13:09:53.515 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 1.5108e-05 2026-02-23T13:09:53.515 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.515 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.515 INFO:teuthology.orchestra.run.trial185.stdout: "event": "waiting for peered", 2026-02-23T13:09:53.515 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.718293+0000", 2026-02-23T13:09:53.515 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 3.7170000000000002e-06 2026-02-23T13:09:53.515 INFO:teuthology.orchestra.run.trial185.stdout: } 2026-02-23T13:09:53.515 INFO:teuthology.orchestra.run.trial185.stdout: ] 2026-02-23T13:09:53.515 INFO:teuthology.orchestra.run.trial185.stdout: } 2026-02-23T13:09:53.515 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.515 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.515 INFO:teuthology.orchestra.run.trial185.stdout: "description": "osd_op(client.14522.0:79268 3.d 3.a4042bad (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e80)", 2026-02-23T13:09:53.515 INFO:teuthology.orchestra.run.trial185.stdout: "initiated_at": "2026-02-23T13:09:52.718282+0000", 2026-02-23T13:09:53.515 INFO:teuthology.orchestra.run.trial185.stdout: "age": 0.68111451099999998, 2026-02-23T13:09:53.515 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 0.68176645700000005, 2026-02-23T13:09:53.515 INFO:teuthology.orchestra.run.trial185.stdout: "continuous": false, 2026-02-23T13:09:53.515 INFO:teuthology.orchestra.run.trial185.stdout: "type_data": { 2026-02-23T13:09:53.515 INFO:teuthology.orchestra.run.trial185.stdout: "flag_point": "waiting for peered", 2026-02-23T13:09:53.515 INFO:teuthology.orchestra.run.trial185.stdout: "client_info": { 2026-02-23T13:09:53.515 INFO:teuthology.orchestra.run.trial185.stdout: "client": "client.14522", 2026-02-23T13:09:53.515 INFO:teuthology.orchestra.run.trial185.stdout: "client_addr": "10.20.193.68:0/2327563370", 2026-02-23T13:09:53.515 INFO:teuthology.orchestra.run.trial185.stdout: "tid": 79268 2026-02-23T13:09:53.515 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.515 INFO:teuthology.orchestra.run.trial185.stdout: "events": [ 2026-02-23T13:09:53.515 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.515 INFO:teuthology.orchestra.run.trial185.stdout: "event": "initiated", 2026-02-23T13:09:53.515 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.718282+0000", 2026-02-23T13:09:53.515 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 0 2026-02-23T13:09:53.515 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.515 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.515 INFO:teuthology.orchestra.run.trial185.stdout: "event": "header_read", 2026-02-23T13:09:53.515 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.718282+0000", 2026-02-23T13:09:53.515 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 0 2026-02-23T13:09:53.515 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.515 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.515 INFO:teuthology.orchestra.run.trial185.stdout: "event": "throttled", 2026-02-23T13:09:53.515 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.718289+0000", 2026-02-23T13:09:53.515 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 6.5230000000000001e-06 2026-02-23T13:09:53.515 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.515 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.515 INFO:teuthology.orchestra.run.trial185.stdout: "event": "all_read", 2026-02-23T13:09:53.515 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.718324+0000", 2026-02-23T13:09:53.516 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 3.5305000000000002e-05 2026-02-23T13:09:53.516 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.516 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.516 INFO:teuthology.orchestra.run.trial185.stdout: "event": "dispatched", 2026-02-23T13:09:53.516 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.718328+0000", 2026-02-23T13:09:53.516 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 3.8070000000000001e-06 2026-02-23T13:09:53.516 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.516 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.516 INFO:teuthology.orchestra.run.trial185.stdout: "event": "queued_for_pg", 2026-02-23T13:09:53.516 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.718329+0000", 2026-02-23T13:09:53.516 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 1.2130000000000001e-06 2026-02-23T13:09:53.516 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.516 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.516 INFO:teuthology.orchestra.run.trial185.stdout: "event": "reached_pg", 2026-02-23T13:09:53.516 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.718354+0000", 2026-02-23T13:09:53.516 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 2.4766e-05 2026-02-23T13:09:53.516 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.516 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.516 INFO:teuthology.orchestra.run.trial185.stdout: "event": "waiting for peered", 2026-02-23T13:09:53.516 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.718356+0000", 2026-02-23T13:09:53.516 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 2.424e-06 2026-02-23T13:09:53.516 INFO:teuthology.orchestra.run.trial185.stdout: } 2026-02-23T13:09:53.516 INFO:teuthology.orchestra.run.trial185.stdout: ] 2026-02-23T13:09:53.516 INFO:teuthology.orchestra.run.trial185.stdout: } 2026-02-23T13:09:53.516 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.516 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.516 INFO:teuthology.orchestra.run.trial185.stdout: "description": "osd_op(client.14522.0:79269 3.d 3.a4042bad (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e80)", 2026-02-23T13:09:53.516 INFO:teuthology.orchestra.run.trial185.stdout: "initiated_at": "2026-02-23T13:09:52.718341+0000", 2026-02-23T13:09:53.516 INFO:teuthology.orchestra.run.trial185.stdout: "age": 0.68105512999999995, 2026-02-23T13:09:53.516 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 0.68172226499999999, 2026-02-23T13:09:53.516 INFO:teuthology.orchestra.run.trial185.stdout: "continuous": false, 2026-02-23T13:09:53.516 INFO:teuthology.orchestra.run.trial185.stdout: "type_data": { 2026-02-23T13:09:53.516 INFO:teuthology.orchestra.run.trial185.stdout: "flag_point": "waiting for peered", 2026-02-23T13:09:53.516 INFO:teuthology.orchestra.run.trial185.stdout: "client_info": { 2026-02-23T13:09:53.516 INFO:teuthology.orchestra.run.trial185.stdout: "client": "client.14522", 2026-02-23T13:09:53.516 INFO:teuthology.orchestra.run.trial185.stdout: "client_addr": "10.20.193.68:0/2327563370", 2026-02-23T13:09:53.516 INFO:teuthology.orchestra.run.trial185.stdout: "tid": 79269 2026-02-23T13:09:53.516 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.516 INFO:teuthology.orchestra.run.trial185.stdout: "events": [ 2026-02-23T13:09:53.516 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.516 INFO:teuthology.orchestra.run.trial185.stdout: "event": "initiated", 2026-02-23T13:09:53.516 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.718341+0000", 2026-02-23T13:09:53.516 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 0 2026-02-23T13:09:53.517 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.517 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.517 INFO:teuthology.orchestra.run.trial185.stdout: "event": "header_read", 2026-02-23T13:09:53.517 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.718341+0000", 2026-02-23T13:09:53.517 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 0 2026-02-23T13:09:53.517 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.517 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.517 INFO:teuthology.orchestra.run.trial185.stdout: "event": "throttled", 2026-02-23T13:09:53.517 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.718359+0000", 2026-02-23T13:09:53.517 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 1.7782999999999999e-05 2026-02-23T13:09:53.517 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.517 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.517 INFO:teuthology.orchestra.run.trial185.stdout: "event": "all_read", 2026-02-23T13:09:53.517 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.718383+0000", 2026-02-23T13:09:53.517 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 2.4195000000000001e-05 2026-02-23T13:09:53.517 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.517 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.517 INFO:teuthology.orchestra.run.trial185.stdout: "event": "dispatched", 2026-02-23T13:09:53.517 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.718387+0000", 2026-02-23T13:09:53.517 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 3.0860000000000002e-06 2026-02-23T13:09:53.517 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.517 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.517 INFO:teuthology.orchestra.run.trial185.stdout: "event": "queued_for_pg", 2026-02-23T13:09:53.517 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.718388+0000", 2026-02-23T13:09:53.517 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 9.6200000000000006e-07 2026-02-23T13:09:53.517 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.517 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.517 INFO:teuthology.orchestra.run.trial185.stdout: "event": "reached_pg", 2026-02-23T13:09:53.517 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.718406+0000", 2026-02-23T13:09:53.517 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 1.8513999999999999e-05 2026-02-23T13:09:53.517 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.517 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.517 INFO:teuthology.orchestra.run.trial185.stdout: "event": "waiting for peered", 2026-02-23T13:09:53.517 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.718408+0000", 2026-02-23T13:09:53.517 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 2.0540000000000002e-06 2026-02-23T13:09:53.517 INFO:teuthology.orchestra.run.trial185.stdout: } 2026-02-23T13:09:53.517 INFO:teuthology.orchestra.run.trial185.stdout: ] 2026-02-23T13:09:53.517 INFO:teuthology.orchestra.run.trial185.stdout: } 2026-02-23T13:09:53.517 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.517 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.517 INFO:teuthology.orchestra.run.trial185.stdout: "description": "osd_op(client.14522.0:79270 3.d 3.a4042bad (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e80)", 2026-02-23T13:09:53.517 INFO:teuthology.orchestra.run.trial185.stdout: "initiated_at": "2026-02-23T13:09:52.718397+0000", 2026-02-23T13:09:53.518 INFO:teuthology.orchestra.run.trial185.stdout: "age": 0.68099995700000004, 2026-02-23T13:09:53.518 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 0.68168759000000001, 2026-02-23T13:09:53.518 INFO:teuthology.orchestra.run.trial185.stdout: "continuous": false, 2026-02-23T13:09:53.518 INFO:teuthology.orchestra.run.trial185.stdout: "type_data": { 2026-02-23T13:09:53.518 INFO:teuthology.orchestra.run.trial185.stdout: "flag_point": "waiting for peered", 2026-02-23T13:09:53.518 INFO:teuthology.orchestra.run.trial185.stdout: "client_info": { 2026-02-23T13:09:53.518 INFO:teuthology.orchestra.run.trial185.stdout: "client": "client.14522", 2026-02-23T13:09:53.518 INFO:teuthology.orchestra.run.trial185.stdout: "client_addr": "10.20.193.68:0/2327563370", 2026-02-23T13:09:53.518 INFO:teuthology.orchestra.run.trial185.stdout: "tid": 79270 2026-02-23T13:09:53.518 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.518 INFO:teuthology.orchestra.run.trial185.stdout: "events": [ 2026-02-23T13:09:53.518 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.518 INFO:teuthology.orchestra.run.trial185.stdout: "event": "initiated", 2026-02-23T13:09:53.518 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.718397+0000", 2026-02-23T13:09:53.518 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 0 2026-02-23T13:09:53.518 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.518 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.518 INFO:teuthology.orchestra.run.trial185.stdout: "event": "header_read", 2026-02-23T13:09:53.518 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.718397+0000", 2026-02-23T13:09:53.518 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 0 2026-02-23T13:09:53.518 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.518 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.518 INFO:teuthology.orchestra.run.trial185.stdout: "event": "throttled", 2026-02-23T13:09:53.518 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.718407+0000", 2026-02-23T13:09:53.518 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 1.0499999999999999e-05 2026-02-23T13:09:53.518 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.518 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.518 INFO:teuthology.orchestra.run.trial185.stdout: "event": "all_read", 2026-02-23T13:09:53.518 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.718427+0000", 2026-02-23T13:09:53.518 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 2.0007e-05 2026-02-23T13:09:53.518 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.518 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.518 INFO:teuthology.orchestra.run.trial185.stdout: "event": "dispatched", 2026-02-23T13:09:53.518 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.718430+0000", 2026-02-23T13:09:53.518 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 3.1360000000000001e-06 2026-02-23T13:09:53.518 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.518 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.518 INFO:teuthology.orchestra.run.trial185.stdout: "event": "queued_for_pg", 2026-02-23T13:09:53.518 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.718431+0000", 2026-02-23T13:09:53.518 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 1.1119999999999999e-06 2026-02-23T13:09:53.518 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.518 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.518 INFO:teuthology.orchestra.run.trial185.stdout: "event": "reached_pg", 2026-02-23T13:09:53.519 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.718448+0000", 2026-02-23T13:09:53.519 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 1.6841e-05 2026-02-23T13:09:53.519 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.519 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.519 INFO:teuthology.orchestra.run.trial185.stdout: "event": "waiting for peered", 2026-02-23T13:09:53.519 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.718450+0000", 2026-02-23T13:09:53.519 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 1.784e-06 2026-02-23T13:09:53.519 INFO:teuthology.orchestra.run.trial185.stdout: } 2026-02-23T13:09:53.519 INFO:teuthology.orchestra.run.trial185.stdout: ] 2026-02-23T13:09:53.519 INFO:teuthology.orchestra.run.trial185.stdout: } 2026-02-23T13:09:53.519 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.519 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.519 INFO:teuthology.orchestra.run.trial185.stdout: "description": "osd_op(client.14522.0:79271 3.d 3.a4042bad (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e80)", 2026-02-23T13:09:53.519 INFO:teuthology.orchestra.run.trial185.stdout: "initiated_at": "2026-02-23T13:09:52.718440+0000", 2026-02-23T13:09:53.519 INFO:teuthology.orchestra.run.trial185.stdout: "age": 0.68095684700000003, 2026-02-23T13:09:53.519 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 0.68166558899999996, 2026-02-23T13:09:53.519 INFO:teuthology.orchestra.run.trial185.stdout: "continuous": false, 2026-02-23T13:09:53.519 INFO:teuthology.orchestra.run.trial185.stdout: "type_data": { 2026-02-23T13:09:53.519 INFO:teuthology.orchestra.run.trial185.stdout: "flag_point": "waiting for peered", 2026-02-23T13:09:53.519 INFO:teuthology.orchestra.run.trial185.stdout: "client_info": { 2026-02-23T13:09:53.519 INFO:teuthology.orchestra.run.trial185.stdout: "client": "client.14522", 2026-02-23T13:09:53.519 INFO:teuthology.orchestra.run.trial185.stdout: "client_addr": "10.20.193.68:0/2327563370", 2026-02-23T13:09:53.519 INFO:teuthology.orchestra.run.trial185.stdout: "tid": 79271 2026-02-23T13:09:53.519 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.519 INFO:teuthology.orchestra.run.trial185.stdout: "events": [ 2026-02-23T13:09:53.519 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.519 INFO:teuthology.orchestra.run.trial185.stdout: "event": "initiated", 2026-02-23T13:09:53.519 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.718440+0000", 2026-02-23T13:09:53.519 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 0 2026-02-23T13:09:53.519 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.519 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.519 INFO:teuthology.orchestra.run.trial185.stdout: "event": "header_read", 2026-02-23T13:09:53.519 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.718440+0000", 2026-02-23T13:09:53.519 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 0 2026-02-23T13:09:53.519 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.519 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.519 INFO:teuthology.orchestra.run.trial185.stdout: "event": "throttled", 2026-02-23T13:09:53.519 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.718446+0000", 2026-02-23T13:09:53.519 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 5.8610000000000003e-06 2026-02-23T13:09:53.519 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.519 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.519 INFO:teuthology.orchestra.run.trial185.stdout: "event": "all_read", 2026-02-23T13:09:53.519 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.718473+0000", 2026-02-23T13:09:53.519 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 2.7141e-05 2026-02-23T13:09:53.520 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.520 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.520 INFO:teuthology.orchestra.run.trial185.stdout: "event": "dispatched", 2026-02-23T13:09:53.520 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.718476+0000", 2026-02-23T13:09:53.520 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 2.8949999999999998e-06 2026-02-23T13:09:53.520 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.520 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.520 INFO:teuthology.orchestra.run.trial185.stdout: "event": "queued_for_pg", 2026-02-23T13:09:53.520 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.718477+0000", 2026-02-23T13:09:53.520 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 1.0219999999999999e-06 2026-02-23T13:09:53.520 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.520 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.520 INFO:teuthology.orchestra.run.trial185.stdout: "event": "reached_pg", 2026-02-23T13:09:53.520 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.718494+0000", 2026-02-23T13:09:53.520 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 1.7132000000000001e-05 2026-02-23T13:09:53.520 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.520 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.520 INFO:teuthology.orchestra.run.trial185.stdout: "event": "waiting for peered", 2026-02-23T13:09:53.520 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.718496+0000", 2026-02-23T13:09:53.520 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 1.844e-06 2026-02-23T13:09:53.520 INFO:teuthology.orchestra.run.trial185.stdout: } 2026-02-23T13:09:53.520 INFO:teuthology.orchestra.run.trial185.stdout: ] 2026-02-23T13:09:53.520 INFO:teuthology.orchestra.run.trial185.stdout: } 2026-02-23T13:09:53.520 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.520 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.520 INFO:teuthology.orchestra.run.trial185.stdout: "description": "osd_op(client.14522.0:79272 3.d 3.a4042bad (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e80)", 2026-02-23T13:09:53.520 INFO:teuthology.orchestra.run.trial185.stdout: "initiated_at": "2026-02-23T13:09:52.718485+0000", 2026-02-23T13:09:53.520 INFO:teuthology.orchestra.run.trial185.stdout: "age": 0.68091121099999996, 2026-02-23T13:09:53.520 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 0.68163746599999997, 2026-02-23T13:09:53.520 INFO:teuthology.orchestra.run.trial185.stdout: "continuous": false, 2026-02-23T13:09:53.520 INFO:teuthology.orchestra.run.trial185.stdout: "type_data": { 2026-02-23T13:09:53.520 INFO:teuthology.orchestra.run.trial185.stdout: "flag_point": "waiting for peered", 2026-02-23T13:09:53.520 INFO:teuthology.orchestra.run.trial185.stdout: "client_info": { 2026-02-23T13:09:53.520 INFO:teuthology.orchestra.run.trial185.stdout: "client": "client.14522", 2026-02-23T13:09:53.520 INFO:teuthology.orchestra.run.trial185.stdout: "client_addr": "10.20.193.68:0/2327563370", 2026-02-23T13:09:53.520 INFO:teuthology.orchestra.run.trial185.stdout: "tid": 79272 2026-02-23T13:09:53.520 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.520 INFO:teuthology.orchestra.run.trial185.stdout: "events": [ 2026-02-23T13:09:53.520 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.520 INFO:teuthology.orchestra.run.trial185.stdout: "event": "initiated", 2026-02-23T13:09:53.520 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.718485+0000", 2026-02-23T13:09:53.520 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 0 2026-02-23T13:09:53.520 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.520 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.520 INFO:teuthology.orchestra.run.trial185.stdout: "event": "header_read", 2026-02-23T13:09:53.521 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.718485+0000", 2026-02-23T13:09:53.521 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 0 2026-02-23T13:09:53.521 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.521 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.521 INFO:teuthology.orchestra.run.trial185.stdout: "event": "throttled", 2026-02-23T13:09:53.521 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.718500+0000", 2026-02-23T13:09:53.521 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 1.4596999999999999e-05 2026-02-23T13:09:53.521 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.521 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.521 INFO:teuthology.orchestra.run.trial185.stdout: "event": "all_read", 2026-02-23T13:09:53.521 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.718521+0000", 2026-02-23T13:09:53.521 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 2.0718000000000001e-05 2026-02-23T13:09:53.521 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.521 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.521 INFO:teuthology.orchestra.run.trial185.stdout: "event": "dispatched", 2026-02-23T13:09:53.521 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.718524+0000", 2026-02-23T13:09:53.521 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 3.106e-06 2026-02-23T13:09:53.521 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.521 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.521 INFO:teuthology.orchestra.run.trial185.stdout: "event": "queued_for_pg", 2026-02-23T13:09:53.521 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.718525+0000", 2026-02-23T13:09:53.521 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 1.0920000000000001e-06 2026-02-23T13:09:53.521 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.521 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.521 INFO:teuthology.orchestra.run.trial185.stdout: "event": "reached_pg", 2026-02-23T13:09:53.521 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.718543+0000", 2026-02-23T13:09:53.521 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 1.7904e-05 2026-02-23T13:09:53.521 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.521 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.521 INFO:teuthology.orchestra.run.trial185.stdout: "event": "waiting for peered", 2026-02-23T13:09:53.521 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.718545+0000", 2026-02-23T13:09:53.521 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 2.3640000000000002e-06 2026-02-23T13:09:53.521 INFO:teuthology.orchestra.run.trial185.stdout: } 2026-02-23T13:09:53.521 INFO:teuthology.orchestra.run.trial185.stdout: ] 2026-02-23T13:09:53.521 INFO:teuthology.orchestra.run.trial185.stdout: } 2026-02-23T13:09:53.521 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.521 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.521 INFO:teuthology.orchestra.run.trial185.stdout: "description": "osd_op(client.14522.0:79273 3.c 3.8e4a5dec (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e80)", 2026-02-23T13:09:53.521 INFO:teuthology.orchestra.run.trial185.stdout: "initiated_at": "2026-02-23T13:09:52.718533+0000", 2026-02-23T13:09:53.521 INFO:teuthology.orchestra.run.trial185.stdout: "age": 0.68086326200000002, 2026-02-23T13:09:53.521 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 0.68160690899999998, 2026-02-23T13:09:53.521 INFO:teuthology.orchestra.run.trial185.stdout: "continuous": false, 2026-02-23T13:09:53.521 INFO:teuthology.orchestra.run.trial185.stdout: "type_data": { 2026-02-23T13:09:53.521 INFO:teuthology.orchestra.run.trial185.stdout: "flag_point": "waiting for peered", 2026-02-23T13:09:53.522 INFO:teuthology.orchestra.run.trial185.stdout: "client_info": { 2026-02-23T13:09:53.522 INFO:teuthology.orchestra.run.trial185.stdout: "client": "client.14522", 2026-02-23T13:09:53.522 INFO:teuthology.orchestra.run.trial185.stdout: "client_addr": "10.20.193.68:0/2327563370", 2026-02-23T13:09:53.522 INFO:teuthology.orchestra.run.trial185.stdout: "tid": 79273 2026-02-23T13:09:53.522 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.522 INFO:teuthology.orchestra.run.trial185.stdout: "events": [ 2026-02-23T13:09:53.522 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.522 INFO:teuthology.orchestra.run.trial185.stdout: "event": "initiated", 2026-02-23T13:09:53.522 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.718533+0000", 2026-02-23T13:09:53.522 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 0 2026-02-23T13:09:53.522 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.522 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.522 INFO:teuthology.orchestra.run.trial185.stdout: "event": "header_read", 2026-02-23T13:09:53.522 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.718533+0000", 2026-02-23T13:09:53.522 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 0 2026-02-23T13:09:53.522 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.522 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.522 INFO:teuthology.orchestra.run.trial185.stdout: "event": "throttled", 2026-02-23T13:09:53.522 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.718541+0000", 2026-02-23T13:09:53.522 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 7.464e-06 2026-02-23T13:09:53.522 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.522 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.522 INFO:teuthology.orchestra.run.trial185.stdout: "event": "all_read", 2026-02-23T13:09:53.522 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.718573+0000", 2026-02-23T13:09:53.522 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 3.1999999999999999e-05 2026-02-23T13:09:53.522 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.522 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.522 INFO:teuthology.orchestra.run.trial185.stdout: "event": "dispatched", 2026-02-23T13:09:53.522 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.718576+0000", 2026-02-23T13:09:53.522 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 3.4259999999999999e-06 2026-02-23T13:09:53.522 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.522 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.522 INFO:teuthology.orchestra.run.trial185.stdout: "event": "queued_for_pg", 2026-02-23T13:09:53.522 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.718577+0000", 2026-02-23T13:09:53.522 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 1.192e-06 2026-02-23T13:09:53.522 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.522 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.522 INFO:teuthology.orchestra.run.trial185.stdout: "event": "reached_pg", 2026-02-23T13:09:53.522 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.718594+0000", 2026-02-23T13:09:53.522 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 1.6280999999999999e-05 2026-02-23T13:09:53.522 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.522 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.522 INFO:teuthology.orchestra.run.trial185.stdout: "event": "waiting for peered", 2026-02-23T13:09:53.522 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.718597+0000", 2026-02-23T13:09:53.523 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 3.0750000000000002e-06 2026-02-23T13:09:53.523 INFO:teuthology.orchestra.run.trial185.stdout: } 2026-02-23T13:09:53.523 INFO:teuthology.orchestra.run.trial185.stdout: ] 2026-02-23T13:09:53.523 INFO:teuthology.orchestra.run.trial185.stdout: } 2026-02-23T13:09:53.523 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.523 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.523 INFO:teuthology.orchestra.run.trial185.stdout: "description": "osd_op(client.14522.0:79274 3.c 3.8e4a5dec (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e80)", 2026-02-23T13:09:53.523 INFO:teuthology.orchestra.run.trial185.stdout: "initiated_at": "2026-02-23T13:09:52.718587+0000", 2026-02-23T13:09:53.523 INFO:teuthology.orchestra.run.trial185.stdout: "age": 0.68080917100000005, 2026-02-23T13:09:53.523 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 0.68156831699999998, 2026-02-23T13:09:53.523 INFO:teuthology.orchestra.run.trial185.stdout: "continuous": false, 2026-02-23T13:09:53.523 INFO:teuthology.orchestra.run.trial185.stdout: "type_data": { 2026-02-23T13:09:53.523 INFO:teuthology.orchestra.run.trial185.stdout: "flag_point": "waiting for peered", 2026-02-23T13:09:53.523 INFO:teuthology.orchestra.run.trial185.stdout: "client_info": { 2026-02-23T13:09:53.523 INFO:teuthology.orchestra.run.trial185.stdout: "client": "client.14522", 2026-02-23T13:09:53.523 INFO:teuthology.orchestra.run.trial185.stdout: "client_addr": "10.20.193.68:0/2327563370", 2026-02-23T13:09:53.523 INFO:teuthology.orchestra.run.trial185.stdout: "tid": 79274 2026-02-23T13:09:53.523 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.523 INFO:teuthology.orchestra.run.trial185.stdout: "events": [ 2026-02-23T13:09:53.523 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.523 INFO:teuthology.orchestra.run.trial185.stdout: "event": "initiated", 2026-02-23T13:09:53.523 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.718587+0000", 2026-02-23T13:09:53.523 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 0 2026-02-23T13:09:53.523 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.523 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.523 INFO:teuthology.orchestra.run.trial185.stdout: "event": "header_read", 2026-02-23T13:09:53.523 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.718587+0000", 2026-02-23T13:09:53.523 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 0 2026-02-23T13:09:53.523 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.523 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.523 INFO:teuthology.orchestra.run.trial185.stdout: "event": "throttled", 2026-02-23T13:09:53.523 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.718600+0000", 2026-02-23T13:09:53.523 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 1.2622999999999999e-05 2026-02-23T13:09:53.523 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.523 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.523 INFO:teuthology.orchestra.run.trial185.stdout: "event": "all_read", 2026-02-23T13:09:53.523 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.718621+0000", 2026-02-23T13:09:53.523 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 2.141e-05 2026-02-23T13:09:53.523 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.523 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.523 INFO:teuthology.orchestra.run.trial185.stdout: "event": "dispatched", 2026-02-23T13:09:53.523 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.718624+0000", 2026-02-23T13:09:53.523 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 2.7760000000000002e-06 2026-02-23T13:09:53.523 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.523 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.524 INFO:teuthology.orchestra.run.trial185.stdout: "event": "queued_for_pg", 2026-02-23T13:09:53.524 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.718627+0000", 2026-02-23T13:09:53.524 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 2.9749999999999999e-06 2026-02-23T13:09:53.524 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.524 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.524 INFO:teuthology.orchestra.run.trial185.stdout: "event": "reached_pg", 2026-02-23T13:09:53.524 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.718657+0000", 2026-02-23T13:09:53.524 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 2.9544999999999999e-05 2026-02-23T13:09:53.524 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.524 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.524 INFO:teuthology.orchestra.run.trial185.stdout: "event": "waiting for peered", 2026-02-23T13:09:53.524 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.718659+0000", 2026-02-23T13:09:53.524 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 2.4449999999999999e-06 2026-02-23T13:09:53.524 INFO:teuthology.orchestra.run.trial185.stdout: } 2026-02-23T13:09:53.524 INFO:teuthology.orchestra.run.trial185.stdout: ] 2026-02-23T13:09:53.524 INFO:teuthology.orchestra.run.trial185.stdout: } 2026-02-23T13:09:53.524 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.524 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.524 INFO:teuthology.orchestra.run.trial185.stdout: "description": "osd_op(client.14522.0:79275 3.c 3.8e4a5dec (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e80)", 2026-02-23T13:09:53.524 INFO:teuthology.orchestra.run.trial185.stdout: "initiated_at": "2026-02-23T13:09:52.718637+0000", 2026-02-23T13:09:53.524 INFO:teuthology.orchestra.run.trial185.stdout: "age": 0.68076009999999998, 2026-02-23T13:09:53.524 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 0.68153643799999997, 2026-02-23T13:09:53.524 INFO:teuthology.orchestra.run.trial185.stdout: "continuous": false, 2026-02-23T13:09:53.524 INFO:teuthology.orchestra.run.trial185.stdout: "type_data": { 2026-02-23T13:09:53.524 INFO:teuthology.orchestra.run.trial185.stdout: "flag_point": "waiting for peered", 2026-02-23T13:09:53.524 INFO:teuthology.orchestra.run.trial185.stdout: "client_info": { 2026-02-23T13:09:53.524 INFO:teuthology.orchestra.run.trial185.stdout: "client": "client.14522", 2026-02-23T13:09:53.524 INFO:teuthology.orchestra.run.trial185.stdout: "client_addr": "10.20.193.68:0/2327563370", 2026-02-23T13:09:53.524 INFO:teuthology.orchestra.run.trial185.stdout: "tid": 79275 2026-02-23T13:09:53.524 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.524 INFO:teuthology.orchestra.run.trial185.stdout: "events": [ 2026-02-23T13:09:53.524 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.524 INFO:teuthology.orchestra.run.trial185.stdout: "event": "initiated", 2026-02-23T13:09:53.524 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.718637+0000", 2026-02-23T13:09:53.524 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 0 2026-02-23T13:09:53.524 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.524 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.524 INFO:teuthology.orchestra.run.trial185.stdout: "event": "header_read", 2026-02-23T13:09:53.524 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.718637+0000", 2026-02-23T13:09:53.524 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 0 2026-02-23T13:09:53.524 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.524 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.524 INFO:teuthology.orchestra.run.trial185.stdout: "event": "throttled", 2026-02-23T13:09:53.524 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.718643+0000", 2026-02-23T13:09:53.525 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 6.2620000000000004e-06 2026-02-23T13:09:53.525 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.525 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.525 INFO:teuthology.orchestra.run.trial185.stdout: "event": "all_read", 2026-02-23T13:09:53.525 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.718665+0000", 2026-02-23T13:09:53.525 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 2.2252e-05 2026-02-23T13:09:53.525 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.525 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.525 INFO:teuthology.orchestra.run.trial185.stdout: "event": "dispatched", 2026-02-23T13:09:53.525 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.718668+0000", 2026-02-23T13:09:53.525 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 3.1149999999999998e-06 2026-02-23T13:09:53.525 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.525 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.525 INFO:teuthology.orchestra.run.trial185.stdout: "event": "queued_for_pg", 2026-02-23T13:09:53.525 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.718669+0000", 2026-02-23T13:09:53.525 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 1.032e-06 2026-02-23T13:09:53.525 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.525 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.525 INFO:teuthology.orchestra.run.trial185.stdout: "event": "reached_pg", 2026-02-23T13:09:53.525 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.718686+0000", 2026-02-23T13:09:53.525 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 1.6521e-05 2026-02-23T13:09:53.525 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.525 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.525 INFO:teuthology.orchestra.run.trial185.stdout: "event": "waiting for peered", 2026-02-23T13:09:53.525 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.718688+0000", 2026-02-23T13:09:53.525 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 2.0640000000000001e-06 2026-02-23T13:09:53.525 INFO:teuthology.orchestra.run.trial185.stdout: } 2026-02-23T13:09:53.525 INFO:teuthology.orchestra.run.trial185.stdout: ] 2026-02-23T13:09:53.525 INFO:teuthology.orchestra.run.trial185.stdout: } 2026-02-23T13:09:53.525 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.525 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.525 INFO:teuthology.orchestra.run.trial185.stdout: "description": "osd_op(client.14522.0:79276 3.c 3.8e4a5dec (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e80)", 2026-02-23T13:09:53.525 INFO:teuthology.orchestra.run.trial185.stdout: "initiated_at": "2026-02-23T13:09:52.718677+0000", 2026-02-23T13:09:53.525 INFO:teuthology.orchestra.run.trial185.stdout: "age": 0.68071974400000002, 2026-02-23T13:09:53.525 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 0.68151134099999999, 2026-02-23T13:09:53.525 INFO:teuthology.orchestra.run.trial185.stdout: "continuous": false, 2026-02-23T13:09:53.525 INFO:teuthology.orchestra.run.trial185.stdout: "type_data": { 2026-02-23T13:09:53.525 INFO:teuthology.orchestra.run.trial185.stdout: "flag_point": "waiting for peered", 2026-02-23T13:09:53.525 INFO:teuthology.orchestra.run.trial185.stdout: "client_info": { 2026-02-23T13:09:53.525 INFO:teuthology.orchestra.run.trial185.stdout: "client": "client.14522", 2026-02-23T13:09:53.525 INFO:teuthology.orchestra.run.trial185.stdout: "client_addr": "10.20.193.68:0/2327563370", 2026-02-23T13:09:53.525 INFO:teuthology.orchestra.run.trial185.stdout: "tid": 79276 2026-02-23T13:09:53.525 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.525 INFO:teuthology.orchestra.run.trial185.stdout: "events": [ 2026-02-23T13:09:53.526 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.526 INFO:teuthology.orchestra.run.trial185.stdout: "event": "initiated", 2026-02-23T13:09:53.526 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.718677+0000", 2026-02-23T13:09:53.526 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 0 2026-02-23T13:09:53.526 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.526 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.526 INFO:teuthology.orchestra.run.trial185.stdout: "event": "header_read", 2026-02-23T13:09:53.526 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.718677+0000", 2026-02-23T13:09:53.526 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 0 2026-02-23T13:09:53.526 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.526 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.526 INFO:teuthology.orchestra.run.trial185.stdout: "event": "throttled", 2026-02-23T13:09:53.526 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.718683+0000", 2026-02-23T13:09:53.526 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 6.1510000000000001e-06 2026-02-23T13:09:53.526 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.526 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.526 INFO:teuthology.orchestra.run.trial185.stdout: "event": "all_read", 2026-02-23T13:09:53.526 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.718711+0000", 2026-02-23T13:09:53.526 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 2.7591999999999999e-05 2026-02-23T13:09:53.526 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.526 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.526 INFO:teuthology.orchestra.run.trial185.stdout: "event": "dispatched", 2026-02-23T13:09:53.526 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.718714+0000", 2026-02-23T13:09:53.526 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 3.0149999999999999e-06 2026-02-23T13:09:53.526 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.526 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.526 INFO:teuthology.orchestra.run.trial185.stdout: "event": "queued_for_pg", 2026-02-23T13:09:53.526 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.718715+0000", 2026-02-23T13:09:53.526 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 1.032e-06 2026-02-23T13:09:53.526 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.526 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.526 INFO:teuthology.orchestra.run.trial185.stdout: "event": "reached_pg", 2026-02-23T13:09:53.526 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.718733+0000", 2026-02-23T13:09:53.526 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 1.8434999999999999e-05 2026-02-23T13:09:53.526 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.526 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.526 INFO:teuthology.orchestra.run.trial185.stdout: "event": "waiting for peered", 2026-02-23T13:09:53.526 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.718736+0000", 2026-02-23T13:09:53.526 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 2.6749999999999998e-06 2026-02-23T13:09:53.526 INFO:teuthology.orchestra.run.trial185.stdout: } 2026-02-23T13:09:53.526 INFO:teuthology.orchestra.run.trial185.stdout: ] 2026-02-23T13:09:53.526 INFO:teuthology.orchestra.run.trial185.stdout: } 2026-02-23T13:09:53.526 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.526 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.527 INFO:teuthology.orchestra.run.trial185.stdout: "description": "osd_op(client.14522.0:79277 3.c 3.8e4a5dec (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e80)", 2026-02-23T13:09:53.527 INFO:teuthology.orchestra.run.trial185.stdout: "initiated_at": "2026-02-23T13:09:52.718723+0000", 2026-02-23T13:09:53.527 INFO:teuthology.orchestra.run.trial185.stdout: "age": 0.68067381900000001, 2026-02-23T13:09:53.527 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 0.68148225699999998, 2026-02-23T13:09:53.527 INFO:teuthology.orchestra.run.trial185.stdout: "continuous": false, 2026-02-23T13:09:53.527 INFO:teuthology.orchestra.run.trial185.stdout: "type_data": { 2026-02-23T13:09:53.527 INFO:teuthology.orchestra.run.trial185.stdout: "flag_point": "waiting for peered", 2026-02-23T13:09:53.527 INFO:teuthology.orchestra.run.trial185.stdout: "client_info": { 2026-02-23T13:09:53.527 INFO:teuthology.orchestra.run.trial185.stdout: "client": "client.14522", 2026-02-23T13:09:53.527 INFO:teuthology.orchestra.run.trial185.stdout: "client_addr": "10.20.193.68:0/2327563370", 2026-02-23T13:09:53.527 INFO:teuthology.orchestra.run.trial185.stdout: "tid": 79277 2026-02-23T13:09:53.527 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.527 INFO:teuthology.orchestra.run.trial185.stdout: "events": [ 2026-02-23T13:09:53.527 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.527 INFO:teuthology.orchestra.run.trial185.stdout: "event": "initiated", 2026-02-23T13:09:53.527 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.718723+0000", 2026-02-23T13:09:53.527 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 0 2026-02-23T13:09:53.527 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.527 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.527 INFO:teuthology.orchestra.run.trial185.stdout: "event": "header_read", 2026-02-23T13:09:53.527 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.718723+0000", 2026-02-23T13:09:53.527 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 0 2026-02-23T13:09:53.527 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.527 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.527 INFO:teuthology.orchestra.run.trial185.stdout: "event": "throttled", 2026-02-23T13:09:53.527 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.718729+0000", 2026-02-23T13:09:53.527 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 6.2020000000000001e-06 2026-02-23T13:09:53.527 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.527 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.527 INFO:teuthology.orchestra.run.trial185.stdout: "event": "all_read", 2026-02-23T13:09:53.527 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.718753+0000", 2026-02-23T13:09:53.527 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 2.3594000000000001e-05 2026-02-23T13:09:53.527 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.527 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.527 INFO:teuthology.orchestra.run.trial185.stdout: "event": "dispatched", 2026-02-23T13:09:53.527 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.718756+0000", 2026-02-23T13:09:53.527 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 3.737e-06 2026-02-23T13:09:53.527 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.527 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.527 INFO:teuthology.orchestra.run.trial185.stdout: "event": "queued_for_pg", 2026-02-23T13:09:53.527 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.718758+0000", 2026-02-23T13:09:53.527 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 1.2720000000000001e-06 2026-02-23T13:09:53.527 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.527 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.528 INFO:teuthology.orchestra.run.trial185.stdout: "event": "reached_pg", 2026-02-23T13:09:53.528 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.718781+0000", 2026-02-23T13:09:53.528 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 2.3244e-05 2026-02-23T13:09:53.528 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:09:53.528 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:09:53.528 INFO:teuthology.orchestra.run.trial185.stdout: "event": "waiting for peered", 2026-02-23T13:09:53.528 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:09:52.718783+0000", 2026-02-23T13:09:53.528 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 2.1339999999999998e-06 2026-02-23T13:09:53.528 INFO:teuthology.orchestra.run.trial185.stdout: } 2026-02-23T13:09:53.528 INFO:teuthology.orchestra.run.trial185.stdout: ] 2026-02-23T13:09:53.528 INFO:teuthology.orchestra.run.trial185.stdout: } 2026-02-23T13:09:53.528 INFO:teuthology.orchestra.run.trial185.stdout: } 2026-02-23T13:09:53.528 INFO:teuthology.orchestra.run.trial185.stdout: ], 2026-02-23T13:09:53.528 INFO:teuthology.orchestra.run.trial185.stdout: "num_ops": 50 2026-02-23T13:09:53.528 INFO:teuthology.orchestra.run.trial185.stdout:} 2026-02-23T13:09:53.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:09:53 trial068 ceph-mon[38805]: pgmap v128: 17 pgs: 6 active+clean, 11 active+undersized+degraded; 519 MiB data, 2.5 GiB used, 2.7 TiB / 2.7 TiB avail; 63 MiB/s wr, 8.02k op/s; 4918/24897 objects degraded (19.753%) 2026-02-23T13:09:53.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:09:53 trial068 ceph-mon[38805]: Health check cleared: OSD_DOWN (was: 1 osds down) 2026-02-23T13:09:53.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:09:53 trial068 ceph-mon[38805]: Health check update: Degraded data redundancy: 4918/24897 objects degraded (19.753%), 11 pgs degraded (PG_DEGRADED) 2026-02-23T13:09:53.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:09:53 trial068 ceph-mon[38805]: osd.3 [v2:10.20.193.185:6808/3696769086,v1:10.20.193.185:6809/3696769086] boot 2026-02-23T13:09:53.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:09:53 trial068 ceph-mon[38805]: osdmap e80: 4 total, 4 up, 4 in 2026-02-23T13:09:53.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:09:53 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2026-02-23T13:09:54.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:09:53 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:09:53] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:09:54.357 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:09:53 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:09:53.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:09:54.400 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:09:54.458 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 10 128 79432 79304 61.9514 0 - 0.0133219 2026-02-23T13:09:54.519 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:09:54.750 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:09:54 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T13:09:54.750 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:09:54 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T13:09:54.750 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:09:54 trial068 ceph-mon[38805]: osdmap e81: 4 total, 4 up, 4 in 2026-02-23T13:09:54.750 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:09:54 trial068 ceph-mon[38805]: osdmap e82: 4 total, 4 up, 4 in 2026-02-23T13:09:55.007 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:09:55.007 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:09:55.008 INFO:teuthology.orchestra.run.trial068.stdout:osd.2: {} 2026-02-23T13:09:55.008 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:09:55.008 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:09:54.925+0000 7f229900d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:09:55.008 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:09:54.925+0000 7f229900d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:09:55.008 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:09:55.009 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:09:55.009 INFO:teuthology.orchestra.run.trial068.stderr:osd.2: osd_enable_op_tracker = '' 2026-02-23T13:09:55.009 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:09:55.458 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 11 128 81659 81531 57.9011 8.69922 0.0130999 0.0171971 2026-02-23T13:09:55.741 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:09:55 trial068 ceph-mon[38805]: pgmap v132: 17 pgs: 6 active+clean, 11 active+undersized+degraded; 519 MiB data, 2.5 GiB used, 2.7 TiB / 2.7 TiB avail; 95 MiB/s wr, 12.19k op/s; 4918/24897 objects degraded (19.753%) 2026-02-23T13:09:55.741 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:09:55 trial068 ceph-mon[38805]: 1.0 scrub starts 2026-02-23T13:09:55.741 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:09:55 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1589306493' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:09:56.458 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 12 128 87536 87408 56.9019 45.9141 0.026236 0.0175481 2026-02-23T13:09:56.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:09:56 trial068 ceph-mon[38805]: 1.0 scrub ok 2026-02-23T13:09:57.458 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 13 128 93237 93109 55.9506 44.5391 0.0401326 0.0178496 2026-02-23T13:09:57.772 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:09:57.797 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:09:57 trial068 ceph-mon[38805]: pgmap v133: 17 pgs: 3 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 6 active+clean, 4 active+undersized+degraded; 606 MiB data, 2.8 GiB used, 2.7 TiB / 2.7 TiB avail; 46 MiB/s wr, 5.83k op/s; 5632/29058 objects degraded (19.382%); 3.3 MiB/s, 52 objects/s recovering 2026-02-23T13:09:57.905 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:09:58.008 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:09:58.007+0000 7f3d0f68e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:09:58.008 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:09:58.007+0000 7f3d0f68e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:09:58.021 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:09:58.021 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:09:58.031 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:09:58.031 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:09:58.041 INFO:teuthology.orchestra.run.trial068.stdout:osd.2: {} 2026-02-23T13:09:58.041 INFO:teuthology.orchestra.run.trial068.stderr:osd.2: osd_enable_op_tracker = '' 2026-02-23T13:09:58.052 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:09:58.052 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:09:58.458 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 14 128 95000 94872 52.9379 13.7734 0.00264973 0.0177882 2026-02-23T13:09:58.529 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 2, 1, 3] 2026-02-23T13:09:58.529 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2026-02-23T13:09:58.530 INFO:tasks.thrashosds.thrasher:Killing osd 2, live_osds are [0, 2, 1, 3] 2026-02-23T13:09:58.530 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2026-02-23T13:09:58.530 DEBUG:teuthology.orchestra.run.trial185:> sudo systemctl stop ceph-5f823084-10b8-11f1-be80-d404e6e7d460@osd.2 2026-02-23T13:09:58.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:09:58 trial068 ceph-mon[38805]: osdmap e83: 4 total, 4 up, 4 in 2026-02-23T13:09:58.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:09:58 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/307237360' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:09:58.856 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:09:58 trial185 systemd[1]: Stopping Ceph osd.2 for 5f823084-10b8-11f1-be80-d404e6e7d460... 2026-02-23T13:09:58.857 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:09:58 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-2[45891]: 2026-02-23T13:09:58.784+0000 7fd5b9861640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --osd-objectstore=bluestore (PID: 1) UID: 0 2026-02-23T13:09:58.857 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:09:58 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-2[45891]: 2026-02-23T13:09:58.784+0000 7fd5b9861640 -1 osd.2 84 *** Got signal Terminated *** 2026-02-23T13:09:58.857 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:09:58 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-2[45891]: 2026-02-23T13:09:58.784+0000 7fd5b9861640 -1 osd.2 84 *** Immediate shutdown (osd_fast_shutdown=true) *** 2026-02-23T13:09:59.458 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 15 128 95864 95736 49.8587 6.75 0.00490456 0.0194012 2026-02-23T13:09:59.482 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:09:59 trial185 podman[70141]: 2026-02-23 13:09:59.150189991 +0000 UTC m=+0.518498401 container died fec9fcc326bc0083c42dd5cd417c7074c110ceebbb983158b517d7483da7fab3 (image=quay.ceph.io/ceph-ci/ceph@sha256:49621faaa845c39b2aae72e4fa24123ee9b2ac3fb8ae834753907068ee2972ea, name=ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-2, org.opencontainers.image.authors=Ceph Release Team , org.opencontainers.image.documentation=https://docs.ceph.com/, CEPH_SHA1=e371433fcd1390a0b82cfb18effc422c1f5d8c1d, CEPH_GIT_REPO=https://github.com/ceph/ceph-ci.git, CEPH_REF=wip-shraddhaag-enable-cephadm-tests, ceph=True, org.label-schema.build-date=20260216, FROM_IMAGE=quay.io/centos/centos:stream9, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, OSD_FLAVOR=default, GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfsganesha-5/, io.buildah.version=1.41.3, org.label-schema.license=GPLv2) 2026-02-23T13:09:59.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:09:59 trial068 ceph-mon[38805]: pgmap v135: 17 pgs: 4 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 7 active+clean; 668 MiB data, 3.0 GiB used, 2.7 TiB / 2.7 TiB avail; 26 MiB/s wr, 3.28k op/s; 5693/32037 objects degraded (17.770%); 4.5 MiB/s, 72 objects/s recovering 2026-02-23T13:09:59.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:09:59 trial068 ceph-mon[38805]: Health check update: Degraded data redundancy: 5693/32037 objects degraded (17.770%), 10 pgs degraded (PG_DEGRADED) 2026-02-23T13:09:59.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:09:59 trial068 ceph-mon[38805]: osdmap e84: 4 total, 4 up, 4 in 2026-02-23T13:09:59.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:09:59 trial068 ceph-mon[38805]: osd.2 marked itself down and dead 2026-02-23T13:09:59.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:09:59 trial068 ceph-mon[38805]: Health check failed: 1 osds down (OSD_DOWN) 2026-02-23T13:09:59.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:09:59 trial068 ceph-mon[38805]: osdmap e85: 4 total, 3 up, 4 in 2026-02-23T13:10:00.077 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:09:59 trial185 podman[70141]: 2026-02-23 13:09:59.772729606 +0000 UTC m=+1.141038026 container remove fec9fcc326bc0083c42dd5cd417c7074c110ceebbb983158b517d7483da7fab3 (image=quay.ceph.io/ceph-ci/ceph@sha256:49621faaa845c39b2aae72e4fa24123ee9b2ac3fb8ae834753907068ee2972ea, name=ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-2, CEPH_GIT_REPO=https://github.com/ceph/ceph-ci.git, ceph=True, io.buildah.version=1.41.3, CEPH_REF=wip-shraddhaag-enable-cephadm-tests, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_SHA1=e371433fcd1390a0b82cfb18effc422c1f5d8c1d, org.opencontainers.image.documentation=https://docs.ceph.com/, OSD_FLAVOR=default, org.label-schema.schema-version=1.0, org.opencontainers.image.authors=Ceph Release Team , org.label-schema.build-date=20260216, FROM_IMAGE=quay.io/centos/centos:stream9, org.label-schema.license=GPLv2, GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfsganesha-5/, org.label-schema.vendor=CentOS) 2026-02-23T13:10:00.077 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:09:59 trial185 bash[70141]: ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-2 2026-02-23T13:10:00.077 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:09:59 trial185 podman[70265]: 2026-02-23 13:09:59.895345914 +0000 UTC m=+0.016025848 container create ed6ac7af63fa8ecb889d40acc37b53b3fa377f228a428d6fab6755deab2fa580 (image=quay.ceph.io/ceph-ci/ceph@sha256:49621faaa845c39b2aae72e4fa24123ee9b2ac3fb8ae834753907068ee2972ea, name=ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-2-deactivate, io.buildah.version=1.41.3, FROM_IMAGE=quay.io/centos/centos:stream9, CEPH_REF=wip-shraddhaag-enable-cephadm-tests, GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfsganesha-5/, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, ceph=True, org.opencontainers.image.authors=Ceph Release Team , CEPH_GIT_REPO=https://github.com/ceph/ceph-ci.git, CEPH_SHA1=e371433fcd1390a0b82cfb18effc422c1f5d8c1d, org.label-schema.vendor=CentOS, OSD_FLAVOR=default, org.opencontainers.image.documentation=https://docs.ceph.com/, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20260216) 2026-02-23T13:10:00.077 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:09:59 trial185 podman[70265]: 2026-02-23 13:09:59.925711323 +0000 UTC m=+0.046391196 container init ed6ac7af63fa8ecb889d40acc37b53b3fa377f228a428d6fab6755deab2fa580 (image=quay.ceph.io/ceph-ci/ceph@sha256:49621faaa845c39b2aae72e4fa24123ee9b2ac3fb8ae834753907068ee2972ea, name=ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-2-deactivate, CEPH_SHA1=e371433fcd1390a0b82cfb18effc422c1f5d8c1d, CEPH_REF=wip-shraddhaag-enable-cephadm-tests, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfsganesha-5/, io.buildah.version=1.41.3, OSD_FLAVOR=default, ceph=True, org.opencontainers.image.authors=Ceph Release Team , org.opencontainers.image.documentation=https://docs.ceph.com/, org.label-schema.build-date=20260216, FROM_IMAGE=quay.io/centos/centos:stream9, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, CEPH_GIT_REPO=https://github.com/ceph/ceph-ci.git) 2026-02-23T13:10:00.078 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:09:59 trial185 podman[70265]: 2026-02-23 13:09:59.928180439 +0000 UTC m=+0.048860322 container start ed6ac7af63fa8ecb889d40acc37b53b3fa377f228a428d6fab6755deab2fa580 (image=quay.ceph.io/ceph-ci/ceph@sha256:49621faaa845c39b2aae72e4fa24123ee9b2ac3fb8ae834753907068ee2972ea, name=ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-2-deactivate, OSD_FLAVOR=default, FROM_IMAGE=quay.io/centos/centos:stream9, ceph=True, CEPH_SHA1=e371433fcd1390a0b82cfb18effc422c1f5d8c1d, org.label-schema.schema-version=1.0, org.opencontainers.image.authors=Ceph Release Team , CEPH_REF=wip-shraddhaag-enable-cephadm-tests, CEPH_GIT_REPO=https://github.com/ceph/ceph-ci.git, org.opencontainers.image.documentation=https://docs.ceph.com/, org.label-schema.name=CentOS Stream 9 Base Image, GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfsganesha-5/, io.buildah.version=1.41.3, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.build-date=20260216) 2026-02-23T13:10:00.078 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:09:59 trial185 podman[70265]: 2026-02-23 13:09:59.92867367 +0000 UTC m=+0.049353553 container attach ed6ac7af63fa8ecb889d40acc37b53b3fa377f228a428d6fab6755deab2fa580 (image=quay.ceph.io/ceph-ci/ceph@sha256:49621faaa845c39b2aae72e4fa24123ee9b2ac3fb8ae834753907068ee2972ea, name=ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-2-deactivate, CEPH_GIT_REPO=https://github.com/ceph/ceph-ci.git, CEPH_SHA1=e371433fcd1390a0b82cfb18effc422c1f5d8c1d, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.opencontainers.image.documentation=https://docs.ceph.com/, io.buildah.version=1.41.3, OSD_FLAVOR=default, org.opencontainers.image.authors=Ceph Release Team , CEPH_REF=wip-shraddhaag-enable-cephadm-tests, org.label-schema.build-date=20260216, GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfsganesha-5/, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, ceph=True, FROM_IMAGE=quay.io/centos/centos:stream9) 2026-02-23T13:10:00.078 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:09:59 trial185 podman[70265]: 2026-02-23 13:09:59.888801626 +0000 UTC m=+0.009481509 image pull 63df4b83db6f95a8ba965919b3671cb27173117c23b5c950aaadc8913f56213c quay.ceph.io/ceph-ci/ceph@sha256:49621faaa845c39b2aae72e4fa24123ee9b2ac3fb8ae834753907068ee2972ea 2026-02-23T13:10:00.078 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:10:00 trial185 podman[70265]: 2026-02-23 13:10:00.075519299 +0000 UTC m=+0.196199172 container died ed6ac7af63fa8ecb889d40acc37b53b3fa377f228a428d6fab6755deab2fa580 (image=quay.ceph.io/ceph-ci/ceph@sha256:49621faaa845c39b2aae72e4fa24123ee9b2ac3fb8ae834753907068ee2972ea, name=ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-2-deactivate, org.opencontainers.image.authors=Ceph Release Team , org.opencontainers.image.documentation=https://docs.ceph.com/, org.label-schema.build-date=20260216, org.label-schema.schema-version=1.0, CEPH_GIT_REPO=https://github.com/ceph/ceph-ci.git, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfsganesha-5/, ceph=True, FROM_IMAGE=quay.io/centos/centos:stream9, io.buildah.version=1.41.3, OSD_FLAVOR=default, CEPH_REF=wip-shraddhaag-enable-cephadm-tests, CEPH_SHA1=e371433fcd1390a0b82cfb18effc422c1f5d8c1d) 2026-02-23T13:10:00.087 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:10:00.122 DEBUG:teuthology.orchestra.run.trial185:> sudo pkill -f 'journalctl -f -n 0 -u ceph-5f823084-10b8-11f1-be80-d404e6e7d460@osd.2.service' 2026-02-23T13:10:00.163 DEBUG:teuthology.orchestra.run:got remote process result: None 2026-02-23T13:10:00.163 INFO:tasks.cephadm.osd.2:Stopped osd.2 2026-02-23T13:10:00.204 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:10:00.308 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:10:00.306+0000 7efe26521640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:10:00.308 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:10:00.306+0000 7efe26521640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:10:00.320 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:10:00.320 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:10:00.330 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:10:00.331 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:10:00.331 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:10:00.341 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:10:00.342 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:10:00.369 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:10:00.369 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:10:00.459 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 16 128 96168 96040 46.891 2.375 0.0039733 0.0204009 2026-02-23T13:10:01.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:10:00 trial068 ceph-mon[38805]: osdmap e86: 4 total, 3 up, 4 in 2026-02-23T13:10:01.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:10:00 trial068 ceph-mon[38805]: pgmap v139: 17 pgs: 3 stale+active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 1 stale+active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 7 active+clean; 668 MiB data, 3.0 GiB used, 2.7 TiB / 2.7 TiB avail; 16 MiB/s wr, 1.99k op/s; 5693/32037 objects degraded (17.770%); 2.6 MiB/s, 40 objects/s recovering 2026-02-23T13:10:01.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:10:00 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/676887409' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:10:01.459 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 17 128 98830 98702 45.3559 20.7969 0.0613816 0.0220229 2026-02-23T13:10:02.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:10:01 trial068 ceph-mon[38805]: osdmap e87: 4 total, 3 up, 4 in 2026-02-23T13:10:02.370 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:10:02.459 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 18 127 104690 104563 45.3798 45.7891 0.00717223 0.0220089 2026-02-23T13:10:02.539 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:10:02.670 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:10:02.668+0000 7f7e446d9640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:10:02.671 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:10:02.670+0000 7f7e446d9640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:10:02.691 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:10:02.691 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:10:02.709 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:10:02.710 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:10:02.710 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:10:02.726 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:10:02.726 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:10:02.771 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:10:02.771 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:10:03.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:10:02 trial068 ceph-mon[38805]: pgmap v141: 17 pgs: 1 active+recovering+undersized+degraded, 2 peering, 1 active+recovering+undersized+degraded+remapped, 11 active+undersized+degraded, 2 active+clean; 780 MiB data, 3.5 GiB used, 2.7 TiB / 2.7 TiB avail; 28 MiB/s wr, 3.59k op/s; 10537/37428 objects degraded (28.153%); 53 MiB/s, 847 objects/s recovering 2026-02-23T13:10:03.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:10:02 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/456908171' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:10:03.459 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 19 128 110227 110099 45.2675 43.25 0.0226617 0.0220689 2026-02-23T13:10:04.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:10:03 trial068 ceph-mon[38805]: Health check update: Degraded data redundancy: 10537/37428 objects degraded (28.153%), 13 pgs degraded (PG_DEGRADED) 2026-02-23T13:10:04.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:10:03 trial068 ceph-mon[38805]: osdmap e88: 4 total, 3 up, 4 in 2026-02-23T13:10:04.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:10:03 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:10:03] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:10:04.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:10:03 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:10:03.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:10:04.459 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T13:10:04.458996+0000 min lat: 0.000737354 max lat: 2.31352 avg lat: 0.0221893 2026-02-23T13:10:04.459 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T13:10:04.459 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 20 128 115409 115281 45.0282 40.4844 0.017534 0.0221893 2026-02-23T13:10:04.772 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:10:04.927 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:10:05.067 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:10:04 trial068 ceph-mon[38805]: pgmap v143: 17 pgs: 1 active+recovering+undersized+degraded, 2 peering, 1 active+recovering+undersized+degraded+remapped, 11 active+undersized+degraded, 2 active+clean; 780 MiB data, 3.5 GiB used, 2.7 TiB / 2.7 TiB avail; 21 MiB/s wr, 2.73k op/s; 10537/37428 objects degraded (28.153%); 40 MiB/s, 642 objects/s recovering 2026-02-23T13:10:05.067 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:10:05.065+0000 7f1fe47ff640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:10:05.067 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:10:05.065+0000 7f1fe47ff640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:10:05.086 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:10:05.087 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:10:05.102 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:10:05.102 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:10:05.103 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:10:05.118 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:10:05.118 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:10:05.163 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:10:05.164 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:10:05.164 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [2] live_osds: [0, 1, 3] 2026-02-23T13:10:05.164 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2026-02-23T13:10:05.164 INFO:tasks.thrashosds.thrasher:Reviving osd 2 2026-02-23T13:10:05.164 INFO:tasks.cephadm.osd.2:Restarting osd.2 (starting--it wasn't running)... 2026-02-23T13:10:05.164 DEBUG:teuthology.orchestra.run.trial185:osd.2> sudo journalctl -f -n 0 -u ceph-5f823084-10b8-11f1-be80-d404e6e7d460@osd.2.service 2026-02-23T13:10:05.167 DEBUG:teuthology.orchestra.run.trial185:> sudo systemctl start ceph-5f823084-10b8-11f1-be80-d404e6e7d460@osd.2 2026-02-23T13:10:05.440 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:10:05 trial185 systemd[1]: Starting Ceph osd.2 for 5f823084-10b8-11f1-be80-d404e6e7d460... 2026-02-23T13:10:05.440 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:10:05 trial185 podman[70451]: 2026-02-23 13:10:05.346798254 +0000 UTC m=+0.016440041 container create 38cb925395540a63c9b2f9b4645a30bde00b536d4557d279c279dc95e0dd70dd (image=quay.ceph.io/ceph-ci/ceph@sha256:49621faaa845c39b2aae72e4fa24123ee9b2ac3fb8ae834753907068ee2972ea, name=ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-2-activate, org.label-schema.schema-version=1.0, CEPH_GIT_REPO=https://github.com/ceph/ceph-ci.git, CEPH_SHA1=e371433fcd1390a0b82cfb18effc422c1f5d8c1d, ceph=True, GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfsganesha-5/, org.label-schema.vendor=CentOS, CEPH_REF=wip-shraddhaag-enable-cephadm-tests, org.opencontainers.image.authors=Ceph Release Team , org.label-schema.build-date=20260216, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, org.opencontainers.image.documentation=https://docs.ceph.com/, io.buildah.version=1.41.3, OSD_FLAVOR=default, FROM_IMAGE=quay.io/centos/centos:stream9) 2026-02-23T13:10:05.440 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:10:05 trial185 podman[70451]: 2026-02-23 13:10:05.390241804 +0000 UTC m=+0.059883591 container init 38cb925395540a63c9b2f9b4645a30bde00b536d4557d279c279dc95e0dd70dd (image=quay.ceph.io/ceph-ci/ceph@sha256:49621faaa845c39b2aae72e4fa24123ee9b2ac3fb8ae834753907068ee2972ea, name=ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-2-activate, CEPH_SHA1=e371433fcd1390a0b82cfb18effc422c1f5d8c1d, CEPH_REF=wip-shraddhaag-enable-cephadm-tests, org.label-schema.license=GPLv2, GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfsganesha-5/, FROM_IMAGE=quay.io/centos/centos:stream9, org.label-schema.schema-version=1.0, io.buildah.version=1.41.3, org.label-schema.build-date=20260216, OSD_FLAVOR=default, org.opencontainers.image.authors=Ceph Release Team , org.label-schema.vendor=CentOS, CEPH_GIT_REPO=https://github.com/ceph/ceph-ci.git, org.label-schema.name=CentOS Stream 9 Base Image, org.opencontainers.image.documentation=https://docs.ceph.com/, ceph=True) 2026-02-23T13:10:05.440 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:10:05 trial185 podman[70451]: 2026-02-23 13:10:05.392987057 +0000 UTC m=+0.062628844 container start 38cb925395540a63c9b2f9b4645a30bde00b536d4557d279c279dc95e0dd70dd (image=quay.ceph.io/ceph-ci/ceph@sha256:49621faaa845c39b2aae72e4fa24123ee9b2ac3fb8ae834753907068ee2972ea, name=ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-2-activate, org.label-schema.schema-version=1.0, CEPH_GIT_REPO=https://github.com/ceph/ceph-ci.git, org.label-schema.build-date=20260216, OSD_FLAVOR=default, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, FROM_IMAGE=quay.io/centos/centos:stream9, org.label-schema.license=GPLv2, org.opencontainers.image.authors=Ceph Release Team , org.opencontainers.image.documentation=https://docs.ceph.com/, CEPH_SHA1=e371433fcd1390a0b82cfb18effc422c1f5d8c1d, io.buildah.version=1.41.3, CEPH_REF=wip-shraddhaag-enable-cephadm-tests, GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfsganesha-5/, org.label-schema.vendor=CentOS) 2026-02-23T13:10:05.440 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:10:05 trial185 podman[70451]: 2026-02-23 13:10:05.39342727 +0000 UTC m=+0.063069067 container attach 38cb925395540a63c9b2f9b4645a30bde00b536d4557d279c279dc95e0dd70dd (image=quay.ceph.io/ceph-ci/ceph@sha256:49621faaa845c39b2aae72e4fa24123ee9b2ac3fb8ae834753907068ee2972ea, name=ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-2-activate, io.buildah.version=1.41.3, CEPH_REF=wip-shraddhaag-enable-cephadm-tests, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, CEPH_GIT_REPO=https://github.com/ceph/ceph-ci.git, org.label-schema.license=GPLv2, org.opencontainers.image.authors=Ceph Release Team , FROM_IMAGE=quay.io/centos/centos:stream9, org.label-schema.schema-version=1.0, CEPH_SHA1=e371433fcd1390a0b82cfb18effc422c1f5d8c1d, ceph=True, GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfsganesha-5/, org.label-schema.build-date=20260216, OSD_FLAVOR=default, org.opencontainers.image.documentation=https://docs.ceph.com/) 2026-02-23T13:10:05.440 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:10:05 trial185 podman[70451]: 2026-02-23 13:10:05.340158719 +0000 UTC m=+0.009800516 image pull 63df4b83db6f95a8ba965919b3671cb27173117c23b5c950aaadc8913f56213c quay.ceph.io/ceph-ci/ceph@sha256:49621faaa845c39b2aae72e4fa24123ee9b2ac3fb8ae834753907068ee2972ea 2026-02-23T13:10:05.459 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 21 128 120570 120442 44.8038 40.3203 0.0469912 0.0222986 2026-02-23T13:10:05.843 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:10:05 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-2-activate[70474]: Running command: /usr/bin/ceph-authtool --gen-print-key 2026-02-23T13:10:05.843 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:10:05 trial185 bash[70451]: Running command: /usr/bin/ceph-authtool --gen-print-key 2026-02-23T13:10:05.843 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:10:05 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-2-activate[70474]: Running command: /usr/bin/ceph-authtool --gen-print-key 2026-02-23T13:10:05.843 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:10:05 trial185 bash[70451]: Running command: /usr/bin/ceph-authtool --gen-print-key 2026-02-23T13:10:05.843 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:10:05 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-2-activate[70474]: --> Failed to activate via raw: did not find any matching OSD to activate 2026-02-23T13:10:05.843 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:10:05 trial185 bash[70451]: --> Failed to activate via raw: did not find any matching OSD to activate 2026-02-23T13:10:06.100 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:10:05 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-2-activate[70474]: Running command: /usr/bin/ceph-authtool --gen-print-key 2026-02-23T13:10:06.100 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:10:05 trial185 bash[70451]: Running command: /usr/bin/ceph-authtool --gen-print-key 2026-02-23T13:10:06.100 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:10:05 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-2-activate[70474]: Running command: /usr/bin/ceph-authtool --gen-print-key 2026-02-23T13:10:06.100 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:10:05 trial185 bash[70451]: Running command: /usr/bin/ceph-authtool --gen-print-key 2026-02-23T13:10:06.100 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:10:05 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-2-activate[70474]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2026-02-23T13:10:06.100 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:10:05 trial185 bash[70451]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2026-02-23T13:10:06.100 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:10:05 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-2-activate[70474]: Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/vg_nvme/lv_4 --path /var/lib/ceph/osd/ceph-2 --no-mon-config 2026-02-23T13:10:06.100 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:10:05 trial185 bash[70451]: Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/vg_nvme/lv_4 --path /var/lib/ceph/osd/ceph-2 --no-mon-config 2026-02-23T13:10:06.100 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:10:05 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-2-activate[70474]: Running command: /usr/bin/ln -snf /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-2/block 2026-02-23T13:10:06.100 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:10:05 trial185 bash[70451]: Running command: /usr/bin/ln -snf /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-2/block 2026-02-23T13:10:06.100 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:10:05 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-2-activate[70474]: Running command: /usr/bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-2/block 2026-02-23T13:10:06.100 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:10:05 trial185 bash[70451]: Running command: /usr/bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-2/block 2026-02-23T13:10:06.100 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:10:05 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-2-activate[70474]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2026-02-23T13:10:06.100 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:10:05 trial185 bash[70451]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2026-02-23T13:10:06.100 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:10:05 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-2-activate[70474]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2026-02-23T13:10:06.100 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:10:05 trial185 bash[70451]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2026-02-23T13:10:06.100 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:10:05 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-2-activate[70474]: --> ceph-volume lvm activate successful for osd ID: 2 2026-02-23T13:10:06.100 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:10:05 trial185 bash[70451]: --> ceph-volume lvm activate successful for osd ID: 2 2026-02-23T13:10:06.100 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:10:05 trial185 podman[70451]: 2026-02-23 13:10:05.992000581 +0000 UTC m=+0.661642378 container died 38cb925395540a63c9b2f9b4645a30bde00b536d4557d279c279dc95e0dd70dd (image=quay.ceph.io/ceph-ci/ceph@sha256:49621faaa845c39b2aae72e4fa24123ee9b2ac3fb8ae834753907068ee2972ea, name=ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-2-activate, GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfsganesha-5/, org.opencontainers.image.authors=Ceph Release Team , org.label-schema.vendor=CentOS, org.opencontainers.image.documentation=https://docs.ceph.com/, org.label-schema.schema-version=1.0, CEPH_GIT_REPO=https://github.com/ceph/ceph-ci.git, io.buildah.version=1.41.3, FROM_IMAGE=quay.io/centos/centos:stream9, CEPH_SHA1=e371433fcd1390a0b82cfb18effc422c1f5d8c1d, OSD_FLAVOR=default, org.label-schema.license=GPLv2, org.label-schema.build-date=20260216, CEPH_REF=wip-shraddhaag-enable-cephadm-tests, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image) 2026-02-23T13:10:06.100 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:10:06 trial185 podman[70451]: 2026-02-23 13:10:06.003005137 +0000 UTC m=+0.672646924 container remove 38cb925395540a63c9b2f9b4645a30bde00b536d4557d279c279dc95e0dd70dd (image=quay.ceph.io/ceph-ci/ceph@sha256:49621faaa845c39b2aae72e4fa24123ee9b2ac3fb8ae834753907068ee2972ea, name=ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-2-activate, org.label-schema.license=GPLv2, FROM_IMAGE=quay.io/centos/centos:stream9, OSD_FLAVOR=default, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_GIT_REPO=https://github.com/ceph/ceph-ci.git, CEPH_SHA1=e371433fcd1390a0b82cfb18effc422c1f5d8c1d, org.label-schema.vendor=CentOS, org.opencontainers.image.documentation=https://docs.ceph.com/, org.opencontainers.image.authors=Ceph Release Team , io.buildah.version=1.41.3, CEPH_REF=wip-shraddhaag-enable-cephadm-tests, GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfsganesha-5/, org.label-schema.schema-version=1.0, org.label-schema.build-date=20260216, ceph=True) 2026-02-23T13:10:06.100 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:10:06 trial185 podman[70686]: 2026-02-23 13:10:06.098315518 +0000 UTC m=+0.016867129 container create f197201319a0b8a31247924fd4ff0c9aef5d95a0d3ce4819befedd322cdb3899 (image=quay.ceph.io/ceph-ci/ceph@sha256:49621faaa845c39b2aae72e4fa24123ee9b2ac3fb8ae834753907068ee2972ea, name=ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-2, org.label-schema.schema-version=1.0, org.opencontainers.image.documentation=https://docs.ceph.com/, GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfsganesha-5/, org.label-schema.build-date=20260216, OSD_FLAVOR=default, ceph=True, org.opencontainers.image.authors=Ceph Release Team , org.label-schema.name=CentOS Stream 9 Base Image, CEPH_GIT_REPO=https://github.com/ceph/ceph-ci.git, io.buildah.version=1.41.3, CEPH_REF=wip-shraddhaag-enable-cephadm-tests, CEPH_SHA1=e371433fcd1390a0b82cfb18effc422c1f5d8c1d, org.label-schema.license=GPLv2, FROM_IMAGE=quay.io/centos/centos:stream9, org.label-schema.vendor=CentOS) 2026-02-23T13:10:06.134 DEBUG:teuthology.orchestra.run.trial185:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph daemon osd.2 dump_ops_in_flight 2026-02-23T13:10:06.260 INFO:teuthology.orchestra.run.trial185.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/config/ceph.conf 2026-02-23T13:10:06.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:10:05 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/4167543660' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:10:06.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:10:05 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T13:10:06.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:10:05 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T13:10:06.357 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:10:06 trial185 podman[70686]: 2026-02-23 13:10:06.124857146 +0000 UTC m=+0.043408767 container init f197201319a0b8a31247924fd4ff0c9aef5d95a0d3ce4819befedd322cdb3899 (image=quay.ceph.io/ceph-ci/ceph@sha256:49621faaa845c39b2aae72e4fa24123ee9b2ac3fb8ae834753907068ee2972ea, name=ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-2, FROM_IMAGE=quay.io/centos/centos:stream9, ceph=True, org.label-schema.vendor=CentOS, org.opencontainers.image.authors=Ceph Release Team , org.label-schema.name=CentOS Stream 9 Base Image, org.opencontainers.image.documentation=https://docs.ceph.com/, GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfsganesha-5/, io.buildah.version=1.41.3, CEPH_SHA1=e371433fcd1390a0b82cfb18effc422c1f5d8c1d, CEPH_GIT_REPO=https://github.com/ceph/ceph-ci.git, org.label-schema.build-date=20260216, OSD_FLAVOR=default, org.label-schema.schema-version=1.0, CEPH_REF=wip-shraddhaag-enable-cephadm-tests, org.label-schema.license=GPLv2) 2026-02-23T13:10:06.357 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:10:06 trial185 podman[70686]: 2026-02-23 13:10:06.127607538 +0000 UTC m=+0.046159140 container start f197201319a0b8a31247924fd4ff0c9aef5d95a0d3ce4819befedd322cdb3899 (image=quay.ceph.io/ceph-ci/ceph@sha256:49621faaa845c39b2aae72e4fa24123ee9b2ac3fb8ae834753907068ee2972ea, name=ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-2, CEPH_GIT_REPO=https://github.com/ceph/ceph-ci.git, CEPH_REF=wip-shraddhaag-enable-cephadm-tests, org.label-schema.build-date=20260216, org.label-schema.license=GPLv2, FROM_IMAGE=quay.io/centos/centos:stream9, org.opencontainers.image.authors=Ceph Release Team , GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfsganesha-5/, CEPH_SHA1=e371433fcd1390a0b82cfb18effc422c1f5d8c1d, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, org.opencontainers.image.documentation=https://docs.ceph.com/, io.buildah.version=1.41.3, OSD_FLAVOR=default) 2026-02-23T13:10:06.358 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:10:06 trial185 bash[70686]: f197201319a0b8a31247924fd4ff0c9aef5d95a0d3ce4819befedd322cdb3899 2026-02-23T13:10:06.358 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:10:06 trial185 podman[70686]: 2026-02-23 13:10:06.092048681 +0000 UTC m=+0.010600292 image pull 63df4b83db6f95a8ba965919b3671cb27173117c23b5c950aaadc8913f56213c quay.ceph.io/ceph-ci/ceph@sha256:49621faaa845c39b2aae72e4fa24123ee9b2ac3fb8ae834753907068ee2972ea 2026-02-23T13:10:06.358 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:10:06 trial185 systemd[1]: Started Ceph osd.2 for 5f823084-10b8-11f1-be80-d404e6e7d460. 2026-02-23T13:10:06.358 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:10:06 trial185 ceph-osd[70713]: -- 10.20.193.185:0/1351563358 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x290bc00 msgr2=0x3748000 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).process 2026-02-23T13:10:06.358 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:10:06 trial185 ceph-osd[70713]: -- 10.20.193.185:0/1351563358 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x290bc00 msgr2=0x3748000 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read continue len=32 2026-02-23T13:10:06.358 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:10:06 trial185 ceph-osd[70713]: --2- 10.20.193.185:0/1351563358 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x290bc00 0x3748000 crc :-1 s=READY pgs=451 gs=1 cs=0 l=1 c_cookie=15490193144510272148 s_cookie=0 reconnecting=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main r=0 2026-02-23T13:10:06.358 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:10:06 trial185 ceph-osd[70713]: --2- 10.20.193.185:0/1351563358 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x290bc00 0x3748000 crc :-1 s=THROTTLE_MESSAGE pgs=451 gs=1 cs=0 l=1 c_cookie=15490193144510272148 s_cookie=0 reconnecting=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).throttle_message 2026-02-23T13:10:06.358 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:10:06 trial185 ceph-osd[70713]: --2- 10.20.193.185:0/1351563358 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x290bc00 0x3748000 crc :-1 s=THROTTLE_BYTES pgs=451 gs=1 cs=0 l=1 c_cookie=15490193144510272148 s_cookie=0 reconnecting=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).throttle_bytes 2026-02-23T13:10:06.358 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:10:06 trial185 ceph-osd[70713]: --2- 10.20.193.185:0/1351563358 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x290bc00 0x3748000 crc :-1 s=THROTTLE_DISPATCH_QUEUE pgs=451 gs=1 cs=0 l=1 c_cookie=15490193144510272148 s_cookie=0 reconnecting=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).throttle_dispatch_queue 2026-02-23T13:10:06.358 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:10:06 trial185 ceph-osd[70713]: --2- 10.20.193.185:0/1351563358 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x290bc00 0x3748000 crc :-1 s=THROTTLE_DONE pgs=451 gs=1 cs=0 l=1 c_cookie=15490193144510272148 s_cookie=0 reconnecting=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).read_frame_segment seg_idx=0 2026-02-23T13:10:06.358 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:10:06 trial185 ceph-osd[70713]: -- 10.20.193.185:0/1351563358 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x290bc00 msgr2=0x3748000 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read start len=45 2026-02-23T13:10:06.359 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:10:06 trial185 ceph-osd[70713]: --2- 10.20.193.185:0/1351563358 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x290bc00 0x3748000 crc :-1 s=THROTTLE_DONE pgs=451 gs=1 cs=0 l=1 c_cookie=15490193144510272148 s_cookie=0 reconnecting=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_segment r=0 2026-02-23T13:10:06.359 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:10:06 trial185 ceph-osd[70713]: --2- 10.20.193.185:0/1351563358 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x290bc00 0x3748000 crc :-1 s=THROTTLE_DONE pgs=451 gs=1 cs=0 l=1 c_cookie=15490193144510272148 s_cookie=0 reconnecting=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).read_frame_segment seg_idx=1 2026-02-23T13:10:06.359 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:10:06 trial185 ceph-osd[70713]: -- 10.20.193.185:0/1351563358 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x290bc00 msgr2=0x3748000 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read start len=194 2026-02-23T13:10:06.359 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:10:06 trial185 ceph-osd[70713]: --2- 10.20.193.185:0/1351563358 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x290bc00 0x3748000 crc :-1 s=THROTTLE_DONE pgs=451 gs=1 cs=0 l=1 c_cookie=15490193144510272148 s_cookie=0 reconnecting=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_segment r=0 2026-02-23T13:10:06.359 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:10:06 trial185 ceph-osd[70713]: -- 10.20.193.185:0/1351563358 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x290bc00 msgr2=0x3748000 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read start len=13 2026-02-23T13:10:06.359 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:10:06 trial185 ceph-osd[70713]: --2- 10.20.193.185:0/1351563358 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x290bc00 0x3748000 crc :-1 s=THROTTLE_DONE pgs=451 gs=1 cs=0 l=1 c_cookie=15490193144510272148 s_cookie=0 reconnecting=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_epilogue_main r=0 2026-02-23T13:10:06.359 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:10:06 trial185 ceph-osd[70713]: --2- 10.20.193.185:0/1351563358 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x290bc00 0x3748000 crc :-1 s=THROTTLE_DONE pgs=451 gs=1 cs=0 l=1 c_cookie=15490193144510272148 s_cookie=0 reconnecting=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_dispatch tag=17 2026-02-23T13:10:06.359 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:10:06 trial185 ceph-osd[70713]: --2- 10.20.193.185:0/1351563358 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x290bc00 0x3748000 crc :-1 s=THROTTLE_DONE pgs=451 gs=1 cs=0 l=1 c_cookie=15490193144510272148 s_cookie=0 reconnecting=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_message 2026-02-23T13:10:06.359 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:10:06 trial185 ceph-osd[70713]: --2- 10.20.193.185:0/1351563358 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x290bc00 0x3748000 crc :-1 s=THROTTLE_DONE pgs=451 gs=1 cs=0 l=1 c_cookie=15490193144510272148 s_cookie=0 reconnecting=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_message got 194 + 0 + 0 byte message. envelope type=18 src mon.0 off 0 2026-02-23T13:10:06.359 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:10:06 trial185 ceph-osd[70713]: --2- 10.20.193.185:0/1351563358 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x290bc00 0x3748000 crc :-1 s=READ_MESSAGE_COMPLETE pgs=451 gs=1 cs=0 l=1 c_cookie=15490193144510272148 s_cookie=0 reconnecting=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_message received message m=0x296f860 seq=4 from=mon.0 type=18 auth_reply(proto 2 0 (0) Success) 2026-02-23T13:10:06.359 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:10:06 trial185 ceph-osd[70713]: -- 10.20.193.185:0/1351563358 queue 0x296f860 prio 196 2026-02-23T13:10:06.359 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:10:06 trial185 ceph-osd[70713]: --2- 10.20.193.185:0/1351563358 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x290bc00 0x3748000 crc :-1 s=READY pgs=451 gs=1 cs=0 l=1 c_cookie=15490193144510272148 s_cookie=0 reconnecting=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).read_frame 2026-02-23T13:10:06.360 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:10:06 trial185 ceph-osd[70713]: -- 10.20.193.185:0/1351563358 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x290bc00 msgr2=0x3748000 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read start len=32 2026-02-23T13:10:06.360 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:10:06 trial185 ceph-osd[70713]: -- 10.20.193.185:0/1351563358 <== mon.0 v2:10.20.193.68:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (crc 0 0 0) 0x296f860 con 0x290bc00 2026-02-23T13:10:06.360 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:10:06 trial185 ceph-osd[70713]: -- 10.20.193.185:0/1351563358 dispatch_throttle_release 194 to dispatch throttler 194/104857600 2026-02-23T13:10:06.360 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:10:06 trial185 ceph-osd[70713]: -- 10.20.193.185:0/1351563358 done calling dispatch on 0x296f860 2026-02-23T13:10:06.360 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:10:06 trial185 ceph-osd[70713]: WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:10:06.360 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:10:06 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-2[70709]: 2026-02-23T13:10:06.142+0000 7f1df943b780 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:10:06.360 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:10:06 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-2[70709]: 2026-02-23T13:10:06.142+0000 7f1df943b780 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:10:06.360 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:10:06 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-2[70709]: 2026-02-23T13:10:06.142+0000 7f1df943b780 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:10:06.392 INFO:teuthology.orchestra.run.trial185.stderr:no valid command found; 10 closest matches: 2026-02-23T13:10:06.392 INFO:teuthology.orchestra.run.trial185.stderr:0 2026-02-23T13:10:06.392 INFO:teuthology.orchestra.run.trial185.stderr:1 2026-02-23T13:10:06.393 INFO:teuthology.orchestra.run.trial185.stderr:2 2026-02-23T13:10:06.393 INFO:teuthology.orchestra.run.trial185.stderr:abort 2026-02-23T13:10:06.393 INFO:teuthology.orchestra.run.trial185.stderr:assert 2026-02-23T13:10:06.393 INFO:teuthology.orchestra.run.trial185.stderr:bluestore collections 2026-02-23T13:10:06.393 INFO:teuthology.orchestra.run.trial185.stderr:bluestore compression stats [] 2026-02-23T13:10:06.393 INFO:teuthology.orchestra.run.trial185.stderr:bluestore list [] [] 2026-02-23T13:10:06.393 INFO:teuthology.orchestra.run.trial185.stderr:bluestore onode metadata 2026-02-23T13:10:06.393 INFO:teuthology.orchestra.run.trial185.stderr:bluestore show sharding 2026-02-23T13:10:06.393 INFO:teuthology.orchestra.run.trial185.stderr:admin_socket: invalid command 2026-02-23T13:10:06.426 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-23T13:10:06.426 INFO:tasks.cephadm.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-23T13:10:06.459 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 22 128 125590 125462 44.5498 39.2188 0.0223212 0.0224275 2026-02-23T13:10:06.856 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:10:06 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-2[70709]: 2026-02-23T13:10:06.397+0000 7f1df943b780 -1 Falling back to public interface 2026-02-23T13:10:07.164 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:10:07.323 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:10:07.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:10:06 trial068 ceph-mon[38805]: pgmap v144: 17 pgs: 1 active+recovery_wait+undersized+degraded, 2 active+recovering+undersized+degraded, 1 active+recovering+undersized+degraded+remapped, 11 active+undersized+degraded, 2 active+clean; 828 MiB data, 3.6 GiB used, 2.7 TiB / 2.7 TiB avail; 26 MiB/s wr, 3.27k op/s; 13086/39738 objects degraded (32.931%); 39 MiB/s, 629 objects/s recovering 2026-02-23T13:10:07.459 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 23 127 130177 130050 44.1712 35.8438 0.063586 0.0226209 2026-02-23T13:10:07.747 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:10:07.700+0000 7fe25dd40640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:10:07.748 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:10:07.700+0000 7fe25dd40640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:10:07.748 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:10:07.748 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:10:07.748 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:10:07.748 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:10:07.748 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:10:07.749 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:10:07.749 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:10:07.875 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:10:07.876 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:10:08.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:10:07 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/34647225' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:10:08.459 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 24 128 134396 134268 43.7037 32.9531 0.0316488 0.0228622 2026-02-23T13:10:09.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:10:08 trial068 ceph-mon[38805]: pgmap v145: 17 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded, 2 active+recovering+undersized+degraded, 11 active+undersized+degraded, 2 active+clean; 980 MiB data, 4.1 GiB used, 2.7 TiB / 2.7 TiB avail; 39 MiB/s wr, 5.00k op/s; 14334/47052 objects degraded (30.464%); 37 MiB/s, 591 objects/s recovering 2026-02-23T13:10:09.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:10:08 trial068 ceph-mon[38805]: Health check update: Degraded data redundancy: 13086/39738 objects degraded (32.931%), 15 pgs degraded (PG_DEGRADED) 2026-02-23T13:10:09.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:10:08 trial068 ceph-mon[38805]: osdmap e89: 4 total, 3 up, 4 in 2026-02-23T13:10:09.459 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 25 128 138716 138588 43.3054 33.75 0.0100841 0.0228964 2026-02-23T13:10:09.877 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:10:09.996 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:10:10.122 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:10:10.120+0000 7f7957bfb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:10:10.122 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:10:10.120+0000 7f7957bfb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:10:10.138 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:10:10.138 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:10:10.154 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:10:10.154 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:10:10.154 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:10:10.168 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:10:10.168 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:10:10.207 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:10:10.207 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:10:10.459 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 26 128 141269 141141 42.4069 19.9453 0.0727953 0.0235599 2026-02-23T13:10:11.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:10:10 trial068 ceph-mon[38805]: osdmap e90: 4 total, 3 up, 4 in 2026-02-23T13:10:11.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:10:10 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/721804891' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:10:11.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:10:10 trial068 ceph-mon[38805]: pgmap v148: 17 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded, 2 active+recovering+undersized+degraded, 11 active+undersized+degraded, 2 active+clean; 980 MiB data, 4.1 GiB used, 2.7 TiB / 2.7 TiB avail; 32 MiB/s wr, 4.09k op/s; 14334/47052 objects degraded (30.464%); 13 MiB/s, 214 objects/s recovering 2026-02-23T13:10:11.427 DEBUG:teuthology.orchestra.run.trial185:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph daemon osd.2 dump_ops_in_flight 2026-02-23T13:10:11.459 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 27 128 145741 145613 42.1301 34.9375 0.0106981 0.0237165 2026-02-23T13:10:11.556 INFO:teuthology.orchestra.run.trial185.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/config/ceph.conf 2026-02-23T13:10:11.682 INFO:teuthology.orchestra.run.trial185.stderr:no valid command found; 10 closest matches: 2026-02-23T13:10:11.683 INFO:teuthology.orchestra.run.trial185.stderr:0 2026-02-23T13:10:11.683 INFO:teuthology.orchestra.run.trial185.stderr:1 2026-02-23T13:10:11.683 INFO:teuthology.orchestra.run.trial185.stderr:2 2026-02-23T13:10:11.683 INFO:teuthology.orchestra.run.trial185.stderr:abort 2026-02-23T13:10:11.683 INFO:teuthology.orchestra.run.trial185.stderr:assert 2026-02-23T13:10:11.683 INFO:teuthology.orchestra.run.trial185.stderr:bluefs debug_inject_read_zeros 2026-02-23T13:10:11.683 INFO:teuthology.orchestra.run.trial185.stderr:bluefs files list 2026-02-23T13:10:11.683 INFO:teuthology.orchestra.run.trial185.stderr:bluefs stats 2026-02-23T13:10:11.683 INFO:teuthology.orchestra.run.trial185.stderr:bluestore bluefs device info [] 2026-02-23T13:10:11.683 INFO:teuthology.orchestra.run.trial185.stderr:bluestore collections 2026-02-23T13:10:11.683 INFO:teuthology.orchestra.run.trial185.stderr:admin_socket: invalid command 2026-02-23T13:10:11.715 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-23T13:10:11.715 INFO:tasks.cephadm.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-23T13:10:12.208 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:10:12.365 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:10:12.460 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 28 128 150106 149978 41.8433 34.1016 0.0481936 0.0238821 2026-02-23T13:10:12.491 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:10:12.487+0000 7f1ec5809640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:10:12.491 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:10:12.487+0000 7f1ec5809640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:10:12.504 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:10:12.505 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:10:12.519 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:10:12.519 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:10:12.519 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:10:12.531 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:10:12.532 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:10:12.570 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:10:12.570 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:10:13.459 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 29 128 154263 154135 41.5202 32.4766 0.0437748 0.0240666 2026-02-23T13:10:13.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:10:13 trial068 ceph-mon[38805]: pgmap v149: 17 pgs: 15 active+undersized+degraded, 2 active+clean; 1.1 GiB data, 4.6 GiB used, 2.7 TiB / 2.7 TiB avail; 45 MiB/s wr, 5.73k op/s; 15938/54630 objects degraded (29.174%); 15 MiB/s, 244 objects/s recovering 2026-02-23T13:10:13.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:10:13 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2446846397' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:10:14.303 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:10:13 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:10:13] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:10:14.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:10:13 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:10:13.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:10:14.460 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 30 128 158256 158128 41.176 31.1953 0.028084 0.0242664 2026-02-23T13:10:14.571 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:10:14.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:10:14 trial068 ceph-mon[38805]: Health check update: Degraded data redundancy: 15938/54630 objects degraded (29.174%), 15 pgs degraded (PG_DEGRADED) 2026-02-23T13:10:14.714 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:10:14.855 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:10:14.853+0000 7fa6a71e8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:10:14.855 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:10:14.853+0000 7fa6a71e8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:10:14.868 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:10:14.869 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:10:14.887 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:10:14.887 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:10:14.887 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:10:14.900 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:10:14.901 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:10:14.936 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:10:14.936 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:10:15.304 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:10:15 trial068 ceph-mon[38805]: pgmap v150: 17 pgs: 15 active+undersized+degraded, 2 active+clean; 1.1 GiB data, 4.6 GiB used, 2.7 TiB / 2.7 TiB avail; 39 MiB/s wr, 4.96k op/s; 15938/54630 objects degraded (29.174%); 11 MiB/s, 174 objects/s recovering 2026-02-23T13:10:15.304 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:10:15 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3146616267' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:10:15.460 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 31 128 162224 162096 40.8476 31 0.0193088 0.0244597 2026-02-23T13:10:16.460 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 32 128 166049 165921 40.5049 29.8828 0.0080592 0.024668 2026-02-23T13:10:16.716 DEBUG:teuthology.orchestra.run.trial185:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph daemon osd.2 dump_ops_in_flight 2026-02-23T13:10:16.843 INFO:teuthology.orchestra.run.trial185.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/config/ceph.conf 2026-02-23T13:10:16.936 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:10:16.963 INFO:teuthology.orchestra.run.trial185.stderr:no valid command found; 10 closest matches: 2026-02-23T13:10:16.964 INFO:teuthology.orchestra.run.trial185.stderr:0 2026-02-23T13:10:16.964 INFO:teuthology.orchestra.run.trial185.stderr:1 2026-02-23T13:10:16.964 INFO:teuthology.orchestra.run.trial185.stderr:2 2026-02-23T13:10:16.964 INFO:teuthology.orchestra.run.trial185.stderr:abort 2026-02-23T13:10:16.964 INFO:teuthology.orchestra.run.trial185.stderr:assert 2026-02-23T13:10:16.964 INFO:teuthology.orchestra.run.trial185.stderr:bluefs debug_inject_read_zeros 2026-02-23T13:10:16.964 INFO:teuthology.orchestra.run.trial185.stderr:bluefs files list 2026-02-23T13:10:16.964 INFO:teuthology.orchestra.run.trial185.stderr:bluefs stats 2026-02-23T13:10:16.964 INFO:teuthology.orchestra.run.trial185.stderr:bluestore bluefs device info [] 2026-02-23T13:10:16.964 INFO:teuthology.orchestra.run.trial185.stderr:bluestore collections 2026-02-23T13:10:16.964 INFO:teuthology.orchestra.run.trial185.stderr:admin_socket: invalid command 2026-02-23T13:10:16.998 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-23T13:10:16.999 INFO:tasks.cephadm.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-23T13:10:17.081 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:10:17.211 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:10:17.210+0000 7f56f157a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:10:17.212 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:10:17.210+0000 7f56f157a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:10:17.232 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:10:17.232 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:10:17.246 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:10:17.247 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:10:17.247 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:10:17.261 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:10:17.261 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:10:17.302 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:10:17.303 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:10:17.460 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 33 128 169739 169611 40.151 28.8281 0.038781 0.0248891 2026-02-23T13:10:17.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:10:17 trial068 ceph-mon[38805]: pgmap v151: 17 pgs: 15 active+undersized+degraded, 2 active+clean; 1.2 GiB data, 4.7 GiB used, 2.7 TiB / 2.7 TiB avail; 25 MiB/s wr, 3.21k op/s; 16616/56664 objects degraded (29.324%); 4.8 MiB/s, 76 objects/s recovering 2026-02-23T13:10:17.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:10:17 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2209591347' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:10:18.460 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 34 127 173686 173559 39.8772 30.8438 0.0614305 0.0250615 2026-02-23T13:10:19.303 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:10:19.443 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:10:19.461 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 35 128 177327 177199 39.5503 28.4375 0.0401418 0.0252631 2026-02-23T13:10:19.561 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:10:19 trial068 ceph-mon[38805]: pgmap v152: 17 pgs: 15 active+undersized+degraded, 2 active+clean; 1.3 GiB data, 5.0 GiB used, 2.7 TiB / 2.7 TiB avail; 34 MiB/s wr, 4.37k op/s; 18188/62259 objects degraded (29.213%); 4.1 MiB/s, 65 objects/s recovering 2026-02-23T13:10:19.561 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:10:19 trial068 ceph-mon[38805]: Health check update: Degraded data redundancy: 18188/62259 objects degraded (29.213%), 15 pgs degraded (PG_DEGRADED) 2026-02-23T13:10:19.578 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:10:19.574+0000 7f9de9d3c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:10:19.578 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:10:19.574+0000 7f9de9d3c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:10:19.592 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:10:19.592 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:10:19.607 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:10:19.607 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:10:19.608 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:10:19.622 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:10:19.622 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:10:19.659 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:10:19.660 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:10:20.460 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 36 128 181114 180986 39.2734 29.5859 0.0235814 0.0254461 2026-02-23T13:10:20.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:10:20 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2641122258' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:10:21.460 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 37 128 184918 184790 39.0151 29.7188 0.0366458 0.0256093 2026-02-23T13:10:21.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:10:21 trial068 ceph-mon[38805]: pgmap v153: 17 pgs: 15 active+undersized+degraded, 2 active+clean; 1.3 GiB data, 5.0 GiB used, 2.7 TiB / 2.7 TiB avail; 31 MiB/s wr, 3.95k op/s; 18188/62259 objects degraded (29.213%); 3.7 MiB/s, 59 objects/s recovering 2026-02-23T13:10:21.661 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:10:21.811 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:10:21.939 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:10:21.937+0000 7f2925de7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:10:21.940 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:10:21.938+0000 7f2925de7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:10:21.956 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:10:21.956 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:10:21.972 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:10:21.972 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:10:21.972 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:10:21.983 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:10:21.983 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:10:22.000 DEBUG:teuthology.orchestra.run.trial185:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph daemon osd.2 dump_ops_in_flight 2026-02-23T13:10:22.020 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:10:22.021 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:10:22.107 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:10:21 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-2[70709]: 2026-02-23T13:10:21.848+0000 7f1df943b780 -1 osd.2 84 log_to_monitors true 2026-02-23T13:10:22.126 INFO:teuthology.orchestra.run.trial185.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/config/ceph.conf 2026-02-23T13:10:22.247 INFO:teuthology.orchestra.run.trial185.stderr:no valid command found; 10 closest matches: 2026-02-23T13:10:22.247 INFO:teuthology.orchestra.run.trial185.stderr:0 2026-02-23T13:10:22.247 INFO:teuthology.orchestra.run.trial185.stderr:1 2026-02-23T13:10:22.247 INFO:teuthology.orchestra.run.trial185.stderr:2 2026-02-23T13:10:22.247 INFO:teuthology.orchestra.run.trial185.stderr:abort 2026-02-23T13:10:22.247 INFO:teuthology.orchestra.run.trial185.stderr:assert 2026-02-23T13:10:22.247 INFO:teuthology.orchestra.run.trial185.stderr:bluefs debug_inject_read_zeros 2026-02-23T13:10:22.247 INFO:teuthology.orchestra.run.trial185.stderr:bluefs files list 2026-02-23T13:10:22.248 INFO:teuthology.orchestra.run.trial185.stderr:bluefs stats 2026-02-23T13:10:22.248 INFO:teuthology.orchestra.run.trial185.stderr:bluestore allocator dump block 2026-02-23T13:10:22.248 INFO:teuthology.orchestra.run.trial185.stderr:bluestore allocator fragmentation block 2026-02-23T13:10:22.248 INFO:teuthology.orchestra.run.trial185.stderr:admin_socket: invalid command 2026-02-23T13:10:22.278 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-23T13:10:22.279 INFO:tasks.cephadm.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-23T13:10:22.460 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 38 128 188584 188456 38.742 28.6406 0.00892142 0.0257968 2026-02-23T13:10:22.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:10:22 trial068 ceph-mon[38805]: from='osd.2 [v2:10.20.193.185:6800/2217358088,v1:10.20.193.185:6801/2217358088]' entity='osd.2' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]} : dispatch 2026-02-23T13:10:22.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:10:22 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3274258800' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:10:22.607 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:10:22 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-2[70709]: 2026-02-23T13:10:22.313+0000 7f1df1284640 -1 osd.2 84 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2026-02-23T13:10:23.460 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 39 128 192273 192145 38.4876 28.8203 0.0603366 0.0259662 2026-02-23T13:10:23.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:10:23 trial068 ceph-mon[38805]: pgmap v154: 17 pgs: 15 active+undersized+degraded, 2 active+clean; 1.4 GiB data, 5.3 GiB used, 2.7 TiB / 2.7 TiB avail; 39 MiB/s wr, 4.95k op/s; 20245/69333 objects degraded (29.200%); 3.2 MiB/s, 50 objects/s recovering 2026-02-23T13:10:23.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:10:23 trial068 ceph-mon[38805]: from='osd.2 [v2:10.20.193.185:6800/2217358088,v1:10.20.193.185:6801/2217358088]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2026-02-23T13:10:23.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:10:23 trial068 ceph-mon[38805]: osdmap e91: 4 total, 3 up, 4 in 2026-02-23T13:10:23.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:10:23 trial068 ceph-mon[38805]: from='osd.2 [v2:10.20.193.185:6800/2217358088,v1:10.20.193.185:6801/2217358088]' entity='osd.2' cmd={"prefix": "osd crush create-or-move", "id": 2, "weight":0.6836, "args": ["host=trial185", "root=default"]} : dispatch 2026-02-23T13:10:24.022 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:10:24.137 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:10:24.237 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:10:24.235+0000 7fc6af30b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:10:24.237 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:10:23 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:10:23] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:10:24.238 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:10:24.235+0000 7fc6af30b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:10:24.249 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:10:24.250 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:10:24.259 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:10:24.260 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:10:24.271 INFO:teuthology.orchestra.run.trial068.stdout:osd.2: {} 2026-02-23T13:10:24.272 INFO:teuthology.orchestra.run.trial068.stderr:osd.2: osd_enable_op_tracker = '' 2026-02-23T13:10:24.284 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:10:24.284 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:10:24.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:10:23 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:10:23.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:10:24.461 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T13:10:24.460586+0000 min lat: 0.000737354 max lat: 2.31352 avg lat: 0.0259992 2026-02-23T13:10:24.461 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T13:10:24.461 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 40 128 193463 193335 37.7578 9.29688 0.0136445 0.0259992 2026-02-23T13:10:24.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:10:24 trial068 ceph-mon[38805]: Health check cleared: OSD_DOWN (was: 1 osds down) 2026-02-23T13:10:24.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:10:24 trial068 ceph-mon[38805]: osd.2 [v2:10.20.193.185:6800/2217358088,v1:10.20.193.185:6801/2217358088] boot 2026-02-23T13:10:24.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:10:24 trial068 ceph-mon[38805]: osdmap e92: 4 total, 4 up, 4 in 2026-02-23T13:10:24.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:10:24 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2026-02-23T13:10:24.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:10:24 trial068 ceph-mon[38805]: Health check update: Degraded data redundancy: 20245/69333 objects degraded (29.200%), 15 pgs degraded (PG_DEGRADED) 2026-02-23T13:10:24.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:10:24 trial068 ceph-mon[38805]: osdmap e93: 4 total, 4 up, 4 in 2026-02-23T13:10:24.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:10:24 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3183703919' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:10:25.461 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 41 128 193921 193793 36.9241 3.57812 0.0449817 0.0270655 2026-02-23T13:10:25.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:10:25 trial068 ceph-mon[38805]: pgmap v158: 17 pgs: 15 active+undersized+degraded, 2 active+clean; 1.4 GiB data, 4.9 GiB used, 2.7 TiB / 2.7 TiB avail; 25 MiB/s wr, 3.14k op/s; 20245/69333 objects degraded (29.200%) 2026-02-23T13:10:25.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:10:25 trial068 ceph-mon[38805]: osdmap e94: 4 total, 4 up, 4 in 2026-02-23T13:10:26.334 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:10:26.461 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 42 128 197134 197006 36.6426 25.1016 0.0380096 0.0272721 2026-02-23T13:10:26.485 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:10:26.637 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:10:26.634+0000 7fa2fb7d2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:10:26.638 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:10:26.636+0000 7fa2fb7d2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:10:26.653 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:10:26.653 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:10:26.667 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:10:26.667 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:10:26.678 INFO:teuthology.orchestra.run.trial068.stdout:osd.2: {} 2026-02-23T13:10:26.679 INFO:teuthology.orchestra.run.trial068.stderr:osd.2: osd_enable_op_tracker = '' 2026-02-23T13:10:26.692 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:10:26.692 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:10:27.280 DEBUG:teuthology.orchestra.run.trial185:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph daemon osd.2 dump_ops_in_flight 2026-02-23T13:10:27.427 INFO:teuthology.orchestra.run.trial185.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/config/ceph.conf 2026-02-23T13:10:27.461 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 43 128 200187 200059 36.3451 23.8516 0.0432425 0.0274963 2026-02-23T13:10:27.585 INFO:teuthology.orchestra.run.trial185.stdout:ERROR: (22) Invalid argument 2026-02-23T13:10:27.586 INFO:teuthology.orchestra.run.trial185.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2026-02-23T13:10:27.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:10:27 trial068 ceph-mon[38805]: pgmap v160: 17 pgs: 4 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 10 active+undersized+degraded, 2 active+clean; 1.4 GiB data, 5.0 GiB used, 2.7 TiB / 2.7 TiB avail; 7.4 MiB/s wr, 952 op/s; 16940/70761 objects degraded (23.940%); 148 KiB/s, 2 objects/s recovering 2026-02-23T13:10:27.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:10:27 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2154928353' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:10:28.461 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 44 128 203497 203369 36.1067 25.8594 0.0178487 0.0276787 2026-02-23T13:10:28.732 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:10:28.882 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:10:29.006 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:10:29.004+0000 7f20eb7a1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:10:29.006 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:10:29.004+0000 7f20eb7a1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:10:29.024 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:10:29.024 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:10:29.040 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:10:29.040 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:10:29.054 INFO:teuthology.orchestra.run.trial068.stdout:osd.2: {} 2026-02-23T13:10:29.054 INFO:teuthology.orchestra.run.trial068.stderr:osd.2: osd_enable_op_tracker = '' 2026-02-23T13:10:29.070 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:10:29.070 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:10:29.461 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 45 128 206436 206308 35.8146 22.9609 0.00860687 0.0278083 2026-02-23T13:10:29.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:10:29 trial068 ceph-mon[38805]: pgmap v161: 17 pgs: 11 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 3 active+clean; 1.5 GiB data, 5.3 GiB used, 2.7 TiB / 2.7 TiB avail; 16 MiB/s wr, 2.02k op/s; 10279/73839 objects degraded (13.921%); 4.5 MiB/s, 71 objects/s recovering 2026-02-23T13:10:29.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:10:29 trial068 ceph-mon[38805]: Health check update: Degraded data redundancy: 10279/73839 objects degraded (13.921%), 14 pgs degraded (PG_DEGRADED) 2026-02-23T13:10:29.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:10:29 trial068 ceph-mon[38805]: osdmap e95: 4 total, 4 up, 4 in 2026-02-23T13:10:29.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:10:29 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/966343632' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:10:30.461 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 46 128 208673 208545 35.4157 17.4766 0.0198621 0.0281356 2026-02-23T13:10:31.118 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:10:31.264 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:10:31.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:10:30 trial068 ceph-mon[38805]: osdmap e96: 4 total, 4 up, 4 in 2026-02-23T13:10:31.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:10:30 trial068 ceph-mon[38805]: pgmap v164: 17 pgs: 11 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 3 active+clean; 1.5 GiB data, 5.3 GiB used, 2.7 TiB / 2.7 TiB avail; 16 MiB/s wr, 2.00k op/s; 10279/73839 objects degraded (13.921%); 4.4 MiB/s, 71 objects/s recovering 2026-02-23T13:10:31.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:10:30 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2026-02-23T13:10:31.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:10:30 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-23T13:10:31.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:10:30 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2026-02-23T13:10:31.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:10:30 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T13:10:31.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:10:30 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2026-02-23T13:10:31.461 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 47 128 210615 210487 34.9849 15.1719 0.0313097 0.0285077 2026-02-23T13:10:31.668 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:10:31.667+0000 7fd8a96ef640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:10:31.669 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:10:31.667+0000 7fd8a96ef640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:10:31.683 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:10:31.683 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:10:31.695 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:10:31.696 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:10:31.707 INFO:teuthology.orchestra.run.trial068.stdout:osd.2: {} 2026-02-23T13:10:31.708 INFO:teuthology.orchestra.run.trial068.stderr:osd.2: osd_enable_op_tracker = '' 2026-02-23T13:10:31.719 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:10:31.720 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:10:32.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:10:31 trial068 ceph-mon[38805]: osdmap e97: 4 total, 4 up, 4 in 2026-02-23T13:10:32.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:10:31 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/4257018915' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:10:32.461 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 48 128 212088 211960 34.4958 11.5078 0.00905634 0.0288231 2026-02-23T13:10:32.646 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 3, 2] 2026-02-23T13:10:32.646 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2026-02-23T13:10:32.647 INFO:tasks.thrashosds.thrasher:Killing osd 0, live_osds are [0, 1, 3, 2] 2026-02-23T13:10:32.647 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2026-02-23T13:10:32.647 DEBUG:teuthology.orchestra.run.trial068:> sudo systemctl stop ceph-5f823084-10b8-11f1-be80-d404e6e7d460@osd.0 2026-02-23T13:10:32.987 INFO:journalctl@ceph.osd.0.trial068.stdout:Feb 23 13:10:32 trial068 systemd[1]: Stopping Ceph osd.0 for 5f823084-10b8-11f1-be80-d404e6e7d460... 2026-02-23T13:10:32.987 INFO:journalctl@ceph.osd.0.trial068.stdout:Feb 23 13:10:32 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-0[54959]: 2026-02-23T13:10:32.744+0000 7f0a66854640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --osd-objectstore=bluestore (PID: 1) UID: 0 2026-02-23T13:10:32.988 INFO:journalctl@ceph.osd.0.trial068.stdout:Feb 23 13:10:32 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-0[54959]: 2026-02-23T13:10:32.744+0000 7f0a66854640 -1 osd.0 98 *** Got signal Terminated *** 2026-02-23T13:10:32.988 INFO:journalctl@ceph.osd.0.trial068.stdout:Feb 23 13:10:32 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-0[54959]: 2026-02-23T13:10:32.744+0000 7f0a66854640 -1 osd.0 98 *** Immediate shutdown (osd_fast_shutdown=true) *** 2026-02-23T13:10:33.325 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:10:32 trial068 ceph-mon[38805]: osdmap e98: 4 total, 4 up, 4 in 2026-02-23T13:10:33.325 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:10:32 trial068 ceph-mon[38805]: pgmap v167: 17 pgs: 7 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+degraded+remapped, 5 active+clean; 1.6 GiB data, 5.5 GiB used, 2.7 TiB / 2.7 TiB avail; 27 MiB/s wr, 3.41k op/s; 7228/78963 objects degraded (9.154%); 48 MiB/s, 762 objects/s recovering 2026-02-23T13:10:33.325 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:10:32 trial068 ceph-mon[38805]: osd.0 marked itself down and dead 2026-02-23T13:10:33.325 INFO:journalctl@ceph.osd.0.trial068.stdout:Feb 23 13:10:33 trial068 podman[91438]: 2026-02-23 13:10:33.190168171 +0000 UTC m=+0.457226921 container died b1da7a11e45f3a66ece3cd04ab6c6fd6fcc4d98550326a734307e2806339d9d3 (image=quay.ceph.io/ceph-ci/ceph@sha256:49621faaa845c39b2aae72e4fa24123ee9b2ac3fb8ae834753907068ee2972ea, name=ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-0, org.label-schema.build-date=20260216, org.opencontainers.image.authors=Ceph Release Team , FROM_IMAGE=quay.io/centos/centos:stream9, ceph=True, io.buildah.version=1.41.3, CEPH_SHA1=e371433fcd1390a0b82cfb18effc422c1f5d8c1d, CEPH_REF=wip-shraddhaag-enable-cephadm-tests, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, org.opencontainers.image.documentation=https://docs.ceph.com/, OSD_FLAVOR=default, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_GIT_REPO=https://github.com/ceph/ceph-ci.git, GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfsganesha-5/) 2026-02-23T13:10:33.325 INFO:journalctl@ceph.osd.0.trial068.stdout:Feb 23 13:10:33 trial068 podman[91438]: 2026-02-23 13:10:33.199548403 +0000 UTC m=+0.466607153 container remove b1da7a11e45f3a66ece3cd04ab6c6fd6fcc4d98550326a734307e2806339d9d3 (image=quay.ceph.io/ceph-ci/ceph@sha256:49621faaa845c39b2aae72e4fa24123ee9b2ac3fb8ae834753907068ee2972ea, name=ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-0, FROM_IMAGE=quay.io/centos/centos:stream9, CEPH_REF=wip-shraddhaag-enable-cephadm-tests, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, org.opencontainers.image.documentation=https://docs.ceph.com/, ceph=True, CEPH_GIT_REPO=https://github.com/ceph/ceph-ci.git, org.label-schema.build-date=20260216, GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfsganesha-5/, io.buildah.version=1.41.3, org.opencontainers.image.authors=Ceph Release Team , CEPH_SHA1=e371433fcd1390a0b82cfb18effc422c1f5d8c1d, org.label-schema.vendor=CentOS, OSD_FLAVOR=default) 2026-02-23T13:10:33.325 INFO:journalctl@ceph.osd.0.trial068.stdout:Feb 23 13:10:33 trial068 bash[91438]: ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-0 2026-02-23T13:10:33.462 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 49 128 212088 211960 33.7918 0 - 0.0288231 2026-02-23T13:10:33.544 DEBUG:teuthology.orchestra.run.trial068:> sudo pkill -f 'journalctl -f -n 0 -u ceph-5f823084-10b8-11f1-be80-d404e6e7d460@osd.0.service' 2026-02-23T13:10:33.582 DEBUG:teuthology.orchestra.run:got remote process result: None 2026-02-23T13:10:33.582 INFO:tasks.cephadm.osd.0:Stopped osd.0 2026-02-23T13:10:33.758 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:10:33.882 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:10:33.993 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:10:33.991+0000 7f6354ce6640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:10:33.993 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:10:33.991+0000 7f6354ce6640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:10:33.995 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.0 2026-02-23T13:10:34.006 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:10:34.006 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:10:34.017 INFO:teuthology.orchestra.run.trial068.stdout:osd.2: {} 2026-02-23T13:10:34.018 INFO:teuthology.orchestra.run.trial068.stderr:osd.2: osd_enable_op_tracker = '' 2026-02-23T13:10:34.030 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:10:34.031 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:10:34.065 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:10:34.066 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:10:34.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:10:33 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:10:33] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:10:34.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:10:33 trial068 ceph-mon[38805]: Health check failed: 1 osds down (OSD_DOWN) 2026-02-23T13:10:34.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:10:33 trial068 ceph-mon[38805]: osdmap e99: 4 total, 3 up, 4 in 2026-02-23T13:10:34.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:10:33 trial068 ceph-mon[38805]: Health check update: Degraded data redundancy: 7228/78963 objects degraded (9.154%), 12 pgs degraded (PG_DEGRADED) 2026-02-23T13:10:34.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:10:33 trial068 ceph-mon[38805]: osdmap e100: 4 total, 3 up, 4 in 2026-02-23T13:10:34.357 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:10:33 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:10:33.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:10:34.461 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 50 128 212480 212352 33.1773 1.53125 0.00842286 0.0299055 2026-02-23T13:10:35.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:10:34 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3362368212' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:10:35.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:10:34 trial068 ceph-mon[38805]: pgmap v170: 17 pgs: 2 stale+active+recovery_wait+undersized+degraded+remapped, 1 peering, 2 stale+active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 5 active+clean; 1.6 GiB data, 5.8 GiB used, 2.7 TiB / 2.7 TiB avail; 27 MiB/s wr, 3.46k op/s; 6818/79029 objects degraded (8.627%); 54 MiB/s, 859 objects/s recovering 2026-02-23T13:10:35.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:10:34 trial068 ceph-mon[38805]: osdmap e101: 4 total, 3 up, 4 in 2026-02-23T13:10:35.461 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 51 128 213792 213664 32.7277 10.25 0.0336467 0.0303948 2026-02-23T13:10:36.067 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:10:36.199 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:10:36.324 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:10:36.322+0000 7f6d0e890640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:10:36.324 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:10:36.322+0000 7f6d0e890640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:10:36.325 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.0 2026-02-23T13:10:36.337 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:10:36.337 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:10:36.350 INFO:teuthology.orchestra.run.trial068.stdout:osd.2: {} 2026-02-23T13:10:36.350 INFO:teuthology.orchestra.run.trial068.stderr:osd.2: osd_enable_op_tracker = '' 2026-02-23T13:10:36.365 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:10:36.365 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:10:36.400 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:10:36.401 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:10:36.462 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 52 127 215681 215554 32.3823 14.7656 0.0456515 0.0308658 2026-02-23T13:10:37.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:10:36 trial068 ceph-mon[38805]: osdmap e102: 4 total, 3 up, 4 in 2026-02-23T13:10:37.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:10:36 trial068 ceph-mon[38805]: pgmap v173: 17 pgs: 2 stale+active+recovery_wait+undersized+degraded+remapped, 1 peering, 2 stale+active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 5 active+clean; 1.6 GiB data, 5.8 GiB used, 2.7 TiB / 2.7 TiB avail; 360 KiB/s wr, 44 op/s; 6818/79029 objects degraded (8.627%); 6.0 MiB/s, 96 objects/s recovering 2026-02-23T13:10:37.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:10:36 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3641443768' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:10:37.462 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 53 127 219702 219575 32.364 31.4141 0.0215175 0.0308822 2026-02-23T13:10:38.402 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:10:38.462 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 54 128 224105 223977 32.4014 34.3906 0.0334834 0.0308498 2026-02-23T13:10:38.536 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:10:38.584 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [0] live_osds: [1, 3, 2] 2026-02-23T13:10:38.584 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2026-02-23T13:10:38.584 INFO:tasks.thrashosds.thrasher:Reviving osd 0 2026-02-23T13:10:38.584 INFO:tasks.cephadm.osd.0:Restarting osd.0 (starting--it wasn't running)... 2026-02-23T13:10:38.584 DEBUG:teuthology.orchestra.run.trial068:osd.0> sudo journalctl -f -n 0 -u ceph-5f823084-10b8-11f1-be80-d404e6e7d460@osd.0.service 2026-02-23T13:10:38.587 DEBUG:teuthology.orchestra.run.trial068:> sudo systemctl start ceph-5f823084-10b8-11f1-be80-d404e6e7d460@osd.0 2026-02-23T13:10:38.647 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:10:38.646+0000 7fb55b558640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:10:38.648 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:10:38.646+0000 7fb55b558640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:10:38.649 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.0 2026-02-23T13:10:38.661 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:10:38.661 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:10:38.673 INFO:teuthology.orchestra.run.trial068.stdout:osd.2: {} 2026-02-23T13:10:38.673 INFO:teuthology.orchestra.run.trial068.stderr:osd.2: osd_enable_op_tracker = '' 2026-02-23T13:10:38.677 INFO:journalctl@ceph.osd.0.trial068.stdout:Feb 23 13:10:38 trial068 systemd[1]: Starting Ceph osd.0 for 5f823084-10b8-11f1-be80-d404e6e7d460... 2026-02-23T13:10:38.686 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:10:38.686 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:10:38.724 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:10:38.725 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:10:39.094 INFO:journalctl@ceph.osd.0.trial068.stdout:Feb 23 13:10:38 trial068 podman[92066]: 2026-02-23 13:10:38.769062151 +0000 UTC m=+0.016845128 container create 34967eff5c036fcbcc95f45e15eba5705b72b487d38a9e23fb910a8c2f29c067 (image=quay.ceph.io/ceph-ci/ceph@sha256:49621faaa845c39b2aae72e4fa24123ee9b2ac3fb8ae834753907068ee2972ea, name=ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-0-activate, CEPH_GIT_REPO=https://github.com/ceph/ceph-ci.git, CEPH_REF=wip-shraddhaag-enable-cephadm-tests, org.label-schema.vendor=CentOS, org.opencontainers.image.authors=Ceph Release Team , org.label-schema.license=GPLv2, FROM_IMAGE=quay.io/centos/centos:stream9, io.buildah.version=1.41.3, CEPH_SHA1=e371433fcd1390a0b82cfb18effc422c1f5d8c1d, OSD_FLAVOR=default, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20260216, GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfsganesha-5/, org.opencontainers.image.documentation=https://docs.ceph.com/, org.label-schema.schema-version=1.0) 2026-02-23T13:10:39.094 INFO:journalctl@ceph.osd.0.trial068.stdout:Feb 23 13:10:38 trial068 podman[92066]: 2026-02-23 13:10:38.813419322 +0000 UTC m=+0.061202309 container init 34967eff5c036fcbcc95f45e15eba5705b72b487d38a9e23fb910a8c2f29c067 (image=quay.ceph.io/ceph-ci/ceph@sha256:49621faaa845c39b2aae72e4fa24123ee9b2ac3fb8ae834753907068ee2972ea, name=ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-0-activate, FROM_IMAGE=quay.io/centos/centos:stream9, ceph=True, CEPH_GIT_REPO=https://github.com/ceph/ceph-ci.git, org.label-schema.vendor=CentOS, org.opencontainers.image.authors=Ceph Release Team , org.label-schema.name=CentOS Stream 9 Base Image, OSD_FLAVOR=default, org.label-schema.build-date=20260216, org.label-schema.license=GPLv2, org.opencontainers.image.documentation=https://docs.ceph.com/, org.label-schema.schema-version=1.0, CEPH_REF=wip-shraddhaag-enable-cephadm-tests, io.buildah.version=1.41.3, GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfsganesha-5/, CEPH_SHA1=e371433fcd1390a0b82cfb18effc422c1f5d8c1d) 2026-02-23T13:10:39.094 INFO:journalctl@ceph.osd.0.trial068.stdout:Feb 23 13:10:38 trial068 podman[92066]: 2026-02-23 13:10:38.816672594 +0000 UTC m=+0.064455571 container start 34967eff5c036fcbcc95f45e15eba5705b72b487d38a9e23fb910a8c2f29c067 (image=quay.ceph.io/ceph-ci/ceph@sha256:49621faaa845c39b2aae72e4fa24123ee9b2ac3fb8ae834753907068ee2972ea, name=ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-0-activate, ceph=True, GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfsganesha-5/, FROM_IMAGE=quay.io/centos/centos:stream9, io.buildah.version=1.41.3, org.label-schema.build-date=20260216, org.label-schema.schema-version=1.0, CEPH_SHA1=e371433fcd1390a0b82cfb18effc422c1f5d8c1d, OSD_FLAVOR=default, CEPH_GIT_REPO=https://github.com/ceph/ceph-ci.git, org.label-schema.name=CentOS Stream 9 Base Image, org.opencontainers.image.documentation=https://docs.ceph.com/, org.opencontainers.image.authors=Ceph Release Team , org.label-schema.license=GPLv2, CEPH_REF=wip-shraddhaag-enable-cephadm-tests, org.label-schema.vendor=CentOS) 2026-02-23T13:10:39.095 INFO:journalctl@ceph.osd.0.trial068.stdout:Feb 23 13:10:38 trial068 podman[92066]: 2026-02-23 13:10:38.817125154 +0000 UTC m=+0.064908151 container attach 34967eff5c036fcbcc95f45e15eba5705b72b487d38a9e23fb910a8c2f29c067 (image=quay.ceph.io/ceph-ci/ceph@sha256:49621faaa845c39b2aae72e4fa24123ee9b2ac3fb8ae834753907068ee2972ea, name=ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-0-activate, org.label-schema.license=GPLv2, CEPH_SHA1=e371433fcd1390a0b82cfb18effc422c1f5d8c1d, org.label-schema.vendor=CentOS, io.buildah.version=1.41.3, org.opencontainers.image.authors=Ceph Release Team , org.label-schema.build-date=20260216, org.label-schema.schema-version=1.0, OSD_FLAVOR=default, CEPH_REF=wip-shraddhaag-enable-cephadm-tests, FROM_IMAGE=quay.io/centos/centos:stream9, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_GIT_REPO=https://github.com/ceph/ceph-ci.git, ceph=True, GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfsganesha-5/, org.opencontainers.image.documentation=https://docs.ceph.com/) 2026-02-23T13:10:39.095 INFO:journalctl@ceph.osd.0.trial068.stdout:Feb 23 13:10:38 trial068 podman[92066]: 2026-02-23 13:10:38.761715739 +0000 UTC m=+0.009498726 image pull 63df4b83db6f95a8ba965919b3671cb27173117c23b5c950aaadc8913f56213c quay.ceph.io/ceph-ci/ceph@sha256:49621faaa845c39b2aae72e4fa24123ee9b2ac3fb8ae834753907068ee2972ea 2026-02-23T13:10:39.095 INFO:journalctl@ceph.osd.0.trial068.stdout:Feb 23 13:10:38 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-0-activate[92085]: Running command: /usr/bin/ceph-authtool --gen-print-key 2026-02-23T13:10:39.095 INFO:journalctl@ceph.osd.0.trial068.stdout:Feb 23 13:10:38 trial068 bash[92066]: Running command: /usr/bin/ceph-authtool --gen-print-key 2026-02-23T13:10:39.095 INFO:journalctl@ceph.osd.0.trial068.stdout:Feb 23 13:10:38 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-0-activate[92085]: Running command: /usr/bin/ceph-authtool --gen-print-key 2026-02-23T13:10:39.095 INFO:journalctl@ceph.osd.0.trial068.stdout:Feb 23 13:10:38 trial068 bash[92066]: Running command: /usr/bin/ceph-authtool --gen-print-key 2026-02-23T13:10:39.462 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 55 128 228237 228109 32.3992 32.2812 0.0123537 0.0308522 2026-02-23T13:10:39.578 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:10:39 trial068 ceph-mon[38805]: pgmap v174: 17 pgs: 4 active+recovering+undersized+degraded, 1 active+recovery_wait+undersized+degraded+remapped, 9 active+undersized+degraded, 3 active+clean; 1.7 GiB data, 6.3 GiB used, 2.7 TiB / 2.7 TiB avail; 10 MiB/s wr, 1.31k op/s; 23350/81624 objects degraded (28.607%); 63 MiB/s, 1.01k objects/s recovering 2026-02-23T13:10:39.578 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:10:39 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1022516300' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:10:39.578 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:10:39 trial068 ceph-mon[38805]: Health check update: Degraded data redundancy: 23350/81624 objects degraded (28.607%), 14 pgs degraded (PG_DEGRADED) 2026-02-23T13:10:39.844 INFO:journalctl@ceph.osd.0.trial068.stdout:Feb 23 13:10:39 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-0-activate[92085]: --> Failed to activate via raw: did not find any matching OSD to activate 2026-02-23T13:10:39.844 INFO:journalctl@ceph.osd.0.trial068.stdout:Feb 23 13:10:39 trial068 bash[92066]: --> Failed to activate via raw: did not find any matching OSD to activate 2026-02-23T13:10:39.844 INFO:journalctl@ceph.osd.0.trial068.stdout:Feb 23 13:10:39 trial068 bash[92066]: Running command: /usr/bin/ceph-authtool --gen-print-key 2026-02-23T13:10:39.845 INFO:journalctl@ceph.osd.0.trial068.stdout:Feb 23 13:10:39 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-0-activate[92085]: Running command: /usr/bin/ceph-authtool --gen-print-key 2026-02-23T13:10:39.845 INFO:journalctl@ceph.osd.0.trial068.stdout:Feb 23 13:10:39 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-0-activate[92085]: Running command: /usr/bin/ceph-authtool --gen-print-key 2026-02-23T13:10:39.845 INFO:journalctl@ceph.osd.0.trial068.stdout:Feb 23 13:10:39 trial068 bash[92066]: Running command: /usr/bin/ceph-authtool --gen-print-key 2026-02-23T13:10:39.845 INFO:journalctl@ceph.osd.0.trial068.stdout:Feb 23 13:10:39 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-0-activate[92085]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2026-02-23T13:10:39.845 INFO:journalctl@ceph.osd.0.trial068.stdout:Feb 23 13:10:39 trial068 bash[92066]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2026-02-23T13:10:39.845 INFO:journalctl@ceph.osd.0.trial068.stdout:Feb 23 13:10:39 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-0-activate[92085]: Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/vg_nvme/lv_4 --path /var/lib/ceph/osd/ceph-0 --no-mon-config 2026-02-23T13:10:39.845 INFO:journalctl@ceph.osd.0.trial068.stdout:Feb 23 13:10:39 trial068 bash[92066]: Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/vg_nvme/lv_4 --path /var/lib/ceph/osd/ceph-0 --no-mon-config 2026-02-23T13:10:39.845 INFO:journalctl@ceph.osd.0.trial068.stdout:Feb 23 13:10:39 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-0-activate[92085]: Running command: /usr/bin/ln -snf /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-0/block 2026-02-23T13:10:39.845 INFO:journalctl@ceph.osd.0.trial068.stdout:Feb 23 13:10:39 trial068 bash[92066]: Running command: /usr/bin/ln -snf /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-0/block 2026-02-23T13:10:39.845 INFO:journalctl@ceph.osd.0.trial068.stdout:Feb 23 13:10:39 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-0-activate[92085]: Running command: /usr/bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-0/block 2026-02-23T13:10:39.845 INFO:journalctl@ceph.osd.0.trial068.stdout:Feb 23 13:10:39 trial068 bash[92066]: Running command: /usr/bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-0/block 2026-02-23T13:10:39.846 INFO:journalctl@ceph.osd.0.trial068.stdout:Feb 23 13:10:39 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-0-activate[92085]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2026-02-23T13:10:39.846 INFO:journalctl@ceph.osd.0.trial068.stdout:Feb 23 13:10:39 trial068 bash[92066]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2026-02-23T13:10:39.846 INFO:journalctl@ceph.osd.0.trial068.stdout:Feb 23 13:10:39 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-0-activate[92085]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2026-02-23T13:10:39.846 INFO:journalctl@ceph.osd.0.trial068.stdout:Feb 23 13:10:39 trial068 bash[92066]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2026-02-23T13:10:39.846 INFO:journalctl@ceph.osd.0.trial068.stdout:Feb 23 13:10:39 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-0-activate[92085]: --> ceph-volume lvm activate successful for osd ID: 0 2026-02-23T13:10:39.846 INFO:journalctl@ceph.osd.0.trial068.stdout:Feb 23 13:10:39 trial068 bash[92066]: --> ceph-volume lvm activate successful for osd ID: 0 2026-02-23T13:10:39.846 INFO:journalctl@ceph.osd.0.trial068.stdout:Feb 23 13:10:39 trial068 podman[92066]: 2026-02-23 13:10:39.729827615 +0000 UTC m=+0.977610602 container died 34967eff5c036fcbcc95f45e15eba5705b72b487d38a9e23fb910a8c2f29c067 (image=quay.ceph.io/ceph-ci/ceph@sha256:49621faaa845c39b2aae72e4fa24123ee9b2ac3fb8ae834753907068ee2972ea, name=ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-0-activate, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_SHA1=e371433fcd1390a0b82cfb18effc422c1f5d8c1d, CEPH_GIT_REPO=https://github.com/ceph/ceph-ci.git, org.opencontainers.image.authors=Ceph Release Team , ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20260216, CEPH_REF=wip-shraddhaag-enable-cephadm-tests, org.label-schema.license=GPLv2, FROM_IMAGE=quay.io/centos/centos:stream9, io.buildah.version=1.41.3, GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfsganesha-5/, org.opencontainers.image.documentation=https://docs.ceph.com/, OSD_FLAVOR=default) 2026-02-23T13:10:39.846 INFO:journalctl@ceph.osd.0.trial068.stdout:Feb 23 13:10:39 trial068 podman[92066]: 2026-02-23 13:10:39.741729855 +0000 UTC m=+0.989512842 container remove 34967eff5c036fcbcc95f45e15eba5705b72b487d38a9e23fb910a8c2f29c067 (image=quay.ceph.io/ceph-ci/ceph@sha256:49621faaa845c39b2aae72e4fa24123ee9b2ac3fb8ae834753907068ee2972ea, name=ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-0-activate, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfsganesha-5/, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.opencontainers.image.authors=Ceph Release Team , io.buildah.version=1.41.3, FROM_IMAGE=quay.io/centos/centos:stream9, org.label-schema.build-date=20260216, CEPH_REF=wip-shraddhaag-enable-cephadm-tests, CEPH_GIT_REPO=https://github.com/ceph/ceph-ci.git, org.opencontainers.image.documentation=https://docs.ceph.com/, CEPH_SHA1=e371433fcd1390a0b82cfb18effc422c1f5d8c1d, OSD_FLAVOR=default, ceph=True) 2026-02-23T13:10:39.846 INFO:journalctl@ceph.osd.0.trial068.stdout:Feb 23 13:10:39 trial068 podman[92313]: 2026-02-23 13:10:39.823701612 +0000 UTC m=+0.017615695 container create f9b4774381c5c1c92e916759cac634c58cf73aaf3d836daa0bd3040a85efe78b (image=quay.ceph.io/ceph-ci/ceph@sha256:49621faaa845c39b2aae72e4fa24123ee9b2ac3fb8ae834753907068ee2972ea, name=ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-0, org.label-schema.schema-version=1.0, CEPH_REF=wip-shraddhaag-enable-cephadm-tests, CEPH_GIT_REPO=https://github.com/ceph/ceph-ci.git, FROM_IMAGE=quay.io/centos/centos:stream9, ceph=True, CEPH_SHA1=e371433fcd1390a0b82cfb18effc422c1f5d8c1d, org.label-schema.build-date=20260216, io.buildah.version=1.41.3, OSD_FLAVOR=default, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.opencontainers.image.documentation=https://docs.ceph.com/, org.opencontainers.image.authors=Ceph Release Team , org.label-schema.name=CentOS Stream 9 Base Image, GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfsganesha-5/) 2026-02-23T13:10:40.032 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph daemon osd.0 dump_ops_in_flight 2026-02-23T13:10:40.170 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:10:40.287 INFO:journalctl@ceph.osd.0.trial068.stdout:Feb 23 13:10:39 trial068 podman[92313]: 2026-02-23 13:10:39.817307319 +0000 UTC m=+0.011221342 image pull 63df4b83db6f95a8ba965919b3671cb27173117c23b5c950aaadc8913f56213c quay.ceph.io/ceph-ci/ceph@sha256:49621faaa845c39b2aae72e4fa24123ee9b2ac3fb8ae834753907068ee2972ea 2026-02-23T13:10:40.288 INFO:journalctl@ceph.osd.0.trial068.stdout:Feb 23 13:10:39 trial068 podman[92313]: 2026-02-23 13:10:39.92050954 +0000 UTC m=+0.114423573 container init f9b4774381c5c1c92e916759cac634c58cf73aaf3d836daa0bd3040a85efe78b (image=quay.ceph.io/ceph-ci/ceph@sha256:49621faaa845c39b2aae72e4fa24123ee9b2ac3fb8ae834753907068ee2972ea, name=ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-0, CEPH_SHA1=e371433fcd1390a0b82cfb18effc422c1f5d8c1d, io.buildah.version=1.41.3, org.label-schema.schema-version=1.0, org.opencontainers.image.authors=Ceph Release Team , FROM_IMAGE=quay.io/centos/centos:stream9, org.label-schema.build-date=20260216, org.opencontainers.image.documentation=https://docs.ceph.com/, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfsganesha-5/, ceph=True, org.label-schema.license=GPLv2, CEPH_GIT_REPO=https://github.com/ceph/ceph-ci.git, CEPH_REF=wip-shraddhaag-enable-cephadm-tests, OSD_FLAVOR=default) 2026-02-23T13:10:40.288 INFO:journalctl@ceph.osd.0.trial068.stdout:Feb 23 13:10:39 trial068 podman[92313]: 2026-02-23 13:10:39.924249176 +0000 UTC m=+0.118163209 container start f9b4774381c5c1c92e916759cac634c58cf73aaf3d836daa0bd3040a85efe78b (image=quay.ceph.io/ceph-ci/ceph@sha256:49621faaa845c39b2aae72e4fa24123ee9b2ac3fb8ae834753907068ee2972ea, name=ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-0, org.opencontainers.image.authors=Ceph Release Team , org.label-schema.vendor=CentOS, FROM_IMAGE=quay.io/centos/centos:stream9, CEPH_SHA1=e371433fcd1390a0b82cfb18effc422c1f5d8c1d, org.opencontainers.image.documentation=https://docs.ceph.com/, org.label-schema.schema-version=1.0, CEPH_REF=wip-shraddhaag-enable-cephadm-tests, CEPH_GIT_REPO=https://github.com/ceph/ceph-ci.git, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.build-date=20260216, io.buildah.version=1.41.3, OSD_FLAVOR=default, org.label-schema.license=GPLv2, GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfsganesha-5/) 2026-02-23T13:10:40.288 INFO:journalctl@ceph.osd.0.trial068.stdout:Feb 23 13:10:39 trial068 ceph-osd[92338]: -- 10.20.193.68:0/1993837559 <== mon.0 v2:10.20.193.68:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (crc 0 0 0) 0x4267860 con 0x4203c00 2026-02-23T13:10:40.288 INFO:journalctl@ceph.osd.0.trial068.stdout:Feb 23 13:10:39 trial068 ceph-osd[92338]: -- 10.20.193.68:0/1993837559 dispatch_throttle_release 194 to dispatch throttler 194/104857600 2026-02-23T13:10:40.288 INFO:journalctl@ceph.osd.0.trial068.stdout:Feb 23 13:10:39 trial068 ceph-osd[92338]: -- 10.20.193.68:0/1993837559 done calling dispatch on 0x4267860 2026-02-23T13:10:40.288 INFO:journalctl@ceph.osd.0.trial068.stdout:Feb 23 13:10:39 trial068 ceph-osd[92338]: WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:10:40.288 INFO:journalctl@ceph.osd.0.trial068.stdout:Feb 23 13:10:39 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-0[92334]: 2026-02-23T13:10:39.940+0000 7f5100ea8780 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:10:40.288 INFO:journalctl@ceph.osd.0.trial068.stdout:Feb 23 13:10:39 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-0[92334]: 2026-02-23T13:10:39.941+0000 7f5100ea8780 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:10:40.289 INFO:journalctl@ceph.osd.0.trial068.stdout:Feb 23 13:10:39 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-0[92334]: 2026-02-23T13:10:39.941+0000 7f5100ea8780 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:10:40.289 INFO:journalctl@ceph.osd.0.trial068.stdout:Feb 23 13:10:40 trial068 bash[92313]: f9b4774381c5c1c92e916759cac634c58cf73aaf3d836daa0bd3040a85efe78b 2026-02-23T13:10:40.289 INFO:journalctl@ceph.osd.0.trial068.stdout:Feb 23 13:10:40 trial068 systemd[1]: Started Ceph osd.0 for 5f823084-10b8-11f1-be80-d404e6e7d460. 2026-02-23T13:10:40.289 INFO:journalctl@ceph.osd.0.trial068.stdout:Feb 23 13:10:40 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-0[92334]: 2026-02-23T13:10:40.163+0000 7f5100ea8780 -1 Falling back to public interface 2026-02-23T13:10:40.462 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 56 128 232089 231961 32.358 30.0938 0.0484949 0.0308893 2026-02-23T13:10:40.593 INFO:teuthology.orchestra.run.trial068.stderr:no valid command found; 10 closest matches: 2026-02-23T13:10:40.594 INFO:teuthology.orchestra.run.trial068.stderr:0 2026-02-23T13:10:40.594 INFO:teuthology.orchestra.run.trial068.stderr:1 2026-02-23T13:10:40.594 INFO:teuthology.orchestra.run.trial068.stderr:2 2026-02-23T13:10:40.594 INFO:teuthology.orchestra.run.trial068.stderr:abort 2026-02-23T13:10:40.594 INFO:teuthology.orchestra.run.trial068.stderr:assert 2026-02-23T13:10:40.594 INFO:teuthology.orchestra.run.trial068.stderr:bluestore collections 2026-02-23T13:10:40.594 INFO:teuthology.orchestra.run.trial068.stderr:bluestore compression stats [] 2026-02-23T13:10:40.594 INFO:teuthology.orchestra.run.trial068.stderr:bluestore list [] [] 2026-02-23T13:10:40.594 INFO:teuthology.orchestra.run.trial068.stderr:bluestore onode metadata 2026-02-23T13:10:40.595 INFO:teuthology.orchestra.run.trial068.stderr:bluestore show sharding 2026-02-23T13:10:40.595 INFO:teuthology.orchestra.run.trial068.stderr:admin_socket: invalid command 2026-02-23T13:10:40.726 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:10:40.990 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:10:40.992 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-23T13:10:40.992 INFO:tasks.cephadm.ceph_manager.ceph:waiting on admin_socket for osd-0, ['dump_ops_in_flight'] 2026-02-23T13:10:41.415 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:10:41.392+0000 7f2bd9f62640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:10:41.415 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:10:41.393+0000 7f2bd9f62640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:10:41.415 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.0 2026-02-23T13:10:41.416 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:10:41.416 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:10:41.421 INFO:teuthology.orchestra.run.trial068.stdout:osd.2: {} 2026-02-23T13:10:41.422 INFO:teuthology.orchestra.run.trial068.stderr:osd.2: osd_enable_op_tracker = '' 2026-02-23T13:10:41.434 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:10:41.434 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:10:41.462 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 57 128 235349 235221 32.2371 25.4688 0.0201998 0.0310028 2026-02-23T13:10:41.523 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:10:41 trial068 ceph-mon[38805]: pgmap v175: 17 pgs: 4 active+recovering+undersized+degraded, 1 active+recovery_wait+undersized+degraded+remapped, 9 active+undersized+degraded, 3 active+clean; 1.7 GiB data, 6.3 GiB used, 2.7 TiB / 2.7 TiB avail; 8.6 MiB/s wr, 1.10k op/s; 23350/81624 objects degraded (28.607%); 53 MiB/s, 844 objects/s recovering 2026-02-23T13:10:41.546 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:10:41.546 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:10:42.462 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 58 128 237201 237073 31.9307 14.4688 0.0319387 0.0312578 2026-02-23T13:10:42.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:10:42 trial068 ceph-mon[38805]: osdmap e103: 4 total, 3 up, 4 in 2026-02-23T13:10:42.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:10:42 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3850799979' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:10:43.462 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 59 128 241363 241235 31.9406 32.5156 0.0242752 0.0312818 2026-02-23T13:10:43.547 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:10:43.570 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:10:43 trial068 ceph-mon[38805]: pgmap v177: 17 pgs: 1 active+recovering+undersized+degraded, 1 active+recovery_wait+undersized+degraded+remapped, 12 active+undersized+degraded, 3 active+clean; 1.8 GiB data, 6.5 GiB used, 2.7 TiB / 2.7 TiB avail; 21 MiB/s wr, 2.72k op/s; 24231/86442 objects degraded (28.032%); 49 MiB/s, 788 objects/s recovering 2026-02-23T13:10:43.571 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:10:43 trial068 ceph-mon[38805]: osdmap e104: 4 total, 3 up, 4 in 2026-02-23T13:10:43.675 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:10:43.794 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:10:43.792+0000 7fb3533e2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:10:43.794 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:10:43.793+0000 7fb3533e2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:10:43.796 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.0 2026-02-23T13:10:43.807 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:10:43.807 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:10:43.820 INFO:teuthology.orchestra.run.trial068.stdout:osd.2: {} 2026-02-23T13:10:43.820 INFO:teuthology.orchestra.run.trial068.stderr:osd.2: osd_enable_op_tracker = '' 2026-02-23T13:10:43.832 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:10:43.832 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:10:43.865 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:10:43.866 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:10:44.093 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:10:43 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:10:43] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:10:44.357 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:10:43 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:10:43.972Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:10:44.462 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T13:10:44.462395+0000 min lat: 0.000737354 max lat: 2.31352 avg lat: 0.0313156 2026-02-23T13:10:44.463 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T13:10:44.463 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 60 128 245316 245188 31.9229 30.8828 0.0569548 0.0313156 2026-02-23T13:10:44.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:10:44 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3754461651' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:10:44.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:10:44 trial068 ceph-mon[38805]: Health check update: Degraded data redundancy: 24231/86442 objects degraded (28.032%), 14 pgs degraded (PG_DEGRADED) 2026-02-23T13:10:45.462 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 61 128 249951 249823 31.9931 36.2109 0.0130969 0.0312424 2026-02-23T13:10:45.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:10:45 trial068 ceph-mon[38805]: pgmap v179: 17 pgs: 14 active+undersized+degraded, 3 active+clean; 1.8 GiB data, 6.8 GiB used, 2.7 TiB / 2.7 TiB avail; 25 MiB/s wr, 3.20k op/s; 24090/88620 objects degraded (27.183%); 50 MiB/s, 807 objects/s recovering 2026-02-23T13:10:45.867 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:10:45.993 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph daemon osd.0 dump_ops_in_flight 2026-02-23T13:10:46.003 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:10:46.117 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:10:46.116+0000 7fd2f38a4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:10:46.118 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:10:46.116+0000 7fd2f38a4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:10:46.119 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.0 2026-02-23T13:10:46.125 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:10:46.131 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:10:46.131 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:10:46.145 INFO:teuthology.orchestra.run.trial068.stdout:osd.2: {} 2026-02-23T13:10:46.145 INFO:teuthology.orchestra.run.trial068.stderr:osd.2: osd_enable_op_tracker = '' 2026-02-23T13:10:46.159 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:10:46.159 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:10:46.202 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:10:46.203 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:10:46.246 INFO:teuthology.orchestra.run.trial068.stderr:no valid command found; 10 closest matches: 2026-02-23T13:10:46.246 INFO:teuthology.orchestra.run.trial068.stderr:0 2026-02-23T13:10:46.246 INFO:teuthology.orchestra.run.trial068.stderr:1 2026-02-23T13:10:46.246 INFO:teuthology.orchestra.run.trial068.stderr:2 2026-02-23T13:10:46.246 INFO:teuthology.orchestra.run.trial068.stderr:abort 2026-02-23T13:10:46.246 INFO:teuthology.orchestra.run.trial068.stderr:assert 2026-02-23T13:10:46.246 INFO:teuthology.orchestra.run.trial068.stderr:bluefs debug_inject_read_zeros 2026-02-23T13:10:46.246 INFO:teuthology.orchestra.run.trial068.stderr:bluefs files list 2026-02-23T13:10:46.246 INFO:teuthology.orchestra.run.trial068.stderr:bluefs stats 2026-02-23T13:10:46.246 INFO:teuthology.orchestra.run.trial068.stderr:bluestore bluefs device info [] 2026-02-23T13:10:46.246 INFO:teuthology.orchestra.run.trial068.stderr:bluestore collections 2026-02-23T13:10:46.246 INFO:teuthology.orchestra.run.trial068.stderr:admin_socket: invalid command 2026-02-23T13:10:46.280 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-23T13:10:46.281 INFO:tasks.cephadm.ceph_manager.ceph:waiting on admin_socket for osd-0, ['dump_ops_in_flight'] 2026-02-23T13:10:46.462 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 62 128 254162 254034 32.0077 32.8984 0.0252862 0.0312313 2026-02-23T13:10:46.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:10:46 trial068 ceph-mon[38805]: 3.2 scrub starts 2026-02-23T13:10:46.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:10:46 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2212217672' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:10:47.463 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 63 128 258715 258587 32.0642 35.5703 0.0116558 0.0311732 2026-02-23T13:10:47.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:10:47 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T13:10:47.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:10:47 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T13:10:47.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:10:47 trial068 ceph-mon[38805]: 3.2 scrub ok 2026-02-23T13:10:47.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:10:47 trial068 ceph-mon[38805]: pgmap v180: 17 pgs: 14 active+undersized+degraded, 3 active+clean; 1.8 GiB data, 6.8 GiB used, 2.7 TiB / 2.7 TiB avail; 18 MiB/s wr, 2.33k op/s; 24090/88620 objects degraded (27.183%); 9.1 MiB/s, 145 objects/s recovering 2026-02-23T13:10:48.204 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:10:48.340 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:10:48.452 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:10:48.450+0000 7f2c0905a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:10:48.452 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:10:48.450+0000 7f2c0905a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:10:48.453 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.0 2026-02-23T13:10:48.463 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 64 128 263295 263167 32.1223 35.7812 0.0908518 0.0311159 2026-02-23T13:10:48.465 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:10:48.466 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:10:48.478 INFO:teuthology.orchestra.run.trial068.stdout:osd.2: {} 2026-02-23T13:10:48.479 INFO:teuthology.orchestra.run.trial068.stderr:osd.2: osd_enable_op_tracker = '' 2026-02-23T13:10:48.479 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:10:48 trial068 ceph-mon[38805]: 3.a scrub starts 2026-02-23T13:10:48.489 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:10:48.490 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:10:48.525 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:10:48.525 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:10:49.463 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 65 128 267617 267489 32.1475 33.7656 0.0187794 0.031095 2026-02-23T13:10:49.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:10:49 trial068 ceph-mon[38805]: 3.a scrub ok 2026-02-23T13:10:49.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:10:49 trial068 ceph-mon[38805]: pgmap v181: 17 pgs: 14 active+undersized+degraded, 3 active+clean; 2.0 GiB data, 7.2 GiB used, 2.7 TiB / 2.7 TiB avail; 38 MiB/s wr, 4.89k op/s; 26175/96306 objects degraded (27.179%); 9.1 MiB/s, 145 objects/s recovering 2026-02-23T13:10:49.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:10:49 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/4138039917' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:10:49.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:10:49 trial068 ceph-mon[38805]: Health check update: Degraded data redundancy: 26175/96306 objects degraded (27.179%), 14 pgs degraded (PG_DEGRADED) 2026-02-23T13:10:50.463 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 66 128 272076 271948 32.1882 34.8359 0.0257415 0.0310563 2026-02-23T13:10:50.527 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:10:50.663 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:10:50.786 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:10:50.784+0000 7f7921e85640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:10:50.786 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:10:50.784+0000 7f7921e85640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:10:50.788 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.0 2026-02-23T13:10:50.801 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:10:50.801 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:10:50.814 INFO:teuthology.orchestra.run.trial068.stdout:osd.2: {} 2026-02-23T13:10:50.814 INFO:teuthology.orchestra.run.trial068.stderr:osd.2: osd_enable_op_tracker = '' 2026-02-23T13:10:50.826 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:10:50.826 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:10:50.864 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:10:50.865 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:10:51.282 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph daemon osd.0 dump_ops_in_flight 2026-02-23T13:10:51.410 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:10:51.463 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 67 127 276246 276119 32.1941 32.5859 0.0261077 0.031048 2026-02-23T13:10:51.533 INFO:teuthology.orchestra.run.trial068.stderr:no valid command found; 10 closest matches: 2026-02-23T13:10:51.533 INFO:teuthology.orchestra.run.trial068.stderr:0 2026-02-23T13:10:51.533 INFO:teuthology.orchestra.run.trial068.stderr:1 2026-02-23T13:10:51.533 INFO:teuthology.orchestra.run.trial068.stderr:2 2026-02-23T13:10:51.533 INFO:teuthology.orchestra.run.trial068.stderr:abort 2026-02-23T13:10:51.533 INFO:teuthology.orchestra.run.trial068.stderr:assert 2026-02-23T13:10:51.533 INFO:teuthology.orchestra.run.trial068.stderr:bluefs debug_inject_read_zeros 2026-02-23T13:10:51.533 INFO:teuthology.orchestra.run.trial068.stderr:bluefs files list 2026-02-23T13:10:51.533 INFO:teuthology.orchestra.run.trial068.stderr:bluefs stats 2026-02-23T13:10:51.534 INFO:teuthology.orchestra.run.trial068.stderr:bluestore bluefs device info [] 2026-02-23T13:10:51.534 INFO:teuthology.orchestra.run.trial068.stderr:bluestore collections 2026-02-23T13:10:51.534 INFO:teuthology.orchestra.run.trial068.stderr:admin_socket: invalid command 2026-02-23T13:10:51.564 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:10:51 trial068 ceph-mon[38805]: pgmap v182: 17 pgs: 14 active+undersized+degraded, 3 active+clean; 2.0 GiB data, 7.2 GiB used, 2.7 TiB / 2.7 TiB avail; 34 MiB/s wr, 4.38k op/s; 26175/96306 objects degraded (27.179%); 8.1 MiB/s, 130 objects/s recovering 2026-02-23T13:10:51.564 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:10:51 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3847092843' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:10:51.564 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:10:51 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T13:10:51.564 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:10:51 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T13:10:51.565 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-23T13:10:51.566 INFO:tasks.cephadm.ceph_manager.ceph:waiting on admin_socket for osd-0, ['dump_ops_in_flight'] 2026-02-23T13:10:52.463 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 68 128 280701 280573 32.2323 34.7969 0.00764062 0.0310111 2026-02-23T13:10:52.866 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:10:52.998 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:10:53.365 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:10:53.363+0000 7f42260ec640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:10:53.366 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:10:53.364+0000 7f42260ec640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:10:53.367 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.0 2026-02-23T13:10:53.381 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:10:53.381 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:10:53.396 INFO:teuthology.orchestra.run.trial068.stdout:osd.2: {} 2026-02-23T13:10:53.396 INFO:teuthology.orchestra.run.trial068.stderr:osd.2: osd_enable_op_tracker = '' 2026-02-23T13:10:53.410 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:10:53.411 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:10:53.411 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:10:53 trial068 ceph-mon[38805]: pgmap v183: 17 pgs: 14 active+undersized+degraded, 3 active+clean; 2.1 GiB data, 7.5 GiB used, 2.7 TiB / 2.7 TiB avail; 32 MiB/s wr, 4.14k op/s; 27720/101967 objects degraded (27.185%); 4.5 MiB/s, 72 objects/s recovering 2026-02-23T13:10:53.443 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:10:53.443 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:10:53.463 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 69 128 284789 284661 32.228 31.9375 0.0117862 0.0310167 2026-02-23T13:10:54.322 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:10:53 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:10:53] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:10:54.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:10:53 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:10:53.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:10:54.463 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 70 127 289135 289008 32.2527 33.9609 0.0192747 0.030995 2026-02-23T13:10:54.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:10:54 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1360865377' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:10:54.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:10:54 trial068 ceph-mon[38805]: Health check update: Degraded data redundancy: 27720/101967 objects degraded (27.185%), 14 pgs degraded (PG_DEGRADED) 2026-02-23T13:10:55.445 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:10:55.464 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 71 128 293262 293134 32.2525 32.2344 0.0102629 0.0309945 2026-02-23T13:10:55.578 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:10:55.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:10:55 trial068 ceph-mon[38805]: pgmap v184: 17 pgs: 14 active+undersized+degraded, 3 active+clean; 2.1 GiB data, 7.7 GiB used, 2.7 TiB / 2.7 TiB avail; 32 MiB/s wr, 4.05k op/s; 28437/104580 objects degraded (27.192%); 3.8 MiB/s, 60 objects/s recovering 2026-02-23T13:10:55.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:10:55 trial068 ceph-mon[38805]: 3.e scrub starts 2026-02-23T13:10:55.689 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:10:55.687+0000 7fafee293640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:10:55.690 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:10:55.688+0000 7fafee293640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:10:55.691 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.0 2026-02-23T13:10:55.702 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:10:55.703 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:10:55.714 INFO:teuthology.orchestra.run.trial068.stdout:osd.2: {} 2026-02-23T13:10:55.714 INFO:teuthology.orchestra.run.trial068.stderr:osd.2: osd_enable_op_tracker = '' 2026-02-23T13:10:55.727 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:10:55.727 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:10:55.760 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:10:55.761 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:10:56.463 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 72 128 297683 297555 32.2842 34.5391 0.00844107 0.0309664 2026-02-23T13:10:56.567 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph daemon osd.0 dump_ops_in_flight 2026-02-23T13:10:56.590 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:10:56 trial068 ceph-mon[38805]: 3.e scrub ok 2026-02-23T13:10:56.590 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:10:56 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3410975966' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:10:56.701 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:10:56.837 INFO:teuthology.orchestra.run.trial068.stderr:no valid command found; 10 closest matches: 2026-02-23T13:10:56.837 INFO:teuthology.orchestra.run.trial068.stderr:0 2026-02-23T13:10:56.838 INFO:teuthology.orchestra.run.trial068.stderr:1 2026-02-23T13:10:56.838 INFO:teuthology.orchestra.run.trial068.stderr:2 2026-02-23T13:10:56.838 INFO:teuthology.orchestra.run.trial068.stderr:abort 2026-02-23T13:10:56.838 INFO:teuthology.orchestra.run.trial068.stderr:assert 2026-02-23T13:10:56.838 INFO:teuthology.orchestra.run.trial068.stderr:bluefs debug_inject_read_zeros 2026-02-23T13:10:56.838 INFO:teuthology.orchestra.run.trial068.stderr:bluefs files list 2026-02-23T13:10:56.838 INFO:teuthology.orchestra.run.trial068.stderr:bluefs stats 2026-02-23T13:10:56.838 INFO:teuthology.orchestra.run.trial068.stderr:bluestore allocator dump block 2026-02-23T13:10:56.838 INFO:teuthology.orchestra.run.trial068.stderr:bluestore allocator fragmentation block 2026-02-23T13:10:56.839 INFO:teuthology.orchestra.run.trial068.stderr:admin_socket: invalid command 2026-02-23T13:10:56.873 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-23T13:10:56.873 INFO:tasks.cephadm.ceph_manager.ceph:waiting on admin_socket for osd-0, ['dump_ops_in_flight'] 2026-02-23T13:10:57.463 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 73 128 301869 301741 32.2899 32.7031 0.0108537 0.0309564 2026-02-23T13:10:57.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:10:57 trial068 ceph-mon[38805]: pgmap v185: 17 pgs: 14 active+undersized+degraded, 3 active+clean; 2.1 GiB data, 7.7 GiB used, 2.7 TiB / 2.7 TiB avail; 28 MiB/s wr, 3.54k op/s; 28437/104580 objects degraded (27.192%) 2026-02-23T13:10:57.762 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:10:57.897 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:10:58.016 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:10:58.014+0000 7f6601774640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:10:58.016 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:10:58.014+0000 7f6601774640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:10:58.017 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.0 2026-02-23T13:10:58.029 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:10:58.029 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:10:58.042 INFO:teuthology.orchestra.run.trial068.stdout:osd.2: {} 2026-02-23T13:10:58.042 INFO:teuthology.orchestra.run.trial068.stderr:osd.2: osd_enable_op_tracker = '' 2026-02-23T13:10:58.053 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:10:58.054 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:10:58.087 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:10:58.088 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:10:58.463 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 74 128 306014 305886 32.2911 32.3828 0.0155562 0.030959 2026-02-23T13:10:58.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:10:58 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/4096442164' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:10:59.463 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 75 128 310222 310094 32.2989 32.875 0.0146663 0.0309487 2026-02-23T13:10:59.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:10:59 trial068 ceph-mon[38805]: pgmap v186: 17 pgs: 14 active+undersized+degraded, 3 active+clean; 2.3 GiB data, 8.1 GiB used, 2.7 TiB / 2.7 TiB avail; 42 MiB/s wr, 5.32k op/s; 30617/112587 objects degraded (27.194%) 2026-02-23T13:10:59.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:10:59 trial068 ceph-mon[38805]: Health check update: Degraded data redundancy: 30617/112587 objects degraded (27.194%), 14 pgs degraded (PG_DEGRADED) 2026-02-23T13:10:59.594 INFO:journalctl@ceph.osd.0.trial068.stdout:Feb 23 13:10:59 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-0[92334]: 2026-02-23T13:10:59.527+0000 7f5100ea8780 -1 osd.0 99 log_to_monitors true 2026-02-23T13:11:00.089 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:11:00.218 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:11:00.333 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:11:00.331+0000 7ff16e9e7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:11:00.333 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:11:00.331+0000 7ff16e9e7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:11:00.336 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.0 2026-02-23T13:11:00.349 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:11:00.349 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:11:00.361 INFO:teuthology.orchestra.run.trial068.stdout:osd.2: {} 2026-02-23T13:11:00.361 INFO:teuthology.orchestra.run.trial068.stderr:osd.2: osd_enable_op_tracker = '' 2026-02-23T13:11:00.371 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:11:00.372 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:11:00.372 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:11:00 trial068 ceph-mon[38805]: from='osd.0 [v2:10.20.193.68:6802/3423688218,v1:10.20.193.68:6803/3423688218]' entity='osd.0' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]} : dispatch 2026-02-23T13:11:00.372 INFO:journalctl@ceph.osd.0.trial068.stdout:Feb 23 13:11:00 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-0[92334]: 2026-02-23T13:11:00.334+0000 7f50f8cf1640 -1 osd.0 99 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2026-02-23T13:11:00.408 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:11:00.409 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:11:00.463 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 76 128 314486 314358 32.3122 33.3125 0.0215241 0.0309349 2026-02-23T13:11:01.464 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 77 128 318180 318052 32.2673 28.8594 0.0595058 0.0309364 2026-02-23T13:11:01.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:11:01 trial068 ceph-mon[38805]: pgmap v187: 17 pgs: 14 active+undersized+degraded, 3 active+clean; 2.3 GiB data, 8.1 GiB used, 2.7 TiB / 2.7 TiB avail; 28 MiB/s wr, 3.62k op/s; 30617/112587 objects degraded (27.194%) 2026-02-23T13:11:01.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:11:01 trial068 ceph-mon[38805]: from='osd.0 [v2:10.20.193.68:6802/3423688218,v1:10.20.193.68:6803/3423688218]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2026-02-23T13:11:01.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:11:01 trial068 ceph-mon[38805]: osdmap e105: 4 total, 3 up, 4 in 2026-02-23T13:11:01.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:11:01 trial068 ceph-mon[38805]: from='osd.0 [v2:10.20.193.68:6802/3423688218,v1:10.20.193.68:6803/3423688218]' entity='osd.0' cmd={"prefix": "osd crush create-or-move", "id": 0, "weight":0.6836, "args": ["host=trial068", "root=default"]} : dispatch 2026-02-23T13:11:01.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:11:01 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/693361348' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:11:01.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:11:01 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T13:11:01.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:11:01 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T13:11:01.874 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph daemon osd.0 dump_ops_in_flight 2026-02-23T13:11:01.993 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:11:02.106 INFO:teuthology.orchestra.run.trial068.stdout:{ 2026-02-23T13:11:02.106 INFO:teuthology.orchestra.run.trial068.stdout: "ops": [ 2026-02-23T13:11:02.106 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:11:02.106 INFO:teuthology.orchestra.run.trial068.stdout: "description": "osd_op(client.14522.0:318153 3.6 3.542230c6 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e106)", 2026-02-23T13:11:02.106 INFO:teuthology.orchestra.run.trial068.stdout: "initiated_at": "2026-02-23T13:11:01.351035+0000", 2026-02-23T13:11:02.106 INFO:teuthology.orchestra.run.trial068.stdout: "age": 0.75311954000000003, 2026-02-23T13:11:02.106 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 0.753135781, 2026-02-23T13:11:02.107 INFO:teuthology.orchestra.run.trial068.stdout: "continuous": false, 2026-02-23T13:11:02.107 INFO:teuthology.orchestra.run.trial068.stdout: "type_data": { 2026-02-23T13:11:02.107 INFO:teuthology.orchestra.run.trial068.stdout: "flag_point": "waiting for peered", 2026-02-23T13:11:02.107 INFO:teuthology.orchestra.run.trial068.stdout: "client_info": { 2026-02-23T13:11:02.107 INFO:teuthology.orchestra.run.trial068.stdout: "client": "client.14522", 2026-02-23T13:11:02.107 INFO:teuthology.orchestra.run.trial068.stdout: "client_addr": "10.20.193.68:0/2327563370", 2026-02-23T13:11:02.107 INFO:teuthology.orchestra.run.trial068.stdout: "tid": 318153 2026-02-23T13:11:02.107 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:11:02.107 INFO:teuthology.orchestra.run.trial068.stdout: "events": [ 2026-02-23T13:11:02.108 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:11:02.108 INFO:teuthology.orchestra.run.trial068.stdout: "event": "initiated", 2026-02-23T13:11:02.108 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:11:01.351035+0000", 2026-02-23T13:11:02.108 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 0 2026-02-23T13:11:02.108 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:11:02.108 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:11:02.108 INFO:teuthology.orchestra.run.trial068.stdout: "event": "header_read", 2026-02-23T13:11:02.108 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:11:01.351035+0000", 2026-02-23T13:11:02.108 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 0 2026-02-23T13:11:02.108 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:11:02.108 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:11:02.109 INFO:teuthology.orchestra.run.trial068.stdout: "event": "throttled", 2026-02-23T13:11:02.109 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:11:01.351042+0000", 2026-02-23T13:11:02.109 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 6.9929999999999998e-06 2026-02-23T13:11:02.109 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:11:02.109 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:11:02.109 INFO:teuthology.orchestra.run.trial068.stdout: "event": "all_read", 2026-02-23T13:11:02.109 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:11:01.351066+0000", 2026-02-23T13:11:02.109 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 2.3493999999999999e-05 2026-02-23T13:11:02.109 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:11:02.109 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:11:02.110 INFO:teuthology.orchestra.run.trial068.stdout: "event": "dispatched", 2026-02-23T13:11:02.110 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:11:01.351071+0000", 2026-02-23T13:11:02.110 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 5.6400000000000002e-06 2026-02-23T13:11:02.110 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:11:02.110 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:11:02.110 INFO:teuthology.orchestra.run.trial068.stdout: "event": "queued_for_pg", 2026-02-23T13:11:02.110 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:11:01.351080+0000", 2026-02-23T13:11:02.110 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 9.0469999999999999e-06 2026-02-23T13:11:02.110 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:11:02.110 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:11:02.111 INFO:teuthology.orchestra.run.trial068.stdout: "event": "reached_pg", 2026-02-23T13:11:02.111 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:11:01.351113+0000", 2026-02-23T13:11:02.111 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 3.2651999999999998e-05 2026-02-23T13:11:02.111 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:11:02.111 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:11:02.111 INFO:teuthology.orchestra.run.trial068.stdout: "event": "waiting for peered", 2026-02-23T13:11:02.111 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:11:01.351121+0000", 2026-02-23T13:11:02.111 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 8.225e-06 2026-02-23T13:11:02.111 INFO:teuthology.orchestra.run.trial068.stdout: } 2026-02-23T13:11:02.111 INFO:teuthology.orchestra.run.trial068.stdout: ] 2026-02-23T13:11:02.112 INFO:teuthology.orchestra.run.trial068.stdout: } 2026-02-23T13:11:02.112 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:11:02.112 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:11:02.112 INFO:teuthology.orchestra.run.trial068.stdout: "description": "osd_op(client.14522.0:318154 3.6 3.542230c6 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e106)", 2026-02-23T13:11:02.112 INFO:teuthology.orchestra.run.trial068.stdout: "initiated_at": "2026-02-23T13:11:01.351097+0000", 2026-02-23T13:11:02.112 INFO:teuthology.orchestra.run.trial068.stdout: "age": 0.75305784499999995, 2026-02-23T13:11:02.112 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 0.75309859199999996, 2026-02-23T13:11:02.112 INFO:teuthology.orchestra.run.trial068.stdout: "continuous": false, 2026-02-23T13:11:02.112 INFO:teuthology.orchestra.run.trial068.stdout: "type_data": { 2026-02-23T13:11:02.112 INFO:teuthology.orchestra.run.trial068.stdout: "flag_point": "waiting for peered", 2026-02-23T13:11:02.113 INFO:teuthology.orchestra.run.trial068.stdout: "client_info": { 2026-02-23T13:11:02.113 INFO:teuthology.orchestra.run.trial068.stdout: "client": "client.14522", 2026-02-23T13:11:02.113 INFO:teuthology.orchestra.run.trial068.stdout: "client_addr": "10.20.193.68:0/2327563370", 2026-02-23T13:11:02.113 INFO:teuthology.orchestra.run.trial068.stdout: "tid": 318154 2026-02-23T13:11:02.113 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:11:02.113 INFO:teuthology.orchestra.run.trial068.stdout: "events": [ 2026-02-23T13:11:02.113 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:11:02.113 INFO:teuthology.orchestra.run.trial068.stdout: "event": "initiated", 2026-02-23T13:11:02.113 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:11:01.351097+0000", 2026-02-23T13:11:02.113 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 0 2026-02-23T13:11:02.114 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:11:02.114 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:11:02.114 INFO:teuthology.orchestra.run.trial068.stdout: "event": "header_read", 2026-02-23T13:11:02.114 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:11:01.351097+0000", 2026-02-23T13:11:02.114 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 0 2026-02-23T13:11:02.114 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:11:02.114 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:11:02.114 INFO:teuthology.orchestra.run.trial068.stdout: "event": "throttled", 2026-02-23T13:11:02.114 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:11:01.351104+0000", 2026-02-23T13:11:02.114 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 7.1439999999999997e-06 2026-02-23T13:11:02.115 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:11:02.115 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:11:02.115 INFO:teuthology.orchestra.run.trial068.stdout: "event": "all_read", 2026-02-23T13:11:02.115 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:11:01.351135+0000", 2026-02-23T13:11:02.115 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 3.1259e-05 2026-02-23T13:11:02.115 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:11:02.115 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:11:02.115 INFO:teuthology.orchestra.run.trial068.stdout: "event": "dispatched", 2026-02-23T13:11:02.115 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:11:01.351139+0000", 2026-02-23T13:11:02.115 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 3.5260000000000002e-06 2026-02-23T13:11:02.116 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:11:02.116 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:11:02.116 INFO:teuthology.orchestra.run.trial068.stdout: "event": "queued_for_pg", 2026-02-23T13:11:02.116 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:11:01.351140+0000", 2026-02-23T13:11:02.116 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 1.3430000000000001e-06 2026-02-23T13:11:02.116 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:11:02.116 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:11:02.116 INFO:teuthology.orchestra.run.trial068.stdout: "event": "reached_pg", 2026-02-23T13:11:02.116 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:11:01.351158+0000", 2026-02-23T13:11:02.116 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 1.7733000000000001e-05 2026-02-23T13:11:02.116 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:11:02.117 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:11:02.117 INFO:teuthology.orchestra.run.trial068.stdout: "event": "waiting for peered", 2026-02-23T13:11:02.117 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:11:01.351160+0000", 2026-02-23T13:11:02.117 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 2.294e-06 2026-02-23T13:11:02.117 INFO:teuthology.orchestra.run.trial068.stdout: } 2026-02-23T13:11:02.117 INFO:teuthology.orchestra.run.trial068.stdout: ] 2026-02-23T13:11:02.117 INFO:teuthology.orchestra.run.trial068.stdout: } 2026-02-23T13:11:02.117 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:11:02.117 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:11:02.117 INFO:teuthology.orchestra.run.trial068.stdout: "description": "osd_op(client.14522.0:318155 3.6 3.542230c6 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e106)", 2026-02-23T13:11:02.117 INFO:teuthology.orchestra.run.trial068.stdout: "initiated_at": "2026-02-23T13:11:01.352691+0000", 2026-02-23T13:11:02.118 INFO:teuthology.orchestra.run.trial068.stdout: "age": 0.75146330900000002, 2026-02-23T13:11:02.118 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 0.75152386299999996, 2026-02-23T13:11:02.118 INFO:teuthology.orchestra.run.trial068.stdout: "continuous": false, 2026-02-23T13:11:02.118 INFO:teuthology.orchestra.run.trial068.stdout: "type_data": { 2026-02-23T13:11:02.118 INFO:teuthology.orchestra.run.trial068.stdout: "flag_point": "waiting for peered", 2026-02-23T13:11:02.118 INFO:teuthology.orchestra.run.trial068.stdout: "client_info": { 2026-02-23T13:11:02.118 INFO:teuthology.orchestra.run.trial068.stdout: "client": "client.14522", 2026-02-23T13:11:02.118 INFO:teuthology.orchestra.run.trial068.stdout: "client_addr": "10.20.193.68:0/2327563370", 2026-02-23T13:11:02.118 INFO:teuthology.orchestra.run.trial068.stdout: "tid": 318155 2026-02-23T13:11:02.119 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:11:02.119 INFO:teuthology.orchestra.run.trial068.stdout: "events": [ 2026-02-23T13:11:02.119 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:11:02.119 INFO:teuthology.orchestra.run.trial068.stdout: "event": "initiated", 2026-02-23T13:11:02.119 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:11:01.352691+0000", 2026-02-23T13:11:02.119 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 0 2026-02-23T13:11:02.119 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:11:02.119 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:11:02.119 INFO:teuthology.orchestra.run.trial068.stdout: "event": "header_read", 2026-02-23T13:11:02.119 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:11:01.352691+0000", 2026-02-23T13:11:02.119 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 0 2026-02-23T13:11:02.120 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:11:02.120 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:11:02.120 INFO:teuthology.orchestra.run.trial068.stdout: "event": "throttled", 2026-02-23T13:11:02.120 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:11:01.352698+0000", 2026-02-23T13:11:02.120 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 6.472e-06 2026-02-23T13:11:02.120 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:11:02.120 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:11:02.120 INFO:teuthology.orchestra.run.trial068.stdout: "event": "all_read", 2026-02-23T13:11:02.120 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:11:01.352725+0000", 2026-02-23T13:11:02.120 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 2.6809999999999999e-05 2026-02-23T13:11:02.121 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:11:02.121 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:11:02.121 INFO:teuthology.orchestra.run.trial068.stdout: "event": "dispatched", 2026-02-23T13:11:02.121 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:11:01.352729+0000", 2026-02-23T13:11:02.121 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 4.4689999999999999e-06 2026-02-23T13:11:02.121 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:11:02.121 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:11:02.121 INFO:teuthology.orchestra.run.trial068.stdout: "event": "queued_for_pg", 2026-02-23T13:11:02.121 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:11:01.352731+0000", 2026-02-23T13:11:02.121 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 1.4020000000000001e-06 2026-02-23T13:11:02.121 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:11:02.122 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:11:02.122 INFO:teuthology.orchestra.run.trial068.stdout: "event": "reached_pg", 2026-02-23T13:11:02.122 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:11:01.352749+0000", 2026-02-23T13:11:02.122 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 1.8365000000000001e-05 2026-02-23T13:11:02.122 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:11:02.122 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:11:02.122 INFO:teuthology.orchestra.run.trial068.stdout: "event": "waiting for peered", 2026-02-23T13:11:02.122 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:11:01.352752+0000", 2026-02-23T13:11:02.122 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 2.6249999999999999e-06 2026-02-23T13:11:02.122 INFO:teuthology.orchestra.run.trial068.stdout: } 2026-02-23T13:11:02.123 INFO:teuthology.orchestra.run.trial068.stdout: ] 2026-02-23T13:11:02.123 INFO:teuthology.orchestra.run.trial068.stdout: } 2026-02-23T13:11:02.123 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:11:02.123 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:11:02.123 INFO:teuthology.orchestra.run.trial068.stdout: "description": "osd_op(client.14522.0:318156 3.6 3.542230c6 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e106)", 2026-02-23T13:11:02.123 INFO:teuthology.orchestra.run.trial068.stdout: "initiated_at": "2026-02-23T13:11:01.353059+0000", 2026-02-23T13:11:02.123 INFO:teuthology.orchestra.run.trial068.stdout: "age": 0.75109605899999998, 2026-02-23T13:11:02.123 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 0.75117228199999997, 2026-02-23T13:11:02.123 INFO:teuthology.orchestra.run.trial068.stdout: "continuous": false, 2026-02-23T13:11:02.123 INFO:teuthology.orchestra.run.trial068.stdout: "type_data": { 2026-02-23T13:11:02.123 INFO:teuthology.orchestra.run.trial068.stdout: "flag_point": "waiting for peered", 2026-02-23T13:11:02.124 INFO:teuthology.orchestra.run.trial068.stdout: "client_info": { 2026-02-23T13:11:02.124 INFO:teuthology.orchestra.run.trial068.stdout: "client": "client.14522", 2026-02-23T13:11:02.124 INFO:teuthology.orchestra.run.trial068.stdout: "client_addr": "10.20.193.68:0/2327563370", 2026-02-23T13:11:02.124 INFO:teuthology.orchestra.run.trial068.stdout: "tid": 318156 2026-02-23T13:11:02.124 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:11:02.124 INFO:teuthology.orchestra.run.trial068.stdout: "events": [ 2026-02-23T13:11:02.124 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:11:02.124 INFO:teuthology.orchestra.run.trial068.stdout: "event": "initiated", 2026-02-23T13:11:02.124 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:11:01.353059+0000", 2026-02-23T13:11:02.124 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 0 2026-02-23T13:11:02.125 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:11:02.125 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:11:02.125 INFO:teuthology.orchestra.run.trial068.stdout: "event": "header_read", 2026-02-23T13:11:02.125 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:11:01.353059+0000", 2026-02-23T13:11:02.125 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 0 2026-02-23T13:11:02.125 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:11:02.125 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:11:02.126 INFO:teuthology.orchestra.run.trial068.stdout: "event": "throttled", 2026-02-23T13:11:02.127 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:11:01.353070+0000", 2026-02-23T13:11:02.127 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 1.1532000000000001e-05 2026-02-23T13:11:02.127 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:11:02.127 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:11:02.127 INFO:teuthology.orchestra.run.trial068.stdout: "event": "all_read", 2026-02-23T13:11:02.127 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:11:01.353097+0000", 2026-02-23T13:11:02.127 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 2.6308999999999998e-05 2026-02-23T13:11:02.127 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:11:02.127 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:11:02.128 INFO:teuthology.orchestra.run.trial068.stdout: "event": "dispatched", 2026-02-23T13:11:02.128 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:11:01.353101+0000", 2026-02-23T13:11:02.128 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 4.9490000000000004e-06 2026-02-23T13:11:02.128 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:11:02.128 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:11:02.128 INFO:teuthology.orchestra.run.trial068.stdout: "event": "queued_for_pg", 2026-02-23T13:11:02.128 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:11:01.353103+0000", 2026-02-23T13:11:02.129 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 1.7129999999999999e-06 2026-02-23T13:11:02.129 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:11:02.129 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:11:02.129 INFO:teuthology.orchestra.run.trial068.stdout: "event": "reached_pg", 2026-02-23T13:11:02.129 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:11:01.353127+0000", 2026-02-23T13:11:02.129 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 2.3515000000000001e-05 2026-02-23T13:11:02.129 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:11:02.129 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:11:02.130 INFO:teuthology.orchestra.run.trial068.stdout: "event": "waiting for peered", 2026-02-23T13:11:02.130 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:11:01.353132+0000", 2026-02-23T13:11:02.130 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 4.8389999999999997e-06 2026-02-23T13:11:02.130 INFO:teuthology.orchestra.run.trial068.stdout: } 2026-02-23T13:11:02.130 INFO:teuthology.orchestra.run.trial068.stdout: ] 2026-02-23T13:11:02.130 INFO:teuthology.orchestra.run.trial068.stdout: } 2026-02-23T13:11:02.130 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:11:02.130 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:11:02.130 INFO:teuthology.orchestra.run.trial068.stdout: "description": "osd_op(client.14522.0:318157 3.6 3.542230c6 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e106)", 2026-02-23T13:11:02.131 INFO:teuthology.orchestra.run.trial068.stdout: "initiated_at": "2026-02-23T13:11:01.356025+0000", 2026-02-23T13:11:02.131 INFO:teuthology.orchestra.run.trial068.stdout: "age": 0.74812976600000003, 2026-02-23T13:11:02.131 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 0.74822230999999995, 2026-02-23T13:11:02.131 INFO:teuthology.orchestra.run.trial068.stdout: "continuous": false, 2026-02-23T13:11:02.131 INFO:teuthology.orchestra.run.trial068.stdout: "type_data": { 2026-02-23T13:11:02.131 INFO:teuthology.orchestra.run.trial068.stdout: "flag_point": "waiting for peered", 2026-02-23T13:11:02.131 INFO:teuthology.orchestra.run.trial068.stdout: "client_info": { 2026-02-23T13:11:02.131 INFO:teuthology.orchestra.run.trial068.stdout: "client": "client.14522", 2026-02-23T13:11:02.131 INFO:teuthology.orchestra.run.trial068.stdout: "client_addr": "10.20.193.68:0/2327563370", 2026-02-23T13:11:02.131 INFO:teuthology.orchestra.run.trial068.stdout: "tid": 318157 2026-02-23T13:11:02.131 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:11:02.132 INFO:teuthology.orchestra.run.trial068.stdout: "events": [ 2026-02-23T13:11:02.132 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:11:02.132 INFO:teuthology.orchestra.run.trial068.stdout: "event": "initiated", 2026-02-23T13:11:02.132 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:11:01.356025+0000", 2026-02-23T13:11:02.132 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 0 2026-02-23T13:11:02.132 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:11:02.132 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:11:02.132 INFO:teuthology.orchestra.run.trial068.stdout: "event": "header_read", 2026-02-23T13:11:02.132 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:11:01.356025+0000", 2026-02-23T13:11:02.132 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 0 2026-02-23T13:11:02.133 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:11:02.133 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:11:02.133 INFO:teuthology.orchestra.run.trial068.stdout: "event": "throttled", 2026-02-23T13:11:02.133 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:11:01.356034+0000", 2026-02-23T13:11:02.133 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 8.9060000000000008e-06 2026-02-23T13:11:02.133 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:11:02.133 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:11:02.133 INFO:teuthology.orchestra.run.trial068.stdout: "event": "all_read", 2026-02-23T13:11:02.133 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:11:01.356061+0000", 2026-02-23T13:11:02.133 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 2.7200999999999999e-05 2026-02-23T13:11:02.134 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:11:02.134 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:11:02.134 INFO:teuthology.orchestra.run.trial068.stdout: "event": "dispatched", 2026-02-23T13:11:02.134 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:11:01.356067+0000", 2026-02-23T13:11:02.134 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 5.4210000000000003e-06 2026-02-23T13:11:02.134 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:11:02.134 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:11:02.134 INFO:teuthology.orchestra.run.trial068.stdout: "event": "queued_for_pg", 2026-02-23T13:11:02.134 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:11:01.356068+0000", 2026-02-23T13:11:02.134 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 1.9429999999999999e-06 2026-02-23T13:11:02.135 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:11:02.135 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:11:02.135 INFO:teuthology.orchestra.run.trial068.stdout: "event": "reached_pg", 2026-02-23T13:11:02.135 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:11:01.356099+0000", 2026-02-23T13:11:02.135 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 3.0346999999999999e-05 2026-02-23T13:11:02.135 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:11:02.135 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:11:02.135 INFO:teuthology.orchestra.run.trial068.stdout: "event": "waiting for peered", 2026-02-23T13:11:02.135 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:11:01.356102+0000", 2026-02-23T13:11:02.135 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 2.8150000000000002e-06 2026-02-23T13:11:02.135 INFO:teuthology.orchestra.run.trial068.stdout: } 2026-02-23T13:11:02.136 INFO:teuthology.orchestra.run.trial068.stdout: ] 2026-02-23T13:11:02.136 INFO:teuthology.orchestra.run.trial068.stdout: } 2026-02-23T13:11:02.136 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:11:02.136 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:11:02.136 INFO:teuthology.orchestra.run.trial068.stdout: "description": "osd_op(client.14522.0:318158 3.6 3.542230c6 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e106)", 2026-02-23T13:11:02.136 INFO:teuthology.orchestra.run.trial068.stdout: "initiated_at": "2026-02-23T13:11:01.356078+0000", 2026-02-23T13:11:02.136 INFO:teuthology.orchestra.run.trial068.stdout: "age": 0.74807656700000003, 2026-02-23T13:11:02.136 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 0.74818463999999996, 2026-02-23T13:11:02.136 INFO:teuthology.orchestra.run.trial068.stdout: "continuous": false, 2026-02-23T13:11:02.136 INFO:teuthology.orchestra.run.trial068.stdout: "type_data": { 2026-02-23T13:11:02.136 INFO:teuthology.orchestra.run.trial068.stdout: "flag_point": "waiting for peered", 2026-02-23T13:11:02.137 INFO:teuthology.orchestra.run.trial068.stdout: "client_info": { 2026-02-23T13:11:02.137 INFO:teuthology.orchestra.run.trial068.stdout: "client": "client.14522", 2026-02-23T13:11:02.137 INFO:teuthology.orchestra.run.trial068.stdout: "client_addr": "10.20.193.68:0/2327563370", 2026-02-23T13:11:02.137 INFO:teuthology.orchestra.run.trial068.stdout: "tid": 318158 2026-02-23T13:11:02.137 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:11:02.137 INFO:teuthology.orchestra.run.trial068.stdout: "events": [ 2026-02-23T13:11:02.137 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:11:02.137 INFO:teuthology.orchestra.run.trial068.stdout: "event": "initiated", 2026-02-23T13:11:02.137 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:11:01.356078+0000", 2026-02-23T13:11:02.137 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 0 2026-02-23T13:11:02.137 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:11:02.138 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:11:02.138 INFO:teuthology.orchestra.run.trial068.stdout: "event": "header_read", 2026-02-23T13:11:02.138 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:11:01.356078+0000", 2026-02-23T13:11:02.138 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 0 2026-02-23T13:11:02.138 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:11:02.138 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:11:02.138 INFO:teuthology.orchestra.run.trial068.stdout: "event": "throttled", 2026-02-23T13:11:02.138 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:11:01.356084+0000", 2026-02-23T13:11:02.138 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 6.0020000000000003e-06 2026-02-23T13:11:02.138 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:11:02.138 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:11:02.138 INFO:teuthology.orchestra.run.trial068.stdout: "event": "all_read", 2026-02-23T13:11:02.139 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:11:01.356103+0000", 2026-02-23T13:11:02.139 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 1.8423999999999998e-05 2026-02-23T13:11:02.139 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:11:02.139 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:11:02.139 INFO:teuthology.orchestra.run.trial068.stdout: "event": "dispatched", 2026-02-23T13:11:02.139 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:11:01.356107+0000", 2026-02-23T13:11:02.139 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 3.968e-06 2026-02-23T13:11:02.139 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:11:02.139 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:11:02.139 INFO:teuthology.orchestra.run.trial068.stdout: "event": "queued_for_pg", 2026-02-23T13:11:02.139 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:11:01.356108+0000", 2026-02-23T13:11:02.140 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 1.252e-06 2026-02-23T13:11:02.140 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:11:02.140 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:11:02.140 INFO:teuthology.orchestra.run.trial068.stdout: "event": "reached_pg", 2026-02-23T13:11:02.140 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:11:01.356128+0000", 2026-02-23T13:11:02.140 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 1.9958000000000001e-05 2026-02-23T13:11:02.140 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:11:02.140 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:11:02.140 INFO:teuthology.orchestra.run.trial068.stdout: "event": "waiting for peered", 2026-02-23T13:11:02.140 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:11:01.356131+0000", 2026-02-23T13:11:02.140 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 2.8550000000000002e-06 2026-02-23T13:11:02.140 INFO:teuthology.orchestra.run.trial068.stdout: } 2026-02-23T13:11:02.140 INFO:teuthology.orchestra.run.trial068.stdout: ] 2026-02-23T13:11:02.140 INFO:teuthology.orchestra.run.trial068.stdout: } 2026-02-23T13:11:02.141 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:11:02.141 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:11:02.141 INFO:teuthology.orchestra.run.trial068.stdout: "description": "osd_op(client.14522.0:318159 3.6 3.542230c6 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e106)", 2026-02-23T13:11:02.141 INFO:teuthology.orchestra.run.trial068.stdout: "initiated_at": "2026-02-23T13:11:01.356118+0000", 2026-02-23T13:11:02.141 INFO:teuthology.orchestra.run.trial068.stdout: "age": 0.74803650099999996, 2026-02-23T13:11:02.141 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 0.74815969299999996, 2026-02-23T13:11:02.141 INFO:teuthology.orchestra.run.trial068.stdout: "continuous": false, 2026-02-23T13:11:02.141 INFO:teuthology.orchestra.run.trial068.stdout: "type_data": { 2026-02-23T13:11:02.141 INFO:teuthology.orchestra.run.trial068.stdout: "flag_point": "waiting for peered", 2026-02-23T13:11:02.141 INFO:teuthology.orchestra.run.trial068.stdout: "client_info": { 2026-02-23T13:11:02.141 INFO:teuthology.orchestra.run.trial068.stdout: "client": "client.14522", 2026-02-23T13:11:02.141 INFO:teuthology.orchestra.run.trial068.stdout: "client_addr": "10.20.193.68:0/2327563370", 2026-02-23T13:11:02.141 INFO:teuthology.orchestra.run.trial068.stdout: "tid": 318159 2026-02-23T13:11:02.142 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:11:02.142 INFO:teuthology.orchestra.run.trial068.stdout: "events": [ 2026-02-23T13:11:02.142 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:11:02.142 INFO:teuthology.orchestra.run.trial068.stdout: "event": "initiated", 2026-02-23T13:11:02.142 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:11:01.356118+0000", 2026-02-23T13:11:02.142 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 0 2026-02-23T13:11:02.142 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:11:02.142 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:11:02.142 INFO:teuthology.orchestra.run.trial068.stdout: "event": "header_read", 2026-02-23T13:11:02.142 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:11:01.356118+0000", 2026-02-23T13:11:02.142 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 0 2026-02-23T13:11:02.142 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:11:02.142 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:11:02.143 INFO:teuthology.orchestra.run.trial068.stdout: "event": "throttled", 2026-02-23T13:11:02.143 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:11:01.356130+0000", 2026-02-23T13:11:02.143 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 1.1671999999999999e-05 2026-02-23T13:11:02.143 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:11:02.143 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:11:02.143 INFO:teuthology.orchestra.run.trial068.stdout: "event": "all_read", 2026-02-23T13:11:02.143 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:11:01.356152+0000", 2026-02-23T13:11:02.143 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 2.2481999999999999e-05 2026-02-23T13:11:02.143 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:11:02.143 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:11:02.143 INFO:teuthology.orchestra.run.trial068.stdout: "event": "dispatched", 2026-02-23T13:11:02.143 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:11:01.356155+0000", 2026-02-23T13:11:02.143 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 2.7049999999999999e-06 2026-02-23T13:11:02.143 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:11:02.144 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:11:02.144 INFO:teuthology.orchestra.run.trial068.stdout: "event": "queued_for_pg", 2026-02-23T13:11:02.144 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:11:01.356156+0000", 2026-02-23T13:11:02.144 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 9.6200000000000006e-07 2026-02-23T13:11:02.144 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:11:02.144 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:11:02.144 INFO:teuthology.orchestra.run.trial068.stdout: "event": "reached_pg", 2026-02-23T13:11:02.144 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:11:01.356168+0000", 2026-02-23T13:11:02.144 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 1.2373e-05 2026-02-23T13:11:02.144 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:11:02.144 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:11:02.144 INFO:teuthology.orchestra.run.trial068.stdout: "event": "waiting for peered", 2026-02-23T13:11:02.144 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:11:01.356170+0000", 2026-02-23T13:11:02.144 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 1.703e-06 2026-02-23T13:11:02.145 INFO:teuthology.orchestra.run.trial068.stdout: } 2026-02-23T13:11:02.145 INFO:teuthology.orchestra.run.trial068.stdout: ] 2026-02-23T13:11:02.145 INFO:teuthology.orchestra.run.trial068.stdout: } 2026-02-23T13:11:02.145 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:11:02.145 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:11:02.145 INFO:teuthology.orchestra.run.trial068.stdout: "description": "osd_op(client.14522.0:318160 3.6 3.542230c6 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e106)", 2026-02-23T13:11:02.145 INFO:teuthology.orchestra.run.trial068.stdout: "initiated_at": "2026-02-23T13:11:01.357840+0000", 2026-02-23T13:11:02.145 INFO:teuthology.orchestra.run.trial068.stdout: "age": 0.74631424599999996, 2026-02-23T13:11:02.145 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 0.74645367799999995, 2026-02-23T13:11:02.145 INFO:teuthology.orchestra.run.trial068.stdout: "continuous": false, 2026-02-23T13:11:02.145 INFO:teuthology.orchestra.run.trial068.stdout: "type_data": { 2026-02-23T13:11:02.145 INFO:teuthology.orchestra.run.trial068.stdout: "flag_point": "waiting for peered", 2026-02-23T13:11:02.145 INFO:teuthology.orchestra.run.trial068.stdout: "client_info": { 2026-02-23T13:11:02.145 INFO:teuthology.orchestra.run.trial068.stdout: "client": "client.14522", 2026-02-23T13:11:02.145 INFO:teuthology.orchestra.run.trial068.stdout: "client_addr": "10.20.193.68:0/2327563370", 2026-02-23T13:11:02.146 INFO:teuthology.orchestra.run.trial068.stdout: "tid": 318160 2026-02-23T13:11:02.146 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:11:02.146 INFO:teuthology.orchestra.run.trial068.stdout: "events": [ 2026-02-23T13:11:02.146 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:11:02.146 INFO:teuthology.orchestra.run.trial068.stdout: "event": "initiated", 2026-02-23T13:11:02.146 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:11:01.357840+0000", 2026-02-23T13:11:02.146 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 0 2026-02-23T13:11:02.146 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:11:02.146 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:11:02.146 INFO:teuthology.orchestra.run.trial068.stdout: "event": "header_read", 2026-02-23T13:11:02.146 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:11:01.357840+0000", 2026-02-23T13:11:02.146 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 0 2026-02-23T13:11:02.146 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:11:02.146 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:11:02.147 INFO:teuthology.orchestra.run.trial068.stdout: "event": "throttled", 2026-02-23T13:11:02.147 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:11:01.357847+0000", 2026-02-23T13:11:02.147 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 6.1619999999999996e-06 2026-02-23T13:11:02.147 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:11:02.147 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:11:02.147 INFO:teuthology.orchestra.run.trial068.stdout: "event": "all_read", 2026-02-23T13:11:02.147 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:11:01.357867+0000", 2026-02-23T13:11:02.147 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 2.0618000000000001e-05 2026-02-23T13:11:02.147 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:11:02.147 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:11:02.147 INFO:teuthology.orchestra.run.trial068.stdout: "event": "dispatched", 2026-02-23T13:11:02.147 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:11:01.357871+0000", 2026-02-23T13:11:02.147 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 3.737e-06 2026-02-23T13:11:02.147 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:11:02.147 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:11:02.147 INFO:teuthology.orchestra.run.trial068.stdout: "event": "queued_for_pg", 2026-02-23T13:11:02.147 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:11:01.357876+0000", 2026-02-23T13:11:02.148 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 4.87e-06 2026-02-23T13:11:02.148 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:11:02.148 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:11:02.148 INFO:teuthology.orchestra.run.trial068.stdout: "event": "reached_pg", 2026-02-23T13:11:02.148 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:11:01.357898+0000", 2026-02-23T13:11:02.148 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 2.2021000000000001e-05 2026-02-23T13:11:02.148 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:11:02.148 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:11:02.148 INFO:teuthology.orchestra.run.trial068.stdout: "event": "waiting for peered", 2026-02-23T13:11:02.148 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:11:01.357901+0000", 2026-02-23T13:11:02.148 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 2.8650000000000001e-06 2026-02-23T13:11:02.148 INFO:teuthology.orchestra.run.trial068.stdout: } 2026-02-23T13:11:02.148 INFO:teuthology.orchestra.run.trial068.stdout: ] 2026-02-23T13:11:02.148 INFO:teuthology.orchestra.run.trial068.stdout: } 2026-02-23T13:11:02.148 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:11:02.148 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:11:02.149 INFO:teuthology.orchestra.run.trial068.stdout: "description": "osd_op(client.14522.0:318177 3.6 3.782bbe76 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e106)", 2026-02-23T13:11:02.149 INFO:teuthology.orchestra.run.trial068.stdout: "initiated_at": "2026-02-23T13:11:01.364075+0000", 2026-02-23T13:11:02.149 INFO:teuthology.orchestra.run.trial068.stdout: "age": 0.74007973299999996, 2026-02-23T13:11:02.149 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 0.740234905, 2026-02-23T13:11:02.149 INFO:teuthology.orchestra.run.trial068.stdout: "continuous": false, 2026-02-23T13:11:02.149 INFO:teuthology.orchestra.run.trial068.stdout: "type_data": { 2026-02-23T13:11:02.149 INFO:teuthology.orchestra.run.trial068.stdout: "flag_point": "waiting for peered", 2026-02-23T13:11:02.149 INFO:teuthology.orchestra.run.trial068.stdout: "client_info": { 2026-02-23T13:11:02.149 INFO:teuthology.orchestra.run.trial068.stdout: "client": "client.14522", 2026-02-23T13:11:02.149 INFO:teuthology.orchestra.run.trial068.stdout: "client_addr": "10.20.193.68:0/2327563370", 2026-02-23T13:11:02.149 INFO:teuthology.orchestra.run.trial068.stdout: "tid": 318177 2026-02-23T13:11:02.149 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:11:02.149 INFO:teuthology.orchestra.run.trial068.stdout: "events": [ 2026-02-23T13:11:02.149 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:11:02.149 INFO:teuthology.orchestra.run.trial068.stdout: "event": "initiated", 2026-02-23T13:11:02.149 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:11:01.364075+0000", 2026-02-23T13:11:02.150 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 0 2026-02-23T13:11:02.150 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:11:02.150 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:11:02.150 INFO:teuthology.orchestra.run.trial068.stdout: "event": "header_read", 2026-02-23T13:11:02.150 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:11:01.364075+0000", 2026-02-23T13:11:02.150 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 0 2026-02-23T13:11:02.150 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:11:02.150 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:11:02.150 INFO:teuthology.orchestra.run.trial068.stdout: "event": "throttled", 2026-02-23T13:11:02.150 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:11:01.364081+0000", 2026-02-23T13:11:02.150 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 6.2009999999999996e-06 2026-02-23T13:11:02.150 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:11:02.150 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:11:02.150 INFO:teuthology.orchestra.run.trial068.stdout: "event": "all_read", 2026-02-23T13:11:02.150 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:11:01.364108+0000", 2026-02-23T13:11:02.150 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 2.7030999999999999e-05 2026-02-23T13:11:02.150 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:11:02.151 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:11:02.151 INFO:teuthology.orchestra.run.trial068.stdout: "event": "dispatched", 2026-02-23T13:11:02.151 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:11:01.364113+0000", 2026-02-23T13:11:02.151 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 4.6589999999999998e-06 2026-02-23T13:11:02.151 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:11:02.151 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:11:02.151 INFO:teuthology.orchestra.run.trial068.stdout: "event": "queued_for_pg", 2026-02-23T13:11:02.151 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:11:01.364115+0000", 2026-02-23T13:11:02.151 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 2.525e-06 2026-02-23T13:11:02.151 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:11:02.151 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:11:02.151 INFO:teuthology.orchestra.run.trial068.stdout: "event": "reached_pg", 2026-02-23T13:11:02.151 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:11:01.364145+0000", 2026-02-23T13:11:02.151 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 2.9745999999999999e-05 2026-02-23T13:11:02.151 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:11:02.151 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:11:02.151 INFO:teuthology.orchestra.run.trial068.stdout: "event": "waiting for peered", 2026-02-23T13:11:02.152 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:11:01.364150+0000", 2026-02-23T13:11:02.152 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 4.7480000000000001e-06 2026-02-23T13:11:02.152 INFO:teuthology.orchestra.run.trial068.stdout: } 2026-02-23T13:11:02.152 INFO:teuthology.orchestra.run.trial068.stdout: ] 2026-02-23T13:11:02.152 INFO:teuthology.orchestra.run.trial068.stdout: } 2026-02-23T13:11:02.152 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:11:02.152 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:11:02.152 INFO:teuthology.orchestra.run.trial068.stdout: "description": "osd_op(client.14522.0:318178 3.6 3.782bbe76 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e106)", 2026-02-23T13:11:02.152 INFO:teuthology.orchestra.run.trial068.stdout: "initiated_at": "2026-02-23T13:11:01.364135+0000", 2026-02-23T13:11:02.152 INFO:teuthology.orchestra.run.trial068.stdout: "age": 0.74001954000000003, 2026-02-23T13:11:02.152 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 0.74018987000000003, 2026-02-23T13:11:02.152 INFO:teuthology.orchestra.run.trial068.stdout: "continuous": false, 2026-02-23T13:11:02.152 INFO:teuthology.orchestra.run.trial068.stdout: "type_data": { 2026-02-23T13:11:02.152 INFO:teuthology.orchestra.run.trial068.stdout: "flag_point": "waiting for peered", 2026-02-23T13:11:02.152 INFO:teuthology.orchestra.run.trial068.stdout: "client_info": { 2026-02-23T13:11:02.152 INFO:teuthology.orchestra.run.trial068.stdout: "client": "client.14522", 2026-02-23T13:11:02.153 INFO:teuthology.orchestra.run.trial068.stdout: "client_addr": "10.20.193.68:0/2327563370", 2026-02-23T13:11:02.153 INFO:teuthology.orchestra.run.trial068.stdout: "tid": 318178 2026-02-23T13:11:02.153 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:11:02.153 INFO:teuthology.orchestra.run.trial068.stdout: "events": [ 2026-02-23T13:11:02.153 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:11:02.153 INFO:teuthology.orchestra.run.trial068.stdout: "event": "initiated", 2026-02-23T13:11:02.153 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:11:01.364135+0000", 2026-02-23T13:11:02.153 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 0 2026-02-23T13:11:02.153 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:11:02.153 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:11:02.153 INFO:teuthology.orchestra.run.trial068.stdout: "event": "header_read", 2026-02-23T13:11:02.153 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:11:01.364135+0000", 2026-02-23T13:11:02.153 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 0 2026-02-23T13:11:02.153 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:11:02.153 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:11:02.153 INFO:teuthology.orchestra.run.trial068.stdout: "event": "throttled", 2026-02-23T13:11:02.153 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:11:01.364144+0000", 2026-02-23T13:11:02.154 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 9.1169999999999993e-06 2026-02-23T13:11:02.154 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:11:02.154 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:11:02.154 INFO:teuthology.orchestra.run.trial068.stdout: "event": "all_read", 2026-02-23T13:11:02.154 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:11:01.364171+0000", 2026-02-23T13:11:02.154 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 2.7041000000000001e-05 2026-02-23T13:11:02.154 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:11:02.154 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:11:02.154 INFO:teuthology.orchestra.run.trial068.stdout: "event": "dispatched", 2026-02-23T13:11:02.154 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:11:01.364175+0000", 2026-02-23T13:11:02.154 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 3.4960000000000001e-06 2026-02-23T13:11:02.154 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:11:02.154 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:11:02.154 INFO:teuthology.orchestra.run.trial068.stdout: "event": "queued_for_pg", 2026-02-23T13:11:02.154 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:11:01.364176+0000", 2026-02-23T13:11:02.154 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 1.5030000000000001e-06 2026-02-23T13:11:02.154 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:11:02.154 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:11:02.155 INFO:teuthology.orchestra.run.trial068.stdout: "event": "reached_pg", 2026-02-23T13:11:02.155 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:11:01.364197+0000", 2026-02-23T13:11:02.155 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 2.0438e-05 2026-02-23T13:11:02.155 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:11:02.155 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:11:02.155 INFO:teuthology.orchestra.run.trial068.stdout: "event": "waiting for peered", 2026-02-23T13:11:02.155 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:11:01.364199+0000", 2026-02-23T13:11:02.155 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 2.1849999999999998e-06 2026-02-23T13:11:02.155 INFO:teuthology.orchestra.run.trial068.stdout: } 2026-02-23T13:11:02.155 INFO:teuthology.orchestra.run.trial068.stdout: ] 2026-02-23T13:11:02.155 INFO:teuthology.orchestra.run.trial068.stdout: } 2026-02-23T13:11:02.155 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:11:02.155 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:11:02.155 INFO:teuthology.orchestra.run.trial068.stdout: "description": "osd_op(client.14522.0:318179 3.6 3.782bbe76 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e106)", 2026-02-23T13:11:02.155 INFO:teuthology.orchestra.run.trial068.stdout: "initiated_at": "2026-02-23T13:11:01.364205+0000", 2026-02-23T13:11:02.155 INFO:teuthology.orchestra.run.trial068.stdout: "age": 0.73995016000000002, 2026-02-23T13:11:02.155 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 0.74013570900000003, 2026-02-23T13:11:02.156 INFO:teuthology.orchestra.run.trial068.stdout: "continuous": false, 2026-02-23T13:11:02.156 INFO:teuthology.orchestra.run.trial068.stdout: "type_data": { 2026-02-23T13:11:02.156 INFO:teuthology.orchestra.run.trial068.stdout: "flag_point": "waiting for peered", 2026-02-23T13:11:02.156 INFO:teuthology.orchestra.run.trial068.stdout: "client_info": { 2026-02-23T13:11:02.156 INFO:teuthology.orchestra.run.trial068.stdout: "client": "client.14522", 2026-02-23T13:11:02.156 INFO:teuthology.orchestra.run.trial068.stdout: "client_addr": "10.20.193.68:0/2327563370", 2026-02-23T13:11:02.156 INFO:teuthology.orchestra.run.trial068.stdout: "tid": 318179 2026-02-23T13:11:02.156 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:11:02.156 INFO:teuthology.orchestra.run.trial068.stdout: "events": [ 2026-02-23T13:11:02.156 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:11:02.156 INFO:teuthology.orchestra.run.trial068.stdout: "event": "initiated", 2026-02-23T13:11:02.156 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:11:01.364205+0000", 2026-02-23T13:11:02.156 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 0 2026-02-23T13:11:02.156 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:11:02.156 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:11:02.156 INFO:teuthology.orchestra.run.trial068.stdout: "event": "header_read", 2026-02-23T13:11:02.156 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:11:01.364205+0000", 2026-02-23T13:11:02.157 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 0 2026-02-23T13:11:02.157 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:11:02.157 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:11:02.157 INFO:teuthology.orchestra.run.trial068.stdout: "event": "throttled", 2026-02-23T13:11:02.157 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:11:01.364210+0000", 2026-02-23T13:11:02.157 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 5.891e-06 2026-02-23T13:11:02.157 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:11:02.157 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:11:02.157 INFO:teuthology.orchestra.run.trial068.stdout: "event": "all_read", 2026-02-23T13:11:02.157 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:11:01.364238+0000", 2026-02-23T13:11:02.157 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 2.7240999999999999e-05 2026-02-23T13:11:02.157 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:11:02.157 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:11:02.157 INFO:teuthology.orchestra.run.trial068.stdout: "event": "dispatched", 2026-02-23T13:11:02.157 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:11:01.364242+0000", 2026-02-23T13:11:02.157 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 4.318e-06 2026-02-23T13:11:02.157 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:11:02.157 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:11:02.157 INFO:teuthology.orchestra.run.trial068.stdout: "event": "queued_for_pg", 2026-02-23T13:11:02.158 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:11:01.364243+0000", 2026-02-23T13:11:02.158 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 1.113e-06 2026-02-23T13:11:02.158 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:11:02.158 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:11:02.158 INFO:teuthology.orchestra.run.trial068.stdout: "event": "reached_pg", 2026-02-23T13:11:02.158 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:11:01.364266+0000", 2026-02-23T13:11:02.158 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 2.2492000000000001e-05 2026-02-23T13:11:02.158 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:11:02.158 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:11:02.158 INFO:teuthology.orchestra.run.trial068.stdout: "event": "waiting for peered", 2026-02-23T13:11:02.158 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:11:01.364268+0000", 2026-02-23T13:11:02.158 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 1.883e-06 2026-02-23T13:11:02.158 INFO:teuthology.orchestra.run.trial068.stdout: } 2026-02-23T13:11:02.158 INFO:teuthology.orchestra.run.trial068.stdout: ] 2026-02-23T13:11:02.158 INFO:teuthology.orchestra.run.trial068.stdout: } 2026-02-23T13:11:02.158 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:11:02.158 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:11:02.158 INFO:teuthology.orchestra.run.trial068.stdout: "description": "osd_op(client.14522.0:318180 3.6 3.782bbe76 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e106)", 2026-02-23T13:11:02.159 INFO:teuthology.orchestra.run.trial068.stdout: "initiated_at": "2026-02-23T13:11:01.364578+0000", 2026-02-23T13:11:02.159 INFO:teuthology.orchestra.run.trial068.stdout: "age": 0.73957653800000001, 2026-02-23T13:11:02.159 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 0.739777095, 2026-02-23T13:11:02.159 INFO:teuthology.orchestra.run.trial068.stdout: "continuous": false, 2026-02-23T13:11:02.159 INFO:teuthology.orchestra.run.trial068.stdout: "type_data": { 2026-02-23T13:11:02.159 INFO:teuthology.orchestra.run.trial068.stdout: "flag_point": "waiting for peered", 2026-02-23T13:11:02.159 INFO:teuthology.orchestra.run.trial068.stdout: "client_info": { 2026-02-23T13:11:02.159 INFO:teuthology.orchestra.run.trial068.stdout: "client": "client.14522", 2026-02-23T13:11:02.159 INFO:teuthology.orchestra.run.trial068.stdout: "client_addr": "10.20.193.68:0/2327563370", 2026-02-23T13:11:02.159 INFO:teuthology.orchestra.run.trial068.stdout: "tid": 318180 2026-02-23T13:11:02.159 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:11:02.159 INFO:teuthology.orchestra.run.trial068.stdout: "events": [ 2026-02-23T13:11:02.159 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:11:02.159 INFO:teuthology.orchestra.run.trial068.stdout: "event": "initiated", 2026-02-23T13:11:02.159 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:11:01.364578+0000", 2026-02-23T13:11:02.159 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 0 2026-02-23T13:11:02.159 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:11:02.159 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:11:02.159 INFO:teuthology.orchestra.run.trial068.stdout: "event": "header_read", 2026-02-23T13:11:02.160 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:11:01.364578+0000", 2026-02-23T13:11:02.160 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 0 2026-02-23T13:11:02.160 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:11:02.160 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:11:02.160 INFO:teuthology.orchestra.run.trial068.stdout: "event": "throttled", 2026-02-23T13:11:02.160 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:11:01.364588+0000", 2026-02-23T13:11:02.160 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 9.6479999999999998e-06 2026-02-23T13:11:02.160 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:11:02.160 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:11:02.160 INFO:teuthology.orchestra.run.trial068.stdout: "event": "all_read", 2026-02-23T13:11:02.160 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:11:01.364610+0000", 2026-02-23T13:11:02.160 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 2.2201999999999999e-05 2026-02-23T13:11:02.160 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:11:02.160 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:11:02.160 INFO:teuthology.orchestra.run.trial068.stdout: "event": "dispatched", 2026-02-23T13:11:02.160 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:11:01.364614+0000", 2026-02-23T13:11:02.160 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 3.907e-06 2026-02-23T13:11:02.160 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:11:02.160 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:11:02.160 INFO:teuthology.orchestra.run.trial068.stdout: "event": "queued_for_pg", 2026-02-23T13:11:02.161 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:11:01.364616+0000", 2026-02-23T13:11:02.161 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 1.753e-06 2026-02-23T13:11:02.161 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:11:02.161 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:11:02.161 INFO:teuthology.orchestra.run.trial068.stdout: "event": "reached_pg", 2026-02-23T13:11:02.161 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:11:01.364634+0000", 2026-02-23T13:11:02.161 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 1.8134e-05 2026-02-23T13:11:02.161 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:11:02.161 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:11:02.161 INFO:teuthology.orchestra.run.trial068.stdout: "event": "waiting for peered", 2026-02-23T13:11:02.161 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:11:01.364637+0000", 2026-02-23T13:11:02.161 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 3.4170000000000001e-06 2026-02-23T13:11:02.161 INFO:teuthology.orchestra.run.trial068.stdout: } 2026-02-23T13:11:02.161 INFO:teuthology.orchestra.run.trial068.stdout: ] 2026-02-23T13:11:02.161 INFO:teuthology.orchestra.run.trial068.stdout: } 2026-02-23T13:11:02.161 INFO:teuthology.orchestra.run.trial068.stdout: } 2026-02-23T13:11:02.161 INFO:teuthology.orchestra.run.trial068.stdout: ], 2026-02-23T13:11:02.161 INFO:teuthology.orchestra.run.trial068.stdout: "num_ops": 12 2026-02-23T13:11:02.161 INFO:teuthology.orchestra.run.trial068.stdout:} 2026-02-23T13:11:02.410 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:11:02.464 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 78 128 318180 318052 31.8536 0 - 0.0309364 2026-02-23T13:11:02.526 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:11:02.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:11:02 trial068 ceph-mon[38805]: Health check cleared: OSD_DOWN (was: 1 osds down) 2026-02-23T13:11:02.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:11:02 trial068 ceph-mon[38805]: osd.0 [v2:10.20.193.68:6802/3423688218,v1:10.20.193.68:6803/3423688218] boot 2026-02-23T13:11:02.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:11:02 trial068 ceph-mon[38805]: osdmap e106: 4 total, 4 up, 4 in 2026-02-23T13:11:02.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:11:02 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2026-02-23T13:11:02.622 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:11:02.620+0000 7f88f92dd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:11:02.622 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:11:02.620+0000 7f88f92dd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:11:02.634 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:11:02.634 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:11:02.644 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:11:02.644 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:11:02.656 INFO:teuthology.orchestra.run.trial068.stdout:osd.2: {} 2026-02-23T13:11:02.656 INFO:teuthology.orchestra.run.trial068.stderr:osd.2: osd_enable_op_tracker = '' 2026-02-23T13:11:02.667 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:11:02.667 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:11:03.464 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 79 128 318180 318052 31.4504 0 - 0.0309364 2026-02-23T13:11:03.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:11:03 trial068 ceph-mon[38805]: pgmap v190: 17 pgs: 9 peering, 5 active+undersized+degraded, 3 active+clean; 2.4 GiB data, 7.7 GiB used, 2.7 TiB / 2.7 TiB avail; 32 MiB/s wr, 4.10k op/s; 11883/116886 objects degraded (10.166%) 2026-02-23T13:11:03.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:11:03 trial068 ceph-mon[38805]: osdmap e107: 4 total, 4 up, 4 in 2026-02-23T13:11:03.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:11:03 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/54687571' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:11:04.265 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:11:03 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:11:03.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:11:04.342 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:11:03 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:11:03] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:11:04.464 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T13:11:04.463812+0000 min lat: 0.000737354 max lat: 2.63923 avg lat: 0.0320066 2026-02-23T13:11:04.464 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T13:11:04.464 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 80 127 319850 319723 31.2205 4.35156 0.00854833 0.0320066 2026-02-23T13:11:04.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:11:04 trial068 ceph-mon[38805]: osdmap e108: 4 total, 4 up, 4 in 2026-02-23T13:11:04.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:11:04 trial068 ceph-mon[38805]: Health check update: Degraded data redundancy: 11883/116886 objects degraded (10.166%), 5 pgs degraded (PG_DEGRADED) 2026-02-23T13:11:04.703 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:11:04.861 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:11:04.990 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:11:04.988+0000 7f0a25bbe640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:11:04.990 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:11:04.988+0000 7f0a25bbe640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:11:05.004 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:11:05.004 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:11:05.016 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:11:05.017 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:11:05.029 INFO:teuthology.orchestra.run.trial068.stdout:osd.2: {} 2026-02-23T13:11:05.030 INFO:teuthology.orchestra.run.trial068.stderr:osd.2: osd_enable_op_tracker = '' 2026-02-23T13:11:05.042 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:11:05.042 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:11:05.464 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 81 128 322695 322567 31.1093 22.2188 0.109461 0.0321266 2026-02-23T13:11:05.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:11:05 trial068 ceph-mon[38805]: pgmap v193: 17 pgs: 3 remapped+peering, 9 peering, 2 active+undersized+degraded, 3 active+clean; 2.4 GiB data, 7.9 GiB used, 2.7 TiB / 2.7 TiB avail; 31 MiB/s wr, 3.93k op/s; 4807/118479 objects degraded (4.057%) 2026-02-23T13:11:05.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:11:05 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/843863327' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:11:06.464 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 82 128 325474 325346 30.9947 21.7109 0.00604732 0.0322456 2026-02-23T13:11:07.090 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:11:07.163 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [1, 3, 2, 0] 2026-02-23T13:11:07.164 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2026-02-23T13:11:07.164 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph pg dump --format=json 2026-02-23T13:11:07.238 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:11:07.313 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:11:07.370 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:11:07.368+0000 7ff8505c7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:11:07.371 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:11:07.369+0000 7ff8505c7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:11:07.371 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:11:07 trial068 ceph-mon[38805]: pgmap v194: 17 pgs: 3 remapped+peering, 9 peering, 2 active+undersized+degraded, 3 active+clean; 2.4 GiB data, 7.9 GiB used, 2.7 TiB / 2.7 TiB avail; 21 MiB/s wr, 2.65k op/s; 4807/118479 objects degraded (4.057%) 2026-02-23T13:11:07.386 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:11:07.387 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:11:07.400 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:11:07.401 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:11:07.417 INFO:teuthology.orchestra.run.trial068.stdout:osd.2: {} 2026-02-23T13:11:07.417 INFO:teuthology.orchestra.run.trial068.stderr:osd.2: osd_enable_op_tracker = '' 2026-02-23T13:11:07.432 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:11:07.432 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:11:07.451 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:11:07.449+0000 7f52592ee640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:11:07.451 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:11:07.450+0000 7f52592ee640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:11:07.464 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 83 127 328245 328118 30.8822 21.6562 0.10096 0.0323625 2026-02-23T13:11:07.607 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T13:11:07.608 INFO:teuthology.orchestra.run.trial068.stderr:dumped all 2026-02-23T13:11:07.655 INFO:teuthology.orchestra.run.trial068.stdout:{"pg_ready":true,"pg_map":{"version":195,"stamp":"2026-02-23T13:11:07.346042+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2657444384,"num_objects":40545,"num_object_clones":0,"num_object_copies":121635,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7190,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40545,"num_whiteouts":0,"num_read":122,"num_read_kb":228,"num_write":324458,"num_write_kb":2596734,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17184,"num_bytes_recovered":1126122016,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160722,"ondisk_log_size":160722,"up":51,"acting":38,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":51,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":2936012800,"kb_used":8395052,"kb_used_data":7055452,"kb_used_omap":83,"kb_used_meta":1339436,"kb_avail":2927617748,"statfs":{"total":3006477107200,"available":2997880573952,"internally_reserved":0,"allocated":7224782848,"data_stored":7221759413,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":85775,"internal_metadata":1371582705},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,9,5,13,17,11,16,15],"upper_bound":256},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":197574656,"num_objects":3016,"num_object_clones":0,"num_object_copies":9048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-23427,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3016,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":24118,"num_write_kb":192944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1091,"num_bytes_recovered":71426048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.014653"},"pg_stats":[{"pgid":"3.f","version":"105'19740","reported_seq":22568,"reported_epoch":107,"state":"remapped+peering","last_fresh":"2026-02-23T13:11:02.337475+0000","last_change":"2026-02-23T13:11:02.337475+0000","last_active":"2026-02-23T13:11:01.335328+0000","last_peered":"2026-02-23T13:11:01.303307+0000","last_clean":"2026-02-23T13:09:42.363073+0000","last_became_active":"2026-02-23T13:10:34.986135+0000","last_became_peered":"2026-02-23T13:10:34.986135+0000","last_unstale":"2026-02-23T13:11:02.337475+0000","last_undegraded":"2026-02-23T13:11:02.337475+0000","last_fullsized":"2026-02-23T13:11:02.337475+0000","mapping_epoch":107,"log_start":"90'9700","ondisk_log_start":"90'9700","created":76,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:09:42.363073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:09:42.363073+0000","last_clean_scrub_stamp":"2026-02-23T13:09:42.363073+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":161710080,"num_objects":2468,"num_object_clones":0,"num_object_copies":7404,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2468,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19740,"num_write_kb":157920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1380,"num_bytes_recovered":90357760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,3],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.e","version":"106'19344","reported_seq":21623,"reported_epoch":107,"state":"active+clean","last_fresh":"2026-02-23T13:11:02.336784+0000","last_change":"2026-02-23T13:10:55.337830+0000","last_active":"2026-02-23T13:11:02.336784+0000","last_peered":"2026-02-23T13:11:02.336784+0000","last_clean":"2026-02-23T13:11:02.336784+0000","last_became_active":"2026-02-23T13:10:35.981714+0000","last_became_peered":"2026-02-23T13:10:35.981714+0000","last_unstale":"2026-02-23T13:11:02.336784+0000","last_undegraded":"2026-02-23T13:11:02.336784+0000","last_fullsized":"2026-02-23T13:11:02.336784+0000","mapping_epoch":101,"log_start":"90'9300","ondisk_log_start":"90'9300","created":76,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"104'17744","last_scrub_stamp":"2026-02-23T13:10:55.337787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:09:42.363073+0000","last_clean_scrub_stamp":"2026-02-23T13:10:55.337787+0000","objects_scrubbed":2214,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T13:11:57.088624+0000","scrub_duration":547,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":158466048,"num_objects":2418,"num_object_clones":0,"num_object_copies":7254,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2418,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19344,"num_write_kb":154752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1223,"num_bytes_recovered":80150528,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"108'20520","reported_seq":23028,"reported_epoch":108,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2026-02-23T13:11:06.516440+0000","last_change":"2026-02-23T13:11:03.804832+0000","last_active":"2026-02-23T13:11:06.516440+0000","last_peered":"2026-02-23T13:11:06.516440+0000","last_clean":"2026-02-23T13:10:32.383237+0000","last_became_active":"2026-02-23T13:11:03.802082+0000","last_became_peered":"2026-02-23T13:11:03.802082+0000","last_unstale":"2026-02-23T13:11:06.516440+0000","last_undegraded":"2026-02-23T13:11:03.372220+0000","last_fullsized":"2026-02-23T13:11:03.338314+0000","mapping_epoch":107,"log_start":"90'10500","ondisk_log_start":"90'10500","created":76,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:09:42.363073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:09:42.363073+0000","last_clean_scrub_stamp":"2026-02-23T13:09:42.363073+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":168099840,"num_objects":2565,"num_object_clones":0,"num_object_copies":7695,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":858,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2565,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20520,"num_write_kb":164160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1391,"num_bytes_recovered":91062272,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,2],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":1707}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.c","version":"108'19876","reported_seq":21114,"reported_epoch":108,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2026-02-23T13:11:06.537008+0000","last_change":"2026-02-23T13:11:03.791244+0000","last_active":"2026-02-23T13:11:06.537008+0000","last_peered":"2026-02-23T13:11:06.537008+0000","last_clean":"2026-02-23T13:10:32.288357+0000","last_became_active":"2026-02-23T13:11:03.788839+0000","last_became_peered":"2026-02-23T13:11:03.788839+0000","last_unstale":"2026-02-23T13:11:06.537008+0000","last_undegraded":"2026-02-23T13:11:03.361201+0000","last_fullsized":"2026-02-23T13:11:03.338304+0000","mapping_epoch":107,"log_start":"90'9800","ondisk_log_start":"90'9800","created":76,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:09:42.363073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:09:42.363073+0000","last_clean_scrub_stamp":"2026-02-23T13:09:42.363073+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":162824192,"num_objects":2485,"num_object_clones":0,"num_object_copies":7455,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":831,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2485,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19876,"num_write_kb":159008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":610,"num_bytes_recovered":39976960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,1],"avail_no_missing":["3","1"],"object_location_counts":[{"shards":"1,3","objects":1654}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.b","version":"108'20507","reported_seq":22119,"reported_epoch":108,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2026-02-23T13:11:07.216139+0000","last_change":"2026-02-23T13:11:03.807438+0000","last_active":"2026-02-23T13:11:07.216139+0000","last_peered":"2026-02-23T13:11:07.216139+0000","last_clean":"2026-02-23T13:09:58.726181+0000","last_became_active":"2026-02-23T13:11:03.803230+0000","last_became_peered":"2026-02-23T13:11:03.803230+0000","last_unstale":"2026-02-23T13:11:07.216139+0000","last_undegraded":"2026-02-23T13:11:03.387398+0000","last_fullsized":"2026-02-23T13:11:03.340910+0000","mapping_epoch":107,"log_start":"90'10500","ondisk_log_start":"90'10500","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:09:42.363073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:09:42.363073+0000","last_clean_scrub_stamp":"2026-02-23T13:09:42.363073+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":167993344,"num_objects":2564,"num_object_clones":0,"num_object_copies":7692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":828,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2564,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20507,"num_write_kb":164056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":764,"num_bytes_recovered":50069504,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[{"shards":"1,2","objects":1736}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"108'20184","reported_seq":23072,"reported_epoch":108,"state":"active+clean","last_fresh":"2026-02-23T13:11:06.532123+0000","last_change":"2026-02-23T13:10:47.945257+0000","last_active":"2026-02-23T13:11:06.532123+0000","last_peered":"2026-02-23T13:11:06.532123+0000","last_clean":"2026-02-23T13:11:06.532123+0000","last_became_active":"2026-02-23T13:10:33.983588+0000","last_became_peered":"2026-02-23T13:10:33.983588+0000","last_unstale":"2026-02-23T13:11:06.532123+0000","last_undegraded":"2026-02-23T13:11:06.532123+0000","last_fullsized":"2026-02-23T13:11:06.532123+0000","mapping_epoch":99,"log_start":"90'10100","ondisk_log_start":"90'10100","created":76,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"104'16184","last_scrub_stamp":"2026-02-23T13:10:47.945219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:09:42.363073+0000","last_clean_scrub_stamp":"2026-02-23T13:10:47.945219+0000","objects_scrubbed":2011,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T13:11:49.868548+0000","scrub_duration":909,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":165347328,"num_objects":2523,"num_object_clones":0,"num_object_copies":7569,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2523,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20184,"num_write_kb":161472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1383,"num_bytes_recovered":90578944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.9","version":"108'20584","reported_seq":24134,"reported_epoch":108,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2026-02-23T13:11:06.537462+0000","last_change":"2026-02-23T13:11:03.800504+0000","last_active":"2026-02-23T13:11:06.537462+0000","last_peered":"2026-02-23T13:11:06.537462+0000","last_clean":"2026-02-23T13:10:32.407346+0000","last_became_active":"2026-02-23T13:11:03.788181+0000","last_became_peered":"2026-02-23T13:11:03.788181+0000","last_unstale":"2026-02-23T13:11:06.537462+0000","last_undegraded":"2026-02-23T13:11:03.370472+0000","last_fullsized":"2026-02-23T13:11:03.338089+0000","mapping_epoch":107,"log_start":"90'10500","ondisk_log_start":"90'10500","created":76,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:09:42.363073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:09:42.363073+0000","last_clean_scrub_stamp":"2026-02-23T13:09:42.363073+0000","objects_scrubbed":0,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":168624128,"num_objects":2573,"num_object_clones":0,"num_object_copies":7719,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":194,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2573,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20584,"num_write_kb":164672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2083,"num_bytes_recovered":136404992,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,2],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":2379}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"108'20632","reported_seq":22236,"reported_epoch":108,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2026-02-23T13:11:07.184837+0000","last_change":"2026-02-23T13:11:03.795488+0000","last_active":"2026-02-23T13:11:07.184837+0000","last_peered":"2026-02-23T13:11:07.184837+0000","last_clean":"2026-02-23T13:09:58.743502+0000","last_became_active":"2026-02-23T13:11:03.782836+0000","last_became_peered":"2026-02-23T13:11:03.782836+0000","last_unstale":"2026-02-23T13:11:07.184837+0000","last_undegraded":"2026-02-23T13:11:03.374124+0000","last_fullsized":"2026-02-23T13:11:03.340834+0000","mapping_epoch":107,"log_start":"90'10600","ondisk_log_start":"90'10600","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:09:42.363073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:09:42.363073+0000","last_clean_scrub_stamp":"2026-02-23T13:09:42.363073+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":169017344,"num_objects":2579,"num_object_clones":0,"num_object_copies":7737,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":835,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2579,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20632,"num_write_kb":165056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":757,"num_bytes_recovered":49610752,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[{"shards":"1,2","objects":1744}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"108'20312","reported_seq":21585,"reported_epoch":108,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2026-02-23T13:11:07.187968+0000","last_change":"2026-02-23T13:11:03.792375+0000","last_active":"2026-02-23T13:11:07.187968+0000","last_peered":"2026-02-23T13:11:07.187968+0000","last_clean":"2026-02-23T13:10:32.272970+0000","last_became_active":"2026-02-23T13:11:03.778183+0000","last_became_peered":"2026-02-23T13:11:03.778183+0000","last_unstale":"2026-02-23T13:11:07.187968+0000","last_undegraded":"2026-02-23T13:11:03.371361+0000","last_fullsized":"2026-02-23T13:11:03.340781+0000","mapping_epoch":107,"log_start":"90'10300","ondisk_log_start":"90'10300","created":76,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:09:42.363073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:09:42.363073+0000","last_clean_scrub_stamp":"2026-02-23T13:09:42.363073+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":166395904,"num_objects":2539,"num_object_clones":0,"num_object_copies":7617,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":794,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2539,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20312,"num_write_kb":162496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":591,"num_bytes_recovered":38731776,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[{"shards":"1,3","objects":1745}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.6","version":"108'20929","reported_seq":22525,"reported_epoch":108,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2026-02-23T13:11:07.190297+0000","last_change":"2026-02-23T13:11:03.811837+0000","last_active":"2026-02-23T13:11:07.190297+0000","last_peered":"2026-02-23T13:11:07.190297+0000","last_clean":"2026-02-23T13:09:58.717061+0000","last_became_active":"2026-02-23T13:11:03.803413+0000","last_became_peered":"2026-02-23T13:11:03.803413+0000","last_unstale":"2026-02-23T13:11:07.190297+0000","last_undegraded":"2026-02-23T13:11:03.373253+0000","last_fullsized":"2026-02-23T13:11:03.341107+0000","mapping_epoch":107,"log_start":"90'10900","ondisk_log_start":"90'10900","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:09:42.363073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:09:42.363073+0000","last_clean_scrub_stamp":"2026-02-23T13:09:42.363073+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":171450368,"num_objects":2617,"num_object_clones":0,"num_object_copies":7851,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":826,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2617,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20929,"num_write_kb":167432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":760,"num_bytes_recovered":49725440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[{"shards":"1,2","objects":1791}],"blocked_by":[],"up_primary":0,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"105'19952","reported_seq":22013,"reported_epoch":107,"state":"remapped+peering","last_fresh":"2026-02-23T13:11:02.337484+0000","last_change":"2026-02-23T13:11:02.337484+0000","last_active":"2026-02-23T13:11:01.332451+0000","last_peered":"2026-02-23T13:11:01.325579+0000","last_clean":"2026-02-23T13:09:42.363073+0000","last_became_active":"2026-02-23T13:10:42.311522+0000","last_became_peered":"2026-02-23T13:10:42.311522+0000","last_unstale":"2026-02-23T13:11:02.337484+0000","last_undegraded":"2026-02-23T13:11:02.337484+0000","last_fullsized":"2026-02-23T13:11:02.337484+0000","mapping_epoch":107,"log_start":"90'9900","ondisk_log_start":"90'9900","created":76,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:09:42.363073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:09:42.363073+0000","last_clean_scrub_stamp":"2026-02-23T13:09:42.363073+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":163446784,"num_objects":2494,"num_object_clones":0,"num_object_copies":7482,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2494,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19952,"num_write_kb":159616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":988,"num_bytes_recovered":64749568,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"108'20216","reported_seq":22631,"reported_epoch":108,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2026-02-23T13:11:07.221752+0000","last_change":"2026-02-23T13:11:03.789981+0000","last_active":"2026-02-23T13:11:07.221752+0000","last_peered":"2026-02-23T13:11:07.221752+0000","last_clean":"2026-02-23T13:09:58.738693+0000","last_became_active":"2026-02-23T13:11:03.782329+0000","last_became_peered":"2026-02-23T13:11:03.782329+0000","last_unstale":"2026-02-23T13:11:07.221752+0000","last_undegraded":"2026-02-23T13:11:03.387773+0000","last_fullsized":"2026-02-23T13:11:03.341050+0000","mapping_epoch":107,"log_start":"90'10200","ondisk_log_start":"90'10200","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:09:42.363073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:09:42.363073+0000","last_clean_scrub_stamp":"2026-02-23T13:09:42.363073+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":165609472,"num_objects":2527,"num_object_clones":0,"num_object_copies":7581,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":371,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2527,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20216,"num_write_kb":161728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1162,"num_bytes_recovered":76152832,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[{"shards":"1,2","objects":2156}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"105'20056","reported_seq":22455,"reported_epoch":107,"state":"remapped+peering","last_fresh":"2026-02-23T13:11:02.337252+0000","last_change":"2026-02-23T13:11:02.337252+0000","last_active":"2026-02-23T13:11:01.333059+0000","last_peered":"2026-02-23T13:11:01.275449+0000","last_clean":"2026-02-23T13:09:42.363073+0000","last_became_active":"2026-02-23T13:10:33.982722+0000","last_became_peered":"2026-02-23T13:10:33.982722+0000","last_unstale":"2026-02-23T13:11:02.337252+0000","last_undegraded":"2026-02-23T13:11:02.337252+0000","last_fullsized":"2026-02-23T13:11:02.337252+0000","mapping_epoch":107,"log_start":"90'10000","ondisk_log_start":"90'10000","created":76,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:09:42.363073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:09:42.363073+0000","last_clean_scrub_stamp":"2026-02-23T13:09:42.363073+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":164298752,"num_objects":2507,"num_object_clones":0,"num_object_copies":7521,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2507,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20056,"num_write_kb":160448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1179,"num_bytes_recovered":77266944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"108'20720","reported_seq":22390,"reported_epoch":108,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2026-02-23T13:11:07.170402+0000","last_change":"2026-02-23T13:11:03.791701+0000","last_active":"2026-02-23T13:11:07.170402+0000","last_peered":"2026-02-23T13:11:07.170402+0000","last_clean":"2026-02-23T13:10:32.343256+0000","last_became_active":"2026-02-23T13:11:03.789175+0000","last_became_peered":"2026-02-23T13:11:03.789175+0000","last_unstale":"2026-02-23T13:11:07.170402+0000","last_undegraded":"2026-02-23T13:11:03.373352+0000","last_fullsized":"2026-02-23T13:11:03.340847+0000","mapping_epoch":107,"log_start":"90'10700","ondisk_log_start":"90'10700","created":76,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:09:42.363073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:09:42.363073+0000","last_clean_scrub_stamp":"2026-02-23T13:09:42.363073+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":169738240,"num_objects":2590,"num_object_clones":0,"num_object_copies":7770,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":853,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2590,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20720,"num_write_kb":165760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":789,"num_bytes_recovered":51675136,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[{"shards":"1,2","objects":1737}],"blocked_by":[],"up_primary":0,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"108'20440","reported_seq":22088,"reported_epoch":108,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2026-02-23T13:11:07.212732+0000","last_change":"2026-02-23T13:11:03.792292+0000","last_active":"2026-02-23T13:11:07.212732+0000","last_peered":"2026-02-23T13:11:07.212732+0000","last_clean":"2026-02-23T13:09:58.652894+0000","last_became_active":"2026-02-23T13:11:03.789042+0000","last_became_peered":"2026-02-23T13:11:03.789042+0000","last_unstale":"2026-02-23T13:11:07.212732+0000","last_undegraded":"2026-02-23T13:11:03.382995+0000","last_fullsized":"2026-02-23T13:11:03.340886+0000","mapping_epoch":107,"log_start":"90'10400","ondisk_log_start":"90'10400","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:09:42.363073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:09:42.363073+0000","last_clean_scrub_stamp":"2026-02-23T13:09:42.363073+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":167444480,"num_objects":2555,"num_object_clones":0,"num_object_copies":7665,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":800,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2555,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20440,"num_write_kb":163520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":780,"num_bytes_recovered":51118080,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[{"shards":"1,2","objects":1755}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"108'20311","reported_seq":23116,"reported_epoch":108,"state":"active+clean","last_fresh":"2026-02-23T13:11:07.211469+0000","last_change":"2026-02-23T13:10:46.187555+0000","last_active":"2026-02-23T13:11:07.211469+0000","last_peered":"2026-02-23T13:11:07.211469+0000","last_clean":"2026-02-23T13:11:07.211469+0000","last_became_active":"2026-02-23T13:10:35.982713+0000","last_became_peered":"2026-02-23T13:10:35.982713+0000","last_unstale":"2026-02-23T13:11:07.211469+0000","last_undegraded":"2026-02-23T13:11:07.211469+0000","last_fullsized":"2026-02-23T13:11:07.211469+0000","mapping_epoch":101,"log_start":"90'10300","ondisk_log_start":"90'10300","created":76,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"104'15600","last_scrub_stamp":"2026-02-23T13:10:46.187433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:09:42.363073+0000","last_clean_scrub_stamp":"2026-02-23T13:10:46.187433+0000","objects_scrubbed":1939,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T13:11:54.258632+0000","scrub_duration":1268,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":166387712,"num_objects":2539,"num_object_clones":0,"num_object_copies":7617,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2539,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20311,"num_write_kb":162488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1342,"num_bytes_recovered":87900160,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"28'99","reported_seq":195,"reported_epoch":108,"state":"active+clean","last_fresh":"2026-02-23T13:11:03.338053+0000","last_change":"2026-02-23T13:11:02.340041+0000","last_active":"2026-02-23T13:11:03.338053+0000","last_peered":"2026-02-23T13:11:03.338053+0000","last_clean":"2026-02-23T13:11:03.338053+0000","last_became_active":"2026-02-23T13:11:02.339822+0000","last_became_peered":"2026-02-23T13:11:02.339822+0000","last_unstale":"2026-02-23T13:11:03.338053+0000","last_undegraded":"2026-02-23T13:11:03.338053+0000","last_fullsized":"2026-02-23T13:11:03.338053+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'99","last_scrub_stamp":"2026-02-23T13:09:54.841853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:07:05.619208+0000","last_clean_scrub_stamp":"2026-02-23T13:09:54.841853+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T13:11:19.668596+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":122,"num_read_kb":228,"num_write":135,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":2656854016,"num_objects":40543,"num_object_clones":0,"num_object_copies":121629,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7190,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40543,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":324323,"num_write_kb":2594584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17182,"num_bytes_recovered":1125531648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":7275880448,"data_stored":7275880448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160623,"ondisk_log_size":160623,"up":48,"acting":35,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":122,"num_read_kb":228,"num_write":135,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":80,"seq":343597383696,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":734003200,"kb_used":1977248,"kb_used_data":1619340,"kb_used_omap":25,"kb_used_meta":357862,"kb_avail":732025952,"statfs":{"total":751619276800,"available":749594574848,"internally_reserved":0,"allocated":1658204160,"data_stored":1657443779,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":25955,"internal_metadata":366451357},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":92,"seq":395136991241,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":734003200,"kb_used":2567824,"kb_used_data":2232956,"kb_used_omap":15,"kb_used_meta":334832,"kb_avail":731435376,"statfs":{"total":751619276800,"available":748989825024,"internally_reserved":0,"allocated":2286546944,"data_stored":2285801100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15483,"internal_metadata":342868869},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":64,"seq":274877906967,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":734003200,"kb_used":2327596,"kb_used_data":1790808,"kb_used_omap":36,"kb_used_meta":536731,"kb_avail":731675604,"statfs":{"total":751619276800,"available":749235818496,"internally_reserved":0,"allocated":1833787392,"data_stored":1833030563,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":37333,"internal_metadata":549613099},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,9,5,13,17,11,16,15],"upper_bound":256},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":106,"seq":455266533378,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":734003200,"kb_used":1522384,"kb_used_data":1412348,"kb_used_omap":6,"kb_used_meta":110009,"kb_avail":732480816,"statfs":{"total":751619276800,"available":750060355584,"internally_reserved":0,"allocated":1446244352,"data_stored":1445483971,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7004,"internal_metadata":112649380},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1490477056,"data_stored":1490477056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1839325184,"data_stored":1839325184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":2284756992,"data_stored":2284756992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1661321216,"data_stored":1661321216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-23T13:11:07.657 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph pg force-backfill 3.d 3.1 3.0 2026-02-23T13:11:07.793 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:11:07.917 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:11:07.915+0000 7fd722ed6640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:11:07.917 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:11:07.915+0000 7fd722ed6640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:11:08.050 INFO:teuthology.orchestra.run.trial068.stderr:instructing pg(s) [3.0,3.1] on osd.1 to force-backfill; instructing pg(s) [3.d] on osd.3 to force-backfill; 2026-02-23T13:11:08.050 INFO:teuthology.orchestra.run.trial068.stderr: 2026-02-23T13:11:08.464 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 84 127 330235 330108 30.6996 15.5469 0.00931543 0.0325015 2026-02-23T13:11:08.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:11:08 trial068 ceph-mon[38805]: osdmap e109: 4 total, 4 up, 4 in 2026-02-23T13:11:08.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:11:08 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1167832419' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:11:09.464 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 85 128 332485 332357 30.5451 17.5703 0.502984 0.0326923 2026-02-23T13:11:09.480 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:11:09.622 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:11:09.648 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:11:09 trial068 ceph-mon[38805]: from='client.14600 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2026-02-23T13:11:09.648 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:11:09 trial068 ceph-mon[38805]: from='client.14602 -' entity='client.admin' cmd=[{"prefix": "pg force-backfill", "pgid": ["3.d", "3.1", "3.0"], "target": ["mon-mgr", ""]}]: dispatch 2026-02-23T13:11:09.648 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:11:09 trial068 ceph-mon[38805]: pgmap v196: 17 pgs: 1 peering, 2 active+recovering+undersized+degraded+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 4 active+clean; 2.5 GiB data, 8.1 GiB used, 2.7 TiB / 2.7 TiB avail; 20 MiB/s wr, 2.57k op/s; 8907/122676 objects degraded (7.261%); 20 MiB/s, 318 objects/s recovering 2026-02-23T13:11:09.648 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:11:09 trial068 ceph-mon[38805]: osdmap e110: 4 total, 4 up, 4 in 2026-02-23T13:11:09.648 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:11:09 trial068 ceph-mon[38805]: Health check update: Degraded data redundancy: 8907/122676 objects degraded (7.261%), 12 pgs degraded (PG_DEGRADED) 2026-02-23T13:11:09.746 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:11:09.744+0000 7efd93255640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:11:09.746 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:11:09.744+0000 7efd93255640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:11:09.761 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:11:09.761 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:11:09.775 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:11:09.775 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:11:09.788 INFO:teuthology.orchestra.run.trial068.stdout:osd.2: {} 2026-02-23T13:11:09.788 INFO:teuthology.orchestra.run.trial068.stderr:osd.2: osd_enable_op_tracker = '' 2026-02-23T13:11:09.801 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:11:09.801 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:11:10.464 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 86 128 334146 334018 30.3408 12.9766 0.0132298 0.0328182 2026-02-23T13:11:10.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:11:10 trial068 ceph-mon[38805]: osdmap e111: 4 total, 4 up, 4 in 2026-02-23T13:11:10.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:11:10 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1894178089' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:11:11.464 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 87 128 336082 335954 30.1659 15.125 0.0257262 0.0330773 2026-02-23T13:11:11.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:11:11 trial068 ceph-mon[38805]: pgmap v199: 17 pgs: 1 peering, 2 active+recovering+undersized+degraded+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 4 active+clean; 2.5 GiB data, 8.1 GiB used, 2.7 TiB / 2.7 TiB avail; 15 MiB/s wr, 1.86k op/s; 8907/122676 objects degraded (7.261%); 20 MiB/s, 318 objects/s recovering 2026-02-23T13:11:11.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:11:11 trial068 ceph-mon[38805]: osdmap e112: 4 total, 4 up, 4 in 2026-02-23T13:11:11.845 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:11:11.994 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:11:12.114 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:11:12.112+0000 7f1e74e7a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:11:12.114 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:11:12.112+0000 7f1e74e7a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:11:12.129 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:11:12.130 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:11:12.147 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:11:12.147 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:11:12.161 INFO:teuthology.orchestra.run.trial068.stdout:osd.2: {} 2026-02-23T13:11:12.161 INFO:teuthology.orchestra.run.trial068.stderr:osd.2: osd_enable_op_tracker = '' 2026-02-23T13:11:12.175 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:11:12.175 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:11:12.464 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 88 128 339051 338923 30.0867 23.1953 0.0220279 0.0332231 2026-02-23T13:11:12.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:11:12 trial068 ceph-mon[38805]: osdmap e113: 4 total, 4 up, 4 in 2026-02-23T13:11:12.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:11:12 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1120150867' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:11:13.093 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [1, 3, 2, 0] 2026-02-23T13:11:13.094 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2026-02-23T13:11:13.094 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph pg dump --format=json 2026-02-23T13:11:13.223 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:11:13.339 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:11:13.337+0000 7f67cf193640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:11:13.339 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:11:13.337+0000 7f67cf193640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:11:13.464 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 89 128 340972 340844 29.9172 15.0078 0.94551 0.0332895 2026-02-23T13:11:13.473 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:11:13 trial068 ceph-mon[38805]: pgmap v202: 17 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+undersized+remapped, 2 active+recovering+undersized+degraded+remapped+forced_backfill, 1 peering, 5 active+recovery_wait+undersized+degraded+remapped, 7 active+clean; 2.6 GiB data, 8.5 GiB used, 2.7 TiB / 2.7 TiB avail; 16 MiB/s wr, 2.03k op/s; 5391/125727 objects degraded (4.288%); 55 MiB/s, 878 objects/s recovering 2026-02-23T13:11:13.473 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:11:13 trial068 ceph-mon[38805]: osdmap e114: 4 total, 4 up, 4 in 2026-02-23T13:11:13.473 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T13:11:13.474 INFO:teuthology.orchestra.run.trial068.stderr:dumped all 2026-02-23T13:11:13.517 INFO:teuthology.orchestra.run.trial068.stdout:{"pg_ready":true,"pg_map":{"version":204,"stamp":"2026-02-23T13:11:13.361444+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2768159264,"num_objects":42236,"num_object_clones":0,"num_object_copies":126708,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5391,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42236,"num_whiteouts":0,"num_read":122,"num_read_kb":228,"num_write":337973,"num_write_kb":2704854,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21520,"num_bytes_recovered":1410196000,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":161037,"ondisk_log_size":161037,"up":51,"acting":42,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":51,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":2936012800,"kb_used":9025160,"kb_used_data":7595320,"kb_used_omap":98,"kb_used_meta":1429661,"kb_avail":2926987640,"statfs":{"total":3006477107200,"available":2997235343360,"internally_reserved":0,"allocated":7777607680,"data_stored":7774441301,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":100771,"internal_metadata":1463973469},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,2,4,3,0,1,7,8,40,55,33],"upper_bound":2048},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":87941120,"num_objects":1344,"num_object_clones":0,"num_object_copies":4032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-3516,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1344,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10735,"num_write_kb":85880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3517,"num_bytes_recovered":230432768,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.012490"},"pg_stats":[{"pgid":"3.f","version":"114'21072","reported_seq":25563,"reported_epoch":114,"state":"active+recovery_wait","last_fresh":"2026-02-23T13:11:12.367518+0000","last_change":"2026-02-23T13:11:08.367001+0000","last_active":"2026-02-23T13:11:12.367518+0000","last_peered":"2026-02-23T13:11:12.367518+0000","last_clean":"2026-02-23T13:09:42.363073+0000","last_became_active":"2026-02-23T13:11:08.355920+0000","last_became_peered":"2026-02-23T13:11:08.355920+0000","last_unstale":"2026-02-23T13:11:12.367518+0000","last_undegraded":"2026-02-23T13:11:12.367518+0000","last_fullsized":"2026-02-23T13:11:12.367518+0000","mapping_epoch":109,"log_start":"90'11000","ondisk_log_start":"90'11000","created":76,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:09:42.363073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:09:42.363073+0000","last_clean_scrub_stamp":"2026-02-23T13:09:42.363073+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":172621824,"num_objects":2634,"num_object_clones":0,"num_object_copies":7902,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2634,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21072,"num_write_kb":168576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2200,"num_bytes_recovered":144007168,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":["2","0","3"],"object_location_counts":[{"shards":"0,2,3","objects":2634}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.e","version":"113'20744","reported_seq":23037,"reported_epoch":114,"state":"active+clean","last_fresh":"2026-02-23T13:11:12.358961+0000","last_change":"2026-02-23T13:10:55.337830+0000","last_active":"2026-02-23T13:11:12.358961+0000","last_peered":"2026-02-23T13:11:12.358961+0000","last_clean":"2026-02-23T13:11:12.358961+0000","last_became_active":"2026-02-23T13:10:35.981714+0000","last_became_peered":"2026-02-23T13:10:35.981714+0000","last_unstale":"2026-02-23T13:11:12.358961+0000","last_undegraded":"2026-02-23T13:11:12.358961+0000","last_fullsized":"2026-02-23T13:11:12.358961+0000","mapping_epoch":101,"log_start":"90'10700","ondisk_log_start":"90'10700","created":76,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"104'17744","last_scrub_stamp":"2026-02-23T13:10:55.337787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:09:42.363073+0000","last_clean_scrub_stamp":"2026-02-23T13:10:55.337787+0000","objects_scrubbed":2214,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T13:11:57.088624+0000","scrub_duration":547,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":169934848,"num_objects":2593,"num_object_clones":0,"num_object_copies":7779,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20744,"num_write_kb":165952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1223,"num_bytes_recovered":80150528,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"113'21211","reported_seq":25446,"reported_epoch":113,"state":"active+clean","last_fresh":"2026-02-23T13:11:11.546219+0000","last_change":"2026-02-23T13:11:11.361132+0000","last_active":"2026-02-23T13:11:11.546219+0000","last_peered":"2026-02-23T13:11:11.546219+0000","last_clean":"2026-02-23T13:11:11.546219+0000","last_became_active":"2026-02-23T13:11:11.354765+0000","last_became_peered":"2026-02-23T13:11:11.354765+0000","last_unstale":"2026-02-23T13:11:11.546219+0000","last_undegraded":"2026-02-23T13:11:11.546219+0000","last_fullsized":"2026-02-23T13:11:11.546219+0000","mapping_epoch":112,"log_start":"90'11200","ondisk_log_start":"90'11200","created":76,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:09:42.363073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:09:42.363073+0000","last_clean_scrub_stamp":"2026-02-23T13:09:42.363073+0000","objects_scrubbed":0,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":173760512,"num_objects":2652,"num_object_clones":0,"num_object_copies":7956,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2652,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21211,"num_write_kb":169688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2249,"num_bytes_recovered":147292160,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.c","version":"113'20592","reported_seq":22854,"reported_epoch":113,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2026-02-23T13:11:11.546574+0000","last_change":"2026-02-23T13:11:10.167371+0000","last_active":"2026-02-23T13:11:11.546574+0000","last_peered":"2026-02-23T13:11:11.546574+0000","last_clean":"2026-02-23T13:10:32.288357+0000","last_became_active":"2026-02-23T13:11:03.788839+0000","last_became_peered":"2026-02-23T13:11:03.788839+0000","last_unstale":"2026-02-23T13:11:11.546574+0000","last_undegraded":"2026-02-23T13:11:03.361201+0000","last_fullsized":"2026-02-23T13:11:03.338304+0000","mapping_epoch":107,"log_start":"90'10500","ondisk_log_start":"90'10500","created":76,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:09:42.363073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:09:42.363073+0000","last_clean_scrub_stamp":"2026-02-23T13:09:42.363073+0000","objects_scrubbed":0,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":168689664,"num_objects":2574,"num_object_clones":0,"num_object_copies":7722,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":328,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2574,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20592,"num_write_kb":164736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1113,"num_bytes_recovered":72941568,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,1],"avail_no_missing":["3","1"],"object_location_counts":[{"shards":"1,3","objects":2246}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.b","version":"113'21267","reported_seq":24550,"reported_epoch":113,"state":"active+undersized+remapped","last_fresh":"2026-02-23T13:11:12.240532+0000","last_change":"2026-02-23T13:11:11.776389+0000","last_active":"2026-02-23T13:11:12.240532+0000","last_peered":"2026-02-23T13:11:12.240532+0000","last_clean":"2026-02-23T13:09:58.726181+0000","last_became_active":"2026-02-23T13:11:03.803230+0000","last_became_peered":"2026-02-23T13:11:03.803230+0000","last_unstale":"2026-02-23T13:11:12.240532+0000","last_undegraded":"2026-02-23T13:11:12.240532+0000","last_fullsized":"2026-02-23T13:11:03.340910+0000","mapping_epoch":107,"log_start":"90'11200","ondisk_log_start":"90'11200","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:09:42.363073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:09:42.363073+0000","last_clean_scrub_stamp":"2026-02-23T13:09:42.363073+0000","objects_scrubbed":0,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":174219264,"num_objects":2659,"num_object_clones":0,"num_object_copies":7977,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2659,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21267,"num_write_kb":170136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1592,"num_bytes_recovered":104333312,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,2],"avail_no_missing":["1","0","2"],"object_location_counts":[{"shards":"1,2","objects":2659}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"113'20840","reported_seq":23731,"reported_epoch":113,"state":"active+clean","last_fresh":"2026-02-23T13:11:11.544735+0000","last_change":"2026-02-23T13:10:47.945257+0000","last_active":"2026-02-23T13:11:11.544735+0000","last_peered":"2026-02-23T13:11:11.544735+0000","last_clean":"2026-02-23T13:11:11.544735+0000","last_became_active":"2026-02-23T13:10:33.983588+0000","last_became_peered":"2026-02-23T13:10:33.983588+0000","last_unstale":"2026-02-23T13:11:11.544735+0000","last_undegraded":"2026-02-23T13:11:11.544735+0000","last_fullsized":"2026-02-23T13:11:11.544735+0000","mapping_epoch":99,"log_start":"90'10800","ondisk_log_start":"90'10800","created":76,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"104'16184","last_scrub_stamp":"2026-02-23T13:10:47.945219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:09:42.363073+0000","last_clean_scrub_stamp":"2026-02-23T13:10:47.945219+0000","objects_scrubbed":2011,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T13:11:49.868548+0000","scrub_duration":909,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":170721280,"num_objects":2605,"num_object_clones":0,"num_object_copies":7815,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2605,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20840,"num_write_kb":166720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1383,"num_bytes_recovered":90578944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.9","version":"113'21242","reported_seq":25197,"reported_epoch":113,"state":"active+clean","last_fresh":"2026-02-23T13:11:11.489476+0000","last_change":"2026-02-23T13:11:09.356827+0000","last_active":"2026-02-23T13:11:11.489476+0000","last_peered":"2026-02-23T13:11:11.489476+0000","last_clean":"2026-02-23T13:11:11.489476+0000","last_became_active":"2026-02-23T13:11:09.350369+0000","last_became_peered":"2026-02-23T13:11:09.350369+0000","last_unstale":"2026-02-23T13:11:11.489476+0000","last_undegraded":"2026-02-23T13:11:11.489476+0000","last_fullsized":"2026-02-23T13:11:11.489476+0000","mapping_epoch":110,"log_start":"90'11200","ondisk_log_start":"90'11200","created":76,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:09:42.363073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:09:42.363073+0000","last_clean_scrub_stamp":"2026-02-23T13:09:42.363073+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":174014464,"num_objects":2656,"num_object_clones":0,"num_object_copies":7968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2656,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21242,"num_write_kb":169936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2277,"num_bytes_recovered":149118976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"113'21388","reported_seq":23001,"reported_epoch":113,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2026-02-23T13:11:12.239053+0000","last_change":"2026-02-23T13:11:03.795488+0000","last_active":"2026-02-23T13:11:12.239053+0000","last_peered":"2026-02-23T13:11:12.239053+0000","last_clean":"2026-02-23T13:09:58.743502+0000","last_became_active":"2026-02-23T13:11:03.782836+0000","last_became_peered":"2026-02-23T13:11:03.782836+0000","last_unstale":"2026-02-23T13:11:12.239053+0000","last_undegraded":"2026-02-23T13:11:03.374124+0000","last_fullsized":"2026-02-23T13:11:03.340834+0000","mapping_epoch":107,"log_start":"90'11300","ondisk_log_start":"90'11300","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:09:42.363073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:09:42.363073+0000","last_clean_scrub_stamp":"2026-02-23T13:09:42.363073+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":175210496,"num_objects":2674,"num_object_clones":0,"num_object_copies":8022,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":835,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2674,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21388,"num_write_kb":171104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":757,"num_bytes_recovered":49610752,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[{"shards":"1,2","objects":1839}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"113'20840","reported_seq":22123,"reported_epoch":113,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2026-02-23T13:11:12.232854+0000","last_change":"2026-02-23T13:11:03.792375+0000","last_active":"2026-02-23T13:11:12.232854+0000","last_peered":"2026-02-23T13:11:12.232854+0000","last_clean":"2026-02-23T13:10:32.272970+0000","last_became_active":"2026-02-23T13:11:03.778183+0000","last_became_peered":"2026-02-23T13:11:03.778183+0000","last_unstale":"2026-02-23T13:11:12.232854+0000","last_undegraded":"2026-02-23T13:11:03.371361+0000","last_fullsized":"2026-02-23T13:11:03.340781+0000","mapping_epoch":107,"log_start":"90'10800","ondisk_log_start":"90'10800","created":76,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:09:42.363073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:09:42.363073+0000","last_clean_scrub_stamp":"2026-02-23T13:09:42.363073+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":170721280,"num_objects":2605,"num_object_clones":0,"num_object_copies":7815,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":794,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2605,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20840,"num_write_kb":166720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":591,"num_bytes_recovered":38731776,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[{"shards":"1,3","objects":1811}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.6","version":"113'21595","reported_seq":23199,"reported_epoch":113,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2026-02-23T13:11:12.242225+0000","last_change":"2026-02-23T13:11:03.811837+0000","last_active":"2026-02-23T13:11:12.242225+0000","last_peered":"2026-02-23T13:11:12.242225+0000","last_clean":"2026-02-23T13:09:58.717061+0000","last_became_active":"2026-02-23T13:11:03.803413+0000","last_became_peered":"2026-02-23T13:11:03.803413+0000","last_unstale":"2026-02-23T13:11:12.242225+0000","last_undegraded":"2026-02-23T13:11:03.373253+0000","last_fullsized":"2026-02-23T13:11:03.341107+0000","mapping_epoch":107,"log_start":"90'11500","ondisk_log_start":"90'11500","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:09:42.363073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:09:42.363073+0000","last_clean_scrub_stamp":"2026-02-23T13:09:42.363073+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":176906240,"num_objects":2700,"num_object_clones":0,"num_object_copies":8100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":826,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2700,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21595,"num_write_kb":172760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":760,"num_bytes_recovered":49725440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[{"shards":"1,2","objects":1874}],"blocked_by":[],"up_primary":0,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"113'21224","reported_seq":23304,"reported_epoch":114,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2026-02-23T13:11:12.358522+0000","last_change":"2026-02-23T13:11:03.808821+0000","last_active":"2026-02-23T13:11:12.358522+0000","last_peered":"2026-02-23T13:11:12.358522+0000","last_clean":"2026-02-23T13:09:42.363073+0000","last_became_active":"2026-02-23T13:11:03.802586+0000","last_became_peered":"2026-02-23T13:11:03.802586+0000","last_unstale":"2026-02-23T13:11:12.358522+0000","last_undegraded":"2026-02-23T13:11:03.355862+0000","last_fullsized":"2026-02-23T13:11:03.339490+0000","mapping_epoch":107,"log_start":"90'11200","ondisk_log_start":"90'11200","created":76,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:09:42.363073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:09:42.363073+0000","last_clean_scrub_stamp":"2026-02-23T13:09:42.363073+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":173867008,"num_objects":2653,"num_object_clones":0,"num_object_copies":7959,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":867,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2653,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21224,"num_write_kb":169792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":988,"num_bytes_recovered":64749568,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,3],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":1786}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"113'20860","reported_seq":24035,"reported_epoch":113,"state":"active+clean","last_fresh":"2026-02-23T13:11:12.242262+0000","last_change":"2026-02-23T13:11:10.363095+0000","last_active":"2026-02-23T13:11:12.242262+0000","last_peered":"2026-02-23T13:11:12.242262+0000","last_clean":"2026-02-23T13:11:12.242262+0000","last_became_active":"2026-02-23T13:11:10.355967+0000","last_became_peered":"2026-02-23T13:11:10.355967+0000","last_unstale":"2026-02-23T13:11:12.242262+0000","last_undegraded":"2026-02-23T13:11:12.242262+0000","last_fullsized":"2026-02-23T13:11:12.242262+0000","mapping_epoch":111,"log_start":"90'10800","ondisk_log_start":"90'10800","created":76,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:09:42.363073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:09:42.363073+0000","last_clean_scrub_stamp":"2026-02-23T13:09:42.363073+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":170885120,"num_objects":2608,"num_object_clones":0,"num_object_copies":7824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2608,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20860,"num_write_kb":166880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1533,"num_bytes_recovered":100466688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"113'21448","reported_seq":23866,"reported_epoch":114,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2026-02-23T13:11:12.358521+0000","last_change":"2026-02-23T13:11:03.808237+0000","last_active":"2026-02-23T13:11:12.358521+0000","last_peered":"2026-02-23T13:11:12.358521+0000","last_clean":"2026-02-23T13:09:42.363073+0000","last_became_active":"2026-02-23T13:11:03.802570+0000","last_became_peered":"2026-02-23T13:11:03.802570+0000","last_unstale":"2026-02-23T13:11:12.358521+0000","last_undegraded":"2026-02-23T13:11:03.356118+0000","last_fullsized":"2026-02-23T13:11:03.339502+0000","mapping_epoch":107,"log_start":"90'11400","ondisk_log_start":"90'11400","created":76,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:09:42.363073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:09:42.363073+0000","last_clean_scrub_stamp":"2026-02-23T13:09:42.363073+0000","objects_scrubbed":0,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":175702016,"num_objects":2681,"num_object_clones":0,"num_object_copies":8043,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":850,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2681,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21448,"num_write_kb":171584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1179,"num_bytes_recovered":77266944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,3],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":1831}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"113'21384","reported_seq":23124,"reported_epoch":113,"state":"active+recovering+undersized+degraded+remapped+forced_backfill","last_fresh":"2026-02-23T13:11:12.244127+0000","last_change":"2026-02-23T13:11:11.785395+0000","last_active":"2026-02-23T13:11:12.244127+0000","last_peered":"2026-02-23T13:11:12.244127+0000","last_clean":"2026-02-23T13:10:32.343256+0000","last_became_active":"2026-02-23T13:11:03.789175+0000","last_became_peered":"2026-02-23T13:11:03.789175+0000","last_unstale":"2026-02-23T13:11:12.244127+0000","last_undegraded":"2026-02-23T13:11:03.373352+0000","last_fullsized":"2026-02-23T13:11:03.340847+0000","mapping_epoch":107,"log_start":"90'11300","ondisk_log_start":"90'11300","created":76,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:09:42.363073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:09:42.363073+0000","last_clean_scrub_stamp":"2026-02-23T13:09:42.363073+0000","objects_scrubbed":0,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":175177728,"num_objects":2673,"num_object_clones":0,"num_object_copies":8019,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":824,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2673,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21384,"num_write_kb":171072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":818,"num_bytes_recovered":53575680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[{"shards":"1,2","objects":1849}],"blocked_by":[],"up_primary":0,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"113'21088","reported_seq":24214,"reported_epoch":113,"state":"active+recovering+undersized+degraded+remapped+forced_backfill","last_fresh":"2026-02-23T13:11:12.230153+0000","last_change":"2026-02-23T13:11:08.049059+0000","last_active":"2026-02-23T13:11:12.230153+0000","last_peered":"2026-02-23T13:11:12.230153+0000","last_clean":"2026-02-23T13:09:58.652894+0000","last_became_active":"2026-02-23T13:11:03.789042+0000","last_became_peered":"2026-02-23T13:11:03.789042+0000","last_unstale":"2026-02-23T13:11:12.230153+0000","last_undegraded":"2026-02-23T13:11:03.382995+0000","last_fullsized":"2026-02-23T13:11:03.340886+0000","mapping_epoch":107,"log_start":"90'11000","ondisk_log_start":"90'11000","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:09:42.363073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:09:42.363073+0000","last_clean_scrub_stamp":"2026-02-23T13:09:42.363073+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":172752896,"num_objects":2636,"num_object_clones":0,"num_object_copies":7908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":67,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21088,"num_write_kb":168704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1513,"num_bytes_recovered":99155968,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[{"shards":"1,2","objects":2569}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"113'21043","reported_seq":23859,"reported_epoch":113,"state":"active+clean","last_fresh":"2026-02-23T13:11:12.241031+0000","last_change":"2026-02-23T13:10:46.187555+0000","last_active":"2026-02-23T13:11:12.241031+0000","last_peered":"2026-02-23T13:11:12.241031+0000","last_clean":"2026-02-23T13:11:12.241031+0000","last_became_active":"2026-02-23T13:10:35.982713+0000","last_became_peered":"2026-02-23T13:10:35.982713+0000","last_unstale":"2026-02-23T13:11:12.241031+0000","last_undegraded":"2026-02-23T13:11:12.241031+0000","last_fullsized":"2026-02-23T13:11:12.241031+0000","mapping_epoch":101,"log_start":"90'11000","ondisk_log_start":"90'11000","created":76,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"104'15600","last_scrub_stamp":"2026-02-23T13:10:46.187433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:09:42.363073+0000","last_clean_scrub_stamp":"2026-02-23T13:10:46.187433+0000","objects_scrubbed":1939,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T13:11:54.258632+0000","scrub_duration":1268,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":172384256,"num_objects":2631,"num_object_clones":0,"num_object_copies":7893,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2631,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21043,"num_write_kb":168344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1342,"num_bytes_recovered":87900160,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"28'99","reported_seq":205,"reported_epoch":113,"state":"active+clean","last_fresh":"2026-02-23T13:11:11.353085+0000","last_change":"2026-02-23T13:11:02.340041+0000","last_active":"2026-02-23T13:11:11.353085+0000","last_peered":"2026-02-23T13:11:11.353085+0000","last_clean":"2026-02-23T13:11:11.353085+0000","last_became_active":"2026-02-23T13:11:02.339822+0000","last_became_peered":"2026-02-23T13:11:02.339822+0000","last_unstale":"2026-02-23T13:11:11.353085+0000","last_undegraded":"2026-02-23T13:11:11.353085+0000","last_fullsized":"2026-02-23T13:11:11.353085+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'99","last_scrub_stamp":"2026-02-23T13:09:54.841853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:07:05.619208+0000","last_clean_scrub_stamp":"2026-02-23T13:09:54.841853+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T13:11:19.668596+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":122,"num_read_kb":228,"num_write":135,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":2767568896,"num_objects":42234,"num_object_clones":0,"num_object_copies":126702,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5391,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42234,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":337838,"num_write_kb":2702704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21518,"num_bytes_recovered":1409605632,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":7844118528,"data_stored":7844118528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160938,"ondisk_log_size":160938,"up":48,"acting":39,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":122,"num_read_kb":228,"num_write":135,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":80,"seq":343597383697,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":734003200,"kb_used":2047944,"kb_used_data":1671988,"kb_used_omap":30,"kb_used_meta":375905,"kb_avail":731955256,"statfs":{"total":751619276800,"available":749522182144,"internally_reserved":0,"allocated":1712115712,"data_stored":1711325141,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":31171,"internal_metadata":384927293},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":92,"seq":395136991243,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":734003200,"kb_used":2736156,"kb_used_data":2365192,"kb_used_omap":19,"kb_used_meta":370924,"kb_avail":731267044,"statfs":{"total":751619276800,"available":748817453056,"internally_reserved":0,"allocated":2421956608,"data_stored":2421158419,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20047,"internal_metadata":379826609},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,2,4,3,0,1,7,8,40,24],"upper_bound":1024},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":64,"seq":274877906968,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":734003200,"kb_used":2404896,"kb_used_data":1850060,"kb_used_omap":39,"kb_used_meta":554776,"kb_avail":731598304,"statfs":{"total":751619276800,"available":749156663296,"internally_reserved":0,"allocated":1894461440,"data_stored":1893666803,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":40593,"internal_metadata":568090991},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,31,33],"upper_bound":2048},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":106,"seq":455266533379,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":734003200,"kb_used":1836164,"kb_used_data":1708080,"kb_used_omap":8,"kb_used_meta":128055,"kb_avail":732167036,"statfs":{"total":751619276800,"available":749739044864,"internally_reserved":0,"allocated":1749073920,"data_stored":1748290938,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8960,"internal_metadata":131128576},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1795334144,"data_stored":1795334144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1899200512,"data_stored":1899200512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":2434187264,"data_stored":2434187264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1715396608,"data_stored":1715396608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-23T13:11:14.227 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:11:14.255 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:11:13 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:11:13] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:11:14.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:11:13 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:11:13.972Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:11:14.383 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:11:14.465 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 90 117 343291 343174 29.7871 18.2031 0.0208907 0.0335544 2026-02-23T13:11:14.505 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:11:14.503+0000 7f333511d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:11:14.505 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:11:14.504+0000 7f333511d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:11:14.506 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:11:14 trial068 ceph-mon[38805]: osdmap e115: 4 total, 4 up, 4 in 2026-02-23T13:11:14.506 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:11:14 trial068 ceph-mon[38805]: from='client.14608 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2026-02-23T13:11:14.506 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:11:14 trial068 ceph-mon[38805]: Health check update: Degraded data redundancy: 5391/125727 objects degraded (4.288%), 8 pgs degraded (PG_DEGRADED) 2026-02-23T13:11:14.506 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:11:14 trial068 ceph-mon[38805]: osdmap e116: 4 total, 4 up, 4 in 2026-02-23T13:11:14.518 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:11:14.518 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:11:14.529 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:11:14.529 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:11:14.541 INFO:teuthology.orchestra.run.trial068.stdout:osd.2: {} 2026-02-23T13:11:14.541 INFO:teuthology.orchestra.run.trial068.stderr:osd.2: osd_enable_op_tracker = '' 2026-02-23T13:11:14.553 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:11:14.553 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:11:15.465 INFO:tasks.radosbench.radosbench.0.trial068.stdout:Total time run: 90.0823 2026-02-23T13:11:15.465 INFO:tasks.radosbench.radosbench.0.trial068.stdout:Total writes made: 343291 2026-02-23T13:11:15.465 INFO:tasks.radosbench.radosbench.0.trial068.stdout:Write size: 8192 2026-02-23T13:11:15.465 INFO:tasks.radosbench.radosbench.0.trial068.stdout:Object size: 65536 2026-02-23T13:11:15.466 INFO:tasks.radosbench.radosbench.0.trial068.stdout:Bandwidth (MB/sec): 29.7723 2026-02-23T13:11:15.466 INFO:tasks.radosbench.radosbench.0.trial068.stdout:Stddev Bandwidth: 18.6151 2026-02-23T13:11:15.466 INFO:tasks.radosbench.radosbench.0.trial068.stdout:Max bandwidth (MB/sec): 88.9141 2026-02-23T13:11:15.466 INFO:tasks.radosbench.radosbench.0.trial068.stdout:Min bandwidth (MB/sec): 0 2026-02-23T13:11:15.466 INFO:tasks.radosbench.radosbench.0.trial068.stdout:Average IOPS: 3810 2026-02-23T13:11:15.466 INFO:tasks.radosbench.radosbench.0.trial068.stdout:Stddev IOPS: 2382.74 2026-02-23T13:11:15.466 INFO:tasks.radosbench.radosbench.0.trial068.stdout:Max IOPS: 11381 2026-02-23T13:11:15.466 INFO:tasks.radosbench.radosbench.0.trial068.stdout:Min IOPS: 0 2026-02-23T13:11:15.466 INFO:tasks.radosbench.radosbench.0.trial068.stdout:Average Latency(s): 0.0335656 2026-02-23T13:11:15.466 INFO:tasks.radosbench.radosbench.0.trial068.stdout:Stddev Latency(s): 0.103013 2026-02-23T13:11:15.466 INFO:tasks.radosbench.radosbench.0.trial068.stdout:Max latency(s): 2.63923 2026-02-23T13:11:15.467 INFO:tasks.radosbench.radosbench.0.trial068.stdout:Min latency(s): 0.000737354 2026-02-23T13:11:15.473 INFO:tasks.radosbench.radosbench.0.trial068.stdout:Cleaning up (deleting benchmark objects) 2026-02-23T13:11:15.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:11:15 trial068 ceph-mon[38805]: pgmap v206: 17 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 1 active+undersized+remapped, 2 active+recovering+undersized+degraded+remapped+forced_backfill, 5 active+recovery_wait+undersized+degraded+remapped, 7 active+clean; 2.6 GiB data, 8.6 GiB used, 2.7 TiB / 2.7 TiB avail; 21 MiB/s wr, 2.75k op/s; 5391/126708 objects degraded (4.255%); 56 MiB/s, 900 objects/s recovering 2026-02-23T13:11:15.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:11:15 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1679746265' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:11:16.595 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:11:16.621 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:11:16 trial068 ceph-mon[38805]: osdmap e117: 4 total, 4 up, 4 in 2026-02-23T13:11:16.737 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:11:16.853 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:11:16.852+0000 7f1aa6c6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:11:16.853 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:11:16.852+0000 7f1aa6c6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:11:16.869 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:11:16.869 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:11:16.881 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:11:16.882 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:11:16.895 INFO:teuthology.orchestra.run.trial068.stdout:osd.2: {} 2026-02-23T13:11:16.895 INFO:teuthology.orchestra.run.trial068.stderr:osd.2: osd_enable_op_tracker = '' 2026-02-23T13:11:16.910 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:11:16.910 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:11:17.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:11:17 trial068 ceph-mon[38805]: pgmap v208: 17 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 1 active+undersized+remapped, 2 active+recovering+undersized+degraded+remapped+forced_backfill, 5 active+recovery_wait+undersized+degraded+remapped, 7 active+clean; 2.6 GiB data, 8.9 GiB used, 2.7 TiB / 2.7 TiB avail; 5.1 MiB/s wr, 653 op/s; 5391/126708 objects degraded (4.255%); 2.0 KiB/s, 0 objects/s recovering 2026-02-23T13:11:17.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:11:17 trial068 ceph-mon[38805]: osdmap e118: 4 total, 4 up, 4 in 2026-02-23T13:11:17.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:11:17 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3466524097' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:11:18.520 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [1, 3, 2, 0] 2026-02-23T13:11:18.520 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2026-02-23T13:11:18.520 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph pg dump --format=json 2026-02-23T13:11:18.664 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:11:18.690 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:11:18 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T13:11:18.691 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:11:18 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T13:11:18.691 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:11:18 trial068 ceph-mon[38805]: 3.c scrub starts 2026-02-23T13:11:18.691 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:11:18 trial068 ceph-mon[38805]: 3.c scrub ok 2026-02-23T13:11:18.691 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:11:18 trial068 ceph-mon[38805]: osdmap e119: 4 total, 4 up, 4 in 2026-02-23T13:11:18.792 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:11:18.790+0000 7fc16e505640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:11:18.792 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:11:18.791+0000 7fc16e505640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:11:18.933 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T13:11:18.933 INFO:teuthology.orchestra.run.trial068.stderr:dumped all 2026-02-23T13:11:18.953 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:11:18.974 INFO:teuthology.orchestra.run.trial068.stdout:{"pg_ready":true,"pg_map":{"version":212,"stamp":"2026-02-23T13:11:18.371119+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2671862328,"num_objects":40764,"num_object_clones":0,"num_object_copies":122292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1043,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40764,"num_whiteouts":0,"num_read":123,"num_read_kb":229,"num_write":345173,"num_write_kb":2745599,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25869,"num_bytes_recovered":1695154720,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":161037,"ondisk_log_size":161037,"up":51,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":51,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":2936012800,"kb_used":9255496,"kb_used_data":7807544,"kb_used_omap":111,"kb_used_meta":1447696,"kb_avail":2926757304,"statfs":{"total":3006477107200,"available":2996999479296,"internally_reserved":0,"allocated":7994925056,"data_stored":7991597065,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":113811,"internal_metadata":1482441581},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-96296936,"num_objects":-1472,"num_object_clones":0,"num_object_copies":-4416,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-4348,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1472,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7200,"num_write_kb":40745,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4349,"num_bytes_recovered":284958720,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.112288"},"pg_stats":[{"pgid":"3.f","version":"118'21556","reported_seq":26062,"reported_epoch":119,"state":"active+clean","last_fresh":"2026-02-23T13:11:17.369287+0000","last_change":"2026-02-23T13:11:14.754265+0000","last_active":"2026-02-23T13:11:17.369287+0000","last_peered":"2026-02-23T13:11:17.369287+0000","last_clean":"2026-02-23T13:11:17.369287+0000","last_became_active":"2026-02-23T13:11:08.355920+0000","last_became_peered":"2026-02-23T13:11:08.355920+0000","last_unstale":"2026-02-23T13:11:17.369287+0000","last_undegraded":"2026-02-23T13:11:17.369287+0000","last_fullsized":"2026-02-23T13:11:17.369287+0000","mapping_epoch":109,"log_start":"90'11500","ondisk_log_start":"90'11500","created":76,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:09:42.363073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:09:42.363073+0000","last_clean_scrub_stamp":"2026-02-23T13:09:42.363073+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":165085184,"num_objects":2519,"num_object_clones":0,"num_object_copies":7557,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2519,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21556,"num_write_kb":171200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2200,"num_bytes_recovered":144007168,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.e","version":"118'21257","reported_seq":23560,"reported_epoch":119,"state":"active+clean","last_fresh":"2026-02-23T13:11:17.369309+0000","last_change":"2026-02-23T13:10:55.337830+0000","last_active":"2026-02-23T13:11:17.369309+0000","last_peered":"2026-02-23T13:11:17.369309+0000","last_clean":"2026-02-23T13:11:17.369309+0000","last_became_active":"2026-02-23T13:10:35.981714+0000","last_became_peered":"2026-02-23T13:10:35.981714+0000","last_unstale":"2026-02-23T13:11:17.369309+0000","last_undegraded":"2026-02-23T13:11:17.369309+0000","last_fullsized":"2026-02-23T13:11:17.369309+0000","mapping_epoch":101,"log_start":"90'11200","ondisk_log_start":"90'11200","created":76,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"104'17744","last_scrub_stamp":"2026-02-23T13:10:55.337787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:09:42.363073+0000","last_clean_scrub_stamp":"2026-02-23T13:10:55.337787+0000","objects_scrubbed":2214,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T13:11:57.088624+0000","scrub_duration":547,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":160497664,"num_objects":2449,"num_object_clones":0,"num_object_copies":7347,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2449,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21257,"num_write_kb":168576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1223,"num_bytes_recovered":80150528,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"118'21769","reported_seq":26022,"reported_epoch":118,"state":"active+clean","last_fresh":"2026-02-23T13:11:16.563529+0000","last_change":"2026-02-23T13:11:11.361132+0000","last_active":"2026-02-23T13:11:16.563529+0000","last_peered":"2026-02-23T13:11:16.563529+0000","last_clean":"2026-02-23T13:11:16.563529+0000","last_became_active":"2026-02-23T13:11:11.354765+0000","last_became_peered":"2026-02-23T13:11:11.354765+0000","last_unstale":"2026-02-23T13:11:16.563529+0000","last_undegraded":"2026-02-23T13:11:16.563529+0000","last_fullsized":"2026-02-23T13:11:16.563529+0000","mapping_epoch":112,"log_start":"90'11700","ondisk_log_start":"90'11700","created":76,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:09:42.363073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:09:42.363073+0000","last_clean_scrub_stamp":"2026-02-23T13:09:42.363073+0000","objects_scrubbed":0,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":172425240,"num_objects":2632,"num_object_clones":0,"num_object_copies":7896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2632,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":21769,"num_write_kb":173505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2249,"num_bytes_recovered":147292160,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.c","version":"118'20981","reported_seq":23917,"reported_epoch":118,"state":"active+clean","last_fresh":"2026-02-23T13:11:16.530030+0000","last_change":"2026-02-23T13:11:13.990681+0000","last_active":"2026-02-23T13:11:16.530030+0000","last_peered":"2026-02-23T13:11:16.530030+0000","last_clean":"2026-02-23T13:11:16.530030+0000","last_became_active":"2026-02-23T13:11:13.984902+0000","last_became_peered":"2026-02-23T13:11:13.984902+0000","last_unstale":"2026-02-23T13:11:16.530030+0000","last_undegraded":"2026-02-23T13:11:16.530030+0000","last_fullsized":"2026-02-23T13:11:16.530030+0000","mapping_epoch":115,"log_start":"90'10900","ondisk_log_start":"90'10900","created":76,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:09:42.363073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:09:42.363073+0000","last_clean_scrub_stamp":"2026-02-23T13:09:42.363073+0000","objects_scrubbed":0,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":166199296,"num_objects":2536,"num_object_clones":0,"num_object_copies":7608,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2536,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20981,"num_write_kb":167232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1441,"num_bytes_recovered":94437376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.b","version":"118'21761","reported_seq":25064,"reported_epoch":118,"state":"active+clean","last_fresh":"2026-02-23T13:11:17.057478+0000","last_change":"2026-02-23T13:11:16.209021+0000","last_active":"2026-02-23T13:11:17.057478+0000","last_peered":"2026-02-23T13:11:17.057478+0000","last_clean":"2026-02-23T13:11:17.057478+0000","last_became_active":"2026-02-23T13:11:13.372478+0000","last_became_peered":"2026-02-23T13:11:13.372478+0000","last_unstale":"2026-02-23T13:11:17.057478+0000","last_undegraded":"2026-02-23T13:11:17.057478+0000","last_fullsized":"2026-02-23T13:11:17.057478+0000","mapping_epoch":114,"log_start":"91'11700","ondisk_log_start":"91'11700","created":76,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:09:42.363073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:09:42.363073+0000","last_clean_scrub_stamp":"2026-02-23T13:09:42.363073+0000","objects_scrubbed":0,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":164626432,"num_objects":2512,"num_object_clones":0,"num_object_copies":7536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2512,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21761,"num_write_kb":172608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1593,"num_bytes_recovered":104341504,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"118'21297","reported_seq":24200,"reported_epoch":118,"state":"active+clean","last_fresh":"2026-02-23T13:11:16.554715+0000","last_change":"2026-02-23T13:10:47.945257+0000","last_active":"2026-02-23T13:11:16.554715+0000","last_peered":"2026-02-23T13:11:16.554715+0000","last_clean":"2026-02-23T13:11:16.554715+0000","last_became_active":"2026-02-23T13:10:33.983588+0000","last_became_peered":"2026-02-23T13:10:33.983588+0000","last_unstale":"2026-02-23T13:11:16.554715+0000","last_undegraded":"2026-02-23T13:11:16.554715+0000","last_fullsized":"2026-02-23T13:11:16.554715+0000","mapping_epoch":99,"log_start":"90'11200","ondisk_log_start":"90'11200","created":76,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"104'16184","last_scrub_stamp":"2026-02-23T13:10:47.945219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:09:42.363073+0000","last_clean_scrub_stamp":"2026-02-23T13:10:47.945219+0000","objects_scrubbed":2011,"log_size":10097,"log_dups_size":0,"ondisk_log_size":10097,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T13:11:49.868548+0000","scrub_duration":909,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":167313408,"num_objects":2553,"num_object_clones":0,"num_object_copies":7659,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2553,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21297,"num_write_kb":169600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1383,"num_bytes_recovered":90578944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.9","version":"118'21858","reported_seq":25824,"reported_epoch":118,"state":"active+clean","last_fresh":"2026-02-23T13:11:16.556475+0000","last_change":"2026-02-23T13:11:09.356827+0000","last_active":"2026-02-23T13:11:16.556475+0000","last_peered":"2026-02-23T13:11:16.556475+0000","last_clean":"2026-02-23T13:11:16.556475+0000","last_became_active":"2026-02-23T13:11:09.350369+0000","last_became_peered":"2026-02-23T13:11:09.350369+0000","last_unstale":"2026-02-23T13:11:16.556475+0000","last_undegraded":"2026-02-23T13:11:16.556475+0000","last_fullsized":"2026-02-23T13:11:16.556475+0000","mapping_epoch":110,"log_start":"90'11800","ondisk_log_start":"90'11800","created":76,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:09:42.363073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:09:42.363073+0000","last_clean_scrub_stamp":"2026-02-23T13:09:42.363073+0000","objects_scrubbed":0,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":172425216,"num_objects":2631,"num_object_clones":0,"num_object_copies":7893,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2631,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21858,"num_write_kb":174144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2277,"num_bytes_recovered":149118976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"118'21885","reported_seq":25170,"reported_epoch":118,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2026-02-23T13:11:17.249889+0000","last_change":"2026-02-23T13:11:15.055157+0000","last_active":"2026-02-23T13:11:17.249889+0000","last_peered":"2026-02-23T13:11:17.249889+0000","last_clean":"2026-02-23T13:09:58.743502+0000","last_became_active":"2026-02-23T13:11:03.782836+0000","last_became_peered":"2026-02-23T13:11:03.782836+0000","last_unstale":"2026-02-23T13:11:17.249889+0000","last_undegraded":"2026-02-23T13:11:03.374124+0000","last_fullsized":"2026-02-23T13:11:03.340834+0000","mapping_epoch":107,"log_start":"91'11800","ondisk_log_start":"91'11800","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:09:42.363073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:09:42.363073+0000","last_clean_scrub_stamp":"2026-02-23T13:09:42.363073+0000","objects_scrubbed":0,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":165937152,"num_objects":2532,"num_object_clones":0,"num_object_copies":7596,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2532,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21885,"num_write_kb":173632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1587,"num_bytes_recovered":104005632,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[{"shards":"1,2","objects":2527}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"117'21211","reported_seq":24093,"reported_epoch":117,"state":"peering","last_fresh":"2026-02-23T13:11:16.208170+0000","last_change":"2026-02-23T13:11:16.368138+0000","last_active":"2026-02-23T13:11:16.368138+0000","last_peered":"2026-02-23T13:11:16.208170+0000","last_clean":"2026-02-23T13:10:32.272970+0000","last_became_active":"2026-02-23T13:11:03.778183+0000","last_became_peered":"2026-02-23T13:11:03.778183+0000","last_unstale":"2026-02-23T13:11:16.208170+0000","last_undegraded":"2026-02-23T13:11:16.208170+0000","last_fullsized":"2026-02-23T13:11:03.340781+0000","mapping_epoch":118,"log_start":"90'11200","ondisk_log_start":"90'11200","created":76,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:09:42.363073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:09:42.363073+0000","last_clean_scrub_stamp":"2026-02-23T13:09:42.363073+0000","objects_scrubbed":0,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":170221568,"num_objects":2598,"num_object_clones":0,"num_object_copies":7794,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2598,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21211,"num_write_kb":169304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1385,"num_bytes_recovered":90767360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,3],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.6","version":"118'22101","reported_seq":24942,"reported_epoch":118,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2026-02-23T13:11:17.250694+0000","last_change":"2026-02-23T13:11:15.351411+0000","last_active":"2026-02-23T13:11:17.250694+0000","last_peered":"2026-02-23T13:11:17.250694+0000","last_clean":"2026-02-23T13:09:58.717061+0000","last_became_active":"2026-02-23T13:11:03.803413+0000","last_became_peered":"2026-02-23T13:11:03.803413+0000","last_unstale":"2026-02-23T13:11:17.250694+0000","last_undegraded":"2026-02-23T13:11:03.373253+0000","last_fullsized":"2026-02-23T13:11:03.341107+0000","mapping_epoch":107,"log_start":"90'12100","ondisk_log_start":"90'12100","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:09:42.363073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:09:42.363073+0000","last_clean_scrub_stamp":"2026-02-23T13:09:42.363073+0000","objects_scrubbed":0,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":167706624,"num_objects":2559,"num_object_clones":0,"num_object_copies":7677,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":214,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2559,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22101,"num_write_kb":175360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1372,"num_bytes_recovered":89833472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[{"shards":"1,2","objects":2345}],"blocked_by":[],"up_primary":0,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"118'21682","reported_seq":25515,"reported_epoch":119,"state":"active+clean","last_fresh":"2026-02-23T13:11:17.369232+0000","last_change":"2026-02-23T13:11:16.375505+0000","last_active":"2026-02-23T13:11:17.369232+0000","last_peered":"2026-02-23T13:11:17.369232+0000","last_clean":"2026-02-23T13:11:17.369232+0000","last_became_active":"2026-02-23T13:11:16.368832+0000","last_became_peered":"2026-02-23T13:11:16.368832+0000","last_unstale":"2026-02-23T13:11:17.369232+0000","last_undegraded":"2026-02-23T13:11:17.369232+0000","last_fullsized":"2026-02-23T13:11:17.369232+0000","mapping_epoch":117,"log_start":"91'11600","ondisk_log_start":"91'11600","created":76,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:09:42.363073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:09:42.363073+0000","last_clean_scrub_stamp":"2026-02-23T13:09:42.363073+0000","objects_scrubbed":0,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":165675008,"num_objects":2528,"num_object_clones":0,"num_object_copies":7584,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2528,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21682,"num_write_kb":172160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1855,"num_bytes_recovered":121569280,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"118'21328","reported_seq":24513,"reported_epoch":118,"state":"active+clean","last_fresh":"2026-02-23T13:11:17.053428+0000","last_change":"2026-02-23T13:11:10.363095+0000","last_active":"2026-02-23T13:11:17.053428+0000","last_peered":"2026-02-23T13:11:17.053428+0000","last_clean":"2026-02-23T13:11:17.053428+0000","last_became_active":"2026-02-23T13:11:10.355967+0000","last_became_peered":"2026-02-23T13:11:10.355967+0000","last_unstale":"2026-02-23T13:11:17.053428+0000","last_undegraded":"2026-02-23T13:11:17.053428+0000","last_fullsized":"2026-02-23T13:11:17.053428+0000","mapping_epoch":111,"log_start":"90'11300","ondisk_log_start":"90'11300","created":76,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:09:42.363073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:09:42.363073+0000","last_clean_scrub_stamp":"2026-02-23T13:09:42.363073+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":162332672,"num_objects":2477,"num_object_clones":0,"num_object_copies":7431,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2477,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21328,"num_write_kb":169280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1533,"num_bytes_recovered":100466688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"119'21868","reported_seq":26005,"reported_epoch":119,"state":"active+clean","last_fresh":"2026-02-23T13:11:17.369297+0000","last_change":"2026-02-23T13:11:16.375711+0000","last_active":"2026-02-23T13:11:17.369297+0000","last_peered":"2026-02-23T13:11:17.369297+0000","last_clean":"2026-02-23T13:11:17.369297+0000","last_became_active":"2026-02-23T13:11:16.369132+0000","last_became_peered":"2026-02-23T13:11:16.369132+0000","last_unstale":"2026-02-23T13:11:17.369297+0000","last_undegraded":"2026-02-23T13:11:17.369297+0000","last_fullsized":"2026-02-23T13:11:17.369297+0000","mapping_epoch":117,"log_start":"90'11800","ondisk_log_start":"90'11800","created":76,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:09:42.363073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:09:42.363073+0000","last_clean_scrub_stamp":"2026-02-23T13:09:42.363073+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":168230912,"num_objects":2567,"num_object_clones":0,"num_object_copies":7701,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2567,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21868,"num_write_kb":173760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2029,"num_bytes_recovered":132972544,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"113'21384","reported_seq":23124,"reported_epoch":113,"state":"active+recovering+undersized+degraded+remapped+forced_backfill","last_fresh":"2026-02-23T13:11:12.244127+0000","last_change":"2026-02-23T13:11:11.785395+0000","last_active":"2026-02-23T13:11:12.244127+0000","last_peered":"2026-02-23T13:11:12.244127+0000","last_clean":"2026-02-23T13:10:32.343256+0000","last_became_active":"2026-02-23T13:11:03.789175+0000","last_became_peered":"2026-02-23T13:11:03.789175+0000","last_unstale":"2026-02-23T13:11:12.244127+0000","last_undegraded":"2026-02-23T13:11:03.373352+0000","last_fullsized":"2026-02-23T13:11:03.340847+0000","mapping_epoch":107,"log_start":"90'11300","ondisk_log_start":"90'11300","created":76,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:09:42.363073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:09:42.363073+0000","last_clean_scrub_stamp":"2026-02-23T13:09:42.363073+0000","objects_scrubbed":0,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":175177728,"num_objects":2673,"num_object_clones":0,"num_object_copies":8019,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":824,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2673,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21384,"num_write_kb":171072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":818,"num_bytes_recovered":53575680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[{"shards":"1,2","objects":1849}],"blocked_by":[],"up_primary":0,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"118'21610","reported_seq":24882,"reported_epoch":118,"state":"active+clean","last_fresh":"2026-02-23T13:11:17.123207+0000","last_change":"2026-02-23T13:11:13.989033+0000","last_active":"2026-02-23T13:11:17.123207+0000","last_peered":"2026-02-23T13:11:17.123207+0000","last_clean":"2026-02-23T13:11:17.123207+0000","last_became_active":"2026-02-23T13:11:13.986056+0000","last_became_peered":"2026-02-23T13:11:13.986056+0000","last_unstale":"2026-02-23T13:11:17.123207+0000","last_undegraded":"2026-02-23T13:11:17.123207+0000","last_fullsized":"2026-02-23T13:11:17.123207+0000","mapping_epoch":115,"log_start":"90'11600","ondisk_log_start":"90'11600","created":76,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:09:42.363073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:09:42.363073+0000","last_clean_scrub_stamp":"2026-02-23T13:09:42.363073+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":164495360,"num_objects":2510,"num_object_clones":0,"num_object_copies":7530,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2510,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21610,"num_write_kb":171520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1580,"num_bytes_recovered":103546880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"118'21490","reported_seq":24316,"reported_epoch":118,"state":"active+clean","last_fresh":"2026-02-23T13:11:17.114855+0000","last_change":"2026-02-23T13:10:46.187555+0000","last_active":"2026-02-23T13:11:17.114855+0000","last_peered":"2026-02-23T13:11:17.114855+0000","last_clean":"2026-02-23T13:11:17.114855+0000","last_became_active":"2026-02-23T13:10:35.982713+0000","last_became_peered":"2026-02-23T13:10:35.982713+0000","last_unstale":"2026-02-23T13:11:17.114855+0000","last_undegraded":"2026-02-23T13:11:17.114855+0000","last_fullsized":"2026-02-23T13:11:17.114855+0000","mapping_epoch":101,"log_start":"91'11400","ondisk_log_start":"91'11400","created":76,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"104'15600","last_scrub_stamp":"2026-02-23T13:10:46.187433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:09:42.363073+0000","last_clean_scrub_stamp":"2026-02-23T13:10:46.187433+0000","objects_scrubbed":1939,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T13:11:54.258632+0000","scrub_duration":1268,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":162922496,"num_objects":2486,"num_object_clones":0,"num_object_copies":7458,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2486,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21490,"num_write_kb":170496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1342,"num_bytes_recovered":87900160,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"28'99","reported_seq":215,"reported_epoch":118,"state":"active+clean","last_fresh":"2026-02-23T13:11:16.367983+0000","last_change":"2026-02-23T13:11:02.340041+0000","last_active":"2026-02-23T13:11:16.367983+0000","last_peered":"2026-02-23T13:11:16.367983+0000","last_clean":"2026-02-23T13:11:16.367983+0000","last_became_active":"2026-02-23T13:11:02.339822+0000","last_became_peered":"2026-02-23T13:11:02.339822+0000","last_unstale":"2026-02-23T13:11:16.367983+0000","last_undegraded":"2026-02-23T13:11:16.367983+0000","last_fullsized":"2026-02-23T13:11:16.367983+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'99","last_scrub_stamp":"2026-02-23T13:09:54.841853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:07:05.619208+0000","last_clean_scrub_stamp":"2026-02-23T13:09:54.841853+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T13:11:19.668596+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":122,"num_read_kb":228,"num_write":135,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":2671271960,"num_objects":40762,"num_object_clones":0,"num_object_copies":122286,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1043,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40762,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":345038,"num_write_kb":2743449,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25867,"num_bytes_recovered":1694564352,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":7980007424,"data_stored":7979999280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160938,"ondisk_log_size":160938,"up":48,"acting":45,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":122,"num_read_kb":228,"num_write":135,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":80,"seq":343597383698,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":734003200,"kb_used":2062408,"kb_used_data":1686452,"kb_used_omap":33,"kb_used_meta":375902,"kb_avail":731940792,"statfs":{"total":751619276800,"available":749507371008,"internally_reserved":0,"allocated":1726926848,"data_stored":1726093888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":34431,"internal_metadata":384924033},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":92,"seq":395136991244,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":734003200,"kb_used":2661948,"kb_used_data":2290920,"kb_used_omap":22,"kb_used_meta":370921,"kb_avail":731341252,"statfs":{"total":751619276800,"available":748893442048,"internally_reserved":0,"allocated":2345902080,"data_stored":2345061336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":23307,"internal_metadata":379823349},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":64,"seq":274877906969,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":734003200,"kb_used":2353996,"kb_used_data":1799160,"kb_used_omap":42,"kb_used_meta":554773,"kb_avail":731649204,"statfs":{"total":751619276800,"available":749208784896,"internally_reserved":0,"allocated":1842339840,"data_stored":1841506720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":43853,"internal_metadata":568087731},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":106,"seq":455266533380,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":734003200,"kb_used":2177144,"kb_used_data":2031012,"kb_used_omap":11,"kb_used_meta":146100,"kb_avail":731826056,"statfs":{"total":751619276800,"available":749389881344,"internally_reserved":0,"allocated":2079756288,"data_stored":2078935121,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12220,"internal_metadata":149606468},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2164482048,"data_stored":2164482048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1809342464,"data_stored":1809342464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":2309230592,"data_stored":2309226520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1696952320,"data_stored":1696948248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-23T13:11:18.977 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph pg force-recovery 3.8 2026-02-23T13:11:19.078 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:11:19.141 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:11:19.182 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:11:19.180+0000 7f942a74a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:11:19.182 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:11:19.181+0000 7f942a74a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:11:19.196 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:11:19.196 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:11:19.206 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:11:19.206 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:11:19.217 INFO:teuthology.orchestra.run.trial068.stdout:osd.2: {} 2026-02-23T13:11:19.218 INFO:teuthology.orchestra.run.trial068.stderr:osd.2: osd_enable_op_tracker = '' 2026-02-23T13:11:19.228 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:11:19.228 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:11:19.255 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:11:19.254+0000 7f038a68a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:11:19.256 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:11:19.254+0000 7f038a68a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:11:19.364 INFO:teuthology.orchestra.run.trial068.stderr:instructing pg(s) [3.8] on osd.1 to force-recovery; 2026-02-23T13:11:19.365 INFO:teuthology.orchestra.run.trial068.stderr: 2026-02-23T13:11:19.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:11:19 trial068 ceph-mon[38805]: pgmap v211: 17 pgs: 1 peering, 2 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped+forced_backfill, 13 active+clean; 2.5 GiB data, 8.8 GiB used, 2.7 TiB / 2.7 TiB avail; 239 B/s rd, 9.3 MiB/s wr, 1.68k op/s; 1043/122292 objects degraded (0.853%); 64 MiB/s, 1.02k objects/s recovering 2026-02-23T13:11:19.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:11:19 trial068 ceph-mon[38805]: osdmap e120: 4 total, 4 up, 4 in 2026-02-23T13:11:19.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:11:19 trial068 ceph-mon[38805]: Health check update: Degraded data redundancy: 1043/122292 objects degraded (0.853%), 3 pgs degraded (PG_DEGRADED) 2026-02-23T13:11:19.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:11:19 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2950024517' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:11:20.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:11:20 trial068 ceph-mon[38805]: 3.1 scrub starts 2026-02-23T13:11:20.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:11:20 trial068 ceph-mon[38805]: 3.1 scrub ok 2026-02-23T13:11:20.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:11:20 trial068 ceph-mon[38805]: from='client.14614 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2026-02-23T13:11:20.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:11:20 trial068 ceph-mon[38805]: 3.d scrub starts 2026-02-23T13:11:20.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:11:20 trial068 ceph-mon[38805]: 3.d scrub ok 2026-02-23T13:11:20.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:11:20 trial068 ceph-mon[38805]: from='client.14618 -' entity='client.admin' cmd=[{"prefix": "pg force-recovery", "pgid": ["3.8"], "target": ["mon-mgr", ""]}]: dispatch 2026-02-23T13:11:20.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:11:20 trial068 ceph-mon[38805]: osdmap e121: 4 total, 4 up, 4 in 2026-02-23T13:11:21.263 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:11:21.413 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:11:21.590 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:11:21 trial068 ceph-mon[38805]: 3.6 scrub starts 2026-02-23T13:11:21.590 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:11:21 trial068 ceph-mon[38805]: pgmap v214: 17 pgs: 1 peering, 2 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped+forced_backfill, 13 active+clean; 2.5 GiB data, 8.8 GiB used, 2.7 TiB / 2.7 TiB avail; 255 B/s rd, 9.9 MiB/s wr, 1.80k op/s; 1043/122292 objects degraded (0.853%); 68 MiB/s, 1.09k objects/s recovering 2026-02-23T13:11:21.590 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:11:21 trial068 ceph-mon[38805]: 3.4 scrub starts 2026-02-23T13:11:21.775 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:11:21.773+0000 7f14e2ba6640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:11:21.775 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:11:21.774+0000 7f14e2ba6640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:11:21.792 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:11:21.792 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:11:21.806 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:11:21.806 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:11:21.820 INFO:teuthology.orchestra.run.trial068.stdout:osd.2: {} 2026-02-23T13:11:21.821 INFO:teuthology.orchestra.run.trial068.stderr:osd.2: osd_enable_op_tracker = '' 2026-02-23T13:11:21.835 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:11:21.835 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:11:22.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:11:22 trial068 ceph-mon[38805]: 3.6 scrub ok 2026-02-23T13:11:22.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:11:22 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3547073886' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:11:22.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:11:22 trial068 ceph-mon[38805]: 3.4 scrub ok 2026-02-23T13:11:23.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:11:23 trial068 ceph-mon[38805]: pgmap v215: 17 pgs: 1 active+clean+scrubbing, 1 peering, 1 active+recovering+undersized+degraded+remapped, 14 active+clean; 2.2 GiB data, 8.3 GiB used, 2.7 TiB / 2.7 TiB avail; 173 B/s rd, 7.2 MiB/s wr, 1.98k op/s; 5/110076 objects degraded (0.005%); 57 MiB/s, 914 objects/s recovering 2026-02-23T13:11:23.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:11:23 trial068 ceph-mon[38805]: 3.b scrub starts 2026-02-23T13:11:23.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:11:23 trial068 ceph-mon[38805]: 3.b scrub ok 2026-02-23T13:11:23.907 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:11:24.057 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:11:24.182 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:11:23 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:11:23] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:11:24.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:11:23 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:11:23.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:11:24.403 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [1, 3, 2, 0] 2026-02-23T13:11:24.403 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2026-02-23T13:11:24.403 INFO:tasks.thrashosds.thrasher:Killing osd 2, live_osds are [1, 3, 2, 0] 2026-02-23T13:11:24.403 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2026-02-23T13:11:24.403 DEBUG:teuthology.orchestra.run.trial185:> sudo systemctl stop ceph-5f823084-10b8-11f1-be80-d404e6e7d460@osd.2 2026-02-23T13:11:24.432 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:11:24.397+0000 7fe3a6407640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:11:24.432 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:11:24.397+0000 7fe3a6407640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:11:24.432 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:11:24.433 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:11:24.433 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:11:24.433 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:11:24.433 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:11:24 trial068 ceph-mon[38805]: 3.f scrub starts 2026-02-23T13:11:24.434 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:11:24 trial068 ceph-mon[38805]: 3.7 scrub starts 2026-02-23T13:11:24.434 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:11:24 trial068 ceph-mon[38805]: Health check update: Degraded data redundancy: 5/110076 objects degraded (0.005%), 1 pg degraded (PG_DEGRADED) 2026-02-23T13:11:24.434 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:11:24 trial068 ceph-mon[38805]: osdmap e122: 4 total, 4 up, 4 in 2026-02-23T13:11:24.434 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:11:24 trial068 ceph-mon[38805]: 3.7 scrub ok 2026-02-23T13:11:24.447 INFO:teuthology.orchestra.run.trial068.stdout:osd.2: {} 2026-02-23T13:11:24.447 INFO:teuthology.orchestra.run.trial068.stderr:osd.2: osd_enable_op_tracker = '' 2026-02-23T13:11:24.476 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:11:24.476 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:11:24.857 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:11:24 trial185 systemd[1]: Stopping Ceph osd.2 for 5f823084-10b8-11f1-be80-d404e6e7d460... 2026-02-23T13:11:24.857 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:11:24 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-2[70709]: 2026-02-23T13:11:24.591+0000 7f1df63bf640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --osd-objectstore=bluestore (PID: 1) UID: 0 2026-02-23T13:11:24.858 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:11:24 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-2[70709]: 2026-02-23T13:11:24.591+0000 7f1df63bf640 -1 osd.2 122 *** Got signal Terminated *** 2026-02-23T13:11:24.858 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:11:24 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-2[70709]: 2026-02-23T13:11:24.591+0000 7f1df63bf640 -1 osd.2 122 *** Immediate shutdown (osd_fast_shutdown=true) *** 2026-02-23T13:11:25.823 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:11:25 trial185 podman[71778]: 2026-02-23 13:11:25.534251773 +0000 UTC m=+1.023294103 container died f197201319a0b8a31247924fd4ff0c9aef5d95a0d3ce4819befedd322cdb3899 (image=quay.ceph.io/ceph-ci/ceph@sha256:49621faaa845c39b2aae72e4fa24123ee9b2ac3fb8ae834753907068ee2972ea, name=ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-2, CEPH_SHA1=e371433fcd1390a0b82cfb18effc422c1f5d8c1d, io.buildah.version=1.41.3, FROM_IMAGE=quay.io/centos/centos:stream9, CEPH_REF=wip-shraddhaag-enable-cephadm-tests, OSD_FLAVOR=default, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, org.opencontainers.image.documentation=https://docs.ceph.com/, CEPH_GIT_REPO=https://github.com/ceph/ceph-ci.git, ceph=True, org.label-schema.schema-version=1.0, GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfsganesha-5/, org.label-schema.vendor=CentOS, org.opencontainers.image.authors=Ceph Release Team , org.label-schema.build-date=20260216) 2026-02-23T13:11:25.823 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:11:25 trial185 podman[71778]: 2026-02-23 13:11:25.547376637 +0000 UTC m=+1.036418967 container remove f197201319a0b8a31247924fd4ff0c9aef5d95a0d3ce4819befedd322cdb3899 (image=quay.ceph.io/ceph-ci/ceph@sha256:49621faaa845c39b2aae72e4fa24123ee9b2ac3fb8ae834753907068ee2972ea, name=ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-2, org.opencontainers.image.documentation=https://docs.ceph.com/, GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfsganesha-5/, CEPH_REF=wip-shraddhaag-enable-cephadm-tests, CEPH_GIT_REPO=https://github.com/ceph/ceph-ci.git, org.opencontainers.image.authors=Ceph Release Team , FROM_IMAGE=quay.io/centos/centos:stream9, org.label-schema.build-date=20260216, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.41.3, ceph=True, CEPH_SHA1=e371433fcd1390a0b82cfb18effc422c1f5d8c1d, OSD_FLAVOR=default, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS) 2026-02-23T13:11:25.823 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:11:25 trial185 bash[71778]: ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-2 2026-02-23T13:11:25.824 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:11:25 trial185 podman[71903]: 2026-02-23 13:11:25.67209022 +0000 UTC m=+0.014798878 container create b8d741eac328b52f45a396052ff43ba98cd65de7175cac55386ba09559428d4a (image=quay.ceph.io/ceph-ci/ceph@sha256:49621faaa845c39b2aae72e4fa24123ee9b2ac3fb8ae834753907068ee2972ea, name=ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-2-deactivate, CEPH_SHA1=e371433fcd1390a0b82cfb18effc422c1f5d8c1d, FROM_IMAGE=quay.io/centos/centos:stream9, ceph=True, CEPH_GIT_REPO=https://github.com/ceph/ceph-ci.git, GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfsganesha-5/, io.buildah.version=1.41.3, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, OSD_FLAVOR=default, org.opencontainers.image.authors=Ceph Release Team , org.label-schema.build-date=20260216, CEPH_REF=wip-shraddhaag-enable-cephadm-tests, org.label-schema.vendor=CentOS, org.opencontainers.image.documentation=https://docs.ceph.com/) 2026-02-23T13:11:25.824 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:11:25 trial185 podman[71903]: 2026-02-23 13:11:25.697406653 +0000 UTC m=+0.040115251 container init b8d741eac328b52f45a396052ff43ba98cd65de7175cac55386ba09559428d4a (image=quay.ceph.io/ceph-ci/ceph@sha256:49621faaa845c39b2aae72e4fa24123ee9b2ac3fb8ae834753907068ee2972ea, name=ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-2-deactivate, CEPH_REF=wip-shraddhaag-enable-cephadm-tests, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_SHA1=e371433fcd1390a0b82cfb18effc422c1f5d8c1d, FROM_IMAGE=quay.io/centos/centos:stream9, org.opencontainers.image.documentation=https://docs.ceph.com/, io.buildah.version=1.41.3, ceph=True, OSD_FLAVOR=default, org.label-schema.build-date=20260216, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_GIT_REPO=https://github.com/ceph/ceph-ci.git, GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfsganesha-5/, org.opencontainers.image.authors=Ceph Release Team ) 2026-02-23T13:11:25.824 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:11:25 trial185 podman[71903]: 2026-02-23 13:11:25.69987915 +0000 UTC m=+0.042587738 container start b8d741eac328b52f45a396052ff43ba98cd65de7175cac55386ba09559428d4a (image=quay.ceph.io/ceph-ci/ceph@sha256:49621faaa845c39b2aae72e4fa24123ee9b2ac3fb8ae834753907068ee2972ea, name=ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-2-deactivate, OSD_FLAVOR=default, org.label-schema.build-date=20260216, CEPH_SHA1=e371433fcd1390a0b82cfb18effc422c1f5d8c1d, org.opencontainers.image.authors=Ceph Release Team , org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_GIT_REPO=https://github.com/ceph/ceph-ci.git, org.opencontainers.image.documentation=https://docs.ceph.com/, org.label-schema.schema-version=1.0, GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfsganesha-5/, FROM_IMAGE=quay.io/centos/centos:stream9, io.buildah.version=1.41.3, CEPH_REF=wip-shraddhaag-enable-cephadm-tests, ceph=True, org.label-schema.license=GPLv2) 2026-02-23T13:11:25.824 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:11:25 trial185 podman[71903]: 2026-02-23 13:11:25.700407689 +0000 UTC m=+0.043116297 container attach b8d741eac328b52f45a396052ff43ba98cd65de7175cac55386ba09559428d4a (image=quay.ceph.io/ceph-ci/ceph@sha256:49621faaa845c39b2aae72e4fa24123ee9b2ac3fb8ae834753907068ee2972ea, name=ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-2-deactivate, io.buildah.version=1.41.3, org.label-schema.schema-version=1.0, CEPH_SHA1=e371433fcd1390a0b82cfb18effc422c1f5d8c1d, GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfsganesha-5/, org.opencontainers.image.authors=Ceph Release Team , org.label-schema.build-date=20260216, CEPH_GIT_REPO=https://github.com/ceph/ceph-ci.git, FROM_IMAGE=quay.io/centos/centos:stream9, org.opencontainers.image.documentation=https://docs.ceph.com/, CEPH_REF=wip-shraddhaag-enable-cephadm-tests, org.label-schema.name=CentOS Stream 9 Base Image, OSD_FLAVOR=default, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2) 2026-02-23T13:11:25.824 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:11:25 trial185 podman[71903]: 2026-02-23 13:11:25.666054073 +0000 UTC m=+0.008762671 image pull 63df4b83db6f95a8ba965919b3671cb27173117c23b5c950aaadc8913f56213c quay.ceph.io/ceph-ci/ceph@sha256:49621faaa845c39b2aae72e4fa24123ee9b2ac3fb8ae834753907068ee2972ea 2026-02-23T13:11:25.824 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:11:25 trial185 podman[71903]: 2026-02-23 13:11:25.82146534 +0000 UTC m=+0.164173938 container died b8d741eac328b52f45a396052ff43ba98cd65de7175cac55386ba09559428d4a (image=quay.ceph.io/ceph-ci/ceph@sha256:49621faaa845c39b2aae72e4fa24123ee9b2ac3fb8ae834753907068ee2972ea, name=ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-2-deactivate, OSD_FLAVOR=default, CEPH_REF=wip-shraddhaag-enable-cephadm-tests, CEPH_SHA1=e371433fcd1390a0b82cfb18effc422c1f5d8c1d, FROM_IMAGE=quay.io/centos/centos:stream9, org.opencontainers.image.documentation=https://docs.ceph.com/, GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfsganesha-5/, org.opencontainers.image.authors=Ceph Release Team , org.label-schema.build-date=20260216, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_GIT_REPO=https://github.com/ceph/ceph-ci.git, io.buildah.version=1.41.3, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image) 2026-02-23T13:11:25.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:11:25 trial068 ceph-mon[38805]: 3.f scrub ok 2026-02-23T13:11:25.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:11:25 trial068 ceph-mon[38805]: pgmap v217: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 1.7 GiB data, 7.1 GiB used, 2.7 TiB / 2.7 TiB avail; 480 KiB/s wr, 2.15k op/s; 11 MiB/s, 173 objects/s recovering 2026-02-23T13:11:25.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:11:25 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1164974119' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:11:25.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:11:25 trial068 ceph-mon[38805]: osd.2 marked itself down and dead 2026-02-23T13:11:25.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:11:25 trial068 ceph-mon[38805]: 3.5 scrub starts 2026-02-23T13:11:25.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:11:25 trial068 ceph-mon[38805]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 5/110076 objects degraded (0.005%), 1 pg degraded) 2026-02-23T13:11:25.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:11:25 trial068 ceph-mon[38805]: Cluster is now healthy 2026-02-23T13:11:25.856 DEBUG:teuthology.orchestra.run.trial185:> sudo pkill -f 'journalctl -f -n 0 -u ceph-5f823084-10b8-11f1-be80-d404e6e7d460@osd.2.service' 2026-02-23T13:11:25.894 DEBUG:teuthology.orchestra.run:got remote process result: None 2026-02-23T13:11:25.894 INFO:tasks.cephadm.osd.2:Stopped osd.2 2026-02-23T13:11:26.778 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:11:26.803 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:11:26 trial068 ceph-mon[38805]: Health check failed: 1 osds down (OSD_DOWN) 2026-02-23T13:11:26.803 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:11:26 trial068 ceph-mon[38805]: osdmap e123: 4 total, 3 up, 4 in 2026-02-23T13:11:26.927 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:11:27.205 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:11:27.159+0000 7f71abc94640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:11:27.205 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:11:27.160+0000 7f71abc94640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:11:27.205 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:11:27.206 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:11:27.206 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:11:27.206 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:11:27.206 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:11:27.207 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:11:27.207 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:11:27.374 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:11:27.375 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:11:27.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:11:27 trial068 ceph-mon[38805]: pgmap v219: 17 pgs: 4 stale+active+clean, 13 active+clean; 1.6 GiB data, 6.3 GiB used, 2.7 TiB / 2.7 TiB avail; 418 KiB/s wr, 2.17k op/s; 9.5 MiB/s, 151 objects/s recovering 2026-02-23T13:11:27.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:11:27 trial068 ceph-mon[38805]: osdmap e124: 4 total, 3 up, 4 in 2026-02-23T13:11:27.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:11:27 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1951947237' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:11:28.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:11:28 trial068 ceph-mon[38805]: Health check failed: Degraded data redundancy: 9474/50490 objects degraded (18.764%), 11 pgs degraded (PG_DEGRADED) 2026-02-23T13:11:29.375 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:11:29.526 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:11:29.655 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:11:29.653+0000 7fbb8edfa640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:11:29.656 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:11:29.654+0000 7fbb8edfa640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:11:29.656 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:11:29 trial068 ceph-mon[38805]: pgmap v221: 17 pgs: 2 stale+active+clean, 11 active+undersized+degraded, 4 active+clean; 1.0 GiB data, 5.3 GiB used, 2.7 TiB / 2.7 TiB avail; 0 B/s wr, 3.31k op/s; 9474/50490 objects degraded (18.764%); 53 KiB/s, 1 objects/s recovering 2026-02-23T13:11:29.674 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:11:29.674 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:11:29.689 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:11:29.690 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:11:29.690 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:11:29.704 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:11:29.704 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:11:29.747 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:11:29.748 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:11:30.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:11:30 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3334086493' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:11:30.895 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [2] live_osds: [1, 3, 0] 2026-02-23T13:11:30.895 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2026-02-23T13:11:30.895 INFO:tasks.thrashosds.thrasher:Reviving osd 2 2026-02-23T13:11:30.895 INFO:tasks.cephadm.osd.2:Restarting osd.2 (starting--it wasn't running)... 2026-02-23T13:11:30.896 DEBUG:teuthology.orchestra.run.trial185:osd.2> sudo journalctl -f -n 0 -u ceph-5f823084-10b8-11f1-be80-d404e6e7d460@osd.2.service 2026-02-23T13:11:30.898 DEBUG:teuthology.orchestra.run.trial185:> sudo systemctl start ceph-5f823084-10b8-11f1-be80-d404e6e7d460@osd.2 2026-02-23T13:11:31.175 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:11:30 trial185 systemd[1]: Starting Ceph osd.2 for 5f823084-10b8-11f1-be80-d404e6e7d460... 2026-02-23T13:11:31.176 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:11:31 trial185 podman[72129]: 2026-02-23 13:11:31.071655029 +0000 UTC m=+0.018033522 container create f607700909d788190d56e77036d6e349d18a4f2baa8cfe70c0063f540edb7f40 (image=quay.ceph.io/ceph-ci/ceph@sha256:49621faaa845c39b2aae72e4fa24123ee9b2ac3fb8ae834753907068ee2972ea, name=ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-2-activate, org.label-schema.license=GPLv2, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.opencontainers.image.authors=Ceph Release Team , GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfsganesha-5/, org.label-schema.build-date=20260216, OSD_FLAVOR=default, CEPH_REF=wip-shraddhaag-enable-cephadm-tests, CEPH_SHA1=e371433fcd1390a0b82cfb18effc422c1f5d8c1d, io.buildah.version=1.41.3, FROM_IMAGE=quay.io/centos/centos:stream9, org.label-schema.schema-version=1.0, CEPH_GIT_REPO=https://github.com/ceph/ceph-ci.git, org.label-schema.vendor=CentOS, org.opencontainers.image.documentation=https://docs.ceph.com/) 2026-02-23T13:11:31.176 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:11:31 trial185 podman[72129]: 2026-02-23 13:11:31.091837252 +0000 UTC m=+0.038215685 container init f607700909d788190d56e77036d6e349d18a4f2baa8cfe70c0063f540edb7f40 (image=quay.ceph.io/ceph-ci/ceph@sha256:49621faaa845c39b2aae72e4fa24123ee9b2ac3fb8ae834753907068ee2972ea, name=ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-2-activate, org.label-schema.build-date=20260216, CEPH_GIT_REPO=https://github.com/ceph/ceph-ci.git, CEPH_REF=wip-shraddhaag-enable-cephadm-tests, FROM_IMAGE=quay.io/centos/centos:stream9, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_SHA1=e371433fcd1390a0b82cfb18effc422c1f5d8c1d, GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfsganesha-5/, io.buildah.version=1.41.3, org.opencontainers.image.authors=Ceph Release Team , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, OSD_FLAVOR=default, org.opencontainers.image.documentation=https://docs.ceph.com/) 2026-02-23T13:11:31.176 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:11:31 trial185 podman[72129]: 2026-02-23 13:11:31.094879616 +0000 UTC m=+0.041258039 container start f607700909d788190d56e77036d6e349d18a4f2baa8cfe70c0063f540edb7f40 (image=quay.ceph.io/ceph-ci/ceph@sha256:49621faaa845c39b2aae72e4fa24123ee9b2ac3fb8ae834753907068ee2972ea, name=ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-2-activate, FROM_IMAGE=quay.io/centos/centos:stream9, org.opencontainers.image.authors=Ceph Release Team , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_GIT_REPO=https://github.com/ceph/ceph-ci.git, org.label-schema.vendor=CentOS, org.opencontainers.image.documentation=https://docs.ceph.com/, CEPH_SHA1=e371433fcd1390a0b82cfb18effc422c1f5d8c1d, org.label-schema.build-date=20260216, io.buildah.version=1.41.3, org.label-schema.license=GPLv2, ceph=True, OSD_FLAVOR=default, CEPH_REF=wip-shraddhaag-enable-cephadm-tests, GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfsganesha-5/) 2026-02-23T13:11:31.176 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:11:31 trial185 podman[72129]: 2026-02-23 13:11:31.095478918 +0000 UTC m=+0.041857361 container attach f607700909d788190d56e77036d6e349d18a4f2baa8cfe70c0063f540edb7f40 (image=quay.ceph.io/ceph-ci/ceph@sha256:49621faaa845c39b2aae72e4fa24123ee9b2ac3fb8ae834753907068ee2972ea, name=ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-2-activate, org.label-schema.license=GPLv2, OSD_FLAVOR=default, CEPH_REF=wip-shraddhaag-enable-cephadm-tests, io.buildah.version=1.41.3, org.opencontainers.image.authors=Ceph Release Team , org.opencontainers.image.documentation=https://docs.ceph.com/, GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfsganesha-5/, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20260216, CEPH_GIT_REPO=https://github.com/ceph/ceph-ci.git, CEPH_SHA1=e371433fcd1390a0b82cfb18effc422c1f5d8c1d, org.label-schema.name=CentOS Stream 9 Base Image, FROM_IMAGE=quay.io/centos/centos:stream9) 2026-02-23T13:11:31.176 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:11:31 trial185 podman[72129]: 2026-02-23 13:11:31.064928119 +0000 UTC m=+0.011306542 image pull 63df4b83db6f95a8ba965919b3671cb27173117c23b5c950aaadc8913f56213c quay.ceph.io/ceph-ci/ceph@sha256:49621faaa845c39b2aae72e4fa24123ee9b2ac3fb8ae834753907068ee2972ea 2026-02-23T13:11:31.176 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:11:31 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-2-activate[72171]: Running command: /usr/bin/ceph-authtool --gen-print-key 2026-02-23T13:11:31.318 INFO:tasks.radosbench.radosbench.0.trial068.stdout:Removed 42912 objects 2026-02-23T13:11:31.319 INFO:tasks.radosbench.radosbench.0.trial068.stdout:Clean up completed and total clean up time :15.8458 2026-02-23T13:11:31.475 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:11:31 trial185 bash[72129]: Running command: /usr/bin/ceph-authtool --gen-print-key 2026-02-23T13:11:31.475 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:11:31 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-2-activate[72171]: Running command: /usr/bin/ceph-authtool --gen-print-key 2026-02-23T13:11:31.475 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:11:31 trial185 bash[72129]: Running command: /usr/bin/ceph-authtool --gen-print-key 2026-02-23T13:11:31.475 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:11:31 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-2-activate[72171]: --> Failed to activate via raw: did not find any matching OSD to activate 2026-02-23T13:11:31.726 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:11:31 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-2-activate[72171]: Running command: /usr/bin/ceph-authtool --gen-print-key 2026-02-23T13:11:31.726 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:11:31 trial185 bash[72129]: --> Failed to activate via raw: did not find any matching OSD to activate 2026-02-23T13:11:31.726 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:11:31 trial185 bash[72129]: Running command: /usr/bin/ceph-authtool --gen-print-key 2026-02-23T13:11:31.727 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:11:31 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-2-activate[72171]: Running command: /usr/bin/ceph-authtool --gen-print-key 2026-02-23T13:11:31.727 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:11:31 trial185 bash[72129]: Running command: /usr/bin/ceph-authtool --gen-print-key 2026-02-23T13:11:31.727 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:11:31 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-2-activate[72171]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2026-02-23T13:11:31.727 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:11:31 trial185 bash[72129]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2026-02-23T13:11:31.727 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:11:31 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-2-activate[72171]: Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/vg_nvme/lv_4 --path /var/lib/ceph/osd/ceph-2 --no-mon-config 2026-02-23T13:11:31.727 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:11:31 trial185 bash[72129]: Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/vg_nvme/lv_4 --path /var/lib/ceph/osd/ceph-2 --no-mon-config 2026-02-23T13:11:31.727 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:11:31 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-2-activate[72171]: Running command: /usr/bin/ln -snf /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-2/block 2026-02-23T13:11:31.727 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:11:31 trial185 bash[72129]: Running command: /usr/bin/ln -snf /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-2/block 2026-02-23T13:11:31.727 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:11:31 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-2-activate[72171]: Running command: /usr/bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-2/block 2026-02-23T13:11:31.728 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:11:31 trial185 bash[72129]: Running command: /usr/bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-2/block 2026-02-23T13:11:31.728 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:11:31 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-2-activate[72171]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2026-02-23T13:11:31.728 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:11:31 trial185 bash[72129]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2026-02-23T13:11:31.728 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:11:31 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-2-activate[72171]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2026-02-23T13:11:31.728 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:11:31 trial185 bash[72129]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2026-02-23T13:11:31.728 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:11:31 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-2-activate[72171]: --> ceph-volume lvm activate successful for osd ID: 2 2026-02-23T13:11:31.728 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:11:31 trial185 bash[72129]: --> ceph-volume lvm activate successful for osd ID: 2 2026-02-23T13:11:31.728 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:11:31 trial185 podman[72129]: 2026-02-23 13:11:31.598669627 +0000 UTC m=+0.545048060 container died f607700909d788190d56e77036d6e349d18a4f2baa8cfe70c0063f540edb7f40 (image=quay.ceph.io/ceph-ci/ceph@sha256:49621faaa845c39b2aae72e4fa24123ee9b2ac3fb8ae834753907068ee2972ea, name=ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-2-activate, CEPH_SHA1=e371433fcd1390a0b82cfb18effc422c1f5d8c1d, org.label-schema.license=GPLv2, GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfsganesha-5/, org.label-schema.name=CentOS Stream 9 Base Image, OSD_FLAVOR=default, org.label-schema.schema-version=1.0, org.label-schema.build-date=20260216, io.buildah.version=1.41.3, org.label-schema.vendor=CentOS, org.opencontainers.image.authors=Ceph Release Team , ceph=True, CEPH_GIT_REPO=https://github.com/ceph/ceph-ci.git, CEPH_REF=wip-shraddhaag-enable-cephadm-tests, org.opencontainers.image.documentation=https://docs.ceph.com/, FROM_IMAGE=quay.io/centos/centos:stream9) 2026-02-23T13:11:31.728 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:11:31 trial185 podman[72129]: 2026-02-23 13:11:31.611274528 +0000 UTC m=+0.557652961 container remove f607700909d788190d56e77036d6e349d18a4f2baa8cfe70c0063f540edb7f40 (image=quay.ceph.io/ceph-ci/ceph@sha256:49621faaa845c39b2aae72e4fa24123ee9b2ac3fb8ae834753907068ee2972ea, name=ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-2-activate, org.label-schema.schema-version=1.0, org.opencontainers.image.authors=Ceph Release Team , CEPH_REF=wip-shraddhaag-enable-cephadm-tests, io.buildah.version=1.41.3, OSD_FLAVOR=default, org.label-schema.build-date=20260216, org.label-schema.vendor=CentOS, FROM_IMAGE=quay.io/centos/centos:stream9, org.opencontainers.image.documentation=https://docs.ceph.com/, org.label-schema.license=GPLv2, CEPH_GIT_REPO=https://github.com/ceph/ceph-ci.git, GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfsganesha-5/, ceph=True, CEPH_SHA1=e371433fcd1390a0b82cfb18effc422c1f5d8c1d, org.label-schema.name=CentOS Stream 9 Base Image) 2026-02-23T13:11:31.728 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:11:31 trial185 podman[72417]: 2026-02-23 13:11:31.6872429 +0000 UTC m=+0.015090655 container create 4fccd31daa59408fdd28a1321c8ffdabdb54aa6380b72a4cfa1e27c74bd04cbb (image=quay.ceph.io/ceph-ci/ceph@sha256:49621faaa845c39b2aae72e4fa24123ee9b2ac3fb8ae834753907068ee2972ea, name=ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-2, org.opencontainers.image.authors=Ceph Release Team , CEPH_SHA1=e371433fcd1390a0b82cfb18effc422c1f5d8c1d, OSD_FLAVOR=default, ceph=True, org.label-schema.license=GPLv2, io.buildah.version=1.41.3, org.opencontainers.image.documentation=https://docs.ceph.com/, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfsganesha-5/, CEPH_REF=wip-shraddhaag-enable-cephadm-tests, org.label-schema.name=CentOS Stream 9 Base Image, FROM_IMAGE=quay.io/centos/centos:stream9, CEPH_GIT_REPO=https://github.com/ceph/ceph-ci.git, org.label-schema.build-date=20260216) 2026-02-23T13:11:31.728 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:11:31 trial185 podman[72417]: 2026-02-23 13:11:31.718808698 +0000 UTC m=+0.046656453 container init 4fccd31daa59408fdd28a1321c8ffdabdb54aa6380b72a4cfa1e27c74bd04cbb (image=quay.ceph.io/ceph-ci/ceph@sha256:49621faaa845c39b2aae72e4fa24123ee9b2ac3fb8ae834753907068ee2972ea, name=ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-2, org.label-schema.build-date=20260216, OSD_FLAVOR=default, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, FROM_IMAGE=quay.io/centos/centos:stream9, org.opencontainers.image.documentation=https://docs.ceph.com/, io.buildah.version=1.41.3, CEPH_SHA1=e371433fcd1390a0b82cfb18effc422c1f5d8c1d, GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfsganesha-5/, ceph=True, CEPH_REF=wip-shraddhaag-enable-cephadm-tests, org.label-schema.vendor=CentOS, org.opencontainers.image.authors=Ceph Release Team , org.label-schema.name=CentOS Stream 9 Base Image, CEPH_GIT_REPO=https://github.com/ceph/ceph-ci.git) 2026-02-23T13:11:31.729 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:11:31 trial185 podman[72417]: 2026-02-23 13:11:31.721236272 +0000 UTC m=+0.049084027 container start 4fccd31daa59408fdd28a1321c8ffdabdb54aa6380b72a4cfa1e27c74bd04cbb (image=quay.ceph.io/ceph-ci/ceph@sha256:49621faaa845c39b2aae72e4fa24123ee9b2ac3fb8ae834753907068ee2972ea, name=ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-2, io.buildah.version=1.41.3, CEPH_SHA1=e371433fcd1390a0b82cfb18effc422c1f5d8c1d, OSD_FLAVOR=default, org.label-schema.vendor=CentOS, CEPH_REF=wip-shraddhaag-enable-cephadm-tests, org.opencontainers.image.authors=Ceph Release Team , org.label-schema.build-date=20260216, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, org.opencontainers.image.documentation=https://docs.ceph.com/, FROM_IMAGE=quay.io/centos/centos:stream9, CEPH_GIT_REPO=https://github.com/ceph/ceph-ci.git, GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfsganesha-5/) 2026-02-23T13:11:31.729 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:11:31 trial185 bash[72417]: 4fccd31daa59408fdd28a1321c8ffdabdb54aa6380b72a4cfa1e27c74bd04cbb 2026-02-23T13:11:31.729 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:11:31 trial185 podman[72417]: 2026-02-23 13:11:31.681605399 +0000 UTC m=+0.009453154 image pull 63df4b83db6f95a8ba965919b3671cb27173117c23b5c950aaadc8913f56213c quay.ceph.io/ceph-ci/ceph@sha256:49621faaa845c39b2aae72e4fa24123ee9b2ac3fb8ae834753907068ee2972ea 2026-02-23T13:11:31.729 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:11:31 trial185 systemd[1]: Started Ceph osd.2 for 5f823084-10b8-11f1-be80-d404e6e7d460. 2026-02-23T13:11:31.730 DEBUG:teuthology.orchestra.run.trial185:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph daemon osd.2 dump_ops_in_flight 2026-02-23T13:11:31.748 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:11:31.773 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:11:31 trial068 ceph-mon[38805]: pgmap v222: 17 pgs: 2 stale+active+clean, 11 active+undersized+degraded, 4 active+clean; 1.0 GiB data, 5.3 GiB used, 2.7 TiB / 2.7 TiB avail; 0 B/s wr, 1.82k op/s; 9474/50490 objects degraded (18.764%); 0 B/s, 0 objects/s recovering 2026-02-23T13:11:31.773 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:11:31 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2026-02-23T13:11:31.774 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:11:31 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-23T13:11:31.774 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:11:31 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2026-02-23T13:11:31.774 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:11:31 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T13:11:31.774 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:11:31 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2026-02-23T13:11:31.851 INFO:teuthology.orchestra.run.trial185.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/config/ceph.conf 2026-02-23T13:11:31.866 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:11:31.960 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:11:31.958+0000 7ff6eabfd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:11:31.960 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:11:31.958+0000 7ff6eabfd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:11:31.964 INFO:teuthology.orchestra.run.trial185.stderr:no valid command found; 10 closest matches: 2026-02-23T13:11:31.964 INFO:teuthology.orchestra.run.trial185.stderr:0 2026-02-23T13:11:31.964 INFO:teuthology.orchestra.run.trial185.stderr:1 2026-02-23T13:11:31.964 INFO:teuthology.orchestra.run.trial185.stderr:2 2026-02-23T13:11:31.964 INFO:teuthology.orchestra.run.trial185.stderr:abort 2026-02-23T13:11:31.964 INFO:teuthology.orchestra.run.trial185.stderr:assert 2026-02-23T13:11:31.964 INFO:teuthology.orchestra.run.trial185.stderr:bluestore collections 2026-02-23T13:11:31.964 INFO:teuthology.orchestra.run.trial185.stderr:bluestore compression stats [] 2026-02-23T13:11:31.964 INFO:teuthology.orchestra.run.trial185.stderr:bluestore list [] [] 2026-02-23T13:11:31.965 INFO:teuthology.orchestra.run.trial185.stderr:bluestore onode metadata 2026-02-23T13:11:31.965 INFO:teuthology.orchestra.run.trial185.stderr:bluestore show sharding 2026-02-23T13:11:31.965 INFO:teuthology.orchestra.run.trial185.stderr:admin_socket: invalid command 2026-02-23T13:11:31.972 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:11:31.973 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:11:31.983 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:11:31.983 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:11:31.983 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:11:31.985 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:11:31 trial185 ceph-osd[72443]: -- 10.20.193.185:0/2883988803 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x3c49c00 msgr2=0x4a86000 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).process 2026-02-23T13:11:31.986 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:11:31 trial185 ceph-osd[72443]: -- 10.20.193.185:0/2883988803 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x3c49c00 msgr2=0x4a86000 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read continue len=32 2026-02-23T13:11:31.986 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:11:31 trial185 ceph-osd[72443]: --2- 10.20.193.185:0/2883988803 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x3c49c00 0x4a86000 crc :-1 s=READY pgs=535 gs=1 cs=0 l=1 c_cookie=16974687338774721412 s_cookie=0 reconnecting=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main r=0 2026-02-23T13:11:31.986 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:11:31 trial185 ceph-osd[72443]: --2- 10.20.193.185:0/2883988803 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x3c49c00 0x4a86000 crc :-1 s=THROTTLE_MESSAGE pgs=535 gs=1 cs=0 l=1 c_cookie=16974687338774721412 s_cookie=0 reconnecting=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).throttle_message 2026-02-23T13:11:31.986 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:11:31 trial185 ceph-osd[72443]: --2- 10.20.193.185:0/2883988803 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x3c49c00 0x4a86000 crc :-1 s=THROTTLE_BYTES pgs=535 gs=1 cs=0 l=1 c_cookie=16974687338774721412 s_cookie=0 reconnecting=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).throttle_bytes 2026-02-23T13:11:31.986 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:11:31 trial185 ceph-osd[72443]: --2- 10.20.193.185:0/2883988803 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x3c49c00 0x4a86000 crc :-1 s=THROTTLE_DISPATCH_QUEUE pgs=535 gs=1 cs=0 l=1 c_cookie=16974687338774721412 s_cookie=0 reconnecting=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).throttle_dispatch_queue 2026-02-23T13:11:31.986 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:11:31 trial185 ceph-osd[72443]: --2- 10.20.193.185:0/2883988803 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x3c49c00 0x4a86000 crc :-1 s=THROTTLE_DONE pgs=535 gs=1 cs=0 l=1 c_cookie=16974687338774721412 s_cookie=0 reconnecting=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).read_frame_segment seg_idx=0 2026-02-23T13:11:31.986 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:11:31 trial185 ceph-osd[72443]: -- 10.20.193.185:0/2883988803 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x3c49c00 msgr2=0x4a86000 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read start len=45 2026-02-23T13:11:31.986 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:11:31 trial185 ceph-osd[72443]: --2- 10.20.193.185:0/2883988803 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x3c49c00 0x4a86000 crc :-1 s=THROTTLE_DONE pgs=535 gs=1 cs=0 l=1 c_cookie=16974687338774721412 s_cookie=0 reconnecting=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_segment r=0 2026-02-23T13:11:31.986 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:11:31 trial185 ceph-osd[72443]: --2- 10.20.193.185:0/2883988803 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x3c49c00 0x4a86000 crc :-1 s=THROTTLE_DONE pgs=535 gs=1 cs=0 l=1 c_cookie=16974687338774721412 s_cookie=0 reconnecting=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).read_frame_segment seg_idx=1 2026-02-23T13:11:31.986 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:11:31 trial185 ceph-osd[72443]: -- 10.20.193.185:0/2883988803 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x3c49c00 msgr2=0x4a86000 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read start len=194 2026-02-23T13:11:31.987 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:11:31 trial185 ceph-osd[72443]: --2- 10.20.193.185:0/2883988803 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x3c49c00 0x4a86000 crc :-1 s=THROTTLE_DONE pgs=535 gs=1 cs=0 l=1 c_cookie=16974687338774721412 s_cookie=0 reconnecting=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_segment r=0 2026-02-23T13:11:31.987 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:11:31 trial185 ceph-osd[72443]: -- 10.20.193.185:0/2883988803 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x3c49c00 msgr2=0x4a86000 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read start len=13 2026-02-23T13:11:31.987 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:11:31 trial185 ceph-osd[72443]: --2- 10.20.193.185:0/2883988803 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x3c49c00 0x4a86000 crc :-1 s=THROTTLE_DONE pgs=535 gs=1 cs=0 l=1 c_cookie=16974687338774721412 s_cookie=0 reconnecting=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_epilogue_main r=0 2026-02-23T13:11:31.987 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:11:31 trial185 ceph-osd[72443]: --2- 10.20.193.185:0/2883988803 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x3c49c00 0x4a86000 crc :-1 s=THROTTLE_DONE pgs=535 gs=1 cs=0 l=1 c_cookie=16974687338774721412 s_cookie=0 reconnecting=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_dispatch tag=17 2026-02-23T13:11:31.987 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:11:31 trial185 ceph-osd[72443]: --2- 10.20.193.185:0/2883988803 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x3c49c00 0x4a86000 crc :-1 s=THROTTLE_DONE pgs=535 gs=1 cs=0 l=1 c_cookie=16974687338774721412 s_cookie=0 reconnecting=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_message 2026-02-23T13:11:31.987 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:11:31 trial185 ceph-osd[72443]: --2- 10.20.193.185:0/2883988803 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x3c49c00 0x4a86000 crc :-1 s=THROTTLE_DONE pgs=535 gs=1 cs=0 l=1 c_cookie=16974687338774721412 s_cookie=0 reconnecting=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_message got 194 + 0 + 0 byte message. envelope type=18 src mon.0 off 0 2026-02-23T13:11:31.987 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:11:31 trial185 ceph-osd[72443]: --2- 10.20.193.185:0/2883988803 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x3c49c00 0x4a86000 crc :-1 s=READ_MESSAGE_COMPLETE pgs=535 gs=1 cs=0 l=1 c_cookie=16974687338774721412 s_cookie=0 reconnecting=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_message received message m=0x3cad860 seq=4 from=mon.0 type=18 auth_reply(proto 2 0 (0) Success) 2026-02-23T13:11:31.987 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:11:31 trial185 ceph-osd[72443]: -- 10.20.193.185:0/2883988803 queue 0x3cad860 prio 196 2026-02-23T13:11:31.987 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:11:31 trial185 ceph-osd[72443]: --2- 10.20.193.185:0/2883988803 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x3c49c00 0x4a86000 crc :-1 s=READY pgs=535 gs=1 cs=0 l=1 c_cookie=16974687338774721412 s_cookie=0 reconnecting=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).read_frame 2026-02-23T13:11:31.987 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:11:31 trial185 ceph-osd[72443]: -- 10.20.193.185:0/2883988803 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x3c49c00 msgr2=0x4a86000 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read start len=32 2026-02-23T13:11:31.987 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:11:31 trial185 ceph-osd[72443]: -- 10.20.193.185:0/2883988803 <== mon.0 v2:10.20.193.68:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (crc 0 0 0) 0x3cad860 con 0x3c49c00 2026-02-23T13:11:31.987 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:11:31 trial185 ceph-osd[72443]: -- 10.20.193.185:0/2883988803 dispatch_throttle_release 194 to dispatch throttler 194/104857600 2026-02-23T13:11:31.988 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:11:31 trial185 ceph-osd[72443]: -- 10.20.193.185:0/2883988803 done calling dispatch on 0x3cad860 2026-02-23T13:11:31.988 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:11:31 trial185 ceph-osd[72443]: WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:11:31.988 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:11:31 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-2[72439]: 2026-02-23T13:11:31.736+0000 7fe90ac1d780 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:11:31.988 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:11:31 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-2[72439]: 2026-02-23T13:11:31.736+0000 7fe90ac1d780 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:11:31.988 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:11:31 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-2[72439]: 2026-02-23T13:11:31.736+0000 7fe90ac1d780 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:11:31.988 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:11:31 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-2[72439]: 2026-02-23T13:11:31.972+0000 7fe90ac1d780 -1 Falling back to public interface 2026-02-23T13:11:31.993 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:11:31.994 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:11:32.000 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-23T13:11:32.000 INFO:tasks.cephadm.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-23T13:11:32.023 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:11:32.024 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:11:32.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:11:32 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2395557438' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:11:33.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:11:33 trial068 ceph-mon[38805]: pgmap v223: 17 pgs: 1 active+undersized, 14 active+undersized+degraded, 2 active+clean; 438 MiB data, 4.0 GiB used, 2.7 TiB / 2.7 TiB avail; 0 B/s wr, 2.66k op/s; 6167/21021 objects degraded (29.337%); 0 B/s, 0 objects/s recovering 2026-02-23T13:11:34.025 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:11:34.143 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:11:34.234 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_1 2026-02-23T13:11:34.235 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd pool rm unique_pool_1 unique_pool_1 --yes-i-really-really-mean-it 2026-02-23T13:11:34.238 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:11:34.237+0000 7f8d09d4b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:11:34.239 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:11:34.237+0000 7f8d09d4b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:11:34.239 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:11:33 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:11:33] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:11:34.251 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:11:34.251 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:11:34.262 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:11:34.262 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:11:34.262 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:11:34.273 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:11:34.273 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:11:34.304 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:11:34.305 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:11:34.356 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:11:34.357 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:11:33 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:11:33.974Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:11:34.459 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:11:34.457+0000 7fa5b31f8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:11:34.459 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:11:34.458+0000 7fa5b31f8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:11:34.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:11:34 trial068 ceph-mon[38805]: Health check update: Degraded data redundancy: 6167/21021 objects degraded (29.337%), 14 pgs degraded (PG_DEGRADED) 2026-02-23T13:11:34.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:11:34 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/196602895' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:11:35.448 INFO:teuthology.orchestra.run.trial068.stderr:pool 'unique_pool_1' removed 2026-02-23T13:11:35.556 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2026-02-23T13:11:35.556 INFO:tasks.radosbench:Beginning radosbench... 2026-02-23T13:11:35.556 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_2 2026-02-23T13:11:35.557 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd pool create unique_pool_2 16 2026-02-23T13:11:35.673 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:11:35.725 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:11:35 trial068 ceph-mon[38805]: pgmap v224: 17 pgs: 5 active+undersized, 10 active+undersized+degraded, 2 active+clean; 50 MiB data, 3.3 GiB used, 2.7 TiB / 2.7 TiB avail; 0 B/s wr, 2.86k op/s; 793/2379 objects degraded (33.333%); 0 B/s, 0 objects/s recovering 2026-02-23T13:11:35.725 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:11:35 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/148298768' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true} : dispatch 2026-02-23T13:11:35.784 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:11:35.782+0000 7f5abe33f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:11:35.784 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:11:35.783+0000 7f5abe33f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:11:36.306 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:11:36.424 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:11:36.453 INFO:teuthology.orchestra.run.trial068.stderr:pool 'unique_pool_2' created 2026-02-23T13:11:36.493 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd pool application enable unique_pool_2 rados --yes-i-really-mean-it || true 2026-02-23T13:11:36.525 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:11:36.524+0000 7ffbf7e7a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:11:36.525 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:11:36.524+0000 7ffbf7e7a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:11:36.538 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:11:36.538 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:11:36.548 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:11:36.549 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:11:36.549 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:11:36.559 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:11:36.560 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:11:36.587 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:11:36 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/148298768' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]': finished 2026-02-23T13:11:36.587 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:11:36 trial068 ceph-mon[38805]: osdmap e125: 4 total, 3 up, 4 in 2026-02-23T13:11:36.587 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:11:36 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2526494796' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16} : dispatch 2026-02-23T13:11:36.588 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:11:36.588 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:11:36.611 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:11:36.713 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:11:36.711+0000 7fc140c92640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:11:36.713 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:11:36.711+0000 7fc140c92640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:11:37.002 DEBUG:teuthology.orchestra.run.trial185:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph daemon osd.2 dump_ops_in_flight 2026-02-23T13:11:37.028 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:11:36 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-2[72439]: 2026-02-23T13:11:36.714+0000 7fe90ac1d780 -1 osd.2 122 log_to_monitors true 2026-02-23T13:11:37.123 INFO:teuthology.orchestra.run.trial185.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/config/ceph.conf 2026-02-23T13:11:37.236 INFO:teuthology.orchestra.run.trial185.stderr:no valid command found; 10 closest matches: 2026-02-23T13:11:37.236 INFO:teuthology.orchestra.run.trial185.stderr:0 2026-02-23T13:11:37.236 INFO:teuthology.orchestra.run.trial185.stderr:1 2026-02-23T13:11:37.236 INFO:teuthology.orchestra.run.trial185.stderr:2 2026-02-23T13:11:37.236 INFO:teuthology.orchestra.run.trial185.stderr:abort 2026-02-23T13:11:37.237 INFO:teuthology.orchestra.run.trial185.stderr:assert 2026-02-23T13:11:37.237 INFO:teuthology.orchestra.run.trial185.stderr:bluefs debug_inject_read_zeros 2026-02-23T13:11:37.237 INFO:teuthology.orchestra.run.trial185.stderr:bluefs files list 2026-02-23T13:11:37.237 INFO:teuthology.orchestra.run.trial185.stderr:bluefs stats 2026-02-23T13:11:37.237 INFO:teuthology.orchestra.run.trial185.stderr:bluestore allocator dump block 2026-02-23T13:11:37.237 INFO:teuthology.orchestra.run.trial185.stderr:bluestore allocator fragmentation block 2026-02-23T13:11:37.237 INFO:teuthology.orchestra.run.trial185.stderr:admin_socket: invalid command 2026-02-23T13:11:37.265 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-23T13:11:37.266 INFO:tasks.cephadm.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-23T13:11:37.454 INFO:teuthology.orchestra.run.trial068.stderr:enabled application 'rados' on pool 'unique_pool_2' 2026-02-23T13:11:37.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:11:37 trial068 ceph-mon[38805]: pgmap v226: 1 pgs: 1 active+undersized+degraded; 577 KiB data, 3.1 GiB used, 2.7 TiB / 2.7 TiB avail; 2/6 objects degraded (33.333%) 2026-02-23T13:11:37.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:11:37 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2526494796' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]': finished 2026-02-23T13:11:37.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:11:37 trial068 ceph-mon[38805]: osdmap e126: 4 total, 3 up, 4 in 2026-02-23T13:11:37.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:11:37 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3283009579' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:11:37.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:11:37 trial068 ceph-mon[38805]: from='osd.2 [v2:10.20.193.185:6800/1511048998,v1:10.20.193.185:6801/1511048998]' entity='osd.2' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]} : dispatch 2026-02-23T13:11:37.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:11:37 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/373623974' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2026-02-23T13:11:37.857 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:11:37 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-2[72439]: 2026-02-23T13:11:37.462+0000 7fe8fea5e640 -1 osd.2 122 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2026-02-23T13:11:38.497 DEBUG:teuthology.orchestra.run.trial068:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -t 128 -b 8192 --object-size 65536 -p unique_pool_2 bench 90 write' 2026-02-23T13:11:38.500 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2026-02-23T13:11:38.500 INFO:teuthology.orchestra.run:waiting for 3000 2026-02-23T13:11:38.546 INFO:tasks.radosbench.radosbench.0.trial068.stdout:hints = 1 2026-02-23T13:11:38.546 INFO:tasks.radosbench.radosbench.0.trial068.stdout:Maintaining 128 concurrent writes of 8192 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2026-02-23T13:11:38.547 INFO:tasks.radosbench.radosbench.0.trial068.stdout:Object prefix: benchmark_data_trial068_96731 2026-02-23T13:11:38.547 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T13:11:38.547 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 0 0 0 0 0 0 - 0 2026-02-23T13:11:38.589 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:11:38.703 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:11:38.724 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:11:38 trial068 ceph-mon[38805]: from='osd.2 [v2:10.20.193.185:6800/1511048998,v1:10.20.193.185:6801/1511048998]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2026-02-23T13:11:38.724 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:11:38 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/373623974' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]': finished 2026-02-23T13:11:38.724 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:11:38 trial068 ceph-mon[38805]: osdmap e127: 4 total, 3 up, 4 in 2026-02-23T13:11:38.725 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:11:38 trial068 ceph-mon[38805]: from='osd.2 [v2:10.20.193.185:6800/1511048998,v1:10.20.193.185:6801/1511048998]' entity='osd.2' cmd={"prefix": "osd crush create-or-move", "id": 2, "weight":0.6836, "args": ["host=trial185", "root=default"]} : dispatch 2026-02-23T13:11:38.803 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:11:38.802+0000 7f011004c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:11:38.804 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:11:38.802+0000 7f011004c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:11:38.817 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:11:38.817 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:11:38.828 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:11:38.829 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:11:38.840 INFO:teuthology.orchestra.run.trial068.stdout:osd.2: {} 2026-02-23T13:11:38.840 INFO:teuthology.orchestra.run.trial068.stderr:osd.2: osd_delete_sleep = '' osd_delete_sleep_hdd = '' osd_delete_sleep_hybrid = '' osd_delete_sleep_ssd = '' osd_enable_op_tracker = '' osd_recovery_max_active = '' osd_recovery_max_active_hdd = '' osd_recovery_max_active_ssd = '' osd_recovery_sleep = '' osd_recovery_sleep_degraded = '' osd_recovery_sleep_degraded_hdd = '' osd_recovery_sleep_degraded_hybrid = '' osd_recovery_sleep_degraded_ssd = '' osd_recovery_sleep_hdd = '' osd_recovery_sleep_hybrid = '' osd_recovery_sleep_ssd = '' osd_scrub_sleep = '' osd_snap_trim_sleep = '' osd_snap_trim_sleep_hdd = '' osd_snap_trim_sleep_hybrid = '' osd_snap_trim_sleep_ssd = '' 2026-02-23T13:11:38.851 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:11:38.851 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:11:39.547 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1 128 994 866 6.76508 6.76562 0.00709265 0.147015 2026-02-23T13:11:39.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:11:39 trial068 ceph-mon[38805]: pgmap v229: 17 pgs: 12 creating+peering, 1 active+undersized+degraded, 4 unknown; 577 KiB data, 2.1 GiB used, 2.7 TiB / 2.7 TiB avail; 2/6 objects degraded (33.333%) 2026-02-23T13:11:39.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:11:39 trial068 ceph-mon[38805]: Health check cleared: OSD_DOWN (was: 1 osds down) 2026-02-23T13:11:39.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:11:39 trial068 ceph-mon[38805]: osd.2 [v2:10.20.193.185:6800/1511048998,v1:10.20.193.185:6801/1511048998] boot 2026-02-23T13:11:39.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:11:39 trial068 ceph-mon[38805]: osdmap e128: 4 total, 4 up, 4 in 2026-02-23T13:11:39.845 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:11:39 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2026-02-23T13:11:39.845 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:11:39 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3285707264' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:11:39.845 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:11:39 trial068 ceph-mon[38805]: Health check update: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded (PG_DEGRADED) 2026-02-23T13:11:40.547 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2 128 12110 11982 46.8009 86.8438 0.0170865 0.0212862 2026-02-23T13:11:40.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:11:40 trial068 ceph-mon[38805]: osdmap e129: 4 total, 4 up, 4 in 2026-02-23T13:11:40.886 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:11:41.041 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:11:41.168 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:11:41.166+0000 7f602a546640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:11:41.169 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:11:41.167+0000 7f602a546640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:11:41.186 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:11:41.187 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:11:41.200 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:11:41.200 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:11:41.218 INFO:teuthology.orchestra.run.trial068.stdout:osd.2: {} 2026-02-23T13:11:41.218 INFO:teuthology.orchestra.run.trial068.stderr:osd.2: osd_enable_op_tracker = '' 2026-02-23T13:11:41.231 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:11:41.231 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:11:41.547 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3 127 22273 22146 57.6672 79.4062 0.0218936 0.0172692 2026-02-23T13:11:41.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:11:41 trial068 ceph-mon[38805]: 1.0 scrub starts 2026-02-23T13:11:41.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:11:41 trial068 ceph-mon[38805]: 1.0 scrub ok 2026-02-23T13:11:41.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:11:41 trial068 ceph-mon[38805]: pgmap v232: 17 pgs: 12 creating+peering, 1 active+undersized+degraded, 4 unknown; 577 KiB data, 2.1 GiB used, 2.7 TiB / 2.7 TiB avail; 2/6 objects degraded (33.333%) 2026-02-23T13:11:41.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:11:41 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2182755646' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:11:42.267 DEBUG:teuthology.orchestra.run.trial185:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph daemon osd.2 dump_ops_in_flight 2026-02-23T13:11:42.417 INFO:teuthology.orchestra.run.trial185.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/config/ceph.conf 2026-02-23T13:11:42.547 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4 127 32390 32263 63.0086 79.0391 0.021617 0.0158293 2026-02-23T13:11:42.572 INFO:teuthology.orchestra.run.trial185.stdout:{ 2026-02-23T13:11:42.572 INFO:teuthology.orchestra.run.trial185.stdout: "ops": [ 2026-02-23T13:11:42.573 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.573 INFO:teuthology.orchestra.run.trial185.stdout: "description": "osd_op(client.14643.0:32537 4.1 4:8bfcdf7f:::benchmark_data_trial068_96731_object4067:head [set-alloc-hint object_size 65536 write_size 8192,write 0~8192 in=8192b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e129)", 2026-02-23T13:11:42.573 INFO:teuthology.orchestra.run.trial185.stdout: "initiated_at": "2026-02-23T13:11:42.557939+0000", 2026-02-23T13:11:42.573 INFO:teuthology.orchestra.run.trial185.stdout: "age": 0.010938982999999999, 2026-02-23T13:11:42.573 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 0.010953841000000001, 2026-02-23T13:11:42.573 INFO:teuthology.orchestra.run.trial185.stdout: "continuous": false, 2026-02-23T13:11:42.573 INFO:teuthology.orchestra.run.trial185.stdout: "type_data": { 2026-02-23T13:11:42.573 INFO:teuthology.orchestra.run.trial185.stdout: "flag_point": "waiting for sub ops", 2026-02-23T13:11:42.573 INFO:teuthology.orchestra.run.trial185.stdout: "client_info": { 2026-02-23T13:11:42.573 INFO:teuthology.orchestra.run.trial185.stdout: "client": "client.14643", 2026-02-23T13:11:42.574 INFO:teuthology.orchestra.run.trial185.stdout: "client_addr": "10.20.193.68:0/3888608980", 2026-02-23T13:11:42.574 INFO:teuthology.orchestra.run.trial185.stdout: "tid": 32537 2026-02-23T13:11:42.574 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.574 INFO:teuthology.orchestra.run.trial185.stdout: "events": [ 2026-02-23T13:11:42.574 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.574 INFO:teuthology.orchestra.run.trial185.stdout: "event": "initiated", 2026-02-23T13:11:42.574 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:11:42.557939+0000", 2026-02-23T13:11:42.574 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 0 2026-02-23T13:11:42.574 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.574 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.574 INFO:teuthology.orchestra.run.trial185.stdout: "event": "header_read", 2026-02-23T13:11:42.574 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:11:42.557939+0000", 2026-02-23T13:11:42.575 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 0 2026-02-23T13:11:42.575 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.575 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.575 INFO:teuthology.orchestra.run.trial185.stdout: "event": "throttled", 2026-02-23T13:11:42.575 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:11:42.557949+0000", 2026-02-23T13:11:42.575 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 1.0079e-05 2026-02-23T13:11:42.575 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.575 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.575 INFO:teuthology.orchestra.run.trial185.stdout: "event": "all_read", 2026-02-23T13:11:42.575 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:11:42.557974+0000", 2026-02-23T13:11:42.575 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 2.4997000000000001e-05 2026-02-23T13:11:42.575 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.576 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.576 INFO:teuthology.orchestra.run.trial185.stdout: "event": "dispatched", 2026-02-23T13:11:42.576 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:11:42.557981+0000", 2026-02-23T13:11:42.576 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 6.6819999999999997e-06 2026-02-23T13:11:42.576 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.576 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.576 INFO:teuthology.orchestra.run.trial185.stdout: "event": "queued_for_pg", 2026-02-23T13:11:42.576 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:11:42.557985+0000", 2026-02-23T13:11:42.576 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 3.8670000000000004e-06 2026-02-23T13:11:42.576 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.576 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.576 INFO:teuthology.orchestra.run.trial185.stdout: "event": "reached_pg", 2026-02-23T13:11:42.577 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:11:42.563061+0000", 2026-02-23T13:11:42.577 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 0.0050758499999999998 2026-02-23T13:11:42.577 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.577 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.577 INFO:teuthology.orchestra.run.trial185.stdout: "event": "started", 2026-02-23T13:11:42.577 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:11:42.563135+0000", 2026-02-23T13:11:42.577 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 7.4139000000000003e-05 2026-02-23T13:11:42.577 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.577 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.577 INFO:teuthology.orchestra.run.trial185.stdout: "event": "waiting for subops from 0,3", 2026-02-23T13:11:42.577 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:11:42.563250+0000", 2026-02-23T13:11:42.577 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 0.000115296 2026-02-23T13:11:42.578 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.578 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.578 INFO:teuthology.orchestra.run.trial185.stdout: "event": "op_commit", 2026-02-23T13:11:42.578 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:11:42.565777+0000", 2026-02-23T13:11:42.578 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 0.0025268690000000002 2026-02-23T13:11:42.578 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.578 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.578 INFO:teuthology.orchestra.run.trial185.stdout: "event": "sub_op_commit_rec", 2026-02-23T13:11:42.578 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:11:42.567774+0000", 2026-02-23T13:11:42.578 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 0.0019971279999999999 2026-02-23T13:11:42.578 INFO:teuthology.orchestra.run.trial185.stdout: } 2026-02-23T13:11:42.579 INFO:teuthology.orchestra.run.trial185.stdout: ] 2026-02-23T13:11:42.579 INFO:teuthology.orchestra.run.trial185.stdout: } 2026-02-23T13:11:42.579 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.579 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.579 INFO:teuthology.orchestra.run.trial185.stdout: "description": "osd_op(client.14643.0:32556 4.9 4.8e1fe259 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e129)", 2026-02-23T13:11:42.579 INFO:teuthology.orchestra.run.trial185.stdout: "initiated_at": "2026-02-23T13:11:42.561539+0000", 2026-02-23T13:11:42.579 INFO:teuthology.orchestra.run.trial185.stdout: "age": 0.0073387950000000004, 2026-02-23T13:11:42.579 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 0.0073853319999999997, 2026-02-23T13:11:42.579 INFO:teuthology.orchestra.run.trial185.stdout: "continuous": false, 2026-02-23T13:11:42.579 INFO:teuthology.orchestra.run.trial185.stdout: "type_data": { 2026-02-23T13:11:42.579 INFO:teuthology.orchestra.run.trial185.stdout: "flag_point": "queued for pg", 2026-02-23T13:11:42.580 INFO:teuthology.orchestra.run.trial185.stdout: "client_info": { 2026-02-23T13:11:42.580 INFO:teuthology.orchestra.run.trial185.stdout: "client": "client.14643", 2026-02-23T13:11:42.580 INFO:teuthology.orchestra.run.trial185.stdout: "client_addr": "10.20.193.68:0/3888608980", 2026-02-23T13:11:42.580 INFO:teuthology.orchestra.run.trial185.stdout: "tid": 32556 2026-02-23T13:11:42.580 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.580 INFO:teuthology.orchestra.run.trial185.stdout: "events": [ 2026-02-23T13:11:42.580 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.580 INFO:teuthology.orchestra.run.trial185.stdout: "event": "initiated", 2026-02-23T13:11:42.580 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:11:42.561539+0000", 2026-02-23T13:11:42.580 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 0 2026-02-23T13:11:42.580 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.580 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.580 INFO:teuthology.orchestra.run.trial185.stdout: "event": "header_read", 2026-02-23T13:11:42.581 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:11:42.561539+0000", 2026-02-23T13:11:42.581 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 0 2026-02-23T13:11:42.581 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.581 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.581 INFO:teuthology.orchestra.run.trial185.stdout: "event": "throttled", 2026-02-23T13:11:42.581 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:11:42.561563+0000", 2026-02-23T13:11:42.581 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 2.3754e-05 2026-02-23T13:11:42.581 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.581 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.581 INFO:teuthology.orchestra.run.trial185.stdout: "event": "all_read", 2026-02-23T13:11:42.581 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:11:42.561676+0000", 2026-02-23T13:11:42.581 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 0.00011283200000000001 2026-02-23T13:11:42.582 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.582 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.582 INFO:teuthology.orchestra.run.trial185.stdout: "event": "dispatched", 2026-02-23T13:11:42.582 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:11:42.561688+0000", 2026-02-23T13:11:42.582 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 1.2252999999999999e-05 2026-02-23T13:11:42.582 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.582 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.582 INFO:teuthology.orchestra.run.trial185.stdout: "event": "queued_for_pg", 2026-02-23T13:11:42.582 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:11:42.561691+0000", 2026-02-23T13:11:42.582 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 2.785e-06 2026-02-23T13:11:42.582 INFO:teuthology.orchestra.run.trial185.stdout: } 2026-02-23T13:11:42.582 INFO:teuthology.orchestra.run.trial185.stdout: ] 2026-02-23T13:11:42.582 INFO:teuthology.orchestra.run.trial185.stdout: } 2026-02-23T13:11:42.583 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.583 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.583 INFO:teuthology.orchestra.run.trial185.stdout: "description": "osd_op(client.14643.0:32538 4.1 4:8bfcdf7f:::benchmark_data_trial068_96731_object4067:head [set-alloc-hint object_size 65536 write_size 8192,write 8192~8192 in=8192b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e129)", 2026-02-23T13:11:42.583 INFO:teuthology.orchestra.run.trial185.stdout: "initiated_at": "2026-02-23T13:11:42.557998+0000", 2026-02-23T13:11:42.583 INFO:teuthology.orchestra.run.trial185.stdout: "age": 0.010879742, 2026-02-23T13:11:42.583 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 0.010945155, 2026-02-23T13:11:42.583 INFO:teuthology.orchestra.run.trial185.stdout: "continuous": false, 2026-02-23T13:11:42.583 INFO:teuthology.orchestra.run.trial185.stdout: "type_data": { 2026-02-23T13:11:42.583 INFO:teuthology.orchestra.run.trial185.stdout: "flag_point": "waiting for sub ops", 2026-02-23T13:11:42.583 INFO:teuthology.orchestra.run.trial185.stdout: "client_info": { 2026-02-23T13:11:42.583 INFO:teuthology.orchestra.run.trial185.stdout: "client": "client.14643", 2026-02-23T13:11:42.583 INFO:teuthology.orchestra.run.trial185.stdout: "client_addr": "10.20.193.68:0/3888608980", 2026-02-23T13:11:42.583 INFO:teuthology.orchestra.run.trial185.stdout: "tid": 32538 2026-02-23T13:11:42.584 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.584 INFO:teuthology.orchestra.run.trial185.stdout: "events": [ 2026-02-23T13:11:42.584 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.584 INFO:teuthology.orchestra.run.trial185.stdout: "event": "initiated", 2026-02-23T13:11:42.584 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:11:42.557998+0000", 2026-02-23T13:11:42.584 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 0 2026-02-23T13:11:42.584 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.584 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.584 INFO:teuthology.orchestra.run.trial185.stdout: "event": "header_read", 2026-02-23T13:11:42.584 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:11:42.557998+0000", 2026-02-23T13:11:42.584 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 0 2026-02-23T13:11:42.584 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.585 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.585 INFO:teuthology.orchestra.run.trial185.stdout: "event": "throttled", 2026-02-23T13:11:42.585 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:11:42.558011+0000", 2026-02-23T13:11:42.585 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 1.2784e-05 2026-02-23T13:11:42.585 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.585 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.585 INFO:teuthology.orchestra.run.trial185.stdout: "event": "all_read", 2026-02-23T13:11:42.585 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:11:42.558045+0000", 2026-02-23T13:11:42.585 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 3.4204e-05 2026-02-23T13:11:42.585 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.586 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.586 INFO:teuthology.orchestra.run.trial185.stdout: "event": "dispatched", 2026-02-23T13:11:42.586 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:11:42.558051+0000", 2026-02-23T13:11:42.586 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 5.9009999999999999e-06 2026-02-23T13:11:42.586 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.586 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.586 INFO:teuthology.orchestra.run.trial185.stdout: "event": "queued_for_pg", 2026-02-23T13:11:42.586 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:11:42.558054+0000", 2026-02-23T13:11:42.586 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 2.3439999999999999e-06 2026-02-23T13:11:42.586 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.586 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.586 INFO:teuthology.orchestra.run.trial185.stdout: "event": "reached_pg", 2026-02-23T13:11:42.586 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:11:42.564206+0000", 2026-02-23T13:11:42.587 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 0.0061520940000000003 2026-02-23T13:11:42.587 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.587 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.587 INFO:teuthology.orchestra.run.trial185.stdout: "event": "started", 2026-02-23T13:11:42.587 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:11:42.564272+0000", 2026-02-23T13:11:42.587 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 6.6193999999999997e-05 2026-02-23T13:11:42.587 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.587 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.587 INFO:teuthology.orchestra.run.trial185.stdout: "event": "waiting for subops from 0,3", 2026-02-23T13:11:42.587 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:11:42.564366+0000", 2026-02-23T13:11:42.587 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 9.4407000000000006e-05 2026-02-23T13:11:42.587 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.588 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.588 INFO:teuthology.orchestra.run.trial185.stdout: "event": "op_commit", 2026-02-23T13:11:42.588 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:11:42.565780+0000", 2026-02-23T13:11:42.588 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 0.001414046 2026-02-23T13:11:42.588 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.588 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.588 INFO:teuthology.orchestra.run.trial185.stdout: "event": "sub_op_commit_rec", 2026-02-23T13:11:42.588 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:11:42.567843+0000", 2026-02-23T13:11:42.588 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 0.0020622800000000001 2026-02-23T13:11:42.588 INFO:teuthology.orchestra.run.trial185.stdout: } 2026-02-23T13:11:42.588 INFO:teuthology.orchestra.run.trial185.stdout: ] 2026-02-23T13:11:42.590 INFO:teuthology.orchestra.run.trial185.stdout: } 2026-02-23T13:11:42.590 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.590 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.590 INFO:teuthology.orchestra.run.trial185.stdout: "description": "osd_op(client.14643.0:32557 4.9 4.8e1fe259 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e129)", 2026-02-23T13:11:42.590 INFO:teuthology.orchestra.run.trial185.stdout: "initiated_at": "2026-02-23T13:11:42.561720+0000", 2026-02-23T13:11:42.590 INFO:teuthology.orchestra.run.trial185.stdout: "age": 0.0071584370000000001, 2026-02-23T13:11:42.590 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 0.0072437070000000003, 2026-02-23T13:11:42.590 INFO:teuthology.orchestra.run.trial185.stdout: "continuous": false, 2026-02-23T13:11:42.590 INFO:teuthology.orchestra.run.trial185.stdout: "type_data": { 2026-02-23T13:11:42.590 INFO:teuthology.orchestra.run.trial185.stdout: "flag_point": "queued for pg", 2026-02-23T13:11:42.591 INFO:teuthology.orchestra.run.trial185.stdout: "client_info": { 2026-02-23T13:11:42.591 INFO:teuthology.orchestra.run.trial185.stdout: "client": "client.14643", 2026-02-23T13:11:42.591 INFO:teuthology.orchestra.run.trial185.stdout: "client_addr": "10.20.193.68:0/3888608980", 2026-02-23T13:11:42.591 INFO:teuthology.orchestra.run.trial185.stdout: "tid": 32557 2026-02-23T13:11:42.591 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.591 INFO:teuthology.orchestra.run.trial185.stdout: "events": [ 2026-02-23T13:11:42.591 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.591 INFO:teuthology.orchestra.run.trial185.stdout: "event": "initiated", 2026-02-23T13:11:42.591 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:11:42.561720+0000", 2026-02-23T13:11:42.591 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 0 2026-02-23T13:11:42.591 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.591 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.592 INFO:teuthology.orchestra.run.trial185.stdout: "event": "header_read", 2026-02-23T13:11:42.592 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:11:42.561720+0000", 2026-02-23T13:11:42.592 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 0 2026-02-23T13:11:42.592 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.592 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.592 INFO:teuthology.orchestra.run.trial185.stdout: "event": "throttled", 2026-02-23T13:11:42.592 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:11:42.561745+0000", 2026-02-23T13:11:42.592 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 2.5287e-05 2026-02-23T13:11:42.592 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.592 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.592 INFO:teuthology.orchestra.run.trial185.stdout: "event": "all_read", 2026-02-23T13:11:42.592 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:11:42.561799+0000", 2026-02-23T13:11:42.592 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 5.3891000000000003e-05 2026-02-23T13:11:42.593 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.593 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.593 INFO:teuthology.orchestra.run.trial185.stdout: "event": "dispatched", 2026-02-23T13:11:42.593 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:11:42.561804+0000", 2026-02-23T13:11:42.593 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 4.7690000000000004e-06 2026-02-23T13:11:42.593 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.593 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.593 INFO:teuthology.orchestra.run.trial185.stdout: "event": "queued_for_pg", 2026-02-23T13:11:42.593 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:11:42.561806+0000", 2026-02-23T13:11:42.593 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 2.7149999999999998e-06 2026-02-23T13:11:42.593 INFO:teuthology.orchestra.run.trial185.stdout: } 2026-02-23T13:11:42.593 INFO:teuthology.orchestra.run.trial185.stdout: ] 2026-02-23T13:11:42.594 INFO:teuthology.orchestra.run.trial185.stdout: } 2026-02-23T13:11:42.594 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.594 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.594 INFO:teuthology.orchestra.run.trial185.stdout: "description": "osd_op(client.14643.0:32539 4.1 4:8bfcdf7f:::benchmark_data_trial068_96731_object4067:head [set-alloc-hint object_size 65536 write_size 8192,write 16384~8192 in=8192b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e129)", 2026-02-23T13:11:42.594 INFO:teuthology.orchestra.run.trial185.stdout: "initiated_at": "2026-02-23T13:11:42.558084+0000", 2026-02-23T13:11:42.594 INFO:teuthology.orchestra.run.trial185.stdout: "age": 0.010794533, 2026-02-23T13:11:42.594 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 0.010892506999999999, 2026-02-23T13:11:42.594 INFO:teuthology.orchestra.run.trial185.stdout: "continuous": false, 2026-02-23T13:11:42.594 INFO:teuthology.orchestra.run.trial185.stdout: "type_data": { 2026-02-23T13:11:42.594 INFO:teuthology.orchestra.run.trial185.stdout: "flag_point": "waiting for sub ops", 2026-02-23T13:11:42.595 INFO:teuthology.orchestra.run.trial185.stdout: "client_info": { 2026-02-23T13:11:42.595 INFO:teuthology.orchestra.run.trial185.stdout: "client": "client.14643", 2026-02-23T13:11:42.595 INFO:teuthology.orchestra.run.trial185.stdout: "client_addr": "10.20.193.68:0/3888608980", 2026-02-23T13:11:42.595 INFO:teuthology.orchestra.run.trial185.stdout: "tid": 32539 2026-02-23T13:11:42.595 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.595 INFO:teuthology.orchestra.run.trial185.stdout: "events": [ 2026-02-23T13:11:42.595 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.595 INFO:teuthology.orchestra.run.trial185.stdout: "event": "initiated", 2026-02-23T13:11:42.595 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:11:42.558084+0000", 2026-02-23T13:11:42.595 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 0 2026-02-23T13:11:42.595 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.595 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.596 INFO:teuthology.orchestra.run.trial185.stdout: "event": "header_read", 2026-02-23T13:11:42.596 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:11:42.558084+0000", 2026-02-23T13:11:42.596 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 0 2026-02-23T13:11:42.596 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.596 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.596 INFO:teuthology.orchestra.run.trial185.stdout: "event": "throttled", 2026-02-23T13:11:42.596 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:11:42.558099+0000", 2026-02-23T13:11:42.596 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 1.5099000000000001e-05 2026-02-23T13:11:42.596 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.596 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.596 INFO:teuthology.orchestra.run.trial185.stdout: "event": "all_read", 2026-02-23T13:11:42.596 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:11:42.558131+0000", 2026-02-23T13:11:42.597 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 3.243e-05 2026-02-23T13:11:42.597 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.597 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.597 INFO:teuthology.orchestra.run.trial185.stdout: "event": "dispatched", 2026-02-23T13:11:42.597 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:11:42.558138+0000", 2026-02-23T13:11:42.597 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 7.1439999999999997e-06 2026-02-23T13:11:42.597 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.597 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.597 INFO:teuthology.orchestra.run.trial185.stdout: "event": "queued_for_pg", 2026-02-23T13:11:42.597 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:11:42.558141+0000", 2026-02-23T13:11:42.597 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 3.236e-06 2026-02-23T13:11:42.597 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.598 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.598 INFO:teuthology.orchestra.run.trial185.stdout: "event": "reached_pg", 2026-02-23T13:11:42.598 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:11:42.564796+0000", 2026-02-23T13:11:42.598 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 0.0066543640000000003 2026-02-23T13:11:42.598 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.598 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.598 INFO:teuthology.orchestra.run.trial185.stdout: "event": "started", 2026-02-23T13:11:42.598 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:11:42.564834+0000", 2026-02-23T13:11:42.598 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 3.8402000000000003e-05 2026-02-23T13:11:42.598 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.598 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.598 INFO:teuthology.orchestra.run.trial185.stdout: "event": "waiting for subops from 0,3", 2026-02-23T13:11:42.598 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:11:42.564919+0000", 2026-02-23T13:11:42.599 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 8.4828999999999997e-05 2026-02-23T13:11:42.599 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.599 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.599 INFO:teuthology.orchestra.run.trial185.stdout: "event": "op_commit", 2026-02-23T13:11:42.599 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:11:42.566674+0000", 2026-02-23T13:11:42.599 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 0.001755375 2026-02-23T13:11:42.599 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.599 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.599 INFO:teuthology.orchestra.run.trial185.stdout: "event": "sub_op_commit_rec", 2026-02-23T13:11:42.599 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:11:42.567873+0000", 2026-02-23T13:11:42.600 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 0.0011991040000000001 2026-02-23T13:11:42.600 INFO:teuthology.orchestra.run.trial185.stdout: } 2026-02-23T13:11:42.600 INFO:teuthology.orchestra.run.trial185.stdout: ] 2026-02-23T13:11:42.600 INFO:teuthology.orchestra.run.trial185.stdout: } 2026-02-23T13:11:42.600 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.600 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.600 INFO:teuthology.orchestra.run.trial185.stdout: "description": "osd_op(client.14643.0:32558 4.9 4.8e1fe259 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e129)", 2026-02-23T13:11:42.600 INFO:teuthology.orchestra.run.trial185.stdout: "initiated_at": "2026-02-23T13:11:42.561819+0000", 2026-02-23T13:11:42.601 INFO:teuthology.orchestra.run.trial185.stdout: "age": 0.0070588609999999996, 2026-02-23T13:11:42.601 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 0.007174868, 2026-02-23T13:11:42.601 INFO:teuthology.orchestra.run.trial185.stdout: "continuous": false, 2026-02-23T13:11:42.601 INFO:teuthology.orchestra.run.trial185.stdout: "type_data": { 2026-02-23T13:11:42.601 INFO:teuthology.orchestra.run.trial185.stdout: "flag_point": "queued for pg", 2026-02-23T13:11:42.601 INFO:teuthology.orchestra.run.trial185.stdout: "client_info": { 2026-02-23T13:11:42.601 INFO:teuthology.orchestra.run.trial185.stdout: "client": "client.14643", 2026-02-23T13:11:42.601 INFO:teuthology.orchestra.run.trial185.stdout: "client_addr": "10.20.193.68:0/3888608980", 2026-02-23T13:11:42.601 INFO:teuthology.orchestra.run.trial185.stdout: "tid": 32558 2026-02-23T13:11:42.601 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.601 INFO:teuthology.orchestra.run.trial185.stdout: "events": [ 2026-02-23T13:11:42.601 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.602 INFO:teuthology.orchestra.run.trial185.stdout: "event": "initiated", 2026-02-23T13:11:42.602 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:11:42.561819+0000", 2026-02-23T13:11:42.602 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 0 2026-02-23T13:11:42.602 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.602 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.602 INFO:teuthology.orchestra.run.trial185.stdout: "event": "header_read", 2026-02-23T13:11:42.602 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:11:42.561819+0000", 2026-02-23T13:11:42.602 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 0 2026-02-23T13:11:42.602 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.602 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.602 INFO:teuthology.orchestra.run.trial185.stdout: "event": "throttled", 2026-02-23T13:11:42.602 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:11:42.561836+0000", 2026-02-23T13:11:42.603 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 1.6381000000000001e-05 2026-02-23T13:11:42.603 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.603 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.603 INFO:teuthology.orchestra.run.trial185.stdout: "event": "all_read", 2026-02-23T13:11:42.603 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:11:42.561883+0000", 2026-02-23T13:11:42.603 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 4.7589000000000001e-05 2026-02-23T13:11:42.603 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.603 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.603 INFO:teuthology.orchestra.run.trial185.stdout: "event": "dispatched", 2026-02-23T13:11:42.603 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:11:42.561890+0000", 2026-02-23T13:11:42.604 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 6.832e-06 2026-02-23T13:11:42.604 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.604 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.604 INFO:teuthology.orchestra.run.trial185.stdout: "event": "queued_for_pg", 2026-02-23T13:11:42.604 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:11:42.561892+0000", 2026-02-23T13:11:42.604 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 2.1849999999999998e-06 2026-02-23T13:11:42.604 INFO:teuthology.orchestra.run.trial185.stdout: } 2026-02-23T13:11:42.604 INFO:teuthology.orchestra.run.trial185.stdout: ] 2026-02-23T13:11:42.604 INFO:teuthology.orchestra.run.trial185.stdout: } 2026-02-23T13:11:42.604 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.604 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.604 INFO:teuthology.orchestra.run.trial185.stdout: "description": "osd_op(client.14643.0:32540 4.1 4:8bfcdf7f:::benchmark_data_trial068_96731_object4067:head [set-alloc-hint object_size 65536 write_size 8192,write 24576~8192 in=8192b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e129)", 2026-02-23T13:11:42.605 INFO:teuthology.orchestra.run.trial185.stdout: "initiated_at": "2026-02-23T13:11:42.558171+0000", 2026-02-23T13:11:42.605 INFO:teuthology.orchestra.run.trial185.stdout: "age": 0.010706597999999999, 2026-02-23T13:11:42.605 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 0.010835489, 2026-02-23T13:11:42.605 INFO:teuthology.orchestra.run.trial185.stdout: "continuous": false, 2026-02-23T13:11:42.605 INFO:teuthology.orchestra.run.trial185.stdout: "type_data": { 2026-02-23T13:11:42.605 INFO:teuthology.orchestra.run.trial185.stdout: "flag_point": "waiting for sub ops", 2026-02-23T13:11:42.605 INFO:teuthology.orchestra.run.trial185.stdout: "client_info": { 2026-02-23T13:11:42.605 INFO:teuthology.orchestra.run.trial185.stdout: "client": "client.14643", 2026-02-23T13:11:42.605 INFO:teuthology.orchestra.run.trial185.stdout: "client_addr": "10.20.193.68:0/3888608980", 2026-02-23T13:11:42.605 INFO:teuthology.orchestra.run.trial185.stdout: "tid": 32540 2026-02-23T13:11:42.606 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.606 INFO:teuthology.orchestra.run.trial185.stdout: "events": [ 2026-02-23T13:11:42.606 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.606 INFO:teuthology.orchestra.run.trial185.stdout: "event": "initiated", 2026-02-23T13:11:42.606 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:11:42.558171+0000", 2026-02-23T13:11:42.606 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 0 2026-02-23T13:11:42.606 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.606 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.606 INFO:teuthology.orchestra.run.trial185.stdout: "event": "header_read", 2026-02-23T13:11:42.606 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:11:42.558171+0000", 2026-02-23T13:11:42.606 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 0 2026-02-23T13:11:42.606 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.607 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.607 INFO:teuthology.orchestra.run.trial185.stdout: "event": "throttled", 2026-02-23T13:11:42.607 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:11:42.558199+0000", 2026-02-23T13:11:42.607 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 2.7141e-05 2026-02-23T13:11:42.607 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.607 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.607 INFO:teuthology.orchestra.run.trial185.stdout: "event": "all_read", 2026-02-23T13:11:42.607 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:11:42.558238+0000", 2026-02-23T13:11:42.607 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 3.9063000000000002e-05 2026-02-23T13:11:42.607 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.607 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.607 INFO:teuthology.orchestra.run.trial185.stdout: "event": "dispatched", 2026-02-23T13:11:42.607 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:11:42.558244+0000", 2026-02-23T13:11:42.608 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 6.2920000000000001e-06 2026-02-23T13:11:42.608 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.608 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.608 INFO:teuthology.orchestra.run.trial185.stdout: "event": "queued_for_pg", 2026-02-23T13:11:42.608 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:11:42.558252+0000", 2026-02-23T13:11:42.608 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 8.2849999999999995e-06 2026-02-23T13:11:42.608 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.608 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.608 INFO:teuthology.orchestra.run.trial185.stdout: "event": "reached_pg", 2026-02-23T13:11:42.608 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:11:42.565307+0000", 2026-02-23T13:11:42.608 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 0.007055034 2026-02-23T13:11:42.608 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.608 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.608 INFO:teuthology.orchestra.run.trial185.stdout: "event": "started", 2026-02-23T13:11:42.609 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:11:42.565343+0000", 2026-02-23T13:11:42.609 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 3.5667000000000003e-05 2026-02-23T13:11:42.609 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.609 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.609 INFO:teuthology.orchestra.run.trial185.stdout: "event": "waiting for subops from 0,3", 2026-02-23T13:11:42.609 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:11:42.565417+0000", 2026-02-23T13:11:42.609 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 7.3617999999999999e-05 2026-02-23T13:11:42.609 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.609 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.609 INFO:teuthology.orchestra.run.trial185.stdout: "event": "op_commit", 2026-02-23T13:11:42.609 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:11:42.567738+0000", 2026-02-23T13:11:42.609 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 0.002320964 2026-02-23T13:11:42.609 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.609 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.610 INFO:teuthology.orchestra.run.trial185.stdout: "event": "sub_op_commit_rec", 2026-02-23T13:11:42.610 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:11:42.567902+0000", 2026-02-23T13:11:42.610 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 0.00016496800000000001 2026-02-23T13:11:42.610 INFO:teuthology.orchestra.run.trial185.stdout: } 2026-02-23T13:11:42.610 INFO:teuthology.orchestra.run.trial185.stdout: ] 2026-02-23T13:11:42.610 INFO:teuthology.orchestra.run.trial185.stdout: } 2026-02-23T13:11:42.610 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.610 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.610 INFO:teuthology.orchestra.run.trial185.stdout: "description": "osd_op(client.14643.0:32541 4.1 4:8bfcdf7f:::benchmark_data_trial068_96731_object4067:head [set-alloc-hint object_size 65536 write_size 8192,write 32768~8192 in=8192b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e129)", 2026-02-23T13:11:42.610 INFO:teuthology.orchestra.run.trial185.stdout: "initiated_at": "2026-02-23T13:11:42.558275+0000", 2026-02-23T13:11:42.610 INFO:teuthology.orchestra.run.trial185.stdout: "age": 0.010602583, 2026-02-23T13:11:42.610 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 0.010756290999999999, 2026-02-23T13:11:42.610 INFO:teuthology.orchestra.run.trial185.stdout: "continuous": false, 2026-02-23T13:11:42.610 INFO:teuthology.orchestra.run.trial185.stdout: "type_data": { 2026-02-23T13:11:42.610 INFO:teuthology.orchestra.run.trial185.stdout: "flag_point": "waiting for sub ops", 2026-02-23T13:11:42.611 INFO:teuthology.orchestra.run.trial185.stdout: "client_info": { 2026-02-23T13:11:42.611 INFO:teuthology.orchestra.run.trial185.stdout: "client": "client.14643", 2026-02-23T13:11:42.611 INFO:teuthology.orchestra.run.trial185.stdout: "client_addr": "10.20.193.68:0/3888608980", 2026-02-23T13:11:42.611 INFO:teuthology.orchestra.run.trial185.stdout: "tid": 32541 2026-02-23T13:11:42.611 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.611 INFO:teuthology.orchestra.run.trial185.stdout: "events": [ 2026-02-23T13:11:42.611 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.611 INFO:teuthology.orchestra.run.trial185.stdout: "event": "initiated", 2026-02-23T13:11:42.611 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:11:42.558275+0000", 2026-02-23T13:11:42.611 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 0 2026-02-23T13:11:42.611 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.611 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.611 INFO:teuthology.orchestra.run.trial185.stdout: "event": "header_read", 2026-02-23T13:11:42.611 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:11:42.558275+0000", 2026-02-23T13:11:42.611 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 0 2026-02-23T13:11:42.612 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.612 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.612 INFO:teuthology.orchestra.run.trial185.stdout: "event": "throttled", 2026-02-23T13:11:42.612 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:11:42.558289+0000", 2026-02-23T13:11:42.612 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 1.3335e-05 2026-02-23T13:11:42.612 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.612 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.612 INFO:teuthology.orchestra.run.trial185.stdout: "event": "all_read", 2026-02-23T13:11:42.612 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:11:42.558330+0000", 2026-02-23T13:11:42.612 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 4.0966e-05 2026-02-23T13:11:42.612 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.612 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.612 INFO:teuthology.orchestra.run.trial185.stdout: "event": "dispatched", 2026-02-23T13:11:42.612 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:11:42.558336+0000", 2026-02-23T13:11:42.612 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 5.8610000000000003e-06 2026-02-23T13:11:42.612 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.613 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.613 INFO:teuthology.orchestra.run.trial185.stdout: "event": "queued_for_pg", 2026-02-23T13:11:42.613 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:11:42.558339+0000", 2026-02-23T13:11:42.613 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 3.2160000000000002e-06 2026-02-23T13:11:42.613 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.613 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.613 INFO:teuthology.orchestra.run.trial185.stdout: "event": "reached_pg", 2026-02-23T13:11:42.613 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:11:42.565806+0000", 2026-02-23T13:11:42.613 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 0.0074668249999999999 2026-02-23T13:11:42.613 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.613 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.613 INFO:teuthology.orchestra.run.trial185.stdout: "event": "started", 2026-02-23T13:11:42.613 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:11:42.565837+0000", 2026-02-23T13:11:42.613 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 3.1399000000000002e-05 2026-02-23T13:11:42.613 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.614 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.614 INFO:teuthology.orchestra.run.trial185.stdout: "event": "waiting for subops from 0,3", 2026-02-23T13:11:42.614 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:11:42.565890+0000", 2026-02-23T13:11:42.614 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 5.3078999999999998e-05 2026-02-23T13:11:42.614 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.614 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.614 INFO:teuthology.orchestra.run.trial185.stdout: "event": "op_commit", 2026-02-23T13:11:42.614 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:11:42.567743+0000", 2026-02-23T13:11:42.614 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 0.0018530580000000001 2026-02-23T13:11:42.614 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.614 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.614 INFO:teuthology.orchestra.run.trial185.stdout: "event": "sub_op_commit_rec", 2026-02-23T13:11:42.614 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:11:42.567930+0000", 2026-02-23T13:11:42.614 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 0.00018650900000000001 2026-02-23T13:11:42.614 INFO:teuthology.orchestra.run.trial185.stdout: } 2026-02-23T13:11:42.614 INFO:teuthology.orchestra.run.trial185.stdout: ] 2026-02-23T13:11:42.614 INFO:teuthology.orchestra.run.trial185.stdout: } 2026-02-23T13:11:42.615 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.615 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.615 INFO:teuthology.orchestra.run.trial185.stdout: "description": "osd_op(client.14643.0:32559 4.9 4.8e1fe259 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e129)", 2026-02-23T13:11:42.615 INFO:teuthology.orchestra.run.trial185.stdout: "initiated_at": "2026-02-23T13:11:42.562047+0000", 2026-02-23T13:11:42.615 INFO:teuthology.orchestra.run.trial185.stdout: "age": 0.0068312049999999999, 2026-02-23T13:11:42.615 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 0.0070184160000000004, 2026-02-23T13:11:42.615 INFO:teuthology.orchestra.run.trial185.stdout: "continuous": false, 2026-02-23T13:11:42.615 INFO:teuthology.orchestra.run.trial185.stdout: "type_data": { 2026-02-23T13:11:42.615 INFO:teuthology.orchestra.run.trial185.stdout: "flag_point": "queued for pg", 2026-02-23T13:11:42.615 INFO:teuthology.orchestra.run.trial185.stdout: "client_info": { 2026-02-23T13:11:42.615 INFO:teuthology.orchestra.run.trial185.stdout: "client": "client.14643", 2026-02-23T13:11:42.615 INFO:teuthology.orchestra.run.trial185.stdout: "client_addr": "10.20.193.68:0/3888608980", 2026-02-23T13:11:42.615 INFO:teuthology.orchestra.run.trial185.stdout: "tid": 32559 2026-02-23T13:11:42.615 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.615 INFO:teuthology.orchestra.run.trial185.stdout: "events": [ 2026-02-23T13:11:42.615 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.615 INFO:teuthology.orchestra.run.trial185.stdout: "event": "initiated", 2026-02-23T13:11:42.615 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:11:42.562047+0000", 2026-02-23T13:11:42.616 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 0 2026-02-23T13:11:42.616 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.616 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.616 INFO:teuthology.orchestra.run.trial185.stdout: "event": "header_read", 2026-02-23T13:11:42.616 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:11:42.562047+0000", 2026-02-23T13:11:42.616 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 0 2026-02-23T13:11:42.616 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.616 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.616 INFO:teuthology.orchestra.run.trial185.stdout: "event": "throttled", 2026-02-23T13:11:42.616 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:11:42.562062+0000", 2026-02-23T13:11:42.616 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 1.4688000000000001e-05 2026-02-23T13:11:42.616 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.616 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.616 INFO:teuthology.orchestra.run.trial185.stdout: "event": "all_read", 2026-02-23T13:11:42.616 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:11:42.562198+0000", 2026-02-23T13:11:42.616 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 0.00013605400000000001 2026-02-23T13:11:42.616 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.617 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.617 INFO:teuthology.orchestra.run.trial185.stdout: "event": "dispatched", 2026-02-23T13:11:42.617 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:11:42.562203+0000", 2026-02-23T13:11:42.617 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 5.2399999999999998e-06 2026-02-23T13:11:42.617 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.617 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.617 INFO:teuthology.orchestra.run.trial185.stdout: "event": "queued_for_pg", 2026-02-23T13:11:42.617 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:11:42.562205+0000", 2026-02-23T13:11:42.617 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 2.4650000000000001e-06 2026-02-23T13:11:42.617 INFO:teuthology.orchestra.run.trial185.stdout: } 2026-02-23T13:11:42.617 INFO:teuthology.orchestra.run.trial185.stdout: ] 2026-02-23T13:11:42.617 INFO:teuthology.orchestra.run.trial185.stdout: } 2026-02-23T13:11:42.617 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.617 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.617 INFO:teuthology.orchestra.run.trial185.stdout: "description": "osd_op(client.14643.0:32542 4.1 4:8bfcdf7f:::benchmark_data_trial068_96731_object4067:head [set-alloc-hint object_size 65536 write_size 8192,write 40960~8192 in=8192b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e129)", 2026-02-23T13:11:42.617 INFO:teuthology.orchestra.run.trial185.stdout: "initiated_at": "2026-02-23T13:11:42.558362+0000", 2026-02-23T13:11:42.618 INFO:teuthology.orchestra.run.trial185.stdout: "age": 0.010516101999999999, 2026-02-23T13:11:42.618 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 0.010718922000000001, 2026-02-23T13:11:42.618 INFO:teuthology.orchestra.run.trial185.stdout: "continuous": false, 2026-02-23T13:11:42.618 INFO:teuthology.orchestra.run.trial185.stdout: "type_data": { 2026-02-23T13:11:42.618 INFO:teuthology.orchestra.run.trial185.stdout: "flag_point": "waiting for sub ops", 2026-02-23T13:11:42.618 INFO:teuthology.orchestra.run.trial185.stdout: "client_info": { 2026-02-23T13:11:42.618 INFO:teuthology.orchestra.run.trial185.stdout: "client": "client.14643", 2026-02-23T13:11:42.618 INFO:teuthology.orchestra.run.trial185.stdout: "client_addr": "10.20.193.68:0/3888608980", 2026-02-23T13:11:42.618 INFO:teuthology.orchestra.run.trial185.stdout: "tid": 32542 2026-02-23T13:11:42.618 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.618 INFO:teuthology.orchestra.run.trial185.stdout: "events": [ 2026-02-23T13:11:42.618 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.618 INFO:teuthology.orchestra.run.trial185.stdout: "event": "initiated", 2026-02-23T13:11:42.618 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:11:42.558362+0000", 2026-02-23T13:11:42.618 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 0 2026-02-23T13:11:42.619 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.619 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.619 INFO:teuthology.orchestra.run.trial185.stdout: "event": "header_read", 2026-02-23T13:11:42.619 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:11:42.558362+0000", 2026-02-23T13:11:42.619 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 0 2026-02-23T13:11:42.619 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.619 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.619 INFO:teuthology.orchestra.run.trial185.stdout: "event": "throttled", 2026-02-23T13:11:42.619 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:11:42.558378+0000", 2026-02-23T13:11:42.619 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 1.579e-05 2026-02-23T13:11:42.619 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.619 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.619 INFO:teuthology.orchestra.run.trial185.stdout: "event": "all_read", 2026-02-23T13:11:42.619 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:11:42.558415+0000", 2026-02-23T13:11:42.619 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 3.7109e-05 2026-02-23T13:11:42.619 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.619 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.620 INFO:teuthology.orchestra.run.trial185.stdout: "event": "dispatched", 2026-02-23T13:11:42.620 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:11:42.558425+0000", 2026-02-23T13:11:42.620 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 1.031e-05 2026-02-23T13:11:42.620 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.620 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.620 INFO:teuthology.orchestra.run.trial185.stdout: "event": "queued_for_pg", 2026-02-23T13:11:42.620 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:11:42.558427+0000", 2026-02-23T13:11:42.620 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 2.1339999999999998e-06 2026-02-23T13:11:42.620 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.620 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.620 INFO:teuthology.orchestra.run.trial185.stdout: "event": "reached_pg", 2026-02-23T13:11:42.620 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:11:42.566241+0000", 2026-02-23T13:11:42.620 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 0.0078140339999999992 2026-02-23T13:11:42.620 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.620 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.620 INFO:teuthology.orchestra.run.trial185.stdout: "event": "started", 2026-02-23T13:11:42.620 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:11:42.566275+0000", 2026-02-23T13:11:42.620 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 3.3532999999999999e-05 2026-02-23T13:11:42.621 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.621 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.621 INFO:teuthology.orchestra.run.trial185.stdout: "event": "waiting for subops from 0,3", 2026-02-23T13:11:42.621 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:11:42.566336+0000", 2026-02-23T13:11:42.621 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 6.1455e-05 2026-02-23T13:11:42.621 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.621 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.621 INFO:teuthology.orchestra.run.trial185.stdout: "event": "op_commit", 2026-02-23T13:11:42.621 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:11:42.567805+0000", 2026-02-23T13:11:42.621 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 0.0014690599999999999 2026-02-23T13:11:42.621 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.621 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.621 INFO:teuthology.orchestra.run.trial185.stdout: "event": "sub_op_commit_rec", 2026-02-23T13:11:42.621 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:11:42.567961+0000", 2026-02-23T13:11:42.621 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 0.000156062 2026-02-23T13:11:42.621 INFO:teuthology.orchestra.run.trial185.stdout: } 2026-02-23T13:11:42.621 INFO:teuthology.orchestra.run.trial185.stdout: ] 2026-02-23T13:11:42.621 INFO:teuthology.orchestra.run.trial185.stdout: } 2026-02-23T13:11:42.621 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.621 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.622 INFO:teuthology.orchestra.run.trial185.stdout: "description": "osd_op(client.14643.0:32560 4.9 4.8e1fe259 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e129)", 2026-02-23T13:11:42.622 INFO:teuthology.orchestra.run.trial185.stdout: "initiated_at": "2026-02-23T13:11:42.562303+0000", 2026-02-23T13:11:42.622 INFO:teuthology.orchestra.run.trial185.stdout: "age": 0.0065750349999999999, 2026-02-23T13:11:42.622 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 0.0068129210000000004, 2026-02-23T13:11:42.622 INFO:teuthology.orchestra.run.trial185.stdout: "continuous": false, 2026-02-23T13:11:42.622 INFO:teuthology.orchestra.run.trial185.stdout: "type_data": { 2026-02-23T13:11:42.622 INFO:teuthology.orchestra.run.trial185.stdout: "flag_point": "queued for pg", 2026-02-23T13:11:42.622 INFO:teuthology.orchestra.run.trial185.stdout: "client_info": { 2026-02-23T13:11:42.622 INFO:teuthology.orchestra.run.trial185.stdout: "client": "client.14643", 2026-02-23T13:11:42.622 INFO:teuthology.orchestra.run.trial185.stdout: "client_addr": "10.20.193.68:0/3888608980", 2026-02-23T13:11:42.622 INFO:teuthology.orchestra.run.trial185.stdout: "tid": 32560 2026-02-23T13:11:42.622 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.622 INFO:teuthology.orchestra.run.trial185.stdout: "events": [ 2026-02-23T13:11:42.622 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.622 INFO:teuthology.orchestra.run.trial185.stdout: "event": "initiated", 2026-02-23T13:11:42.622 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:11:42.562303+0000", 2026-02-23T13:11:42.622 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 0 2026-02-23T13:11:42.622 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.622 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.623 INFO:teuthology.orchestra.run.trial185.stdout: "event": "header_read", 2026-02-23T13:11:42.623 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:11:42.562303+0000", 2026-02-23T13:11:42.623 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 0 2026-02-23T13:11:42.623 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.623 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.623 INFO:teuthology.orchestra.run.trial185.stdout: "event": "throttled", 2026-02-23T13:11:42.623 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:11:42.562320+0000", 2026-02-23T13:11:42.623 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 1.6551000000000001e-05 2026-02-23T13:11:42.623 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.623 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.623 INFO:teuthology.orchestra.run.trial185.stdout: "event": "all_read", 2026-02-23T13:11:42.623 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:11:42.562365+0000", 2026-02-23T13:11:42.623 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 4.5584999999999998e-05 2026-02-23T13:11:42.623 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.623 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.623 INFO:teuthology.orchestra.run.trial185.stdout: "event": "dispatched", 2026-02-23T13:11:42.623 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:11:42.562373+0000", 2026-02-23T13:11:42.623 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 7.644e-06 2026-02-23T13:11:42.623 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.623 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.624 INFO:teuthology.orchestra.run.trial185.stdout: "event": "queued_for_pg", 2026-02-23T13:11:42.624 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:11:42.562378+0000", 2026-02-23T13:11:42.624 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 5.1599999999999997e-06 2026-02-23T13:11:42.624 INFO:teuthology.orchestra.run.trial185.stdout: } 2026-02-23T13:11:42.624 INFO:teuthology.orchestra.run.trial185.stdout: ] 2026-02-23T13:11:42.624 INFO:teuthology.orchestra.run.trial185.stdout: } 2026-02-23T13:11:42.624 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.624 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.624 INFO:teuthology.orchestra.run.trial185.stdout: "description": "osd_op(client.14643.0:32586 4.a 4:5eaf47c2:::benchmark_data_trial068_96731_object4073:head [set-alloc-hint object_size 65536 write_size 8192,write 8192~8192 in=8192b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e129)", 2026-02-23T13:11:42.624 INFO:teuthology.orchestra.run.trial185.stdout: "initiated_at": "2026-02-23T13:11:42.564221+0000", 2026-02-23T13:11:42.624 INFO:teuthology.orchestra.run.trial185.stdout: "age": 0.004656795, 2026-02-23T13:11:42.624 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 0.0049121429999999999, 2026-02-23T13:11:42.624 INFO:teuthology.orchestra.run.trial185.stdout: "continuous": false, 2026-02-23T13:11:42.624 INFO:teuthology.orchestra.run.trial185.stdout: "type_data": { 2026-02-23T13:11:42.624 INFO:teuthology.orchestra.run.trial185.stdout: "flag_point": "commit sent; apply or cleanup", 2026-02-23T13:11:42.624 INFO:teuthology.orchestra.run.trial185.stdout: "client_info": { 2026-02-23T13:11:42.624 INFO:teuthology.orchestra.run.trial185.stdout: "client": "client.14643", 2026-02-23T13:11:42.624 INFO:teuthology.orchestra.run.trial185.stdout: "client_addr": "10.20.193.68:0/3888608980", 2026-02-23T13:11:42.624 INFO:teuthology.orchestra.run.trial185.stdout: "tid": 32586 2026-02-23T13:11:42.624 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.625 INFO:teuthology.orchestra.run.trial185.stdout: "events": [ 2026-02-23T13:11:42.625 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.625 INFO:teuthology.orchestra.run.trial185.stdout: "event": "initiated", 2026-02-23T13:11:42.625 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:11:42.564221+0000", 2026-02-23T13:11:42.625 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 0 2026-02-23T13:11:42.625 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.625 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.625 INFO:teuthology.orchestra.run.trial185.stdout: "event": "header_read", 2026-02-23T13:11:42.625 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:11:42.564221+0000", 2026-02-23T13:11:42.625 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 0 2026-02-23T13:11:42.625 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.625 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.625 INFO:teuthology.orchestra.run.trial185.stdout: "event": "throttled", 2026-02-23T13:11:42.625 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:11:42.564231+0000", 2026-02-23T13:11:42.625 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 9.8379999999999997e-06 2026-02-23T13:11:42.625 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.625 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.625 INFO:teuthology.orchestra.run.trial185.stdout: "event": "all_read", 2026-02-23T13:11:42.625 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:11:42.564289+0000", 2026-02-23T13:11:42.625 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 5.8288999999999997e-05 2026-02-23T13:11:42.625 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.625 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.626 INFO:teuthology.orchestra.run.trial185.stdout: "event": "dispatched", 2026-02-23T13:11:42.626 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:11:42.564295+0000", 2026-02-23T13:11:42.626 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 5.5099999999999998e-06 2026-02-23T13:11:42.626 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.626 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.626 INFO:teuthology.orchestra.run.trial185.stdout: "event": "queued_for_pg", 2026-02-23T13:11:42.626 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:11:42.564298+0000", 2026-02-23T13:11:42.626 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 2.8959999999999999e-06 2026-02-23T13:11:42.626 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.626 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.626 INFO:teuthology.orchestra.run.trial185.stdout: "event": "reached_pg", 2026-02-23T13:11:42.626 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:11:42.565024+0000", 2026-02-23T13:11:42.626 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 0.00072648000000000005 2026-02-23T13:11:42.626 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.626 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.626 INFO:teuthology.orchestra.run.trial185.stdout: "event": "started", 2026-02-23T13:11:42.626 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:11:42.565076+0000", 2026-02-23T13:11:42.626 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 5.2206999999999997e-05 2026-02-23T13:11:42.626 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.626 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.626 INFO:teuthology.orchestra.run.trial185.stdout: "event": "waiting for subops from 1,3", 2026-02-23T13:11:42.626 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:11:42.565137+0000", 2026-02-23T13:11:42.627 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 6.0152999999999998e-05 2026-02-23T13:11:42.627 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.627 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.627 INFO:teuthology.orchestra.run.trial185.stdout: "event": "op_commit", 2026-02-23T13:11:42.627 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:11:42.566931+0000", 2026-02-23T13:11:42.627 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 0.0017941179999999999 2026-02-23T13:11:42.627 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.627 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.627 INFO:teuthology.orchestra.run.trial185.stdout: "event": "sub_op_commit_rec", 2026-02-23T13:11:42.627 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:11:42.568635+0000", 2026-02-23T13:11:42.627 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 0.0017040790000000001 2026-02-23T13:11:42.627 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.627 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.627 INFO:teuthology.orchestra.run.trial185.stdout: "event": "sub_op_commit_rec", 2026-02-23T13:11:42.627 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:11:42.569087+0000", 2026-02-23T13:11:42.627 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 0.00045197599999999998 2026-02-23T13:11:42.627 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.627 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.627 INFO:teuthology.orchestra.run.trial185.stdout: "event": "commit_sent", 2026-02-23T13:11:42.627 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:11:42.569114+0000", 2026-02-23T13:11:42.627 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 2.7311e-05 2026-02-23T13:11:42.627 INFO:teuthology.orchestra.run.trial185.stdout: } 2026-02-23T13:11:42.628 INFO:teuthology.orchestra.run.trial185.stdout: ] 2026-02-23T13:11:42.628 INFO:teuthology.orchestra.run.trial185.stdout: } 2026-02-23T13:11:42.628 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.628 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.628 INFO:teuthology.orchestra.run.trial185.stdout: "description": "osd_op(client.14643.0:32543 4.1 4:8bfcdf7f:::benchmark_data_trial068_96731_object4067:head [set-alloc-hint object_size 65536 write_size 8192,write 49152~8192 in=8192b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e129)", 2026-02-23T13:11:42.628 INFO:teuthology.orchestra.run.trial185.stdout: "initiated_at": "2026-02-23T13:11:42.558440+0000", 2026-02-23T13:11:42.628 INFO:teuthology.orchestra.run.trial185.stdout: "age": 0.010437905000000001, 2026-02-23T13:11:42.628 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 0.010716096, 2026-02-23T13:11:42.628 INFO:teuthology.orchestra.run.trial185.stdout: "continuous": false, 2026-02-23T13:11:42.628 INFO:teuthology.orchestra.run.trial185.stdout: "type_data": { 2026-02-23T13:11:42.628 INFO:teuthology.orchestra.run.trial185.stdout: "flag_point": "waiting for sub ops", 2026-02-23T13:11:42.628 INFO:teuthology.orchestra.run.trial185.stdout: "client_info": { 2026-02-23T13:11:42.628 INFO:teuthology.orchestra.run.trial185.stdout: "client": "client.14643", 2026-02-23T13:11:42.628 INFO:teuthology.orchestra.run.trial185.stdout: "client_addr": "10.20.193.68:0/3888608980", 2026-02-23T13:11:42.628 INFO:teuthology.orchestra.run.trial185.stdout: "tid": 32543 2026-02-23T13:11:42.628 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.628 INFO:teuthology.orchestra.run.trial185.stdout: "events": [ 2026-02-23T13:11:42.628 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.628 INFO:teuthology.orchestra.run.trial185.stdout: "event": "initiated", 2026-02-23T13:11:42.628 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:11:42.558440+0000", 2026-02-23T13:11:42.628 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 0 2026-02-23T13:11:42.628 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.628 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.629 INFO:teuthology.orchestra.run.trial185.stdout: "event": "header_read", 2026-02-23T13:11:42.629 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:11:42.558440+0000", 2026-02-23T13:11:42.629 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 0 2026-02-23T13:11:42.629 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.629 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.629 INFO:teuthology.orchestra.run.trial185.stdout: "event": "throttled", 2026-02-23T13:11:42.629 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:11:42.558452+0000", 2026-02-23T13:11:42.629 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 1.1642e-05 2026-02-23T13:11:42.629 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.629 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.629 INFO:teuthology.orchestra.run.trial185.stdout: "event": "all_read", 2026-02-23T13:11:42.629 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:11:42.558481+0000", 2026-02-23T13:11:42.629 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 2.9485e-05 2026-02-23T13:11:42.629 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.629 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.629 INFO:teuthology.orchestra.run.trial185.stdout: "event": "dispatched", 2026-02-23T13:11:42.629 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:11:42.558485+0000", 2026-02-23T13:11:42.629 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 3.2959999999999999e-06 2026-02-23T13:11:42.629 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.629 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.629 INFO:teuthology.orchestra.run.trial185.stdout: "event": "queued_for_pg", 2026-02-23T13:11:42.629 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:11:42.558487+0000", 2026-02-23T13:11:42.629 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 2.024e-06 2026-02-23T13:11:42.630 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.630 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.630 INFO:teuthology.orchestra.run.trial185.stdout: "event": "reached_pg", 2026-02-23T13:11:42.630 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:11:42.566714+0000", 2026-02-23T13:11:42.630 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 0.0082274679999999999 2026-02-23T13:11:42.630 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.630 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.630 INFO:teuthology.orchestra.run.trial185.stdout: "event": "started", 2026-02-23T13:11:42.630 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:11:42.566759+0000", 2026-02-23T13:11:42.630 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 4.4573999999999997e-05 2026-02-23T13:11:42.630 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.630 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.630 INFO:teuthology.orchestra.run.trial185.stdout: "event": "waiting for subops from 0,3", 2026-02-23T13:11:42.630 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:11:42.566818+0000", 2026-02-23T13:11:42.630 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 5.9179999999999999e-05 2026-02-23T13:11:42.630 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.630 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.630 INFO:teuthology.orchestra.run.trial185.stdout: "event": "op_commit", 2026-02-23T13:11:42.630 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:11:42.567811+0000", 2026-02-23T13:11:42.630 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 0.00099280799999999997 2026-02-23T13:11:42.630 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.630 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.630 INFO:teuthology.orchestra.run.trial185.stdout: "event": "sub_op_commit_rec", 2026-02-23T13:11:42.631 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:11:42.568006+0000", 2026-02-23T13:11:42.631 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 0.00019506499999999999 2026-02-23T13:11:42.631 INFO:teuthology.orchestra.run.trial185.stdout: } 2026-02-23T13:11:42.631 INFO:teuthology.orchestra.run.trial185.stdout: ] 2026-02-23T13:11:42.631 INFO:teuthology.orchestra.run.trial185.stdout: } 2026-02-23T13:11:42.631 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.631 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.631 INFO:teuthology.orchestra.run.trial185.stdout: "description": "osd_op(client.14643.0:32553 4.9 4:9a47f871:::benchmark_data_trial068_96731_object4069:head [set-alloc-hint object_size 65536 write_size 8192,write 0~8192 in=8192b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e129)", 2026-02-23T13:11:42.631 INFO:teuthology.orchestra.run.trial185.stdout: "initiated_at": "2026-02-23T13:11:42.560315+0000", 2026-02-23T13:11:42.631 INFO:teuthology.orchestra.run.trial185.stdout: "age": 0.0085625649999999994, 2026-02-23T13:11:42.631 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 0.0088617750000000006, 2026-02-23T13:11:42.631 INFO:teuthology.orchestra.run.trial185.stdout: "continuous": false, 2026-02-23T13:11:42.631 INFO:teuthology.orchestra.run.trial185.stdout: "type_data": { 2026-02-23T13:11:42.631 INFO:teuthology.orchestra.run.trial185.stdout: "flag_point": "waiting for sub ops", 2026-02-23T13:11:42.631 INFO:teuthology.orchestra.run.trial185.stdout: "client_info": { 2026-02-23T13:11:42.631 INFO:teuthology.orchestra.run.trial185.stdout: "client": "client.14643", 2026-02-23T13:11:42.631 INFO:teuthology.orchestra.run.trial185.stdout: "client_addr": "10.20.193.68:0/3888608980", 2026-02-23T13:11:42.631 INFO:teuthology.orchestra.run.trial185.stdout: "tid": 32553 2026-02-23T13:11:42.631 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.631 INFO:teuthology.orchestra.run.trial185.stdout: "events": [ 2026-02-23T13:11:42.631 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.631 INFO:teuthology.orchestra.run.trial185.stdout: "event": "initiated", 2026-02-23T13:11:42.631 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:11:42.560315+0000", 2026-02-23T13:11:42.632 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 0 2026-02-23T13:11:42.632 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.632 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.632 INFO:teuthology.orchestra.run.trial185.stdout: "event": "header_read", 2026-02-23T13:11:42.632 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:11:42.560315+0000", 2026-02-23T13:11:42.632 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 0 2026-02-23T13:11:42.632 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.632 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.632 INFO:teuthology.orchestra.run.trial185.stdout: "event": "throttled", 2026-02-23T13:11:42.632 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:11:42.560331+0000", 2026-02-23T13:11:42.632 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 1.5588999999999999e-05 2026-02-23T13:11:42.632 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.632 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.632 INFO:teuthology.orchestra.run.trial185.stdout: "event": "all_read", 2026-02-23T13:11:42.632 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:11:42.560376+0000", 2026-02-23T13:11:42.632 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 4.5043999999999998e-05 2026-02-23T13:11:42.632 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.632 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.632 INFO:teuthology.orchestra.run.trial185.stdout: "event": "dispatched", 2026-02-23T13:11:42.632 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:11:42.560383+0000", 2026-02-23T13:11:42.632 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 6.4119999999999998e-06 2026-02-23T13:11:42.632 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.632 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.632 INFO:teuthology.orchestra.run.trial185.stdout: "event": "queued_for_pg", 2026-02-23T13:11:42.632 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:11:42.560386+0000", 2026-02-23T13:11:42.632 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 3.4869999999999998e-06 2026-02-23T13:11:42.633 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.633 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.633 INFO:teuthology.orchestra.run.trial185.stdout: "event": "reached_pg", 2026-02-23T13:11:42.633 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:11:42.567283+0000", 2026-02-23T13:11:42.633 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 0.0068970979999999999 2026-02-23T13:11:42.633 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.633 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.633 INFO:teuthology.orchestra.run.trial185.stdout: "event": "started", 2026-02-23T13:11:42.633 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:11:42.567355+0000", 2026-02-23T13:11:42.633 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 7.1765000000000002e-05 2026-02-23T13:11:42.633 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.633 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.633 INFO:teuthology.orchestra.run.trial185.stdout: "event": "waiting for subops from 0,1", 2026-02-23T13:11:42.633 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:11:42.567434+0000", 2026-02-23T13:11:42.633 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 7.9297999999999999e-05 2026-02-23T13:11:42.633 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.633 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.633 INFO:teuthology.orchestra.run.trial185.stdout: "event": "op_commit", 2026-02-23T13:11:42.633 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:11:42.567944+0000", 2026-02-23T13:11:42.633 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 0.00050963400000000002 2026-02-23T13:11:42.633 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.633 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.633 INFO:teuthology.orchestra.run.trial185.stdout: "event": "sub_op_commit_rec", 2026-02-23T13:11:42.634 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:11:42.569034+0000", 2026-02-23T13:11:42.634 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 0.0010900199999999999 2026-02-23T13:11:42.634 INFO:teuthology.orchestra.run.trial185.stdout: } 2026-02-23T13:11:42.634 INFO:teuthology.orchestra.run.trial185.stdout: ] 2026-02-23T13:11:42.634 INFO:teuthology.orchestra.run.trial185.stdout: } 2026-02-23T13:11:42.634 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.634 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.634 INFO:teuthology.orchestra.run.trial185.stdout: "description": "osd_op(client.14643.0:32561 4.9 4.55b198c9 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e129)", 2026-02-23T13:11:42.634 INFO:teuthology.orchestra.run.trial185.stdout: "initiated_at": "2026-02-23T13:11:42.562398+0000", 2026-02-23T13:11:42.634 INFO:teuthology.orchestra.run.trial185.stdout: "age": 0.0064803279999999996, 2026-02-23T13:11:42.634 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 0.0068017900000000003, 2026-02-23T13:11:42.634 INFO:teuthology.orchestra.run.trial185.stdout: "continuous": false, 2026-02-23T13:11:42.634 INFO:teuthology.orchestra.run.trial185.stdout: "type_data": { 2026-02-23T13:11:42.634 INFO:teuthology.orchestra.run.trial185.stdout: "flag_point": "queued for pg", 2026-02-23T13:11:42.634 INFO:teuthology.orchestra.run.trial185.stdout: "client_info": { 2026-02-23T13:11:42.634 INFO:teuthology.orchestra.run.trial185.stdout: "client": "client.14643", 2026-02-23T13:11:42.634 INFO:teuthology.orchestra.run.trial185.stdout: "client_addr": "10.20.193.68:0/3888608980", 2026-02-23T13:11:42.634 INFO:teuthology.orchestra.run.trial185.stdout: "tid": 32561 2026-02-23T13:11:42.634 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.634 INFO:teuthology.orchestra.run.trial185.stdout: "events": [ 2026-02-23T13:11:42.634 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.634 INFO:teuthology.orchestra.run.trial185.stdout: "event": "initiated", 2026-02-23T13:11:42.634 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:11:42.562398+0000", 2026-02-23T13:11:42.634 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 0 2026-02-23T13:11:42.634 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.634 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.635 INFO:teuthology.orchestra.run.trial185.stdout: "event": "header_read", 2026-02-23T13:11:42.635 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:11:42.562398+0000", 2026-02-23T13:11:42.635 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 0 2026-02-23T13:11:42.635 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.635 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.635 INFO:teuthology.orchestra.run.trial185.stdout: "event": "throttled", 2026-02-23T13:11:42.635 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:11:42.562410+0000", 2026-02-23T13:11:42.635 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 1.2714000000000001e-05 2026-02-23T13:11:42.635 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.635 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.635 INFO:teuthology.orchestra.run.trial185.stdout: "event": "all_read", 2026-02-23T13:11:42.635 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:11:42.562452+0000", 2026-02-23T13:11:42.635 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 4.1196999999999999e-05 2026-02-23T13:11:42.635 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.635 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.635 INFO:teuthology.orchestra.run.trial185.stdout: "event": "dispatched", 2026-02-23T13:11:42.635 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:11:42.562458+0000", 2026-02-23T13:11:42.635 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 5.9109999999999998e-06 2026-02-23T13:11:42.635 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.635 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.635 INFO:teuthology.orchestra.run.trial185.stdout: "event": "queued_for_pg", 2026-02-23T13:11:42.635 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:11:42.562462+0000", 2026-02-23T13:11:42.635 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 4.6789999999999996e-06 2026-02-23T13:11:42.635 INFO:teuthology.orchestra.run.trial185.stdout: } 2026-02-23T13:11:42.635 INFO:teuthology.orchestra.run.trial185.stdout: ] 2026-02-23T13:11:42.635 INFO:teuthology.orchestra.run.trial185.stdout: } 2026-02-23T13:11:42.635 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.636 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.636 INFO:teuthology.orchestra.run.trial185.stdout: "description": "osd_op(client.14643.0:32587 4.a 4:5eaf47c2:::benchmark_data_trial068_96731_object4073:head [set-alloc-hint object_size 65536 write_size 8192,write 16384~8192 in=8192b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e129)", 2026-02-23T13:11:42.636 INFO:teuthology.orchestra.run.trial185.stdout: "initiated_at": "2026-02-23T13:11:42.564316+0000", 2026-02-23T13:11:42.636 INFO:teuthology.orchestra.run.trial185.stdout: "age": 0.0045621180000000004, 2026-02-23T13:11:42.636 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 0.004900331, 2026-02-23T13:11:42.636 INFO:teuthology.orchestra.run.trial185.stdout: "continuous": false, 2026-02-23T13:11:42.636 INFO:teuthology.orchestra.run.trial185.stdout: "type_data": { 2026-02-23T13:11:42.636 INFO:teuthology.orchestra.run.trial185.stdout: "flag_point": "waiting for sub ops", 2026-02-23T13:11:42.636 INFO:teuthology.orchestra.run.trial185.stdout: "client_info": { 2026-02-23T13:11:42.636 INFO:teuthology.orchestra.run.trial185.stdout: "client": "client.14643", 2026-02-23T13:11:42.636 INFO:teuthology.orchestra.run.trial185.stdout: "client_addr": "10.20.193.68:0/3888608980", 2026-02-23T13:11:42.636 INFO:teuthology.orchestra.run.trial185.stdout: "tid": 32587 2026-02-23T13:11:42.636 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.636 INFO:teuthology.orchestra.run.trial185.stdout: "events": [ 2026-02-23T13:11:42.636 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.636 INFO:teuthology.orchestra.run.trial185.stdout: "event": "initiated", 2026-02-23T13:11:42.636 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:11:42.564316+0000", 2026-02-23T13:11:42.636 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 0 2026-02-23T13:11:42.636 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.636 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.636 INFO:teuthology.orchestra.run.trial185.stdout: "event": "header_read", 2026-02-23T13:11:42.636 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:11:42.564316+0000", 2026-02-23T13:11:42.636 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 0 2026-02-23T13:11:42.636 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.636 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.636 INFO:teuthology.orchestra.run.trial185.stdout: "event": "throttled", 2026-02-23T13:11:42.637 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:11:42.564327+0000", 2026-02-23T13:11:42.637 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 1.081e-05 2026-02-23T13:11:42.637 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.637 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.637 INFO:teuthology.orchestra.run.trial185.stdout: "event": "all_read", 2026-02-23T13:11:42.637 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:11:42.564368+0000", 2026-02-23T13:11:42.637 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 4.1737999999999999e-05 2026-02-23T13:11:42.637 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.637 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.637 INFO:teuthology.orchestra.run.trial185.stdout: "event": "dispatched", 2026-02-23T13:11:42.637 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:11:42.564374+0000", 2026-02-23T13:11:42.637 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 5.891e-06 2026-02-23T13:11:42.637 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.637 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.637 INFO:teuthology.orchestra.run.trial185.stdout: "event": "queued_for_pg", 2026-02-23T13:11:42.637 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:11:42.564377+0000", 2026-02-23T13:11:42.637 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 2.5349999999999999e-06 2026-02-23T13:11:42.637 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.637 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.637 INFO:teuthology.orchestra.run.trial185.stdout: "event": "reached_pg", 2026-02-23T13:11:42.637 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:11:42.565621+0000", 2026-02-23T13:11:42.637 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 0.0012438180000000001 2026-02-23T13:11:42.637 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.637 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.637 INFO:teuthology.orchestra.run.trial185.stdout: "event": "started", 2026-02-23T13:11:42.638 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:11:42.565663+0000", 2026-02-23T13:11:42.638 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 4.2409e-05 2026-02-23T13:11:42.638 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.638 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.638 INFO:teuthology.orchestra.run.trial185.stdout: "event": "waiting for subops from 1,3", 2026-02-23T13:11:42.638 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:11:42.565709+0000", 2026-02-23T13:11:42.638 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 4.5816000000000003e-05 2026-02-23T13:11:42.638 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.638 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.638 INFO:teuthology.orchestra.run.trial185.stdout: "event": "op_commit", 2026-02-23T13:11:42.638 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:11:42.567510+0000", 2026-02-23T13:11:42.638 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 0.0018006400000000001 2026-02-23T13:11:42.638 INFO:teuthology.orchestra.run.trial185.stdout: } 2026-02-23T13:11:42.638 INFO:teuthology.orchestra.run.trial185.stdout: ] 2026-02-23T13:11:42.638 INFO:teuthology.orchestra.run.trial185.stdout: } 2026-02-23T13:11:42.638 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.638 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.638 INFO:teuthology.orchestra.run.trial185.stdout: "description": "osd_op(client.14643.0:32554 4.9 4:9a47f871:::benchmark_data_trial068_96731_object4069:head [set-alloc-hint object_size 65536 write_size 8192,write 8192~8192 in=8192b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e129)", 2026-02-23T13:11:42.638 INFO:teuthology.orchestra.run.trial185.stdout: "initiated_at": "2026-02-23T13:11:42.560418+0000", 2026-02-23T13:11:42.638 INFO:teuthology.orchestra.run.trial185.stdout: "age": 0.0084602940000000001, 2026-02-23T13:11:42.638 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 0.0088179639999999993, 2026-02-23T13:11:42.638 INFO:teuthology.orchestra.run.trial185.stdout: "continuous": false, 2026-02-23T13:11:42.638 INFO:teuthology.orchestra.run.trial185.stdout: "type_data": { 2026-02-23T13:11:42.638 INFO:teuthology.orchestra.run.trial185.stdout: "flag_point": "waiting for sub ops", 2026-02-23T13:11:42.638 INFO:teuthology.orchestra.run.trial185.stdout: "client_info": { 2026-02-23T13:11:42.638 INFO:teuthology.orchestra.run.trial185.stdout: "client": "client.14643", 2026-02-23T13:11:42.638 INFO:teuthology.orchestra.run.trial185.stdout: "client_addr": "10.20.193.68:0/3888608980", 2026-02-23T13:11:42.638 INFO:teuthology.orchestra.run.trial185.stdout: "tid": 32554 2026-02-23T13:11:42.639 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.639 INFO:teuthology.orchestra.run.trial185.stdout: "events": [ 2026-02-23T13:11:42.639 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.639 INFO:teuthology.orchestra.run.trial185.stdout: "event": "initiated", 2026-02-23T13:11:42.639 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:11:42.560418+0000", 2026-02-23T13:11:42.639 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 0 2026-02-23T13:11:42.639 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.639 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.639 INFO:teuthology.orchestra.run.trial185.stdout: "event": "header_read", 2026-02-23T13:11:42.639 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:11:42.560418+0000", 2026-02-23T13:11:42.639 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 0 2026-02-23T13:11:42.639 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.639 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.639 INFO:teuthology.orchestra.run.trial185.stdout: "event": "throttled", 2026-02-23T13:11:42.639 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:11:42.560429+0000", 2026-02-23T13:11:42.639 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 1.1742e-05 2026-02-23T13:11:42.639 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.639 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.639 INFO:teuthology.orchestra.run.trial185.stdout: "event": "all_read", 2026-02-23T13:11:42.639 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:11:42.560465+0000", 2026-02-23T13:11:42.639 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 3.5036000000000001e-05 2026-02-23T13:11:42.640 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.640 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.640 INFO:teuthology.orchestra.run.trial185.stdout: "event": "dispatched", 2026-02-23T13:11:42.640 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:11:42.560469+0000", 2026-02-23T13:11:42.640 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 4.899e-06 2026-02-23T13:11:42.640 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.640 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.640 INFO:teuthology.orchestra.run.trial185.stdout: "event": "queued_for_pg", 2026-02-23T13:11:42.640 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:11:42.560472+0000", 2026-02-23T13:11:42.640 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 2.875e-06 2026-02-23T13:11:42.640 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.640 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.640 INFO:teuthology.orchestra.run.trial185.stdout: "event": "reached_pg", 2026-02-23T13:11:42.640 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:11:42.567989+0000", 2026-02-23T13:11:42.640 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 0.007516277 2026-02-23T13:11:42.640 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.640 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.640 INFO:teuthology.orchestra.run.trial185.stdout: "event": "started", 2026-02-23T13:11:42.640 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:11:42.568032+0000", 2026-02-23T13:11:42.640 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 4.3522000000000001e-05 2026-02-23T13:11:42.640 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.640 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.640 INFO:teuthology.orchestra.run.trial185.stdout: "event": "waiting for subops from 0,1", 2026-02-23T13:11:42.640 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:11:42.568158+0000", 2026-02-23T13:11:42.640 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 0.00012604599999999999 2026-02-23T13:11:42.640 INFO:teuthology.orchestra.run.trial185.stdout: } 2026-02-23T13:11:42.640 INFO:teuthology.orchestra.run.trial185.stdout: ] 2026-02-23T13:11:42.640 INFO:teuthology.orchestra.run.trial185.stdout: } 2026-02-23T13:11:42.640 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.641 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.641 INFO:teuthology.orchestra.run.trial185.stdout: "description": "osd_op(client.14643.0:32562 4.9 4.55b198c9 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e129)", 2026-02-23T13:11:42.641 INFO:teuthology.orchestra.run.trial185.stdout: "initiated_at": "2026-02-23T13:11:42.562480+0000", 2026-02-23T13:11:42.641 INFO:teuthology.orchestra.run.trial185.stdout: "age": 0.0063978840000000004, 2026-02-23T13:11:42.641 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 0.0067726959999999999, 2026-02-23T13:11:42.641 INFO:teuthology.orchestra.run.trial185.stdout: "continuous": false, 2026-02-23T13:11:42.641 INFO:teuthology.orchestra.run.trial185.stdout: "type_data": { 2026-02-23T13:11:42.641 INFO:teuthology.orchestra.run.trial185.stdout: "flag_point": "queued for pg", 2026-02-23T13:11:42.641 INFO:teuthology.orchestra.run.trial185.stdout: "client_info": { 2026-02-23T13:11:42.641 INFO:teuthology.orchestra.run.trial185.stdout: "client": "client.14643", 2026-02-23T13:11:42.641 INFO:teuthology.orchestra.run.trial185.stdout: "client_addr": "10.20.193.68:0/3888608980", 2026-02-23T13:11:42.641 INFO:teuthology.orchestra.run.trial185.stdout: "tid": 32562 2026-02-23T13:11:42.641 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.641 INFO:teuthology.orchestra.run.trial185.stdout: "events": [ 2026-02-23T13:11:42.641 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.641 INFO:teuthology.orchestra.run.trial185.stdout: "event": "initiated", 2026-02-23T13:11:42.641 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:11:42.562480+0000", 2026-02-23T13:11:42.641 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 0 2026-02-23T13:11:42.641 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.641 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.641 INFO:teuthology.orchestra.run.trial185.stdout: "event": "header_read", 2026-02-23T13:11:42.641 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:11:42.562480+0000", 2026-02-23T13:11:42.641 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 0 2026-02-23T13:11:42.641 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.641 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.641 INFO:teuthology.orchestra.run.trial185.stdout: "event": "throttled", 2026-02-23T13:11:42.641 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:11:42.562492+0000", 2026-02-23T13:11:42.641 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 1.2303000000000001e-05 2026-02-23T13:11:42.641 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.642 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.642 INFO:teuthology.orchestra.run.trial185.stdout: "event": "all_read", 2026-02-23T13:11:42.642 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:11:42.562528+0000", 2026-02-23T13:11:42.642 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 3.5336000000000002e-05 2026-02-23T13:11:42.642 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.642 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.642 INFO:teuthology.orchestra.run.trial185.stdout: "event": "dispatched", 2026-02-23T13:11:42.642 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:11:42.562533+0000", 2026-02-23T13:11:42.642 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 5.1000000000000003e-06 2026-02-23T13:11:42.642 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.642 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.642 INFO:teuthology.orchestra.run.trial185.stdout: "event": "queued_for_pg", 2026-02-23T13:11:42.642 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:11:42.562536+0000", 2026-02-23T13:11:42.642 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 3.4460000000000001e-06 2026-02-23T13:11:42.642 INFO:teuthology.orchestra.run.trial185.stdout: } 2026-02-23T13:11:42.642 INFO:teuthology.orchestra.run.trial185.stdout: ] 2026-02-23T13:11:42.642 INFO:teuthology.orchestra.run.trial185.stdout: } 2026-02-23T13:11:42.642 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.642 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.642 INFO:teuthology.orchestra.run.trial185.stdout: "description": "osd_op(client.14643.0:32544 4.1 4:8bfcdf7f:::benchmark_data_trial068_96731_object4067:head [set-alloc-hint object_size 65536 write_size 8192,write 57344~8192 in=8192b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e129)", 2026-02-23T13:11:42.642 INFO:teuthology.orchestra.run.trial185.stdout: "initiated_at": "2026-02-23T13:11:42.558507+0000", 2026-02-23T13:11:42.642 INFO:teuthology.orchestra.run.trial185.stdout: "age": 0.010371461, 2026-02-23T13:11:42.642 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 0.010758515999999999, 2026-02-23T13:11:42.642 INFO:teuthology.orchestra.run.trial185.stdout: "continuous": false, 2026-02-23T13:11:42.642 INFO:teuthology.orchestra.run.trial185.stdout: "type_data": { 2026-02-23T13:11:42.642 INFO:teuthology.orchestra.run.trial185.stdout: "flag_point": "waiting for sub ops", 2026-02-23T13:11:42.642 INFO:teuthology.orchestra.run.trial185.stdout: "client_info": { 2026-02-23T13:11:42.642 INFO:teuthology.orchestra.run.trial185.stdout: "client": "client.14643", 2026-02-23T13:11:42.642 INFO:teuthology.orchestra.run.trial185.stdout: "client_addr": "10.20.193.68:0/3888608980", 2026-02-23T13:11:42.642 INFO:teuthology.orchestra.run.trial185.stdout: "tid": 32544 2026-02-23T13:11:42.642 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.643 INFO:teuthology.orchestra.run.trial185.stdout: "events": [ 2026-02-23T13:11:42.643 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.643 INFO:teuthology.orchestra.run.trial185.stdout: "event": "initiated", 2026-02-23T13:11:42.643 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:11:42.558507+0000", 2026-02-23T13:11:42.643 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 0 2026-02-23T13:11:42.643 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.643 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.643 INFO:teuthology.orchestra.run.trial185.stdout: "event": "header_read", 2026-02-23T13:11:42.643 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:11:42.558507+0000", 2026-02-23T13:11:42.643 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 0 2026-02-23T13:11:42.643 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.643 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.643 INFO:teuthology.orchestra.run.trial185.stdout: "event": "throttled", 2026-02-23T13:11:42.643 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:11:42.558516+0000", 2026-02-23T13:11:42.643 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 9.1069999999999994e-06 2026-02-23T13:11:42.643 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.644 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.644 INFO:teuthology.orchestra.run.trial185.stdout: "event": "all_read", 2026-02-23T13:11:42.644 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:11:42.558544+0000", 2026-02-23T13:11:42.644 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 2.8362999999999999e-05 2026-02-23T13:11:42.644 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.644 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.644 INFO:teuthology.orchestra.run.trial185.stdout: "event": "dispatched", 2026-02-23T13:11:42.644 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:11:42.558548+0000", 2026-02-23T13:11:42.644 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 3.7469999999999999e-06 2026-02-23T13:11:42.644 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.644 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.644 INFO:teuthology.orchestra.run.trial185.stdout: "event": "queued_for_pg", 2026-02-23T13:11:42.644 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:11:42.558550+0000", 2026-02-23T13:11:42.644 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 2.4650000000000001e-06 2026-02-23T13:11:42.644 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.644 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.644 INFO:teuthology.orchestra.run.trial185.stdout: "event": "reached_pg", 2026-02-23T13:11:42.644 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:11:42.567212+0000", 2026-02-23T13:11:42.644 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 0.0086615909999999997 2026-02-23T13:11:42.644 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.644 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.644 INFO:teuthology.orchestra.run.trial185.stdout: "event": "started", 2026-02-23T13:11:42.644 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:11:42.567266+0000", 2026-02-23T13:11:42.644 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 5.3971000000000002e-05 2026-02-23T13:11:42.644 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.644 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.644 INFO:teuthology.orchestra.run.trial185.stdout: "event": "waiting for subops from 0,3", 2026-02-23T13:11:42.644 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:11:42.567366+0000", 2026-02-23T13:11:42.644 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 0.000100358 2026-02-23T13:11:42.645 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.645 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.645 INFO:teuthology.orchestra.run.trial185.stdout: "event": "op_commit", 2026-02-23T13:11:42.645 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:11:42.567938+0000", 2026-02-23T13:11:42.645 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 0.00057191999999999996 2026-02-23T13:11:42.645 INFO:teuthology.orchestra.run.trial185.stdout: } 2026-02-23T13:11:42.645 INFO:teuthology.orchestra.run.trial185.stdout: ] 2026-02-23T13:11:42.645 INFO:teuthology.orchestra.run.trial185.stdout: } 2026-02-23T13:11:42.645 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.645 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.645 INFO:teuthology.orchestra.run.trial185.stdout: "description": "osd_op(client.14643.0:32563 4.9 4.55b198c9 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e129)", 2026-02-23T13:11:42.645 INFO:teuthology.orchestra.run.trial185.stdout: "initiated_at": "2026-02-23T13:11:42.562548+0000", 2026-02-23T13:11:42.645 INFO:teuthology.orchestra.run.trial185.stdout: "age": 0.0063299269999999999, 2026-02-23T13:11:42.645 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 0.0067356170000000002, 2026-02-23T13:11:42.645 INFO:teuthology.orchestra.run.trial185.stdout: "continuous": false, 2026-02-23T13:11:42.645 INFO:teuthology.orchestra.run.trial185.stdout: "type_data": { 2026-02-23T13:11:42.645 INFO:teuthology.orchestra.run.trial185.stdout: "flag_point": "queued for pg", 2026-02-23T13:11:42.645 INFO:teuthology.orchestra.run.trial185.stdout: "client_info": { 2026-02-23T13:11:42.645 INFO:teuthology.orchestra.run.trial185.stdout: "client": "client.14643", 2026-02-23T13:11:42.645 INFO:teuthology.orchestra.run.trial185.stdout: "client_addr": "10.20.193.68:0/3888608980", 2026-02-23T13:11:42.645 INFO:teuthology.orchestra.run.trial185.stdout: "tid": 32563 2026-02-23T13:11:42.645 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.645 INFO:teuthology.orchestra.run.trial185.stdout: "events": [ 2026-02-23T13:11:42.645 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.645 INFO:teuthology.orchestra.run.trial185.stdout: "event": "initiated", 2026-02-23T13:11:42.645 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:11:42.562548+0000", 2026-02-23T13:11:42.645 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 0 2026-02-23T13:11:42.645 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.645 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.645 INFO:teuthology.orchestra.run.trial185.stdout: "event": "header_read", 2026-02-23T13:11:42.645 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:11:42.562548+0000", 2026-02-23T13:11:42.646 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 0 2026-02-23T13:11:42.646 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.646 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.646 INFO:teuthology.orchestra.run.trial185.stdout: "event": "throttled", 2026-02-23T13:11:42.646 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:11:42.562559+0000", 2026-02-23T13:11:42.646 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 1.1041e-05 2026-02-23T13:11:42.646 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.646 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.646 INFO:teuthology.orchestra.run.trial185.stdout: "event": "all_read", 2026-02-23T13:11:42.646 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:11:42.562607+0000", 2026-02-23T13:11:42.646 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 4.7488999999999999e-05 2026-02-23T13:11:42.646 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.646 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.646 INFO:teuthology.orchestra.run.trial185.stdout: "event": "dispatched", 2026-02-23T13:11:42.646 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:11:42.562612+0000", 2026-02-23T13:11:42.646 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 4.9989999999999999e-06 2026-02-23T13:11:42.646 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.646 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.646 INFO:teuthology.orchestra.run.trial185.stdout: "event": "queued_for_pg", 2026-02-23T13:11:42.646 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:11:42.562616+0000", 2026-02-23T13:11:42.646 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 4.1679999999999997e-06 2026-02-23T13:11:42.646 INFO:teuthology.orchestra.run.trial185.stdout: } 2026-02-23T13:11:42.646 INFO:teuthology.orchestra.run.trial185.stdout: ] 2026-02-23T13:11:42.646 INFO:teuthology.orchestra.run.trial185.stdout: } 2026-02-23T13:11:42.646 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.646 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.646 INFO:teuthology.orchestra.run.trial185.stdout: "description": "osd_op(client.14643.0:32588 4.a 4:5eaf47c2:::benchmark_data_trial068_96731_object4073:head [set-alloc-hint object_size 65536 write_size 8192,write 24576~8192 in=8192b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e129)", 2026-02-23T13:11:42.646 INFO:teuthology.orchestra.run.trial185.stdout: "initiated_at": "2026-02-23T13:11:42.564396+0000", 2026-02-23T13:11:42.646 INFO:teuthology.orchestra.run.trial185.stdout: "age": 0.0044823190000000002, 2026-02-23T13:11:42.646 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 0.0049089880000000004, 2026-02-23T13:11:42.646 INFO:teuthology.orchestra.run.trial185.stdout: "continuous": false, 2026-02-23T13:11:42.646 INFO:teuthology.orchestra.run.trial185.stdout: "type_data": { 2026-02-23T13:11:42.646 INFO:teuthology.orchestra.run.trial185.stdout: "flag_point": "waiting for sub ops", 2026-02-23T13:11:42.647 INFO:teuthology.orchestra.run.trial185.stdout: "client_info": { 2026-02-23T13:11:42.647 INFO:teuthology.orchestra.run.trial185.stdout: "client": "client.14643", 2026-02-23T13:11:42.647 INFO:teuthology.orchestra.run.trial185.stdout: "client_addr": "10.20.193.68:0/3888608980", 2026-02-23T13:11:42.647 INFO:teuthology.orchestra.run.trial185.stdout: "tid": 32588 2026-02-23T13:11:42.647 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.647 INFO:teuthology.orchestra.run.trial185.stdout: "events": [ 2026-02-23T13:11:42.647 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.647 INFO:teuthology.orchestra.run.trial185.stdout: "event": "initiated", 2026-02-23T13:11:42.647 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:11:42.564396+0000", 2026-02-23T13:11:42.647 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 0 2026-02-23T13:11:42.647 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.647 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.647 INFO:teuthology.orchestra.run.trial185.stdout: "event": "header_read", 2026-02-23T13:11:42.647 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:11:42.564396+0000", 2026-02-23T13:11:42.647 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 0 2026-02-23T13:11:42.647 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.647 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.647 INFO:teuthology.orchestra.run.trial185.stdout: "event": "throttled", 2026-02-23T13:11:42.647 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:11:42.564418+0000", 2026-02-23T13:11:42.647 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 2.2141999999999999e-05 2026-02-23T13:11:42.647 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.647 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.647 INFO:teuthology.orchestra.run.trial185.stdout: "event": "all_read", 2026-02-23T13:11:42.647 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:11:42.564470+0000", 2026-02-23T13:11:42.647 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 5.2448000000000003e-05 2026-02-23T13:11:42.647 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.647 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.647 INFO:teuthology.orchestra.run.trial185.stdout: "event": "dispatched", 2026-02-23T13:11:42.647 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:11:42.564473+0000", 2026-02-23T13:11:42.647 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 3.0759999999999999e-06 2026-02-23T13:11:42.647 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.647 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.647 INFO:teuthology.orchestra.run.trial185.stdout: "event": "queued_for_pg", 2026-02-23T13:11:42.647 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:11:42.564475+0000", 2026-02-23T13:11:42.648 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 1.412e-06 2026-02-23T13:11:42.648 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.648 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.648 INFO:teuthology.orchestra.run.trial185.stdout: "event": "reached_pg", 2026-02-23T13:11:42.648 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:11:42.566011+0000", 2026-02-23T13:11:42.648 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 0.001535965 2026-02-23T13:11:42.648 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.648 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.648 INFO:teuthology.orchestra.run.trial185.stdout: "event": "started", 2026-02-23T13:11:42.648 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:11:42.566057+0000", 2026-02-23T13:11:42.648 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 4.6597000000000001e-05 2026-02-23T13:11:42.648 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.648 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.648 INFO:teuthology.orchestra.run.trial185.stdout: "event": "waiting for subops from 1,3", 2026-02-23T13:11:42.648 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:11:42.566149+0000", 2026-02-23T13:11:42.648 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 9.1261000000000006e-05 2026-02-23T13:11:42.648 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.648 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.648 INFO:teuthology.orchestra.run.trial185.stdout: "event": "op_commit", 2026-02-23T13:11:42.648 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:11:42.567522+0000", 2026-02-23T13:11:42.648 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 0.0013730599999999999 2026-02-23T13:11:42.648 INFO:teuthology.orchestra.run.trial185.stdout: } 2026-02-23T13:11:42.648 INFO:teuthology.orchestra.run.trial185.stdout: ] 2026-02-23T13:11:42.648 INFO:teuthology.orchestra.run.trial185.stdout: } 2026-02-23T13:11:42.648 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.648 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.648 INFO:teuthology.orchestra.run.trial185.stdout: "description": "osd_op(client.14643.0:32555 4.9 4:9a47f871:::benchmark_data_trial068_96731_object4069:head [set-alloc-hint object_size 65536 write_size 8192,write 16384~8192 in=8192b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e129)", 2026-02-23T13:11:42.648 INFO:teuthology.orchestra.run.trial185.stdout: "initiated_at": "2026-02-23T13:11:42.560493+0000", 2026-02-23T13:11:42.648 INFO:teuthology.orchestra.run.trial185.stdout: "age": 0.0083852739999999999, 2026-02-23T13:11:42.648 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 0.0088383620000000006, 2026-02-23T13:11:42.648 INFO:teuthology.orchestra.run.trial185.stdout: "continuous": false, 2026-02-23T13:11:42.648 INFO:teuthology.orchestra.run.trial185.stdout: "type_data": { 2026-02-23T13:11:42.648 INFO:teuthology.orchestra.run.trial185.stdout: "flag_point": "waiting for sub ops", 2026-02-23T13:11:42.648 INFO:teuthology.orchestra.run.trial185.stdout: "client_info": { 2026-02-23T13:11:42.649 INFO:teuthology.orchestra.run.trial185.stdout: "client": "client.14643", 2026-02-23T13:11:42.649 INFO:teuthology.orchestra.run.trial185.stdout: "client_addr": "10.20.193.68:0/3888608980", 2026-02-23T13:11:42.649 INFO:teuthology.orchestra.run.trial185.stdout: "tid": 32555 2026-02-23T13:11:42.649 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.649 INFO:teuthology.orchestra.run.trial185.stdout: "events": [ 2026-02-23T13:11:42.649 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.649 INFO:teuthology.orchestra.run.trial185.stdout: "event": "initiated", 2026-02-23T13:11:42.649 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:11:42.560493+0000", 2026-02-23T13:11:42.649 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 0 2026-02-23T13:11:42.649 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.649 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.649 INFO:teuthology.orchestra.run.trial185.stdout: "event": "header_read", 2026-02-23T13:11:42.649 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:11:42.560493+0000", 2026-02-23T13:11:42.649 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 0 2026-02-23T13:11:42.649 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.649 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.649 INFO:teuthology.orchestra.run.trial185.stdout: "event": "throttled", 2026-02-23T13:11:42.649 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:11:42.560506+0000", 2026-02-23T13:11:42.649 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 1.3436e-05 2026-02-23T13:11:42.649 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.649 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.649 INFO:teuthology.orchestra.run.trial185.stdout: "event": "all_read", 2026-02-23T13:11:42.649 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:11:42.560558+0000", 2026-02-23T13:11:42.649 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 5.2017e-05 2026-02-23T13:11:42.649 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.649 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.649 INFO:teuthology.orchestra.run.trial185.stdout: "event": "dispatched", 2026-02-23T13:11:42.649 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:11:42.560564+0000", 2026-02-23T13:11:42.649 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 6.0510000000000002e-06 2026-02-23T13:11:42.649 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.649 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.649 INFO:teuthology.orchestra.run.trial185.stdout: "event": "queued_for_pg", 2026-02-23T13:11:42.649 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:11:42.560567+0000", 2026-02-23T13:11:42.649 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 2.305e-06 2026-02-23T13:11:42.650 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.650 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.650 INFO:teuthology.orchestra.run.trial185.stdout: "event": "reached_pg", 2026-02-23T13:11:42.650 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:11:42.568552+0000", 2026-02-23T13:11:42.650 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 0.0079849840000000005 2026-02-23T13:11:42.650 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.650 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.650 INFO:teuthology.orchestra.run.trial185.stdout: "event": "started", 2026-02-23T13:11:42.650 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:11:42.568599+0000", 2026-02-23T13:11:42.650 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 4.7429000000000003e-05 2026-02-23T13:11:42.650 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.650 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.650 INFO:teuthology.orchestra.run.trial185.stdout: "event": "waiting for subops from 0,1", 2026-02-23T13:11:42.650 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:11:42.568684+0000", 2026-02-23T13:11:42.650 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 8.5500000000000005e-05 2026-02-23T13:11:42.650 INFO:teuthology.orchestra.run.trial185.stdout: } 2026-02-23T13:11:42.650 INFO:teuthology.orchestra.run.trial185.stdout: ] 2026-02-23T13:11:42.650 INFO:teuthology.orchestra.run.trial185.stdout: } 2026-02-23T13:11:42.650 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.650 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.650 INFO:teuthology.orchestra.run.trial185.stdout: "description": "osd_op(client.14643.0:32589 4.a 4:5eaf47c2:::benchmark_data_trial068_96731_object4073:head [set-alloc-hint object_size 65536 write_size 8192,write 32768~8192 in=8192b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e129)", 2026-02-23T13:11:42.650 INFO:teuthology.orchestra.run.trial185.stdout: "initiated_at": "2026-02-23T13:11:42.564484+0000", 2026-02-23T13:11:42.650 INFO:teuthology.orchestra.run.trial185.stdout: "age": 0.0043944539999999999, 2026-02-23T13:11:42.650 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 0.0048741319999999998, 2026-02-23T13:11:42.650 INFO:teuthology.orchestra.run.trial185.stdout: "continuous": false, 2026-02-23T13:11:42.650 INFO:teuthology.orchestra.run.trial185.stdout: "type_data": { 2026-02-23T13:11:42.650 INFO:teuthology.orchestra.run.trial185.stdout: "flag_point": "waiting for sub ops", 2026-02-23T13:11:42.650 INFO:teuthology.orchestra.run.trial185.stdout: "client_info": { 2026-02-23T13:11:42.650 INFO:teuthology.orchestra.run.trial185.stdout: "client": "client.14643", 2026-02-23T13:11:42.650 INFO:teuthology.orchestra.run.trial185.stdout: "client_addr": "10.20.193.68:0/3888608980", 2026-02-23T13:11:42.650 INFO:teuthology.orchestra.run.trial185.stdout: "tid": 32589 2026-02-23T13:11:42.650 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.650 INFO:teuthology.orchestra.run.trial185.stdout: "events": [ 2026-02-23T13:11:42.650 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.650 INFO:teuthology.orchestra.run.trial185.stdout: "event": "initiated", 2026-02-23T13:11:42.651 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:11:42.564484+0000", 2026-02-23T13:11:42.651 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 0 2026-02-23T13:11:42.651 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.651 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.651 INFO:teuthology.orchestra.run.trial185.stdout: "event": "header_read", 2026-02-23T13:11:42.651 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:11:42.564484+0000", 2026-02-23T13:11:42.651 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 0 2026-02-23T13:11:42.651 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.651 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.651 INFO:teuthology.orchestra.run.trial185.stdout: "event": "throttled", 2026-02-23T13:11:42.651 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:11:42.564493+0000", 2026-02-23T13:11:42.651 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 9.1670000000000005e-06 2026-02-23T13:11:42.651 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.651 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.651 INFO:teuthology.orchestra.run.trial185.stdout: "event": "all_read", 2026-02-23T13:11:42.651 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:11:42.564518+0000", 2026-02-23T13:11:42.651 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 2.4836000000000001e-05 2026-02-23T13:11:42.651 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.651 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.651 INFO:teuthology.orchestra.run.trial185.stdout: "event": "dispatched", 2026-02-23T13:11:42.651 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:11:42.564529+0000", 2026-02-23T13:11:42.651 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 1.1482e-05 2026-02-23T13:11:42.651 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.651 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.651 INFO:teuthology.orchestra.run.trial185.stdout: "event": "queued_for_pg", 2026-02-23T13:11:42.651 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:11:42.564535+0000", 2026-02-23T13:11:42.651 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 5.9209999999999997e-06 2026-02-23T13:11:42.651 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.651 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.651 INFO:teuthology.orchestra.run.trial185.stdout: "event": "reached_pg", 2026-02-23T13:11:42.651 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:11:42.566531+0000", 2026-02-23T13:11:42.651 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 0.001996416 2026-02-23T13:11:42.651 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.651 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.651 INFO:teuthology.orchestra.run.trial185.stdout: "event": "started", 2026-02-23T13:11:42.652 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:11:42.566564+0000", 2026-02-23T13:11:42.652 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 3.2781999999999999e-05 2026-02-23T13:11:42.652 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.652 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.652 INFO:teuthology.orchestra.run.trial185.stdout: "event": "waiting for subops from 1,3", 2026-02-23T13:11:42.652 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:11:42.566612+0000", 2026-02-23T13:11:42.652 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 4.7889000000000002e-05 2026-02-23T13:11:42.652 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.652 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.652 INFO:teuthology.orchestra.run.trial185.stdout: "event": "op_commit", 2026-02-23T13:11:42.652 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:11:42.568567+0000", 2026-02-23T13:11:42.652 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 0.001954608 2026-02-23T13:11:42.652 INFO:teuthology.orchestra.run.trial185.stdout: } 2026-02-23T13:11:42.652 INFO:teuthology.orchestra.run.trial185.stdout: ] 2026-02-23T13:11:42.652 INFO:teuthology.orchestra.run.trial185.stdout: } 2026-02-23T13:11:42.652 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.652 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.652 INFO:teuthology.orchestra.run.trial185.stdout: "description": "osd_op(client.14643.0:32564 4.9 4.55b198c9 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e129)", 2026-02-23T13:11:42.652 INFO:teuthology.orchestra.run.trial185.stdout: "initiated_at": "2026-02-23T13:11:42.562628+0000", 2026-02-23T13:11:42.652 INFO:teuthology.orchestra.run.trial185.stdout: "age": 0.0062499469999999996, 2026-02-23T13:11:42.652 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 0.0067527380000000003, 2026-02-23T13:11:42.652 INFO:teuthology.orchestra.run.trial185.stdout: "continuous": false, 2026-02-23T13:11:42.652 INFO:teuthology.orchestra.run.trial185.stdout: "type_data": { 2026-02-23T13:11:42.652 INFO:teuthology.orchestra.run.trial185.stdout: "flag_point": "queued for pg", 2026-02-23T13:11:42.652 INFO:teuthology.orchestra.run.trial185.stdout: "client_info": { 2026-02-23T13:11:42.652 INFO:teuthology.orchestra.run.trial185.stdout: "client": "client.14643", 2026-02-23T13:11:42.652 INFO:teuthology.orchestra.run.trial185.stdout: "client_addr": "10.20.193.68:0/3888608980", 2026-02-23T13:11:42.652 INFO:teuthology.orchestra.run.trial185.stdout: "tid": 32564 2026-02-23T13:11:42.652 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.652 INFO:teuthology.orchestra.run.trial185.stdout: "events": [ 2026-02-23T13:11:42.652 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.652 INFO:teuthology.orchestra.run.trial185.stdout: "event": "initiated", 2026-02-23T13:11:42.652 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:11:42.562628+0000", 2026-02-23T13:11:42.652 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 0 2026-02-23T13:11:42.652 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.652 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.653 INFO:teuthology.orchestra.run.trial185.stdout: "event": "header_read", 2026-02-23T13:11:42.653 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:11:42.562628+0000", 2026-02-23T13:11:42.653 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 0 2026-02-23T13:11:42.653 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.653 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.653 INFO:teuthology.orchestra.run.trial185.stdout: "event": "throttled", 2026-02-23T13:11:42.653 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:11:42.562641+0000", 2026-02-23T13:11:42.653 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 1.3004e-05 2026-02-23T13:11:42.653 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.653 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.653 INFO:teuthology.orchestra.run.trial185.stdout: "event": "all_read", 2026-02-23T13:11:42.653 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:11:42.562686+0000", 2026-02-23T13:11:42.653 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 4.4694000000000003e-05 2026-02-23T13:11:42.653 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.653 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.653 INFO:teuthology.orchestra.run.trial185.stdout: "event": "dispatched", 2026-02-23T13:11:42.653 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:11:42.562692+0000", 2026-02-23T13:11:42.653 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 5.7910000000000001e-06 2026-02-23T13:11:42.653 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.653 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.653 INFO:teuthology.orchestra.run.trial185.stdout: "event": "queued_for_pg", 2026-02-23T13:11:42.653 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:11:42.562694+0000", 2026-02-23T13:11:42.653 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 2.7750000000000001e-06 2026-02-23T13:11:42.653 INFO:teuthology.orchestra.run.trial185.stdout: } 2026-02-23T13:11:42.653 INFO:teuthology.orchestra.run.trial185.stdout: ] 2026-02-23T13:11:42.653 INFO:teuthology.orchestra.run.trial185.stdout: } 2026-02-23T13:11:42.653 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.653 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.653 INFO:teuthology.orchestra.run.trial185.stdout: "description": "osd_op(client.14643.0:32565 4.9 4.55b198c9 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e129)", 2026-02-23T13:11:42.653 INFO:teuthology.orchestra.run.trial185.stdout: "initiated_at": "2026-02-23T13:11:42.562720+0000", 2026-02-23T13:11:42.653 INFO:teuthology.orchestra.run.trial185.stdout: "age": 0.0061575939999999997, 2026-02-23T13:11:42.653 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 0.0066794209999999996, 2026-02-23T13:11:42.653 INFO:teuthology.orchestra.run.trial185.stdout: "continuous": false, 2026-02-23T13:11:42.653 INFO:teuthology.orchestra.run.trial185.stdout: "type_data": { 2026-02-23T13:11:42.653 INFO:teuthology.orchestra.run.trial185.stdout: "flag_point": "queued for pg", 2026-02-23T13:11:42.653 INFO:teuthology.orchestra.run.trial185.stdout: "client_info": { 2026-02-23T13:11:42.654 INFO:teuthology.orchestra.run.trial185.stdout: "client": "client.14643", 2026-02-23T13:11:42.654 INFO:teuthology.orchestra.run.trial185.stdout: "client_addr": "10.20.193.68:0/3888608980", 2026-02-23T13:11:42.654 INFO:teuthology.orchestra.run.trial185.stdout: "tid": 32565 2026-02-23T13:11:42.654 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.654 INFO:teuthology.orchestra.run.trial185.stdout: "events": [ 2026-02-23T13:11:42.654 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.654 INFO:teuthology.orchestra.run.trial185.stdout: "event": "initiated", 2026-02-23T13:11:42.654 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:11:42.562720+0000", 2026-02-23T13:11:42.654 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 0 2026-02-23T13:11:42.654 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.654 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.654 INFO:teuthology.orchestra.run.trial185.stdout: "event": "header_read", 2026-02-23T13:11:42.654 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:11:42.562720+0000", 2026-02-23T13:11:42.654 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 0 2026-02-23T13:11:42.654 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.654 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.654 INFO:teuthology.orchestra.run.trial185.stdout: "event": "throttled", 2026-02-23T13:11:42.654 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:11:42.562735+0000", 2026-02-23T13:11:42.654 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 1.4858000000000001e-05 2026-02-23T13:11:42.654 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.654 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.654 INFO:teuthology.orchestra.run.trial185.stdout: "event": "all_read", 2026-02-23T13:11:42.654 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:11:42.562799+0000", 2026-02-23T13:11:42.654 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 6.3818999999999993e-05 2026-02-23T13:11:42.654 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.654 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.654 INFO:teuthology.orchestra.run.trial185.stdout: "event": "dispatched", 2026-02-23T13:11:42.654 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:11:42.562804+0000", 2026-02-23T13:11:42.654 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 4.9490000000000004e-06 2026-02-23T13:11:42.654 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.654 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.654 INFO:teuthology.orchestra.run.trial185.stdout: "event": "queued_for_pg", 2026-02-23T13:11:42.654 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:11:42.562807+0000", 2026-02-23T13:11:42.654 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 3.1659999999999998e-06 2026-02-23T13:11:42.654 INFO:teuthology.orchestra.run.trial185.stdout: } 2026-02-23T13:11:42.654 INFO:teuthology.orchestra.run.trial185.stdout: ] 2026-02-23T13:11:42.654 INFO:teuthology.orchestra.run.trial185.stdout: } 2026-02-23T13:11:42.655 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.655 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.655 INFO:teuthology.orchestra.run.trial185.stdout: "description": "osd_op(client.14643.0:32590 4.a 4:5eaf47c2:::benchmark_data_trial068_96731_object4073:head [set-alloc-hint object_size 65536 write_size 8192,write 40960~8192 in=8192b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e129)", 2026-02-23T13:11:42.655 INFO:teuthology.orchestra.run.trial185.stdout: "initiated_at": "2026-02-23T13:11:42.564542+0000", 2026-02-23T13:11:42.655 INFO:teuthology.orchestra.run.trial185.stdout: "age": 0.0043358149999999998, 2026-02-23T13:11:42.655 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 0.0048705160000000001, 2026-02-23T13:11:42.655 INFO:teuthology.orchestra.run.trial185.stdout: "continuous": false, 2026-02-23T13:11:42.655 INFO:teuthology.orchestra.run.trial185.stdout: "type_data": { 2026-02-23T13:11:42.655 INFO:teuthology.orchestra.run.trial185.stdout: "flag_point": "waiting for sub ops", 2026-02-23T13:11:42.655 INFO:teuthology.orchestra.run.trial185.stdout: "client_info": { 2026-02-23T13:11:42.655 INFO:teuthology.orchestra.run.trial185.stdout: "client": "client.14643", 2026-02-23T13:11:42.655 INFO:teuthology.orchestra.run.trial185.stdout: "client_addr": "10.20.193.68:0/3888608980", 2026-02-23T13:11:42.655 INFO:teuthology.orchestra.run.trial185.stdout: "tid": 32590 2026-02-23T13:11:42.655 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.655 INFO:teuthology.orchestra.run.trial185.stdout: "events": [ 2026-02-23T13:11:42.655 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.655 INFO:teuthology.orchestra.run.trial185.stdout: "event": "initiated", 2026-02-23T13:11:42.655 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:11:42.564542+0000", 2026-02-23T13:11:42.655 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 0 2026-02-23T13:11:42.655 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.655 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.655 INFO:teuthology.orchestra.run.trial185.stdout: "event": "header_read", 2026-02-23T13:11:42.655 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:11:42.564542+0000", 2026-02-23T13:11:42.655 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 0 2026-02-23T13:11:42.655 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.655 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.655 INFO:teuthology.orchestra.run.trial185.stdout: "event": "throttled", 2026-02-23T13:11:42.655 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:11:42.564550+0000", 2026-02-23T13:11:42.655 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 8.1659999999999994e-06 2026-02-23T13:11:42.655 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.655 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.655 INFO:teuthology.orchestra.run.trial185.stdout: "event": "all_read", 2026-02-23T13:11:42.655 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:11:42.564574+0000", 2026-02-23T13:11:42.655 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 2.3714e-05 2026-02-23T13:11:42.655 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.655 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.655 INFO:teuthology.orchestra.run.trial185.stdout: "event": "dispatched", 2026-02-23T13:11:42.655 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:11:42.564583+0000", 2026-02-23T13:11:42.656 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 8.8470000000000002e-06 2026-02-23T13:11:42.656 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.656 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.656 INFO:teuthology.orchestra.run.trial185.stdout: "event": "queued_for_pg", 2026-02-23T13:11:42.656 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:11:42.564584+0000", 2026-02-23T13:11:42.656 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 1.3319999999999999e-06 2026-02-23T13:11:42.656 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.656 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.656 INFO:teuthology.orchestra.run.trial185.stdout: "event": "reached_pg", 2026-02-23T13:11:42.656 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:11:42.566994+0000", 2026-02-23T13:11:42.656 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 0.0024092290000000001 2026-02-23T13:11:42.656 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.656 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.656 INFO:teuthology.orchestra.run.trial185.stdout: "event": "started", 2026-02-23T13:11:42.656 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:11:42.567036+0000", 2026-02-23T13:11:42.656 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 4.2870999999999997e-05 2026-02-23T13:11:42.656 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.656 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.656 INFO:teuthology.orchestra.run.trial185.stdout: "event": "waiting for subops from 1,3", 2026-02-23T13:11:42.656 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:11:42.567123+0000", 2026-02-23T13:11:42.656 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 8.7012e-05 2026-02-23T13:11:42.656 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.656 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.656 INFO:teuthology.orchestra.run.trial185.stdout: "event": "op_commit", 2026-02-23T13:11:42.656 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:11:42.568578+0000", 2026-02-23T13:11:42.656 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 0.0014549630000000001 2026-02-23T13:11:42.656 INFO:teuthology.orchestra.run.trial185.stdout: } 2026-02-23T13:11:42.656 INFO:teuthology.orchestra.run.trial185.stdout: ] 2026-02-23T13:11:42.656 INFO:teuthology.orchestra.run.trial185.stdout: } 2026-02-23T13:11:42.656 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.656 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.656 INFO:teuthology.orchestra.run.trial185.stdout: "description": "osd_op(client.14643.0:32566 4.9 4.55b198c9 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e129)", 2026-02-23T13:11:42.656 INFO:teuthology.orchestra.run.trial185.stdout: "initiated_at": "2026-02-23T13:11:42.562826+0000", 2026-02-23T13:11:42.656 INFO:teuthology.orchestra.run.trial185.stdout: "age": 0.0060523870000000002, 2026-02-23T13:11:42.656 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 0.0066378829999999998, 2026-02-23T13:11:42.656 INFO:teuthology.orchestra.run.trial185.stdout: "continuous": false, 2026-02-23T13:11:42.656 INFO:teuthology.orchestra.run.trial185.stdout: "type_data": { 2026-02-23T13:11:42.656 INFO:teuthology.orchestra.run.trial185.stdout: "flag_point": "queued for pg", 2026-02-23T13:11:42.657 INFO:teuthology.orchestra.run.trial185.stdout: "client_info": { 2026-02-23T13:11:42.657 INFO:teuthology.orchestra.run.trial185.stdout: "client": "client.14643", 2026-02-23T13:11:42.657 INFO:teuthology.orchestra.run.trial185.stdout: "client_addr": "10.20.193.68:0/3888608980", 2026-02-23T13:11:42.657 INFO:teuthology.orchestra.run.trial185.stdout: "tid": 32566 2026-02-23T13:11:42.657 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.657 INFO:teuthology.orchestra.run.trial185.stdout: "events": [ 2026-02-23T13:11:42.657 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.657 INFO:teuthology.orchestra.run.trial185.stdout: "event": "initiated", 2026-02-23T13:11:42.657 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:11:42.562826+0000", 2026-02-23T13:11:42.657 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 0 2026-02-23T13:11:42.657 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.657 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.657 INFO:teuthology.orchestra.run.trial185.stdout: "event": "header_read", 2026-02-23T13:11:42.657 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:11:42.562826+0000", 2026-02-23T13:11:42.657 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 0 2026-02-23T13:11:42.657 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.657 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.657 INFO:teuthology.orchestra.run.trial185.stdout: "event": "throttled", 2026-02-23T13:11:42.657 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:11:42.562842+0000", 2026-02-23T13:11:42.657 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 1.6290000000000002e-05 2026-02-23T13:11:42.657 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.657 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.657 INFO:teuthology.orchestra.run.trial185.stdout: "event": "all_read", 2026-02-23T13:11:42.657 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:11:42.562889+0000", 2026-02-23T13:11:42.657 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 4.7488999999999999e-05 2026-02-23T13:11:42.657 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.657 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.657 INFO:teuthology.orchestra.run.trial185.stdout: "event": "dispatched", 2026-02-23T13:11:42.657 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:11:42.562901+0000", 2026-02-23T13:11:42.657 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 1.1341e-05 2026-02-23T13:11:42.657 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.657 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.657 INFO:teuthology.orchestra.run.trial185.stdout: "event": "queued_for_pg", 2026-02-23T13:11:42.657 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:11:42.562903+0000", 2026-02-23T13:11:42.657 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 2.2050000000000001e-06 2026-02-23T13:11:42.657 INFO:teuthology.orchestra.run.trial185.stdout: } 2026-02-23T13:11:42.657 INFO:teuthology.orchestra.run.trial185.stdout: ] 2026-02-23T13:11:42.657 INFO:teuthology.orchestra.run.trial185.stdout: } 2026-02-23T13:11:42.657 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.658 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.658 INFO:teuthology.orchestra.run.trial185.stdout: "description": "osd_op(client.14643.0:32567 4.9 4.55b198c9 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e129)", 2026-02-23T13:11:42.658 INFO:teuthology.orchestra.run.trial185.stdout: "initiated_at": "2026-02-23T13:11:42.562918+0000", 2026-02-23T13:11:42.658 INFO:teuthology.orchestra.run.trial185.stdout: "age": 0.0059598940000000003, 2026-02-23T13:11:42.658 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 0.0065617510000000002, 2026-02-23T13:11:42.658 INFO:teuthology.orchestra.run.trial185.stdout: "continuous": false, 2026-02-23T13:11:42.658 INFO:teuthology.orchestra.run.trial185.stdout: "type_data": { 2026-02-23T13:11:42.658 INFO:teuthology.orchestra.run.trial185.stdout: "flag_point": "queued for pg", 2026-02-23T13:11:42.658 INFO:teuthology.orchestra.run.trial185.stdout: "client_info": { 2026-02-23T13:11:42.658 INFO:teuthology.orchestra.run.trial185.stdout: "client": "client.14643", 2026-02-23T13:11:42.658 INFO:teuthology.orchestra.run.trial185.stdout: "client_addr": "10.20.193.68:0/3888608980", 2026-02-23T13:11:42.658 INFO:teuthology.orchestra.run.trial185.stdout: "tid": 32567 2026-02-23T13:11:42.658 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.658 INFO:teuthology.orchestra.run.trial185.stdout: "events": [ 2026-02-23T13:11:42.658 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.658 INFO:teuthology.orchestra.run.trial185.stdout: "event": "initiated", 2026-02-23T13:11:42.658 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:11:42.562918+0000", 2026-02-23T13:11:42.658 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 0 2026-02-23T13:11:42.658 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.658 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.658 INFO:teuthology.orchestra.run.trial185.stdout: "event": "header_read", 2026-02-23T13:11:42.658 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:11:42.562918+0000", 2026-02-23T13:11:42.658 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 0 2026-02-23T13:11:42.658 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.658 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.658 INFO:teuthology.orchestra.run.trial185.stdout: "event": "throttled", 2026-02-23T13:11:42.658 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:11:42.562937+0000", 2026-02-23T13:11:42.658 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 1.8403999999999999e-05 2026-02-23T13:11:42.658 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.658 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.658 INFO:teuthology.orchestra.run.trial185.stdout: "event": "all_read", 2026-02-23T13:11:42.658 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:11:42.562982+0000", 2026-02-23T13:11:42.658 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 4.5495999999999999e-05 2026-02-23T13:11:42.658 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.658 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.658 INFO:teuthology.orchestra.run.trial185.stdout: "event": "dispatched", 2026-02-23T13:11:42.658 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:11:42.562989+0000", 2026-02-23T13:11:42.658 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 7.3629999999999996e-06 2026-02-23T13:11:42.658 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.658 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.659 INFO:teuthology.orchestra.run.trial185.stdout: "event": "queued_for_pg", 2026-02-23T13:11:42.659 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:11:42.562991+0000", 2026-02-23T13:11:42.659 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 1.9139999999999998e-06 2026-02-23T13:11:42.659 INFO:teuthology.orchestra.run.trial185.stdout: } 2026-02-23T13:11:42.659 INFO:teuthology.orchestra.run.trial185.stdout: ] 2026-02-23T13:11:42.659 INFO:teuthology.orchestra.run.trial185.stdout: } 2026-02-23T13:11:42.659 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.659 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.659 INFO:teuthology.orchestra.run.trial185.stdout: "description": "osd_op(client.14643.0:32591 4.a 4:5eaf47c2:::benchmark_data_trial068_96731_object4073:head [set-alloc-hint object_size 65536 write_size 8192,write 49152~8192 in=8192b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e129)", 2026-02-23T13:11:42.659 INFO:teuthology.orchestra.run.trial185.stdout: "initiated_at": "2026-02-23T13:11:42.564857+0000", 2026-02-23T13:11:42.659 INFO:teuthology.orchestra.run.trial185.stdout: "age": 0.0040205750000000002, 2026-02-23T13:11:42.659 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 0.0046351049999999996, 2026-02-23T13:11:42.659 INFO:teuthology.orchestra.run.trial185.stdout: "continuous": false, 2026-02-23T13:11:42.659 INFO:teuthology.orchestra.run.trial185.stdout: "type_data": { 2026-02-23T13:11:42.659 INFO:teuthology.orchestra.run.trial185.stdout: "flag_point": "waiting for sub ops", 2026-02-23T13:11:42.659 INFO:teuthology.orchestra.run.trial185.stdout: "client_info": { 2026-02-23T13:11:42.659 INFO:teuthology.orchestra.run.trial185.stdout: "client": "client.14643", 2026-02-23T13:11:42.659 INFO:teuthology.orchestra.run.trial185.stdout: "client_addr": "10.20.193.68:0/3888608980", 2026-02-23T13:11:42.659 INFO:teuthology.orchestra.run.trial185.stdout: "tid": 32591 2026-02-23T13:11:42.659 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.659 INFO:teuthology.orchestra.run.trial185.stdout: "events": [ 2026-02-23T13:11:42.659 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.659 INFO:teuthology.orchestra.run.trial185.stdout: "event": "initiated", 2026-02-23T13:11:42.659 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:11:42.564857+0000", 2026-02-23T13:11:42.659 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 0 2026-02-23T13:11:42.659 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.659 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.659 INFO:teuthology.orchestra.run.trial185.stdout: "event": "header_read", 2026-02-23T13:11:42.659 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:11:42.564857+0000", 2026-02-23T13:11:42.659 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 0 2026-02-23T13:11:42.659 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.659 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.659 INFO:teuthology.orchestra.run.trial185.stdout: "event": "throttled", 2026-02-23T13:11:42.659 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:11:42.564869+0000", 2026-02-23T13:11:42.659 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 1.1141000000000001e-05 2026-02-23T13:11:42.659 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.659 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.659 INFO:teuthology.orchestra.run.trial185.stdout: "event": "all_read", 2026-02-23T13:11:42.659 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:11:42.564910+0000", 2026-02-23T13:11:42.659 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 4.1347999999999998e-05 2026-02-23T13:11:42.659 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.659 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.660 INFO:teuthology.orchestra.run.trial185.stdout: "event": "dispatched", 2026-02-23T13:11:42.660 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:11:42.564916+0000", 2026-02-23T13:11:42.660 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 6.1210000000000003e-06 2026-02-23T13:11:42.660 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.660 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.660 INFO:teuthology.orchestra.run.trial185.stdout: "event": "queued_for_pg", 2026-02-23T13:11:42.660 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:11:42.564919+0000", 2026-02-23T13:11:42.660 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 2.5849999999999998e-06 2026-02-23T13:11:42.660 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.660 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.660 INFO:teuthology.orchestra.run.trial185.stdout: "event": "reached_pg", 2026-02-23T13:11:42.660 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:11:42.567554+0000", 2026-02-23T13:11:42.660 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 0.0026354820000000002 2026-02-23T13:11:42.660 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.660 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.660 INFO:teuthology.orchestra.run.trial185.stdout: "event": "started", 2026-02-23T13:11:42.660 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:11:42.567595+0000", 2026-02-23T13:11:42.660 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 4.0645999999999997e-05 2026-02-23T13:11:42.660 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.660 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.660 INFO:teuthology.orchestra.run.trial185.stdout: "event": "waiting for subops from 1,3", 2026-02-23T13:11:42.660 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:11:42.567660+0000", 2026-02-23T13:11:42.660 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 6.4782000000000004e-05 2026-02-23T13:11:42.660 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.660 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.660 INFO:teuthology.orchestra.run.trial185.stdout: "event": "op_commit", 2026-02-23T13:11:42.660 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:11:42.568584+0000", 2026-02-23T13:11:42.660 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 0.00092425000000000005 2026-02-23T13:11:42.660 INFO:teuthology.orchestra.run.trial185.stdout: } 2026-02-23T13:11:42.660 INFO:teuthology.orchestra.run.trial185.stdout: ] 2026-02-23T13:11:42.660 INFO:teuthology.orchestra.run.trial185.stdout: } 2026-02-23T13:11:42.660 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.660 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.660 INFO:teuthology.orchestra.run.trial185.stdout: "description": "osd_op(client.14643.0:32568 4.9 4.55b198c9 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e129)", 2026-02-23T13:11:42.660 INFO:teuthology.orchestra.run.trial185.stdout: "initiated_at": "2026-02-23T13:11:42.563109+0000", 2026-02-23T13:11:42.660 INFO:teuthology.orchestra.run.trial185.stdout: "age": 0.005768736, 2026-02-23T13:11:42.660 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 0.0064020309999999999, 2026-02-23T13:11:42.660 INFO:teuthology.orchestra.run.trial185.stdout: "continuous": false, 2026-02-23T13:11:42.660 INFO:teuthology.orchestra.run.trial185.stdout: "type_data": { 2026-02-23T13:11:42.660 INFO:teuthology.orchestra.run.trial185.stdout: "flag_point": "queued for pg", 2026-02-23T13:11:42.661 INFO:teuthology.orchestra.run.trial185.stdout: "client_info": { 2026-02-23T13:11:42.661 INFO:teuthology.orchestra.run.trial185.stdout: "client": "client.14643", 2026-02-23T13:11:42.661 INFO:teuthology.orchestra.run.trial185.stdout: "client_addr": "10.20.193.68:0/3888608980", 2026-02-23T13:11:42.661 INFO:teuthology.orchestra.run.trial185.stdout: "tid": 32568 2026-02-23T13:11:42.661 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.661 INFO:teuthology.orchestra.run.trial185.stdout: "events": [ 2026-02-23T13:11:42.661 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.661 INFO:teuthology.orchestra.run.trial185.stdout: "event": "initiated", 2026-02-23T13:11:42.661 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:11:42.563109+0000", 2026-02-23T13:11:42.661 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 0 2026-02-23T13:11:42.661 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.661 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.661 INFO:teuthology.orchestra.run.trial185.stdout: "event": "header_read", 2026-02-23T13:11:42.661 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:11:42.563109+0000", 2026-02-23T13:11:42.661 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 0 2026-02-23T13:11:42.661 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.661 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.661 INFO:teuthology.orchestra.run.trial185.stdout: "event": "throttled", 2026-02-23T13:11:42.661 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:11:42.563131+0000", 2026-02-23T13:11:42.661 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 2.128e-05 2026-02-23T13:11:42.661 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.661 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.661 INFO:teuthology.orchestra.run.trial185.stdout: "event": "all_read", 2026-02-23T13:11:42.661 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:11:42.563240+0000", 2026-02-23T13:11:42.661 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 0.000109535 2026-02-23T13:11:42.661 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.661 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.661 INFO:teuthology.orchestra.run.trial185.stdout: "event": "dispatched", 2026-02-23T13:11:42.661 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:11:42.563248+0000", 2026-02-23T13:11:42.661 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 7.9740000000000002e-06 2026-02-23T13:11:42.661 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.661 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.661 INFO:teuthology.orchestra.run.trial185.stdout: "event": "queued_for_pg", 2026-02-23T13:11:42.661 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:11:42.563251+0000", 2026-02-23T13:11:42.661 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 2.655e-06 2026-02-23T13:11:42.661 INFO:teuthology.orchestra.run.trial185.stdout: } 2026-02-23T13:11:42.661 INFO:teuthology.orchestra.run.trial185.stdout: ] 2026-02-23T13:11:42.661 INFO:teuthology.orchestra.run.trial185.stdout: } 2026-02-23T13:11:42.661 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.661 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.661 INFO:teuthology.orchestra.run.trial185.stdout: "description": "osd_op(client.14643.0:32592 4.a 4:5eaf47c2:::benchmark_data_trial068_96731_object4073:head [set-alloc-hint object_size 65536 write_size 8192,write 57344~8192 in=8192b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e129)", 2026-02-23T13:11:42.662 INFO:teuthology.orchestra.run.trial185.stdout: "initiated_at": "2026-02-23T13:11:42.564993+0000", 2026-02-23T13:11:42.662 INFO:teuthology.orchestra.run.trial185.stdout: "age": 0.0038848400000000001, 2026-02-23T13:11:42.662 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 0.0045339849999999999, 2026-02-23T13:11:42.662 INFO:teuthology.orchestra.run.trial185.stdout: "continuous": false, 2026-02-23T13:11:42.662 INFO:teuthology.orchestra.run.trial185.stdout: "type_data": { 2026-02-23T13:11:42.662 INFO:teuthology.orchestra.run.trial185.stdout: "flag_point": "waiting for sub ops", 2026-02-23T13:11:42.662 INFO:teuthology.orchestra.run.trial185.stdout: "client_info": { 2026-02-23T13:11:42.662 INFO:teuthology.orchestra.run.trial185.stdout: "client": "client.14643", 2026-02-23T13:11:42.662 INFO:teuthology.orchestra.run.trial185.stdout: "client_addr": "10.20.193.68:0/3888608980", 2026-02-23T13:11:42.662 INFO:teuthology.orchestra.run.trial185.stdout: "tid": 32592 2026-02-23T13:11:42.662 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.662 INFO:teuthology.orchestra.run.trial185.stdout: "events": [ 2026-02-23T13:11:42.662 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.662 INFO:teuthology.orchestra.run.trial185.stdout: "event": "initiated", 2026-02-23T13:11:42.662 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:11:42.564993+0000", 2026-02-23T13:11:42.662 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 0 2026-02-23T13:11:42.662 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.662 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.662 INFO:teuthology.orchestra.run.trial185.stdout: "event": "header_read", 2026-02-23T13:11:42.662 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:11:42.564993+0000", 2026-02-23T13:11:42.662 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 0 2026-02-23T13:11:42.662 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.662 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.662 INFO:teuthology.orchestra.run.trial185.stdout: "event": "throttled", 2026-02-23T13:11:42.662 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:11:42.565004+0000", 2026-02-23T13:11:42.662 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 1.0869999999999999e-05 2026-02-23T13:11:42.662 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.662 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.662 INFO:teuthology.orchestra.run.trial185.stdout: "event": "all_read", 2026-02-23T13:11:42.662 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:11:42.565050+0000", 2026-02-23T13:11:42.662 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 4.6205999999999998e-05 2026-02-23T13:11:42.662 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.662 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.662 INFO:teuthology.orchestra.run.trial185.stdout: "event": "dispatched", 2026-02-23T13:11:42.662 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:11:42.565061+0000", 2026-02-23T13:11:42.662 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 1.027e-05 2026-02-23T13:11:42.662 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.662 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.662 INFO:teuthology.orchestra.run.trial185.stdout: "event": "queued_for_pg", 2026-02-23T13:11:42.662 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:11:42.565063+0000", 2026-02-23T13:11:42.662 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 2.204e-06 2026-02-23T13:11:42.662 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.662 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.663 INFO:teuthology.orchestra.run.trial185.stdout: "event": "reached_pg", 2026-02-23T13:11:42.663 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:11:42.567985+0000", 2026-02-23T13:11:42.663 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 0.0029222390000000001 2026-02-23T13:11:42.663 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.663 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.663 INFO:teuthology.orchestra.run.trial185.stdout: "event": "started", 2026-02-23T13:11:42.663 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:11:42.568033+0000", 2026-02-23T13:11:42.663 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 4.8099999999999997e-05 2026-02-23T13:11:42.663 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.663 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.663 INFO:teuthology.orchestra.run.trial185.stdout: "event": "waiting for subops from 1,3", 2026-02-23T13:11:42.663 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:11:42.568165+0000", 2026-02-23T13:11:42.663 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 0.000132037 2026-02-23T13:11:42.663 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.663 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.663 INFO:teuthology.orchestra.run.trial185.stdout: "event": "op_commit", 2026-02-23T13:11:42.663 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:11:42.568750+0000", 2026-02-23T13:11:42.663 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 0.000584985 2026-02-23T13:11:42.663 INFO:teuthology.orchestra.run.trial185.stdout: } 2026-02-23T13:11:42.663 INFO:teuthology.orchestra.run.trial185.stdout: ] 2026-02-23T13:11:42.663 INFO:teuthology.orchestra.run.trial185.stdout: } 2026-02-23T13:11:42.663 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.663 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.663 INFO:teuthology.orchestra.run.trial185.stdout: "description": "osd_repop_reply(client.14643.0:32537 4.1 e129/128 ondisk, result = 0)", 2026-02-23T13:11:42.663 INFO:teuthology.orchestra.run.trial185.stdout: "initiated_at": "2026-02-23T13:11:42.569072+0000", 2026-02-23T13:11:42.663 INFO:teuthology.orchestra.run.trial185.stdout: "age": 4294967295.9998059, 2026-02-23T13:11:42.663 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 0.00047710299999999999, 2026-02-23T13:11:42.663 INFO:teuthology.orchestra.run.trial185.stdout: "continuous": false, 2026-02-23T13:11:42.663 INFO:teuthology.orchestra.run.trial185.stdout: "type_data": { 2026-02-23T13:11:42.663 INFO:teuthology.orchestra.run.trial185.stdout: "flag_point": "started", 2026-02-23T13:11:42.663 INFO:teuthology.orchestra.run.trial185.stdout: "events": [ 2026-02-23T13:11:42.663 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.663 INFO:teuthology.orchestra.run.trial185.stdout: "event": "initiated", 2026-02-23T13:11:42.663 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:11:42.569072+0000", 2026-02-23T13:11:42.663 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 0 2026-02-23T13:11:42.663 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.663 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.663 INFO:teuthology.orchestra.run.trial185.stdout: "event": "header_read", 2026-02-23T13:11:42.663 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:11:42.569072+0000", 2026-02-23T13:11:42.663 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 0 2026-02-23T13:11:42.663 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.663 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.663 INFO:teuthology.orchestra.run.trial185.stdout: "event": "throttled", 2026-02-23T13:11:42.664 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:11:42.569079+0000", 2026-02-23T13:11:42.664 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 7.2139999999999999e-06 2026-02-23T13:11:42.664 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.664 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.664 INFO:teuthology.orchestra.run.trial185.stdout: "event": "all_read", 2026-02-23T13:11:42.664 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:11:42.569130+0000", 2026-02-23T13:11:42.664 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 5.1226e-05 2026-02-23T13:11:42.664 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.664 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.664 INFO:teuthology.orchestra.run.trial185.stdout: "event": "dispatched", 2026-02-23T13:11:42.664 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:11:42.569135+0000", 2026-02-23T13:11:42.664 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 4.177e-06 2026-02-23T13:11:42.664 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.664 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.664 INFO:teuthology.orchestra.run.trial185.stdout: "event": "queued_for_pg", 2026-02-23T13:11:42.664 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:11:42.569138+0000", 2026-02-23T13:11:42.664 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 3.4170000000000001e-06 2026-02-23T13:11:42.664 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.664 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.664 INFO:teuthology.orchestra.run.trial185.stdout: "event": "reached_pg", 2026-02-23T13:11:42.664 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:11:42.569530+0000", 2026-02-23T13:11:42.664 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 0.00039227400000000002 2026-02-23T13:11:42.664 INFO:teuthology.orchestra.run.trial185.stdout: }, 2026-02-23T13:11:42.664 INFO:teuthology.orchestra.run.trial185.stdout: { 2026-02-23T13:11:42.664 INFO:teuthology.orchestra.run.trial185.stdout: "event": "started", 2026-02-23T13:11:42.664 INFO:teuthology.orchestra.run.trial185.stdout: "time": "2026-02-23T13:11:42.569533+0000", 2026-02-23T13:11:42.664 INFO:teuthology.orchestra.run.trial185.stdout: "duration": 2.2340000000000001e-06 2026-02-23T13:11:42.664 INFO:teuthology.orchestra.run.trial185.stdout: } 2026-02-23T13:11:42.664 INFO:teuthology.orchestra.run.trial185.stdout: ] 2026-02-23T13:11:42.664 INFO:teuthology.orchestra.run.trial185.stdout: } 2026-02-23T13:11:42.664 INFO:teuthology.orchestra.run.trial185.stdout: } 2026-02-23T13:11:42.664 INFO:teuthology.orchestra.run.trial185.stdout: ], 2026-02-23T13:11:42.664 INFO:teuthology.orchestra.run.trial185.stdout: "num_ops": 32 2026-02-23T13:11:42.664 INFO:teuthology.orchestra.run.trial185.stdout:} 2026-02-23T13:11:42.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:11:42 trial068 ceph-mon[38805]: pgmap v233: 17 pgs: 9 active+clean, 8 creating+peering; 66 MiB data, 2.3 GiB used, 2.7 TiB / 2.7 TiB avail; 11 MiB/s wr, 1.43k op/s 2026-02-23T13:11:42.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:11:42 trial068 ceph-mon[38805]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded) 2026-02-23T13:11:42.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:11:42 trial068 ceph-mon[38805]: Cluster is now healthy 2026-02-23T13:11:43.280 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:11:43.444 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:11:43.547 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5 128 42098 41970 65.5729 75.8359 0.0224115 0.0152241 2026-02-23T13:11:43.578 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:11:43.576+0000 7f7cc4577640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:11:43.578 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:11:43.576+0000 7f7cc4577640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:11:43.600 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:11:43.601 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:11:43.617 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:11:43.617 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:11:43.634 INFO:teuthology.orchestra.run.trial068.stdout:osd.2: {} 2026-02-23T13:11:43.635 INFO:teuthology.orchestra.run.trial068.stderr:osd.2: osd_enable_op_tracker = '' 2026-02-23T13:11:43.649 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:11:43.649 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:11:43.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:11:43 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3801833019' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:11:44.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:11:43 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:11:43] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:11:44.357 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:11:43 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:11:43.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:11:44.547 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6 128 51484 51356 66.8645 73.3281 0.0142839 0.0149302 2026-02-23T13:11:45.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:11:44 trial068 ceph-mon[38805]: osdmap e130: 4 total, 4 up, 4 in 2026-02-23T13:11:45.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:11:44 trial068 ceph-mon[38805]: pgmap v235: 17 pgs: 17 active+clean; 187 MiB data, 1.7 GiB used, 2.7 TiB / 2.7 TiB avail; 31 MiB/s wr, 3.97k op/s 2026-02-23T13:11:45.547 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7 127 60090 59963 66.9178 67.2422 0.0383829 0.0149142 2026-02-23T13:11:45.698 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:11:45.855 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:11:45.985 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:11:45.982+0000 7fba92c26640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:11:45.985 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:11:45.983+0000 7fba92c26640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:11:46.005 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:11:46.006 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:11:46.019 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:11:46.019 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:11:46.031 INFO:teuthology.orchestra.run.trial068.stdout:osd.2: {} 2026-02-23T13:11:46.032 INFO:teuthology.orchestra.run.trial068.stderr:osd.2: osd_enable_op_tracker = '' 2026-02-23T13:11:46.044 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:11:46.044 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:11:46.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:11:45 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/4123656699' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:11:46.547 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8 128 67916 67788 66.194 61.1328 0.0463373 0.0150697 2026-02-23T13:11:47.245 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:11:46 trial068 ceph-mon[38805]: pgmap v236: 17 pgs: 17 active+clean; 278 MiB data, 2.1 GiB used, 2.7 TiB / 2.7 TiB avail; 36 MiB/s wr, 4.55k op/s 2026-02-23T13:11:47.548 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9 128 75529 75401 65.4472 59.4766 0.0393083 0.0152571 2026-02-23T13:11:47.666 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [1, 3, 0, 2] 2026-02-23T13:11:47.666 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2026-02-23T13:11:47.666 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph pg dump --format=json 2026-02-23T13:11:47.832 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:11:47.968 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:11:47.966+0000 7f78a9b7f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:11:47.968 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:11:47.966+0000 7f78a9b7f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:11:48.092 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:11:48.127 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T13:11:48.128 INFO:teuthology.orchestra.run.trial068.stderr:dumped all 2026-02-23T13:11:48.182 INFO:teuthology.orchestra.run.trial068.stdout:{"pg_ready":true,"pg_map":{"version":236,"stamp":"2026-02-23T13:11:46.263930+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":291660320,"num_objects":4445,"num_object_clones":0,"num_object_copies":13335,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4445,"num_whiteouts":0,"num_read":122,"num_read_kb":228,"num_write":35666,"num_write_kb":286398,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":35630,"ondisk_log_size":35630,"up":51,"acting":51,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":51,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":2936012800,"kb_used":2160216,"kb_used_data":707080,"kb_used_omap":120,"kb_used_meta":1452935,"kb_avail":2933852584,"statfs":{"total":3006477107200,"available":3004265046016,"internally_reserved":0,"allocated":724049920,"data_stored":720363257,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":123283,"internal_metadata":1487806061},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[7,14,9,9,10],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":291069952,"num_objects":4443,"num_object_clones":0,"num_object_copies":13329,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4443,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":35531,"num_write_kb":284248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.802828"},"pg_stats":[{"pgid":"4.f","version":"129'1967","reported_seq":1977,"reported_epoch":129,"state":"active+clean","last_fresh":"2026-02-23T13:11:42.301282+0000","last_change":"2026-02-23T13:11:39.470208+0000","last_active":"2026-02-23T13:11:42.301282+0000","last_peered":"2026-02-23T13:11:42.301282+0000","last_clean":"2026-02-23T13:11:42.301282+0000","last_became_active":"2026-02-23T13:11:39.469391+0000","last_became_peered":"2026-02-23T13:11:39.469391+0000","last_unstale":"2026-02-23T13:11:42.301282+0000","last_undegraded":"2026-02-23T13:11:42.301282+0000","last_fullsized":"2026-02-23T13:11:42.301282+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:11:36.450546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:11:36.450546+0000","last_clean_scrub_stamp":"2026-02-23T13:11:36.450546+0000","objects_scrubbed":0,"log_size":1967,"log_dups_size":0,"ondisk_log_size":1967,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T13:13:01.564918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16113664,"num_objects":246,"num_object_clones":0,"num_object_copies":738,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":246,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1967,"num_write_kb":15736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"129'2160","reported_seq":2170,"reported_epoch":129,"state":"active+clean","last_fresh":"2026-02-23T13:11:42.465188+0000","last_change":"2026-02-23T13:11:39.470782+0000","last_active":"2026-02-23T13:11:42.465188+0000","last_peered":"2026-02-23T13:11:42.465188+0000","last_clean":"2026-02-23T13:11:42.465188+0000","last_became_active":"2026-02-23T13:11:39.470325+0000","last_became_peered":"2026-02-23T13:11:39.470325+0000","last_unstale":"2026-02-23T13:11:42.465188+0000","last_undegraded":"2026-02-23T13:11:42.465188+0000","last_fullsized":"2026-02-23T13:11:42.465188+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:11:36.450546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:11:36.450546+0000","last_clean_scrub_stamp":"2026-02-23T13:11:36.450546+0000","objects_scrubbed":0,"log_size":2160,"log_dups_size":0,"ondisk_log_size":2160,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T13:12:44.787419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17694720,"num_objects":270,"num_object_clones":0,"num_object_copies":810,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":270,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2160,"num_write_kb":17280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"129'1276","reported_seq":1287,"reported_epoch":129,"state":"active+clean","last_fresh":"2026-02-23T13:11:41.581316+0000","last_change":"2026-02-23T13:11:39.472097+0000","last_active":"2026-02-23T13:11:41.581316+0000","last_peered":"2026-02-23T13:11:41.581316+0000","last_clean":"2026-02-23T13:11:41.581316+0000","last_became_active":"2026-02-23T13:11:39.471785+0000","last_became_peered":"2026-02-23T13:11:39.471785+0000","last_unstale":"2026-02-23T13:11:41.581316+0000","last_undegraded":"2026-02-23T13:11:41.581316+0000","last_fullsized":"2026-02-23T13:11:41.581316+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:11:36.450546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:11:36.450546+0000","last_clean_scrub_stamp":"2026-02-23T13:11:36.450546+0000","objects_scrubbed":0,"log_size":1276,"log_dups_size":0,"ondisk_log_size":1276,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T13:12:51.086185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10452992,"num_objects":160,"num_object_clones":0,"num_object_copies":480,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":160,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1276,"num_write_kb":10208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"129'1440","reported_seq":1458,"reported_epoch":129,"state":"active+clean","last_fresh":"2026-02-23T13:11:41.581637+0000","last_change":"2026-02-23T13:11:39.471967+0000","last_active":"2026-02-23T13:11:41.581637+0000","last_peered":"2026-02-23T13:11:41.581637+0000","last_clean":"2026-02-23T13:11:41.581637+0000","last_became_active":"2026-02-23T13:11:39.471615+0000","last_became_peered":"2026-02-23T13:11:39.471615+0000","last_unstale":"2026-02-23T13:11:41.581637+0000","last_undegraded":"2026-02-23T13:11:41.581637+0000","last_fullsized":"2026-02-23T13:11:41.581637+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:11:36.450546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:11:36.450546+0000","last_clean_scrub_stamp":"2026-02-23T13:11:36.450546+0000","objects_scrubbed":0,"log_size":1440,"log_dups_size":0,"ondisk_log_size":1440,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T13:12:44.823499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11796480,"num_objects":180,"num_object_clones":0,"num_object_copies":540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":180,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1440,"num_write_kb":11520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"130'3516","reported_seq":3536,"reported_epoch":130,"state":"active+clean","last_fresh":"2026-02-23T13:11:45.364109+0000","last_change":"2026-02-23T13:11:39.472140+0000","last_active":"2026-02-23T13:11:45.364109+0000","last_peered":"2026-02-23T13:11:45.364109+0000","last_clean":"2026-02-23T13:11:45.364109+0000","last_became_active":"2026-02-23T13:11:39.471739+0000","last_became_peered":"2026-02-23T13:11:39.471739+0000","last_unstale":"2026-02-23T13:11:45.364109+0000","last_undegraded":"2026-02-23T13:11:45.364109+0000","last_fullsized":"2026-02-23T13:11:45.364109+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:11:36.450546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:11:36.450546+0000","last_clean_scrub_stamp":"2026-02-23T13:11:36.450546+0000","objects_scrubbed":0,"log_size":3516,"log_dups_size":0,"ondisk_log_size":3516,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T13:13:04.763480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28803072,"num_objects":440,"num_object_clones":0,"num_object_copies":1320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":440,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3516,"num_write_kb":28128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"129'1920","reported_seq":1932,"reported_epoch":129,"state":"active+clean","last_fresh":"2026-02-23T13:11:42.454567+0000","last_change":"2026-02-23T13:11:39.471766+0000","last_active":"2026-02-23T13:11:42.454567+0000","last_peered":"2026-02-23T13:11:42.454567+0000","last_clean":"2026-02-23T13:11:42.454567+0000","last_became_active":"2026-02-23T13:11:39.470612+0000","last_became_peered":"2026-02-23T13:11:39.470612+0000","last_unstale":"2026-02-23T13:11:42.454567+0000","last_undegraded":"2026-02-23T13:11:42.454567+0000","last_fullsized":"2026-02-23T13:11:42.454567+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:11:36.450546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:11:36.450546+0000","last_clean_scrub_stamp":"2026-02-23T13:11:36.450546+0000","objects_scrubbed":0,"log_size":1920,"log_dups_size":0,"ondisk_log_size":1920,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T13:12:55.159735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15728640,"num_objects":240,"num_object_clones":0,"num_object_copies":720,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":240,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1920,"num_write_kb":15360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"129'1748","reported_seq":1753,"reported_epoch":129,"state":"active+clean","last_fresh":"2026-02-23T13:11:42.465169+0000","last_change":"2026-02-23T13:11:39.469555+0000","last_active":"2026-02-23T13:11:42.465169+0000","last_peered":"2026-02-23T13:11:42.465169+0000","last_clean":"2026-02-23T13:11:42.465169+0000","last_became_active":"2026-02-23T13:11:39.468085+0000","last_became_peered":"2026-02-23T13:11:39.468085+0000","last_unstale":"2026-02-23T13:11:42.465169+0000","last_undegraded":"2026-02-23T13:11:42.465169+0000","last_fullsized":"2026-02-23T13:11:42.465169+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:11:36.450546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:11:36.450546+0000","last_clean_scrub_stamp":"2026-02-23T13:11:36.450546+0000","objects_scrubbed":0,"log_size":1748,"log_dups_size":0,"ondisk_log_size":1748,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T13:12:49.678369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14319616,"num_objects":219,"num_object_clones":0,"num_object_copies":657,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":219,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1748,"num_write_kb":13984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"129'1424","reported_seq":1436,"reported_epoch":129,"state":"active+clean","last_fresh":"2026-02-23T13:11:41.578693+0000","last_change":"2026-02-23T13:11:39.471636+0000","last_active":"2026-02-23T13:11:41.578693+0000","last_peered":"2026-02-23T13:11:41.578693+0000","last_clean":"2026-02-23T13:11:41.578693+0000","last_became_active":"2026-02-23T13:11:39.471085+0000","last_became_peered":"2026-02-23T13:11:39.471085+0000","last_unstale":"2026-02-23T13:11:41.578693+0000","last_undegraded":"2026-02-23T13:11:41.578693+0000","last_fullsized":"2026-02-23T13:11:41.578693+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:11:36.450546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:11:36.450546+0000","last_clean_scrub_stamp":"2026-02-23T13:11:36.450546+0000","objects_scrubbed":0,"log_size":1424,"log_dups_size":0,"ondisk_log_size":1424,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T13:12:59.331985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11665408,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1424,"num_write_kb":11392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.7","version":"129'2096","reported_seq":2115,"reported_epoch":129,"state":"active+clean","last_fresh":"2026-02-23T13:11:42.296728+0000","last_change":"2026-02-23T13:11:39.470174+0000","last_active":"2026-02-23T13:11:42.296728+0000","last_peered":"2026-02-23T13:11:42.296728+0000","last_clean":"2026-02-23T13:11:42.296728+0000","last_became_active":"2026-02-23T13:11:39.469529+0000","last_became_peered":"2026-02-23T13:11:39.469529+0000","last_unstale":"2026-02-23T13:11:42.296728+0000","last_undegraded":"2026-02-23T13:11:42.296728+0000","last_fullsized":"2026-02-23T13:11:42.296728+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:11:36.450546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:11:36.450546+0000","last_clean_scrub_stamp":"2026-02-23T13:11:36.450546+0000","objects_scrubbed":0,"log_size":2096,"log_dups_size":0,"ondisk_log_size":2096,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T13:12:54.680850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17170432,"num_objects":262,"num_object_clones":0,"num_object_copies":786,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":262,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2096,"num_write_kb":16768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"130'3616","reported_seq":3637,"reported_epoch":130,"state":"active+clean","last_fresh":"2026-02-23T13:11:45.357037+0000","last_change":"2026-02-23T13:11:39.472075+0000","last_active":"2026-02-23T13:11:45.357037+0000","last_peered":"2026-02-23T13:11:45.357037+0000","last_clean":"2026-02-23T13:11:45.357037+0000","last_became_active":"2026-02-23T13:11:39.471827+0000","last_became_peered":"2026-02-23T13:11:39.471827+0000","last_unstale":"2026-02-23T13:11:45.357037+0000","last_undegraded":"2026-02-23T13:11:45.357037+0000","last_fullsized":"2026-02-23T13:11:45.357037+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:11:36.450546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:11:36.450546+0000","last_clean_scrub_stamp":"2026-02-23T13:11:36.450546+0000","objects_scrubbed":0,"log_size":3616,"log_dups_size":0,"ondisk_log_size":3616,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T13:12:49.133699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29622272,"num_objects":452,"num_object_clones":0,"num_object_copies":1356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":452,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3616,"num_write_kb":28928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"129'1768","reported_seq":1784,"reported_epoch":129,"state":"active+clean","last_fresh":"2026-02-23T13:11:42.299285+0000","last_change":"2026-02-23T13:11:37.460679+0000","last_active":"2026-02-23T13:11:42.299285+0000","last_peered":"2026-02-23T13:11:42.299285+0000","last_clean":"2026-02-23T13:11:42.299285+0000","last_became_active":"2026-02-23T13:11:37.460441+0000","last_became_peered":"2026-02-23T13:11:37.460441+0000","last_unstale":"2026-02-23T13:11:42.299285+0000","last_undegraded":"2026-02-23T13:11:42.299285+0000","last_fullsized":"2026-02-23T13:11:42.299285+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:11:36.450546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:11:36.450546+0000","last_clean_scrub_stamp":"2026-02-23T13:11:36.450546+0000","objects_scrubbed":0,"log_size":1768,"log_dups_size":0,"ondisk_log_size":1768,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T13:12:41.196043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14483456,"num_objects":221,"num_object_clones":0,"num_object_copies":663,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":221,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1768,"num_write_kb":14144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"130'3584","reported_seq":3600,"reported_epoch":130,"state":"active+clean","last_fresh":"2026-02-23T13:11:45.360521+0000","last_change":"2026-02-23T13:11:39.472121+0000","last_active":"2026-02-23T13:11:45.360521+0000","last_peered":"2026-02-23T13:11:45.360521+0000","last_clean":"2026-02-23T13:11:45.360521+0000","last_became_active":"2026-02-23T13:11:39.471723+0000","last_became_peered":"2026-02-23T13:11:39.471723+0000","last_unstale":"2026-02-23T13:11:45.360521+0000","last_undegraded":"2026-02-23T13:11:45.360521+0000","last_fullsized":"2026-02-23T13:11:45.360521+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:11:36.450546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:11:36.450546+0000","last_clean_scrub_stamp":"2026-02-23T13:11:36.450546+0000","objects_scrubbed":0,"log_size":3584,"log_dups_size":0,"ondisk_log_size":3584,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T13:12:55.202666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29360128,"num_objects":448,"num_object_clones":0,"num_object_copies":1344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":448,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3584,"num_write_kb":28672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"129'1952","reported_seq":1966,"reported_epoch":129,"state":"active+clean","last_fresh":"2026-02-23T13:11:42.300738+0000","last_change":"2026-02-23T13:11:37.460191+0000","last_active":"2026-02-23T13:11:42.300738+0000","last_peered":"2026-02-23T13:11:42.300738+0000","last_clean":"2026-02-23T13:11:42.300738+0000","last_became_active":"2026-02-23T13:11:37.459917+0000","last_became_peered":"2026-02-23T13:11:37.459917+0000","last_unstale":"2026-02-23T13:11:42.300738+0000","last_undegraded":"2026-02-23T13:11:42.300738+0000","last_fullsized":"2026-02-23T13:11:42.300738+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:11:36.450546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:11:36.450546+0000","last_clean_scrub_stamp":"2026-02-23T13:11:36.450546+0000","objects_scrubbed":0,"log_size":1952,"log_dups_size":0,"ondisk_log_size":1952,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T13:12:42.231079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15990784,"num_objects":244,"num_object_clones":0,"num_object_copies":732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":244,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1952,"num_write_kb":15616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1","version":"129'1896","reported_seq":1908,"reported_epoch":129,"state":"active+clean","last_fresh":"2026-02-23T13:11:42.462378+0000","last_change":"2026-02-23T13:11:39.472209+0000","last_active":"2026-02-23T13:11:42.462378+0000","last_peered":"2026-02-23T13:11:42.462378+0000","last_clean":"2026-02-23T13:11:42.462378+0000","last_became_active":"2026-02-23T13:11:39.471455+0000","last_became_peered":"2026-02-23T13:11:39.471455+0000","last_unstale":"2026-02-23T13:11:42.462378+0000","last_undegraded":"2026-02-23T13:11:42.462378+0000","last_fullsized":"2026-02-23T13:11:42.462378+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:11:36.450546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:11:36.450546+0000","last_clean_scrub_stamp":"2026-02-23T13:11:36.450546+0000","objects_scrubbed":0,"log_size":1896,"log_dups_size":0,"ondisk_log_size":1896,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T13:12:56.259027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15532032,"num_objects":237,"num_object_clones":0,"num_object_copies":711,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":237,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1896,"num_write_kb":15168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.0","version":"130'3856","reported_seq":3874,"reported_epoch":130,"state":"active+clean","last_fresh":"2026-02-23T13:11:45.359224+0000","last_change":"2026-02-23T13:11:39.468708+0000","last_active":"2026-02-23T13:11:45.359224+0000","last_peered":"2026-02-23T13:11:45.359224+0000","last_clean":"2026-02-23T13:11:45.359224+0000","last_became_active":"2026-02-23T13:11:39.468560+0000","last_became_peered":"2026-02-23T13:11:39.468560+0000","last_unstale":"2026-02-23T13:11:45.359224+0000","last_undegraded":"2026-02-23T13:11:45.359224+0000","last_fullsized":"2026-02-23T13:11:45.359224+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:11:36.450546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:11:36.450546+0000","last_clean_scrub_stamp":"2026-02-23T13:11:36.450546+0000","objects_scrubbed":0,"log_size":3856,"log_dups_size":0,"ondisk_log_size":3856,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T13:12:54.241416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31588352,"num_objects":482,"num_object_clones":0,"num_object_copies":1446,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3856,"num_write_kb":30848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,3],"acting":[0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.5","version":"129'1312","reported_seq":1331,"reported_epoch":129,"state":"active+clean","last_fresh":"2026-02-23T13:11:41.572330+0000","last_change":"2026-02-23T13:11:39.472161+0000","last_active":"2026-02-23T13:11:41.572330+0000","last_peered":"2026-02-23T13:11:41.572330+0000","last_clean":"2026-02-23T13:11:41.572330+0000","last_became_active":"2026-02-23T13:11:39.471840+0000","last_became_peered":"2026-02-23T13:11:39.471840+0000","last_unstale":"2026-02-23T13:11:41.572330+0000","last_undegraded":"2026-02-23T13:11:41.572330+0000","last_fullsized":"2026-02-23T13:11:41.572330+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:11:36.450546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:11:36.450546+0000","last_clean_scrub_stamp":"2026-02-23T13:11:36.450546+0000","objects_scrubbed":0,"log_size":1312,"log_dups_size":0,"ondisk_log_size":1312,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T13:12:47.743969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10747904,"num_objects":164,"num_object_clones":0,"num_object_copies":492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":164,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1312,"num_write_kb":10496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"28'99","reported_seq":256,"reported_epoch":129,"state":"active+clean","last_fresh":"2026-02-23T13:11:39.969840+0000","last_change":"2026-02-23T13:11:39.969840+0000","last_active":"2026-02-23T13:11:39.969840+0000","last_peered":"2026-02-23T13:11:39.969840+0000","last_clean":"2026-02-23T13:11:39.969840+0000","last_became_active":"2026-02-23T13:11:39.471449+0000","last_became_peered":"2026-02-23T13:11:39.471449+0000","last_unstale":"2026-02-23T13:11:39.969840+0000","last_undegraded":"2026-02-23T13:11:39.969840+0000","last_fullsized":"2026-02-23T13:11:39.969840+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'99","last_scrub_stamp":"2026-02-23T13:11:39.969814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:07:05.619208+0000","last_clean_scrub_stamp":"2026-02-23T13:11:39.969814+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T13:12:57.741599+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":122,"num_read_kb":228,"num_write":135,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":16,"stat_sum":{"num_bytes":291069952,"num_objects":4443,"num_object_clones":0,"num_object_copies":13329,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4443,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":35531,"num_write_kb":284248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":875298816,"data_stored":875298816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":35531,"ondisk_log_size":35531,"up":48,"acting":48,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":122,"num_read_kb":228,"num_write":135,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":80,"seq":343597383703,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":734003200,"kb_used":564828,"kb_used_data":98632,"kb_used_omap":41,"kb_used_meta":466134,"kb_avail":733438372,"statfs":{"total":751619276800,"available":751040892928,"internally_reserved":0,"allocated":100999168,"data_stored":100078586,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":42255,"internal_metadata":477321969},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":128,"seq":549755813890,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":734003200,"kb_used":278636,"kb_used_data":173144,"kb_used_omap":7,"kb_used_meta":105464,"kb_avail":733724564,"statfs":{"total":751619276800,"available":751333953536,"internally_reserved":0,"allocated":177299456,"data_stored":176378874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8003,"internal_metadata":107995325},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":64,"seq":274877906974,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":734003200,"kb_used":746360,"kb_used_data":119332,"kb_used_omap":49,"kb_used_meta":626958,"kb_avail":733256840,"statfs":{"total":751619276800,"available":750855004160,"internally_reserved":0,"allocated":122195968,"data_stored":121278938,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":51025,"internal_metadata":642005167},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[7,14,9,9,10],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":106,"seq":455266533386,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":734003200,"kb_used":570392,"kb_used_data":315972,"kb_used_omap":21,"kb_used_meta":254378,"kb_avail":733432808,"statfs":{"total":751619276800,"available":751035195392,"internally_reserved":0,"allocated":323555328,"data_stored":322626859,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":22000,"internal_metadata":260483600},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":357031936,"data_stored":357031936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":153477120,"data_stored":153477120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":226131968,"data_stored":226131968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":138657792,"data_stored":138657792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-23T13:11:48.247 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:11:48.377 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:11:48.375+0000 7fc13bb7a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:11:48.377 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:11:48.375+0000 7fc13bb7a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:11:48.395 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:11:48.395 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:11:48.409 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:11:48.409 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:11:48.425 INFO:teuthology.orchestra.run.trial068.stdout:osd.2: {} 2026-02-23T13:11:48.425 INFO:teuthology.orchestra.run.trial068.stderr:osd.2: osd_enable_op_tracker = '' 2026-02-23T13:11:48.437 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:11:48.437 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:11:48.548 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 10 128 82309 82181 64.1989 52.9688 0.0374366 0.0155502 2026-02-23T13:11:49.548 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 11 128 88940 88812 63.0718 51.8047 0.016234 0.0158293 2026-02-23T13:11:49.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:11:49 trial068 ceph-mon[38805]: from='client.14653 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2026-02-23T13:11:49.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:11:49 trial068 ceph-mon[38805]: pgmap v237: 17 pgs: 17 active+clean; 537 MiB data, 3.0 GiB used, 2.7 TiB / 2.7 TiB avail; 61 MiB/s wr, 7.81k op/s 2026-02-23T13:11:49.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:11:49 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3943561619' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:11:50.488 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:11:50.548 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 12 128 95314 95186 61.9653 49.7969 0.0179623 0.0161187 2026-02-23T13:11:50.644 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:11:50.767 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:11:50.764+0000 7f0ce78ac640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:11:50.767 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:11:50.765+0000 7f0ce78ac640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:11:50.782 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:11:50.783 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:11:50.796 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:11:50.796 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:11:50.812 INFO:teuthology.orchestra.run.trial068.stdout:osd.2: {} 2026-02-23T13:11:50.813 INFO:teuthology.orchestra.run.trial068.stderr:osd.2: osd_enable_op_tracker = '' 2026-02-23T13:11:50.825 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:11:50.826 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:11:51.548 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 13 127 100880 100753 60.544 43.4922 0.0351461 0.0164909 2026-02-23T13:11:51.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:11:51 trial068 ceph-mon[38805]: pgmap v238: 17 pgs: 17 active+clean; 537 MiB data, 3.0 GiB used, 2.7 TiB / 2.7 TiB avail; 54 MiB/s wr, 6.87k op/s 2026-02-23T13:11:51.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:11:51 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/4119233694' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:11:52.548 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 14 128 106734 106606 59.4854 45.7266 0.0122264 0.0167936 2026-02-23T13:11:52.869 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:11:53.021 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:11:53.154 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:11:53.152+0000 7f508b7a5640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:11:53.154 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:11:53.152+0000 7f508b7a5640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:11:53.171 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:11:53.172 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:11:53.184 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [1, 3, 0, 2] 2026-02-23T13:11:53.184 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2026-02-23T13:11:53.184 INFO:tasks.thrashosds.thrasher:Killing osd 2, live_osds are [1, 3, 0, 2] 2026-02-23T13:11:53.184 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2026-02-23T13:11:53.184 DEBUG:teuthology.orchestra.run.trial185:> sudo systemctl stop ceph-5f823084-10b8-11f1-be80-d404e6e7d460@osd.2 2026-02-23T13:11:53.187 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:11:53.187 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:11:53.202 INFO:teuthology.orchestra.run.trial068.stdout:osd.2: {} 2026-02-23T13:11:53.203 INFO:teuthology.orchestra.run.trial068.stderr:osd.2: osd_enable_op_tracker = '' 2026-02-23T13:11:53.218 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:11:53.218 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:11:53.464 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:11:53 trial185 systemd[1]: Stopping Ceph osd.2 for 5f823084-10b8-11f1-be80-d404e6e7d460... 2026-02-23T13:11:53.464 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:11:53 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-2[72439]: 2026-02-23T13:11:53.308+0000 7fe907ba1640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --osd-objectstore=bluestore (PID: 1) UID: 0 2026-02-23T13:11:53.464 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:11:53 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-2[72439]: 2026-02-23T13:11:53.308+0000 7fe907ba1640 -1 osd.2 130 *** Got signal Terminated *** 2026-02-23T13:11:53.464 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:11:53 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-2[72439]: 2026-02-23T13:11:53.308+0000 7fe907ba1640 -1 osd.2 130 *** Immediate shutdown (osd_fast_shutdown=true) *** 2026-02-23T13:11:53.464 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:11:53 trial185 podman[73255]: 2026-02-23 13:11:53.430375425 +0000 UTC m=+0.138460433 container died 4fccd31daa59408fdd28a1321c8ffdabdb54aa6380b72a4cfa1e27c74bd04cbb (image=quay.ceph.io/ceph-ci/ceph@sha256:49621faaa845c39b2aae72e4fa24123ee9b2ac3fb8ae834753907068ee2972ea, name=ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-2, org.label-schema.vendor=CentOS, org.opencontainers.image.documentation=https://docs.ceph.com/, org.label-schema.license=GPLv2, io.buildah.version=1.41.3, org.label-schema.build-date=20260216, CEPH_REF=wip-shraddhaag-enable-cephadm-tests, CEPH_SHA1=e371433fcd1390a0b82cfb18effc422c1f5d8c1d, ceph=True, org.opencontainers.image.authors=Ceph Release Team , OSD_FLAVOR=default, CEPH_GIT_REPO=https://github.com/ceph/ceph-ci.git, GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfsganesha-5/, org.label-schema.name=CentOS Stream 9 Base Image, FROM_IMAGE=quay.io/centos/centos:stream9, org.label-schema.schema-version=1.0) 2026-02-23T13:11:53.464 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:11:53 trial185 podman[73255]: 2026-02-23 13:11:53.439207595 +0000 UTC m=+0.147292593 container remove 4fccd31daa59408fdd28a1321c8ffdabdb54aa6380b72a4cfa1e27c74bd04cbb (image=quay.ceph.io/ceph-ci/ceph@sha256:49621faaa845c39b2aae72e4fa24123ee9b2ac3fb8ae834753907068ee2972ea, name=ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-2, FROM_IMAGE=quay.io/centos/centos:stream9, GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfsganesha-5/, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_GIT_REPO=https://github.com/ceph/ceph-ci.git, CEPH_SHA1=e371433fcd1390a0b82cfb18effc422c1f5d8c1d, OSD_FLAVOR=default, org.label-schema.vendor=CentOS, ceph=True, CEPH_REF=wip-shraddhaag-enable-cephadm-tests, org.label-schema.license=GPLv2, io.buildah.version=1.41.3, org.opencontainers.image.authors=Ceph Release Team , org.opencontainers.image.documentation=https://docs.ceph.com/, org.label-schema.build-date=20260216, org.label-schema.schema-version=1.0) 2026-02-23T13:11:53.464 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:11:53 trial185 bash[73255]: ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-2 2026-02-23T13:11:53.548 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 15 128 110948 110820 57.7143 32.9219 0.0198544 0.0170422 2026-02-23T13:11:53.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:11:53 trial068 ceph-mon[38805]: pgmap v239: 17 pgs: 17 active+clean; 670 MiB data, 3.5 GiB used, 2.7 TiB / 2.7 TiB avail; 60 MiB/s wr, 7.74k op/s 2026-02-23T13:11:53.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:11:53 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1058186582' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:11:53.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:11:53 trial068 ceph-mon[38805]: osd.2 marked itself down and dead 2026-02-23T13:11:53.767 DEBUG:teuthology.orchestra.run.trial185:> sudo pkill -f 'journalctl -f -n 0 -u ceph-5f823084-10b8-11f1-be80-d404e6e7d460@osd.2.service' 2026-02-23T13:11:53.802 DEBUG:teuthology.orchestra.run:got remote process result: None 2026-02-23T13:11:53.802 INFO:tasks.cephadm.osd.2:Stopped osd.2 2026-02-23T13:11:54.318 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:11:53 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:11:53] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:11:54.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:11:53 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:11:53.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:11:54.548 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 16 127 114315 114188 55.7515 26.3125 0.00992612 0.0179108 2026-02-23T13:11:54.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:11:54 trial068 ceph-mon[38805]: Health check failed: 1 osds down (OSD_DOWN) 2026-02-23T13:11:54.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:11:54 trial068 ceph-mon[38805]: osdmap e131: 4 total, 3 up, 4 in 2026-02-23T13:11:54.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:11:54 trial068 ceph-mon[38805]: osdmap e132: 4 total, 3 up, 4 in 2026-02-23T13:11:55.264 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:11:55.422 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:11:55.548 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 17 128 119870 119742 55.0242 43.3906 0.00832753 0.0181232 2026-02-23T13:11:55.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:11:55 trial068 ceph-mon[38805]: pgmap v242: 17 pgs: 4 stale+active+clean, 13 active+clean; 794 MiB data, 4.0 GiB used, 2.7 TiB / 2.7 TiB avail; 64 MiB/s wr, 8.25k op/s 2026-02-23T13:11:55.832 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:11:55.807+0000 7fc7a0400640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:11:55.832 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:11:55.807+0000 7fc7a0400640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:11:55.832 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:11:55.832 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:11:55.838 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:11:55.839 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:11:55.839 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:11:55.855 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:11:55.855 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:11:56.548 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 18 127 124097 123970 53.8023 33.0312 0.0244833 0.0185722 2026-02-23T13:11:56.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:11:56 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3748248719' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:11:56.727 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:11:56.727 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:11:57.548 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 19 128 129559 129431 53.2159 42.6641 0.0119802 0.0187769 2026-02-23T13:11:57.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:11:57 trial068 ceph-mon[38805]: pgmap v243: 17 pgs: 4 stale+active+clean, 4 active+undersized+degraded, 9 active+clean; 844 MiB data, 4.2 GiB used, 2.7 TiB / 2.7 TiB avail; 38 MiB/s wr, 4.91k op/s; 3727/40509 objects degraded (9.200%) 2026-02-23T13:11:57.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:11:57 trial068 ceph-mon[38805]: Health check failed: Degraded data redundancy: 3727/40509 objects degraded (9.200%), 4 pgs degraded (PG_DEGRADED) 2026-02-23T13:11:58.548 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T13:11:58.548457+0000 min lat: 0.0010091 max lat: 0.941385 avg lat: 0.0190146 2026-02-23T13:11:58.549 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T13:11:58.549 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 20 128 134641 134513 52.5401 39.7031 0.0387745 0.0190146 2026-02-23T13:11:58.728 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:11:58.803 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [2] live_osds: [1, 3, 0] 2026-02-23T13:11:58.803 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2026-02-23T13:11:58.803 INFO:tasks.thrashosds.thrasher:Reviving osd 2 2026-02-23T13:11:58.803 INFO:tasks.cephadm.osd.2:Restarting osd.2 (starting--it wasn't running)... 2026-02-23T13:11:58.803 DEBUG:teuthology.orchestra.run.trial185:osd.2> sudo journalctl -f -n 0 -u ceph-5f823084-10b8-11f1-be80-d404e6e7d460@osd.2.service 2026-02-23T13:11:58.806 DEBUG:teuthology.orchestra.run.trial185:> sudo systemctl start ceph-5f823084-10b8-11f1-be80-d404e6e7d460@osd.2 2026-02-23T13:11:58.885 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:11:59.075 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:11:58 trial185 systemd[1]: Starting Ceph osd.2 for 5f823084-10b8-11f1-be80-d404e6e7d460... 2026-02-23T13:11:59.075 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:11:58 trial185 podman[73567]: 2026-02-23 13:11:58.981688833 +0000 UTC m=+0.017825383 container create 36349b362754f1932a2cee909345ba13081e7c1f5ab79eb486016ef78244e52f (image=quay.ceph.io/ceph-ci/ceph@sha256:49621faaa845c39b2aae72e4fa24123ee9b2ac3fb8ae834753907068ee2972ea, name=ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-2-activate, org.label-schema.vendor=CentOS, CEPH_GIT_REPO=https://github.com/ceph/ceph-ci.git, OSD_FLAVOR=default, CEPH_SHA1=e371433fcd1390a0b82cfb18effc422c1f5d8c1d, org.label-schema.schema-version=1.0, org.label-schema.build-date=20260216, org.opencontainers.image.authors=Ceph Release Team , CEPH_REF=wip-shraddhaag-enable-cephadm-tests, io.buildah.version=1.41.3, org.label-schema.license=GPLv2, ceph=True, org.opencontainers.image.documentation=https://docs.ceph.com/, org.label-schema.name=CentOS Stream 9 Base Image, GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfsganesha-5/, FROM_IMAGE=quay.io/centos/centos:stream9) 2026-02-23T13:11:59.075 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:11:59 trial185 podman[73567]: 2026-02-23 13:11:59.017922057 +0000 UTC m=+0.054058607 container init 36349b362754f1932a2cee909345ba13081e7c1f5ab79eb486016ef78244e52f (image=quay.ceph.io/ceph-ci/ceph@sha256:49621faaa845c39b2aae72e4fa24123ee9b2ac3fb8ae834753907068ee2972ea, name=ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-2-activate, FROM_IMAGE=quay.io/centos/centos:stream9, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, OSD_FLAVOR=default, org.label-schema.build-date=20260216, GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfsganesha-5/, CEPH_GIT_REPO=https://github.com/ceph/ceph-ci.git, io.buildah.version=1.41.3, org.label-schema.license=GPLv2, CEPH_SHA1=e371433fcd1390a0b82cfb18effc422c1f5d8c1d, org.opencontainers.image.authors=Ceph Release Team , org.opencontainers.image.documentation=https://docs.ceph.com/, CEPH_REF=wip-shraddhaag-enable-cephadm-tests, org.label-schema.vendor=CentOS) 2026-02-23T13:11:59.076 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:11:59 trial185 podman[73567]: 2026-02-23 13:11:59.020477399 +0000 UTC m=+0.056613949 container start 36349b362754f1932a2cee909345ba13081e7c1f5ab79eb486016ef78244e52f (image=quay.ceph.io/ceph-ci/ceph@sha256:49621faaa845c39b2aae72e4fa24123ee9b2ac3fb8ae834753907068ee2972ea, name=ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-2-activate, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20260216, org.label-schema.vendor=CentOS, GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfsganesha-5/, FROM_IMAGE=quay.io/centos/centos:stream9, CEPH_SHA1=e371433fcd1390a0b82cfb18effc422c1f5d8c1d, org.opencontainers.image.documentation=https://docs.ceph.com/, CEPH_REF=wip-shraddhaag-enable-cephadm-tests, OSD_FLAVOR=default, io.buildah.version=1.41.3, org.label-schema.license=GPLv2, org.opencontainers.image.authors=Ceph Release Team , org.label-schema.schema-version=1.0, CEPH_GIT_REPO=https://github.com/ceph/ceph-ci.git) 2026-02-23T13:11:59.076 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:11:59 trial185 podman[73567]: 2026-02-23 13:11:59.020976553 +0000 UTC m=+0.057113103 container attach 36349b362754f1932a2cee909345ba13081e7c1f5ab79eb486016ef78244e52f (image=quay.ceph.io/ceph-ci/ceph@sha256:49621faaa845c39b2aae72e4fa24123ee9b2ac3fb8ae834753907068ee2972ea, name=ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-2-activate, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20260216, CEPH_REF=wip-shraddhaag-enable-cephadm-tests, io.buildah.version=1.41.3, FROM_IMAGE=quay.io/centos/centos:stream9, org.opencontainers.image.documentation=https://docs.ceph.com/, GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfsganesha-5/, CEPH_GIT_REPO=https://github.com/ceph/ceph-ci.git, OSD_FLAVOR=default, CEPH_SHA1=e371433fcd1390a0b82cfb18effc422c1f5d8c1d, org.opencontainers.image.authors=Ceph Release Team , ceph=True) 2026-02-23T13:11:59.076 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:11:59 trial185 podman[73567]: 2026-02-23 13:11:58.974812543 +0000 UTC m=+0.010949093 image pull 63df4b83db6f95a8ba965919b3671cb27173117c23b5c950aaadc8913f56213c quay.ceph.io/ceph-ci/ceph@sha256:49621faaa845c39b2aae72e4fa24123ee9b2ac3fb8ae834753907068ee2972ea 2026-02-23T13:11:59.276 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:11:59.274+0000 7faab91b5640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:11:59.277 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:11:59.275+0000 7faab91b5640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:11:59.291 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:11:59.291 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:11:59.309 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:11:59.309 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:11:59.309 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:11:59.323 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:11:59.324 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:11:59.357 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:11:59 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-2-activate[73587]: Running command: /usr/bin/ceph-authtool --gen-print-key 2026-02-23T13:11:59.357 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:11:59 trial185 bash[73567]: Running command: /usr/bin/ceph-authtool --gen-print-key 2026-02-23T13:11:59.357 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:11:59 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-2-activate[73587]: Running command: /usr/bin/ceph-authtool --gen-print-key 2026-02-23T13:11:59.357 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:11:59 trial185 bash[73567]: Running command: /usr/bin/ceph-authtool --gen-print-key 2026-02-23T13:11:59.370 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:11:59.371 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:11:59.548 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 21 128 139446 139318 51.8256 37.5391 0.0222137 0.0192816 2026-02-23T13:11:59.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:11:59 trial068 ceph-mon[38805]: pgmap v244: 17 pgs: 15 active+undersized+degraded, 2 active+clean; 971 MiB data, 4.5 GiB used, 2.7 TiB / 2.7 TiB avail; 54 MiB/s wr, 6.94k op/s; 13550/46590 objects degraded (29.083%); 2.0 KiB/s, 0 objects/s recovering 2026-02-23T13:11:59.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:11:59 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/610192663' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:11:59.709 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:11:59 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-2-activate[73587]: --> Failed to activate via raw: did not find any matching OSD to activate 2026-02-23T13:11:59.709 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:11:59 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-2-activate[73587]: Running command: /usr/bin/ceph-authtool --gen-print-key 2026-02-23T13:11:59.709 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:11:59 trial185 bash[73567]: --> Failed to activate via raw: did not find any matching OSD to activate 2026-02-23T13:11:59.709 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:11:59 trial185 bash[73567]: Running command: /usr/bin/ceph-authtool --gen-print-key 2026-02-23T13:11:59.709 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:11:59 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-2-activate[73587]: Running command: /usr/bin/ceph-authtool --gen-print-key 2026-02-23T13:11:59.709 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:11:59 trial185 bash[73567]: Running command: /usr/bin/ceph-authtool --gen-print-key 2026-02-23T13:11:59.709 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:11:59 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-2-activate[73587]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2026-02-23T13:11:59.709 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:11:59 trial185 bash[73567]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2026-02-23T13:11:59.709 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:11:59 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-2-activate[73587]: Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/vg_nvme/lv_4 --path /var/lib/ceph/osd/ceph-2 --no-mon-config 2026-02-23T13:11:59.709 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:11:59 trial185 bash[73567]: Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/vg_nvme/lv_4 --path /var/lib/ceph/osd/ceph-2 --no-mon-config 2026-02-23T13:11:59.709 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:11:59 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-2-activate[73587]: Running command: /usr/bin/ln -snf /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-2/block 2026-02-23T13:11:59.709 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:11:59 trial185 bash[73567]: Running command: /usr/bin/ln -snf /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-2/block 2026-02-23T13:11:59.709 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:11:59 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-2-activate[73587]: Running command: /usr/bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-2/block 2026-02-23T13:11:59.709 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:11:59 trial185 bash[73567]: Running command: /usr/bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-2/block 2026-02-23T13:11:59.709 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:11:59 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-2-activate[73587]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2026-02-23T13:11:59.709 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:11:59 trial185 bash[73567]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2026-02-23T13:11:59.709 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:11:59 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-2-activate[73587]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2026-02-23T13:11:59.710 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:11:59 trial185 bash[73567]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2026-02-23T13:11:59.710 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:11:59 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-2-activate[73587]: --> ceph-volume lvm activate successful for osd ID: 2 2026-02-23T13:11:59.710 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:11:59 trial185 bash[73567]: --> ceph-volume lvm activate successful for osd ID: 2 2026-02-23T13:11:59.710 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:11:59 trial185 podman[73567]: 2026-02-23 13:11:59.609251386 +0000 UTC m=+0.645387926 container died 36349b362754f1932a2cee909345ba13081e7c1f5ab79eb486016ef78244e52f (image=quay.ceph.io/ceph-ci/ceph@sha256:49621faaa845c39b2aae72e4fa24123ee9b2ac3fb8ae834753907068ee2972ea, name=ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-2-activate, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, FROM_IMAGE=quay.io/centos/centos:stream9, ceph=True, CEPH_REF=wip-shraddhaag-enable-cephadm-tests, org.label-schema.vendor=CentOS, GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfsganesha-5/, org.opencontainers.image.authors=Ceph Release Team , org.label-schema.build-date=20260216, CEPH_GIT_REPO=https://github.com/ceph/ceph-ci.git, io.buildah.version=1.41.3, CEPH_SHA1=e371433fcd1390a0b82cfb18effc422c1f5d8c1d, org.opencontainers.image.documentation=https://docs.ceph.com/, OSD_FLAVOR=default, org.label-schema.license=GPLv2) 2026-02-23T13:11:59.710 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:11:59 trial185 podman[73567]: 2026-02-23 13:11:59.621636375 +0000 UTC m=+0.657772915 container remove 36349b362754f1932a2cee909345ba13081e7c1f5ab79eb486016ef78244e52f (image=quay.ceph.io/ceph-ci/ceph@sha256:49621faaa845c39b2aae72e4fa24123ee9b2ac3fb8ae834753907068ee2972ea, name=ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-2-activate, FROM_IMAGE=quay.io/centos/centos:stream9, org.label-schema.schema-version=1.0, org.opencontainers.image.documentation=https://docs.ceph.com/, ceph=True, org.label-schema.vendor=CentOS, org.opencontainers.image.authors=Ceph Release Team , CEPH_SHA1=e371433fcd1390a0b82cfb18effc422c1f5d8c1d, OSD_FLAVOR=default, org.label-schema.license=GPLv2, org.label-schema.build-date=20260216, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_REF=wip-shraddhaag-enable-cephadm-tests, CEPH_GIT_REPO=https://github.com/ceph/ceph-ci.git, io.buildah.version=1.41.3, GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfsganesha-5/) 2026-02-23T13:11:59.710 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:11:59 trial185 podman[73801]: 2026-02-23 13:11:59.707155292 +0000 UTC m=+0.017938354 container create 8c29a9eb095cb1bd71e78d15ec0ff9260c98404be09bd1d6d932d1e25985d352 (image=quay.ceph.io/ceph-ci/ceph@sha256:49621faaa845c39b2aae72e4fa24123ee9b2ac3fb8ae834753907068ee2972ea, name=ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-2, org.label-schema.build-date=20260216, CEPH_REF=wip-shraddhaag-enable-cephadm-tests, OSD_FLAVOR=default, ceph=True, GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfsganesha-5/, org.opencontainers.image.authors=Ceph Release Team , org.label-schema.vendor=CentOS, FROM_IMAGE=quay.io/centos/centos:stream9, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_SHA1=e371433fcd1390a0b82cfb18effc422c1f5d8c1d, CEPH_GIT_REPO=https://github.com/ceph/ceph-ci.git, org.label-schema.schema-version=1.0, io.buildah.version=1.41.3, org.opencontainers.image.documentation=https://docs.ceph.com/) 2026-02-23T13:11:59.755 DEBUG:teuthology.orchestra.run.trial185:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph daemon osd.2 dump_ops_in_flight 2026-02-23T13:11:59.884 INFO:teuthology.orchestra.run.trial185.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/config/ceph.conf 2026-02-23T13:11:59.959 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:11:59 trial185 podman[73801]: 2026-02-23 13:11:59.743482321 +0000 UTC m=+0.054265323 container init 8c29a9eb095cb1bd71e78d15ec0ff9260c98404be09bd1d6d932d1e25985d352 (image=quay.ceph.io/ceph-ci/ceph@sha256:49621faaa845c39b2aae72e4fa24123ee9b2ac3fb8ae834753907068ee2972ea, name=ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-2, org.opencontainers.image.documentation=https://docs.ceph.com/, GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfsganesha-5/, CEPH_GIT_REPO=https://github.com/ceph/ceph-ci.git, io.buildah.version=1.41.3, org.label-schema.schema-version=1.0, org.opencontainers.image.authors=Ceph Release Team , OSD_FLAVOR=default, CEPH_REF=wip-shraddhaag-enable-cephadm-tests, CEPH_SHA1=e371433fcd1390a0b82cfb18effc422c1f5d8c1d, org.label-schema.vendor=CentOS, FROM_IMAGE=quay.io/centos/centos:stream9, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20260216, ceph=True, org.label-schema.license=GPLv2) 2026-02-23T13:11:59.960 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:11:59 trial185 podman[73801]: 2026-02-23 13:11:59.745984144 +0000 UTC m=+0.056767126 container start 8c29a9eb095cb1bd71e78d15ec0ff9260c98404be09bd1d6d932d1e25985d352 (image=quay.ceph.io/ceph-ci/ceph@sha256:49621faaa845c39b2aae72e4fa24123ee9b2ac3fb8ae834753907068ee2972ea, name=ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-2, org.opencontainers.image.documentation=https://docs.ceph.com/, GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfsganesha-5/, FROM_IMAGE=quay.io/centos/centos:stream9, org.label-schema.build-date=20260216, CEPH_REF=wip-shraddhaag-enable-cephadm-tests, org.opencontainers.image.authors=Ceph Release Team , org.label-schema.license=GPLv2, CEPH_GIT_REPO=https://github.com/ceph/ceph-ci.git, ceph=True, CEPH_SHA1=e371433fcd1390a0b82cfb18effc422c1f5d8c1d, org.label-schema.vendor=CentOS, io.buildah.version=1.41.3, OSD_FLAVOR=default, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0) 2026-02-23T13:11:59.960 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:11:59 trial185 bash[73801]: 8c29a9eb095cb1bd71e78d15ec0ff9260c98404be09bd1d6d932d1e25985d352 2026-02-23T13:11:59.960 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:11:59 trial185 podman[73801]: 2026-02-23 13:11:59.700357389 +0000 UTC m=+0.011140391 image pull 63df4b83db6f95a8ba965919b3671cb27173117c23b5c950aaadc8913f56213c quay.ceph.io/ceph-ci/ceph@sha256:49621faaa845c39b2aae72e4fa24123ee9b2ac3fb8ae834753907068ee2972ea 2026-02-23T13:11:59.960 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:11:59 trial185 systemd[1]: Started Ceph osd.2 for 5f823084-10b8-11f1-be80-d404e6e7d460. 2026-02-23T13:11:59.960 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:11:59 trial185 ceph-osd[73827]: -- 10.20.193.185:0/2813634164 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x2d81c00 msgr2=0x3bbe000 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).process 2026-02-23T13:11:59.960 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:11:59 trial185 ceph-osd[73827]: -- 10.20.193.185:0/2813634164 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x2d81c00 msgr2=0x3bbe000 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read continue len=32 2026-02-23T13:11:59.960 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:11:59 trial185 ceph-osd[73827]: --2- 10.20.193.185:0/2813634164 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x2d81c00 0x3bbe000 crc :-1 s=READY pgs=572 gs=1 cs=0 l=1 c_cookie=13901835509592696745 s_cookie=0 reconnecting=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main r=0 2026-02-23T13:11:59.960 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:11:59 trial185 ceph-osd[73827]: --2- 10.20.193.185:0/2813634164 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x2d81c00 0x3bbe000 crc :-1 s=THROTTLE_MESSAGE pgs=572 gs=1 cs=0 l=1 c_cookie=13901835509592696745 s_cookie=0 reconnecting=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).throttle_message 2026-02-23T13:11:59.961 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:11:59 trial185 ceph-osd[73827]: --2- 10.20.193.185:0/2813634164 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x2d81c00 0x3bbe000 crc :-1 s=THROTTLE_BYTES pgs=572 gs=1 cs=0 l=1 c_cookie=13901835509592696745 s_cookie=0 reconnecting=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).throttle_bytes 2026-02-23T13:11:59.961 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:11:59 trial185 ceph-osd[73827]: --2- 10.20.193.185:0/2813634164 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x2d81c00 0x3bbe000 crc :-1 s=THROTTLE_DISPATCH_QUEUE pgs=572 gs=1 cs=0 l=1 c_cookie=13901835509592696745 s_cookie=0 reconnecting=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).throttle_dispatch_queue 2026-02-23T13:11:59.961 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:11:59 trial185 ceph-osd[73827]: --2- 10.20.193.185:0/2813634164 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x2d81c00 0x3bbe000 crc :-1 s=THROTTLE_DONE pgs=572 gs=1 cs=0 l=1 c_cookie=13901835509592696745 s_cookie=0 reconnecting=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).read_frame_segment seg_idx=0 2026-02-23T13:11:59.961 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:11:59 trial185 ceph-osd[73827]: -- 10.20.193.185:0/2813634164 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x2d81c00 msgr2=0x3bbe000 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read start len=45 2026-02-23T13:11:59.961 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:11:59 trial185 ceph-osd[73827]: --2- 10.20.193.185:0/2813634164 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x2d81c00 0x3bbe000 crc :-1 s=THROTTLE_DONE pgs=572 gs=1 cs=0 l=1 c_cookie=13901835509592696745 s_cookie=0 reconnecting=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_segment r=0 2026-02-23T13:11:59.961 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:11:59 trial185 ceph-osd[73827]: --2- 10.20.193.185:0/2813634164 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x2d81c00 0x3bbe000 crc :-1 s=THROTTLE_DONE pgs=572 gs=1 cs=0 l=1 c_cookie=13901835509592696745 s_cookie=0 reconnecting=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).read_frame_segment seg_idx=1 2026-02-23T13:11:59.961 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:11:59 trial185 ceph-osd[73827]: -- 10.20.193.185:0/2813634164 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x2d81c00 msgr2=0x3bbe000 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read start len=194 2026-02-23T13:11:59.961 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:11:59 trial185 ceph-osd[73827]: --2- 10.20.193.185:0/2813634164 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x2d81c00 0x3bbe000 crc :-1 s=THROTTLE_DONE pgs=572 gs=1 cs=0 l=1 c_cookie=13901835509592696745 s_cookie=0 reconnecting=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_segment r=0 2026-02-23T13:11:59.961 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:11:59 trial185 ceph-osd[73827]: -- 10.20.193.185:0/2813634164 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x2d81c00 msgr2=0x3bbe000 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read start len=13 2026-02-23T13:11:59.961 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:11:59 trial185 ceph-osd[73827]: --2- 10.20.193.185:0/2813634164 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x2d81c00 0x3bbe000 crc :-1 s=THROTTLE_DONE pgs=572 gs=1 cs=0 l=1 c_cookie=13901835509592696745 s_cookie=0 reconnecting=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_epilogue_main r=0 2026-02-23T13:11:59.962 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:11:59 trial185 ceph-osd[73827]: --2- 10.20.193.185:0/2813634164 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x2d81c00 0x3bbe000 crc :-1 s=THROTTLE_DONE pgs=572 gs=1 cs=0 l=1 c_cookie=13901835509592696745 s_cookie=0 reconnecting=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_dispatch tag=17 2026-02-23T13:11:59.962 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:11:59 trial185 ceph-osd[73827]: --2- 10.20.193.185:0/2813634164 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x2d81c00 0x3bbe000 crc :-1 s=THROTTLE_DONE pgs=572 gs=1 cs=0 l=1 c_cookie=13901835509592696745 s_cookie=0 reconnecting=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_message 2026-02-23T13:11:59.962 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:11:59 trial185 ceph-osd[73827]: --2- 10.20.193.185:0/2813634164 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x2d81c00 0x3bbe000 crc :-1 s=THROTTLE_DONE pgs=572 gs=1 cs=0 l=1 c_cookie=13901835509592696745 s_cookie=0 reconnecting=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_message got 194 + 0 + 0 byte message. envelope type=18 src mon.0 off 0 2026-02-23T13:11:59.962 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:11:59 trial185 ceph-osd[73827]: --2- 10.20.193.185:0/2813634164 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x2d81c00 0x3bbe000 crc :-1 s=READ_MESSAGE_COMPLETE pgs=572 gs=1 cs=0 l=1 c_cookie=13901835509592696745 s_cookie=0 reconnecting=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_message received message m=0x2de5860 seq=4 from=mon.0 type=18 auth_reply(proto 2 0 (0) Success) 2026-02-23T13:11:59.962 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:11:59 trial185 ceph-osd[73827]: -- 10.20.193.185:0/2813634164 queue 0x2de5860 prio 196 2026-02-23T13:11:59.962 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:11:59 trial185 ceph-osd[73827]: --2- 10.20.193.185:0/2813634164 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x2d81c00 0x3bbe000 crc :-1 s=READY pgs=572 gs=1 cs=0 l=1 c_cookie=13901835509592696745 s_cookie=0 reconnecting=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).read_frame 2026-02-23T13:11:59.962 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:11:59 trial185 ceph-osd[73827]: -- 10.20.193.185:0/2813634164 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x2d81c00 msgr2=0x3bbe000 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read start len=32 2026-02-23T13:11:59.962 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:11:59 trial185 ceph-osd[73827]: -- 10.20.193.185:0/2813634164 <== mon.0 v2:10.20.193.68:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (crc 0 0 0) 0x2de5860 con 0x2d81c00 2026-02-23T13:11:59.963 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:11:59 trial185 ceph-osd[73827]: -- 10.20.193.185:0/2813634164 dispatch_throttle_release 194 to dispatch throttler 194/104857600 2026-02-23T13:11:59.963 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:11:59 trial185 ceph-osd[73827]: -- 10.20.193.185:0/2813634164 done calling dispatch on 0x2de5860 2026-02-23T13:11:59.963 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:11:59 trial185 ceph-osd[73827]: WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:11:59.963 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:11:59 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-2[73823]: 2026-02-23T13:11:59.759+0000 7f7385a61780 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:11:59.963 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:11:59 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-2[73823]: 2026-02-23T13:11:59.759+0000 7f7385a61780 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:11:59.963 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:11:59 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-2[73823]: 2026-02-23T13:11:59.760+0000 7f7385a61780 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:12:00.017 INFO:teuthology.orchestra.run.trial185.stderr:no valid command found; 10 closest matches: 2026-02-23T13:12:00.018 INFO:teuthology.orchestra.run.trial185.stderr:0 2026-02-23T13:12:00.018 INFO:teuthology.orchestra.run.trial185.stderr:1 2026-02-23T13:12:00.018 INFO:teuthology.orchestra.run.trial185.stderr:2 2026-02-23T13:12:00.018 INFO:teuthology.orchestra.run.trial185.stderr:abort 2026-02-23T13:12:00.018 INFO:teuthology.orchestra.run.trial185.stderr:assert 2026-02-23T13:12:00.018 INFO:teuthology.orchestra.run.trial185.stderr:bluestore collections 2026-02-23T13:12:00.018 INFO:teuthology.orchestra.run.trial185.stderr:bluestore compression stats [] 2026-02-23T13:12:00.019 INFO:teuthology.orchestra.run.trial185.stderr:bluestore list [] [] 2026-02-23T13:12:00.019 INFO:teuthology.orchestra.run.trial185.stderr:bluestore onode metadata 2026-02-23T13:12:00.019 INFO:teuthology.orchestra.run.trial185.stderr:bluestore show sharding 2026-02-23T13:12:00.019 INFO:teuthology.orchestra.run.trial185.stderr:admin_socket: invalid command 2026-02-23T13:12:00.051 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-23T13:12:00.052 INFO:tasks.cephadm.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-23T13:12:00.357 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:12:00 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-2[73823]: 2026-02-23T13:12:00.007+0000 7f7385a61780 -1 Falling back to public interface 2026-02-23T13:12:00.548 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 22 127 144208 144081 51.1611 37.2109 0.0278292 0.0195246 2026-02-23T13:12:01.372 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:12:01.527 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:12:01.548 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 23 128 148819 148691 50.5025 36.0156 0.0210945 0.0197828 2026-02-23T13:12:01.580 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:12:01 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T13:12:01.581 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:12:01 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T13:12:01.581 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:12:01 trial068 ceph-mon[38805]: pgmap v245: 17 pgs: 15 active+undersized+degraded, 2 active+clean; 971 MiB data, 4.5 GiB used, 2.7 TiB / 2.7 TiB avail; 38 MiB/s wr, 4.81k op/s; 13550/46590 objects degraded (29.083%); 2.0 KiB/s, 0 objects/s recovering 2026-02-23T13:12:01.658 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:12:01.656+0000 7fd9fd276640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:12:01.658 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:12:01.657+0000 7fd9fd276640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:12:01.675 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:12:01.675 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:12:01.689 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:12:01.689 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:12:01.690 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:12:01.702 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:12:01.702 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:12:01.739 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:12:01.739 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:12:02.549 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 24 128 153278 153150 49.8496 34.8359 0.0233183 0.0200363 2026-02-23T13:12:02.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:12:02 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/65889426' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:12:03.549 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 25 127 157337 157210 49.1236 31.7188 0.0130702 0.0203343 2026-02-23T13:12:03.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:12:03 trial068 ceph-mon[38805]: pgmap v246: 17 pgs: 15 active+undersized+degraded, 2 active+clean; 1.1 GiB data, 4.8 GiB used, 2.7 TiB / 2.7 TiB avail; 35 MiB/s wr, 4.42k op/s; 15662/52926 objects degraded (29.592%); 1.8 KiB/s, 0 objects/s recovering 2026-02-23T13:12:03.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:12:03 trial068 ceph-mon[38805]: Health check update: Degraded data redundancy: 15662/52926 objects degraded (29.592%), 15 pgs degraded (PG_DEGRADED) 2026-02-23T13:12:03.741 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:12:03.886 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:12:04.008 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:12:04.006+0000 7fa9662c9640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:12:04.008 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:12:04.007+0000 7fa9662c9640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:12:04.023 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:12:04.023 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:12:04.024 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:12:03 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:12:03] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:12:04.037 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:12:04.037 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:12:04.037 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:12:04.053 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:12:04.053 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:12:04.096 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:12:04.096 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:12:04.357 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:12:03 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:12:03.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:12:04.549 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 26 128 161270 161142 48.4156 30.7188 0.0174448 0.020629 2026-02-23T13:12:04.998 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:12:04 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-2[73823]: 2026-02-23T13:12:04.690+0000 7f7385a61780 -1 osd.2 130 log_to_monitors true 2026-02-23T13:12:04.998 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:12:04 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-2[73823]: 2026-02-23T13:12:04.995+0000 7f73798a2640 -1 osd.2 130 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2026-02-23T13:12:05.052 DEBUG:teuthology.orchestra.run.trial185:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph daemon osd.2 dump_ops_in_flight 2026-02-23T13:12:05.187 INFO:teuthology.orchestra.run.trial185.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/config/ceph.conf 2026-02-23T13:12:05.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:12:04 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1517381232' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:12:05.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:12:04 trial068 ceph-mon[38805]: pgmap v247: 17 pgs: 15 active+undersized+degraded, 2 active+clean; 1.1 GiB data, 5.0 GiB used, 2.7 TiB / 2.7 TiB avail; 36 MiB/s wr, 4.59k op/s; 16201/55761 objects degraded (29.054%); 1.6 KiB/s, 0 objects/s recovering 2026-02-23T13:12:05.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:12:04 trial068 ceph-mon[38805]: from='osd.2 [v2:10.20.193.185:6800/1398591049,v1:10.20.193.185:6801/1398591049]' entity='osd.2' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]} : dispatch 2026-02-23T13:12:05.420 INFO:teuthology.orchestra.run.trial185.stdout:{ 2026-02-23T13:12:05.420 INFO:teuthology.orchestra.run.trial185.stdout: "ops": [], 2026-02-23T13:12:05.420 INFO:teuthology.orchestra.run.trial185.stdout: "num_ops": 0 2026-02-23T13:12:05.420 INFO:teuthology.orchestra.run.trial185.stdout:} 2026-02-23T13:12:05.549 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 27 128 165347 165219 47.8021 31.8516 0.0283366 0.0209 2026-02-23T13:12:06.098 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:12:06.215 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:12:06.264 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:12:05 trial068 ceph-mon[38805]: from='osd.2 [v2:10.20.193.185:6800/1398591049,v1:10.20.193.185:6801/1398591049]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2026-02-23T13:12:06.264 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:12:05 trial068 ceph-mon[38805]: osdmap e133: 4 total, 3 up, 4 in 2026-02-23T13:12:06.264 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:12:05 trial068 ceph-mon[38805]: from='osd.2 [v2:10.20.193.185:6800/1398591049,v1:10.20.193.185:6801/1398591049]' entity='osd.2' cmd={"prefix": "osd crush create-or-move", "id": 2, "weight":0.6836, "args": ["host=trial185", "root=default"]} : dispatch 2026-02-23T13:12:06.323 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:12:06.321+0000 7f934d901640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:12:06.323 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:12:06.321+0000 7f934d901640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:12:06.335 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:12:06.335 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:12:06.346 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:12:06.346 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:12:06.358 INFO:teuthology.orchestra.run.trial068.stdout:osd.2: {} 2026-02-23T13:12:06.358 INFO:teuthology.orchestra.run.trial068.stderr:osd.2: osd_delete_sleep = '' osd_delete_sleep_hdd = '' osd_delete_sleep_hybrid = '' osd_delete_sleep_ssd = '' osd_enable_op_tracker = '' osd_recovery_max_active = '' osd_recovery_max_active_hdd = '' osd_recovery_max_active_ssd = '' osd_recovery_sleep = '' osd_recovery_sleep_degraded = '' osd_recovery_sleep_degraded_hdd = '' osd_recovery_sleep_degraded_hybrid = '' osd_recovery_sleep_degraded_ssd = '' osd_recovery_sleep_hdd = '' osd_recovery_sleep_hybrid = '' osd_recovery_sleep_ssd = '' osd_scrub_sleep = '' osd_snap_trim_sleep = '' osd_snap_trim_sleep_hdd = '' osd_snap_trim_sleep_hybrid = '' osd_snap_trim_sleep_ssd = '' 2026-02-23T13:12:06.369 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:12:06.370 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:12:06.550 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 28 128 167272 167144 46.632 15.0391 0.0049391 0.021005 2026-02-23T13:12:07.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:12:06 trial068 ceph-mon[38805]: Health check cleared: OSD_DOWN (was: 1 osds down) 2026-02-23T13:12:07.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:12:06 trial068 ceph-mon[38805]: osd.2 [v2:10.20.193.185:6800/1398591049,v1:10.20.193.185:6801/1398591049] boot 2026-02-23T13:12:07.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:12:06 trial068 ceph-mon[38805]: osdmap e134: 4 total, 4 up, 4 in 2026-02-23T13:12:07.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:12:06 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2026-02-23T13:12:07.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:12:06 trial068 ceph-mon[38805]: pgmap v250: 17 pgs: 15 active+undersized+degraded, 2 active+clean; 1.2 GiB data, 5.0 GiB used, 2.7 TiB / 2.7 TiB avail; 29 MiB/s wr, 3.69k op/s; 16838/57672 objects degraded (29.196%) 2026-02-23T13:12:07.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:12:06 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2944459739' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:12:07.549 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 29 128 167272 167144 45.024 0 - 0.021005 2026-02-23T13:12:08.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:12:08 trial068 ceph-mon[38805]: osdmap e135: 4 total, 4 up, 4 in 2026-02-23T13:12:08.406 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:12:08.549 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 30 128 167858 167730 43.6758 2.28906 0.0668847 0.0228742 2026-02-23T13:12:08.558 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:12:08.687 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:12:08.685+0000 7f19af643640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:12:08.688 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:12:08.686+0000 7f19af643640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:12:08.703 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:12:08.703 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:12:08.716 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:12:08.716 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:12:08.731 INFO:teuthology.orchestra.run.trial068.stdout:osd.2: {} 2026-02-23T13:12:08.731 INFO:teuthology.orchestra.run.trial068.stderr:osd.2: osd_enable_op_tracker = '' 2026-02-23T13:12:08.743 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:12:08.744 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:12:09.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:12:09 trial068 ceph-mon[38805]: osdmap e136: 4 total, 4 up, 4 in 2026-02-23T13:12:09.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:12:09 trial068 ceph-mon[38805]: pgmap v253: 17 pgs: 3 remapped+peering, 5 peering, 7 active+undersized+degraded, 2 active+clean; 1.2 GiB data, 5.3 GiB used, 2.7 TiB / 2.7 TiB avail; 28 MiB/s wr, 3.63k op/s; 8569/61203 objects degraded (14.001%) 2026-02-23T13:12:09.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:12:09 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T13:12:09.345 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:12:09 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T13:12:09.345 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:12:09 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3135923167' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:12:09.345 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:12:09 trial068 ceph-mon[38805]: Health check update: Degraded data redundancy: 16838/57672 objects degraded (29.196%), 15 pgs degraded (PG_DEGRADED) 2026-02-23T13:12:09.549 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 31 128 171452 171324 43.1726 28.0781 0.0210481 0.0231399 2026-02-23T13:12:10.472 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [1, 3, 0, 2] 2026-02-23T13:12:10.472 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2026-02-23T13:12:10.472 INFO:tasks.thrashosds.thrasher:Killing osd 1, live_osds are [1, 3, 0, 2] 2026-02-23T13:12:10.472 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2026-02-23T13:12:10.473 DEBUG:teuthology.orchestra.run.trial068:> sudo systemctl stop ceph-5f823084-10b8-11f1-be80-d404e6e7d460@osd.1 2026-02-23T13:12:10.550 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 32 128 174830 174702 42.6481 26.3906 0.0327804 0.0234252 2026-02-23T13:12:10.784 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:12:10.812 INFO:journalctl@ceph.osd.1.trial068.stdout:Feb 23 13:12:10 trial068 systemd[1]: Stopping Ceph osd.1 for 5f823084-10b8-11f1-be80-d404e6e7d460... 2026-02-23T13:12:10.812 INFO:journalctl@ceph.osd.1.trial068.stdout:Feb 23 13:12:10 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-1[85558]: 2026-02-23T13:12:10.600+0000 7f0ce87c4640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --osd-objectstore=bluestore (PID: 1) UID: 0 2026-02-23T13:12:10.812 INFO:journalctl@ceph.osd.1.trial068.stdout:Feb 23 13:12:10 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-1[85558]: 2026-02-23T13:12:10.600+0000 7f0ce87c4640 -1 osd.1 136 *** Got signal Terminated *** 2026-02-23T13:12:10.812 INFO:journalctl@ceph.osd.1.trial068.stdout:Feb 23 13:12:10 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-1[85558]: 2026-02-23T13:12:10.600+0000 7f0ce87c4640 -1 osd.1 136 *** Immediate shutdown (osd_fast_shutdown=true) *** 2026-02-23T13:12:10.932 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:12:11.061 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:12:11.058+0000 7f454cfe3640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:12:11.061 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:12:11.059+0000 7f454cfe3640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:12:11.074 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:12:11.074 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:12:11.089 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:12:11.089 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:12:11.104 INFO:teuthology.orchestra.run.trial068.stdout:osd.2: {} 2026-02-23T13:12:11.104 INFO:teuthology.orchestra.run.trial068.stderr:osd.2: osd_enable_op_tracker = '' 2026-02-23T13:12:11.119 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:12:11.119 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:12:11.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:12:11 trial068 ceph-mon[38805]: pgmap v254: 17 pgs: 3 remapped+peering, 5 peering, 7 active+undersized+degraded, 2 active+clean; 1.2 GiB data, 5.3 GiB used, 2.7 TiB / 2.7 TiB avail; 21 MiB/s wr, 2.75k op/s; 8569/61203 objects degraded (14.001%) 2026-02-23T13:12:11.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:12:11 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T13:12:11.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:12:11 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T13:12:11.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:12:11 trial068 ceph-mon[38805]: osd.1 marked itself down and dead 2026-02-23T13:12:11.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:12:11 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/319280729' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:12:11.550 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 33 128 177495 177367 41.9866 20.8203 0.041752 0.0236581 2026-02-23T13:12:11.844 INFO:journalctl@ceph.osd.1.trial068.stdout:Feb 23 13:12:11 trial068 podman[98279]: 2026-02-23 13:12:11.523227023 +0000 UTC m=+0.936567455 container died 1db5dbbd33423db9ff24044ed084029c9c425941c10daf030d13b0c68271c85c (image=quay.ceph.io/ceph-ci/ceph@sha256:49621faaa845c39b2aae72e4fa24123ee9b2ac3fb8ae834753907068ee2972ea, name=ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-1, org.opencontainers.image.authors=Ceph Release Team , org.label-schema.schema-version=1.0, CEPH_SHA1=e371433fcd1390a0b82cfb18effc422c1f5d8c1d, OSD_FLAVOR=default, org.label-schema.vendor=CentOS, CEPH_GIT_REPO=https://github.com/ceph/ceph-ci.git, org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20260216, io.buildah.version=1.41.3, org.opencontainers.image.documentation=https://docs.ceph.com/, FROM_IMAGE=quay.io/centos/centos:stream9, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_REF=wip-shraddhaag-enable-cephadm-tests, GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfsganesha-5/) 2026-02-23T13:12:11.845 INFO:journalctl@ceph.osd.1.trial068.stdout:Feb 23 13:12:11 trial068 podman[98279]: 2026-02-23 13:12:11.532731746 +0000 UTC m=+0.946072168 container remove 1db5dbbd33423db9ff24044ed084029c9c425941c10daf030d13b0c68271c85c (image=quay.ceph.io/ceph-ci/ceph@sha256:49621faaa845c39b2aae72e4fa24123ee9b2ac3fb8ae834753907068ee2972ea, name=ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-1, ceph=True, CEPH_SHA1=e371433fcd1390a0b82cfb18effc422c1f5d8c1d, org.label-schema.license=GPLv2, GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfsganesha-5/, org.opencontainers.image.documentation=https://docs.ceph.com/, FROM_IMAGE=quay.io/centos/centos:stream9, CEPH_REF=wip-shraddhaag-enable-cephadm-tests, org.label-schema.vendor=CentOS, org.label-schema.build-date=20260216, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, OSD_FLAVOR=default, io.buildah.version=1.41.3, CEPH_GIT_REPO=https://github.com/ceph/ceph-ci.git, org.opencontainers.image.authors=Ceph Release Team ) 2026-02-23T13:12:11.845 INFO:journalctl@ceph.osd.1.trial068.stdout:Feb 23 13:12:11 trial068 bash[98279]: ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-1 2026-02-23T13:12:11.845 INFO:journalctl@ceph.osd.1.trial068.stdout:Feb 23 13:12:11 trial068 podman[98506]: 2026-02-23 13:12:11.662047721 +0000 UTC m=+0.015060531 container create eb86caeb697aa1e19f34bfe0e58e8d03068deebd8641171c20750b5a45966461 (image=quay.ceph.io/ceph-ci/ceph@sha256:49621faaa845c39b2aae72e4fa24123ee9b2ac3fb8ae834753907068ee2972ea, name=ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-1-deactivate, CEPH_REF=wip-shraddhaag-enable-cephadm-tests, org.opencontainers.image.authors=Ceph Release Team , OSD_FLAVOR=default, io.buildah.version=1.41.3, CEPH_SHA1=e371433fcd1390a0b82cfb18effc422c1f5d8c1d, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfsganesha-5/, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20260216, org.opencontainers.image.documentation=https://docs.ceph.com/, CEPH_GIT_REPO=https://github.com/ceph/ceph-ci.git, FROM_IMAGE=quay.io/centos/centos:stream9) 2026-02-23T13:12:11.845 INFO:journalctl@ceph.osd.1.trial068.stdout:Feb 23 13:12:11 trial068 podman[98506]: 2026-02-23 13:12:11.697254884 +0000 UTC m=+0.050267644 container init eb86caeb697aa1e19f34bfe0e58e8d03068deebd8641171c20750b5a45966461 (image=quay.ceph.io/ceph-ci/ceph@sha256:49621faaa845c39b2aae72e4fa24123ee9b2ac3fb8ae834753907068ee2972ea, name=ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-1-deactivate, GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfsganesha-5/, org.opencontainers.image.authors=Ceph Release Team , io.buildah.version=1.41.3, CEPH_GIT_REPO=https://github.com/ceph/ceph-ci.git, FROM_IMAGE=quay.io/centos/centos:stream9, CEPH_SHA1=e371433fcd1390a0b82cfb18effc422c1f5d8c1d, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20260216, org.label-schema.schema-version=1.0, CEPH_REF=wip-shraddhaag-enable-cephadm-tests, OSD_FLAVOR=default, org.opencontainers.image.documentation=https://docs.ceph.com/, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image) 2026-02-23T13:12:11.845 INFO:journalctl@ceph.osd.1.trial068.stdout:Feb 23 13:12:11 trial068 podman[98506]: 2026-02-23 13:12:11.699813425 +0000 UTC m=+0.052826165 container start eb86caeb697aa1e19f34bfe0e58e8d03068deebd8641171c20750b5a45966461 (image=quay.ceph.io/ceph-ci/ceph@sha256:49621faaa845c39b2aae72e4fa24123ee9b2ac3fb8ae834753907068ee2972ea, name=ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-1-deactivate, org.opencontainers.image.documentation=https://docs.ceph.com/, org.label-schema.vendor=CentOS, CEPH_REF=wip-shraddhaag-enable-cephadm-tests, GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfsganesha-5/, OSD_FLAVOR=default, org.label-schema.schema-version=1.0, CEPH_GIT_REPO=https://github.com/ceph/ceph-ci.git, FROM_IMAGE=quay.io/centos/centos:stream9, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_SHA1=e371433fcd1390a0b82cfb18effc422c1f5d8c1d, org.label-schema.license=GPLv2, org.label-schema.build-date=20260216, io.buildah.version=1.41.3, org.opencontainers.image.authors=Ceph Release Team ) 2026-02-23T13:12:11.846 INFO:journalctl@ceph.osd.1.trial068.stdout:Feb 23 13:12:11 trial068 podman[98506]: 2026-02-23 13:12:11.700281335 +0000 UTC m=+0.053294095 container attach eb86caeb697aa1e19f34bfe0e58e8d03068deebd8641171c20750b5a45966461 (image=quay.ceph.io/ceph-ci/ceph@sha256:49621faaa845c39b2aae72e4fa24123ee9b2ac3fb8ae834753907068ee2972ea, name=ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-1-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfsganesha-5/, io.buildah.version=1.41.3, org.opencontainers.image.documentation=https://docs.ceph.com/, CEPH_REF=wip-shraddhaag-enable-cephadm-tests, org.label-schema.license=GPLv2, org.opencontainers.image.authors=Ceph Release Team , OSD_FLAVOR=default, CEPH_GIT_REPO=https://github.com/ceph/ceph-ci.git, ceph=True, org.label-schema.schema-version=1.0, FROM_IMAGE=quay.io/centos/centos:stream9, CEPH_SHA1=e371433fcd1390a0b82cfb18effc422c1f5d8c1d, org.label-schema.build-date=20260216, org.label-schema.vendor=CentOS) 2026-02-23T13:12:11.846 INFO:journalctl@ceph.osd.1.trial068.stdout:Feb 23 13:12:11 trial068 podman[98506]: 2026-02-23 13:12:11.655796426 +0000 UTC m=+0.008809186 image pull 63df4b83db6f95a8ba965919b3671cb27173117c23b5c950aaadc8913f56213c quay.ceph.io/ceph-ci/ceph@sha256:49621faaa845c39b2aae72e4fa24123ee9b2ac3fb8ae834753907068ee2972ea 2026-02-23T13:12:11.883 DEBUG:teuthology.orchestra.run.trial068:> sudo pkill -f 'journalctl -f -n 0 -u ceph-5f823084-10b8-11f1-be80-d404e6e7d460@osd.1.service' 2026-02-23T13:12:11.920 DEBUG:teuthology.orchestra.run:got remote process result: None 2026-02-23T13:12:11.920 INFO:tasks.cephadm.osd.1:Stopped osd.1 2026-02-23T13:12:12.550 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 34 128 177563 177435 40.7672 0.53125 0.0024748 0.0239538 2026-02-23T13:12:12.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:12:12 trial068 ceph-mon[38805]: Health check failed: 1 osds down (OSD_DOWN) 2026-02-23T13:12:12.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:12:12 trial068 ceph-mon[38805]: osdmap e137: 4 total, 3 up, 4 in 2026-02-23T13:12:13.162 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:12:13.282 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:12:13.391 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:12:13.389+0000 7fe9d1264640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:12:13.392 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:12:13.390+0000 7fe9d1264640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:12:13.404 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:12:13.404 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:12:13.404 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.1 2026-02-23T13:12:13.416 INFO:teuthology.orchestra.run.trial068.stdout:osd.2: {} 2026-02-23T13:12:13.416 INFO:teuthology.orchestra.run.trial068.stderr:osd.2: osd_enable_op_tracker = '' 2026-02-23T13:12:13.428 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:12:13.428 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:12:13.448 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:12:13 trial068 ceph-mon[38805]: pgmap v256: 17 pgs: 2 remapped+peering, 2 active+recovering+undersized+degraded+remapped, 1 remapped, 1 stale+active+clean, 7 active+recovery_wait+undersized+degraded+remapped, 1 peering, 1 active+clean, 1 stale+remapped+peering, 1 active+undersized+degraded; 1.3 GiB data, 5.4 GiB used, 2.7 TiB / 2.7 TiB avail; 25 MiB/s wr, 3.23k op/s; 4594/64935 objects degraded (7.075%); 5.2 MiB/s, 83 objects/s recovering 2026-02-23T13:12:13.448 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:12:13 trial068 ceph-mon[38805]: osdmap e138: 4 total, 3 up, 4 in 2026-02-23T13:12:13.448 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:12:13 trial068 ceph-mon[38805]: osdmap e139: 4 total, 3 up, 4 in 2026-02-23T13:12:13.477 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:12:13.478 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:12:13.550 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 35 128 178071 177943 39.7158 3.96875 0.0717003 0.0250773 2026-02-23T13:12:14.324 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:12:13 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:12:13] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:12:14.357 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:12:13 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:12:13.972Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:12:14.550 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 36 128 180070 179942 39.0464 15.6172 0.056569 0.0255819 2026-02-23T13:12:14.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:12:14 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3478784585' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:12:14.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:12:14 trial068 ceph-mon[38805]: Health check update: Degraded data redundancy: 4594/64935 objects degraded (7.075%), 10 pgs degraded (PG_DEGRADED) 2026-02-23T13:12:14.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:12:14 trial068 ceph-mon[38805]: osdmap e140: 4 total, 3 up, 4 in 2026-02-23T13:12:15.479 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:12:15.550 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 37 128 183607 183479 38.7378 27.6328 0.0123667 0.0257959 2026-02-23T13:12:15.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:12:15 trial068 ceph-mon[38805]: pgmap v260: 17 pgs: 2 remapped+peering, 2 active+recovering+undersized+degraded+remapped, 1 remapped, 1 stale+active+clean, 7 active+recovery_wait+undersized+degraded+remapped, 1 peering, 1 active+clean, 1 stale+remapped+peering, 1 active+undersized+degraded; 1.3 GiB data, 5.4 GiB used, 2.7 TiB / 2.7 TiB avail; 19 MiB/s wr, 2.48k op/s; 4594/64935 objects degraded (7.075%); 7.8 MiB/s, 125 objects/s recovering 2026-02-23T13:12:15.604 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:12:15.878 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:12:15.828+0000 7fddba7bd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:12:15.878 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:12:15.828+0000 7fddba7bd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:12:15.878 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:12:15.878 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.1 2026-02-23T13:12:15.878 INFO:teuthology.orchestra.run.trial068.stderr:osd.2: osd_enable_op_tracker = '' 2026-02-23T13:12:15.878 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:12:15.878 INFO:teuthology.orchestra.run.trial068.stdout:osd.2: {} 2026-02-23T13:12:15.878 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:12:15.878 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:12:15.952 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:12:15.953 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:12:16.550 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 38 128 187338 187210 38.4854 29.1484 0.0238486 0.0259679 2026-02-23T13:12:16.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:12:16 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2224799282' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:12:16.920 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [1] live_osds: [3, 0, 2] 2026-02-23T13:12:16.921 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2026-02-23T13:12:16.921 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph pg dump --format=json 2026-02-23T13:12:17.049 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:12:17.161 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:12:17.160+0000 7fbcd0a0d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:12:17.162 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:12:17.160+0000 7fbcd0a0d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:12:17.296 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T13:12:17.297 INFO:teuthology.orchestra.run.trial068.stderr:dumped all 2026-02-23T13:12:17.342 INFO:teuthology.orchestra.run.trial068.stdout:{"pg_ready":true,"pg_map":{"version":261,"stamp":"2026-02-23T13:12:16.268496+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1481220640,"num_objects":22597,"num_object_clones":0,"num_object_copies":67791,"num_objects_missing_on_primary":807,"num_objects_missing":807,"num_objects_degraded":12979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22597,"num_whiteouts":0,"num_read":122,"num_read_kb":228,"num_write":180876,"num_write_kb":1448078,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3595,"num_bytes_recovered":235741728,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160840,"ondisk_log_size":160840,"up":41,"acting":37,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":51,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":2936012800,"kb_used":6177932,"kb_used_data":4187004,"kb_used_omap":141,"kb_used_meta":1990706,"kb_avail":2929834868,"statfs":{"total":3006477107200,"available":3000150904832,"internally_reserved":0,"allocated":4287492096,"data_stored":4283558802,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":144491,"internal_metadata":2038483861},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,0,0,0,0,0,0,0,23],"upper_bound":2048},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":113008640,"num_objects":1724,"num_object_clones":0,"num_object_copies":5172,"num_objects_missing_on_primary":807,"num_objects_missing":807,"num_objects_degraded":8536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1724,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13795,"num_write_kb":110360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3591,"num_bytes_recovered":235069440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.948881"},"pg_stats":[{"pgid":"4.f","version":"140'11528","reported_seq":12089,"reported_epoch":140,"state":"active+recovering+undersized+degraded","last_fresh":"2026-02-23T13:12:15.019107+0000","last_change":"2026-02-23T13:12:13.331940+0000","last_active":"2026-02-23T13:12:15.019107+0000","last_peered":"2026-02-23T13:12:15.019107+0000","last_clean":"2026-02-23T13:11:53.314008+0000","last_became_active":"2026-02-23T13:12:13.328498+0000","last_became_peered":"2026-02-23T13:12:13.328498+0000","last_unstale":"2026-02-23T13:12:15.019107+0000","last_undegraded":"2026-02-23T13:12:13.324788+0000","last_fullsized":"2026-02-23T13:12:13.321976+0000","mapping_epoch":138,"log_start":"129'1500","ondisk_log_start":"129'1500","created":126,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:11:36.450546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:11:36.450546+0000","last_clean_scrub_stamp":"2026-02-23T13:11:36.450546+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":94437376,"num_objects":1441,"num_object_clones":0,"num_object_copies":4323,"num_objects_missing_on_primary":183,"num_objects_missing":183,"num_objects_degraded":1624,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1441,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11528,"num_write_kb":92224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":255,"num_bytes_recovered":16711680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["3"],"object_location_counts":[{"shards":"2,3","objects":1258},{"shards":"3","objects":183}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.e","version":"140'11539","reported_seq":11634,"reported_epoch":140,"state":"active+recovering+undersized+degraded","last_fresh":"2026-02-23T13:12:15.002538+0000","last_change":"2026-02-23T13:12:14.705764+0000","last_active":"2026-02-23T13:12:15.002538+0000","last_peered":"2026-02-23T13:12:15.002538+0000","last_clean":"2026-02-23T13:11:53.296540+0000","last_became_active":"2026-02-23T13:12:13.326690+0000","last_became_peered":"2026-02-23T13:12:13.326690+0000","last_unstale":"2026-02-23T13:12:15.002538+0000","last_undegraded":"2026-02-23T13:12:13.323536+0000","last_fullsized":"2026-02-23T13:12:13.322025+0000","mapping_epoch":138,"log_start":"129'1500","ondisk_log_start":"129'1500","created":126,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:11:36.450546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:11:36.450546+0000","last_clean_scrub_stamp":"2026-02-23T13:11:36.450546+0000","objects_scrubbed":0,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":94527488,"num_objects":1443,"num_object_clones":0,"num_object_copies":4329,"num_objects_missing_on_primary":397,"num_objects_missing":397,"num_objects_degraded":1840,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1443,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11539,"num_write_kb":92312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":1957888,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["3"],"object_location_counts":[{"shards":"2,3","objects":1045},{"shards":"3","objects":397}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"137'10952","reported_seq":11008,"reported_epoch":137,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2026-02-23T13:12:11.362547+0000","last_change":"2026-02-23T13:12:08.377362+0000","last_active":"2026-02-23T13:12:11.362547+0000","last_peered":"2026-02-23T13:12:11.362547+0000","last_clean":"2026-02-23T13:11:53.307568+0000","last_became_active":"2026-02-23T13:12:08.342935+0000","last_became_peered":"2026-02-23T13:12:08.342935+0000","last_unstale":"2026-02-23T13:12:11.362547+0000","last_undegraded":"2026-02-23T13:12:08.018992+0000","last_fullsized":"2026-02-23T13:12:08.000039+0000","mapping_epoch":135,"log_start":"129'900","ondisk_log_start":"129'900","created":126,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:11:36.450546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:11:36.450546+0000","last_clean_scrub_stamp":"2026-02-23T13:11:36.450546+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":89718784,"num_objects":1369,"num_object_clones":0,"num_object_copies":4107,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":418,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1369,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10952,"num_write_kb":87616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[{"shards":"0,3","objects":951}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"136'11626","reported_seq":11661,"reported_epoch":136,"state":"remapped+peering","last_fresh":"2026-02-23T13:12:11.316884+0000","last_change":"2026-02-23T13:12:11.325285+0000","last_active":"2026-02-23T13:12:11.325285+0000","last_peered":"2026-02-23T13:12:11.316884+0000","last_clean":"2026-02-23T13:11:53.292743+0000","last_became_active":"2026-02-23T13:12:08.358478+0000","last_became_peered":"2026-02-23T13:12:08.358478+0000","last_unstale":"2026-02-23T13:12:11.316884+0000","last_undegraded":"2026-02-23T13:12:08.026860+0000","last_fullsized":"2026-02-23T13:12:08.000228+0000","mapping_epoch":137,"log_start":"129'1600","ondisk_log_start":"129'1600","created":126,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:11:36.450546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:11:36.450546+0000","last_clean_scrub_stamp":"2026-02-23T13:11:36.450546+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":95240192,"num_objects":1454,"num_object_clones":0,"num_object_copies":4362,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1454,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11626,"num_write_kb":93008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":40960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"140'11512","reported_seq":12433,"reported_epoch":140,"state":"active+undersized+degraded","last_fresh":"2026-02-23T13:12:15.414693+0000","last_change":"2026-02-23T13:12:14.909959+0000","last_active":"2026-02-23T13:12:15.414693+0000","last_peered":"2026-02-23T13:12:15.414693+0000","last_clean":"2026-02-23T13:11:53.319354+0000","last_became_active":"2026-02-23T13:12:13.325738+0000","last_became_peered":"2026-02-23T13:12:13.325738+0000","last_unstale":"2026-02-23T13:12:15.414693+0000","last_undegraded":"2026-02-23T13:12:13.323178+0000","last_fullsized":"2026-02-23T13:12:13.322763+0000","mapping_epoch":138,"log_start":"129'1500","ondisk_log_start":"129'1500","created":126,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:11:36.450546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:11:36.450546+0000","last_clean_scrub_stamp":"2026-02-23T13:11:36.450546+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T13:13:01.303751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":94306304,"num_objects":1439,"num_object_clones":0,"num_object_copies":4317,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1439,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1439,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11512,"num_write_kb":92096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":435,"num_bytes_recovered":28508160,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,2","objects":1439}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"140'11136","reported_seq":12051,"reported_epoch":140,"state":"active+undersized+degraded","last_fresh":"2026-02-23T13:12:14.995631+0000","last_change":"2026-02-23T13:12:12.326368+0000","last_active":"2026-02-23T13:12:14.995631+0000","last_peered":"2026-02-23T13:12:14.995631+0000","last_clean":"2026-02-23T13:11:53.299069+0000","last_became_active":"2026-02-23T13:12:12.326368+0000","last_became_peered":"2026-02-23T13:12:12.326368+0000","last_unstale":"2026-02-23T13:12:14.995631+0000","last_undegraded":"2026-02-23T13:12:12.320392+0000","last_fullsized":"2026-02-23T13:12:12.320334+0000","mapping_epoch":137,"log_start":"129'1100","ondisk_log_start":"129'1100","created":126,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:11:36.450546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:11:36.450546+0000","last_clean_scrub_stamp":"2026-02-23T13:11:36.450546+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T13:13:02.518656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":91226112,"num_objects":1392,"num_object_clones":0,"num_object_copies":4176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1392,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1392,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11136,"num_write_kb":89088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":443,"num_bytes_recovered":29032448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":1392}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"140'11486","reported_seq":11986,"reported_epoch":140,"state":"active+recovering+undersized+degraded","last_fresh":"2026-02-23T13:12:15.011438+0000","last_change":"2026-02-23T13:12:13.331259+0000","last_active":"2026-02-23T13:12:15.011438+0000","last_peered":"2026-02-23T13:12:15.011438+0000","last_clean":"2026-02-23T13:11:53.315507+0000","last_became_active":"2026-02-23T13:12:13.326071+0000","last_became_peered":"2026-02-23T13:12:13.326071+0000","last_unstale":"2026-02-23T13:12:15.011438+0000","last_undegraded":"2026-02-23T13:12:13.324588+0000","last_fullsized":"2026-02-23T13:12:13.322148+0000","mapping_epoch":138,"log_start":"129'1400","ondisk_log_start":"129'1400","created":126,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:11:36.450546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:11:36.450546+0000","last_clean_scrub_stamp":"2026-02-23T13:11:36.450546+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":94093312,"num_objects":1436,"num_object_clones":0,"num_object_copies":4308,"num_objects_missing_on_primary":227,"num_objects_missing":227,"num_objects_degraded":1663,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1436,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11486,"num_write_kb":91888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":234,"num_bytes_recovered":15335424,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":["0"],"object_location_counts":[{"shards":"0","objects":227},{"shards":"0,2","objects":1208}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"137'11056","reported_seq":11376,"reported_epoch":137,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2026-02-23T13:12:11.609030+0000","last_change":"2026-02-23T13:12:11.324313+0000","last_active":"2026-02-23T13:12:11.609030+0000","last_peered":"2026-02-23T13:12:11.609030+0000","last_clean":"2026-02-23T13:11:53.322095+0000","last_became_active":"2026-02-23T13:12:08.349412+0000","last_became_peered":"2026-02-23T13:12:08.349412+0000","last_unstale":"2026-02-23T13:12:11.609030+0000","last_undegraded":"2026-02-23T13:12:08.019799+0000","last_fullsized":"2026-02-23T13:12:07.999976+0000","mapping_epoch":135,"log_start":"129'1000","ondisk_log_start":"129'1000","created":126,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:11:36.450546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:11:36.450546+0000","last_clean_scrub_stamp":"2026-02-23T13:11:36.450546+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90570752,"num_objects":1382,"num_object_clones":0,"num_object_copies":4146,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":299,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1382,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11056,"num_write_kb":88448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":137,"num_bytes_recovered":8888320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[{"shards":"0,3","objects":1083}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.7","version":"140'11248","reported_seq":12074,"reported_epoch":140,"state":"active+undersized+degraded","last_fresh":"2026-02-23T13:12:14.991140+0000","last_change":"2026-02-23T13:12:14.704459+0000","last_active":"2026-02-23T13:12:14.991140+0000","last_peered":"2026-02-23T13:12:14.991140+0000","last_clean":"2026-02-23T13:11:53.320930+0000","last_became_active":"2026-02-23T13:12:13.325541+0000","last_became_peered":"2026-02-23T13:12:13.325541+0000","last_unstale":"2026-02-23T13:12:14.991140+0000","last_undegraded":"2026-02-23T13:12:13.323717+0000","last_fullsized":"2026-02-23T13:12:13.322544+0000","mapping_epoch":138,"log_start":"129'1200","ondisk_log_start":"129'1200","created":126,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:11:36.450546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:11:36.450546+0000","last_clean_scrub_stamp":"2026-02-23T13:11:36.450546+0000","objects_scrubbed":0,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T13:12:40.517403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92143616,"num_objects":1406,"num_object_clones":0,"num_object_copies":4218,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1406,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1406,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11248,"num_write_kb":89984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":388,"num_bytes_recovered":25378816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":1406}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.6","version":"140'11296","reported_seq":12216,"reported_epoch":140,"state":"active+undersized+degraded","last_fresh":"2026-02-23T13:12:15.403417+0000","last_change":"2026-02-23T13:12:15.402162+0000","last_active":"2026-02-23T13:12:15.403417+0000","last_peered":"2026-02-23T13:12:15.403417+0000","last_clean":"2026-02-23T13:11:53.287549+0000","last_became_active":"2026-02-23T13:12:13.326599+0000","last_became_peered":"2026-02-23T13:12:13.326599+0000","last_unstale":"2026-02-23T13:12:15.403417+0000","last_undegraded":"2026-02-23T13:12:13.323377+0000","last_fullsized":"2026-02-23T13:12:13.322881+0000","mapping_epoch":138,"log_start":"129'1200","ondisk_log_start":"129'1200","created":126,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:11:36.450546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:11:36.450546+0000","last_clean_scrub_stamp":"2026-02-23T13:11:36.450546+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T13:12:41.724138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92536832,"num_objects":1412,"num_object_clones":0,"num_object_copies":4236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1412,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1412,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11296,"num_write_kb":90368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":434,"num_bytes_recovered":28442624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,2","objects":1412}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"140'11160","reported_seq":11195,"reported_epoch":140,"state":"active+undersized+degraded","last_fresh":"2026-02-23T13:12:15.418276+0000","last_change":"2026-02-23T13:12:12.322755+0000","last_active":"2026-02-23T13:12:15.418276+0000","last_peered":"2026-02-23T13:12:15.418276+0000","last_clean":"2026-02-23T13:12:11.190891+0000","last_became_active":"2026-02-23T13:12:12.322755+0000","last_became_peered":"2026-02-23T13:12:12.322755+0000","last_unstale":"2026-02-23T13:12:15.418276+0000","last_undegraded":"2026-02-23T13:12:12.321637+0000","last_fullsized":"2026-02-23T13:12:12.321512+0000","mapping_epoch":137,"log_start":"129'1100","ondisk_log_start":"129'1100","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:11:36.450546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:11:36.450546+0000","last_clean_scrub_stamp":"2026-02-23T13:11:36.450546+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T13:12:49.466771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":91422720,"num_objects":1395,"num_object_clones":0,"num_object_copies":4185,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1395,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1395,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11160,"num_write_kb":89280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,3","objects":1395}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3","version":"140'11204","reported_seq":12113,"reported_epoch":140,"state":"active+clean","last_fresh":"2026-02-23T13:12:15.402798+0000","last_change":"2026-02-23T13:12:14.000576+0000","last_active":"2026-02-23T13:12:15.402798+0000","last_peered":"2026-02-23T13:12:15.402798+0000","last_clean":"2026-02-23T13:12:15.402798+0000","last_became_active":"2026-02-23T13:12:13.995043+0000","last_became_peered":"2026-02-23T13:12:13.995043+0000","last_unstale":"2026-02-23T13:12:15.402798+0000","last_undegraded":"2026-02-23T13:12:15.402798+0000","last_fullsized":"2026-02-23T13:12:15.402798+0000","mapping_epoch":139,"log_start":"129'1200","ondisk_log_start":"129'1200","created":126,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:11:36.450546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:11:36.450546+0000","last_clean_scrub_stamp":"2026-02-23T13:11:36.450546+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T13:12:38.075846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":91783168,"num_objects":1401,"num_object_clones":0,"num_object_copies":4203,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1401,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11204,"num_write_kb":89632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":432,"num_bytes_recovered":28311552,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"136'11334","reported_seq":11362,"reported_epoch":137,"state":"peering","last_fresh":"2026-02-23T13:12:11.320657+0000","last_change":"2026-02-23T13:12:11.320657+0000","last_active":"2026-02-23T13:12:11.320657+0000","last_peered":"2026-02-23T13:12:11.311607+0000","last_clean":"2026-02-23T13:12:11.311607+0000","last_became_active":"2026-02-23T13:11:37.459917+0000","last_became_peered":"2026-02-23T13:11:37.459917+0000","last_unstale":"2026-02-23T13:12:11.320657+0000","last_undegraded":"2026-02-23T13:12:11.320657+0000","last_fullsized":"2026-02-23T13:12:11.320657+0000","mapping_epoch":137,"log_start":"129'1300","ondisk_log_start":"129'1300","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:11:36.450546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:11:36.450546+0000","last_clean_scrub_stamp":"2026-02-23T13:11:36.450546+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92848128,"num_objects":1417,"num_object_clones":0,"num_object_copies":4251,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1417,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11334,"num_write_kb":90672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"140'11256","reported_seq":11297,"reported_epoch":140,"state":"active+clean","last_fresh":"2026-02-23T13:12:15.006401+0000","last_change":"2026-02-23T13:12:13.996820+0000","last_active":"2026-02-23T13:12:15.006401+0000","last_peered":"2026-02-23T13:12:15.006401+0000","last_clean":"2026-02-23T13:12:15.006401+0000","last_became_active":"2026-02-23T13:12:13.993163+0000","last_became_peered":"2026-02-23T13:12:13.993163+0000","last_unstale":"2026-02-23T13:12:15.006401+0000","last_undegraded":"2026-02-23T13:12:15.006401+0000","last_fullsized":"2026-02-23T13:12:15.006401+0000","mapping_epoch":139,"log_start":"129'1200","ondisk_log_start":"129'1200","created":126,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:11:36.450546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:11:36.450546+0000","last_clean_scrub_stamp":"2026-02-23T13:11:36.450546+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T13:12:37.344053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92209152,"num_objects":1407,"num_object_clones":0,"num_object_copies":4221,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1407,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11256,"num_write_kb":90048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":8192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.0","version":"140'11536","reported_seq":12471,"reported_epoch":140,"state":"active+clean","last_fresh":"2026-02-23T13:12:15.381931+0000","last_change":"2026-02-23T13:12:13.327494+0000","last_active":"2026-02-23T13:12:15.381931+0000","last_peered":"2026-02-23T13:12:15.381931+0000","last_clean":"2026-02-23T13:12:15.381931+0000","last_became_active":"2026-02-23T13:12:13.325079+0000","last_became_peered":"2026-02-23T13:12:13.325079+0000","last_unstale":"2026-02-23T13:12:15.381931+0000","last_undegraded":"2026-02-23T13:12:15.381931+0000","last_fullsized":"2026-02-23T13:12:15.381931+0000","mapping_epoch":138,"log_start":"129'1500","ondisk_log_start":"129'1500","created":126,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:11:36.450546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:11:36.450546+0000","last_clean_scrub_stamp":"2026-02-23T13:11:36.450546+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T13:12:39.220485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":94502912,"num_objects":1442,"num_object_clones":0,"num_object_copies":4326,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1442,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11536,"num_write_kb":92288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":438,"num_bytes_recovered":28704768,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,3],"acting":[0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.5","version":"137'10872","reported_seq":11630,"reported_epoch":137,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2026-02-23T13:12:11.609192+0000","last_change":"2026-02-23T13:12:08.383175+0000","last_active":"2026-02-23T13:12:11.609192+0000","last_peered":"2026-02-23T13:12:11.609192+0000","last_clean":"2026-02-23T13:11:53.290169+0000","last_became_active":"2026-02-23T13:12:08.349527+0000","last_became_peered":"2026-02-23T13:12:08.349527+0000","last_unstale":"2026-02-23T13:12:11.609192+0000","last_undegraded":"2026-02-23T13:12:08.018234+0000","last_fullsized":"2026-02-23T13:12:08.000209+0000","mapping_epoch":135,"log_start":"129'800","ondisk_log_start":"129'800","created":126,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:11:36.450546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:11:36.450546+0000","last_clean_scrub_stamp":"2026-02-23T13:11:36.450546+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":89063424,"num_objects":1359,"num_object_clones":0,"num_object_copies":4077,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":91,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1359,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10872,"num_write_kb":86976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":359,"num_bytes_recovered":23502848,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[{"shards":"0,3","objects":1268}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"28'99","reported_seq":284,"reported_epoch":137,"state":"active+clean","last_fresh":"2026-02-23T13:12:11.323936+0000","last_change":"2026-02-23T13:12:07.006552+0000","last_active":"2026-02-23T13:12:11.323936+0000","last_peered":"2026-02-23T13:12:11.323936+0000","last_clean":"2026-02-23T13:12:11.323936+0000","last_became_active":"2026-02-23T13:12:07.006352+0000","last_became_peered":"2026-02-23T13:12:07.006352+0000","last_unstale":"2026-02-23T13:12:11.323936+0000","last_undegraded":"2026-02-23T13:12:11.323936+0000","last_fullsized":"2026-02-23T13:12:11.323936+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'99","last_scrub_stamp":"2026-02-23T13:11:39.969814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:07:05.619208+0000","last_clean_scrub_stamp":"2026-02-23T13:11:39.969814+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T13:12:45.543132+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":122,"num_read_kb":228,"num_write":135,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":16,"stat_sum":{"num_bytes":1480630272,"num_objects":22595,"num_object_clones":0,"num_object_copies":67785,"num_objects_missing_on_primary":807,"num_objects_missing":807,"num_objects_degraded":12979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22595,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":180741,"num_write_kb":1445928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3593,"num_bytes_recovered":235151360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4308402176,"data_stored":4308402176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160741,"ondisk_log_size":160741,"up":38,"acting":34,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":122,"num_read_kb":228,"num_write":135,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":80,"seq":343597383709,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":734003200,"kb_used":1740768,"kb_used_data":1057996,"kb_used_omap":46,"kb_used_meta":682705,"kb_avail":732262432,"statfs":{"total":751619276800,"available":749836730368,"internally_reserved":0,"allocated":1083387904,"data_stored":1082415271,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":47471,"internal_metadata":699090577},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":134,"seq":575525617667,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":734003200,"kb_used":1308648,"kb_used_data":1206996,"kb_used_omap":13,"kb_used_meta":101618,"kb_avail":732694552,"statfs":{"total":751619276800,"available":750279221248,"internally_reserved":0,"allocated":1235963904,"data_stored":1234960864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13563,"internal_metadata":104057605},"hb_peers":[0,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,0,0,0,0,0,0,0,23],"upper_bound":2048},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":64,"seq":274877906979,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":734003200,"kb_used":1613900,"kb_used_data":842296,"kb_used_omap":53,"kb_used_meta":771530,"kb_avail":732389300,"statfs":{"total":751619276800,"available":749966643200,"internally_reserved":0,"allocated":862511104,"data_stored":861556523,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":54937,"internal_metadata":790047079},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":106,"seq":455266533392,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":734003200,"kb_used":1514616,"kb_used_data":1079716,"kb_used_omap":27,"kb_used_meta":434852,"kb_avail":732488584,"statfs":{"total":751619276800,"available":750068310016,"internally_reserved":0,"allocated":1105629184,"data_stored":1104626144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28520,"internal_metadata":445288600},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1119354880,"data_stored":1119354880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":860971008,"data_stored":860971008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1237245952,"data_stored":1237245952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1090830336,"data_stored":1090830336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-23T13:12:17.550 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 39 128 191187 191059 38.2696 30.0703 0.0999877 0.0261133 2026-02-23T13:12:17.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:12:17 trial068 ceph-mon[38805]: pgmap v261: 17 pgs: 3 active+recovering+undersized+degraded, 1 remapped+peering, 2 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 peering, 4 active+clean, 5 active+undersized+degraded; 1.4 GiB data, 5.9 GiB used, 2.7 TiB / 2.7 TiB avail; 22 MiB/s wr, 2.79k op/s; 12979/67791 objects degraded (19.146%); 45 MiB/s, 725 objects/s recovering 2026-02-23T13:12:17.954 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:12:18.090 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:12:18.208 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:12:18.206+0000 7f1eb722a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:12:18.208 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:12:18.206+0000 7f1eb722a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:12:18.221 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:12:18.222 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:12:18.222 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.1 2026-02-23T13:12:18.235 INFO:teuthology.orchestra.run.trial068.stdout:osd.2: {} 2026-02-23T13:12:18.236 INFO:teuthology.orchestra.run.trial068.stderr:osd.2: osd_enable_op_tracker = '' 2026-02-23T13:12:18.247 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:12:18.248 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:12:18.285 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:12:18.285 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:12:18.550 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T13:12:18.550491+0000 min lat: 0.0010091 max lat: 2.47028 avg lat: 0.0262493 2026-02-23T13:12:18.551 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T13:12:18.551 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 40 128 195057 194929 38.0687 30.2344 0.0232893 0.0262493 2026-02-23T13:12:18.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:12:18 trial068 ceph-mon[38805]: from='client.14681 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2026-02-23T13:12:18.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:12:18 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3858517995' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:12:19.550 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 41 128 198898 198770 37.872 30.0078 0.0103631 0.0263869 2026-02-23T13:12:19.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:12:19 trial068 ceph-mon[38805]: pgmap v262: 17 pgs: 4 active+recovering+undersized+degraded, 7 active+clean, 6 active+undersized+degraded; 1.4 GiB data, 5.9 GiB used, 2.7 TiB / 2.7 TiB avail; 14 MiB/s wr, 1.78k op/s; 15229/68943 objects degraded (22.089%); 45 MiB/s, 725 objects/s recovering 2026-02-23T13:12:19.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:12:19 trial068 ceph-mon[38805]: Health check update: Degraded data redundancy: 15229/68943 objects degraded (22.089%), 10 pgs degraded (PG_DEGRADED) 2026-02-23T13:12:20.287 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:12:20.416 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:12:20.529 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:12:20.527+0000 7f36a1bc6640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:12:20.529 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:12:20.528+0000 7f36a1bc6640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:12:20.544 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:12:20.544 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:12:20.544 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.1 2026-02-23T13:12:20.550 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 42 128 202645 202517 37.6672 29.2734 0.0246402 0.0265316 2026-02-23T13:12:20.559 INFO:teuthology.orchestra.run.trial068.stdout:osd.2: {} 2026-02-23T13:12:20.559 INFO:teuthology.orchestra.run.trial068.stderr:osd.2: osd_enable_op_tracker = '' 2026-02-23T13:12:20.572 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:12:20.572 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:12:20.611 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:12:20.611 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:12:21.551 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 43 128 206547 206419 37.5001 30.4844 0.0598422 0.0266534 2026-02-23T13:12:21.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:12:21 trial068 ceph-mon[38805]: pgmap v263: 17 pgs: 1 active+recovering+undersized+degraded, 7 active+clean, 9 active+undersized+degraded; 1.5 GiB data, 6.1 GiB used, 2.7 TiB / 2.7 TiB avail; 18 MiB/s wr, 2.27k op/s; 15191/71709 objects degraded (21.184%); 41 MiB/s, 649 objects/s recovering 2026-02-23T13:12:21.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:12:21 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1914752026' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:12:22.345 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [1] live_osds: [3, 0, 2] 2026-02-23T13:12:22.345 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2026-02-23T13:12:22.345 INFO:tasks.thrashosds.thrasher:Killing osd 2, live_osds are [3, 0, 2] 2026-02-23T13:12:22.345 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2026-02-23T13:12:22.346 DEBUG:teuthology.orchestra.run.trial185:> sudo systemctl stop ceph-5f823084-10b8-11f1-be80-d404e6e7d460@osd.2 2026-02-23T13:12:22.551 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 44 128 210153 210025 37.2881 28.1719 0.0337281 0.026805 2026-02-23T13:12:22.613 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:12:22.740 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:12:22.857 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:12:22 trial185 systemd[1]: Stopping Ceph osd.2 for 5f823084-10b8-11f1-be80-d404e6e7d460... 2026-02-23T13:12:22.857 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:12:22 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-2[73823]: 2026-02-23T13:12:22.468+0000 7f73829e5640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --osd-objectstore=bluestore (PID: 1) UID: 0 2026-02-23T13:12:22.857 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:12:22 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-2[73823]: 2026-02-23T13:12:22.468+0000 7f73829e5640 -1 osd.2 140 *** Got signal Terminated *** 2026-02-23T13:12:22.857 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:12:22 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-2[73823]: 2026-02-23T13:12:22.468+0000 7f73829e5640 -1 osd.2 140 *** Immediate shutdown (osd_fast_shutdown=true) *** 2026-02-23T13:12:22.859 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:12:22.857+0000 7fc9b5af4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:12:22.859 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:12:22.858+0000 7fc9b5af4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:12:22.873 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:12:22.874 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:12:22.874 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.1 2026-02-23T13:12:22.887 INFO:teuthology.orchestra.run.trial068.stdout:osd.2: {} 2026-02-23T13:12:22.888 INFO:teuthology.orchestra.run.trial068.stderr:osd.2: osd_enable_op_tracker = '' 2026-02-23T13:12:22.902 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:12:22.903 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:12:22.938 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:12:22.938 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:12:23.551 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 45 128 213318 213190 37.0086 24.7266 0.10976 0.0268724 2026-02-23T13:12:23.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:12:23 trial068 ceph-mon[38805]: pgmap v264: 17 pgs: 7 active+clean, 10 active+undersized+degraded; 1.5 GiB data, 6.4 GiB used, 2.7 TiB / 2.7 TiB avail; 26 MiB/s wr, 3.35k op/s; 15939/76167 objects degraded (20.926%); 36 MiB/s, 580 objects/s recovering 2026-02-23T13:12:23.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:12:23 trial068 ceph-mon[38805]: osd.2 marked itself down and dead 2026-02-23T13:12:23.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:12:23 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/178697974' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:12:23.731 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:12:23 trial185 podman[74554]: 2026-02-23 13:12:23.432846542 +0000 UTC m=+0.979916334 container died 8c29a9eb095cb1bd71e78d15ec0ff9260c98404be09bd1d6d932d1e25985d352 (image=quay.ceph.io/ceph-ci/ceph@sha256:49621faaa845c39b2aae72e4fa24123ee9b2ac3fb8ae834753907068ee2972ea, name=ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-2, OSD_FLAVOR=default, org.label-schema.name=CentOS Stream 9 Base Image, GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfsganesha-5/, org.opencontainers.image.documentation=https://docs.ceph.com/, org.label-schema.vendor=CentOS, CEPH_GIT_REPO=https://github.com/ceph/ceph-ci.git, org.opencontainers.image.authors=Ceph Release Team , org.label-schema.schema-version=1.0, CEPH_REF=wip-shraddhaag-enable-cephadm-tests, FROM_IMAGE=quay.io/centos/centos:stream9, io.buildah.version=1.41.3, org.label-schema.license=GPLv2, CEPH_SHA1=e371433fcd1390a0b82cfb18effc422c1f5d8c1d, org.label-schema.build-date=20260216, ceph=True) 2026-02-23T13:12:23.731 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:12:23 trial185 podman[74554]: 2026-02-23 13:12:23.441553132 +0000 UTC m=+0.988622913 container remove 8c29a9eb095cb1bd71e78d15ec0ff9260c98404be09bd1d6d932d1e25985d352 (image=quay.ceph.io/ceph-ci/ceph@sha256:49621faaa845c39b2aae72e4fa24123ee9b2ac3fb8ae834753907068ee2972ea, name=ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-2, ceph=True, FROM_IMAGE=quay.io/centos/centos:stream9, GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfsganesha-5/, org.label-schema.name=CentOS Stream 9 Base Image, OSD_FLAVOR=default, CEPH_GIT_REPO=https://github.com/ceph/ceph-ci.git, CEPH_SHA1=e371433fcd1390a0b82cfb18effc422c1f5d8c1d, CEPH_REF=wip-shraddhaag-enable-cephadm-tests, org.label-schema.vendor=CentOS, org.label-schema.build-date=20260216, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.opencontainers.image.authors=Ceph Release Team , io.buildah.version=1.41.3, org.opencontainers.image.documentation=https://docs.ceph.com/) 2026-02-23T13:12:23.732 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:12:23 trial185 bash[74554]: ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-2 2026-02-23T13:12:23.732 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:12:23 trial185 podman[74677]: 2026-02-23 13:12:23.572356934 +0000 UTC m=+0.016428082 container create 9acb38ec42d43d60a701cbc7280eef21ee6650ab034d46bcdb395e03cb5e7542 (image=quay.ceph.io/ceph-ci/ceph@sha256:49621faaa845c39b2aae72e4fa24123ee9b2ac3fb8ae834753907068ee2972ea, name=ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-2-deactivate, GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfsganesha-5/, io.buildah.version=1.41.3, org.label-schema.schema-version=1.0, OSD_FLAVOR=default, CEPH_REF=wip-shraddhaag-enable-cephadm-tests, FROM_IMAGE=quay.io/centos/centos:stream9, org.opencontainers.image.authors=Ceph Release Team , CEPH_SHA1=e371433fcd1390a0b82cfb18effc422c1f5d8c1d, org.label-schema.license=GPLv2, org.opencontainers.image.documentation=https://docs.ceph.com/, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20260216, org.label-schema.vendor=CentOS, CEPH_GIT_REPO=https://github.com/ceph/ceph-ci.git, ceph=True) 2026-02-23T13:12:23.732 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:12:23 trial185 podman[74677]: 2026-02-23 13:12:23.601526411 +0000 UTC m=+0.045597569 container init 9acb38ec42d43d60a701cbc7280eef21ee6650ab034d46bcdb395e03cb5e7542 (image=quay.ceph.io/ceph-ci/ceph@sha256:49621faaa845c39b2aae72e4fa24123ee9b2ac3fb8ae834753907068ee2972ea, name=ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-2-deactivate, CEPH_SHA1=e371433fcd1390a0b82cfb18effc422c1f5d8c1d, org.label-schema.schema-version=1.0, FROM_IMAGE=quay.io/centos/centos:stream9, OSD_FLAVOR=default, CEPH_REF=wip-shraddhaag-enable-cephadm-tests, GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfsganesha-5/, org.label-schema.build-date=20260216, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, org.opencontainers.image.documentation=https://docs.ceph.com/, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.41.3, org.opencontainers.image.authors=Ceph Release Team , CEPH_GIT_REPO=https://github.com/ceph/ceph-ci.git) 2026-02-23T13:12:23.732 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:12:23 trial185 podman[74677]: 2026-02-23 13:12:23.603871099 +0000 UTC m=+0.047942267 container start 9acb38ec42d43d60a701cbc7280eef21ee6650ab034d46bcdb395e03cb5e7542 (image=quay.ceph.io/ceph-ci/ceph@sha256:49621faaa845c39b2aae72e4fa24123ee9b2ac3fb8ae834753907068ee2972ea, name=ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-2-deactivate, GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfsganesha-5/, org.label-schema.build-date=20260216, io.buildah.version=1.41.3, CEPH_SHA1=e371433fcd1390a0b82cfb18effc422c1f5d8c1d, CEPH_REF=wip-shraddhaag-enable-cephadm-tests, ceph=True, org.label-schema.schema-version=1.0, OSD_FLAVOR=default, FROM_IMAGE=quay.io/centos/centos:stream9, org.label-schema.license=GPLv2, org.opencontainers.image.documentation=https://docs.ceph.com/, org.label-schema.vendor=CentOS, org.opencontainers.image.authors=Ceph Release Team , CEPH_GIT_REPO=https://github.com/ceph/ceph-ci.git, org.label-schema.name=CentOS Stream 9 Base Image) 2026-02-23T13:12:23.732 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:12:23 trial185 podman[74677]: 2026-02-23 13:12:23.604441829 +0000 UTC m=+0.048512977 container attach 9acb38ec42d43d60a701cbc7280eef21ee6650ab034d46bcdb395e03cb5e7542 (image=quay.ceph.io/ceph-ci/ceph@sha256:49621faaa845c39b2aae72e4fa24123ee9b2ac3fb8ae834753907068ee2972ea, name=ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-2-deactivate, org.opencontainers.image.documentation=https://docs.ceph.com/, org.label-schema.vendor=CentOS, CEPH_GIT_REPO=https://github.com/ceph/ceph-ci.git, FROM_IMAGE=quay.io/centos/centos:stream9, GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfsganesha-5/, org.label-schema.build-date=20260216, io.buildah.version=1.41.3, org.opencontainers.image.authors=Ceph Release Team , CEPH_REF=wip-shraddhaag-enable-cephadm-tests, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, OSD_FLAVOR=default, CEPH_SHA1=e371433fcd1390a0b82cfb18effc422c1f5d8c1d, ceph=True, org.label-schema.license=GPLv2) 2026-02-23T13:12:23.732 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:12:23 trial185 podman[74677]: 2026-02-23 13:12:23.56595662 +0000 UTC m=+0.010027778 image pull 63df4b83db6f95a8ba965919b3671cb27173117c23b5c950aaadc8913f56213c quay.ceph.io/ceph-ci/ceph@sha256:49621faaa845c39b2aae72e4fa24123ee9b2ac3fb8ae834753907068ee2972ea 2026-02-23T13:12:23.732 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:12:23 trial185 podman[74677]: 2026-02-23 13:12:23.729436356 +0000 UTC m=+0.173507514 container died 9acb38ec42d43d60a701cbc7280eef21ee6650ab034d46bcdb395e03cb5e7542 (image=quay.ceph.io/ceph-ci/ceph@sha256:49621faaa845c39b2aae72e4fa24123ee9b2ac3fb8ae834753907068ee2972ea, name=ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-2-deactivate, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.opencontainers.image.documentation=https://docs.ceph.com/, org.label-schema.build-date=20260216, io.buildah.version=1.41.3, CEPH_REF=wip-shraddhaag-enable-cephadm-tests, FROM_IMAGE=quay.io/centos/centos:stream9, org.label-schema.name=CentOS Stream 9 Base Image, org.opencontainers.image.authors=Ceph Release Team , CEPH_GIT_REPO=https://github.com/ceph/ceph-ci.git, ceph=True, GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfsganesha-5/, org.label-schema.vendor=CentOS, OSD_FLAVOR=default, CEPH_SHA1=e371433fcd1390a0b82cfb18effc422c1f5d8c1d) 2026-02-23T13:12:23.769 DEBUG:teuthology.orchestra.run.trial185:> sudo pkill -f 'journalctl -f -n 0 -u ceph-5f823084-10b8-11f1-be80-d404e6e7d460@osd.2.service' 2026-02-23T13:12:23.805 DEBUG:teuthology.orchestra.run:got remote process result: None 2026-02-23T13:12:23.805 INFO:tasks.cephadm.osd.2:Stopped osd.2 2026-02-23T13:12:24.107 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:12:23 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:12:23.974Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:12:24.333 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:12:23 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:12:23] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:12:24.551 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 46 128 213402 213274 36.2184 0.65625 0.00970422 0.0270304 2026-02-23T13:12:24.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:12:24 trial068 ceph-mon[38805]: Health check update: 2 osds down (OSD_DOWN) 2026-02-23T13:12:24.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:12:24 trial068 ceph-mon[38805]: osdmap e141: 4 total, 2 up, 4 in 2026-02-23T13:12:24.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:12:24 trial068 ceph-mon[38805]: Health check update: Degraded data redundancy: 15939/76167 objects degraded (20.926%), 10 pgs degraded (PG_DEGRADED) 2026-02-23T13:12:24.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:12:24 trial068 ceph-mon[38805]: osdmap e142: 4 total, 2 up, 4 in 2026-02-23T13:12:24.939 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:12:25.055 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:12:25.159 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:12:25.157+0000 7fca35da0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:12:25.159 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:12:25.157+0000 7fca35da0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:12:25.171 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:12:25.171 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:12:25.172 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.1 2026-02-23T13:12:25.172 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:12:25.183 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:12:25.184 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:12:25.211 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:12:25.212 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:12:25.551 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 47 128 213402 213274 35.4478 0 - 0.0270304 2026-02-23T13:12:25.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:12:25 trial068 ceph-mon[38805]: pgmap v267: 17 pgs: 5 stale+active+undersized+degraded, 1 stale+active+clean, 6 active+clean, 5 active+undersized+degraded; 1.5 GiB data, 6.4 GiB used, 2.7 TiB / 2.7 TiB avail; 22 MiB/s wr, 2.79k op/s; 15939/76167 objects degraded (20.926%); 16 MiB/s, 263 objects/s recovering 2026-02-23T13:12:25.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:12:25 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2141588687' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:12:26.551 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 48 128 213402 213274 34.7093 0 - 0.0270304 2026-02-23T13:12:27.212 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:12:27.329 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:12:27.434 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:12:27.432+0000 7f8308aad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:12:27.434 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:12:27.432+0000 7f8308aad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:12:27.446 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:12:27.446 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:12:27.447 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.1 2026-02-23T13:12:27.447 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:12:27.458 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:12:27.458 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:12:27.486 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:12:27.486 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:12:27.551 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 49 128 213402 213274 34.001 0 - 0.0270304 2026-02-23T13:12:27.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:12:27 trial068 ceph-mon[38805]: pgmap v268: 17 pgs: 3 stale+active+undersized+degraded, 1 stale+active+clean, 4 undersized+degraded+peered, 4 active+clean, 5 active+undersized+degraded; 1.6 GiB data, 6.5 GiB used, 2.7 TiB / 2.7 TiB avail; 24 MiB/s wr, 3.02k op/s; 26369/78009 objects degraded (33.803%); 6.6 MiB/s, 105 objects/s recovering 2026-02-23T13:12:28.551 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 50 128 213402 213274 33.321 0 - 0.0270304 2026-02-23T13:12:28.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:12:28 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1425660584' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:12:28.806 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [1, 2] live_osds: [3, 0] 2026-02-23T13:12:28.806 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2026-02-23T13:12:28.806 INFO:tasks.thrashosds.thrasher:Reviving osd 1 2026-02-23T13:12:28.806 INFO:tasks.cephadm.osd.1:Restarting osd.1 (starting--it wasn't running)... 2026-02-23T13:12:28.807 DEBUG:teuthology.orchestra.run.trial068:osd.1> sudo journalctl -f -n 0 -u ceph-5f823084-10b8-11f1-be80-d404e6e7d460@osd.1.service 2026-02-23T13:12:28.810 DEBUG:teuthology.orchestra.run.trial068:> sudo systemctl start ceph-5f823084-10b8-11f1-be80-d404e6e7d460@osd.1 2026-02-23T13:12:29.085 INFO:journalctl@ceph.osd.1.trial068.stdout:Feb 23 13:12:28 trial068 systemd[1]: Starting Ceph osd.1 for 5f823084-10b8-11f1-be80-d404e6e7d460... 2026-02-23T13:12:29.085 INFO:journalctl@ceph.osd.1.trial068.stdout:Feb 23 13:12:28 trial068 podman[99555]: 2026-02-23 13:12:28.974960695 +0000 UTC m=+0.015632592 container create 8327ac2e2619434c9fd53b86e882a555feecc65a9946098a4d146115d3a6f088 (image=quay.ceph.io/ceph-ci/ceph@sha256:49621faaa845c39b2aae72e4fa24123ee9b2ac3fb8ae834753907068ee2972ea, name=ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-1-activate, CEPH_REF=wip-shraddhaag-enable-cephadm-tests, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20260216, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.opencontainers.image.authors=Ceph Release Team , org.label-schema.vendor=CentOS, org.opencontainers.image.documentation=https://docs.ceph.com/, CEPH_GIT_REPO=https://github.com/ceph/ceph-ci.git, OSD_FLAVOR=default, CEPH_SHA1=e371433fcd1390a0b82cfb18effc422c1f5d8c1d, FROM_IMAGE=quay.io/centos/centos:stream9, io.buildah.version=1.41.3, GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfsganesha-5/, ceph=True) 2026-02-23T13:12:29.085 INFO:journalctl@ceph.osd.1.trial068.stdout:Feb 23 13:12:29 trial068 podman[99555]: 2026-02-23 13:12:29.012009955 +0000 UTC m=+0.052681862 container init 8327ac2e2619434c9fd53b86e882a555feecc65a9946098a4d146115d3a6f088 (image=quay.ceph.io/ceph-ci/ceph@sha256:49621faaa845c39b2aae72e4fa24123ee9b2ac3fb8ae834753907068ee2972ea, name=ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-1-activate, CEPH_GIT_REPO=https://github.com/ceph/ceph-ci.git, GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfsganesha-5/, CEPH_SHA1=e371433fcd1390a0b82cfb18effc422c1f5d8c1d, CEPH_REF=wip-shraddhaag-enable-cephadm-tests, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, org.opencontainers.image.authors=Ceph Release Team , org.opencontainers.image.documentation=https://docs.ceph.com/, ceph=True, org.label-schema.build-date=20260216, FROM_IMAGE=quay.io/centos/centos:stream9, OSD_FLAVOR=default, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, io.buildah.version=1.41.3) 2026-02-23T13:12:29.085 INFO:journalctl@ceph.osd.1.trial068.stdout:Feb 23 13:12:29 trial068 podman[99555]: 2026-02-23 13:12:29.01437065 +0000 UTC m=+0.055042557 container start 8327ac2e2619434c9fd53b86e882a555feecc65a9946098a4d146115d3a6f088 (image=quay.ceph.io/ceph-ci/ceph@sha256:49621faaa845c39b2aae72e4fa24123ee9b2ac3fb8ae834753907068ee2972ea, name=ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-1-activate, org.label-schema.schema-version=1.0, OSD_FLAVOR=default, io.buildah.version=1.41.3, org.label-schema.vendor=CentOS, GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfsganesha-5/, CEPH_SHA1=e371433fcd1390a0b82cfb18effc422c1f5d8c1d, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, org.opencontainers.image.authors=Ceph Release Team , CEPH_GIT_REPO=https://github.com/ceph/ceph-ci.git, org.opencontainers.image.documentation=https://docs.ceph.com/, ceph=True, CEPH_REF=wip-shraddhaag-enable-cephadm-tests, org.label-schema.build-date=20260216, FROM_IMAGE=quay.io/centos/centos:stream9) 2026-02-23T13:12:29.085 INFO:journalctl@ceph.osd.1.trial068.stdout:Feb 23 13:12:29 trial068 podman[99555]: 2026-02-23 13:12:29.014940581 +0000 UTC m=+0.055612478 container attach 8327ac2e2619434c9fd53b86e882a555feecc65a9946098a4d146115d3a6f088 (image=quay.ceph.io/ceph-ci/ceph@sha256:49621faaa845c39b2aae72e4fa24123ee9b2ac3fb8ae834753907068ee2972ea, name=ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-1-activate, FROM_IMAGE=quay.io/centos/centos:stream9, org.label-schema.name=CentOS Stream 9 Base Image, OSD_FLAVOR=default, org.label-schema.vendor=CentOS, CEPH_SHA1=e371433fcd1390a0b82cfb18effc422c1f5d8c1d, org.label-schema.build-date=20260216, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.41.3, org.opencontainers.image.authors=Ceph Release Team , CEPH_GIT_REPO=https://github.com/ceph/ceph-ci.git, org.opencontainers.image.documentation=https://docs.ceph.com/, GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfsganesha-5/, CEPH_REF=wip-shraddhaag-enable-cephadm-tests, ceph=True) 2026-02-23T13:12:29.085 INFO:journalctl@ceph.osd.1.trial068.stdout:Feb 23 13:12:29 trial068 podman[99555]: 2026-02-23 13:12:28.968734427 +0000 UTC m=+0.009406344 image pull 63df4b83db6f95a8ba965919b3671cb27173117c23b5c950aaadc8913f56213c quay.ceph.io/ceph-ci/ceph@sha256:49621faaa845c39b2aae72e4fa24123ee9b2ac3fb8ae834753907068ee2972ea 2026-02-23T13:12:29.086 INFO:journalctl@ceph.osd.1.trial068.stdout:Feb 23 13:12:29 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-1-activate[99580]: Running command: /usr/bin/ceph-authtool --gen-print-key 2026-02-23T13:12:29.086 INFO:journalctl@ceph.osd.1.trial068.stdout:Feb 23 13:12:29 trial068 bash[99555]: Running command: /usr/bin/ceph-authtool --gen-print-key 2026-02-23T13:12:29.339 INFO:journalctl@ceph.osd.1.trial068.stdout:Feb 23 13:12:29 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-1-activate[99580]: Running command: /usr/bin/ceph-authtool --gen-print-key 2026-02-23T13:12:29.339 INFO:journalctl@ceph.osd.1.trial068.stdout:Feb 23 13:12:29 trial068 bash[99555]: Running command: /usr/bin/ceph-authtool --gen-print-key 2026-02-23T13:12:29.487 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:12:29.551 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 51 128 213402 213274 32.6677 0 - 0.0270304 2026-02-23T13:12:29.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:12:29 trial068 ceph-mon[38805]: pgmap v269: 17 pgs: 8 undersized+degraded+peered, 9 active+undersized+degraded; 1.6 GiB data, 6.6 GiB used, 2.7 TiB / 2.7 TiB avail; 22 MiB/s wr, 2.76k op/s; 40092/79998 objects degraded (50.116%); 268 KiB/s, 4 objects/s recovering 2026-02-23T13:12:29.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:12:29 trial068 ceph-mon[38805]: Health check update: Degraded data redundancy: 40092/79998 objects degraded (50.116%), 17 pgs degraded (PG_DEGRADED) 2026-02-23T13:12:29.606 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:12:29.723 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:12:29.721+0000 7ff16315d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:12:29.723 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:12:29.721+0000 7ff16315d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:12:29.735 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:12:29.736 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:12:29.736 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.1 2026-02-23T13:12:29.736 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:12:29.747 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:12:29.747 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:12:29.776 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:12:29.777 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:12:29.900 INFO:journalctl@ceph.osd.1.trial068.stdout:Feb 23 13:12:29 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-1-activate[99580]: --> Failed to activate via raw: did not find any matching OSD to activate 2026-02-23T13:12:29.900 INFO:journalctl@ceph.osd.1.trial068.stdout:Feb 23 13:12:29 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-1-activate[99580]: Running command: /usr/bin/ceph-authtool --gen-print-key 2026-02-23T13:12:29.900 INFO:journalctl@ceph.osd.1.trial068.stdout:Feb 23 13:12:29 trial068 bash[99555]: --> Failed to activate via raw: did not find any matching OSD to activate 2026-02-23T13:12:29.900 INFO:journalctl@ceph.osd.1.trial068.stdout:Feb 23 13:12:29 trial068 bash[99555]: Running command: /usr/bin/ceph-authtool --gen-print-key 2026-02-23T13:12:29.900 INFO:journalctl@ceph.osd.1.trial068.stdout:Feb 23 13:12:29 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-1-activate[99580]: Running command: /usr/bin/ceph-authtool --gen-print-key 2026-02-23T13:12:29.900 INFO:journalctl@ceph.osd.1.trial068.stdout:Feb 23 13:12:29 trial068 bash[99555]: Running command: /usr/bin/ceph-authtool --gen-print-key 2026-02-23T13:12:29.900 INFO:journalctl@ceph.osd.1.trial068.stdout:Feb 23 13:12:29 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-1-activate[99580]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2026-02-23T13:12:29.900 INFO:journalctl@ceph.osd.1.trial068.stdout:Feb 23 13:12:29 trial068 bash[99555]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2026-02-23T13:12:29.901 INFO:journalctl@ceph.osd.1.trial068.stdout:Feb 23 13:12:29 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-1-activate[99580]: Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/vg_nvme/lv_3 --path /var/lib/ceph/osd/ceph-1 --no-mon-config 2026-02-23T13:12:29.901 INFO:journalctl@ceph.osd.1.trial068.stdout:Feb 23 13:12:29 trial068 bash[99555]: Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/vg_nvme/lv_3 --path /var/lib/ceph/osd/ceph-1 --no-mon-config 2026-02-23T13:12:29.901 INFO:journalctl@ceph.osd.1.trial068.stdout:Feb 23 13:12:29 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-1-activate[99580]: Running command: /usr/bin/ln -snf /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-1/block 2026-02-23T13:12:29.901 INFO:journalctl@ceph.osd.1.trial068.stdout:Feb 23 13:12:29 trial068 bash[99555]: Running command: /usr/bin/ln -snf /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-1/block 2026-02-23T13:12:29.901 INFO:journalctl@ceph.osd.1.trial068.stdout:Feb 23 13:12:29 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-1-activate[99580]: Running command: /usr/bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-1/block 2026-02-23T13:12:29.901 INFO:journalctl@ceph.osd.1.trial068.stdout:Feb 23 13:12:29 trial068 bash[99555]: Running command: /usr/bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-1/block 2026-02-23T13:12:29.901 INFO:journalctl@ceph.osd.1.trial068.stdout:Feb 23 13:12:29 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-1-activate[99580]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2026-02-23T13:12:29.901 INFO:journalctl@ceph.osd.1.trial068.stdout:Feb 23 13:12:29 trial068 bash[99555]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2026-02-23T13:12:29.901 INFO:journalctl@ceph.osd.1.trial068.stdout:Feb 23 13:12:29 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-1-activate[99580]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2026-02-23T13:12:29.901 INFO:journalctl@ceph.osd.1.trial068.stdout:Feb 23 13:12:29 trial068 bash[99555]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2026-02-23T13:12:29.901 INFO:journalctl@ceph.osd.1.trial068.stdout:Feb 23 13:12:29 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-1-activate[99580]: --> ceph-volume lvm activate successful for osd ID: 1 2026-02-23T13:12:29.901 INFO:journalctl@ceph.osd.1.trial068.stdout:Feb 23 13:12:29 trial068 bash[99555]: --> ceph-volume lvm activate successful for osd ID: 1 2026-02-23T13:12:29.901 INFO:journalctl@ceph.osd.1.trial068.stdout:Feb 23 13:12:29 trial068 podman[99555]: 2026-02-23 13:12:29.789190491 +0000 UTC m=+0.829862398 container died 8327ac2e2619434c9fd53b86e882a555feecc65a9946098a4d146115d3a6f088 (image=quay.ceph.io/ceph-ci/ceph@sha256:49621faaa845c39b2aae72e4fa24123ee9b2ac3fb8ae834753907068ee2972ea, name=ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-1-activate, FROM_IMAGE=quay.io/centos/centos:stream9, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_GIT_REPO=https://github.com/ceph/ceph-ci.git, GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfsganesha-5/, CEPH_SHA1=e371433fcd1390a0b82cfb18effc422c1f5d8c1d, org.label-schema.build-date=20260216, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.opencontainers.image.documentation=https://docs.ceph.com/, ceph=True, io.buildah.version=1.41.3, org.opencontainers.image.authors=Ceph Release Team , OSD_FLAVOR=default, CEPH_REF=wip-shraddhaag-enable-cephadm-tests) 2026-02-23T13:12:29.901 INFO:journalctl@ceph.osd.1.trial068.stdout:Feb 23 13:12:29 trial068 podman[99555]: 2026-02-23 13:12:29.80064915 +0000 UTC m=+0.841321057 container remove 8327ac2e2619434c9fd53b86e882a555feecc65a9946098a4d146115d3a6f088 (image=quay.ceph.io/ceph-ci/ceph@sha256:49621faaa845c39b2aae72e4fa24123ee9b2ac3fb8ae834753907068ee2972ea, name=ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-1-activate, ceph=True, org.label-schema.schema-version=1.0, org.opencontainers.image.documentation=https://docs.ceph.com/, CEPH_GIT_REPO=https://github.com/ceph/ceph-ci.git, CEPH_SHA1=e371433fcd1390a0b82cfb18effc422c1f5d8c1d, org.label-schema.name=CentOS Stream 9 Base Image, org.opencontainers.image.authors=Ceph Release Team , org.label-schema.license=GPLv2, GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfsganesha-5/, FROM_IMAGE=quay.io/centos/centos:stream9, org.label-schema.vendor=CentOS, org.label-schema.build-date=20260216, io.buildah.version=1.41.3, CEPH_REF=wip-shraddhaag-enable-cephadm-tests, OSD_FLAVOR=default) 2026-02-23T13:12:29.901 INFO:journalctl@ceph.osd.1.trial068.stdout:Feb 23 13:12:29 trial068 podman[99927]: 2026-02-23 13:12:29.876245365 +0000 UTC m=+0.015298595 container create e32560e4f15a411ce6f25e4db870081696bda35865d4c9993d0fa174fdef5a0a (image=quay.ceph.io/ceph-ci/ceph@sha256:49621faaa845c39b2aae72e4fa24123ee9b2ac3fb8ae834753907068ee2972ea, name=ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-1, org.label-schema.license=GPLv2, org.opencontainers.image.documentation=https://docs.ceph.com/, io.buildah.version=1.41.3, org.label-schema.build-date=20260216, CEPH_GIT_REPO=https://github.com/ceph/ceph-ci.git, org.opencontainers.image.authors=Ceph Release Team , org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, CEPH_SHA1=e371433fcd1390a0b82cfb18effc422c1f5d8c1d, OSD_FLAVOR=default, org.label-schema.vendor=CentOS, FROM_IMAGE=quay.io/centos/centos:stream9, GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfsganesha-5/, org.label-schema.schema-version=1.0, CEPH_REF=wip-shraddhaag-enable-cephadm-tests) 2026-02-23T13:12:29.909 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph daemon osd.1 dump_ops_in_flight 2026-02-23T13:12:30.031 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:12:30.149 INFO:teuthology.orchestra.run.trial068.stderr:no valid command found; 10 closest matches: 2026-02-23T13:12:30.149 INFO:teuthology.orchestra.run.trial068.stderr:0 2026-02-23T13:12:30.149 INFO:teuthology.orchestra.run.trial068.stderr:1 2026-02-23T13:12:30.149 INFO:teuthology.orchestra.run.trial068.stderr:2 2026-02-23T13:12:30.149 INFO:teuthology.orchestra.run.trial068.stderr:abort 2026-02-23T13:12:30.149 INFO:teuthology.orchestra.run.trial068.stderr:assert 2026-02-23T13:12:30.149 INFO:teuthology.orchestra.run.trial068.stderr:bluestore collections 2026-02-23T13:12:30.149 INFO:teuthology.orchestra.run.trial068.stderr:bluestore compression stats [] 2026-02-23T13:12:30.149 INFO:teuthology.orchestra.run.trial068.stderr:bluestore list [] [] 2026-02-23T13:12:30.149 INFO:teuthology.orchestra.run.trial068.stderr:bluestore onode metadata 2026-02-23T13:12:30.149 INFO:teuthology.orchestra.run.trial068.stderr:bluestore show sharding 2026-02-23T13:12:30.149 INFO:teuthology.orchestra.run.trial068.stderr:admin_socket: invalid command 2026-02-23T13:12:30.157 INFO:journalctl@ceph.osd.1.trial068.stdout:Feb 23 13:12:29 trial068 podman[99927]: 2026-02-23 13:12:29.900584625 +0000 UTC m=+0.039637845 container init e32560e4f15a411ce6f25e4db870081696bda35865d4c9993d0fa174fdef5a0a (image=quay.ceph.io/ceph-ci/ceph@sha256:49621faaa845c39b2aae72e4fa24123ee9b2ac3fb8ae834753907068ee2972ea, name=ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-1, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, OSD_FLAVOR=default, CEPH_GIT_REPO=https://github.com/ceph/ceph-ci.git, org.label-schema.name=CentOS Stream 9 Base Image, org.opencontainers.image.documentation=https://docs.ceph.com/, ceph=True, CEPH_REF=wip-shraddhaag-enable-cephadm-tests, FROM_IMAGE=quay.io/centos/centos:stream9, io.buildah.version=1.41.3, org.label-schema.build-date=20260216, org.opencontainers.image.authors=Ceph Release Team , GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfsganesha-5/, CEPH_SHA1=e371433fcd1390a0b82cfb18effc422c1f5d8c1d) 2026-02-23T13:12:30.157 INFO:journalctl@ceph.osd.1.trial068.stdout:Feb 23 13:12:29 trial068 podman[99927]: 2026-02-23 13:12:29.903312671 +0000 UTC m=+0.042365901 container start e32560e4f15a411ce6f25e4db870081696bda35865d4c9993d0fa174fdef5a0a (image=quay.ceph.io/ceph-ci/ceph@sha256:49621faaa845c39b2aae72e4fa24123ee9b2ac3fb8ae834753907068ee2972ea, name=ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-1, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, FROM_IMAGE=quay.io/centos/centos:stream9, org.opencontainers.image.documentation=https://docs.ceph.com/, CEPH_GIT_REPO=https://github.com/ceph/ceph-ci.git, org.opencontainers.image.authors=Ceph Release Team , GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfsganesha-5/, OSD_FLAVOR=default, org.label-schema.schema-version=1.0, CEPH_REF=wip-shraddhaag-enable-cephadm-tests, io.buildah.version=1.41.3, org.label-schema.build-date=20260216, ceph=True, CEPH_SHA1=e371433fcd1390a0b82cfb18effc422c1f5d8c1d) 2026-02-23T13:12:30.158 INFO:journalctl@ceph.osd.1.trial068.stdout:Feb 23 13:12:29 trial068 bash[99927]: e32560e4f15a411ce6f25e4db870081696bda35865d4c9993d0fa174fdef5a0a 2026-02-23T13:12:30.158 INFO:journalctl@ceph.osd.1.trial068.stdout:Feb 23 13:12:29 trial068 podman[99927]: 2026-02-23 13:12:29.870101782 +0000 UTC m=+0.009155012 image pull 63df4b83db6f95a8ba965919b3671cb27173117c23b5c950aaadc8913f56213c quay.ceph.io/ceph-ci/ceph@sha256:49621faaa845c39b2aae72e4fa24123ee9b2ac3fb8ae834753907068ee2972ea 2026-02-23T13:12:30.158 INFO:journalctl@ceph.osd.1.trial068.stdout:Feb 23 13:12:29 trial068 systemd[1]: Started Ceph osd.1 for 5f823084-10b8-11f1-be80-d404e6e7d460. 2026-02-23T13:12:30.158 INFO:journalctl@ceph.osd.1.trial068.stdout:Feb 23 13:12:29 trial068 ceph-osd[99956]: -- 10.20.193.68:0/1146350090 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x4221c00 msgr2=0x505e000 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).process 2026-02-23T13:12:30.158 INFO:journalctl@ceph.osd.1.trial068.stdout:Feb 23 13:12:29 trial068 ceph-osd[99956]: -- 10.20.193.68:0/1146350090 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x4221c00 msgr2=0x505e000 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read continue len=32 2026-02-23T13:12:30.158 INFO:journalctl@ceph.osd.1.trial068.stdout:Feb 23 13:12:29 trial068 ceph-osd[99956]: --2- 10.20.193.68:0/1146350090 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x4221c00 0x505e000 crc :-1 s=READY pgs=602 gs=1 cs=0 l=1 c_cookie=7137512895559948888 s_cookie=0 reconnecting=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main r=0 2026-02-23T13:12:30.158 INFO:journalctl@ceph.osd.1.trial068.stdout:Feb 23 13:12:29 trial068 ceph-osd[99956]: --2- 10.20.193.68:0/1146350090 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x4221c00 0x505e000 crc :-1 s=THROTTLE_MESSAGE pgs=602 gs=1 cs=0 l=1 c_cookie=7137512895559948888 s_cookie=0 reconnecting=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).throttle_message 2026-02-23T13:12:30.158 INFO:journalctl@ceph.osd.1.trial068.stdout:Feb 23 13:12:29 trial068 ceph-osd[99956]: --2- 10.20.193.68:0/1146350090 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x4221c00 0x505e000 crc :-1 s=THROTTLE_BYTES pgs=602 gs=1 cs=0 l=1 c_cookie=7137512895559948888 s_cookie=0 reconnecting=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).throttle_bytes 2026-02-23T13:12:30.158 INFO:journalctl@ceph.osd.1.trial068.stdout:Feb 23 13:12:29 trial068 ceph-osd[99956]: --2- 10.20.193.68:0/1146350090 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x4221c00 0x505e000 crc :-1 s=THROTTLE_DISPATCH_QUEUE pgs=602 gs=1 cs=0 l=1 c_cookie=7137512895559948888 s_cookie=0 reconnecting=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).throttle_dispatch_queue 2026-02-23T13:12:30.158 INFO:journalctl@ceph.osd.1.trial068.stdout:Feb 23 13:12:29 trial068 ceph-osd[99956]: --2- 10.20.193.68:0/1146350090 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x4221c00 0x505e000 crc :-1 s=THROTTLE_DONE pgs=602 gs=1 cs=0 l=1 c_cookie=7137512895559948888 s_cookie=0 reconnecting=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).read_frame_segment seg_idx=0 2026-02-23T13:12:30.158 INFO:journalctl@ceph.osd.1.trial068.stdout:Feb 23 13:12:29 trial068 ceph-osd[99956]: -- 10.20.193.68:0/1146350090 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x4221c00 msgr2=0x505e000 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read start len=45 2026-02-23T13:12:30.158 INFO:journalctl@ceph.osd.1.trial068.stdout:Feb 23 13:12:29 trial068 ceph-osd[99956]: --2- 10.20.193.68:0/1146350090 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x4221c00 0x505e000 crc :-1 s=THROTTLE_DONE pgs=602 gs=1 cs=0 l=1 c_cookie=7137512895559948888 s_cookie=0 reconnecting=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_segment r=0 2026-02-23T13:12:30.159 INFO:journalctl@ceph.osd.1.trial068.stdout:Feb 23 13:12:29 trial068 ceph-osd[99956]: --2- 10.20.193.68:0/1146350090 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x4221c00 0x505e000 crc :-1 s=THROTTLE_DONE pgs=602 gs=1 cs=0 l=1 c_cookie=7137512895559948888 s_cookie=0 reconnecting=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).read_frame_segment seg_idx=1 2026-02-23T13:12:30.159 INFO:journalctl@ceph.osd.1.trial068.stdout:Feb 23 13:12:29 trial068 ceph-osd[99956]: -- 10.20.193.68:0/1146350090 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x4221c00 msgr2=0x505e000 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read start len=194 2026-02-23T13:12:30.159 INFO:journalctl@ceph.osd.1.trial068.stdout:Feb 23 13:12:29 trial068 ceph-osd[99956]: --2- 10.20.193.68:0/1146350090 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x4221c00 0x505e000 crc :-1 s=THROTTLE_DONE pgs=602 gs=1 cs=0 l=1 c_cookie=7137512895559948888 s_cookie=0 reconnecting=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_segment r=0 2026-02-23T13:12:30.159 INFO:journalctl@ceph.osd.1.trial068.stdout:Feb 23 13:12:29 trial068 ceph-osd[99956]: -- 10.20.193.68:0/1146350090 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x4221c00 msgr2=0x505e000 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read start len=13 2026-02-23T13:12:30.159 INFO:journalctl@ceph.osd.1.trial068.stdout:Feb 23 13:12:29 trial068 ceph-osd[99956]: --2- 10.20.193.68:0/1146350090 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x4221c00 0x505e000 crc :-1 s=THROTTLE_DONE pgs=602 gs=1 cs=0 l=1 c_cookie=7137512895559948888 s_cookie=0 reconnecting=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_epilogue_main r=0 2026-02-23T13:12:30.159 INFO:journalctl@ceph.osd.1.trial068.stdout:Feb 23 13:12:29 trial068 ceph-osd[99956]: --2- 10.20.193.68:0/1146350090 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x4221c00 0x505e000 crc :-1 s=THROTTLE_DONE pgs=602 gs=1 cs=0 l=1 c_cookie=7137512895559948888 s_cookie=0 reconnecting=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_dispatch tag=17 2026-02-23T13:12:30.159 INFO:journalctl@ceph.osd.1.trial068.stdout:Feb 23 13:12:29 trial068 ceph-osd[99956]: --2- 10.20.193.68:0/1146350090 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x4221c00 0x505e000 crc :-1 s=THROTTLE_DONE pgs=602 gs=1 cs=0 l=1 c_cookie=7137512895559948888 s_cookie=0 reconnecting=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_message 2026-02-23T13:12:30.159 INFO:journalctl@ceph.osd.1.trial068.stdout:Feb 23 13:12:29 trial068 ceph-osd[99956]: --2- 10.20.193.68:0/1146350090 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x4221c00 0x505e000 crc :-1 s=THROTTLE_DONE pgs=602 gs=1 cs=0 l=1 c_cookie=7137512895559948888 s_cookie=0 reconnecting=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_message got 194 + 0 + 0 byte message. envelope type=18 src mon.0 off 0 2026-02-23T13:12:30.159 INFO:journalctl@ceph.osd.1.trial068.stdout:Feb 23 13:12:29 trial068 ceph-osd[99956]: --2- 10.20.193.68:0/1146350090 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x4221c00 0x505e000 crc :-1 s=READ_MESSAGE_COMPLETE pgs=602 gs=1 cs=0 l=1 c_cookie=7137512895559948888 s_cookie=0 reconnecting=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_message received message m=0x4285860 seq=4 from=mon.0 type=18 auth_reply(proto 2 0 (0) Success) 2026-02-23T13:12:30.159 INFO:journalctl@ceph.osd.1.trial068.stdout:Feb 23 13:12:29 trial068 ceph-osd[99956]: -- 10.20.193.68:0/1146350090 queue 0x4285860 prio 196 2026-02-23T13:12:30.159 INFO:journalctl@ceph.osd.1.trial068.stdout:Feb 23 13:12:29 trial068 ceph-osd[99956]: --2- 10.20.193.68:0/1146350090 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x4221c00 0x505e000 crc :-1 s=READY pgs=602 gs=1 cs=0 l=1 c_cookie=7137512895559948888 s_cookie=0 reconnecting=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).read_frame 2026-02-23T13:12:30.159 INFO:journalctl@ceph.osd.1.trial068.stdout:Feb 23 13:12:29 trial068 ceph-osd[99956]: -- 10.20.193.68:0/1146350090 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x4221c00 msgr2=0x505e000 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read start len=32 2026-02-23T13:12:30.160 INFO:journalctl@ceph.osd.1.trial068.stdout:Feb 23 13:12:29 trial068 ceph-osd[99956]: -- 10.20.193.68:0/1146350090 <== mon.0 v2:10.20.193.68:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (crc 0 0 0) 0x4285860 con 0x4221c00 2026-02-23T13:12:30.160 INFO:journalctl@ceph.osd.1.trial068.stdout:Feb 23 13:12:29 trial068 ceph-osd[99956]: -- 10.20.193.68:0/1146350090 dispatch_throttle_release 194 to dispatch throttler 194/104857600 2026-02-23T13:12:30.160 INFO:journalctl@ceph.osd.1.trial068.stdout:Feb 23 13:12:29 trial068 ceph-osd[99956]: -- 10.20.193.68:0/1146350090 done calling dispatch on 0x4285860 2026-02-23T13:12:30.160 INFO:journalctl@ceph.osd.1.trial068.stdout:Feb 23 13:12:29 trial068 ceph-osd[99956]: WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:12:30.160 INFO:journalctl@ceph.osd.1.trial068.stdout:Feb 23 13:12:29 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-1[99952]: 2026-02-23T13:12:29.917+0000 7f760f1d4780 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:12:30.160 INFO:journalctl@ceph.osd.1.trial068.stdout:Feb 23 13:12:29 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-1[99952]: 2026-02-23T13:12:29.917+0000 7f760f1d4780 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:12:30.160 INFO:journalctl@ceph.osd.1.trial068.stdout:Feb 23 13:12:29 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-1[99952]: 2026-02-23T13:12:29.917+0000 7f760f1d4780 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:12:30.160 INFO:journalctl@ceph.osd.1.trial068.stdout:Feb 23 13:12:30 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-1[99952]: 2026-02-23T13:12:30.140+0000 7f760f1d4780 -1 Falling back to public interface 2026-02-23T13:12:30.178 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-23T13:12:30.179 INFO:tasks.cephadm.ceph_manager.ceph:waiting on admin_socket for osd-1, ['dump_ops_in_flight'] 2026-02-23T13:12:30.551 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 52 128 213402 213274 32.0395 0 - 0.0270304 2026-02-23T13:12:30.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:12:30 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1052392829' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:12:31.552 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 53 128 213402 213274 31.435 0 - 0.0270304 2026-02-23T13:12:31.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:12:31 trial068 ceph-mon[38805]: pgmap v270: 17 pgs: 8 undersized+degraded+peered, 9 active+undersized+degraded; 1.6 GiB data, 6.6 GiB used, 2.7 TiB / 2.7 TiB avail; 10 MiB/s wr, 1.28k op/s; 40092/79998 objects degraded (50.116%); 4.0 KiB/s, 0 objects/s recovering 2026-02-23T13:12:31.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:12:31 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2026-02-23T13:12:31.778 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:12:31.899 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:12:32.008 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:12:32.006+0000 7fa91cbf1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:12:32.008 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:12:32.006+0000 7fa91cbf1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:12:32.021 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:12:32.021 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:12:32.022 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.1 2026-02-23T13:12:32.022 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:12:32.033 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:12:32.033 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:12:32.064 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:12:32.065 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:12:32.552 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 54 128 213402 213274 30.8529 0 - 0.0270304 2026-02-23T13:12:32.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:12:32 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-23T13:12:32.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:12:32 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2026-02-23T13:12:32.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:12:32 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T13:12:32.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:12:32 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2026-02-23T13:12:32.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:12:32 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2241251735' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:12:33.552 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 55 128 213402 213274 30.2919 0 - 0.0270304 2026-02-23T13:12:33.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:12:33 trial068 ceph-mon[38805]: pgmap v271: 17 pgs: 8 undersized+degraded+peered, 9 active+undersized+degraded; 1.6 GiB data, 6.6 GiB used, 2.7 TiB / 2.7 TiB avail; 8.9 MiB/s wr, 1.14k op/s; 40092/79998 objects degraded (50.116%); 3.6 KiB/s, 0 objects/s recovering 2026-02-23T13:12:34.065 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:12:34.185 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:12:34.239 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:12:33 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:12:33] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:12:34.329 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:12:34.327+0000 7fc7a26e2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:12:34.329 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:12:34.327+0000 7fc7a26e2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:12:34.341 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:12:34.341 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:12:34.341 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.1 2026-02-23T13:12:34.341 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:12:34.352 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:12:34.352 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:12:34.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:12:33 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:12:33.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:12:34.381 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:12:34.381 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:12:34.552 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 56 128 213402 213274 29.751 0 - 0.0270304 2026-02-23T13:12:34.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:12:34 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1767243342' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:12:35.180 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph daemon osd.1 dump_ops_in_flight 2026-02-23T13:12:35.297 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:12:35.413 INFO:teuthology.orchestra.run.trial068.stderr:no valid command found; 10 closest matches: 2026-02-23T13:12:35.413 INFO:teuthology.orchestra.run.trial068.stderr:0 2026-02-23T13:12:35.413 INFO:teuthology.orchestra.run.trial068.stderr:1 2026-02-23T13:12:35.413 INFO:teuthology.orchestra.run.trial068.stderr:2 2026-02-23T13:12:35.413 INFO:teuthology.orchestra.run.trial068.stderr:abort 2026-02-23T13:12:35.413 INFO:teuthology.orchestra.run.trial068.stderr:assert 2026-02-23T13:12:35.413 INFO:teuthology.orchestra.run.trial068.stderr:bluefs debug_inject_read_zeros 2026-02-23T13:12:35.413 INFO:teuthology.orchestra.run.trial068.stderr:bluefs files list 2026-02-23T13:12:35.414 INFO:teuthology.orchestra.run.trial068.stderr:bluefs stats 2026-02-23T13:12:35.414 INFO:teuthology.orchestra.run.trial068.stderr:bluestore bluefs device info [] 2026-02-23T13:12:35.414 INFO:teuthology.orchestra.run.trial068.stderr:bluestore collections 2026-02-23T13:12:35.414 INFO:teuthology.orchestra.run.trial068.stderr:admin_socket: invalid command 2026-02-23T13:12:35.444 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-23T13:12:35.445 INFO:tasks.cephadm.ceph_manager.ceph:waiting on admin_socket for osd-1, ['dump_ops_in_flight'] 2026-02-23T13:12:35.552 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 57 128 213402 213274 29.2291 0 - 0.0270304 2026-02-23T13:12:35.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:12:35 trial068 ceph-mon[38805]: pgmap v272: 17 pgs: 8 undersized+degraded+peered, 9 active+undersized+degraded; 1.6 GiB data, 6.6 GiB used, 2.7 TiB / 2.7 TiB avail; 7.8 MiB/s wr, 994 op/s; 40092/79998 objects degraded (50.116%); 3.1 KiB/s, 0 objects/s recovering 2026-02-23T13:12:36.382 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:12:36.501 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:12:36.552 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 58 128 213402 213274 28.7251 0 - 0.0270304 2026-02-23T13:12:36.606 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:12:36.605+0000 7fe6bef40640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:12:36.606 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:12:36.605+0000 7fe6bef40640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:12:36.619 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:12:36.620 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:12:36.620 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.1 2026-02-23T13:12:36.620 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:12:36.630 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:12:36.631 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:12:36.661 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:12:36.661 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:12:37.552 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 59 128 213402 213274 28.2383 0 - 0.0270304 2026-02-23T13:12:37.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:12:37 trial068 ceph-mon[38805]: pgmap v273: 17 pgs: 8 undersized+degraded+peered, 9 active+undersized+degraded; 1.6 GiB data, 6.6 GiB used, 2.7 TiB / 2.7 TiB avail; 6.7 MiB/s wr, 851 op/s; 40092/79998 objects degraded (50.116%); 2.7 KiB/s, 0 objects/s recovering 2026-02-23T13:12:37.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:12:37 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/163126195' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:12:38.552 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T13:12:38.552136+0000 min lat: 0.0010091 max lat: 2.47028 avg lat: 0.0270304 2026-02-23T13:12:38.552 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T13:12:38.552 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 60 128 213402 213274 27.7676 0 - 0.0270304 2026-02-23T13:12:38.662 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:12:38.780 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:12:38.879 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:12:38.878+0000 7fa129c40640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:12:38.880 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:12:38.878+0000 7fa129c40640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:12:38.892 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:12:38.892 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:12:38.892 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.1 2026-02-23T13:12:38.892 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:12:38.902 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:12:38.903 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:12:38.933 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:12:38.933 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:12:39.552 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 61 128 213402 213274 27.3124 0 - 0.0270304 2026-02-23T13:12:39.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:12:39 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T13:12:39.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:12:39 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T13:12:39.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:12:39 trial068 ceph-mon[38805]: pgmap v274: 17 pgs: 8 undersized+degraded+peered, 9 active+undersized+degraded; 1.6 GiB data, 6.6 GiB used, 2.7 TiB / 2.7 TiB avail; 3.5 MiB/s wr, 442 op/s; 40092/79998 objects degraded (50.116%); 2.7 KiB/s, 0 objects/s recovering 2026-02-23T13:12:39.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:12:39 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/83872413' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:12:39.844 INFO:journalctl@ceph.osd.1.trial068.stdout:Feb 23 13:12:39 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-1[99952]: 2026-02-23T13:12:39.697+0000 7f760f1d4780 -1 osd.1 136 log_to_monitors true 2026-02-23T13:12:40.446 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph daemon osd.1 dump_ops_in_flight 2026-02-23T13:12:40.552 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 62 128 213402 213274 26.8719 0 - 0.0270304 2026-02-23T13:12:40.564 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:12:40.683 INFO:teuthology.orchestra.run.trial068.stdout:{ 2026-02-23T13:12:40.683 INFO:teuthology.orchestra.run.trial068.stdout: "ops": [], 2026-02-23T13:12:40.684 INFO:teuthology.orchestra.run.trial068.stdout: "num_ops": 0 2026-02-23T13:12:40.684 INFO:teuthology.orchestra.run.trial068.stdout:} 2026-02-23T13:12:40.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:12:40 trial068 ceph-mon[38805]: from='osd.1 [v2:10.20.193.68:6810/2748534589,v1:10.20.193.68:6811/2748534589]' entity='osd.1' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]} : dispatch 2026-02-23T13:12:40.844 INFO:journalctl@ceph.osd.1.trial068.stdout:Feb 23 13:12:40 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-1[99952]: 2026-02-23T13:12:40.477+0000 7f760701d640 -1 osd.1 136 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2026-02-23T13:12:40.934 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:12:41.053 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:12:41.153 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:12:41.151+0000 7fa10d108640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:12:41.153 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:12:41.152+0000 7fa10d108640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:12:41.165 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:12:41.166 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:12:41.166 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.1 2026-02-23T13:12:41.166 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:12:41.176 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:12:41.176 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:12:41.207 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:12:41.207 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:12:41.552 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 63 128 213402 213274 26.4454 0 - 0.0270304 2026-02-23T13:12:41.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:12:41 trial068 ceph-mon[38805]: pgmap v275: 17 pgs: 8 undersized+degraded+peered, 9 active+undersized+degraded; 1.6 GiB data, 6.6 GiB used, 2.7 TiB / 2.7 TiB avail; 40092/79998 objects degraded (50.116%) 2026-02-23T13:12:41.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:12:41 trial068 ceph-mon[38805]: from='osd.1 [v2:10.20.193.68:6810/2748534589,v1:10.20.193.68:6811/2748534589]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2026-02-23T13:12:41.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:12:41 trial068 ceph-mon[38805]: osdmap e143: 4 total, 2 up, 4 in 2026-02-23T13:12:41.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:12:41 trial068 ceph-mon[38805]: from='osd.1 [v2:10.20.193.68:6810/2748534589,v1:10.20.193.68:6811/2748534589]' entity='osd.1' cmd={"prefix": "osd crush create-or-move", "id": 1, "weight":0.6836, "args": ["host=trial068", "root=default"]} : dispatch 2026-02-23T13:12:41.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:12:41 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/698182673' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:12:42.552 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 64 128 213450 213322 26.038 0.0208333 18.514 0.0312343 2026-02-23T13:12:42.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:12:42 trial068 ceph-mon[38805]: Health check update: 1 osds down (OSD_DOWN) 2026-02-23T13:12:42.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:12:42 trial068 ceph-mon[38805]: osd.1 [v2:10.20.193.68:6810/2748534589,v1:10.20.193.68:6811/2748534589] boot 2026-02-23T13:12:42.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:12:42 trial068 ceph-mon[38805]: osdmap e144: 4 total, 3 up, 4 in 2026-02-23T13:12:42.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:12:42 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2026-02-23T13:12:43.208 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:12:43.333 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:12:43.447 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:12:43.445+0000 7f81ed190640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:12:43.447 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:12:43.445+0000 7f81ed190640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:12:43.460 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:12:43.460 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:12:43.470 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:12:43.471 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:12:43.471 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:12:43.482 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:12:43.482 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:12:43.518 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:12:43.518 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:12:43.552 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 65 128 213783 213655 25.6775 2.60156 1.01607 0.037073 2026-02-23T13:12:43.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:12:43 trial068 ceph-mon[38805]: pgmap v278: 17 pgs: 8 undersized+degraded+peered, 9 active+undersized+degraded; 1.6 GiB data, 6.0 GiB used, 2.7 TiB / 2.7 TiB avail; 40092/79998 objects degraded (50.116%) 2026-02-23T13:12:43.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:12:43 trial068 ceph-mon[38805]: osdmap e145: 4 total, 3 up, 4 in 2026-02-23T13:12:43.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:12:43 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1484006755' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:12:44.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:12:43 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:12:43] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:12:44.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:12:43 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:12:43.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:12:44.553 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 66 127 217758 217631 25.759 31.0625 0.0108291 0.0388074 2026-02-23T13:12:44.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:12:44 trial068 ceph-mon[38805]: osdmap e146: 4 total, 3 up, 4 in 2026-02-23T13:12:45.519 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:12:45.553 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 67 128 221445 221317 25.8043 28.7969 0.125932 0.0387224 2026-02-23T13:12:45.663 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:12:45.716 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [2] live_osds: [3, 0, 1] 2026-02-23T13:12:45.716 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2026-02-23T13:12:45.716 INFO:tasks.thrashosds.thrasher:Reviving osd 2 2026-02-23T13:12:45.716 INFO:tasks.cephadm.osd.2:Restarting osd.2 (starting--it wasn't running)... 2026-02-23T13:12:45.716 DEBUG:teuthology.orchestra.run.trial185:osd.2> sudo journalctl -f -n 0 -u ceph-5f823084-10b8-11f1-be80-d404e6e7d460@osd.2.service 2026-02-23T13:12:45.719 DEBUG:teuthology.orchestra.run.trial185:> sudo systemctl start ceph-5f823084-10b8-11f1-be80-d404e6e7d460@osd.2 2026-02-23T13:12:45.737 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:12:45 trial068 ceph-mon[38805]: pgmap v281: 17 pgs: 8 undersized+degraded+peered, 9 active+undersized+degraded; 1.6 GiB data, 6.0 GiB used, 2.7 TiB / 2.7 TiB avail; 40092/79998 objects degraded (50.116%) 2026-02-23T13:12:45.799 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:12:45.797+0000 7f5d1e954640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:12:45.799 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:12:45.798+0000 7f5d1e954640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:12:45.816 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:12:45.817 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:12:45.829 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:12:45.829 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:12:45.829 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:12:45.843 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:12:45.843 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:12:45.881 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:12:45.882 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:12:46.015 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:12:45 trial185 systemd[1]: Starting Ceph osd.2 for 5f823084-10b8-11f1-be80-d404e6e7d460... 2026-02-23T13:12:46.016 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:12:45 trial185 podman[74964]: 2026-02-23 13:12:45.890507379 +0000 UTC m=+0.016074108 container create 718f475d44f8402df64d1d124c3c942b9c743a259ae6256e618671955c1decd6 (image=quay.ceph.io/ceph-ci/ceph@sha256:49621faaa845c39b2aae72e4fa24123ee9b2ac3fb8ae834753907068ee2972ea, name=ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-2-activate, org.opencontainers.image.documentation=https://docs.ceph.com/, FROM_IMAGE=quay.io/centos/centos:stream9, OSD_FLAVOR=default, org.label-schema.vendor=CentOS, CEPH_GIT_REPO=https://github.com/ceph/ceph-ci.git, CEPH_SHA1=e371433fcd1390a0b82cfb18effc422c1f5d8c1d, org.opencontainers.image.authors=Ceph Release Team , io.buildah.version=1.41.3, GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfsganesha-5/, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, CEPH_REF=wip-shraddhaag-enable-cephadm-tests, org.label-schema.build-date=20260216, org.label-schema.schema-version=1.0) 2026-02-23T13:12:46.016 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:12:45 trial185 podman[74964]: 2026-02-23 13:12:45.931769376 +0000 UTC m=+0.057336105 container init 718f475d44f8402df64d1d124c3c942b9c743a259ae6256e618671955c1decd6 (image=quay.ceph.io/ceph-ci/ceph@sha256:49621faaa845c39b2aae72e4fa24123ee9b2ac3fb8ae834753907068ee2972ea, name=ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-2-activate, GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfsganesha-5/, org.opencontainers.image.documentation=https://docs.ceph.com/, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_GIT_REPO=https://github.com/ceph/ceph-ci.git, FROM_IMAGE=quay.io/centos/centos:stream9, CEPH_REF=wip-shraddhaag-enable-cephadm-tests, org.label-schema.vendor=CentOS, org.opencontainers.image.authors=Ceph Release Team , ceph=True, org.label-schema.build-date=20260216, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_SHA1=e371433fcd1390a0b82cfb18effc422c1f5d8c1d, io.buildah.version=1.41.3, OSD_FLAVOR=default) 2026-02-23T13:12:46.016 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:12:45 trial185 podman[74964]: 2026-02-23 13:12:45.934280395 +0000 UTC m=+0.059847124 container start 718f475d44f8402df64d1d124c3c942b9c743a259ae6256e618671955c1decd6 (image=quay.ceph.io/ceph-ci/ceph@sha256:49621faaa845c39b2aae72e4fa24123ee9b2ac3fb8ae834753907068ee2972ea, name=ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-2-activate, ceph=True, org.label-schema.build-date=20260216, org.label-schema.schema-version=1.0, GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfsganesha-5/, CEPH_GIT_REPO=https://github.com/ceph/ceph-ci.git, OSD_FLAVOR=default, CEPH_REF=wip-shraddhaag-enable-cephadm-tests, org.label-schema.license=GPLv2, org.opencontainers.image.authors=Ceph Release Team , org.opencontainers.image.documentation=https://docs.ceph.com/, FROM_IMAGE=quay.io/centos/centos:stream9, io.buildah.version=1.41.3, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, CEPH_SHA1=e371433fcd1390a0b82cfb18effc422c1f5d8c1d) 2026-02-23T13:12:46.016 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:12:45 trial185 podman[74964]: 2026-02-23 13:12:45.934833813 +0000 UTC m=+0.060400552 container attach 718f475d44f8402df64d1d124c3c942b9c743a259ae6256e618671955c1decd6 (image=quay.ceph.io/ceph-ci/ceph@sha256:49621faaa845c39b2aae72e4fa24123ee9b2ac3fb8ae834753907068ee2972ea, name=ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-2-activate, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.41.3, OSD_FLAVOR=default, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_REF=wip-shraddhaag-enable-cephadm-tests, CEPH_GIT_REPO=https://github.com/ceph/ceph-ci.git, CEPH_SHA1=e371433fcd1390a0b82cfb18effc422c1f5d8c1d, org.opencontainers.image.authors=Ceph Release Team , org.opencontainers.image.documentation=https://docs.ceph.com/, org.label-schema.vendor=CentOS, org.label-schema.build-date=20260216, org.label-schema.license=GPLv2, FROM_IMAGE=quay.io/centos/centos:stream9, GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfsganesha-5/) 2026-02-23T13:12:46.016 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:12:45 trial185 podman[74964]: 2026-02-23 13:12:45.883892673 +0000 UTC m=+0.009459402 image pull 63df4b83db6f95a8ba965919b3671cb27173117c23b5c950aaadc8913f56213c quay.ceph.io/ceph-ci/ceph@sha256:49621faaa845c39b2aae72e4fa24123ee9b2ac3fb8ae834753907068ee2972ea 2026-02-23T13:12:46.016 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:12:46 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-2-activate[74985]: Running command: /usr/bin/ceph-authtool --gen-print-key 2026-02-23T13:12:46.016 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:12:46 trial185 bash[74964]: Running command: /usr/bin/ceph-authtool --gen-print-key 2026-02-23T13:12:46.319 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:12:46 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-2-activate[74985]: Running command: /usr/bin/ceph-authtool --gen-print-key 2026-02-23T13:12:46.319 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:12:46 trial185 bash[74964]: Running command: /usr/bin/ceph-authtool --gen-print-key 2026-02-23T13:12:46.319 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:12:46 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-2-activate[74985]: --> Failed to activate via raw: did not find any matching OSD to activate 2026-02-23T13:12:46.319 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:12:46 trial185 bash[74964]: --> Failed to activate via raw: did not find any matching OSD to activate 2026-02-23T13:12:46.319 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:12:46 trial185 bash[74964]: Running command: /usr/bin/ceph-authtool --gen-print-key 2026-02-23T13:12:46.553 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 68 128 223109 222981 25.616 13 0.718245 0.0388127 2026-02-23T13:12:46.570 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:12:46 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-2-activate[74985]: Running command: /usr/bin/ceph-authtool --gen-print-key 2026-02-23T13:12:46.571 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:12:46 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-2-activate[74985]: Running command: /usr/bin/ceph-authtool --gen-print-key 2026-02-23T13:12:46.571 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:12:46 trial185 bash[74964]: Running command: /usr/bin/ceph-authtool --gen-print-key 2026-02-23T13:12:46.571 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:12:46 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-2-activate[74985]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2026-02-23T13:12:46.571 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:12:46 trial185 bash[74964]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2026-02-23T13:12:46.571 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:12:46 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-2-activate[74985]: Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/vg_nvme/lv_4 --path /var/lib/ceph/osd/ceph-2 --no-mon-config 2026-02-23T13:12:46.571 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:12:46 trial185 bash[74964]: Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/vg_nvme/lv_4 --path /var/lib/ceph/osd/ceph-2 --no-mon-config 2026-02-23T13:12:46.571 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:12:46 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-2-activate[74985]: Running command: /usr/bin/ln -snf /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-2/block 2026-02-23T13:12:46.571 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:12:46 trial185 bash[74964]: Running command: /usr/bin/ln -snf /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-2/block 2026-02-23T13:12:46.571 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:12:46 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-2-activate[74985]: Running command: /usr/bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-2/block 2026-02-23T13:12:46.571 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:12:46 trial185 bash[74964]: Running command: /usr/bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-2/block 2026-02-23T13:12:46.572 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:12:46 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-2-activate[74985]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2026-02-23T13:12:46.572 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:12:46 trial185 bash[74964]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2026-02-23T13:12:46.572 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:12:46 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-2-activate[74985]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2026-02-23T13:12:46.572 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:12:46 trial185 bash[74964]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2026-02-23T13:12:46.572 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:12:46 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-2-activate[74985]: --> ceph-volume lvm activate successful for osd ID: 2 2026-02-23T13:12:46.572 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:12:46 trial185 bash[74964]: --> ceph-volume lvm activate successful for osd ID: 2 2026-02-23T13:12:46.572 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:12:46 trial185 podman[74964]: 2026-02-23 13:12:46.436347391 +0000 UTC m=+0.561914140 container died 718f475d44f8402df64d1d124c3c942b9c743a259ae6256e618671955c1decd6 (image=quay.ceph.io/ceph-ci/ceph@sha256:49621faaa845c39b2aae72e4fa24123ee9b2ac3fb8ae834753907068ee2972ea, name=ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-2-activate, ceph=True, GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfsganesha-5/, CEPH_SHA1=e371433fcd1390a0b82cfb18effc422c1f5d8c1d, CEPH_REF=wip-shraddhaag-enable-cephadm-tests, org.label-schema.license=GPLv2, org.opencontainers.image.documentation=https://docs.ceph.com/, io.buildah.version=1.41.3, OSD_FLAVOR=default, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_GIT_REPO=https://github.com/ceph/ceph-ci.git, FROM_IMAGE=quay.io/centos/centos:stream9, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20260216, org.opencontainers.image.authors=Ceph Release Team ) 2026-02-23T13:12:46.572 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:12:46 trial185 podman[74964]: 2026-02-23 13:12:46.447312266 +0000 UTC m=+0.572878995 container remove 718f475d44f8402df64d1d124c3c942b9c743a259ae6256e618671955c1decd6 (image=quay.ceph.io/ceph-ci/ceph@sha256:49621faaa845c39b2aae72e4fa24123ee9b2ac3fb8ae834753907068ee2972ea, name=ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-2-activate, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_SHA1=e371433fcd1390a0b82cfb18effc422c1f5d8c1d, org.label-schema.license=GPLv2, CEPH_GIT_REPO=https://github.com/ceph/ceph-ci.git, FROM_IMAGE=quay.io/centos/centos:stream9, org.opencontainers.image.authors=Ceph Release Team , OSD_FLAVOR=default, org.opencontainers.image.documentation=https://docs.ceph.com/, CEPH_REF=wip-shraddhaag-enable-cephadm-tests, GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfsganesha-5/, org.label-schema.vendor=CentOS, org.label-schema.build-date=20260216, org.label-schema.schema-version=1.0, io.buildah.version=1.41.3) 2026-02-23T13:12:46.572 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:12:46 trial185 podman[75197]: 2026-02-23 13:12:46.538620735 +0000 UTC m=+0.017151919 container create 1d17780e12e57ff76768afce1dc67bc526adc9883ba076bc4e9e0a3973d274f1 (image=quay.ceph.io/ceph-ci/ceph@sha256:49621faaa845c39b2aae72e4fa24123ee9b2ac3fb8ae834753907068ee2972ea, name=ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-2, org.opencontainers.image.authors=Ceph Release Team , CEPH_REF=wip-shraddhaag-enable-cephadm-tests, org.label-schema.name=CentOS Stream 9 Base Image, org.opencontainers.image.documentation=https://docs.ceph.com/, GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfsganesha-5/, CEPH_GIT_REPO=https://github.com/ceph/ceph-ci.git, CEPH_SHA1=e371433fcd1390a0b82cfb18effc422c1f5d8c1d, FROM_IMAGE=quay.io/centos/centos:stream9, ceph=True, io.buildah.version=1.41.3, org.label-schema.vendor=CentOS, org.label-schema.build-date=20260216, OSD_FLAVOR=default, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2) 2026-02-23T13:12:46.572 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:12:46 trial185 podman[75197]: 2026-02-23 13:12:46.566486828 +0000 UTC m=+0.045018032 container init 1d17780e12e57ff76768afce1dc67bc526adc9883ba076bc4e9e0a3973d274f1 (image=quay.ceph.io/ceph-ci/ceph@sha256:49621faaa845c39b2aae72e4fa24123ee9b2ac3fb8ae834753907068ee2972ea, name=ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-2, CEPH_REF=wip-shraddhaag-enable-cephadm-tests, CEPH_GIT_REPO=https://github.com/ceph/ceph-ci.git, ceph=True, OSD_FLAVOR=default, FROM_IMAGE=quay.io/centos/centos:stream9, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, io.buildah.version=1.41.3, GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfsganesha-5/, org.label-schema.build-date=20260216, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_SHA1=e371433fcd1390a0b82cfb18effc422c1f5d8c1d, org.opencontainers.image.documentation=https://docs.ceph.com/, org.label-schema.vendor=CentOS, org.opencontainers.image.authors=Ceph Release Team ) 2026-02-23T13:12:46.572 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:12:46 trial185 podman[75197]: 2026-02-23 13:12:46.568756725 +0000 UTC m=+0.047287919 container start 1d17780e12e57ff76768afce1dc67bc526adc9883ba076bc4e9e0a3973d274f1 (image=quay.ceph.io/ceph-ci/ceph@sha256:49621faaa845c39b2aae72e4fa24123ee9b2ac3fb8ae834753907068ee2972ea, name=ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-2, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, OSD_FLAVOR=default, CEPH_SHA1=e371433fcd1390a0b82cfb18effc422c1f5d8c1d, org.label-schema.build-date=20260216, CEPH_GIT_REPO=https://github.com/ceph/ceph-ci.git, ceph=True, io.buildah.version=1.41.3, FROM_IMAGE=quay.io/centos/centos:stream9, org.opencontainers.image.documentation=https://docs.ceph.com/, GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfsganesha-5/, org.opencontainers.image.authors=Ceph Release Team , CEPH_REF=wip-shraddhaag-enable-cephadm-tests, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2026-02-23T13:12:46.576 DEBUG:teuthology.orchestra.run.trial185:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph daemon osd.2 dump_ops_in_flight 2026-02-23T13:12:46.704 INFO:teuthology.orchestra.run.trial185.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/config/ceph.conf 2026-02-23T13:12:46.823 INFO:teuthology.orchestra.run.trial185.stderr:no valid command found; 10 closest matches: 2026-02-23T13:12:46.823 INFO:teuthology.orchestra.run.trial185.stderr:0 2026-02-23T13:12:46.823 INFO:teuthology.orchestra.run.trial185.stderr:1 2026-02-23T13:12:46.824 INFO:teuthology.orchestra.run.trial185.stderr:2 2026-02-23T13:12:46.824 INFO:teuthology.orchestra.run.trial185.stderr:abort 2026-02-23T13:12:46.824 INFO:teuthology.orchestra.run.trial185.stderr:assert 2026-02-23T13:12:46.824 INFO:teuthology.orchestra.run.trial185.stderr:bluestore collections 2026-02-23T13:12:46.824 INFO:teuthology.orchestra.run.trial185.stderr:bluestore compression stats [] 2026-02-23T13:12:46.824 INFO:teuthology.orchestra.run.trial185.stderr:bluestore list [] [] 2026-02-23T13:12:46.825 INFO:teuthology.orchestra.run.trial185.stderr:bluestore onode metadata 2026-02-23T13:12:46.825 INFO:teuthology.orchestra.run.trial185.stderr:bluestore show sharding 2026-02-23T13:12:46.825 INFO:teuthology.orchestra.run.trial185.stderr:admin_socket: invalid command 2026-02-23T13:12:46.826 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:12:46 trial185 bash[75197]: 1d17780e12e57ff76768afce1dc67bc526adc9883ba076bc4e9e0a3973d274f1 2026-02-23T13:12:46.826 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:12:46 trial185 podman[75197]: 2026-02-23 13:12:46.53212365 +0000 UTC m=+0.010654854 image pull 63df4b83db6f95a8ba965919b3671cb27173117c23b5c950aaadc8913f56213c quay.ceph.io/ceph-ci/ceph@sha256:49621faaa845c39b2aae72e4fa24123ee9b2ac3fb8ae834753907068ee2972ea 2026-02-23T13:12:46.826 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:12:46 trial185 systemd[1]: Started Ceph osd.2 for 5f823084-10b8-11f1-be80-d404e6e7d460. 2026-02-23T13:12:46.826 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:12:46 trial185 ceph-osd[75221]: -- 10.20.193.185:0/2159500948 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x3493c00 msgr2=0x42d0000 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).process 2026-02-23T13:12:46.827 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:12:46 trial185 ceph-osd[75221]: -- 10.20.193.185:0/2159500948 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x3493c00 msgr2=0x42d0000 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read continue len=32 2026-02-23T13:12:46.827 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:12:46 trial185 ceph-osd[75221]: --2- 10.20.193.185:0/2159500948 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x3493c00 0x42d0000 crc :-1 s=READY pgs=618 gs=1 cs=0 l=1 c_cookie=1971582492419620938 s_cookie=0 reconnecting=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main r=0 2026-02-23T13:12:46.827 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:12:46 trial185 ceph-osd[75221]: --2- 10.20.193.185:0/2159500948 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x3493c00 0x42d0000 crc :-1 s=THROTTLE_MESSAGE pgs=618 gs=1 cs=0 l=1 c_cookie=1971582492419620938 s_cookie=0 reconnecting=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).throttle_message 2026-02-23T13:12:46.827 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:12:46 trial185 ceph-osd[75221]: --2- 10.20.193.185:0/2159500948 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x3493c00 0x42d0000 crc :-1 s=THROTTLE_BYTES pgs=618 gs=1 cs=0 l=1 c_cookie=1971582492419620938 s_cookie=0 reconnecting=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).throttle_bytes 2026-02-23T13:12:46.827 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:12:46 trial185 ceph-osd[75221]: --2- 10.20.193.185:0/2159500948 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x3493c00 0x42d0000 crc :-1 s=THROTTLE_DISPATCH_QUEUE pgs=618 gs=1 cs=0 l=1 c_cookie=1971582492419620938 s_cookie=0 reconnecting=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).throttle_dispatch_queue 2026-02-23T13:12:46.828 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:12:46 trial185 ceph-osd[75221]: --2- 10.20.193.185:0/2159500948 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x3493c00 0x42d0000 crc :-1 s=THROTTLE_DONE pgs=618 gs=1 cs=0 l=1 c_cookie=1971582492419620938 s_cookie=0 reconnecting=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).read_frame_segment seg_idx=0 2026-02-23T13:12:46.828 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:12:46 trial185 ceph-osd[75221]: -- 10.20.193.185:0/2159500948 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x3493c00 msgr2=0x42d0000 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read start len=45 2026-02-23T13:12:46.828 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:12:46 trial185 ceph-osd[75221]: --2- 10.20.193.185:0/2159500948 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x3493c00 0x42d0000 crc :-1 s=THROTTLE_DONE pgs=618 gs=1 cs=0 l=1 c_cookie=1971582492419620938 s_cookie=0 reconnecting=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_segment r=0 2026-02-23T13:12:46.828 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:12:46 trial185 ceph-osd[75221]: --2- 10.20.193.185:0/2159500948 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x3493c00 0x42d0000 crc :-1 s=THROTTLE_DONE pgs=618 gs=1 cs=0 l=1 c_cookie=1971582492419620938 s_cookie=0 reconnecting=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).read_frame_segment seg_idx=1 2026-02-23T13:12:46.828 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:12:46 trial185 ceph-osd[75221]: -- 10.20.193.185:0/2159500948 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x3493c00 msgr2=0x42d0000 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read start len=194 2026-02-23T13:12:46.828 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:12:46 trial185 ceph-osd[75221]: --2- 10.20.193.185:0/2159500948 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x3493c00 0x42d0000 crc :-1 s=THROTTLE_DONE pgs=618 gs=1 cs=0 l=1 c_cookie=1971582492419620938 s_cookie=0 reconnecting=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_segment r=0 2026-02-23T13:12:46.829 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:12:46 trial185 ceph-osd[75221]: -- 10.20.193.185:0/2159500948 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x3493c00 msgr2=0x42d0000 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read start len=13 2026-02-23T13:12:46.829 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:12:46 trial185 ceph-osd[75221]: --2- 10.20.193.185:0/2159500948 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x3493c00 0x42d0000 crc :-1 s=THROTTLE_DONE pgs=618 gs=1 cs=0 l=1 c_cookie=1971582492419620938 s_cookie=0 reconnecting=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_epilogue_main r=0 2026-02-23T13:12:46.829 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:12:46 trial185 ceph-osd[75221]: --2- 10.20.193.185:0/2159500948 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x3493c00 0x42d0000 crc :-1 s=THROTTLE_DONE pgs=618 gs=1 cs=0 l=1 c_cookie=1971582492419620938 s_cookie=0 reconnecting=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_dispatch tag=17 2026-02-23T13:12:46.829 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:12:46 trial185 ceph-osd[75221]: --2- 10.20.193.185:0/2159500948 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x3493c00 0x42d0000 crc :-1 s=THROTTLE_DONE pgs=618 gs=1 cs=0 l=1 c_cookie=1971582492419620938 s_cookie=0 reconnecting=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_message 2026-02-23T13:12:46.829 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:12:46 trial185 ceph-osd[75221]: --2- 10.20.193.185:0/2159500948 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x3493c00 0x42d0000 crc :-1 s=THROTTLE_DONE pgs=618 gs=1 cs=0 l=1 c_cookie=1971582492419620938 s_cookie=0 reconnecting=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_message got 194 + 0 + 0 byte message. envelope type=18 src mon.0 off 0 2026-02-23T13:12:46.829 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:12:46 trial185 ceph-osd[75221]: --2- 10.20.193.185:0/2159500948 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x3493c00 0x42d0000 crc :-1 s=READ_MESSAGE_COMPLETE pgs=618 gs=1 cs=0 l=1 c_cookie=1971582492419620938 s_cookie=0 reconnecting=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_message received message m=0x34f7860 seq=4 from=mon.0 type=18 auth_reply(proto 2 0 (0) Success) 2026-02-23T13:12:46.829 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:12:46 trial185 ceph-osd[75221]: -- 10.20.193.185:0/2159500948 queue 0x34f7860 prio 196 2026-02-23T13:12:46.829 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:12:46 trial185 ceph-osd[75221]: --2- 10.20.193.185:0/2159500948 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x3493c00 0x42d0000 crc :-1 s=READY pgs=618 gs=1 cs=0 l=1 c_cookie=1971582492419620938 s_cookie=0 reconnecting=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).read_frame 2026-02-23T13:12:46.829 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:12:46 trial185 ceph-osd[75221]: -- 10.20.193.185:0/2159500948 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x3493c00 msgr2=0x42d0000 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read start len=32 2026-02-23T13:12:46.830 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:12:46 trial185 ceph-osd[75221]: -- 10.20.193.185:0/2159500948 <== mon.0 v2:10.20.193.68:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (crc 0 0 0) 0x34f7860 con 0x3493c00 2026-02-23T13:12:46.830 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:12:46 trial185 ceph-osd[75221]: -- 10.20.193.185:0/2159500948 dispatch_throttle_release 194 to dispatch throttler 194/104857600 2026-02-23T13:12:46.830 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:12:46 trial185 ceph-osd[75221]: -- 10.20.193.185:0/2159500948 done calling dispatch on 0x34f7860 2026-02-23T13:12:46.830 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:12:46 trial185 ceph-osd[75221]: WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:12:46.830 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:12:46 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-2[75217]: 2026-02-23T13:12:46.582+0000 7f1b9fd6b780 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:12:46.830 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:12:46 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-2[75217]: 2026-02-23T13:12:46.583+0000 7f1b9fd6b780 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:12:46.830 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:12:46 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-2[75217]: 2026-02-23T13:12:46.583+0000 7f1b9fd6b780 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:12:46.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:12:46 trial068 ceph-mon[38805]: osdmap e147: 4 total, 3 up, 4 in 2026-02-23T13:12:46.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:12:46 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2343264193' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:12:46.869 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-23T13:12:46.869 INFO:tasks.cephadm.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-23T13:12:47.106 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:12:46 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-2[75217]: 2026-02-23T13:12:46.833+0000 7f1b9fd6b780 -1 Falling back to public interface 2026-02-23T13:12:47.553 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 69 128 223563 223435 25.2962 3.54688 0.017091 0.0394099 2026-02-23T13:12:47.745 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:12:47 trial068 ceph-mon[38805]: pgmap v283: 17 pgs: 3 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 4 undersized+degraded+peered, 9 active+undersized+degraded; 1.7 GiB data, 6.2 GiB used, 2.7 TiB / 2.7 TiB avail; 6.5 MiB/s wr, 836 op/s; 31084/81504 objects degraded (38.138%); 14 MiB/s, 221 objects/s recovering 2026-02-23T13:12:47.745 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:12:47 trial068 ceph-mon[38805]: Health check update: Degraded data redundancy: 31084/81504 objects degraded (38.138%), 17 pgs degraded (PG_DEGRADED) 2026-02-23T13:12:47.746 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:12:47 trial068 ceph-mon[38805]: osdmap e148: 4 total, 3 up, 4 in 2026-02-23T13:12:47.883 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:12:48.036 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:12:48.163 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:12:48.161+0000 7f3f4c881640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:12:48.164 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:12:48.162+0000 7f3f4c881640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:12:48.179 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:12:48.179 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:12:48.193 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:12:48.194 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:12:48.194 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:12:48.208 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:12:48.208 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:12:48.249 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:12:48.250 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:12:48.553 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 70 128 227905 227777 25.4194 33.9219 0.02869 0.0393257 2026-02-23T13:12:48.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:12:48 trial068 ceph-mon[38805]: osdmap e149: 4 total, 3 up, 4 in 2026-02-23T13:12:48.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:12:48 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/13192954' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:12:49.553 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 71 128 232125 231997 25.5257 32.9688 0.122666 0.0391548 2026-02-23T13:12:49.606 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:12:49 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-2[75217]: 2026-02-23T13:12:49.291+0000 7f1b9fd6b780 -1 osd.2 140 log_to_monitors true 2026-02-23T13:12:49.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:12:49 trial068 ceph-mon[38805]: pgmap v286: 17 pgs: 3 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 undersized+degraded+peered, 12 active+undersized+degraded; 1.7 GiB data, 6.2 GiB used, 2.7 TiB / 2.7 TiB avail; 13 MiB/s wr, 1.71k op/s; 26550/83061 objects degraded (31.964%); 25 MiB/s, 395 objects/s recovering 2026-02-23T13:12:49.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:12:49 trial068 ceph-mon[38805]: osdmap e150: 4 total, 3 up, 4 in 2026-02-23T13:12:49.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:12:49 trial068 ceph-mon[38805]: from='osd.2 [v2:10.20.193.185:6800/177831092,v1:10.20.193.185:6801/177831092]' entity='osd.2' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]} : dispatch 2026-02-23T13:12:50.251 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:12:50.357 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:12:50 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-2[75217]: 2026-02-23T13:12:49.999+0000 7f1b93bac640 -1 osd.2 140 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2026-02-23T13:12:50.400 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:12:50.516 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:12:50.515+0000 7ff8b86b2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:12:50.516 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:12:50.515+0000 7ff8b86b2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:12:50.530 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:12:50.531 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:12:50.544 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:12:50.545 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:12:50.545 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:12:50.553 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 72 128 236083 235955 25.6006 30.9219 0.10922 0.0390391 2026-02-23T13:12:50.559 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:12:50.559 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:12:50.604 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:12:50.605 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:12:51.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:12:50 trial068 ceph-mon[38805]: from='osd.2 [v2:10.20.193.185:6800/177831092,v1:10.20.193.185:6801/177831092]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2026-02-23T13:12:51.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:12:50 trial068 ceph-mon[38805]: osdmap e151: 4 total, 3 up, 4 in 2026-02-23T13:12:51.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:12:50 trial068 ceph-mon[38805]: from='osd.2 [v2:10.20.193.185:6800/177831092,v1:10.20.193.185:6801/177831092]' entity='osd.2' cmd={"prefix": "osd crush create-or-move", "id": 2, "weight":0.6836, "args": ["host=trial185", "root=default"]} : dispatch 2026-02-23T13:12:51.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:12:50 trial068 ceph-mon[38805]: pgmap v289: 17 pgs: 3 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 undersized+degraded+peered, 12 active+undersized+degraded; 1.7 GiB data, 6.3 GiB used, 2.7 TiB / 2.7 TiB avail; 8.1 MiB/s wr, 1.04k op/s; 26550/83061 objects degraded (31.964%); 13 MiB/s, 207 objects/s recovering 2026-02-23T13:12:51.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:12:50 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3829482772' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:12:51.553 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 73 128 238068 237940 25.4623 15.5078 0.0224237 0.0389588 2026-02-23T13:12:51.870 DEBUG:teuthology.orchestra.run.trial185:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph daemon osd.2 dump_ops_in_flight 2026-02-23T13:12:51.989 INFO:teuthology.orchestra.run.trial185.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/config/ceph.conf 2026-02-23T13:12:52.115 INFO:teuthology.orchestra.run.trial185.stdout:{ 2026-02-23T13:12:52.115 INFO:teuthology.orchestra.run.trial185.stdout: "ops": [], 2026-02-23T13:12:52.115 INFO:teuthology.orchestra.run.trial185.stdout: "num_ops": 0 2026-02-23T13:12:52.115 INFO:teuthology.orchestra.run.trial185.stdout:} 2026-02-23T13:12:52.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:12:52 trial068 ceph-mon[38805]: Health check cleared: OSD_DOWN (was: 1 osds down) 2026-02-23T13:12:52.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:12:52 trial068 ceph-mon[38805]: osd.2 [v2:10.20.193.185:6800/177831092,v1:10.20.193.185:6801/177831092] boot 2026-02-23T13:12:52.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:12:52 trial068 ceph-mon[38805]: osdmap e152: 4 total, 4 up, 4 in 2026-02-23T13:12:52.345 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:12:52 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2026-02-23T13:12:52.553 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 74 128 238068 237940 25.1182 0 - 0.0389588 2026-02-23T13:12:52.606 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:12:52.723 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:12:52.825 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:12:52.823+0000 7f3dca0bb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:12:52.825 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:12:52.823+0000 7f3dca0bb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:12:52.837 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:12:52.837 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:12:52.847 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:12:52.847 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:12:52.858 INFO:teuthology.orchestra.run.trial068.stdout:osd.2: {} 2026-02-23T13:12:52.859 INFO:teuthology.orchestra.run.trial068.stderr:osd.2: osd_delete_sleep = '' osd_delete_sleep_hdd = '' osd_delete_sleep_hybrid = '' osd_delete_sleep_ssd = '' osd_enable_op_tracker = '' osd_recovery_max_active = '' osd_recovery_max_active_hdd = '' osd_recovery_max_active_ssd = '' osd_recovery_sleep = '' osd_recovery_sleep_degraded = '' osd_recovery_sleep_degraded_hdd = '' osd_recovery_sleep_degraded_hybrid = '' osd_recovery_sleep_degraded_ssd = '' osd_recovery_sleep_hdd = '' osd_recovery_sleep_hybrid = '' osd_recovery_sleep_ssd = '' osd_scrub_sleep = '' osd_snap_trim_sleep = '' osd_snap_trim_sleep_hdd = '' osd_snap_trim_sleep_hybrid = '' osd_snap_trim_sleep_ssd = '' 2026-02-23T13:12:52.870 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:12:52.870 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:12:53.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:12:53 trial068 ceph-mon[38805]: 4.4 scrub starts 2026-02-23T13:12:53.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:12:53 trial068 ceph-mon[38805]: 4.4 scrub ok 2026-02-23T13:12:53.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:12:53 trial068 ceph-mon[38805]: osdmap e153: 4 total, 4 up, 4 in 2026-02-23T13:12:53.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:12:53 trial068 ceph-mon[38805]: pgmap v292: 17 pgs: 5 peering, 2 active+clean, 10 active+undersized+degraded; 1.8 GiB data, 6.6 GiB used, 2.7 TiB / 2.7 TiB avail; 25 MiB/s wr, 3.16k op/s; 18133/87804 objects degraded (20.652%); 15 MiB/s, 235 objects/s recovering 2026-02-23T13:12:53.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:12:53 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1499009358' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:12:53.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:12:53 trial068 ceph-mon[38805]: osdmap e154: 4 total, 4 up, 4 in 2026-02-23T13:12:53.553 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 75 128 238514 238386 24.8298 1.74219 0.0175892 0.0402613 2026-02-23T13:12:54.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:12:53 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:12:53] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:12:54.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:12:54 trial068 ceph-mon[38805]: Health check update: Degraded data redundancy: 18133/87804 objects degraded (20.652%), 10 pgs degraded (PG_DEGRADED) 2026-02-23T13:12:54.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:12:53 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:12:53.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:12:54.553 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 76 128 242259 242131 24.888 29.2578 0.00971298 0.040165 2026-02-23T13:12:54.905 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:12:55.052 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:12:55.186 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:12:55.184+0000 7f4d48704640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:12:55.186 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:12:55.184+0000 7f4d48704640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:12:55.187 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:12:55 trial068 ceph-mon[38805]: pgmap v294: 17 pgs: 5 peering, 2 active+clean, 10 active+undersized+degraded; 1.8 GiB data, 6.6 GiB used, 2.7 TiB / 2.7 TiB avail; 23 MiB/s wr, 2.95k op/s; 18133/87804 objects degraded (20.652%); 14 MiB/s, 220 objects/s recovering 2026-02-23T13:12:55.199 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:12:55.199 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:12:55.209 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:12:55.210 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:12:55.220 INFO:teuthology.orchestra.run.trial068.stdout:osd.2: {} 2026-02-23T13:12:55.220 INFO:teuthology.orchestra.run.trial068.stderr:osd.2: osd_enable_op_tracker = '' 2026-02-23T13:12:55.230 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:12:55.230 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:12:55.553 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 77 128 244418 244290 24.7838 16.8672 0.00332982 0.0401039 2026-02-23T13:12:56.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:12:56 trial068 ceph-mon[38805]: osdmap e155: 4 total, 4 up, 4 in 2026-02-23T13:12:56.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:12:56 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3468233295' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:12:56.553 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 78 128 244648 244520 24.4891 1.79688 0.0217966 0.0405771 2026-02-23T13:12:57.151 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [3, 0, 1, 2] 2026-02-23T13:12:57.152 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd reweight 0 1 2026-02-23T13:12:57.267 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:12:57.274 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:12:57.289 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:12:57 trial068 ceph-mon[38805]: osdmap e156: 4 total, 4 up, 4 in 2026-02-23T13:12:57.290 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:12:57 trial068 ceph-mon[38805]: pgmap v297: 17 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 3 active+undersized+remapped, 7 peering, 2 active+clean, 1 active+undersized+degraded; 1.8 GiB data, 6.8 GiB used, 2.7 TiB / 2.7 TiB avail; 13 MiB/s wr, 1.70k op/s; 2423/90528 objects degraded (2.677%); 15 MiB/s, 243 objects/s recovering 2026-02-23T13:12:57.384 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:12:57.382+0000 7fdd1ca54640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:12:57.384 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:12:57.382+0000 7fdd1ca54640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:12:57.384 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:12:57.481 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:12:57.480+0000 7f5078e34640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:12:57.482 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:12:57.480+0000 7f5078e34640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:12:57.493 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:12:57.494 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:12:57.503 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:12:57.504 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:12:57.514 INFO:teuthology.orchestra.run.trial068.stdout:osd.2: {} 2026-02-23T13:12:57.514 INFO:teuthology.orchestra.run.trial068.stderr:osd.2: osd_enable_op_tracker = '' 2026-02-23T13:12:57.526 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:12:57.526 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:12:57.553 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 79 128 245080 244952 24.2219 3.375 0.00403746 0.0410381 2026-02-23T13:12:58.008 INFO:teuthology.orchestra.run.trial068.stderr:reweighted osd.0 to 1 (10000) 2026-02-23T13:12:58.050 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd reweight 1 1 2026-02-23T13:12:58.194 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:12:58.261 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:12:58 trial068 ceph-mon[38805]: osdmap e157: 4 total, 4 up, 4 in 2026-02-23T13:12:58.261 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:12:58 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2998452619' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:12:58.261 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:12:58 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3094154368' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2026-02-23T13:12:58.261 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:12:58 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3094154368' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2026-02-23T13:12:58.261 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:12:58 trial068 ceph-mon[38805]: osdmap e158: 4 total, 4 up, 4 in 2026-02-23T13:12:58.318 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:12:58.317+0000 7f8ac7ba3640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:12:58.319 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:12:58.317+0000 7f8ac7ba3640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:12:58.554 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T13:12:58.553616+0000 min lat: 0.0010091 max lat: 20.2504 avg lat: 0.0414637 2026-02-23T13:12:58.554 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T13:12:58.554 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 80 128 247043 246915 24.1108 15.3359 0.0294426 0.0414637 2026-02-23T13:12:59.014 INFO:teuthology.orchestra.run.trial068.stderr:reweighted osd.1 to 1 (10000) 2026-02-23T13:12:59.061 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd reweight 2 1 2026-02-23T13:12:59.198 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:12:59.267 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:12:59 trial068 ceph-mon[38805]: 4.d scrub starts 2026-02-23T13:12:59.267 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:12:59 trial068 ceph-mon[38805]: pgmap v300: 17 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 5 active+undersized+remapped, 4 peering, 3 active+clean, 1 active+undersized+degraded; 1.9 GiB data, 6.9 GiB used, 2.7 TiB / 2.7 TiB avail; 18 MiB/s wr, 2.24k op/s; 2423/91158 objects degraded (2.658%); 28 MiB/s, 452 objects/s recovering 2026-02-23T13:12:59.267 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:12:59 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1580778881' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2026-02-23T13:12:59.267 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:12:59 trial068 ceph-mon[38805]: Health check update: Degraded data redundancy: 2423/90528 objects degraded (2.677%), 5 pgs degraded (PG_DEGRADED) 2026-02-23T13:12:59.328 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:12:59.326+0000 7f81fa0e9640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:12:59.328 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:12:59.326+0000 7f81fa0e9640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:12:59.553 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 81 128 250678 250550 24.1637 28.3984 0.0379129 0.0413704 2026-02-23T13:12:59.562 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:12:59.704 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:12:59.828 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:12:59.826+0000 7f50dff4b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:12:59.828 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:12:59.826+0000 7f50dff4b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:12:59.842 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:12:59.842 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:12:59.856 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:12:59.856 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:12:59.872 INFO:teuthology.orchestra.run.trial068.stdout:osd.2: {} 2026-02-23T13:12:59.872 INFO:teuthology.orchestra.run.trial068.stderr:osd.2: osd_enable_op_tracker = '' 2026-02-23T13:12:59.887 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:12:59.887 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:13:00.017 INFO:teuthology.orchestra.run.trial068.stderr:reweighted osd.2 to 1 (10000) 2026-02-23T13:13:00.061 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd reweight 3 1 2026-02-23T13:13:00.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:13:00 trial068 ceph-mon[38805]: 4.e scrub starts 2026-02-23T13:13:00.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:13:00 trial068 ceph-mon[38805]: 4.6 scrub starts 2026-02-23T13:13:00.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:13:00 trial068 ceph-mon[38805]: 4.e scrub ok 2026-02-23T13:13:00.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:13:00 trial068 ceph-mon[38805]: 4.6 scrub ok 2026-02-23T13:13:00.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:13:00 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1580778881' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2026-02-23T13:13:00.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:13:00 trial068 ceph-mon[38805]: osdmap e159: 4 total, 4 up, 4 in 2026-02-23T13:13:00.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:13:00 trial068 ceph-mon[38805]: 4.5 scrub starts 2026-02-23T13:13:00.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:13:00 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2899941949' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2026-02-23T13:13:00.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:13:00 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/4183541957' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:13:00.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:13:00 trial068 ceph-mon[38805]: 4.d scrub ok 2026-02-23T13:13:00.206 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:13:00.455 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:13:00.453+0000 7fd70093b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:13:00.455 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:13:00.454+0000 7fd70093b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:13:00.554 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 82 128 254333 254205 24.2172 28.5547 0.0390248 0.0412819 2026-02-23T13:13:01.019 INFO:teuthology.orchestra.run.trial068.stderr:reweighted osd.3 to 1 (10000) 2026-02-23T13:13:01.067 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2026-02-23T13:13:01.068 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph pg dump --format=json 2026-02-23T13:13:01.254 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:13:01.282 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:13:01 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2899941949' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2026-02-23T13:13:01.282 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:13:01 trial068 ceph-mon[38805]: osdmap e160: 4 total, 4 up, 4 in 2026-02-23T13:13:01.282 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:13:01 trial068 ceph-mon[38805]: 4.5 scrub ok 2026-02-23T13:13:01.282 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:13:01 trial068 ceph-mon[38805]: pgmap v303: 17 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+undersized+remapped, 4 peering, 7 active+clean; 1.9 GiB data, 7.1 GiB used, 2.7 TiB / 2.7 TiB avail; 9.5 MiB/s wr, 1.22k op/s; 530/92355 objects degraded (0.574%); 17 MiB/s, 276 objects/s recovering 2026-02-23T13:13:01.282 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:13:01 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/4032459634' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2026-02-23T13:13:01.379 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:13:01.377+0000 7ff04e94a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:13:01.379 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:13:01.377+0000 7ff04e94a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:13:01.519 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T13:13:01.519 INFO:teuthology.orchestra.run.trial068.stderr:dumped all 2026-02-23T13:13:01.554 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 83 128 258203 258075 24.2897 30.2344 0.0148454 0.041154 2026-02-23T13:13:01.568 INFO:teuthology.orchestra.run.trial068.stdout:{"pg_ready":true,"pg_map":{"version":304,"stamp":"2026-02-23T13:13:01.019024+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2057224736,"num_objects":31385,"num_object_clones":0,"num_object_copies":94155,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31385,"num_whiteouts":0,"num_read":122,"num_read_kb":228,"num_write":251189,"num_write_kb":2010582,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11222,"num_bytes_recovered":734601760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160953,"ondisk_log_size":160953,"up":51,"acting":49,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":51,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":2936012800,"kb_used":7563584,"kb_used_data":5986992,"kb_used_omap":132,"kb_used_meta":1576379,"kb_avail":2928449216,"statfs":{"total":3006477107200,"available":2998731997184,"internally_reserved":0,"allocated":6130679808,"data_stored":6126119489,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":136051,"internal_metadata":1614212237},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[6,5,13,6,22,12,14,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":65429504,"num_objects":999,"num_object_clones":0,"num_object_copies":2997,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2423,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":999,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7987,"num_write_kb":63896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":869,"num_bytes_recovered":56844288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.011014"},"pg_stats":[{"pgid":"4.f","version":"160'16040","reported_seq":17967,"reported_epoch":160,"state":"active+clean","last_fresh":"2026-02-23T13:13:00.496031+0000","last_change":"2026-02-23T13:12:58.015180+0000","last_active":"2026-02-23T13:13:00.496031+0000","last_peered":"2026-02-23T13:13:00.496031+0000","last_clean":"2026-02-23T13:13:00.496031+0000","last_became_active":"2026-02-23T13:12:58.011500+0000","last_became_peered":"2026-02-23T13:12:58.011500+0000","last_unstale":"2026-02-23T13:13:00.496031+0000","last_undegraded":"2026-02-23T13:13:00.496031+0000","last_fullsized":"2026-02-23T13:13:00.496031+0000","mapping_epoch":157,"log_start":"130'6000","ondisk_log_start":"130'6000","created":126,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:11:36.450546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:11:36.450546+0000","last_clean_scrub_stamp":"2026-02-23T13:11:36.450546+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":131399680,"num_objects":2005,"num_object_clones":0,"num_object_copies":6015,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2005,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16040,"num_write_kb":128320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":909,"num_bytes_recovered":59572224,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"159'16032","reported_seq":17577,"reported_epoch":159,"state":"active+clean","last_fresh":"2026-02-23T13:12:59.958820+0000","last_change":"2026-02-23T13:12:58.792221+0000","last_active":"2026-02-23T13:12:59.958820+0000","last_peered":"2026-02-23T13:12:59.958820+0000","last_clean":"2026-02-23T13:12:59.958820+0000","last_became_active":"2026-02-23T13:12:57.011288+0000","last_became_peered":"2026-02-23T13:12:57.011288+0000","last_unstale":"2026-02-23T13:12:59.958820+0000","last_undegraded":"2026-02-23T13:12:59.958820+0000","last_fullsized":"2026-02-23T13:12:59.958820+0000","mapping_epoch":156,"log_start":"130'6000","ondisk_log_start":"130'6000","created":126,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"158'15752","last_scrub_stamp":"2026-02-23T13:12:58.792154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:11:36.450546+0000","last_clean_scrub_stamp":"2026-02-23T13:12:58.792154+0000","objects_scrubbed":1964,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T13:14:17.745746+0000","scrub_duration":465,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":131334144,"num_objects":2004,"num_object_clones":0,"num_object_copies":6012,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2004,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16032,"num_write_kb":128256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":905,"num_bytes_recovered":59301888,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"155'14992","reported_seq":16306,"reported_epoch":155,"state":"peering","last_fresh":"2026-02-23T13:12:55.588966+0000","last_change":"2026-02-23T13:12:56.008533+0000","last_active":"2026-02-23T13:12:56.008533+0000","last_peered":"2026-02-23T13:12:55.588966+0000","last_clean":"2026-02-23T13:12:23.333062+0000","last_became_active":"2026-02-23T13:12:53.415830+0000","last_became_peered":"2026-02-23T13:12:53.415830+0000","last_unstale":"2026-02-23T13:12:55.588966+0000","last_undegraded":"2026-02-23T13:12:55.588966+0000","last_fullsized":"2026-02-23T13:12:53.001216+0000","mapping_epoch":156,"log_start":"130'4900","ondisk_log_start":"130'4900","created":126,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:11:36.450546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:11:36.450546+0000","last_clean_scrub_stamp":"2026-02-23T13:11:36.450546+0000","objects_scrubbed":0,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":122814464,"num_objects":1874,"num_object_clones":0,"num_object_copies":5622,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1874,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14992,"num_write_kb":119936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":610,"num_bytes_recovered":39763968,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,2],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"156'15816","reported_seq":17818,"reported_epoch":156,"state":"active+undersized+remapped","last_fresh":"2026-02-23T13:12:56.408850+0000","last_change":"2026-02-23T13:12:56.408965+0000","last_active":"2026-02-23T13:12:56.408850+0000","last_peered":"2026-02-23T13:12:56.408850+0000","last_clean":"2026-02-23T13:11:53.292743+0000","last_became_active":"2026-02-23T13:12:53.421310+0000","last_became_peered":"2026-02-23T13:12:53.421310+0000","last_unstale":"2026-02-23T13:12:56.408850+0000","last_undegraded":"2026-02-23T13:12:53.011461+0000","last_fullsized":"2026-02-23T13:12:53.001258+0000","mapping_epoch":153,"log_start":"130'5800","ondisk_log_start":"130'5800","created":126,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:11:36.450546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:11:36.450546+0000","last_clean_scrub_stamp":"2026-02-23T13:11:36.450546+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":129564672,"num_objects":1977,"num_object_clones":0,"num_object_copies":5931,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1977,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15816,"num_write_kb":126528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":957,"num_bytes_recovered":62480384,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1],"avail_no_missing":["3","1","2"],"object_location_counts":[{"shards":"1,3","objects":1977}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"160'15982","reported_seq":17945,"reported_epoch":160,"state":"active+clean","last_fresh":"2026-02-23T13:13:00.488240+0000","last_change":"2026-02-23T13:12:58.017947+0000","last_active":"2026-02-23T13:13:00.488240+0000","last_peered":"2026-02-23T13:13:00.488240+0000","last_clean":"2026-02-23T13:13:00.488240+0000","last_became_active":"2026-02-23T13:12:58.011138+0000","last_became_peered":"2026-02-23T13:12:58.011138+0000","last_unstale":"2026-02-23T13:13:00.488240+0000","last_undegraded":"2026-02-23T13:13:00.488240+0000","last_fullsized":"2026-02-23T13:13:00.488240+0000","mapping_epoch":157,"log_start":"130'5900","ondisk_log_start":"130'5900","created":126,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:11:36.450546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:11:36.450546+0000","last_clean_scrub_stamp":"2026-02-23T13:11:36.450546+0000","objects_scrubbed":0,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":130924544,"num_objects":1998,"num_object_clones":0,"num_object_copies":5994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1998,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15982,"num_write_kb":127856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":926,"num_bytes_recovered":60604416,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"159'15256","reported_seq":16841,"reported_epoch":159,"state":"active+clean","last_fresh":"2026-02-23T13:13:00.002267+0000","last_change":"2026-02-23T13:12:57.015183+0000","last_active":"2026-02-23T13:13:00.002267+0000","last_peered":"2026-02-23T13:13:00.002267+0000","last_clean":"2026-02-23T13:13:00.002267+0000","last_became_active":"2026-02-23T13:12:57.011012+0000","last_became_peered":"2026-02-23T13:12:57.011012+0000","last_unstale":"2026-02-23T13:13:00.002267+0000","last_undegraded":"2026-02-23T13:13:00.002267+0000","last_fullsized":"2026-02-23T13:13:00.002267+0000","mapping_epoch":156,"log_start":"130'5200","ondisk_log_start":"130'5200","created":126,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:11:36.450546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:11:36.450546+0000","last_clean_scrub_stamp":"2026-02-23T13:11:36.450546+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":124977152,"num_objects":1907,"num_object_clones":0,"num_object_copies":5721,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1907,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15256,"num_write_kb":122048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":896,"num_bytes_recovered":58646528,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"159'16040","reported_seq":17880,"reported_epoch":159,"state":"active+clean","last_fresh":"2026-02-23T13:12:59.978677+0000","last_change":"2026-02-23T13:12:57.015222+0000","last_active":"2026-02-23T13:12:59.978677+0000","last_peered":"2026-02-23T13:12:59.978677+0000","last_clean":"2026-02-23T13:12:59.978677+0000","last_became_active":"2026-02-23T13:12:57.011352+0000","last_became_peered":"2026-02-23T13:12:57.011352+0000","last_unstale":"2026-02-23T13:12:59.978677+0000","last_undegraded":"2026-02-23T13:12:59.978677+0000","last_fullsized":"2026-02-23T13:12:59.978677+0000","mapping_epoch":156,"log_start":"130'6000","ondisk_log_start":"130'6000","created":126,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:11:36.450546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:11:36.450546+0000","last_clean_scrub_stamp":"2026-02-23T13:11:36.450546+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":131399680,"num_objects":2005,"num_object_clones":0,"num_object_copies":6015,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2005,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16040,"num_write_kb":128320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":947,"num_bytes_recovered":62062592,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"155'15192","reported_seq":16543,"reported_epoch":155,"state":"peering","last_fresh":"2026-02-23T13:12:55.911303+0000","last_change":"2026-02-23T13:12:56.008414+0000","last_active":"2026-02-23T13:12:56.008414+0000","last_peered":"2026-02-23T13:12:55.911303+0000","last_clean":"2026-02-23T13:12:23.334565+0000","last_became_active":"2026-02-23T13:12:53.418187+0000","last_became_peered":"2026-02-23T13:12:53.418187+0000","last_unstale":"2026-02-23T13:12:55.911303+0000","last_undegraded":"2026-02-23T13:12:55.911303+0000","last_fullsized":"2026-02-23T13:12:53.001219+0000","mapping_epoch":156,"log_start":"130'5100","ondisk_log_start":"130'5100","created":126,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:11:36.450546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:11:36.450546+0000","last_clean_scrub_stamp":"2026-02-23T13:11:36.450546+0000","objects_scrubbed":0,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":124452864,"num_objects":1899,"num_object_clones":0,"num_object_copies":5697,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1899,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15192,"num_write_kb":121536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":636,"num_bytes_recovered":41402368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,2],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.7","version":"160'15570","reported_seq":17365,"reported_epoch":160,"state":"active+clean","last_fresh":"2026-02-23T13:13:00.498627+0000","last_change":"2026-02-23T13:12:56.015592+0000","last_active":"2026-02-23T13:13:00.498627+0000","last_peered":"2026-02-23T13:13:00.498627+0000","last_clean":"2026-02-23T13:13:00.498627+0000","last_became_active":"2026-02-23T13:12:56.011425+0000","last_became_peered":"2026-02-23T13:12:56.011425+0000","last_unstale":"2026-02-23T13:13:00.498627+0000","last_undegraded":"2026-02-23T13:13:00.498627+0000","last_fullsized":"2026-02-23T13:13:00.498627+0000","mapping_epoch":155,"log_start":"130'5500","ondisk_log_start":"130'5500","created":126,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:11:36.450546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:11:36.450546+0000","last_clean_scrub_stamp":"2026-02-23T13:11:36.450546+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":127549440,"num_objects":1947,"num_object_clones":0,"num_object_copies":5841,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1947,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15570,"num_write_kb":124560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":851,"num_bytes_recovered":55615488,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"160'15894","reported_seq":17904,"reported_epoch":160,"state":"active+clean","last_fresh":"2026-02-23T13:13:00.488458+0000","last_change":"2026-02-23T13:12:58.990117+0000","last_active":"2026-02-23T13:13:00.488458+0000","last_peered":"2026-02-23T13:13:00.488458+0000","last_clean":"2026-02-23T13:13:00.488458+0000","last_became_active":"2026-02-23T13:12:57.012274+0000","last_became_peered":"2026-02-23T13:12:57.012274+0000","last_unstale":"2026-02-23T13:13:00.488458+0000","last_undegraded":"2026-02-23T13:13:00.488458+0000","last_fullsized":"2026-02-23T13:13:00.488458+0000","mapping_epoch":156,"log_start":"130'5800","ondisk_log_start":"130'5800","created":126,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"158'15592","last_scrub_stamp":"2026-02-23T13:12:58.990065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:11:36.450546+0000","last_clean_scrub_stamp":"2026-02-23T13:12:58.990065+0000","objects_scrubbed":1945,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T13:14:01.221621+0000","scrub_duration":369,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":130203648,"num_objects":1987,"num_object_clones":0,"num_object_copies":5961,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1987,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15894,"num_write_kb":127152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":938,"num_bytes_recovered":61456384,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"160'15824","reported_seq":16435,"reported_epoch":160,"state":"active+clean","last_fresh":"2026-02-23T13:13:00.437531+0000","last_change":"2026-02-23T13:12:51.932089+0000","last_active":"2026-02-23T13:13:00.437531+0000","last_peered":"2026-02-23T13:13:00.437531+0000","last_clean":"2026-02-23T13:13:00.437531+0000","last_became_active":"2026-02-23T13:12:47.496034+0000","last_became_peered":"2026-02-23T13:12:47.496034+0000","last_unstale":"2026-02-23T13:13:00.437531+0000","last_undegraded":"2026-02-23T13:13:00.437531+0000","last_fullsized":"2026-02-23T13:13:00.437531+0000","mapping_epoch":148,"log_start":"130'5800","ondisk_log_start":"130'5800","created":126,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"151'14880","last_scrub_stamp":"2026-02-23T13:12:51.932039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:11:36.450546+0000","last_clean_scrub_stamp":"2026-02-23T13:12:51.932039+0000","objects_scrubbed":1860,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T13:14:01.557089+0000","scrub_duration":94,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":129630208,"num_objects":1978,"num_object_clones":0,"num_object_copies":5934,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1978,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15824,"num_write_kb":126592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":279,"num_bytes_recovered":18284544,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"160'15752","reported_seq":17120,"reported_epoch":160,"state":"active+clean","last_fresh":"2026-02-23T13:13:00.447474+0000","last_change":"2026-02-23T13:12:57.018841+0000","last_active":"2026-02-23T13:13:00.447474+0000","last_peered":"2026-02-23T13:13:00.447474+0000","last_clean":"2026-02-23T13:13:00.447474+0000","last_became_active":"2026-02-23T13:12:57.012156+0000","last_became_peered":"2026-02-23T13:12:57.012156+0000","last_unstale":"2026-02-23T13:13:00.447474+0000","last_undegraded":"2026-02-23T13:13:00.447474+0000","last_fullsized":"2026-02-23T13:13:00.447474+0000","mapping_epoch":156,"log_start":"130'5700","ondisk_log_start":"130'5700","created":126,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:11:36.450546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:11:36.450546+0000","last_clean_scrub_stamp":"2026-02-23T13:11:36.450546+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T13:13:05.429853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":129040384,"num_objects":1969,"num_object_clones":0,"num_object_copies":5907,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1969,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15752,"num_write_kb":126016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":628,"num_bytes_recovered":41156608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"160'16400","reported_seq":16673,"reported_epoch":160,"state":"active+clean","last_fresh":"2026-02-23T13:13:00.471910+0000","last_change":"2026-02-23T13:12:47.499686+0000","last_active":"2026-02-23T13:13:00.471910+0000","last_peered":"2026-02-23T13:13:00.471910+0000","last_clean":"2026-02-23T13:13:00.471910+0000","last_became_active":"2026-02-23T13:12:47.496124+0000","last_became_peered":"2026-02-23T13:12:47.496124+0000","last_unstale":"2026-02-23T13:13:00.471910+0000","last_undegraded":"2026-02-23T13:13:00.471910+0000","last_fullsized":"2026-02-23T13:13:00.471910+0000","mapping_epoch":148,"log_start":"130'6300","ondisk_log_start":"130'6300","created":126,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:11:36.450546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:11:36.450546+0000","last_clean_scrub_stamp":"2026-02-23T13:11:36.450546+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":134348800,"num_objects":2050,"num_object_clones":0,"num_object_copies":6150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2050,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16400,"num_write_kb":131200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":294,"num_bytes_recovered":19218432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1","version":"159'15504","reported_seq":15930,"reported_epoch":159,"state":"active+clean","last_fresh":"2026-02-23T13:12:59.969972+0000","last_change":"2026-02-23T13:12:56.013478+0000","last_active":"2026-02-23T13:12:59.969972+0000","last_peered":"2026-02-23T13:12:59.969972+0000","last_clean":"2026-02-23T13:12:59.969972+0000","last_became_active":"2026-02-23T13:12:56.010005+0000","last_became_peered":"2026-02-23T13:12:56.010005+0000","last_unstale":"2026-02-23T13:12:59.969972+0000","last_undegraded":"2026-02-23T13:12:59.969972+0000","last_fullsized":"2026-02-23T13:12:59.969972+0000","mapping_epoch":155,"log_start":"130'5500","ondisk_log_start":"130'5500","created":126,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:11:36.450546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:11:36.450546+0000","last_clean_scrub_stamp":"2026-02-23T13:11:36.450546+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":127008768,"num_objects":1938,"num_object_clones":0,"num_object_copies":5814,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1938,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15504,"num_write_kb":124032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":175,"num_bytes_recovered":11362304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.0","version":"160'15920","reported_seq":17308,"reported_epoch":160,"state":"active+clean","last_fresh":"2026-02-23T13:13:00.485879+0000","last_change":"2026-02-23T13:12:56.013162+0000","last_active":"2026-02-23T13:13:00.485879+0000","last_peered":"2026-02-23T13:13:00.485879+0000","last_clean":"2026-02-23T13:13:00.485879+0000","last_became_active":"2026-02-23T13:12:56.010072+0000","last_became_peered":"2026-02-23T13:12:56.010072+0000","last_unstale":"2026-02-23T13:13:00.485879+0000","last_undegraded":"2026-02-23T13:13:00.485879+0000","last_fullsized":"2026-02-23T13:13:00.485879+0000","mapping_epoch":155,"log_start":"130'5900","ondisk_log_start":"130'5900","created":126,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:11:36.450546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:11:36.450546+0000","last_clean_scrub_stamp":"2026-02-23T13:11:36.450546+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T13:13:06.181031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":130416640,"num_objects":1990,"num_object_clones":0,"num_object_copies":5970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1990,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15920,"num_write_kb":127360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":638,"num_bytes_recovered":41754624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,3],"acting":[0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.5","version":"156'14840","reported_seq":16201,"reported_epoch":156,"state":"active+undersized+remapped","last_fresh":"2026-02-23T13:12:56.088356+0000","last_change":"2026-02-23T13:12:56.068735+0000","last_active":"2026-02-23T13:12:56.088356+0000","last_peered":"2026-02-23T13:12:56.088356+0000","last_clean":"2026-02-23T13:12:23.256807+0000","last_became_active":"2026-02-23T13:12:53.415848+0000","last_became_peered":"2026-02-23T13:12:53.415848+0000","last_unstale":"2026-02-23T13:12:56.088356+0000","last_undegraded":"2026-02-23T13:12:56.088356+0000","last_fullsized":"2026-02-23T13:12:53.001243+0000","mapping_epoch":153,"log_start":"130'4800","ondisk_log_start":"130'4800","created":126,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:11:36.450546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:11:36.450546+0000","last_clean_scrub_stamp":"2026-02-23T13:11:36.450546+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":121569280,"num_objects":1855,"num_object_clones":0,"num_object_copies":5565,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1855,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14840,"num_write_kb":118720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":631,"num_bytes_recovered":41328640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,0],"avail_no_missing":["3","0","2"],"object_location_counts":[{"shards":"0,3","objects":1855}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"28'99","reported_seq":333,"reported_epoch":156,"state":"active+clean","last_fresh":"2026-02-23T13:12:56.008521+0000","last_change":"2026-02-23T13:12:52.005066+0000","last_active":"2026-02-23T13:12:56.008521+0000","last_peered":"2026-02-23T13:12:56.008521+0000","last_clean":"2026-02-23T13:12:56.008521+0000","last_became_active":"2026-02-23T13:12:52.004933+0000","last_became_peered":"2026-02-23T13:12:52.004933+0000","last_unstale":"2026-02-23T13:12:56.008521+0000","last_undegraded":"2026-02-23T13:12:56.008521+0000","last_fullsized":"2026-02-23T13:12:56.008521+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'99","last_scrub_stamp":"2026-02-23T13:11:39.969814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:07:05.619208+0000","last_clean_scrub_stamp":"2026-02-23T13:11:39.969814+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T13:12:59.396325+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":122,"num_read_kb":228,"num_write":135,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":16,"stat_sum":{"num_bytes":2056634368,"num_objects":31383,"num_object_clones":0,"num_object_copies":94149,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31383,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":251054,"num_write_kb":2008432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11220,"num_bytes_recovered":734011392,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":6172606464,"data_stored":6172606464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160854,"ondisk_log_size":160854,"up":48,"acting":46,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":122,"num_read_kb":228,"num_write":135,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":80,"seq":343597383718,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":734003200,"kb_used":2240680,"kb_used_data":1467668,"kb_used_omap":59,"kb_used_meta":772932,"kb_avail":731762520,"statfs":{"total":751619276800,"available":749324820480,"internally_reserved":0,"allocated":1502892032,"data_stored":1501768693,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60511,"internal_metadata":791483297},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":152,"seq":652835028995,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":734003200,"kb_used":1848940,"kb_used_data":1748952,"kb_used_omap":14,"kb_used_meta":99953,"kb_avail":732154260,"statfs":{"total":751619276800,"available":749725962240,"internally_reserved":0,"allocated":1790926848,"data_stored":1789780004,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15212,"internal_metadata":102352020},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,2,1,0,6,2,6],"upper_bound":128},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":144,"seq":618475290629,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":734003200,"kb_used":1408688,"kb_used_data":1266332,"kb_used_omap":18,"kb_used_meta":142317,"kb_avail":732594512,"statfs":{"total":751619276800,"available":750176780288,"internally_reserved":0,"allocated":1296723968,"data_stored":1295580676,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18768,"internal_metadata":145733296},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":106,"seq":455266533401,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":734003200,"kb_used":2065276,"kb_used_data":1504040,"kb_used_omap":40,"kb_used_meta":561175,"kb_avail":731937924,"statfs":{"total":751619276800,"available":749504434176,"internally_reserved":0,"allocated":1540136960,"data_stored":1538990116,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":41560,"internal_metadata":574643624},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,3,12,6,16,10,8,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1556766720,"data_stored":1556766720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1309392896,"data_stored":1309392896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1804623872,"data_stored":1804623872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1501822976,"data_stored":1501822976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-23T13:13:01.570 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph pg dump --format=json 2026-02-23T13:13:01.725 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:13:01.846 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:13:01.844+0000 7efd8aa95640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:13:01.846 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:13:01.844+0000 7efd8aa95640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:13:01.933 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:13:01.994 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T13:13:01.994 INFO:teuthology.orchestra.run.trial068.stderr:dumped all 2026-02-23T13:13:02.045 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:13:02 trial068 ceph-mon[38805]: 4.7 scrub starts 2026-02-23T13:13:02.045 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:13:02 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/4032459634' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2026-02-23T13:13:02.045 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:13:02 trial068 ceph-mon[38805]: osdmap e161: 4 total, 4 up, 4 in 2026-02-23T13:13:02.045 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:13:02 trial068 ceph-mon[38805]: 4.7 scrub ok 2026-02-23T13:13:02.045 INFO:teuthology.orchestra.run.trial068.stdout:{"pg_ready":true,"pg_map":{"version":304,"stamp":"2026-02-23T13:13:01.019024+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2057224736,"num_objects":31385,"num_object_clones":0,"num_object_copies":94155,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31385,"num_whiteouts":0,"num_read":122,"num_read_kb":228,"num_write":251189,"num_write_kb":2010582,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11222,"num_bytes_recovered":734601760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160953,"ondisk_log_size":160953,"up":51,"acting":49,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":51,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":2936012800,"kb_used":7563584,"kb_used_data":5986992,"kb_used_omap":132,"kb_used_meta":1576379,"kb_avail":2928449216,"statfs":{"total":3006477107200,"available":2998731997184,"internally_reserved":0,"allocated":6130679808,"data_stored":6126119489,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":136051,"internal_metadata":1614212237},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[6,5,13,6,22,12,14,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":65429504,"num_objects":999,"num_object_clones":0,"num_object_copies":2997,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2423,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":999,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7987,"num_write_kb":63896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":869,"num_bytes_recovered":56844288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.011014"},"pg_stats":[{"pgid":"4.f","version":"160'16040","reported_seq":17967,"reported_epoch":160,"state":"active+clean","last_fresh":"2026-02-23T13:13:00.496031+0000","last_change":"2026-02-23T13:12:58.015180+0000","last_active":"2026-02-23T13:13:00.496031+0000","last_peered":"2026-02-23T13:13:00.496031+0000","last_clean":"2026-02-23T13:13:00.496031+0000","last_became_active":"2026-02-23T13:12:58.011500+0000","last_became_peered":"2026-02-23T13:12:58.011500+0000","last_unstale":"2026-02-23T13:13:00.496031+0000","last_undegraded":"2026-02-23T13:13:00.496031+0000","last_fullsized":"2026-02-23T13:13:00.496031+0000","mapping_epoch":157,"log_start":"130'6000","ondisk_log_start":"130'6000","created":126,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:11:36.450546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:11:36.450546+0000","last_clean_scrub_stamp":"2026-02-23T13:11:36.450546+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":131399680,"num_objects":2005,"num_object_clones":0,"num_object_copies":6015,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2005,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16040,"num_write_kb":128320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":909,"num_bytes_recovered":59572224,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"159'16032","reported_seq":17577,"reported_epoch":159,"state":"active+clean","last_fresh":"2026-02-23T13:12:59.958820+0000","last_change":"2026-02-23T13:12:58.792221+0000","last_active":"2026-02-23T13:12:59.958820+0000","last_peered":"2026-02-23T13:12:59.958820+0000","last_clean":"2026-02-23T13:12:59.958820+0000","last_became_active":"2026-02-23T13:12:57.011288+0000","last_became_peered":"2026-02-23T13:12:57.011288+0000","last_unstale":"2026-02-23T13:12:59.958820+0000","last_undegraded":"2026-02-23T13:12:59.958820+0000","last_fullsized":"2026-02-23T13:12:59.958820+0000","mapping_epoch":156,"log_start":"130'6000","ondisk_log_start":"130'6000","created":126,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"158'15752","last_scrub_stamp":"2026-02-23T13:12:58.792154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:11:36.450546+0000","last_clean_scrub_stamp":"2026-02-23T13:12:58.792154+0000","objects_scrubbed":1964,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T13:14:17.745746+0000","scrub_duration":465,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":131334144,"num_objects":2004,"num_object_clones":0,"num_object_copies":6012,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2004,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16032,"num_write_kb":128256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":905,"num_bytes_recovered":59301888,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"155'14992","reported_seq":16306,"reported_epoch":155,"state":"peering","last_fresh":"2026-02-23T13:12:55.588966+0000","last_change":"2026-02-23T13:12:56.008533+0000","last_active":"2026-02-23T13:12:56.008533+0000","last_peered":"2026-02-23T13:12:55.588966+0000","last_clean":"2026-02-23T13:12:23.333062+0000","last_became_active":"2026-02-23T13:12:53.415830+0000","last_became_peered":"2026-02-23T13:12:53.415830+0000","last_unstale":"2026-02-23T13:12:55.588966+0000","last_undegraded":"2026-02-23T13:12:55.588966+0000","last_fullsized":"2026-02-23T13:12:53.001216+0000","mapping_epoch":156,"log_start":"130'4900","ondisk_log_start":"130'4900","created":126,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:11:36.450546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:11:36.450546+0000","last_clean_scrub_stamp":"2026-02-23T13:11:36.450546+0000","objects_scrubbed":0,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":122814464,"num_objects":1874,"num_object_clones":0,"num_object_copies":5622,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1874,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14992,"num_write_kb":119936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":610,"num_bytes_recovered":39763968,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,2],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"156'15816","reported_seq":17818,"reported_epoch":156,"state":"active+undersized+remapped","last_fresh":"2026-02-23T13:12:56.408850+0000","last_change":"2026-02-23T13:12:56.408965+0000","last_active":"2026-02-23T13:12:56.408850+0000","last_peered":"2026-02-23T13:12:56.408850+0000","last_clean":"2026-02-23T13:11:53.292743+0000","last_became_active":"2026-02-23T13:12:53.421310+0000","last_became_peered":"2026-02-23T13:12:53.421310+0000","last_unstale":"2026-02-23T13:12:56.408850+0000","last_undegraded":"2026-02-23T13:12:53.011461+0000","last_fullsized":"2026-02-23T13:12:53.001258+0000","mapping_epoch":153,"log_start":"130'5800","ondisk_log_start":"130'5800","created":126,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:11:36.450546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:11:36.450546+0000","last_clean_scrub_stamp":"2026-02-23T13:11:36.450546+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":129564672,"num_objects":1977,"num_object_clones":0,"num_object_copies":5931,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1977,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15816,"num_write_kb":126528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":957,"num_bytes_recovered":62480384,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1],"avail_no_missing":["3","1","2"],"object_location_counts":[{"shards":"1,3","objects":1977}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"160'15982","reported_seq":17945,"reported_epoch":160,"state":"active+clean","last_fresh":"2026-02-23T13:13:00.488240+0000","last_change":"2026-02-23T13:12:58.017947+0000","last_active":"2026-02-23T13:13:00.488240+0000","last_peered":"2026-02-23T13:13:00.488240+0000","last_clean":"2026-02-23T13:13:00.488240+0000","last_became_active":"2026-02-23T13:12:58.011138+0000","last_became_peered":"2026-02-23T13:12:58.011138+0000","last_unstale":"2026-02-23T13:13:00.488240+0000","last_undegraded":"2026-02-23T13:13:00.488240+0000","last_fullsized":"2026-02-23T13:13:00.488240+0000","mapping_epoch":157,"log_start":"130'5900","ondisk_log_start":"130'5900","created":126,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:11:36.450546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:11:36.450546+0000","last_clean_scrub_stamp":"2026-02-23T13:11:36.450546+0000","objects_scrubbed":0,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":130924544,"num_objects":1998,"num_object_clones":0,"num_object_copies":5994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1998,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15982,"num_write_kb":127856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":926,"num_bytes_recovered":60604416,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"159'15256","reported_seq":16841,"reported_epoch":159,"state":"active+clean","last_fresh":"2026-02-23T13:13:00.002267+0000","last_change":"2026-02-23T13:12:57.015183+0000","last_active":"2026-02-23T13:13:00.002267+0000","last_peered":"2026-02-23T13:13:00.002267+0000","last_clean":"2026-02-23T13:13:00.002267+0000","last_became_active":"2026-02-23T13:12:57.011012+0000","last_became_peered":"2026-02-23T13:12:57.011012+0000","last_unstale":"2026-02-23T13:13:00.002267+0000","last_undegraded":"2026-02-23T13:13:00.002267+0000","last_fullsized":"2026-02-23T13:13:00.002267+0000","mapping_epoch":156,"log_start":"130'5200","ondisk_log_start":"130'5200","created":126,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:11:36.450546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:11:36.450546+0000","last_clean_scrub_stamp":"2026-02-23T13:11:36.450546+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":124977152,"num_objects":1907,"num_object_clones":0,"num_object_copies":5721,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1907,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15256,"num_write_kb":122048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":896,"num_bytes_recovered":58646528,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"159'16040","reported_seq":17880,"reported_epoch":159,"state":"active+clean","last_fresh":"2026-02-23T13:12:59.978677+0000","last_change":"2026-02-23T13:12:57.015222+0000","last_active":"2026-02-23T13:12:59.978677+0000","last_peered":"2026-02-23T13:12:59.978677+0000","last_clean":"2026-02-23T13:12:59.978677+0000","last_became_active":"2026-02-23T13:12:57.011352+0000","last_became_peered":"2026-02-23T13:12:57.011352+0000","last_unstale":"2026-02-23T13:12:59.978677+0000","last_undegraded":"2026-02-23T13:12:59.978677+0000","last_fullsized":"2026-02-23T13:12:59.978677+0000","mapping_epoch":156,"log_start":"130'6000","ondisk_log_start":"130'6000","created":126,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:11:36.450546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:11:36.450546+0000","last_clean_scrub_stamp":"2026-02-23T13:11:36.450546+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":131399680,"num_objects":2005,"num_object_clones":0,"num_object_copies":6015,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2005,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16040,"num_write_kb":128320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":947,"num_bytes_recovered":62062592,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"155'15192","reported_seq":16543,"reported_epoch":155,"state":"peering","last_fresh":"2026-02-23T13:12:55.911303+0000","last_change":"2026-02-23T13:12:56.008414+0000","last_active":"2026-02-23T13:12:56.008414+0000","last_peered":"2026-02-23T13:12:55.911303+0000","last_clean":"2026-02-23T13:12:23.334565+0000","last_became_active":"2026-02-23T13:12:53.418187+0000","last_became_peered":"2026-02-23T13:12:53.418187+0000","last_unstale":"2026-02-23T13:12:55.911303+0000","last_undegraded":"2026-02-23T13:12:55.911303+0000","last_fullsized":"2026-02-23T13:12:53.001219+0000","mapping_epoch":156,"log_start":"130'5100","ondisk_log_start":"130'5100","created":126,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:11:36.450546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:11:36.450546+0000","last_clean_scrub_stamp":"2026-02-23T13:11:36.450546+0000","objects_scrubbed":0,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":124452864,"num_objects":1899,"num_object_clones":0,"num_object_copies":5697,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1899,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15192,"num_write_kb":121536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":636,"num_bytes_recovered":41402368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,2],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.7","version":"160'15570","reported_seq":17365,"reported_epoch":160,"state":"active+clean","last_fresh":"2026-02-23T13:13:00.498627+0000","last_change":"2026-02-23T13:12:56.015592+0000","last_active":"2026-02-23T13:13:00.498627+0000","last_peered":"2026-02-23T13:13:00.498627+0000","last_clean":"2026-02-23T13:13:00.498627+0000","last_became_active":"2026-02-23T13:12:56.011425+0000","last_became_peered":"2026-02-23T13:12:56.011425+0000","last_unstale":"2026-02-23T13:13:00.498627+0000","last_undegraded":"2026-02-23T13:13:00.498627+0000","last_fullsized":"2026-02-23T13:13:00.498627+0000","mapping_epoch":155,"log_start":"130'5500","ondisk_log_start":"130'5500","created":126,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:11:36.450546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:11:36.450546+0000","last_clean_scrub_stamp":"2026-02-23T13:11:36.450546+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":127549440,"num_objects":1947,"num_object_clones":0,"num_object_copies":5841,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1947,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15570,"num_write_kb":124560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":851,"num_bytes_recovered":55615488,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"160'15894","reported_seq":17904,"reported_epoch":160,"state":"active+clean","last_fresh":"2026-02-23T13:13:00.488458+0000","last_change":"2026-02-23T13:12:58.990117+0000","last_active":"2026-02-23T13:13:00.488458+0000","last_peered":"2026-02-23T13:13:00.488458+0000","last_clean":"2026-02-23T13:13:00.488458+0000","last_became_active":"2026-02-23T13:12:57.012274+0000","last_became_peered":"2026-02-23T13:12:57.012274+0000","last_unstale":"2026-02-23T13:13:00.488458+0000","last_undegraded":"2026-02-23T13:13:00.488458+0000","last_fullsized":"2026-02-23T13:13:00.488458+0000","mapping_epoch":156,"log_start":"130'5800","ondisk_log_start":"130'5800","created":126,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"158'15592","last_scrub_stamp":"2026-02-23T13:12:58.990065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:11:36.450546+0000","last_clean_scrub_stamp":"2026-02-23T13:12:58.990065+0000","objects_scrubbed":1945,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T13:14:01.221621+0000","scrub_duration":369,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":130203648,"num_objects":1987,"num_object_clones":0,"num_object_copies":5961,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1987,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15894,"num_write_kb":127152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":938,"num_bytes_recovered":61456384,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"160'15824","reported_seq":16435,"reported_epoch":160,"state":"active+clean","last_fresh":"2026-02-23T13:13:00.437531+0000","last_change":"2026-02-23T13:12:51.932089+0000","last_active":"2026-02-23T13:13:00.437531+0000","last_peered":"2026-02-23T13:13:00.437531+0000","last_clean":"2026-02-23T13:13:00.437531+0000","last_became_active":"2026-02-23T13:12:47.496034+0000","last_became_peered":"2026-02-23T13:12:47.496034+0000","last_unstale":"2026-02-23T13:13:00.437531+0000","last_undegraded":"2026-02-23T13:13:00.437531+0000","last_fullsized":"2026-02-23T13:13:00.437531+0000","mapping_epoch":148,"log_start":"130'5800","ondisk_log_start":"130'5800","created":126,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"151'14880","last_scrub_stamp":"2026-02-23T13:12:51.932039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:11:36.450546+0000","last_clean_scrub_stamp":"2026-02-23T13:12:51.932039+0000","objects_scrubbed":1860,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T13:14:01.557089+0000","scrub_duration":94,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":129630208,"num_objects":1978,"num_object_clones":0,"num_object_copies":5934,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1978,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15824,"num_write_kb":126592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":279,"num_bytes_recovered":18284544,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"160'15752","reported_seq":17120,"reported_epoch":160,"state":"active+clean","last_fresh":"2026-02-23T13:13:00.447474+0000","last_change":"2026-02-23T13:12:57.018841+0000","last_active":"2026-02-23T13:13:00.447474+0000","last_peered":"2026-02-23T13:13:00.447474+0000","last_clean":"2026-02-23T13:13:00.447474+0000","last_became_active":"2026-02-23T13:12:57.012156+0000","last_became_peered":"2026-02-23T13:12:57.012156+0000","last_unstale":"2026-02-23T13:13:00.447474+0000","last_undegraded":"2026-02-23T13:13:00.447474+0000","last_fullsized":"2026-02-23T13:13:00.447474+0000","mapping_epoch":156,"log_start":"130'5700","ondisk_log_start":"130'5700","created":126,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:11:36.450546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:11:36.450546+0000","last_clean_scrub_stamp":"2026-02-23T13:11:36.450546+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T13:13:05.429853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":129040384,"num_objects":1969,"num_object_clones":0,"num_object_copies":5907,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1969,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15752,"num_write_kb":126016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":628,"num_bytes_recovered":41156608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"160'16400","reported_seq":16673,"reported_epoch":160,"state":"active+clean","last_fresh":"2026-02-23T13:13:00.471910+0000","last_change":"2026-02-23T13:12:47.499686+0000","last_active":"2026-02-23T13:13:00.471910+0000","last_peered":"2026-02-23T13:13:00.471910+0000","last_clean":"2026-02-23T13:13:00.471910+0000","last_became_active":"2026-02-23T13:12:47.496124+0000","last_became_peered":"2026-02-23T13:12:47.496124+0000","last_unstale":"2026-02-23T13:13:00.471910+0000","last_undegraded":"2026-02-23T13:13:00.471910+0000","last_fullsized":"2026-02-23T13:13:00.471910+0000","mapping_epoch":148,"log_start":"130'6300","ondisk_log_start":"130'6300","created":126,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:11:36.450546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:11:36.450546+0000","last_clean_scrub_stamp":"2026-02-23T13:11:36.450546+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":134348800,"num_objects":2050,"num_object_clones":0,"num_object_copies":6150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2050,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16400,"num_write_kb":131200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":294,"num_bytes_recovered":19218432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1","version":"159'15504","reported_seq":15930,"reported_epoch":159,"state":"active+clean","last_fresh":"2026-02-23T13:12:59.969972+0000","last_change":"2026-02-23T13:12:56.013478+0000","last_active":"2026-02-23T13:12:59.969972+0000","last_peered":"2026-02-23T13:12:59.969972+0000","last_clean":"2026-02-23T13:12:59.969972+0000","last_became_active":"2026-02-23T13:12:56.010005+0000","last_became_peered":"2026-02-23T13:12:56.010005+0000","last_unstale":"2026-02-23T13:12:59.969972+0000","last_undegraded":"2026-02-23T13:12:59.969972+0000","last_fullsized":"2026-02-23T13:12:59.969972+0000","mapping_epoch":155,"log_start":"130'5500","ondisk_log_start":"130'5500","created":126,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:11:36.450546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:11:36.450546+0000","last_clean_scrub_stamp":"2026-02-23T13:11:36.450546+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":127008768,"num_objects":1938,"num_object_clones":0,"num_object_copies":5814,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1938,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15504,"num_write_kb":124032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":175,"num_bytes_recovered":11362304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.0","version":"160'15920","reported_seq":17308,"reported_epoch":160,"state":"active+clean","last_fresh":"2026-02-23T13:13:00.485879+0000","last_change":"2026-02-23T13:12:56.013162+0000","last_active":"2026-02-23T13:13:00.485879+0000","last_peered":"2026-02-23T13:13:00.485879+0000","last_clean":"2026-02-23T13:13:00.485879+0000","last_became_active":"2026-02-23T13:12:56.010072+0000","last_became_peered":"2026-02-23T13:12:56.010072+0000","last_unstale":"2026-02-23T13:13:00.485879+0000","last_undegraded":"2026-02-23T13:13:00.485879+0000","last_fullsized":"2026-02-23T13:13:00.485879+0000","mapping_epoch":155,"log_start":"130'5900","ondisk_log_start":"130'5900","created":126,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:11:36.450546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:11:36.450546+0000","last_clean_scrub_stamp":"2026-02-23T13:11:36.450546+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T13:13:06.181031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":130416640,"num_objects":1990,"num_object_clones":0,"num_object_copies":5970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1990,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15920,"num_write_kb":127360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":638,"num_bytes_recovered":41754624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,3],"acting":[0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.5","version":"156'14840","reported_seq":16201,"reported_epoch":156,"state":"active+undersized+remapped","last_fresh":"2026-02-23T13:12:56.088356+0000","last_change":"2026-02-23T13:12:56.068735+0000","last_active":"2026-02-23T13:12:56.088356+0000","last_peered":"2026-02-23T13:12:56.088356+0000","last_clean":"2026-02-23T13:12:23.256807+0000","last_became_active":"2026-02-23T13:12:53.415848+0000","last_became_peered":"2026-02-23T13:12:53.415848+0000","last_unstale":"2026-02-23T13:12:56.088356+0000","last_undegraded":"2026-02-23T13:12:56.088356+0000","last_fullsized":"2026-02-23T13:12:53.001243+0000","mapping_epoch":153,"log_start":"130'4800","ondisk_log_start":"130'4800","created":126,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:11:36.450546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:11:36.450546+0000","last_clean_scrub_stamp":"2026-02-23T13:11:36.450546+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":121569280,"num_objects":1855,"num_object_clones":0,"num_object_copies":5565,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1855,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14840,"num_write_kb":118720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":631,"num_bytes_recovered":41328640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,0],"avail_no_missing":["3","0","2"],"object_location_counts":[{"shards":"0,3","objects":1855}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"28'99","reported_seq":333,"reported_epoch":156,"state":"active+clean","last_fresh":"2026-02-23T13:12:56.008521+0000","last_change":"2026-02-23T13:12:52.005066+0000","last_active":"2026-02-23T13:12:56.008521+0000","last_peered":"2026-02-23T13:12:56.008521+0000","last_clean":"2026-02-23T13:12:56.008521+0000","last_became_active":"2026-02-23T13:12:52.004933+0000","last_became_peered":"2026-02-23T13:12:52.004933+0000","last_unstale":"2026-02-23T13:12:56.008521+0000","last_undegraded":"2026-02-23T13:12:56.008521+0000","last_fullsized":"2026-02-23T13:12:56.008521+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'99","last_scrub_stamp":"2026-02-23T13:11:39.969814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:07:05.619208+0000","last_clean_scrub_stamp":"2026-02-23T13:11:39.969814+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T13:12:59.396325+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":122,"num_read_kb":228,"num_write":135,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":16,"stat_sum":{"num_bytes":2056634368,"num_objects":31383,"num_object_clones":0,"num_object_copies":94149,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31383,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":251054,"num_write_kb":2008432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11220,"num_bytes_recovered":734011392,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":6172606464,"data_stored":6172606464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160854,"ondisk_log_size":160854,"up":48,"acting":46,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":122,"num_read_kb":228,"num_write":135,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":80,"seq":343597383718,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":734003200,"kb_used":2240680,"kb_used_data":1467668,"kb_used_omap":59,"kb_used_meta":772932,"kb_avail":731762520,"statfs":{"total":751619276800,"available":749324820480,"internally_reserved":0,"allocated":1502892032,"data_stored":1501768693,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60511,"internal_metadata":791483297},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":152,"seq":652835028995,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":734003200,"kb_used":1848940,"kb_used_data":1748952,"kb_used_omap":14,"kb_used_meta":99953,"kb_avail":732154260,"statfs":{"total":751619276800,"available":749725962240,"internally_reserved":0,"allocated":1790926848,"data_stored":1789780004,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15212,"internal_metadata":102352020},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,2,1,0,6,2,6],"upper_bound":128},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":144,"seq":618475290629,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":734003200,"kb_used":1408688,"kb_used_data":1266332,"kb_used_omap":18,"kb_used_meta":142317,"kb_avail":732594512,"statfs":{"total":751619276800,"available":750176780288,"internally_reserved":0,"allocated":1296723968,"data_stored":1295580676,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18768,"internal_metadata":145733296},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":106,"seq":455266533401,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":734003200,"kb_used":2065276,"kb_used_data":1504040,"kb_used_omap":40,"kb_used_meta":561175,"kb_avail":731937924,"statfs":{"total":751619276800,"available":749504434176,"internally_reserved":0,"allocated":1540136960,"data_stored":1538990116,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":41560,"internal_metadata":574643624},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,3,12,6,16,10,8,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1556766720,"data_stored":1556766720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1309392896,"data_stored":1309392896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1804623872,"data_stored":1804623872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1501822976,"data_stored":1501822976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-23T13:13:02.046 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph status --format=json 2026-02-23T13:13:02.079 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:13:02.195 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:13:02.207 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:13:02.205+0000 7f0073efe640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:13:02.207 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:13:02.205+0000 7f0073efe640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:13:02.225 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:13:02.226 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:13:02.241 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:13:02.241 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:13:02.254 INFO:teuthology.orchestra.run.trial068.stdout:osd.2: {} 2026-02-23T13:13:02.254 INFO:teuthology.orchestra.run.trial068.stderr:osd.2: osd_enable_op_tracker = '' 2026-02-23T13:13:02.272 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:13:02.272 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:13:02.327 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:13:02.326+0000 7fa49e4fa640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:13:02.327 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:13:02.326+0000 7fa49e4fa640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:13:02.514 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T13:13:02.514 INFO:teuthology.orchestra.run.trial068.stdout:{"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 530/92355 objects degraded (0.574%), 3 pgs degraded","count":3},"muted":false}},"mutes":[]},"election_epoch":5,"quorum":[0],"quorum_names":["a"],"quorum_age":418,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":1},"osdmap":{"epoch":161,"num_osds":4,"num_up_osds":4,"osd_up_since":1771852370,"num_in_osds":4,"osd_in_since":1771852023,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":7},{"state_name":"peering","count":4},{"state_name":"active+undersized+remapped","count":3},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":2},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":17,"num_pools":2,"num_objects":30785,"data_bytes":2017944096,"bytes_used":7612014592,"bytes_avail":2998865092608,"bytes_total":3006477107200,"inactive_pgs_ratio":0.23529411852359772,"degraded_objects":530,"degraded_total":92355,"degraded_ratio":0.0057387255698121377,"recovering_objects_per_sec":276,"recovering_bytes_per_sec":18040275,"recovering_keys_per_sec":0,"num_objects_recovered":1107,"num_bytes_recovered":72171520,"num_keys_recovered":0,"write_bytes_sec":10003035,"read_op_per_sec":0,"write_op_per_sec":1221},"fsmap":{"epoch":1,"btime":"2026-02-23T13:06:03:098534+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["cephadm","dashboard","iostat","nfs","prometheus"],"services":{"dashboard":"https://10.20.193.68:8443/","prometheus":"http://10.20.193.68:9283/"}},"servicemap":{"epoch":7,"modified":"2026-02-23T13:12:02.266447+0000","services":{"osd":{"daemons":{"summary":"","2":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2026-02-23T13:13:02.554 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 84 127 261635 261508 24.3198 26.8203 0.0163152 0.0410964 2026-02-23T13:13:02.560 INFO:tasks.cephadm.ceph_manager.ceph:making progress, resetting timeout 2026-02-23T13:13:02.560 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph pg dump --format=json 2026-02-23T13:13:02.735 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:13:02.856 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:13:02.854+0000 7faf875c1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:13:02.856 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:13:02.854+0000 7faf875c1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:13:03.003 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T13:13:03.004 INFO:teuthology.orchestra.run.trial068.stderr:dumped all 2026-02-23T13:13:03.048 INFO:teuthology.orchestra.run.trial068.stdout:{"pg_ready":true,"pg_map":{"version":305,"stamp":"2026-02-23T13:13:02.276111+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2087305760,"num_objects":31845,"num_object_clones":0,"num_object_copies":95535,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31845,"num_whiteouts":0,"num_read":122,"num_read_kb":228,"num_write":254861,"num_write_kb":2039958,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11222,"num_bytes_recovered":734601760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160925,"ondisk_log_size":160925,"up":51,"acting":51,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":51,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":2936012800,"kb_used":7668248,"kb_used_data":6055552,"kb_used_omap":135,"kb_used_meta":1612472,"kb_avail":2928344552,"statfs":{"total":3006477107200,"available":2998624821248,"internally_reserved":0,"allocated":6200885248,"data_stored":6196293545,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":138659,"internal_metadata":1651171933},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[6,5,13,6,22,12,14,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":95510528,"num_objects":1459,"num_object_clones":0,"num_object_copies":4377,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2423,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1459,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11659,"num_write_kb":93272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":869,"num_bytes_recovered":56844288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.267528"},"pg_stats":[{"pgid":"4.f","version":"160'16040","reported_seq":17967,"reported_epoch":160,"state":"active+clean","last_fresh":"2026-02-23T13:13:00.496031+0000","last_change":"2026-02-23T13:12:58.015180+0000","last_active":"2026-02-23T13:13:00.496031+0000","last_peered":"2026-02-23T13:13:00.496031+0000","last_clean":"2026-02-23T13:13:00.496031+0000","last_became_active":"2026-02-23T13:12:58.011500+0000","last_became_peered":"2026-02-23T13:12:58.011500+0000","last_unstale":"2026-02-23T13:13:00.496031+0000","last_undegraded":"2026-02-23T13:13:00.496031+0000","last_fullsized":"2026-02-23T13:13:00.496031+0000","mapping_epoch":157,"log_start":"130'6000","ondisk_log_start":"130'6000","created":126,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:11:36.450546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:11:36.450546+0000","last_clean_scrub_stamp":"2026-02-23T13:11:36.450546+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":131399680,"num_objects":2005,"num_object_clones":0,"num_object_copies":6015,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2005,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16040,"num_write_kb":128320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":909,"num_bytes_recovered":59572224,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"159'16032","reported_seq":17577,"reported_epoch":159,"state":"active+clean","last_fresh":"2026-02-23T13:12:59.958820+0000","last_change":"2026-02-23T13:12:58.792221+0000","last_active":"2026-02-23T13:12:59.958820+0000","last_peered":"2026-02-23T13:12:59.958820+0000","last_clean":"2026-02-23T13:12:59.958820+0000","last_became_active":"2026-02-23T13:12:57.011288+0000","last_became_peered":"2026-02-23T13:12:57.011288+0000","last_unstale":"2026-02-23T13:12:59.958820+0000","last_undegraded":"2026-02-23T13:12:59.958820+0000","last_fullsized":"2026-02-23T13:12:59.958820+0000","mapping_epoch":156,"log_start":"130'6000","ondisk_log_start":"130'6000","created":126,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"158'15752","last_scrub_stamp":"2026-02-23T13:12:58.792154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:11:36.450546+0000","last_clean_scrub_stamp":"2026-02-23T13:12:58.792154+0000","objects_scrubbed":1964,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T13:14:17.745746+0000","scrub_duration":465,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":131334144,"num_objects":2004,"num_object_clones":0,"num_object_copies":6012,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2004,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16032,"num_write_kb":128256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":905,"num_bytes_recovered":59301888,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"161'15880","reported_seq":17239,"reported_epoch":161,"state":"active+clean","last_fresh":"2026-02-23T13:13:01.664893+0000","last_change":"2026-02-23T13:12:59.853778+0000","last_active":"2026-02-23T13:13:01.664893+0000","last_peered":"2026-02-23T13:13:01.664893+0000","last_clean":"2026-02-23T13:13:01.664893+0000","last_became_active":"2026-02-23T13:12:57.011232+0000","last_became_peered":"2026-02-23T13:12:57.011232+0000","last_unstale":"2026-02-23T13:13:01.664893+0000","last_undegraded":"2026-02-23T13:13:01.664893+0000","last_fullsized":"2026-02-23T13:13:01.664893+0000","mapping_epoch":156,"log_start":"130'5800","ondisk_log_start":"130'5800","created":126,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"159'15496","last_scrub_stamp":"2026-02-23T13:12:59.853734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:11:36.450546+0000","last_clean_scrub_stamp":"2026-02-23T13:12:59.853734+0000","objects_scrubbed":1912,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T13:14:05.363974+0000","scrub_duration":1854,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":130088960,"num_objects":1985,"num_object_clones":0,"num_object_copies":5955,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15880,"num_write_kb":127040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":610,"num_bytes_recovered":39763968,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"161'16710","reported_seq":18727,"reported_epoch":161,"state":"active+clean","last_fresh":"2026-02-23T13:13:01.684921+0000","last_change":"2026-02-23T13:12:58.016130+0000","last_active":"2026-02-23T13:13:01.684921+0000","last_peered":"2026-02-23T13:13:01.684921+0000","last_clean":"2026-02-23T13:13:01.684921+0000","last_became_active":"2026-02-23T13:12:58.010060+0000","last_became_peered":"2026-02-23T13:12:58.010060+0000","last_unstale":"2026-02-23T13:13:01.684921+0000","last_undegraded":"2026-02-23T13:13:01.684921+0000","last_fullsized":"2026-02-23T13:13:01.684921+0000","mapping_epoch":157,"log_start":"130'6700","ondisk_log_start":"130'6700","created":126,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:11:36.450546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:11:36.450546+0000","last_clean_scrub_stamp":"2026-02-23T13:11:36.450546+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":136888320,"num_objects":2089,"num_object_clones":0,"num_object_copies":6267,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2089,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16710,"num_write_kb":133680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":957,"num_bytes_recovered":62480384,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"160'15982","reported_seq":17945,"reported_epoch":160,"state":"active+clean","last_fresh":"2026-02-23T13:13:00.488240+0000","last_change":"2026-02-23T13:12:58.017947+0000","last_active":"2026-02-23T13:13:00.488240+0000","last_peered":"2026-02-23T13:13:00.488240+0000","last_clean":"2026-02-23T13:13:00.488240+0000","last_became_active":"2026-02-23T13:12:58.011138+0000","last_became_peered":"2026-02-23T13:12:58.011138+0000","last_unstale":"2026-02-23T13:13:00.488240+0000","last_undegraded":"2026-02-23T13:13:00.488240+0000","last_fullsized":"2026-02-23T13:13:00.488240+0000","mapping_epoch":157,"log_start":"130'5900","ondisk_log_start":"130'5900","created":126,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:11:36.450546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:11:36.450546+0000","last_clean_scrub_stamp":"2026-02-23T13:11:36.450546+0000","objects_scrubbed":0,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":130924544,"num_objects":1998,"num_object_clones":0,"num_object_copies":5994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1998,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15982,"num_write_kb":127856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":926,"num_bytes_recovered":60604416,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"159'15256","reported_seq":16841,"reported_epoch":159,"state":"active+clean","last_fresh":"2026-02-23T13:13:00.002267+0000","last_change":"2026-02-23T13:12:57.015183+0000","last_active":"2026-02-23T13:13:00.002267+0000","last_peered":"2026-02-23T13:13:00.002267+0000","last_clean":"2026-02-23T13:13:00.002267+0000","last_became_active":"2026-02-23T13:12:57.011012+0000","last_became_peered":"2026-02-23T13:12:57.011012+0000","last_unstale":"2026-02-23T13:13:00.002267+0000","last_undegraded":"2026-02-23T13:13:00.002267+0000","last_fullsized":"2026-02-23T13:13:00.002267+0000","mapping_epoch":156,"log_start":"130'5200","ondisk_log_start":"130'5200","created":126,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:11:36.450546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:11:36.450546+0000","last_clean_scrub_stamp":"2026-02-23T13:11:36.450546+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":124977152,"num_objects":1907,"num_object_clones":0,"num_object_copies":5721,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1907,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15256,"num_write_kb":122048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":896,"num_bytes_recovered":58646528,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"159'16040","reported_seq":17880,"reported_epoch":159,"state":"active+clean","last_fresh":"2026-02-23T13:12:59.978677+0000","last_change":"2026-02-23T13:12:57.015222+0000","last_active":"2026-02-23T13:12:59.978677+0000","last_peered":"2026-02-23T13:12:59.978677+0000","last_clean":"2026-02-23T13:12:59.978677+0000","last_became_active":"2026-02-23T13:12:57.011352+0000","last_became_peered":"2026-02-23T13:12:57.011352+0000","last_unstale":"2026-02-23T13:12:59.978677+0000","last_undegraded":"2026-02-23T13:12:59.978677+0000","last_fullsized":"2026-02-23T13:12:59.978677+0000","mapping_epoch":156,"log_start":"130'6000","ondisk_log_start":"130'6000","created":126,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:11:36.450546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:11:36.450546+0000","last_clean_scrub_stamp":"2026-02-23T13:11:36.450546+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":131399680,"num_objects":2005,"num_object_clones":0,"num_object_copies":6015,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2005,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16040,"num_write_kb":128320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":947,"num_bytes_recovered":62062592,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"161'16138","reported_seq":17507,"reported_epoch":161,"state":"active+clean","last_fresh":"2026-02-23T13:13:01.676276+0000","last_change":"2026-02-23T13:12:57.018171+0000","last_active":"2026-02-23T13:13:01.676276+0000","last_peered":"2026-02-23T13:13:01.676276+0000","last_clean":"2026-02-23T13:13:01.676276+0000","last_became_active":"2026-02-23T13:12:57.011216+0000","last_became_peered":"2026-02-23T13:12:57.011216+0000","last_unstale":"2026-02-23T13:13:01.676276+0000","last_undegraded":"2026-02-23T13:13:01.676276+0000","last_fullsized":"2026-02-23T13:13:01.676276+0000","mapping_epoch":156,"log_start":"130'6100","ondisk_log_start":"130'6100","created":126,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:11:36.450546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:11:36.450546+0000","last_clean_scrub_stamp":"2026-02-23T13:11:36.450546+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":132202496,"num_objects":2018,"num_object_clones":0,"num_object_copies":6054,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2018,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16138,"num_write_kb":129104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":636,"num_bytes_recovered":41402368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.7","version":"160'15570","reported_seq":17365,"reported_epoch":160,"state":"active+clean","last_fresh":"2026-02-23T13:13:00.498627+0000","last_change":"2026-02-23T13:12:56.015592+0000","last_active":"2026-02-23T13:13:00.498627+0000","last_peered":"2026-02-23T13:13:00.498627+0000","last_clean":"2026-02-23T13:13:00.498627+0000","last_became_active":"2026-02-23T13:12:56.011425+0000","last_became_peered":"2026-02-23T13:12:56.011425+0000","last_unstale":"2026-02-23T13:13:00.498627+0000","last_undegraded":"2026-02-23T13:13:00.498627+0000","last_fullsized":"2026-02-23T13:13:00.498627+0000","mapping_epoch":155,"log_start":"130'5500","ondisk_log_start":"130'5500","created":126,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:11:36.450546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:11:36.450546+0000","last_clean_scrub_stamp":"2026-02-23T13:11:36.450546+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":127549440,"num_objects":1947,"num_object_clones":0,"num_object_copies":5841,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1947,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15570,"num_write_kb":124560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":851,"num_bytes_recovered":55615488,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"160'15894","reported_seq":17904,"reported_epoch":160,"state":"active+clean","last_fresh":"2026-02-23T13:13:00.488458+0000","last_change":"2026-02-23T13:12:58.990117+0000","last_active":"2026-02-23T13:13:00.488458+0000","last_peered":"2026-02-23T13:13:00.488458+0000","last_clean":"2026-02-23T13:13:00.488458+0000","last_became_active":"2026-02-23T13:12:57.012274+0000","last_became_peered":"2026-02-23T13:12:57.012274+0000","last_unstale":"2026-02-23T13:13:00.488458+0000","last_undegraded":"2026-02-23T13:13:00.488458+0000","last_fullsized":"2026-02-23T13:13:00.488458+0000","mapping_epoch":156,"log_start":"130'5800","ondisk_log_start":"130'5800","created":126,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"158'15592","last_scrub_stamp":"2026-02-23T13:12:58.990065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:11:36.450546+0000","last_clean_scrub_stamp":"2026-02-23T13:12:58.990065+0000","objects_scrubbed":1945,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T13:14:01.221621+0000","scrub_duration":369,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":130203648,"num_objects":1987,"num_object_clones":0,"num_object_copies":5961,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1987,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15894,"num_write_kb":127152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":938,"num_bytes_recovered":61456384,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"160'15824","reported_seq":16435,"reported_epoch":160,"state":"active+clean","last_fresh":"2026-02-23T13:13:00.437531+0000","last_change":"2026-02-23T13:12:51.932089+0000","last_active":"2026-02-23T13:13:00.437531+0000","last_peered":"2026-02-23T13:13:00.437531+0000","last_clean":"2026-02-23T13:13:00.437531+0000","last_became_active":"2026-02-23T13:12:47.496034+0000","last_became_peered":"2026-02-23T13:12:47.496034+0000","last_unstale":"2026-02-23T13:13:00.437531+0000","last_undegraded":"2026-02-23T13:13:00.437531+0000","last_fullsized":"2026-02-23T13:13:00.437531+0000","mapping_epoch":148,"log_start":"130'5800","ondisk_log_start":"130'5800","created":126,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"151'14880","last_scrub_stamp":"2026-02-23T13:12:51.932039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:11:36.450546+0000","last_clean_scrub_stamp":"2026-02-23T13:12:51.932039+0000","objects_scrubbed":1860,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T13:14:01.557089+0000","scrub_duration":94,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":129630208,"num_objects":1978,"num_object_clones":0,"num_object_copies":5934,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1978,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15824,"num_write_kb":126592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":279,"num_bytes_recovered":18284544,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"160'15752","reported_seq":17120,"reported_epoch":160,"state":"active+clean","last_fresh":"2026-02-23T13:13:00.447474+0000","last_change":"2026-02-23T13:12:57.018841+0000","last_active":"2026-02-23T13:13:00.447474+0000","last_peered":"2026-02-23T13:13:00.447474+0000","last_clean":"2026-02-23T13:13:00.447474+0000","last_became_active":"2026-02-23T13:12:57.012156+0000","last_became_peered":"2026-02-23T13:12:57.012156+0000","last_unstale":"2026-02-23T13:13:00.447474+0000","last_undegraded":"2026-02-23T13:13:00.447474+0000","last_fullsized":"2026-02-23T13:13:00.447474+0000","mapping_epoch":156,"log_start":"130'5700","ondisk_log_start":"130'5700","created":126,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:11:36.450546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:11:36.450546+0000","last_clean_scrub_stamp":"2026-02-23T13:11:36.450546+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T13:13:05.429853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":129040384,"num_objects":1969,"num_object_clones":0,"num_object_copies":5907,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1969,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15752,"num_write_kb":126016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":628,"num_bytes_recovered":41156608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"160'16400","reported_seq":16673,"reported_epoch":160,"state":"active+clean","last_fresh":"2026-02-23T13:13:00.471910+0000","last_change":"2026-02-23T13:12:47.499686+0000","last_active":"2026-02-23T13:13:00.471910+0000","last_peered":"2026-02-23T13:13:00.471910+0000","last_clean":"2026-02-23T13:13:00.471910+0000","last_became_active":"2026-02-23T13:12:47.496124+0000","last_became_peered":"2026-02-23T13:12:47.496124+0000","last_unstale":"2026-02-23T13:13:00.471910+0000","last_undegraded":"2026-02-23T13:13:00.471910+0000","last_fullsized":"2026-02-23T13:13:00.471910+0000","mapping_epoch":148,"log_start":"130'6300","ondisk_log_start":"130'6300","created":126,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:11:36.450546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:11:36.450546+0000","last_clean_scrub_stamp":"2026-02-23T13:11:36.450546+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":134348800,"num_objects":2050,"num_object_clones":0,"num_object_copies":6150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2050,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16400,"num_write_kb":131200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":294,"num_bytes_recovered":19218432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1","version":"159'15504","reported_seq":15930,"reported_epoch":159,"state":"active+clean","last_fresh":"2026-02-23T13:12:59.969972+0000","last_change":"2026-02-23T13:12:56.013478+0000","last_active":"2026-02-23T13:12:59.969972+0000","last_peered":"2026-02-23T13:12:59.969972+0000","last_clean":"2026-02-23T13:12:59.969972+0000","last_became_active":"2026-02-23T13:12:56.010005+0000","last_became_peered":"2026-02-23T13:12:56.010005+0000","last_unstale":"2026-02-23T13:12:59.969972+0000","last_undegraded":"2026-02-23T13:12:59.969972+0000","last_fullsized":"2026-02-23T13:12:59.969972+0000","mapping_epoch":155,"log_start":"130'5500","ondisk_log_start":"130'5500","created":126,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:11:36.450546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:11:36.450546+0000","last_clean_scrub_stamp":"2026-02-23T13:11:36.450546+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":127008768,"num_objects":1938,"num_object_clones":0,"num_object_copies":5814,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1938,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15504,"num_write_kb":124032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":175,"num_bytes_recovered":11362304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.0","version":"160'15920","reported_seq":17308,"reported_epoch":160,"state":"active+clean","last_fresh":"2026-02-23T13:13:00.485879+0000","last_change":"2026-02-23T13:12:56.013162+0000","last_active":"2026-02-23T13:13:00.485879+0000","last_peered":"2026-02-23T13:13:00.485879+0000","last_clean":"2026-02-23T13:13:00.485879+0000","last_became_active":"2026-02-23T13:12:56.010072+0000","last_became_peered":"2026-02-23T13:12:56.010072+0000","last_unstale":"2026-02-23T13:13:00.485879+0000","last_undegraded":"2026-02-23T13:13:00.485879+0000","last_fullsized":"2026-02-23T13:13:00.485879+0000","mapping_epoch":155,"log_start":"130'5900","ondisk_log_start":"130'5900","created":126,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:11:36.450546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:11:36.450546+0000","last_clean_scrub_stamp":"2026-02-23T13:11:36.450546+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T13:13:06.181031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":130416640,"num_objects":1990,"num_object_clones":0,"num_object_copies":5970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1990,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15920,"num_write_kb":127360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":638,"num_bytes_recovered":41754624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,3],"acting":[0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.5","version":"161'15784","reported_seq":17190,"reported_epoch":161,"state":"active+clean","last_fresh":"2026-02-23T13:13:01.678765+0000","last_change":"2026-02-23T13:13:00.145165+0000","last_active":"2026-02-23T13:13:01.678765+0000","last_peered":"2026-02-23T13:13:01.678765+0000","last_clean":"2026-02-23T13:13:01.678765+0000","last_became_active":"2026-02-23T13:12:58.010232+0000","last_became_peered":"2026-02-23T13:12:58.010232+0000","last_unstale":"2026-02-23T13:13:01.678765+0000","last_undegraded":"2026-02-23T13:13:01.678765+0000","last_fullsized":"2026-02-23T13:13:01.678765+0000","mapping_epoch":157,"log_start":"130'5700","ondisk_log_start":"130'5700","created":126,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"160'15384","last_scrub_stamp":"2026-02-23T13:13:00.145123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:11:36.450546+0000","last_clean_scrub_stamp":"2026-02-23T13:13:00.145123+0000","objects_scrubbed":1915,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T13:14:02.486042+0000","scrub_duration":1115,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":129302528,"num_objects":1973,"num_object_clones":0,"num_object_copies":5919,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1973,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15784,"num_write_kb":126272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":631,"num_bytes_recovered":41328640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"28'99","reported_seq":343,"reported_epoch":161,"state":"active+clean","last_fresh":"2026-02-23T13:13:01.022564+0000","last_change":"2026-02-23T13:12:52.005066+0000","last_active":"2026-02-23T13:13:01.022564+0000","last_peered":"2026-02-23T13:13:01.022564+0000","last_clean":"2026-02-23T13:13:01.022564+0000","last_became_active":"2026-02-23T13:12:52.004933+0000","last_became_peered":"2026-02-23T13:12:52.004933+0000","last_unstale":"2026-02-23T13:13:01.022564+0000","last_undegraded":"2026-02-23T13:13:01.022564+0000","last_fullsized":"2026-02-23T13:13:01.022564+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'99","last_scrub_stamp":"2026-02-23T13:11:39.969814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:07:05.619208+0000","last_clean_scrub_stamp":"2026-02-23T13:11:39.969814+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":122,"num_read_kb":228,"num_write":135,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":16,"stat_sum":{"num_bytes":2086715392,"num_objects":31843,"num_object_clones":0,"num_object_copies":95529,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31843,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":254726,"num_write_kb":2037808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11220,"num_bytes_recovered":734011392,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":6259851264,"data_stored":6259851264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160826,"ondisk_log_size":160826,"up":48,"acting":48,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":122,"num_read_kb":228,"num_write":135,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":80,"seq":343597383719,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":734003200,"kb_used":2345344,"kb_used_data":1536228,"kb_used_omap":61,"kb_used_meta":809026,"kb_avail":731657856,"statfs":{"total":751619276800,"available":749217644544,"internally_reserved":0,"allocated":1573097472,"data_stored":1571942749,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63119,"internal_metadata":828442993},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":152,"seq":652835028995,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":734003200,"kb_used":1848940,"kb_used_data":1748952,"kb_used_omap":14,"kb_used_meta":99953,"kb_avail":732154260,"statfs":{"total":751619276800,"available":749725962240,"internally_reserved":0,"allocated":1790926848,"data_stored":1789780004,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15212,"internal_metadata":102352020},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,2,1,0,6,2,6],"upper_bound":128},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":144,"seq":618475290629,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":734003200,"kb_used":1408688,"kb_used_data":1266332,"kb_used_omap":18,"kb_used_meta":142317,"kb_avail":732594512,"statfs":{"total":751619276800,"available":750176780288,"internally_reserved":0,"allocated":1296723968,"data_stored":1295580676,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18768,"internal_metadata":145733296},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":106,"seq":455266533401,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":734003200,"kb_used":2065276,"kb_used_data":1504040,"kb_used_omap":40,"kb_used_meta":561175,"kb_avail":731937924,"statfs":{"total":751619276800,"available":749504434176,"internally_reserved":0,"allocated":1540136960,"data_stored":1538990116,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":41560,"internal_metadata":574643624},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,3,12,6,16,10,8,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1556766720,"data_stored":1556766720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1309392896,"data_stored":1309392896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1804623872,"data_stored":1804623872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1589067776,"data_stored":1589067776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-23T13:13:03.111 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:13:03 trial068 ceph-mon[38805]: from='client.14733 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2026-02-23T13:13:03.111 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:13:03 trial068 ceph-mon[38805]: 4.b scrub starts 2026-02-23T13:13:03.111 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:13:03 trial068 ceph-mon[38805]: 4.f scrub starts 2026-02-23T13:13:03.111 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:13:03 trial068 ceph-mon[38805]: from='client.14735 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2026-02-23T13:13:03.112 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:13:03 trial068 ceph-mon[38805]: 4.8 scrub starts 2026-02-23T13:13:03.112 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:13:03 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/133740227' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:13:03.112 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:13:03 trial068 ceph-mon[38805]: 4.f scrub ok 2026-02-23T13:13:03.112 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:13:03 trial068 ceph-mon[38805]: pgmap v305: 17 pgs: 17 active+clean; 1.9 GiB data, 7.3 GiB used, 2.7 TiB / 2.7 TiB avail; 21 MiB/s wr, 2.73k op/s; 13 MiB/s, 203 objects/s recovering 2026-02-23T13:13:03.112 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:13:03 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1796236850' entity='client.admin' cmd={"prefix": "status", "format": "json"} : dispatch 2026-02-23T13:13:03.554 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 85 128 265066 264938 24.3489 26.7969 0.11814 0.0410565 2026-02-23T13:13:04.315 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:13:04.338 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:13:03 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:13:03] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:13:04.338 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:13:04 trial068 ceph-mon[38805]: 4.2 scrub starts 2026-02-23T13:13:04.339 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:13:04 trial068 ceph-mon[38805]: 4.b scrub ok 2026-02-23T13:13:04.339 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:13:04 trial068 ceph-mon[38805]: 4.8 scrub ok 2026-02-23T13:13:04.339 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:13:04 trial068 ceph-mon[38805]: from='client.14741 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2026-02-23T13:13:04.339 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:13:04 trial068 ceph-mon[38805]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 530/92355 objects degraded (0.574%), 3 pgs degraded) 2026-02-23T13:13:04.339 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:13:04 trial068 ceph-mon[38805]: Cluster is now healthy 2026-02-23T13:13:04.339 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:13:04 trial068 ceph-mon[38805]: 4.2 scrub ok 2026-02-23T13:13:04.339 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:13:04 trial068 ceph-mon[38805]: osdmap e162: 4 total, 4 up, 4 in 2026-02-23T13:13:04.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:13:03 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:13:03.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:13:04.444 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:13:04.554 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 86 128 268550 268422 24.3822 27.2188 0.0135496 0.0409759 2026-02-23T13:13:04.785 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:13:04.783+0000 7fa680f33640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:13:04.785 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:13:04.784+0000 7fa680f33640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:13:04.798 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:13:04.799 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:13:04.810 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:13:04.810 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:13:04.824 INFO:teuthology.orchestra.run.trial068.stdout:osd.2: {} 2026-02-23T13:13:04.824 INFO:teuthology.orchestra.run.trial068.stderr:osd.2: osd_enable_op_tracker = '' 2026-02-23T13:13:04.839 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:13:04.839 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:13:05.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:13:05 trial068 ceph-mon[38805]: pgmap v307: 17 pgs: 17 active+clean; 1.9 GiB data, 7.3 GiB used, 2.7 TiB / 2.7 TiB avail; 17 MiB/s wr, 2.21k op/s; 10 MiB/s, 165 objects/s recovering 2026-02-23T13:13:05.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:13:05 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3211568680' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:13:05.554 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 87 128 272099 271971 24.4206 27.7266 0.012181 0.0409398 2026-02-23T13:13:06.051 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph pg dump --format=json 2026-02-23T13:13:06.195 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:13:06.313 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:13:06 trial068 ceph-mon[38805]: 1.0 scrub starts 2026-02-23T13:13:06.313 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:13:06 trial068 ceph-mon[38805]: 1.0 scrub ok 2026-02-23T13:13:06.313 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:13:06 trial068 ceph-mon[38805]: 4.1 scrub starts 2026-02-23T13:13:06.313 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:13:06 trial068 ceph-mon[38805]: 4.c scrub starts 2026-02-23T13:13:06.554 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 88 128 275556 275428 24.45 27.0078 0.0811105 0.0408824 2026-02-23T13:13:06.573 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:13:06.563+0000 7f1b89625640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:13:06.573 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:13:06.564+0000 7f1b89625640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:13:06.711 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T13:13:06.711 INFO:teuthology.orchestra.run.trial068.stderr:dumped all 2026-02-23T13:13:06.884 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:13:06.916 INFO:teuthology.orchestra.run.trial068.stdout:{"pg_ready":true,"pg_map":{"version":308,"stamp":"2026-02-23T13:13:06.276653+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2195399200,"num_objects":33496,"num_object_clones":0,"num_object_copies":100488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33496,"num_whiteouts":0,"num_read":122,"num_read_kb":228,"num_write":268056,"num_write_kb":2145518,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11222,"num_bytes_recovered":734601760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160920,"ondisk_log_size":160920,"up":51,"acting":51,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":51,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":2936012800,"kb_used":8042572,"kb_used_data":6375732,"kb_used_omap":141,"kb_used_meta":1666610,"kb_avail":2927970228,"statfs":{"total":3006477107200,"available":2998241513472,"internally_reserved":0,"allocated":6528749568,"data_stored":6524086930,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":144527,"internal_metadata":1706609521},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,3,3,4,7],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":177455104,"num_objects":2711,"num_object_clones":0,"num_object_copies":8133,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-530,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2711,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21662,"num_write_kb":173296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":530,"num_bytes_recovered":34676736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.260392"},"pg_stats":[{"pgid":"4.f","version":"162'17144","reported_seq":19103,"reported_epoch":162,"state":"active+clean","last_fresh":"2026-02-23T13:13:05.495240+0000","last_change":"2026-02-23T13:13:02.216187+0000","last_active":"2026-02-23T13:13:05.495240+0000","last_peered":"2026-02-23T13:13:05.495240+0000","last_clean":"2026-02-23T13:13:05.495240+0000","last_became_active":"2026-02-23T13:12:58.011500+0000","last_became_peered":"2026-02-23T13:12:58.011500+0000","last_unstale":"2026-02-23T13:13:05.495240+0000","last_undegraded":"2026-02-23T13:13:05.495240+0000","last_fullsized":"2026-02-23T13:13:05.495240+0000","mapping_epoch":157,"log_start":"130'7100","ondisk_log_start":"130'7100","created":126,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"161'16416","last_scrub_stamp":"2026-02-23T13:13:02.216104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:11:36.450546+0000","last_clean_scrub_stamp":"2026-02-23T13:13:02.216104+0000","objects_scrubbed":2050,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T13:14:13.493839+0000","scrub_duration":420,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":140443648,"num_objects":2143,"num_object_clones":0,"num_object_copies":6429,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2143,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17144,"num_write_kb":137152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":909,"num_bytes_recovered":59572224,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"162'17175","reported_seq":18720,"reported_epoch":162,"state":"active+clean","last_fresh":"2026-02-23T13:13:04.985481+0000","last_change":"2026-02-23T13:12:58.792221+0000","last_active":"2026-02-23T13:13:04.985481+0000","last_peered":"2026-02-23T13:13:04.985481+0000","last_clean":"2026-02-23T13:13:04.985481+0000","last_became_active":"2026-02-23T13:12:57.011288+0000","last_became_peered":"2026-02-23T13:12:57.011288+0000","last_unstale":"2026-02-23T13:13:04.985481+0000","last_undegraded":"2026-02-23T13:13:04.985481+0000","last_fullsized":"2026-02-23T13:13:04.985481+0000","mapping_epoch":156,"log_start":"130'7100","ondisk_log_start":"130'7100","created":126,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"158'15752","last_scrub_stamp":"2026-02-23T13:12:58.792154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:11:36.450546+0000","last_clean_scrub_stamp":"2026-02-23T13:12:58.792154+0000","objects_scrubbed":1964,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T13:14:17.745746+0000","scrub_duration":465,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":140697600,"num_objects":2147,"num_object_clones":0,"num_object_copies":6441,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2147,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17175,"num_write_kb":137400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":905,"num_bytes_recovered":59301888,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"161'15880","reported_seq":17239,"reported_epoch":161,"state":"active+clean","last_fresh":"2026-02-23T13:13:01.664893+0000","last_change":"2026-02-23T13:12:59.853778+0000","last_active":"2026-02-23T13:13:01.664893+0000","last_peered":"2026-02-23T13:13:01.664893+0000","last_clean":"2026-02-23T13:13:01.664893+0000","last_became_active":"2026-02-23T13:12:57.011232+0000","last_became_peered":"2026-02-23T13:12:57.011232+0000","last_unstale":"2026-02-23T13:13:01.664893+0000","last_undegraded":"2026-02-23T13:13:01.664893+0000","last_fullsized":"2026-02-23T13:13:01.664893+0000","mapping_epoch":156,"log_start":"130'5800","ondisk_log_start":"130'5800","created":126,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"159'15496","last_scrub_stamp":"2026-02-23T13:12:59.853734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:11:36.450546+0000","last_clean_scrub_stamp":"2026-02-23T13:12:59.853734+0000","objects_scrubbed":1912,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T13:14:05.363974+0000","scrub_duration":1854,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":130088960,"num_objects":1985,"num_object_clones":0,"num_object_copies":5955,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15880,"num_write_kb":127040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":610,"num_bytes_recovered":39763968,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"161'16710","reported_seq":18727,"reported_epoch":161,"state":"active+clean","last_fresh":"2026-02-23T13:13:01.684921+0000","last_change":"2026-02-23T13:12:58.016130+0000","last_active":"2026-02-23T13:13:01.684921+0000","last_peered":"2026-02-23T13:13:01.684921+0000","last_clean":"2026-02-23T13:13:01.684921+0000","last_became_active":"2026-02-23T13:12:58.010060+0000","last_became_peered":"2026-02-23T13:12:58.010060+0000","last_unstale":"2026-02-23T13:13:01.684921+0000","last_undegraded":"2026-02-23T13:13:01.684921+0000","last_fullsized":"2026-02-23T13:13:01.684921+0000","mapping_epoch":157,"log_start":"130'6700","ondisk_log_start":"130'6700","created":126,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:11:36.450546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:11:36.450546+0000","last_clean_scrub_stamp":"2026-02-23T13:11:36.450546+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":136888320,"num_objects":2089,"num_object_clones":0,"num_object_copies":6267,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2089,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16710,"num_write_kb":133680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":957,"num_bytes_recovered":62480384,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"162'17030","reported_seq":19027,"reported_epoch":162,"state":"active+clean","last_fresh":"2026-02-23T13:13:05.502339+0000","last_change":"2026-02-23T13:13:02.854359+0000","last_active":"2026-02-23T13:13:05.502339+0000","last_peered":"2026-02-23T13:13:05.502339+0000","last_clean":"2026-02-23T13:13:05.502339+0000","last_became_active":"2026-02-23T13:12:58.011138+0000","last_became_peered":"2026-02-23T13:12:58.011138+0000","last_unstale":"2026-02-23T13:13:05.502339+0000","last_undegraded":"2026-02-23T13:13:05.502339+0000","last_fullsized":"2026-02-23T13:13:05.502339+0000","mapping_epoch":157,"log_start":"132'7000","ondisk_log_start":"132'7000","created":126,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"161'16421","last_scrub_stamp":"2026-02-23T13:13:02.854315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:11:36.450546+0000","last_clean_scrub_stamp":"2026-02-23T13:13:02.854315+0000","objects_scrubbed":2023,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T13:14:17.401026+0000","scrub_duration":1280,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":139509760,"num_objects":2129,"num_object_clones":0,"num_object_copies":6387,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2129,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17030,"num_write_kb":136240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":926,"num_bytes_recovered":60604416,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"162'16344","reported_seq":17962,"reported_epoch":162,"state":"active+clean","last_fresh":"2026-02-23T13:13:05.007902+0000","last_change":"2026-02-23T13:12:57.015183+0000","last_active":"2026-02-23T13:13:05.007902+0000","last_peered":"2026-02-23T13:13:05.007902+0000","last_clean":"2026-02-23T13:13:05.007902+0000","last_became_active":"2026-02-23T13:12:57.011012+0000","last_became_peered":"2026-02-23T13:12:57.011012+0000","last_unstale":"2026-02-23T13:13:05.007902+0000","last_undegraded":"2026-02-23T13:13:05.007902+0000","last_fullsized":"2026-02-23T13:13:05.007902+0000","mapping_epoch":156,"log_start":"130'6300","ondisk_log_start":"130'6300","created":126,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:11:36.450546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:11:36.450546+0000","last_clean_scrub_stamp":"2026-02-23T13:11:36.450546+0000","objects_scrubbed":0,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":133890048,"num_objects":2043,"num_object_clones":0,"num_object_copies":6129,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2043,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16344,"num_write_kb":130752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":896,"num_bytes_recovered":58646528,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"162'17144","reported_seq":18990,"reported_epoch":162,"state":"active+clean","last_fresh":"2026-02-23T13:13:04.951668+0000","last_change":"2026-02-23T13:12:57.015222+0000","last_active":"2026-02-23T13:13:04.951668+0000","last_peered":"2026-02-23T13:13:04.951668+0000","last_clean":"2026-02-23T13:13:04.951668+0000","last_became_active":"2026-02-23T13:12:57.011352+0000","last_became_peered":"2026-02-23T13:12:57.011352+0000","last_unstale":"2026-02-23T13:13:04.951668+0000","last_undegraded":"2026-02-23T13:13:04.951668+0000","last_fullsized":"2026-02-23T13:13:04.951668+0000","mapping_epoch":156,"log_start":"132'7100","ondisk_log_start":"132'7100","created":126,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:11:36.450546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:11:36.450546+0000","last_clean_scrub_stamp":"2026-02-23T13:11:36.450546+0000","objects_scrubbed":0,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":140443648,"num_objects":2143,"num_object_clones":0,"num_object_copies":6429,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2143,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17144,"num_write_kb":137152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":947,"num_bytes_recovered":62062592,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"161'16138","reported_seq":17507,"reported_epoch":161,"state":"active+clean","last_fresh":"2026-02-23T13:13:01.676276+0000","last_change":"2026-02-23T13:12:57.018171+0000","last_active":"2026-02-23T13:13:01.676276+0000","last_peered":"2026-02-23T13:13:01.676276+0000","last_clean":"2026-02-23T13:13:01.676276+0000","last_became_active":"2026-02-23T13:12:57.011216+0000","last_became_peered":"2026-02-23T13:12:57.011216+0000","last_unstale":"2026-02-23T13:13:01.676276+0000","last_undegraded":"2026-02-23T13:13:01.676276+0000","last_fullsized":"2026-02-23T13:13:01.676276+0000","mapping_epoch":156,"log_start":"130'6100","ondisk_log_start":"130'6100","created":126,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:11:36.450546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:11:36.450546+0000","last_clean_scrub_stamp":"2026-02-23T13:11:36.450546+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":132202496,"num_objects":2018,"num_object_clones":0,"num_object_copies":6054,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2018,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16138,"num_write_kb":129104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":636,"num_bytes_recovered":41402368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.7","version":"162'16755","reported_seq":18581,"reported_epoch":162,"state":"active+clean","last_fresh":"2026-02-23T13:13:05.509761+0000","last_change":"2026-02-23T13:13:01.303496+0000","last_active":"2026-02-23T13:13:05.509761+0000","last_peered":"2026-02-23T13:13:05.509761+0000","last_clean":"2026-02-23T13:13:05.509761+0000","last_became_active":"2026-02-23T13:12:56.011425+0000","last_became_peered":"2026-02-23T13:12:56.011425+0000","last_unstale":"2026-02-23T13:13:05.509761+0000","last_undegraded":"2026-02-23T13:13:05.509761+0000","last_fullsized":"2026-02-23T13:13:05.509761+0000","mapping_epoch":155,"log_start":"130'6700","ondisk_log_start":"130'6700","created":126,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"161'15792","last_scrub_stamp":"2026-02-23T13:13:01.303465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:11:36.450546+0000","last_clean_scrub_stamp":"2026-02-23T13:13:01.303465+0000","objects_scrubbed":1963,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T13:14:29.576810+0000","scrub_duration":563,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":137256960,"num_objects":2095,"num_object_clones":0,"num_object_copies":6285,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2095,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16755,"num_write_kb":134040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":851,"num_bytes_recovered":55615488,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"162'17010","reported_seq":19026,"reported_epoch":162,"state":"active+clean","last_fresh":"2026-02-23T13:13:05.503577+0000","last_change":"2026-02-23T13:12:58.990117+0000","last_active":"2026-02-23T13:13:05.503577+0000","last_peered":"2026-02-23T13:13:05.503577+0000","last_clean":"2026-02-23T13:13:05.503577+0000","last_became_active":"2026-02-23T13:12:57.012274+0000","last_became_peered":"2026-02-23T13:12:57.012274+0000","last_unstale":"2026-02-23T13:13:05.503577+0000","last_undegraded":"2026-02-23T13:13:05.503577+0000","last_fullsized":"2026-02-23T13:13:05.503577+0000","mapping_epoch":156,"log_start":"132'7000","ondisk_log_start":"132'7000","created":126,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"158'15592","last_scrub_stamp":"2026-02-23T13:12:58.990065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:11:36.450546+0000","last_clean_scrub_stamp":"2026-02-23T13:12:58.990065+0000","objects_scrubbed":1945,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T13:14:01.221621+0000","scrub_duration":369,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":139345920,"num_objects":2127,"num_object_clones":0,"num_object_copies":6381,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2127,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17010,"num_write_kb":136080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":938,"num_bytes_recovered":61456384,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"162'16941","reported_seq":17554,"reported_epoch":162,"state":"active+clean","last_fresh":"2026-02-23T13:13:05.506707+0000","last_change":"2026-02-23T13:12:51.932089+0000","last_active":"2026-02-23T13:13:05.506707+0000","last_peered":"2026-02-23T13:13:05.506707+0000","last_clean":"2026-02-23T13:13:05.506707+0000","last_became_active":"2026-02-23T13:12:47.496034+0000","last_became_peered":"2026-02-23T13:12:47.496034+0000","last_unstale":"2026-02-23T13:13:05.506707+0000","last_undegraded":"2026-02-23T13:13:05.506707+0000","last_fullsized":"2026-02-23T13:13:05.506707+0000","mapping_epoch":148,"log_start":"132'6900","ondisk_log_start":"132'6900","created":126,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"151'14880","last_scrub_stamp":"2026-02-23T13:12:51.932039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:11:36.450546+0000","last_clean_scrub_stamp":"2026-02-23T13:12:51.932039+0000","objects_scrubbed":1860,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T13:14:01.557089+0000","scrub_duration":94,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":138780672,"num_objects":2118,"num_object_clones":0,"num_object_copies":6354,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2118,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16941,"num_write_kb":135528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":279,"num_bytes_recovered":18284544,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"162'16952","reported_seq":18324,"reported_epoch":162,"state":"active+clean","last_fresh":"2026-02-23T13:13:05.473277+0000","last_change":"2026-02-23T13:12:57.018841+0000","last_active":"2026-02-23T13:13:05.473277+0000","last_peered":"2026-02-23T13:13:05.473277+0000","last_clean":"2026-02-23T13:13:05.473277+0000","last_became_active":"2026-02-23T13:12:57.012156+0000","last_became_peered":"2026-02-23T13:12:57.012156+0000","last_unstale":"2026-02-23T13:13:05.473277+0000","last_undegraded":"2026-02-23T13:13:05.473277+0000","last_fullsized":"2026-02-23T13:13:05.473277+0000","mapping_epoch":156,"log_start":"132'6900","ondisk_log_start":"132'6900","created":126,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:11:36.450546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:11:36.450546+0000","last_clean_scrub_stamp":"2026-02-23T13:11:36.450546+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":138870784,"num_objects":2119,"num_object_clones":0,"num_object_copies":6357,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2119,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16952,"num_write_kb":135616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":628,"num_bytes_recovered":41156608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"162'17288","reported_seq":17594,"reported_epoch":162,"state":"active+clean","last_fresh":"2026-02-23T13:13:05.383504+0000","last_change":"2026-02-23T13:13:03.100218+0000","last_active":"2026-02-23T13:13:05.383504+0000","last_peered":"2026-02-23T13:13:05.383504+0000","last_clean":"2026-02-23T13:13:05.383504+0000","last_became_active":"2026-02-23T13:12:47.496124+0000","last_became_peered":"2026-02-23T13:12:47.496124+0000","last_unstale":"2026-02-23T13:13:05.383504+0000","last_undegraded":"2026-02-23T13:13:05.383504+0000","last_fullsized":"2026-02-23T13:13:05.383504+0000","mapping_epoch":148,"log_start":"132'7200","ondisk_log_start":"132'7200","created":126,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"161'16888","last_scrub_stamp":"2026-02-23T13:13:03.100166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:11:36.450546+0000","last_clean_scrub_stamp":"2026-02-23T13:13:03.100166+0000","objects_scrubbed":2109,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T13:14:15.208020+0000","scrub_duration":322,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":141623296,"num_objects":2161,"num_object_clones":0,"num_object_copies":6483,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2161,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17288,"num_write_kb":138304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":294,"num_bytes_recovered":19218432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1","version":"162'16560","reported_seq":16987,"reported_epoch":162,"state":"active+clean","last_fresh":"2026-02-23T13:13:05.012241+0000","last_change":"2026-02-23T13:12:56.013478+0000","last_active":"2026-02-23T13:13:05.012241+0000","last_peered":"2026-02-23T13:13:05.012241+0000","last_clean":"2026-02-23T13:13:05.012241+0000","last_became_active":"2026-02-23T13:12:56.010005+0000","last_became_peered":"2026-02-23T13:12:56.010005+0000","last_unstale":"2026-02-23T13:13:05.012241+0000","last_undegraded":"2026-02-23T13:13:05.012241+0000","last_fullsized":"2026-02-23T13:13:05.012241+0000","mapping_epoch":155,"log_start":"132'6500","ondisk_log_start":"132'6500","created":126,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:11:36.450546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:11:36.450546+0000","last_clean_scrub_stamp":"2026-02-23T13:11:36.450546+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":135659520,"num_objects":2070,"num_object_clones":0,"num_object_copies":6210,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2070,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16560,"num_write_kb":132480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":175,"num_bytes_recovered":11362304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.0","version":"162'17066","reported_seq":18460,"reported_epoch":162,"state":"active+clean","last_fresh":"2026-02-23T13:13:05.494640+0000","last_change":"2026-02-23T13:12:56.013162+0000","last_active":"2026-02-23T13:13:05.494640+0000","last_peered":"2026-02-23T13:13:05.494640+0000","last_clean":"2026-02-23T13:13:05.494640+0000","last_became_active":"2026-02-23T13:12:56.010072+0000","last_became_peered":"2026-02-23T13:12:56.010072+0000","last_unstale":"2026-02-23T13:13:05.494640+0000","last_undegraded":"2026-02-23T13:13:05.494640+0000","last_fullsized":"2026-02-23T13:13:05.494640+0000","mapping_epoch":155,"log_start":"130'7000","ondisk_log_start":"130'7000","created":126,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T13:11:36.450546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:11:36.450546+0000","last_clean_scrub_stamp":"2026-02-23T13:11:36.450546+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T13:13:06.181031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":139804672,"num_objects":2134,"num_object_clones":0,"num_object_copies":6402,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2134,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17066,"num_write_kb":136528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":638,"num_bytes_recovered":41754624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,3],"acting":[0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.5","version":"161'15784","reported_seq":17190,"reported_epoch":161,"state":"active+clean","last_fresh":"2026-02-23T13:13:01.678765+0000","last_change":"2026-02-23T13:13:00.145165+0000","last_active":"2026-02-23T13:13:01.678765+0000","last_peered":"2026-02-23T13:13:01.678765+0000","last_clean":"2026-02-23T13:13:01.678765+0000","last_became_active":"2026-02-23T13:12:58.010232+0000","last_became_peered":"2026-02-23T13:12:58.010232+0000","last_unstale":"2026-02-23T13:13:01.678765+0000","last_undegraded":"2026-02-23T13:13:01.678765+0000","last_fullsized":"2026-02-23T13:13:01.678765+0000","mapping_epoch":157,"log_start":"130'5700","ondisk_log_start":"130'5700","created":126,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"160'15384","last_scrub_stamp":"2026-02-23T13:13:00.145123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:11:36.450546+0000","last_clean_scrub_stamp":"2026-02-23T13:13:00.145123+0000","objects_scrubbed":1915,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T13:14:02.486042+0000","scrub_duration":1115,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":129302528,"num_objects":1973,"num_object_clones":0,"num_object_copies":5919,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1973,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15784,"num_write_kb":126272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":631,"num_bytes_recovered":41328640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"28'99","reported_seq":343,"reported_epoch":161,"state":"active+clean","last_fresh":"2026-02-23T13:13:01.022564+0000","last_change":"2026-02-23T13:12:52.005066+0000","last_active":"2026-02-23T13:13:01.022564+0000","last_peered":"2026-02-23T13:13:01.022564+0000","last_clean":"2026-02-23T13:13:01.022564+0000","last_became_active":"2026-02-23T13:12:52.004933+0000","last_became_peered":"2026-02-23T13:12:52.004933+0000","last_unstale":"2026-02-23T13:13:01.022564+0000","last_undegraded":"2026-02-23T13:13:01.022564+0000","last_fullsized":"2026-02-23T13:13:01.022564+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'99","last_scrub_stamp":"2026-02-23T13:11:39.969814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T13:07:05.619208+0000","last_clean_scrub_stamp":"2026-02-23T13:11:39.969814+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":122,"num_read_kb":228,"num_write":135,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":16,"stat_sum":{"num_bytes":2194808832,"num_objects":33494,"num_object_clones":0,"num_object_copies":100482,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33494,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":267921,"num_write_kb":2143368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11220,"num_bytes_recovered":734011392,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":6586433536,"data_stored":6586433536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160821,"ondisk_log_size":160821,"up":48,"acting":48,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":122,"num_read_kb":228,"num_write":135,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":80,"seq":343597383719,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":734003200,"kb_used":2345344,"kb_used_data":1536228,"kb_used_omap":61,"kb_used_meta":809026,"kb_avail":731657856,"statfs":{"total":751619276800,"available":749217644544,"internally_reserved":0,"allocated":1573097472,"data_stored":1571942749,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63119,"internal_metadata":828442993},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":152,"seq":652835028996,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":734003200,"kb_used":1992928,"kb_used_data":1874892,"kb_used_omap":16,"kb_used_meta":117999,"kb_avail":732010272,"statfs":{"total":751619276800,"available":749578518528,"internally_reserved":0,"allocated":1919889408,"data_stored":1918718919,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17168,"internal_metadata":120831216},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":144,"seq":618475290630,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":734003200,"kb_used":1514348,"kb_used_data":1353944,"kb_used_omap":20,"kb_used_meta":160363,"kb_avail":732488852,"statfs":{"total":751619276800,"available":750068584448,"internally_reserved":0,"allocated":1386438656,"data_stored":1385271719,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20724,"internal_metadata":164212492},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":106,"seq":455266533402,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":734003200,"kb_used":2189952,"kb_used_data":1610668,"kb_used_omap":42,"kb_used_meta":579221,"kb_avail":731813248,"statfs":{"total":751619276800,"available":749376765952,"internally_reserved":0,"allocated":1649324032,"data_stored":1648153543,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":43516,"internal_metadata":593122820},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,2,3,4,7],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1666637824,"data_stored":1666637824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1397817344,"data_stored":1397817344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1932910592,"data_stored":1932910592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1589067776,"data_stored":1589067776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-23T13:13:06.918 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2026-02-23T13:13:06.918 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2026-02-23T13:13:06.919 INFO:tasks.thrashosds.thrasher:Killing osd 2, live_osds are [3, 0, 1, 2] 2026-02-23T13:13:06.919 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2026-02-23T13:13:06.919 DEBUG:teuthology.orchestra.run.trial185:> sudo systemctl stop ceph-5f823084-10b8-11f1-be80-d404e6e7d460@osd.2 2026-02-23T13:13:07.033 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:13:07.096 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:13:07 trial068 ceph-mon[38805]: 4.1 scrub ok 2026-02-23T13:13:07.096 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:13:07 trial068 ceph-mon[38805]: pgmap v308: 17 pgs: 17 active+clean; 2.0 GiB data, 7.7 GiB used, 2.7 TiB / 2.7 TiB avail; 27 MiB/s wr, 3.46k op/s; 5.3 MiB/s, 84 objects/s recovering 2026-02-23T13:13:07.096 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:13:07 trial068 ceph-mon[38805]: 4.c scrub ok 2026-02-23T13:13:07.165 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:13:07.163+0000 7fc6aedaf640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:13:07.165 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:13:07.164+0000 7fc6aedaf640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:13:07.183 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:13:07.184 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:13:07.198 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:13:07.198 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:13:07.215 INFO:teuthology.orchestra.run.trial068.stdout:osd.2: {} 2026-02-23T13:13:07.215 INFO:teuthology.orchestra.run.trial068.stderr:osd.2: osd_enable_op_tracker = '' 2026-02-23T13:13:07.231 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:13:07.231 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:13:07.357 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:13:06 trial185 systemd[1]: Stopping Ceph osd.2 for 5f823084-10b8-11f1-be80-d404e6e7d460... 2026-02-23T13:13:07.357 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:13:07 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-2[75217]: 2026-02-23T13:13:07.044+0000 7f1b9ccef640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --osd-objectstore=bluestore (PID: 1) UID: 0 2026-02-23T13:13:07.357 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:13:07 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-2[75217]: 2026-02-23T13:13:07.044+0000 7f1b9ccef640 -1 osd.2 162 *** Got signal Terminated *** 2026-02-23T13:13:07.357 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:13:07 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-2[75217]: 2026-02-23T13:13:07.044+0000 7f1b9ccef640 -1 osd.2 162 *** Immediate shutdown (osd_fast_shutdown=true) *** 2026-02-23T13:13:07.554 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 89 128 279219 279091 24.4968 28.6172 0.0215509 0.0408094 2026-02-23T13:13:08.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:13:08 trial068 ceph-mon[38805]: from='client.14745 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2026-02-23T13:13:08.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:13:08 trial068 ceph-mon[38805]: osd.2 marked itself down and dead 2026-02-23T13:13:08.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:13:08 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2514601344' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:13:08.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:13:08 trial068 ceph-mon[38805]: 4.9 scrub starts 2026-02-23T13:13:08.409 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:13:08 trial185 podman[75960]: 2026-02-23 13:13:08.12051713 +0000 UTC m=+1.091942718 container died 1d17780e12e57ff76768afce1dc67bc526adc9883ba076bc4e9e0a3973d274f1 (image=quay.ceph.io/ceph-ci/ceph@sha256:49621faaa845c39b2aae72e4fa24123ee9b2ac3fb8ae834753907068ee2972ea, name=ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-2, FROM_IMAGE=quay.io/centos/centos:stream9, OSD_FLAVOR=default, org.label-schema.schema-version=1.0, org.opencontainers.image.documentation=https://docs.ceph.com/, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_SHA1=e371433fcd1390a0b82cfb18effc422c1f5d8c1d, GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfsganesha-5/, io.buildah.version=1.41.3, org.label-schema.build-date=20260216, org.label-schema.license=GPLv2, ceph=True, CEPH_REF=wip-shraddhaag-enable-cephadm-tests, org.opencontainers.image.authors=Ceph Release Team , CEPH_GIT_REPO=https://github.com/ceph/ceph-ci.git) 2026-02-23T13:13:08.409 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:13:08 trial185 podman[75960]: 2026-02-23 13:13:08.130069555 +0000 UTC m=+1.101495153 container remove 1d17780e12e57ff76768afce1dc67bc526adc9883ba076bc4e9e0a3973d274f1 (image=quay.ceph.io/ceph-ci/ceph@sha256:49621faaa845c39b2aae72e4fa24123ee9b2ac3fb8ae834753907068ee2972ea, name=ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-2, org.opencontainers.image.authors=Ceph Release Team , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, CEPH_GIT_REPO=https://github.com/ceph/ceph-ci.git, CEPH_SHA1=e371433fcd1390a0b82cfb18effc422c1f5d8c1d, OSD_FLAVOR=default, FROM_IMAGE=quay.io/centos/centos:stream9, GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfsganesha-5/, org.label-schema.vendor=CentOS, ceph=True, CEPH_REF=wip-shraddhaag-enable-cephadm-tests, org.opencontainers.image.documentation=https://docs.ceph.com/, org.label-schema.build-date=20260216, org.label-schema.schema-version=1.0, io.buildah.version=1.41.3) 2026-02-23T13:13:08.409 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:13:08 trial185 bash[75960]: ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-2 2026-02-23T13:13:08.409 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:13:08 trial185 podman[76081]: 2026-02-23 13:13:08.256163284 +0000 UTC m=+0.016584806 container create 39050f872736470672f2afc76ce4fc3979b3ee87ffb69a354ee62bf00fc585f2 (image=quay.ceph.io/ceph-ci/ceph@sha256:49621faaa845c39b2aae72e4fa24123ee9b2ac3fb8ae834753907068ee2972ea, name=ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-2-deactivate, OSD_FLAVOR=default, org.label-schema.build-date=20260216, org.label-schema.schema-version=1.0, FROM_IMAGE=quay.io/centos/centos:stream9, org.opencontainers.image.authors=Ceph Release Team , CEPH_SHA1=e371433fcd1390a0b82cfb18effc422c1f5d8c1d, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, CEPH_REF=wip-shraddhaag-enable-cephadm-tests, org.label-schema.license=GPLv2, io.buildah.version=1.41.3, org.opencontainers.image.documentation=https://docs.ceph.com/, GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfsganesha-5/, CEPH_GIT_REPO=https://github.com/ceph/ceph-ci.git) 2026-02-23T13:13:08.409 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:13:08 trial185 podman[76081]: 2026-02-23 13:13:08.287302155 +0000 UTC m=+0.047723667 container init 39050f872736470672f2afc76ce4fc3979b3ee87ffb69a354ee62bf00fc585f2 (image=quay.ceph.io/ceph-ci/ceph@sha256:49621faaa845c39b2aae72e4fa24123ee9b2ac3fb8ae834753907068ee2972ea, name=ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-2-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.41.3, FROM_IMAGE=quay.io/centos/centos:stream9, org.opencontainers.image.authors=Ceph Release Team , CEPH_SHA1=e371433fcd1390a0b82cfb18effc422c1f5d8c1d, GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfsganesha-5/, OSD_FLAVOR=default, org.label-schema.build-date=20260216, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_GIT_REPO=https://github.com/ceph/ceph-ci.git, CEPH_REF=wip-shraddhaag-enable-cephadm-tests, org.opencontainers.image.documentation=https://docs.ceph.com/) 2026-02-23T13:13:08.409 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:13:08 trial185 podman[76081]: 2026-02-23 13:13:08.28980543 +0000 UTC m=+0.050226952 container start 39050f872736470672f2afc76ce4fc3979b3ee87ffb69a354ee62bf00fc585f2 (image=quay.ceph.io/ceph-ci/ceph@sha256:49621faaa845c39b2aae72e4fa24123ee9b2ac3fb8ae834753907068ee2972ea, name=ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-2-deactivate, io.buildah.version=1.41.3, org.label-schema.schema-version=1.0, FROM_IMAGE=quay.io/centos/centos:stream9, org.opencontainers.image.authors=Ceph Release Team , CEPH_SHA1=e371433fcd1390a0b82cfb18effc422c1f5d8c1d, org.opencontainers.image.documentation=https://docs.ceph.com/, CEPH_GIT_REPO=https://github.com/ceph/ceph-ci.git, ceph=True, GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfsganesha-5/, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, OSD_FLAVOR=default, org.label-schema.build-date=20260216, CEPH_REF=wip-shraddhaag-enable-cephadm-tests, org.label-schema.license=GPLv2) 2026-02-23T13:13:08.409 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:13:08 trial185 podman[76081]: 2026-02-23 13:13:08.290326096 +0000 UTC m=+0.050747618 container attach 39050f872736470672f2afc76ce4fc3979b3ee87ffb69a354ee62bf00fc585f2 (image=quay.ceph.io/ceph-ci/ceph@sha256:49621faaa845c39b2aae72e4fa24123ee9b2ac3fb8ae834753907068ee2972ea, name=ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-2-deactivate, CEPH_GIT_REPO=https://github.com/ceph/ceph-ci.git, org.label-schema.build-date=20260216, CEPH_SHA1=e371433fcd1390a0b82cfb18effc422c1f5d8c1d, io.buildah.version=1.41.3, FROM_IMAGE=quay.io/centos/centos:stream9, org.opencontainers.image.authors=Ceph Release Team , org.label-schema.schema-version=1.0, CEPH_REF=wip-shraddhaag-enable-cephadm-tests, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, OSD_FLAVOR=default, org.opencontainers.image.documentation=https://docs.ceph.com/, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfsganesha-5/) 2026-02-23T13:13:08.409 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:13:08 trial185 podman[76081]: 2026-02-23 13:13:08.249139551 +0000 UTC m=+0.009561073 image pull 63df4b83db6f95a8ba965919b3671cb27173117c23b5c950aaadc8913f56213c quay.ceph.io/ceph-ci/ceph@sha256:49621faaa845c39b2aae72e4fa24123ee9b2ac3fb8ae834753907068ee2972ea 2026-02-23T13:13:08.443 DEBUG:teuthology.orchestra.run.trial185:> sudo pkill -f 'journalctl -f -n 0 -u ceph-5f823084-10b8-11f1-be80-d404e6e7d460@osd.2.service' 2026-02-23T13:13:08.479 DEBUG:teuthology.orchestra.run:got remote process result: None 2026-02-23T13:13:08.479 INFO:tasks.cephadm.osd.2:Stopped osd.2 2026-02-23T13:13:08.554 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 90 128 280783 280655 24.3604 12.2188 0.0373109 0.0408016 2026-02-23T13:13:09.280 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:13:09.304 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:13:09 trial068 ceph-mon[38805]: 4.3 scrub starts 2026-02-23T13:13:09.304 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:13:09 trial068 ceph-mon[38805]: Health check failed: 1 osds down (OSD_DOWN) 2026-02-23T13:13:09.304 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:13:09 trial068 ceph-mon[38805]: osdmap e163: 4 total, 3 up, 4 in 2026-02-23T13:13:09.304 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:13:09 trial068 ceph-mon[38805]: pgmap v310: 17 pgs: 1 active+clean+scrubbing, 4 stale+active+clean, 12 active+clean; 2.1 GiB data, 7.8 GiB used, 2.7 TiB / 2.7 TiB avail; 23 MiB/s wr, 2.95k op/s 2026-02-23T13:13:09.304 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:13:09 trial068 ceph-mon[38805]: osdmap e164: 4 total, 3 up, 4 in 2026-02-23T13:13:09.395 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:13:09.490 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:13:09.488+0000 7fd4ea5e6640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:13:09.490 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:13:09.489+0000 7fd4ea5e6640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:13:09.502 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:13:09.502 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:13:09.512 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:13:09.513 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:13:09.513 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:13:09.524 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:13:09.524 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:13:09.553 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:13:09.553 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:13:09.555 INFO:tasks.radosbench.radosbench.0.trial068.stdout:Total time run: 90.5172 2026-02-23T13:13:09.555 INFO:tasks.radosbench.radosbench.0.trial068.stdout:Total writes made: 280783 2026-02-23T13:13:09.555 INFO:tasks.radosbench.radosbench.0.trial068.stdout:Write size: 8192 2026-02-23T13:13:09.555 INFO:tasks.radosbench.radosbench.0.trial068.stdout:Object size: 65536 2026-02-23T13:13:09.555 INFO:tasks.radosbench.radosbench.0.trial068.stdout:Bandwidth (MB/sec): 24.2343 2026-02-23T13:13:09.555 INFO:tasks.radosbench.radosbench.0.trial068.stdout:Stddev Bandwidth: 21.7627 2026-02-23T13:13:09.555 INFO:tasks.radosbench.radosbench.0.trial068.stdout:Max bandwidth (MB/sec): 86.8438 2026-02-23T13:13:09.555 INFO:tasks.radosbench.radosbench.0.trial068.stdout:Min bandwidth (MB/sec): 0 2026-02-23T13:13:09.555 INFO:tasks.radosbench.radosbench.0.trial068.stdout:Average IOPS: 3101 2026-02-23T13:13:09.555 INFO:tasks.radosbench.radosbench.0.trial068.stdout:Stddev IOPS: 2785.63 2026-02-23T13:13:09.556 INFO:tasks.radosbench.radosbench.0.trial068.stdout:Max IOPS: 11116 2026-02-23T13:13:09.556 INFO:tasks.radosbench.radosbench.0.trial068.stdout:Min IOPS: 0 2026-02-23T13:13:09.556 INFO:tasks.radosbench.radosbench.0.trial068.stdout:Average Latency(s): 0.041247 2026-02-23T13:13:09.556 INFO:tasks.radosbench.radosbench.0.trial068.stdout:Stddev Latency(s): 0.422172 2026-02-23T13:13:09.556 INFO:tasks.radosbench.radosbench.0.trial068.stdout:Max latency(s): 20.2504 2026-02-23T13:13:09.556 INFO:tasks.radosbench.radosbench.0.trial068.stdout:Min latency(s): 0.0010091 2026-02-23T13:13:09.560 INFO:tasks.radosbench.radosbench.0.trial068.stdout:Cleaning up (deleting benchmark objects) 2026-02-23T13:13:10.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:13:10 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T13:13:10.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:13:10 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T13:13:10.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:13:10 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/4115360595' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:13:11.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:13:11 trial068 ceph-mon[38805]: pgmap v312: 17 pgs: 1 active+clean+scrubbing, 4 stale+active+clean, 12 active+clean; 2.1 GiB data, 7.8 GiB used, 2.7 TiB / 2.7 TiB avail; 22 MiB/s wr, 2.82k op/s 2026-02-23T13:13:11.554 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:13:11.702 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:13:11.817 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:13:11.815+0000 7f0529790640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:13:11.817 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:13:11.815+0000 7f0529790640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:13:11.832 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:13:11.832 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:13:11.848 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:13:11.848 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:13:11.848 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:13:11.865 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:13:11.865 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:13:11.904 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:13:11.904 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:13:12.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:13:12 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3209026938' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:13:13.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:13:13 trial068 ceph-mon[38805]: pgmap v313: 17 pgs: 15 active+undersized+degraded, 2 active+clean; 1.8 GiB data, 7.7 GiB used, 2.7 TiB / 2.7 TiB avail; 127 B/s rd, 25 MiB/s wr, 4.04k op/s; 24925/86475 objects degraded (28.823%); 12 KiB/s, 0 objects/s recovering 2026-02-23T13:13:13.480 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [2] live_osds: [3, 0, 1] 2026-02-23T13:13:13.480 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2026-02-23T13:13:13.480 INFO:tasks.thrashosds.thrasher:Reviving osd 2 2026-02-23T13:13:13.480 INFO:tasks.cephadm.osd.2:Restarting osd.2 (starting--it wasn't running)... 2026-02-23T13:13:13.480 DEBUG:teuthology.orchestra.run.trial185:osd.2> sudo journalctl -f -n 0 -u ceph-5f823084-10b8-11f1-be80-d404e6e7d460@osd.2.service 2026-02-23T13:13:13.483 DEBUG:teuthology.orchestra.run.trial185:> sudo systemctl start ceph-5f823084-10b8-11f1-be80-d404e6e7d460@osd.2 2026-02-23T13:13:13.755 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:13:13 trial185 systemd[1]: Starting Ceph osd.2 for 5f823084-10b8-11f1-be80-d404e6e7d460... 2026-02-23T13:13:13.755 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:13:13 trial185 podman[76260]: 2026-02-23 13:13:13.661849042 +0000 UTC m=+0.018454412 container create 77bbee286819e1714d9e8ea8d186d907eb8f125b852a279719efa15fcec2ca19 (image=quay.ceph.io/ceph-ci/ceph@sha256:49621faaa845c39b2aae72e4fa24123ee9b2ac3fb8ae834753907068ee2972ea, name=ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-2-activate, org.opencontainers.image.authors=Ceph Release Team , CEPH_GIT_REPO=https://github.com/ceph/ceph-ci.git, ceph=True, org.label-schema.build-date=20260216, org.label-schema.vendor=CentOS, GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfsganesha-5/, io.buildah.version=1.41.3, CEPH_REF=wip-shraddhaag-enable-cephadm-tests, org.opencontainers.image.documentation=https://docs.ceph.com/, FROM_IMAGE=quay.io/centos/centos:stream9, OSD_FLAVOR=default, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_SHA1=e371433fcd1390a0b82cfb18effc422c1f5d8c1d) 2026-02-23T13:13:13.756 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:13:13 trial185 podman[76260]: 2026-02-23 13:13:13.703313369 +0000 UTC m=+0.059918749 container init 77bbee286819e1714d9e8ea8d186d907eb8f125b852a279719efa15fcec2ca19 (image=quay.ceph.io/ceph-ci/ceph@sha256:49621faaa845c39b2aae72e4fa24123ee9b2ac3fb8ae834753907068ee2972ea, name=ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-2-activate, org.label-schema.license=GPLv2, GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfsganesha-5/, CEPH_GIT_REPO=https://github.com/ceph/ceph-ci.git, CEPH_SHA1=e371433fcd1390a0b82cfb18effc422c1f5d8c1d, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.opencontainers.image.authors=Ceph Release Team , CEPH_REF=wip-shraddhaag-enable-cephadm-tests, org.opencontainers.image.documentation=https://docs.ceph.com/, OSD_FLAVOR=default, io.buildah.version=1.41.3, FROM_IMAGE=quay.io/centos/centos:stream9, org.label-schema.schema-version=1.0, org.label-schema.build-date=20260216, org.label-schema.vendor=CentOS) 2026-02-23T13:13:13.756 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:13:13 trial185 podman[76260]: 2026-02-23 13:13:13.705877437 +0000 UTC m=+0.062482817 container start 77bbee286819e1714d9e8ea8d186d907eb8f125b852a279719efa15fcec2ca19 (image=quay.ceph.io/ceph-ci/ceph@sha256:49621faaa845c39b2aae72e4fa24123ee9b2ac3fb8ae834753907068ee2972ea, name=ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-2-activate, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, OSD_FLAVOR=default, org.label-schema.vendor=CentOS, CEPH_GIT_REPO=https://github.com/ceph/ceph-ci.git, CEPH_REF=wip-shraddhaag-enable-cephadm-tests, CEPH_SHA1=e371433fcd1390a0b82cfb18effc422c1f5d8c1d, org.opencontainers.image.documentation=https://docs.ceph.com/, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.opencontainers.image.authors=Ceph Release Team , org.label-schema.build-date=20260216, GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfsganesha-5/, io.buildah.version=1.41.3, FROM_IMAGE=quay.io/centos/centos:stream9) 2026-02-23T13:13:13.756 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:13:13 trial185 podman[76260]: 2026-02-23 13:13:13.706417971 +0000 UTC m=+0.063023351 container attach 77bbee286819e1714d9e8ea8d186d907eb8f125b852a279719efa15fcec2ca19 (image=quay.ceph.io/ceph-ci/ceph@sha256:49621faaa845c39b2aae72e4fa24123ee9b2ac3fb8ae834753907068ee2972ea, name=ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-2-activate, io.buildah.version=1.41.3, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.vendor=CentOS, org.opencontainers.image.documentation=https://docs.ceph.com/, org.label-schema.license=GPLv2, org.opencontainers.image.authors=Ceph Release Team , org.label-schema.build-date=20260216, CEPH_GIT_REPO=https://github.com/ceph/ceph-ci.git, CEPH_SHA1=e371433fcd1390a0b82cfb18effc422c1f5d8c1d, OSD_FLAVOR=default, GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfsganesha-5/, CEPH_REF=wip-shraddhaag-enable-cephadm-tests, FROM_IMAGE=quay.io/centos/centos:stream9) 2026-02-23T13:13:13.756 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:13:13 trial185 podman[76260]: 2026-02-23 13:13:13.65541752 +0000 UTC m=+0.012022900 image pull 63df4b83db6f95a8ba965919b3671cb27173117c23b5c950aaadc8913f56213c quay.ceph.io/ceph-ci/ceph@sha256:49621faaa845c39b2aae72e4fa24123ee9b2ac3fb8ae834753907068ee2972ea 2026-02-23T13:13:13.905 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:13:14.062 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:13:14.107 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:13:13 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:13:13.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:13:14.107 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:13:13 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-2-activate[76280]: Running command: /usr/bin/ceph-authtool --gen-print-key 2026-02-23T13:13:14.108 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:13:13 trial185 bash[76260]: Running command: /usr/bin/ceph-authtool --gen-print-key 2026-02-23T13:13:14.108 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:13:13 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-2-activate[76280]: Running command: /usr/bin/ceph-authtool --gen-print-key 2026-02-23T13:13:14.108 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:13:13 trial185 bash[76260]: Running command: /usr/bin/ceph-authtool --gen-print-key 2026-02-23T13:13:14.222 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:13:14.220+0000 7fcae6f47640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:13:14.222 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:13:14.220+0000 7fcae6f47640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:13:14.222 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:13:13 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:13:13] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:13:14.222 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:13:14 trial068 ceph-mon[38805]: Health check failed: Degraded data redundancy: 24925/86475 objects degraded (28.823%), 15 pgs degraded (PG_DEGRADED) 2026-02-23T13:13:14.243 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:13:14.243 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:13:14.267 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:13:14.268 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:13:14.268 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:13:14.284 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:13:14.284 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:13:14.325 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:13:14.326 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:13:14.454 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:13:14 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-2-activate[76280]: --> Failed to activate via raw: did not find any matching OSD to activate 2026-02-23T13:13:14.454 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:13:14 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-2-activate[76280]: Running command: /usr/bin/ceph-authtool --gen-print-key 2026-02-23T13:13:14.454 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:13:14 trial185 bash[76260]: --> Failed to activate via raw: did not find any matching OSD to activate 2026-02-23T13:13:14.454 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:13:14 trial185 bash[76260]: Running command: /usr/bin/ceph-authtool --gen-print-key 2026-02-23T13:13:14.454 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:13:14 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-2-activate[76280]: Running command: /usr/bin/ceph-authtool --gen-print-key 2026-02-23T13:13:14.454 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:13:14 trial185 bash[76260]: Running command: /usr/bin/ceph-authtool --gen-print-key 2026-02-23T13:13:14.455 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:13:14 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-2-activate[76280]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2026-02-23T13:13:14.455 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:13:14 trial185 bash[76260]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2026-02-23T13:13:14.455 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:13:14 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-2-activate[76280]: Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/vg_nvme/lv_4 --path /var/lib/ceph/osd/ceph-2 --no-mon-config 2026-02-23T13:13:14.455 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:13:14 trial185 bash[76260]: Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/vg_nvme/lv_4 --path /var/lib/ceph/osd/ceph-2 --no-mon-config 2026-02-23T13:13:14.455 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:13:14 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-2-activate[76280]: Running command: /usr/bin/ln -snf /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-2/block 2026-02-23T13:13:14.455 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:13:14 trial185 bash[76260]: Running command: /usr/bin/ln -snf /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-2/block 2026-02-23T13:13:14.455 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:13:14 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-2-activate[76280]: Running command: /usr/bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-2/block 2026-02-23T13:13:14.455 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:13:14 trial185 bash[76260]: Running command: /usr/bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-2/block 2026-02-23T13:13:14.455 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:13:14 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-2-activate[76280]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2026-02-23T13:13:14.455 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:13:14 trial185 bash[76260]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2026-02-23T13:13:14.455 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:13:14 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-2-activate[76280]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2026-02-23T13:13:14.455 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:13:14 trial185 bash[76260]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2026-02-23T13:13:14.456 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:13:14 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-2-activate[76280]: --> ceph-volume lvm activate successful for osd ID: 2 2026-02-23T13:13:14.456 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:13:14 trial185 bash[76260]: --> ceph-volume lvm activate successful for osd ID: 2 2026-02-23T13:13:14.456 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:13:14 trial185 podman[76260]: 2026-02-23 13:13:14.353266916 +0000 UTC m=+0.709872296 container died 77bbee286819e1714d9e8ea8d186d907eb8f125b852a279719efa15fcec2ca19 (image=quay.ceph.io/ceph-ci/ceph@sha256:49621faaa845c39b2aae72e4fa24123ee9b2ac3fb8ae834753907068ee2972ea, name=ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-2-activate, io.buildah.version=1.41.3, FROM_IMAGE=quay.io/centos/centos:stream9, CEPH_REF=wip-shraddhaag-enable-cephadm-tests, CEPH_SHA1=e371433fcd1390a0b82cfb18effc422c1f5d8c1d, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, OSD_FLAVOR=default, GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfsganesha-5/, org.label-schema.schema-version=1.0, org.label-schema.build-date=20260216, CEPH_GIT_REPO=https://github.com/ceph/ceph-ci.git, org.opencontainers.image.documentation=https://docs.ceph.com/, org.label-schema.license=GPLv2, org.opencontainers.image.authors=Ceph Release Team , org.label-schema.vendor=CentOS) 2026-02-23T13:13:14.456 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:13:14 trial185 podman[76260]: 2026-02-23 13:13:14.364741626 +0000 UTC m=+0.721347007 container remove 77bbee286819e1714d9e8ea8d186d907eb8f125b852a279719efa15fcec2ca19 (image=quay.ceph.io/ceph-ci/ceph@sha256:49621faaa845c39b2aae72e4fa24123ee9b2ac3fb8ae834753907068ee2972ea, name=ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-2-activate, org.label-schema.build-date=20260216, OSD_FLAVOR=default, io.buildah.version=1.41.3, org.opencontainers.image.authors=Ceph Release Team , ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.opencontainers.image.documentation=https://docs.ceph.com/, CEPH_GIT_REPO=https://github.com/ceph/ceph-ci.git, GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfsganesha-5/, FROM_IMAGE=quay.io/centos/centos:stream9, CEPH_SHA1=e371433fcd1390a0b82cfb18effc422c1f5d8c1d, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_REF=wip-shraddhaag-enable-cephadm-tests, org.label-schema.vendor=CentOS) 2026-02-23T13:13:14.456 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:13:14 trial185 podman[76479]: 2026-02-23 13:13:14.451373274 +0000 UTC m=+0.016420247 container create 1b2663ad5b4f5fe41e99da7641a26011278c89ab49abbf0d9def63f68ec27ced (image=quay.ceph.io/ceph-ci/ceph@sha256:49621faaa845c39b2aae72e4fa24123ee9b2ac3fb8ae834753907068ee2972ea, name=ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-2, org.label-schema.license=GPLv2, io.buildah.version=1.41.3, org.opencontainers.image.documentation=https://docs.ceph.com/, org.opencontainers.image.authors=Ceph Release Team , org.label-schema.vendor=CentOS, OSD_FLAVOR=default, FROM_IMAGE=quay.io/centos/centos:stream9, CEPH_GIT_REPO=https://github.com/ceph/ceph-ci.git, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, CEPH_SHA1=e371433fcd1390a0b82cfb18effc422c1f5d8c1d, CEPH_REF=wip-shraddhaag-enable-cephadm-tests, org.label-schema.schema-version=1.0, GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfsganesha-5/, org.label-schema.build-date=20260216) 2026-02-23T13:13:14.490 DEBUG:teuthology.orchestra.run.trial185:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph daemon osd.2 dump_ops_in_flight 2026-02-23T13:13:14.629 INFO:teuthology.orchestra.run.trial185.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/config/ceph.conf 2026-02-23T13:13:14.706 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:13:14 trial185 podman[76479]: 2026-02-23 13:13:14.480993026 +0000 UTC m=+0.046039939 container init 1b2663ad5b4f5fe41e99da7641a26011278c89ab49abbf0d9def63f68ec27ced (image=quay.ceph.io/ceph-ci/ceph@sha256:49621faaa845c39b2aae72e4fa24123ee9b2ac3fb8ae834753907068ee2972ea, name=ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-2, org.label-schema.license=GPLv2, io.buildah.version=1.41.3, org.label-schema.name=CentOS Stream 9 Base Image, GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfsganesha-5/, CEPH_GIT_REPO=https://github.com/ceph/ceph-ci.git, org.opencontainers.image.documentation=https://docs.ceph.com/, org.label-schema.build-date=20260216, ceph=True, FROM_IMAGE=quay.io/centos/centos:stream9, org.opencontainers.image.authors=Ceph Release Team , CEPH_REF=wip-shraddhaag-enable-cephadm-tests, CEPH_SHA1=e371433fcd1390a0b82cfb18effc422c1f5d8c1d, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, OSD_FLAVOR=default) 2026-02-23T13:13:14.706 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:13:14 trial185 podman[76479]: 2026-02-23 13:13:14.483889518 +0000 UTC m=+0.048936431 container start 1b2663ad5b4f5fe41e99da7641a26011278c89ab49abbf0d9def63f68ec27ced (image=quay.ceph.io/ceph-ci/ceph@sha256:49621faaa845c39b2aae72e4fa24123ee9b2ac3fb8ae834753907068ee2972ea, name=ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-2, FROM_IMAGE=quay.io/centos/centos:stream9, io.buildah.version=1.41.3, org.opencontainers.image.authors=Ceph Release Team , OSD_FLAVOR=default, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfsganesha-5/, org.label-schema.license=GPLv2, org.opencontainers.image.documentation=https://docs.ceph.com/, CEPH_REF=wip-shraddhaag-enable-cephadm-tests, CEPH_SHA1=e371433fcd1390a0b82cfb18effc422c1f5d8c1d, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_GIT_REPO=https://github.com/ceph/ceph-ci.git, org.label-schema.build-date=20260216) 2026-02-23T13:13:14.706 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:13:14 trial185 bash[76479]: 1b2663ad5b4f5fe41e99da7641a26011278c89ab49abbf0d9def63f68ec27ced 2026-02-23T13:13:14.706 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:13:14 trial185 podman[76479]: 2026-02-23 13:13:14.445166734 +0000 UTC m=+0.010213657 image pull 63df4b83db6f95a8ba965919b3671cb27173117c23b5c950aaadc8913f56213c quay.ceph.io/ceph-ci/ceph@sha256:49621faaa845c39b2aae72e4fa24123ee9b2ac3fb8ae834753907068ee2972ea 2026-02-23T13:13:14.706 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:13:14 trial185 systemd[1]: Started Ceph osd.2 for 5f823084-10b8-11f1-be80-d404e6e7d460. 2026-02-23T13:13:14.706 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:13:14 trial185 ceph-osd[76505]: WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:13:14.706 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:13:14 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-2[76501]: 2026-02-23T13:13:14.498+0000 7f3f0414e780 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:13:14.706 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:13:14 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-2[76501]: 2026-02-23T13:13:14.499+0000 7f3f0414e780 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:13:14.707 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:13:14 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-2[76501]: 2026-02-23T13:13:14.499+0000 7f3f0414e780 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:13:14.766 INFO:teuthology.orchestra.run.trial185.stderr:no valid command found; 10 closest matches: 2026-02-23T13:13:14.766 INFO:teuthology.orchestra.run.trial185.stderr:0 2026-02-23T13:13:14.766 INFO:teuthology.orchestra.run.trial185.stderr:1 2026-02-23T13:13:14.766 INFO:teuthology.orchestra.run.trial185.stderr:2 2026-02-23T13:13:14.766 INFO:teuthology.orchestra.run.trial185.stderr:abort 2026-02-23T13:13:14.767 INFO:teuthology.orchestra.run.trial185.stderr:assert 2026-02-23T13:13:14.767 INFO:teuthology.orchestra.run.trial185.stderr:bluestore collections 2026-02-23T13:13:14.767 INFO:teuthology.orchestra.run.trial185.stderr:bluestore compression stats [] 2026-02-23T13:13:14.767 INFO:teuthology.orchestra.run.trial185.stderr:bluestore list [] [] 2026-02-23T13:13:14.767 INFO:teuthology.orchestra.run.trial185.stderr:bluestore onode metadata 2026-02-23T13:13:14.767 INFO:teuthology.orchestra.run.trial185.stderr:bluestore show sharding 2026-02-23T13:13:14.767 INFO:teuthology.orchestra.run.trial185.stderr:admin_socket: invalid command 2026-02-23T13:13:14.800 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-23T13:13:14.800 INFO:tasks.cephadm.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-23T13:13:15.106 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:13:14 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-2[76501]: 2026-02-23T13:13:14.744+0000 7f3f0414e780 -1 Falling back to public interface 2026-02-23T13:13:15.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:13:15 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1885304302' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:13:15.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:13:15 trial068 ceph-mon[38805]: pgmap v314: 17 pgs: 15 active+undersized+degraded, 2 active+clean; 1.8 GiB data, 7.7 GiB used, 2.7 TiB / 2.7 TiB avail; 127 B/s rd, 13 MiB/s wr, 2.39k op/s; 24925/86475 objects degraded (28.823%); 12 KiB/s, 0 objects/s recovering 2026-02-23T13:13:16.327 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:13:16.477 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:13:16.611 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:13:16.609+0000 7fadc6e51640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:13:16.611 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:13:16.609+0000 7fadc6e51640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:13:16.627 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:13:16.628 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:13:16.642 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:13:16.643 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:13:16.643 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:13:16.658 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:13:16.658 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:13:16.696 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:13:16.697 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:13:17.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:13:17 trial068 ceph-mon[38805]: pgmap v315: 17 pgs: 15 active+undersized+degraded, 2 active+clean; 1.1 GiB data, 6.0 GiB used, 2.7 TiB / 2.7 TiB avail; 124 B/s rd, 7.9 MiB/s wr, 3.18k op/s; 15884/51603 objects degraded (30.781%); 12 KiB/s, 0 objects/s recovering 2026-02-23T13:13:17.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:13:17 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3242815516' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:13:18.106 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:13:17 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-2[76501]: 2026-02-23T13:13:17.632+0000 7f3f0414e780 -1 osd.2 162 log_to_monitors true 2026-02-23T13:13:18.227 INFO:tasks.radosbench.radosbench.0.trial068.stdout:Removed 35098 objects 2026-02-23T13:13:18.228 INFO:tasks.radosbench.radosbench.0.trial068.stdout:Clean up completed and total clean up time :8.66726 2026-02-23T13:13:18.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:13:18 trial068 ceph-mon[38805]: from='osd.2 [v2:10.20.193.185:6800/309948183,v1:10.20.193.185:6801/309948183]' entity='osd.2' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]} : dispatch 2026-02-23T13:13:18.606 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:13:18 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-2[76501]: 2026-02-23T13:13:18.334+0000 7f3ef7f8f640 -1 osd.2 162 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2026-02-23T13:13:18.698 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:13:18.816 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:13:18.915 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:13:18.913+0000 7f40be517640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:13:18.916 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:13:18.914+0000 7f40be517640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:13:18.927 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:13:18.928 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:13:18.937 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:13:18.937 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:13:18.937 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:13:18.949 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:13:18.949 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:13:18.979 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:13:18.979 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:13:19.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:13:19 trial068 ceph-mon[38805]: pgmap v316: 17 pgs: 15 active+undersized+degraded, 2 active+clean; 524 MiB data, 5.1 GiB used, 2.7 TiB / 2.7 TiB avail; 102 B/s rd, 6.5 MiB/s wr, 3.50k op/s; 7061/25134 objects degraded (28.093%); 9.6 KiB/s, 0 objects/s recovering 2026-02-23T13:13:19.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:13:19 trial068 ceph-mon[38805]: Health check update: Degraded data redundancy: 7061/25134 objects degraded (28.093%), 15 pgs degraded (PG_DEGRADED) 2026-02-23T13:13:19.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:13:19 trial068 ceph-mon[38805]: from='osd.2 [v2:10.20.193.185:6800/309948183,v1:10.20.193.185:6801/309948183]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2026-02-23T13:13:19.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:13:19 trial068 ceph-mon[38805]: osdmap e165: 4 total, 3 up, 4 in 2026-02-23T13:13:19.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:13:19 trial068 ceph-mon[38805]: from='osd.2 [v2:10.20.193.185:6800/309948183,v1:10.20.193.185:6801/309948183]' entity='osd.2' cmd={"prefix": "osd crush create-or-move", "id": 2, "weight":0.6836, "args": ["host=trial185", "root=default"]} : dispatch 2026-02-23T13:13:19.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:13:19 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3925315261' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:13:19.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:13:19 trial068 ceph-mon[38805]: Health check cleared: OSD_DOWN (was: 1 osds down) 2026-02-23T13:13:19.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:13:19 trial068 ceph-mon[38805]: osd.2 [v2:10.20.193.185:6800/309948183,v1:10.20.193.185:6801/309948183] boot 2026-02-23T13:13:19.595 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:13:19 trial068 ceph-mon[38805]: osdmap e166: 4 total, 4 up, 4 in 2026-02-23T13:13:19.595 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:13:19 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2026-02-23T13:13:19.801 DEBUG:teuthology.orchestra.run.trial185:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph daemon osd.2 dump_ops_in_flight 2026-02-23T13:13:19.921 INFO:teuthology.orchestra.run.trial185.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/config/ceph.conf 2026-02-23T13:13:20.045 INFO:teuthology.orchestra.run.trial185.stdout:{ 2026-02-23T13:13:20.046 INFO:teuthology.orchestra.run.trial185.stdout: "ops": [], 2026-02-23T13:13:20.046 INFO:teuthology.orchestra.run.trial185.stdout: "num_ops": 0 2026-02-23T13:13:20.046 INFO:teuthology.orchestra.run.trial185.stdout:} 2026-02-23T13:13:20.980 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:13:21.103 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:13:21.310 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:13:21 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T13:13:21.310 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:13:21 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T13:13:21.310 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:13:21 trial068 ceph-mon[38805]: osdmap e167: 4 total, 4 up, 4 in 2026-02-23T13:13:21.310 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:13:21 trial068 ceph-mon[38805]: pgmap v320: 17 pgs: 15 active+undersized+degraded, 2 active+clean; 524 MiB data, 5.1 GiB used, 2.7 TiB / 2.7 TiB avail; 0 B/s wr, 3.41k op/s; 7061/25134 objects degraded (28.093%) 2026-02-23T13:13:21.370 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:13:21.368+0000 7fc5b4c3f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:13:21.370 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:13:21.369+0000 7fc5b4c3f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:13:21.382 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:13:21.382 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:13:21.392 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:13:21.392 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:13:21.403 INFO:teuthology.orchestra.run.trial068.stdout:osd.2: {} 2026-02-23T13:13:21.404 INFO:teuthology.orchestra.run.trial068.stderr:osd.2: osd_delete_sleep = '' osd_delete_sleep_hdd = '' osd_delete_sleep_hybrid = '' osd_delete_sleep_ssd = '' osd_enable_op_tracker = '' osd_recovery_max_active = '' osd_recovery_max_active_hdd = '' osd_recovery_max_active_ssd = '' osd_recovery_sleep = '' osd_recovery_sleep_degraded = '' osd_recovery_sleep_degraded_hdd = '' osd_recovery_sleep_degraded_hybrid = '' osd_recovery_sleep_degraded_ssd = '' osd_recovery_sleep_hdd = '' osd_recovery_sleep_hybrid = '' osd_recovery_sleep_ssd = '' osd_scrub_sleep = '' osd_snap_trim_sleep = '' osd_snap_trim_sleep_hdd = '' osd_snap_trim_sleep_hybrid = '' osd_snap_trim_sleep_ssd = '' 2026-02-23T13:13:21.414 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:13:21.414 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:13:22.223 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_2 2026-02-23T13:13:22.223 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd pool rm unique_pool_2 unique_pool_2 --yes-i-really-really-mean-it 2026-02-23T13:13:22.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:13:22 trial068 ceph-mon[38805]: osdmap e168: 4 total, 4 up, 4 in 2026-02-23T13:13:22.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:13:22 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1719852617' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:13:22.346 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:13:22.454 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:13:22.452+0000 7f95e92e1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:13:22.454 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:13:22.452+0000 7f95e92e1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:13:23.006 INFO:teuthology.orchestra.run.trial068.stderr:pool 'unique_pool_2' removed 2026-02-23T13:13:23.047 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2026-02-23T13:13:23.047 INFO:tasks.radosbench:Beginning radosbench... 2026-02-23T13:13:23.047 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_3 2026-02-23T13:13:23.048 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd pool create unique_pool_3 16 2026-02-23T13:13:23.166 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:13:23.261 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:13:23 trial068 ceph-mon[38805]: pgmap v322: 17 pgs: 4 active+recovery_wait+undersized+degraded+remapped, 10 remapped+peering, 3 active+clean; 577 KiB data, 3.6 GiB used, 2.7 TiB / 2.7 TiB avail; 0 B/s wr, 2.09k op/s; 8777/6 objects degraded (146283.333%) 2026-02-23T13:13:23.262 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:13:23 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3595065159' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true} : dispatch 2026-02-23T13:13:23.347 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:13:23.345+0000 7f2178609640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:13:23.347 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:13:23.345+0000 7f2178609640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:13:23.453 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:13:23.573 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:13:23.688 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:13:23.686+0000 7f187f07f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:13:23.688 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:13:23.686+0000 7f187f07f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:13:23.701 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:13:23.701 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:13:23.712 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:13:23.713 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:13:23.724 INFO:teuthology.orchestra.run.trial068.stdout:osd.2: {} 2026-02-23T13:13:23.724 INFO:teuthology.orchestra.run.trial068.stderr:osd.2: osd_enable_op_tracker = '' 2026-02-23T13:13:23.735 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:13:23.735 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:13:24.002 INFO:teuthology.orchestra.run.trial068.stderr:pool 'unique_pool_3' created 2026-02-23T13:13:24.042 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd pool application enable unique_pool_3 rados --yes-i-really-mean-it || true 2026-02-23T13:13:24.158 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:13:24.268 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:13:24.266+0000 7feafdd37640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:13:24.268 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:13:24.266+0000 7feafdd37640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:13:24.268 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:13:23 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:13:23] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:13:24.269 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:13:24 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3595065159' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]': finished 2026-02-23T13:13:24.269 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:13:24 trial068 ceph-mon[38805]: osdmap e169: 4 total, 4 up, 4 in 2026-02-23T13:13:24.269 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:13:24 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2212874263' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16} : dispatch 2026-02-23T13:13:24.269 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:13:24 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2029426387' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:13:24.269 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:13:24 trial068 ceph-mon[38805]: Health check update: Degraded data redundancy: 8777/6 objects degraded (146283.333%), 4 pgs degraded (PG_DEGRADED) 2026-02-23T13:13:24.269 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:13:24 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2212874263' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]': finished 2026-02-23T13:13:24.269 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:13:24 trial068 ceph-mon[38805]: osdmap e170: 4 total, 4 up, 4 in 2026-02-23T13:13:24.357 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:13:23 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:13:23.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:13:25.005 INFO:teuthology.orchestra.run.trial068.stderr:enabled application 'rados' on pool 'unique_pool_3' 2026-02-23T13:13:25.083 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [3, 0, 1, 2] 2026-02-23T13:13:25.083 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2026-02-23T13:13:25.083 INFO:tasks.thrashosds.thrasher:Killing osd 1, live_osds are [3, 0, 1, 2] 2026-02-23T13:13:25.084 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2026-02-23T13:13:25.084 DEBUG:teuthology.orchestra.run.trial068:> sudo systemctl stop ceph-5f823084-10b8-11f1-be80-d404e6e7d460@osd.1 2026-02-23T13:13:25.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:13:25 trial068 ceph-mon[38805]: pgmap v325: 17 pgs: 16 unknown, 1 active+clean; 577 KiB data, 3.4 GiB used, 2.7 TiB / 2.7 TiB avail 2026-02-23T13:13:25.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:13:25 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2351092605' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2026-02-23T13:13:25.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:13:25 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2351092605' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]': finished 2026-02-23T13:13:25.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:13:25 trial068 ceph-mon[38805]: osdmap e171: 4 total, 4 up, 4 in 2026-02-23T13:13:25.345 INFO:journalctl@ceph.osd.1.trial068.stdout:Feb 23 13:13:25 trial068 systemd[1]: Stopping Ceph osd.1 for 5f823084-10b8-11f1-be80-d404e6e7d460... 2026-02-23T13:13:25.345 INFO:journalctl@ceph.osd.1.trial068.stdout:Feb 23 13:13:25 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-1[99952]: 2026-02-23T13:13:25.182+0000 7f760c158640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --osd-objectstore=bluestore (PID: 1) UID: 0 2026-02-23T13:13:25.345 INFO:journalctl@ceph.osd.1.trial068.stdout:Feb 23 13:13:25 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-1[99952]: 2026-02-23T13:13:25.182+0000 7f760c158640 -1 osd.1 171 *** Got signal Terminated *** 2026-02-23T13:13:25.345 INFO:journalctl@ceph.osd.1.trial068.stdout:Feb 23 13:13:25 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-1[99952]: 2026-02-23T13:13:25.182+0000 7f760c158640 -1 osd.1 171 *** Immediate shutdown (osd_fast_shutdown=true) *** 2026-02-23T13:13:25.770 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:13:25.887 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:13:25.982 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:13:25.980+0000 7f67488d1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:13:25.982 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:13:25.980+0000 7f67488d1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:13:25.994 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:13:25.995 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:13:26.005 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:13:26.006 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:13:26.019 INFO:teuthology.orchestra.run.trial068.stdout:osd.2: {} 2026-02-23T13:13:26.019 INFO:teuthology.orchestra.run.trial068.stderr:osd.2: osd_enable_op_tracker = '' 2026-02-23T13:13:26.031 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:13:26.031 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:13:26.046 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:13:26 trial068 ceph-mon[38805]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 8777/6 objects degraded (146283.333%), 4 pgs degraded) 2026-02-23T13:13:26.047 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:13:26 trial068 ceph-mon[38805]: Cluster is now healthy 2026-02-23T13:13:26.047 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:13:26 trial068 ceph-mon[38805]: osd.1 marked itself down and dead 2026-02-23T13:13:26.047 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:13:26 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2618744403' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:13:26.048 DEBUG:teuthology.orchestra.run.trial068:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -t 128 -b 8192 --object-size 65536 -p unique_pool_3 bench 90 write' 2026-02-23T13:13:26.051 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2026-02-23T13:13:26.051 INFO:teuthology.orchestra.run:waiting for 3000 2026-02-23T13:13:26.097 INFO:tasks.radosbench.radosbench.0.trial068.stdout:hints = 1 2026-02-23T13:13:26.098 INFO:tasks.radosbench.radosbench.0.trial068.stdout:Maintaining 128 concurrent writes of 8192 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2026-02-23T13:13:26.098 INFO:tasks.radosbench.radosbench.0.trial068.stdout:Object prefix: benchmark_data_trial068_104694 2026-02-23T13:13:26.098 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T13:13:26.098 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 0 0 0 0 0 0 - 0 2026-02-23T13:13:26.315 INFO:journalctl@ceph.osd.1.trial068.stdout:Feb 23 13:13:26 trial068 podman[104563]: 2026-02-23 13:13:26.17495752 +0000 UTC m=+1.004660190 container died e32560e4f15a411ce6f25e4db870081696bda35865d4c9993d0fa174fdef5a0a (image=quay.ceph.io/ceph-ci/ceph@sha256:49621faaa845c39b2aae72e4fa24123ee9b2ac3fb8ae834753907068ee2972ea, name=ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-1, GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfsganesha-5/, io.buildah.version=1.41.3, org.label-schema.schema-version=1.0, FROM_IMAGE=quay.io/centos/centos:stream9, OSD_FLAVOR=default, CEPH_GIT_REPO=https://github.com/ceph/ceph-ci.git, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_SHA1=e371433fcd1390a0b82cfb18effc422c1f5d8c1d, org.opencontainers.image.authors=Ceph Release Team , ceph=True, CEPH_REF=wip-shraddhaag-enable-cephadm-tests, org.label-schema.build-date=20260216, org.opencontainers.image.documentation=https://docs.ceph.com/, org.label-schema.license=GPLv2) 2026-02-23T13:13:26.315 INFO:journalctl@ceph.osd.1.trial068.stdout:Feb 23 13:13:26 trial068 podman[104563]: 2026-02-23 13:13:26.183631636 +0000 UTC m=+1.013334296 container remove e32560e4f15a411ce6f25e4db870081696bda35865d4c9993d0fa174fdef5a0a (image=quay.ceph.io/ceph-ci/ceph@sha256:49621faaa845c39b2aae72e4fa24123ee9b2ac3fb8ae834753907068ee2972ea, name=ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-1, org.label-schema.vendor=CentOS, CEPH_GIT_REPO=https://github.com/ceph/ceph-ci.git, org.label-schema.name=CentOS Stream 9 Base Image, org.opencontainers.image.documentation=https://docs.ceph.com/, GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfsganesha-5/, CEPH_REF=wip-shraddhaag-enable-cephadm-tests, io.buildah.version=1.41.3, org.label-schema.schema-version=1.0, ceph=True, OSD_FLAVOR=default, CEPH_SHA1=e371433fcd1390a0b82cfb18effc422c1f5d8c1d, org.label-schema.license=GPLv2, FROM_IMAGE=quay.io/centos/centos:stream9, org.label-schema.build-date=20260216, org.opencontainers.image.authors=Ceph Release Team ) 2026-02-23T13:13:26.315 INFO:journalctl@ceph.osd.1.trial068.stdout:Feb 23 13:13:26 trial068 bash[104563]: ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-1 2026-02-23T13:13:26.509 DEBUG:teuthology.orchestra.run.trial068:> sudo pkill -f 'journalctl -f -n 0 -u ceph-5f823084-10b8-11f1-be80-d404e6e7d460@osd.1.service' 2026-02-23T13:13:26.570 INFO:journalctl@ceph.osd.1.trial068.stdout:Feb 23 13:13:26 trial068 podman[104849]: 2026-02-23 13:13:26.314272838 +0000 UTC m=+0.016277601 container create 09c5a34505179cfb5b52383d04e2e2edf4d2c426f86c4f00d5b0a2b61a8ef06c (image=quay.ceph.io/ceph-ci/ceph@sha256:49621faaa845c39b2aae72e4fa24123ee9b2ac3fb8ae834753907068ee2972ea, name=ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-1-deactivate, CEPH_GIT_REPO=https://github.com/ceph/ceph-ci.git, org.label-schema.name=CentOS Stream 9 Base Image, FROM_IMAGE=quay.io/centos/centos:stream9, io.buildah.version=1.41.3, CEPH_REF=wip-shraddhaag-enable-cephadm-tests, org.opencontainers.image.documentation=https://docs.ceph.com/, org.label-schema.build-date=20260216, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_SHA1=e371433fcd1390a0b82cfb18effc422c1f5d8c1d, ceph=True, org.label-schema.vendor=CentOS, org.opencontainers.image.authors=Ceph Release Team , OSD_FLAVOR=default, GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfsganesha-5/) 2026-02-23T13:13:26.571 INFO:journalctl@ceph.osd.1.trial068.stdout:Feb 23 13:13:26 trial068 podman[104849]: 2026-02-23 13:13:26.347116571 +0000 UTC m=+0.049121284 container init 09c5a34505179cfb5b52383d04e2e2edf4d2c426f86c4f00d5b0a2b61a8ef06c (image=quay.ceph.io/ceph-ci/ceph@sha256:49621faaa845c39b2aae72e4fa24123ee9b2ac3fb8ae834753907068ee2972ea, name=ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-1-deactivate, io.buildah.version=1.41.3, org.label-schema.build-date=20260216, CEPH_GIT_REPO=https://github.com/ceph/ceph-ci.git, ceph=True, FROM_IMAGE=quay.io/centos/centos:stream9, org.label-schema.vendor=CentOS, CEPH_REF=wip-shraddhaag-enable-cephadm-tests, org.opencontainers.image.documentation=https://docs.ceph.com/, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_SHA1=e371433fcd1390a0b82cfb18effc422c1f5d8c1d, org.opencontainers.image.authors=Ceph Release Team , OSD_FLAVOR=default, GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfsganesha-5/, org.label-schema.license=GPLv2) 2026-02-23T13:13:26.571 INFO:journalctl@ceph.osd.1.trial068.stdout:Feb 23 13:13:26 trial068 podman[104849]: 2026-02-23 13:13:26.349631506 +0000 UTC m=+0.051636209 container start 09c5a34505179cfb5b52383d04e2e2edf4d2c426f86c4f00d5b0a2b61a8ef06c (image=quay.ceph.io/ceph-ci/ceph@sha256:49621faaa845c39b2aae72e4fa24123ee9b2ac3fb8ae834753907068ee2972ea, name=ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-1-deactivate, org.opencontainers.image.authors=Ceph Release Team , OSD_FLAVOR=default, GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfsganesha-5/, CEPH_REF=wip-shraddhaag-enable-cephadm-tests, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_SHA1=e371433fcd1390a0b82cfb18effc422c1f5d8c1d, org.label-schema.vendor=CentOS, org.label-schema.build-date=20260216, org.label-schema.license=GPLv2, org.opencontainers.image.documentation=https://docs.ceph.com/, CEPH_GIT_REPO=https://github.com/ceph/ceph-ci.git, FROM_IMAGE=quay.io/centos/centos:stream9, org.label-schema.schema-version=1.0, io.buildah.version=1.41.3) 2026-02-23T13:13:26.571 INFO:journalctl@ceph.osd.1.trial068.stdout:Feb 23 13:13:26 trial068 podman[104849]: 2026-02-23 13:13:26.350351058 +0000 UTC m=+0.052355781 container attach 09c5a34505179cfb5b52383d04e2e2edf4d2c426f86c4f00d5b0a2b61a8ef06c (image=quay.ceph.io/ceph-ci/ceph@sha256:49621faaa845c39b2aae72e4fa24123ee9b2ac3fb8ae834753907068ee2972ea, name=ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-1-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_SHA1=e371433fcd1390a0b82cfb18effc422c1f5d8c1d, FROM_IMAGE=quay.io/centos/centos:stream9, OSD_FLAVOR=default, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.41.3, org.label-schema.vendor=CentOS, org.opencontainers.image.authors=Ceph Release Team , GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfsganesha-5/, org.label-schema.build-date=20260216, org.opencontainers.image.documentation=https://docs.ceph.com/, CEPH_GIT_REPO=https://github.com/ceph/ceph-ci.git, CEPH_REF=wip-shraddhaag-enable-cephadm-tests, org.label-schema.schema-version=1.0) 2026-02-23T13:13:26.571 INFO:journalctl@ceph.osd.1.trial068.stdout:Feb 23 13:13:26 trial068 podman[104849]: 2026-02-23 13:13:26.308308914 +0000 UTC m=+0.010313627 image pull 63df4b83db6f95a8ba965919b3671cb27173117c23b5c950aaadc8913f56213c quay.ceph.io/ceph-ci/ceph@sha256:49621faaa845c39b2aae72e4fa24123ee9b2ac3fb8ae834753907068ee2972ea 2026-02-23T13:13:26.571 INFO:journalctl@ceph.osd.1.trial068.stdout:Feb 23 13:13:26 trial068 podman[104849]: 2026-02-23 13:13:26.477501542 +0000 UTC m=+0.179506265 container died 09c5a34505179cfb5b52383d04e2e2edf4d2c426f86c4f00d5b0a2b61a8ef06c (image=quay.ceph.io/ceph-ci/ceph@sha256:49621faaa845c39b2aae72e4fa24123ee9b2ac3fb8ae834753907068ee2972ea, name=ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-1-deactivate, ceph=True, io.buildah.version=1.41.3, org.label-schema.build-date=20260216, org.label-schema.schema-version=1.0, CEPH_GIT_REPO=https://github.com/ceph/ceph-ci.git, org.label-schema.name=CentOS Stream 9 Base Image, org.opencontainers.image.authors=Ceph Release Team , OSD_FLAVOR=default, GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfsganesha-5/, org.opencontainers.image.documentation=https://docs.ceph.com/, org.label-schema.license=GPLv2, CEPH_SHA1=e371433fcd1390a0b82cfb18effc422c1f5d8c1d, FROM_IMAGE=quay.io/centos/centos:stream9, org.label-schema.vendor=CentOS, CEPH_REF=wip-shraddhaag-enable-cephadm-tests) 2026-02-23T13:13:26.571 INFO:journalctl@ceph.osd.1.trial068.stdout:Feb 23 13:13:26 trial068 podman[104849]: 2026-02-23 13:13:26.489377082 +0000 UTC m=+0.191381785 container remove 09c5a34505179cfb5b52383d04e2e2edf4d2c426f86c4f00d5b0a2b61a8ef06c (image=quay.ceph.io/ceph-ci/ceph@sha256:49621faaa845c39b2aae72e4fa24123ee9b2ac3fb8ae834753907068ee2972ea, name=ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-1-deactivate, org.label-schema.build-date=20260216, org.opencontainers.image.authors=Ceph Release Team , org.label-schema.vendor=CentOS, CEPH_GIT_REPO=https://github.com/ceph/ceph-ci.git, CEPH_REF=wip-shraddhaag-enable-cephadm-tests, FROM_IMAGE=quay.io/centos/centos:stream9, ceph=True, CEPH_SHA1=e371433fcd1390a0b82cfb18effc422c1f5d8c1d, org.label-schema.license=GPLv2, GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfsganesha-5/, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.41.3, org.opencontainers.image.documentation=https://docs.ceph.com/, OSD_FLAVOR=default) 2026-02-23T13:13:26.571 INFO:journalctl@ceph.osd.1.trial068.stdout:Feb 23 13:13:26 trial068 systemd[1]: ceph-5f823084-10b8-11f1-be80-d404e6e7d460@osd.1.service: Deactivated successfully. 2026-02-23T13:13:26.571 INFO:journalctl@ceph.osd.1.trial068.stdout:Feb 23 13:13:26 trial068 systemd[1]: Stopped Ceph osd.1 for 5f823084-10b8-11f1-be80-d404e6e7d460. 2026-02-23T13:13:26.571 INFO:journalctl@ceph.osd.1.trial068.stdout:Feb 23 13:13:26 trial068 systemd[1]: ceph-5f823084-10b8-11f1-be80-d404e6e7d460@osd.1.service: Consumed 2min 27.692s CPU time. 2026-02-23T13:13:26.586 DEBUG:teuthology.orchestra.run:got remote process result: None 2026-02-23T13:13:26.586 INFO:tasks.cephadm.osd.1:Stopped osd.1 2026-02-23T13:13:27.098 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1 128 1031 903 7.05407 7.05469 0.0030326 0.140374 2026-02-23T13:13:27.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:13:27 trial068 ceph-mon[38805]: Health check failed: 1 osds down (OSD_DOWN) 2026-02-23T13:13:27.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:13:27 trial068 ceph-mon[38805]: osdmap e172: 4 total, 3 up, 4 in 2026-02-23T13:13:27.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:13:27 trial068 ceph-mon[38805]: pgmap v328: 17 pgs: 7 unknown, 10 active+clean; 577 KiB data, 3.4 GiB used, 2.7 TiB / 2.7 TiB avail 2026-02-23T13:13:28.069 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:13:28.098 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2 128 10152 10024 39.154 71.2578 0.00212648 0.0254132 2026-02-23T13:13:28.203 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:13:28.269 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:13:28 trial068 ceph-mon[38805]: osdmap e173: 4 total, 3 up, 4 in 2026-02-23T13:13:28.325 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:13:28.323+0000 7f1830b22640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:13:28.325 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:13:28.323+0000 7f1830b22640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:13:28.341 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:13:28.342 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:13:28.342 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.1 2026-02-23T13:13:28.356 INFO:teuthology.orchestra.run.trial068.stdout:osd.2: {} 2026-02-23T13:13:28.356 INFO:teuthology.orchestra.run.trial068.stderr:osd.2: osd_enable_op_tracker = '' 2026-02-23T13:13:28.372 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:13:28.372 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:13:28.406 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:13:28.407 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:13:29.098 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3 128 19749 19621 51.0932 74.9766 0.00369277 0.0195192 2026-02-23T13:13:29.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:13:29 trial068 ceph-mon[38805]: pgmap v330: 17 pgs: 3 peering, 2 stale+active+clean, 2 unknown, 10 active+clean; 961 KiB data, 3.4 GiB used, 2.7 TiB / 2.7 TiB avail; 90 KiB/s wr, 11 op/s 2026-02-23T13:13:29.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:13:29 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/4070743057' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:13:29.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:13:29 trial068 ceph-mon[38805]: osdmap e174: 4 total, 3 up, 4 in 2026-02-23T13:13:30.098 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4 128 30045 29917 58.4279 80.4375 0.0150644 0.0170781 2026-02-23T13:13:30.408 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:13:30.549 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:13:30.665 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:13:30 trial068 ceph-mon[38805]: Health check failed: Degraded data redundancy: 262/1113 objects degraded (23.540%), 3 pgs degraded (PG_DEGRADED) 2026-02-23T13:13:30.673 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:13:30.671+0000 7f5c21d80640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:13:30.673 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:13:30.671+0000 7f5c21d80640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:13:30.687 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:13:30.688 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:13:30.688 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.1 2026-02-23T13:13:30.701 INFO:teuthology.orchestra.run.trial068.stdout:osd.2: {} 2026-02-23T13:13:30.702 INFO:teuthology.orchestra.run.trial068.stderr:osd.2: osd_enable_op_tracker = '' 2026-02-23T13:13:30.717 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:13:30.717 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:13:30.753 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:13:30.754 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:13:31.098 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5 128 39915 39787 62.1631 77.1094 0.0198559 0.0160403 2026-02-23T13:13:31.586 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [1] live_osds: [3, 0, 2] 2026-02-23T13:13:31.587 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2026-02-23T13:13:31.587 INFO:tasks.thrashosds.thrasher:Reviving osd 1 2026-02-23T13:13:31.587 INFO:tasks.cephadm.osd.1:Restarting osd.1 (starting--it wasn't running)... 2026-02-23T13:13:31.587 DEBUG:teuthology.orchestra.run.trial068:osd.1> sudo journalctl -f -n 0 -u ceph-5f823084-10b8-11f1-be80-d404e6e7d460@osd.1.service 2026-02-23T13:13:31.590 DEBUG:teuthology.orchestra.run.trial068:> sudo systemctl start ceph-5f823084-10b8-11f1-be80-d404e6e7d460@osd.1 2026-02-23T13:13:31.654 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:13:31 trial068 ceph-mon[38805]: pgmap v332: 17 pgs: 3 peering, 3 active+undersized+degraded, 11 active+clean; 24 MiB data, 1.8 GiB used, 2.7 TiB / 2.7 TiB avail; 4.4 MiB/s wr, 559 op/s; 262/1113 objects degraded (23.540%) 2026-02-23T13:13:31.655 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:13:31 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2404779424' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:13:31.913 INFO:journalctl@ceph.osd.1.trial068.stdout:Feb 23 13:13:31 trial068 systemd[1]: Starting Ceph osd.1 for 5f823084-10b8-11f1-be80-d404e6e7d460... 2026-02-23T13:13:31.913 INFO:journalctl@ceph.osd.1.trial068.stdout:Feb 23 13:13:31 trial068 podman[105335]: 2026-02-23 13:13:31.77707196 +0000 UTC m=+0.018173803 container create 61ec2fbe65be8e6a581650212eb5a015af3c4dca30c325b1b2a6187bdf22031d (image=quay.ceph.io/ceph-ci/ceph@sha256:49621faaa845c39b2aae72e4fa24123ee9b2ac3fb8ae834753907068ee2972ea, name=ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-1-activate, org.label-schema.name=CentOS Stream 9 Base Image, OSD_FLAVOR=default, org.label-schema.build-date=20260216, GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfsganesha-5/, org.opencontainers.image.authors=Ceph Release Team , CEPH_GIT_REPO=https://github.com/ceph/ceph-ci.git, CEPH_SHA1=e371433fcd1390a0b82cfb18effc422c1f5d8c1d, CEPH_REF=wip-shraddhaag-enable-cephadm-tests, org.label-schema.schema-version=1.0, io.buildah.version=1.41.3, org.label-schema.vendor=CentOS, FROM_IMAGE=quay.io/centos/centos:stream9, org.opencontainers.image.documentation=https://docs.ceph.com/, ceph=True, org.label-schema.license=GPLv2) 2026-02-23T13:13:31.914 INFO:journalctl@ceph.osd.1.trial068.stdout:Feb 23 13:13:31 trial068 podman[105335]: 2026-02-23 13:13:31.811265659 +0000 UTC m=+0.052367442 container init 61ec2fbe65be8e6a581650212eb5a015af3c4dca30c325b1b2a6187bdf22031d (image=quay.ceph.io/ceph-ci/ceph@sha256:49621faaa845c39b2aae72e4fa24123ee9b2ac3fb8ae834753907068ee2972ea, name=ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-1-activate, org.opencontainers.image.authors=Ceph Release Team , io.buildah.version=1.41.3, CEPH_GIT_REPO=https://github.com/ceph/ceph-ci.git, org.label-schema.schema-version=1.0, GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfsganesha-5/, CEPH_REF=wip-shraddhaag-enable-cephadm-tests, org.label-schema.build-date=20260216, CEPH_SHA1=e371433fcd1390a0b82cfb18effc422c1f5d8c1d, FROM_IMAGE=quay.io/centos/centos:stream9, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, OSD_FLAVOR=default, org.label-schema.vendor=CentOS, org.opencontainers.image.documentation=https://docs.ceph.com/) 2026-02-23T13:13:31.914 INFO:journalctl@ceph.osd.1.trial068.stdout:Feb 23 13:13:31 trial068 podman[105335]: 2026-02-23 13:13:31.814741799 +0000 UTC m=+0.055843592 container start 61ec2fbe65be8e6a581650212eb5a015af3c4dca30c325b1b2a6187bdf22031d (image=quay.ceph.io/ceph-ci/ceph@sha256:49621faaa845c39b2aae72e4fa24123ee9b2ac3fb8ae834753907068ee2972ea, name=ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-1-activate, org.label-schema.build-date=20260216, CEPH_GIT_REPO=https://github.com/ceph/ceph-ci.git, org.label-schema.vendor=CentOS, io.buildah.version=1.41.3, org.opencontainers.image.documentation=https://docs.ceph.com/, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_SHA1=e371433fcd1390a0b82cfb18effc422c1f5d8c1d, FROM_IMAGE=quay.io/centos/centos:stream9, org.label-schema.name=CentOS Stream 9 Base Image, OSD_FLAVOR=default, GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfsganesha-5/, org.opencontainers.image.authors=Ceph Release Team , ceph=True, CEPH_REF=wip-shraddhaag-enable-cephadm-tests) 2026-02-23T13:13:31.914 INFO:journalctl@ceph.osd.1.trial068.stdout:Feb 23 13:13:31 trial068 podman[105335]: 2026-02-23 13:13:31.81524788 +0000 UTC m=+0.056349683 container attach 61ec2fbe65be8e6a581650212eb5a015af3c4dca30c325b1b2a6187bdf22031d (image=quay.ceph.io/ceph-ci/ceph@sha256:49621faaa845c39b2aae72e4fa24123ee9b2ac3fb8ae834753907068ee2972ea, name=ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-1-activate, org.label-schema.build-date=20260216, org.label-schema.license=GPLv2, OSD_FLAVOR=default, CEPH_GIT_REPO=https://github.com/ceph/ceph-ci.git, org.opencontainers.image.documentation=https://docs.ceph.com/, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfsganesha-5/, io.buildah.version=1.41.3, org.opencontainers.image.authors=Ceph Release Team , CEPH_SHA1=e371433fcd1390a0b82cfb18effc422c1f5d8c1d, org.label-schema.vendor=CentOS, FROM_IMAGE=quay.io/centos/centos:stream9, ceph=True, CEPH_REF=wip-shraddhaag-enable-cephadm-tests) 2026-02-23T13:13:31.914 INFO:journalctl@ceph.osd.1.trial068.stdout:Feb 23 13:13:31 trial068 podman[105335]: 2026-02-23 13:13:31.769184683 +0000 UTC m=+0.010286486 image pull 63df4b83db6f95a8ba965919b3671cb27173117c23b5c950aaadc8913f56213c quay.ceph.io/ceph-ci/ceph@sha256:49621faaa845c39b2aae72e4fa24123ee9b2ac3fb8ae834753907068ee2972ea 2026-02-23T13:13:31.914 INFO:journalctl@ceph.osd.1.trial068.stdout:Feb 23 13:13:31 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-1-activate[105355]: Running command: /usr/bin/ceph-authtool --gen-print-key 2026-02-23T13:13:31.914 INFO:journalctl@ceph.osd.1.trial068.stdout:Feb 23 13:13:31 trial068 bash[105335]: Running command: /usr/bin/ceph-authtool --gen-print-key 2026-02-23T13:13:31.914 INFO:journalctl@ceph.osd.1.trial068.stdout:Feb 23 13:13:31 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-1-activate[105355]: Running command: /usr/bin/ceph-authtool --gen-print-key 2026-02-23T13:13:31.914 INFO:journalctl@ceph.osd.1.trial068.stdout:Feb 23 13:13:31 trial068 bash[105335]: Running command: /usr/bin/ceph-authtool --gen-print-key 2026-02-23T13:13:32.098 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6 127 49003 48876 63.6364 71.0078 0.00492038 0.0156926 2026-02-23T13:13:32.658 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:13:32 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2026-02-23T13:13:32.659 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:13:32 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-23T13:13:32.659 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:13:32 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2026-02-23T13:13:32.659 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:13:32 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T13:13:32.659 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:13:32 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2026-02-23T13:13:32.659 INFO:journalctl@ceph.osd.1.trial068.stdout:Feb 23 13:13:32 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-1-activate[105355]: --> Failed to activate via raw: did not find any matching OSD to activate 2026-02-23T13:13:32.659 INFO:journalctl@ceph.osd.1.trial068.stdout:Feb 23 13:13:32 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-1-activate[105355]: Running command: /usr/bin/ceph-authtool --gen-print-key 2026-02-23T13:13:32.659 INFO:journalctl@ceph.osd.1.trial068.stdout:Feb 23 13:13:32 trial068 bash[105335]: --> Failed to activate via raw: did not find any matching OSD to activate 2026-02-23T13:13:32.660 INFO:journalctl@ceph.osd.1.trial068.stdout:Feb 23 13:13:32 trial068 bash[105335]: Running command: /usr/bin/ceph-authtool --gen-print-key 2026-02-23T13:13:32.660 INFO:journalctl@ceph.osd.1.trial068.stdout:Feb 23 13:13:32 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-1-activate[105355]: Running command: /usr/bin/ceph-authtool --gen-print-key 2026-02-23T13:13:32.660 INFO:journalctl@ceph.osd.1.trial068.stdout:Feb 23 13:13:32 trial068 bash[105335]: Running command: /usr/bin/ceph-authtool --gen-print-key 2026-02-23T13:13:32.660 INFO:journalctl@ceph.osd.1.trial068.stdout:Feb 23 13:13:32 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-1-activate[105355]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2026-02-23T13:13:32.756 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:13:32.880 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph daemon osd.1 dump_ops_in_flight 2026-02-23T13:13:32.904 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:13:32.932 INFO:journalctl@ceph.osd.1.trial068.stdout:Feb 23 13:13:32 trial068 bash[105335]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2026-02-23T13:13:32.932 INFO:journalctl@ceph.osd.1.trial068.stdout:Feb 23 13:13:32 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-1-activate[105355]: Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/vg_nvme/lv_3 --path /var/lib/ceph/osd/ceph-1 --no-mon-config 2026-02-23T13:13:32.932 INFO:journalctl@ceph.osd.1.trial068.stdout:Feb 23 13:13:32 trial068 bash[105335]: Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/vg_nvme/lv_3 --path /var/lib/ceph/osd/ceph-1 --no-mon-config 2026-02-23T13:13:32.932 INFO:journalctl@ceph.osd.1.trial068.stdout:Feb 23 13:13:32 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-1-activate[105355]: Running command: /usr/bin/ln -snf /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-1/block 2026-02-23T13:13:32.932 INFO:journalctl@ceph.osd.1.trial068.stdout:Feb 23 13:13:32 trial068 bash[105335]: Running command: /usr/bin/ln -snf /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-1/block 2026-02-23T13:13:32.932 INFO:journalctl@ceph.osd.1.trial068.stdout:Feb 23 13:13:32 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-1-activate[105355]: Running command: /usr/bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-1/block 2026-02-23T13:13:32.932 INFO:journalctl@ceph.osd.1.trial068.stdout:Feb 23 13:13:32 trial068 bash[105335]: Running command: /usr/bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-1/block 2026-02-23T13:13:32.932 INFO:journalctl@ceph.osd.1.trial068.stdout:Feb 23 13:13:32 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-1-activate[105355]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2026-02-23T13:13:32.932 INFO:journalctl@ceph.osd.1.trial068.stdout:Feb 23 13:13:32 trial068 bash[105335]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2026-02-23T13:13:32.932 INFO:journalctl@ceph.osd.1.trial068.stdout:Feb 23 13:13:32 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-1-activate[105355]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2026-02-23T13:13:32.932 INFO:journalctl@ceph.osd.1.trial068.stdout:Feb 23 13:13:32 trial068 bash[105335]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2026-02-23T13:13:32.932 INFO:journalctl@ceph.osd.1.trial068.stdout:Feb 23 13:13:32 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-1-activate[105355]: --> ceph-volume lvm activate successful for osd ID: 1 2026-02-23T13:13:32.933 INFO:journalctl@ceph.osd.1.trial068.stdout:Feb 23 13:13:32 trial068 bash[105335]: --> ceph-volume lvm activate successful for osd ID: 1 2026-02-23T13:13:32.933 INFO:journalctl@ceph.osd.1.trial068.stdout:Feb 23 13:13:32 trial068 podman[105335]: 2026-02-23 13:13:32.733119793 +0000 UTC m=+0.974221586 container died 61ec2fbe65be8e6a581650212eb5a015af3c4dca30c325b1b2a6187bdf22031d (image=quay.ceph.io/ceph-ci/ceph@sha256:49621faaa845c39b2aae72e4fa24123ee9b2ac3fb8ae834753907068ee2972ea, name=ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-1-activate, org.label-schema.build-date=20260216, io.buildah.version=1.41.3, org.opencontainers.image.documentation=https://docs.ceph.com/, CEPH_REF=wip-shraddhaag-enable-cephadm-tests, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfsganesha-5/, org.opencontainers.image.authors=Ceph Release Team , CEPH_GIT_REPO=https://github.com/ceph/ceph-ci.git, CEPH_SHA1=e371433fcd1390a0b82cfb18effc422c1f5d8c1d, ceph=True, OSD_FLAVOR=default, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, FROM_IMAGE=quay.io/centos/centos:stream9) 2026-02-23T13:13:32.933 INFO:journalctl@ceph.osd.1.trial068.stdout:Feb 23 13:13:32 trial068 podman[105335]: 2026-02-23 13:13:32.744896287 +0000 UTC m=+0.985998080 container remove 61ec2fbe65be8e6a581650212eb5a015af3c4dca30c325b1b2a6187bdf22031d (image=quay.ceph.io/ceph-ci/ceph@sha256:49621faaa845c39b2aae72e4fa24123ee9b2ac3fb8ae834753907068ee2972ea, name=ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-1-activate, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, OSD_FLAVOR=default, org.opencontainers.image.authors=Ceph Release Team , CEPH_SHA1=e371433fcd1390a0b82cfb18effc422c1f5d8c1d, GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfsganesha-5/, org.label-schema.build-date=20260216, CEPH_REF=wip-shraddhaag-enable-cephadm-tests, org.opencontainers.image.documentation=https://docs.ceph.com/, CEPH_GIT_REPO=https://github.com/ceph/ceph-ci.git, FROM_IMAGE=quay.io/centos/centos:stream9, io.buildah.version=1.41.3, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2) 2026-02-23T13:13:32.933 INFO:journalctl@ceph.osd.1.trial068.stdout:Feb 23 13:13:32 trial068 podman[105611]: 2026-02-23 13:13:32.834992228 +0000 UTC m=+0.018692769 container create 77fde433ffa7b44bd863bd612e273a7f96fac9cbfd6debb719c4b7a5b2644f3f (image=quay.ceph.io/ceph-ci/ceph@sha256:49621faaa845c39b2aae72e4fa24123ee9b2ac3fb8ae834753907068ee2972ea, name=ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-1, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20260216, org.opencontainers.image.authors=Ceph Release Team , org.label-schema.vendor=CentOS, OSD_FLAVOR=default, org.label-schema.license=GPLv2, FROM_IMAGE=quay.io/centos/centos:stream9, CEPH_REF=wip-shraddhaag-enable-cephadm-tests, CEPH_SHA1=e371433fcd1390a0b82cfb18effc422c1f5d8c1d, GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfsganesha-5/, org.opencontainers.image.documentation=https://docs.ceph.com/, io.buildah.version=1.41.3, CEPH_GIT_REPO=https://github.com/ceph/ceph-ci.git) 2026-02-23T13:13:32.933 INFO:journalctl@ceph.osd.1.trial068.stdout:Feb 23 13:13:32 trial068 podman[105611]: 2026-02-23 13:13:32.869383658 +0000 UTC m=+0.053084199 container init 77fde433ffa7b44bd863bd612e273a7f96fac9cbfd6debb719c4b7a5b2644f3f (image=quay.ceph.io/ceph-ci/ceph@sha256:49621faaa845c39b2aae72e4fa24123ee9b2ac3fb8ae834753907068ee2972ea, name=ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-1, CEPH_GIT_REPO=https://github.com/ceph/ceph-ci.git, org.label-schema.build-date=20260216, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, org.opencontainers.image.documentation=https://docs.ceph.com/, CEPH_SHA1=e371433fcd1390a0b82cfb18effc422c1f5d8c1d, org.label-schema.license=GPLv2, ceph=True, GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfsganesha-5/, io.buildah.version=1.41.3, FROM_IMAGE=quay.io/centos/centos:stream9, org.opencontainers.image.authors=Ceph Release Team , OSD_FLAVOR=default, CEPH_REF=wip-shraddhaag-enable-cephadm-tests, org.label-schema.vendor=CentOS) 2026-02-23T13:13:32.933 INFO:journalctl@ceph.osd.1.trial068.stdout:Feb 23 13:13:32 trial068 podman[105611]: 2026-02-23 13:13:32.872524579 +0000 UTC m=+0.056225120 container start 77fde433ffa7b44bd863bd612e273a7f96fac9cbfd6debb719c4b7a5b2644f3f (image=quay.ceph.io/ceph-ci/ceph@sha256:49621faaa845c39b2aae72e4fa24123ee9b2ac3fb8ae834753907068ee2972ea, name=ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-1, CEPH_REF=wip-shraddhaag-enable-cephadm-tests, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.41.3, FROM_IMAGE=quay.io/centos/centos:stream9, org.label-schema.schema-version=1.0, org.opencontainers.image.documentation=https://docs.ceph.com/, org.label-schema.build-date=20260216, OSD_FLAVOR=default, CEPH_SHA1=e371433fcd1390a0b82cfb18effc422c1f5d8c1d, GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfsganesha-5/, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, CEPH_GIT_REPO=https://github.com/ceph/ceph-ci.git, org.opencontainers.image.authors=Ceph Release Team ) 2026-02-23T13:13:32.933 INFO:journalctl@ceph.osd.1.trial068.stdout:Feb 23 13:13:32 trial068 bash[105611]: 77fde433ffa7b44bd863bd612e273a7f96fac9cbfd6debb719c4b7a5b2644f3f 2026-02-23T13:13:32.933 INFO:journalctl@ceph.osd.1.trial068.stdout:Feb 23 13:13:32 trial068 podman[105611]: 2026-02-23 13:13:32.828217591 +0000 UTC m=+0.011918142 image pull 63df4b83db6f95a8ba965919b3671cb27173117c23b5c950aaadc8913f56213c quay.ceph.io/ceph-ci/ceph@sha256:49621faaa845c39b2aae72e4fa24123ee9b2ac3fb8ae834753907068ee2972ea 2026-02-23T13:13:32.933 INFO:journalctl@ceph.osd.1.trial068.stdout:Feb 23 13:13:32 trial068 systemd[1]: Started Ceph osd.1 for 5f823084-10b8-11f1-be80-d404e6e7d460. 2026-02-23T13:13:32.933 INFO:journalctl@ceph.osd.1.trial068.stdout:Feb 23 13:13:32 trial068 ceph-osd[105635]: -- 10.20.193.68:0/2110907179 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x24e7c00 msgr2=0x3324000 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).process 2026-02-23T13:13:32.933 INFO:journalctl@ceph.osd.1.trial068.stdout:Feb 23 13:13:32 trial068 ceph-osd[105635]: -- 10.20.193.68:0/2110907179 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x24e7c00 msgr2=0x3324000 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read continue len=32 2026-02-23T13:13:32.933 INFO:journalctl@ceph.osd.1.trial068.stdout:Feb 23 13:13:32 trial068 ceph-osd[105635]: --2- 10.20.193.68:0/2110907179 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x24e7c00 0x3324000 crc :-1 s=READY pgs=687 gs=1 cs=0 l=1 c_cookie=1816659872685296942 s_cookie=0 reconnecting=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main r=0 2026-02-23T13:13:32.933 INFO:journalctl@ceph.osd.1.trial068.stdout:Feb 23 13:13:32 trial068 ceph-osd[105635]: --2- 10.20.193.68:0/2110907179 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x24e7c00 0x3324000 crc :-1 s=THROTTLE_MESSAGE pgs=687 gs=1 cs=0 l=1 c_cookie=1816659872685296942 s_cookie=0 reconnecting=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).throttle_message 2026-02-23T13:13:32.933 INFO:journalctl@ceph.osd.1.trial068.stdout:Feb 23 13:13:32 trial068 ceph-osd[105635]: --2- 10.20.193.68:0/2110907179 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x24e7c00 0x3324000 crc :-1 s=THROTTLE_BYTES pgs=687 gs=1 cs=0 l=1 c_cookie=1816659872685296942 s_cookie=0 reconnecting=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).throttle_bytes 2026-02-23T13:13:32.933 INFO:journalctl@ceph.osd.1.trial068.stdout:Feb 23 13:13:32 trial068 ceph-osd[105635]: --2- 10.20.193.68:0/2110907179 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x24e7c00 0x3324000 crc :-1 s=THROTTLE_DISPATCH_QUEUE pgs=687 gs=1 cs=0 l=1 c_cookie=1816659872685296942 s_cookie=0 reconnecting=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).throttle_dispatch_queue 2026-02-23T13:13:32.933 INFO:journalctl@ceph.osd.1.trial068.stdout:Feb 23 13:13:32 trial068 ceph-osd[105635]: --2- 10.20.193.68:0/2110907179 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x24e7c00 0x3324000 crc :-1 s=THROTTLE_DONE pgs=687 gs=1 cs=0 l=1 c_cookie=1816659872685296942 s_cookie=0 reconnecting=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).read_frame_segment seg_idx=0 2026-02-23T13:13:32.933 INFO:journalctl@ceph.osd.1.trial068.stdout:Feb 23 13:13:32 trial068 ceph-osd[105635]: -- 10.20.193.68:0/2110907179 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x24e7c00 msgr2=0x3324000 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read start len=45 2026-02-23T13:13:32.933 INFO:journalctl@ceph.osd.1.trial068.stdout:Feb 23 13:13:32 trial068 ceph-osd[105635]: --2- 10.20.193.68:0/2110907179 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x24e7c00 0x3324000 crc :-1 s=THROTTLE_DONE pgs=687 gs=1 cs=0 l=1 c_cookie=1816659872685296942 s_cookie=0 reconnecting=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_segment r=0 2026-02-23T13:13:32.933 INFO:journalctl@ceph.osd.1.trial068.stdout:Feb 23 13:13:32 trial068 ceph-osd[105635]: --2- 10.20.193.68:0/2110907179 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x24e7c00 0x3324000 crc :-1 s=THROTTLE_DONE pgs=687 gs=1 cs=0 l=1 c_cookie=1816659872685296942 s_cookie=0 reconnecting=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).read_frame_segment seg_idx=1 2026-02-23T13:13:32.933 INFO:journalctl@ceph.osd.1.trial068.stdout:Feb 23 13:13:32 trial068 ceph-osd[105635]: -- 10.20.193.68:0/2110907179 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x24e7c00 msgr2=0x3324000 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read start len=194 2026-02-23T13:13:32.933 INFO:journalctl@ceph.osd.1.trial068.stdout:Feb 23 13:13:32 trial068 ceph-osd[105635]: --2- 10.20.193.68:0/2110907179 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x24e7c00 0x3324000 crc :-1 s=THROTTLE_DONE pgs=687 gs=1 cs=0 l=1 c_cookie=1816659872685296942 s_cookie=0 reconnecting=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_segment r=0 2026-02-23T13:13:32.934 INFO:journalctl@ceph.osd.1.trial068.stdout:Feb 23 13:13:32 trial068 ceph-osd[105635]: -- 10.20.193.68:0/2110907179 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x24e7c00 msgr2=0x3324000 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read start len=13 2026-02-23T13:13:32.934 INFO:journalctl@ceph.osd.1.trial068.stdout:Feb 23 13:13:32 trial068 ceph-osd[105635]: --2- 10.20.193.68:0/2110907179 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x24e7c00 0x3324000 crc :-1 s=THROTTLE_DONE pgs=687 gs=1 cs=0 l=1 c_cookie=1816659872685296942 s_cookie=0 reconnecting=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_epilogue_main r=0 2026-02-23T13:13:32.934 INFO:journalctl@ceph.osd.1.trial068.stdout:Feb 23 13:13:32 trial068 ceph-osd[105635]: --2- 10.20.193.68:0/2110907179 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x24e7c00 0x3324000 crc :-1 s=THROTTLE_DONE pgs=687 gs=1 cs=0 l=1 c_cookie=1816659872685296942 s_cookie=0 reconnecting=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_dispatch tag=17 2026-02-23T13:13:32.934 INFO:journalctl@ceph.osd.1.trial068.stdout:Feb 23 13:13:32 trial068 ceph-osd[105635]: --2- 10.20.193.68:0/2110907179 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x24e7c00 0x3324000 crc :-1 s=THROTTLE_DONE pgs=687 gs=1 cs=0 l=1 c_cookie=1816659872685296942 s_cookie=0 reconnecting=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_message 2026-02-23T13:13:32.934 INFO:journalctl@ceph.osd.1.trial068.stdout:Feb 23 13:13:32 trial068 ceph-osd[105635]: --2- 10.20.193.68:0/2110907179 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x24e7c00 0x3324000 crc :-1 s=THROTTLE_DONE pgs=687 gs=1 cs=0 l=1 c_cookie=1816659872685296942 s_cookie=0 reconnecting=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_message got 194 + 0 + 0 byte message. envelope type=18 src mon.0 off 0 2026-02-23T13:13:32.934 INFO:journalctl@ceph.osd.1.trial068.stdout:Feb 23 13:13:32 trial068 ceph-osd[105635]: --2- 10.20.193.68:0/2110907179 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x24e7c00 0x3324000 crc :-1 s=READ_MESSAGE_COMPLETE pgs=687 gs=1 cs=0 l=1 c_cookie=1816659872685296942 s_cookie=0 reconnecting=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_message received message m=0x254b860 seq=4 from=mon.0 type=18 auth_reply(proto 2 0 (0) Success) 2026-02-23T13:13:32.934 INFO:journalctl@ceph.osd.1.trial068.stdout:Feb 23 13:13:32 trial068 ceph-osd[105635]: -- 10.20.193.68:0/2110907179 queue 0x254b860 prio 196 2026-02-23T13:13:32.934 INFO:journalctl@ceph.osd.1.trial068.stdout:Feb 23 13:13:32 trial068 ceph-osd[105635]: --2- 10.20.193.68:0/2110907179 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x24e7c00 0x3324000 crc :-1 s=READY pgs=687 gs=1 cs=0 l=1 c_cookie=1816659872685296942 s_cookie=0 reconnecting=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).read_frame 2026-02-23T13:13:32.934 INFO:journalctl@ceph.osd.1.trial068.stdout:Feb 23 13:13:32 trial068 ceph-osd[105635]: -- 10.20.193.68:0/2110907179 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x24e7c00 msgr2=0x3324000 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read start len=32 2026-02-23T13:13:32.934 INFO:journalctl@ceph.osd.1.trial068.stdout:Feb 23 13:13:32 trial068 ceph-osd[105635]: -- 10.20.193.68:0/2110907179 <== mon.0 v2:10.20.193.68:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (crc 0 0 0) 0x254b860 con 0x24e7c00 2026-02-23T13:13:32.934 INFO:journalctl@ceph.osd.1.trial068.stdout:Feb 23 13:13:32 trial068 ceph-osd[105635]: -- 10.20.193.68:0/2110907179 dispatch_throttle_release 194 to dispatch throttler 194/104857600 2026-02-23T13:13:32.934 INFO:journalctl@ceph.osd.1.trial068.stdout:Feb 23 13:13:32 trial068 ceph-osd[105635]: -- 10.20.193.68:0/2110907179 done calling dispatch on 0x254b860 2026-02-23T13:13:32.934 INFO:journalctl@ceph.osd.1.trial068.stdout:Feb 23 13:13:32 trial068 ceph-osd[105635]: WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:13:32.934 INFO:journalctl@ceph.osd.1.trial068.stdout:Feb 23 13:13:32 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-1[105631]: 2026-02-23T13:13:32.888+0000 7fbac04c5780 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:13:32.934 INFO:journalctl@ceph.osd.1.trial068.stdout:Feb 23 13:13:32 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-1[105631]: 2026-02-23T13:13:32.888+0000 7fbac04c5780 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:13:32.934 INFO:journalctl@ceph.osd.1.trial068.stdout:Feb 23 13:13:32 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-1[105631]: 2026-02-23T13:13:32.888+0000 7fbac04c5780 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:13:33.019 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:13:33.037 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:13:33.035+0000 7fd1c5d4a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:13:33.037 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:13:33.035+0000 7fd1c5d4a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:13:33.054 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:13:33.054 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:13:33.055 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.1 2026-02-23T13:13:33.068 INFO:teuthology.orchestra.run.trial068.stdout:osd.2: {} 2026-02-23T13:13:33.068 INFO:teuthology.orchestra.run.trial068.stderr:osd.2: osd_enable_op_tracker = '' 2026-02-23T13:13:33.080 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:13:33.080 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:13:33.098 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7 128 57570 57442 64.1051 66.9219 0.0106448 0.0155636 2026-02-23T13:13:33.118 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:13:33.119 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:13:33.147 INFO:teuthology.orchestra.run.trial068.stderr:no valid command found; 10 closest matches: 2026-02-23T13:13:33.147 INFO:teuthology.orchestra.run.trial068.stderr:0 2026-02-23T13:13:33.148 INFO:teuthology.orchestra.run.trial068.stderr:1 2026-02-23T13:13:33.148 INFO:teuthology.orchestra.run.trial068.stderr:2 2026-02-23T13:13:33.148 INFO:teuthology.orchestra.run.trial068.stderr:abort 2026-02-23T13:13:33.148 INFO:teuthology.orchestra.run.trial068.stderr:assert 2026-02-23T13:13:33.148 INFO:teuthology.orchestra.run.trial068.stderr:bluestore collections 2026-02-23T13:13:33.148 INFO:teuthology.orchestra.run.trial068.stderr:bluestore compression stats [] 2026-02-23T13:13:33.148 INFO:teuthology.orchestra.run.trial068.stderr:bluestore list [] [] 2026-02-23T13:13:33.148 INFO:teuthology.orchestra.run.trial068.stderr:bluestore onode metadata 2026-02-23T13:13:33.148 INFO:teuthology.orchestra.run.trial068.stderr:bluestore show sharding 2026-02-23T13:13:33.149 INFO:teuthology.orchestra.run.trial068.stderr:admin_socket: invalid command 2026-02-23T13:13:33.184 INFO:journalctl@ceph.osd.1.trial068.stdout:Feb 23 13:13:33 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-1[105631]: 2026-02-23T13:13:33.080+0000 7fbac04c5780 -1 Falling back to public interface 2026-02-23T13:13:33.185 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-23T13:13:33.185 INFO:tasks.cephadm.ceph_manager.ceph:waiting on admin_socket for osd-1, ['dump_ops_in_flight'] 2026-02-23T13:13:33.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:13:33 trial068 ceph-mon[38805]: pgmap v333: 17 pgs: 10 active+undersized+degraded, 7 active+clean; 253 MiB data, 2.3 GiB used, 2.7 TiB / 2.7 TiB avail; 40 MiB/s wr, 5.16k op/s; 2394/12135 objects degraded (19.728%) 2026-02-23T13:13:33.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:13:33 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3889186917' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:13:34.098 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8 128 65714 65586 64.0444 63.625 0.0169183 0.0155875 2026-02-23T13:13:34.263 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:13:33 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:13:33.972Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:13:34.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:13:33 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:13:33] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:13:35.098 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9 128 73644 73516 63.8115 61.9531 0.00570423 0.0156467 2026-02-23T13:13:35.120 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:13:35.264 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:13:35.384 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:13:35.382+0000 7f6de7549640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:13:35.385 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:13:35.383+0000 7f6de7549640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:13:35.397 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:13:35.398 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:13:35.398 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:13:35 trial068 ceph-mon[38805]: pgmap v334: 17 pgs: 10 active+undersized+degraded, 7 active+clean; 346 MiB data, 2.6 GiB used, 2.7 TiB / 2.7 TiB avail; 43 MiB/s wr, 5.53k op/s; 3508/16611 objects degraded (21.119%) 2026-02-23T13:13:35.399 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.1 2026-02-23T13:13:35.412 INFO:teuthology.orchestra.run.trial068.stdout:osd.2: {} 2026-02-23T13:13:35.412 INFO:teuthology.orchestra.run.trial068.stderr:osd.2: osd_enable_op_tracker = '' 2026-02-23T13:13:35.424 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:13:35.424 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:13:35.459 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:13:35.460 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:13:36.099 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 10 127 80555 80428 62.83 54 0.00587004 0.0158969 2026-02-23T13:13:36.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:13:36 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3184190907' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:13:36.844 INFO:journalctl@ceph.osd.1.trial068.stdout:Feb 23 13:13:36 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-1[105631]: 2026-02-23T13:13:36.522+0000 7fbac04c5780 -1 osd.1 171 log_to_monitors true 2026-02-23T13:13:37.098 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 11 128 87584 87456 62.1093 54.9062 0.0110772 0.0160798 2026-02-23T13:13:37.461 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:13:37.601 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:13:37.645 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:13:37 trial068 ceph-mon[38805]: pgmap v335: 17 pgs: 10 active+undersized+degraded, 7 active+clean; 493 MiB data, 2.9 GiB used, 2.7 TiB / 2.7 TiB avail; 53 MiB/s wr, 6.79k op/s; 4874/23643 objects degraded (20.615%) 2026-02-23T13:13:37.645 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:13:37 trial068 ceph-mon[38805]: Health check update: Degraded data redundancy: 4874/23643 objects degraded (20.615%), 10 pgs degraded (PG_DEGRADED) 2026-02-23T13:13:37.645 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:13:37 trial068 ceph-mon[38805]: from='osd.1 [v2:10.20.193.68:6810/3818277133,v1:10.20.193.68:6811/3818277133]' entity='osd.1' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]} : dispatch 2026-02-23T13:13:37.645 INFO:journalctl@ceph.osd.1.trial068.stdout:Feb 23 13:13:37 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-1[105631]: 2026-02-23T13:13:37.391+0000 7fbab4306640 -1 osd.1 171 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2026-02-23T13:13:37.711 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:13:37.709+0000 7fa2608c1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:13:37.711 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:13:37.709+0000 7fa2608c1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:13:37.725 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:13:37.726 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:13:37.726 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.1 2026-02-23T13:13:37.738 INFO:teuthology.orchestra.run.trial068.stdout:osd.2: {} 2026-02-23T13:13:37.738 INFO:teuthology.orchestra.run.trial068.stderr:osd.2: osd_enable_op_tracker = '' 2026-02-23T13:13:37.752 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:13:37.752 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:13:37.789 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:13:37.789 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:13:38.098 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 12 128 94414 94286 61.3798 53.3594 0.0101293 0.0162747 2026-02-23T13:13:38.186 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph daemon osd.1 dump_ops_in_flight 2026-02-23T13:13:38.322 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:13:38.448 INFO:teuthology.orchestra.run.trial068.stdout:{ 2026-02-23T13:13:38.449 INFO:teuthology.orchestra.run.trial068.stdout: "ops": [ 2026-02-23T13:13:38.449 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:13:38.449 INFO:teuthology.orchestra.run.trial068.stdout: "description": "osd_op(client.14774.0:96261 5.6 5.412d8096 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e176)", 2026-02-23T13:13:38.449 INFO:teuthology.orchestra.run.trial068.stdout: "initiated_at": "2026-02-23T13:13:38.407735+0000", 2026-02-23T13:13:38.449 INFO:teuthology.orchestra.run.trial068.stdout: "age": 0.038163415999999999, 2026-02-23T13:13:38.449 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 0.038180388000000003, 2026-02-23T13:13:38.450 INFO:teuthology.orchestra.run.trial068.stdout: "continuous": false, 2026-02-23T13:13:38.450 INFO:teuthology.orchestra.run.trial068.stdout: "type_data": { 2026-02-23T13:13:38.450 INFO:teuthology.orchestra.run.trial068.stdout: "flag_point": "waiting for peered", 2026-02-23T13:13:38.450 INFO:teuthology.orchestra.run.trial068.stdout: "client_info": { 2026-02-23T13:13:38.450 INFO:teuthology.orchestra.run.trial068.stdout: "client": "client.14774", 2026-02-23T13:13:38.450 INFO:teuthology.orchestra.run.trial068.stdout: "client_addr": "10.20.193.68:0/3517244806", 2026-02-23T13:13:38.450 INFO:teuthology.orchestra.run.trial068.stdout: "tid": 96261 2026-02-23T13:13:38.450 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:13:38.451 INFO:teuthology.orchestra.run.trial068.stdout: "events": [ 2026-02-23T13:13:38.451 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:13:38.451 INFO:teuthology.orchestra.run.trial068.stdout: "event": "initiated", 2026-02-23T13:13:38.451 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:13:38.407735+0000", 2026-02-23T13:13:38.451 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 0 2026-02-23T13:13:38.451 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:13:38.451 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:13:38.451 INFO:teuthology.orchestra.run.trial068.stdout: "event": "header_read", 2026-02-23T13:13:38.451 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:13:38.407735+0000", 2026-02-23T13:13:38.451 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 0 2026-02-23T13:13:38.451 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:13:38.451 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:13:38.451 INFO:teuthology.orchestra.run.trial068.stdout: "event": "throttled", 2026-02-23T13:13:38.451 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:13:38.407753+0000", 2026-02-23T13:13:38.451 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 1.8584999999999999e-05 2026-02-23T13:13:38.451 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:13:38.452 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:13:38.452 INFO:teuthology.orchestra.run.trial068.stdout: "event": "all_read", 2026-02-23T13:13:38.452 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:13:38.407987+0000", 2026-02-23T13:13:38.452 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 0.00023354800000000001 2026-02-23T13:13:38.452 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:13:38.452 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:13:38.452 INFO:teuthology.orchestra.run.trial068.stdout: "event": "dispatched", 2026-02-23T13:13:38.452 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:13:38.407993+0000", 2026-02-23T13:13:38.452 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 5.6710000000000004e-06 2026-02-23T13:13:38.452 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:13:38.452 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:13:38.452 INFO:teuthology.orchestra.run.trial068.stdout: "event": "queued_for_pg", 2026-02-23T13:13:38.452 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:13:38.408005+0000", 2026-02-23T13:13:38.452 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 1.2724e-05 2026-02-23T13:13:38.452 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:13:38.452 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:13:38.452 INFO:teuthology.orchestra.run.trial068.stdout: "event": "reached_pg", 2026-02-23T13:13:38.452 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:13:38.408041+0000", 2026-02-23T13:13:38.452 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 3.5806999999999998e-05 2026-02-23T13:13:38.453 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:13:38.453 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:13:38.453 INFO:teuthology.orchestra.run.trial068.stdout: "event": "waiting for peered", 2026-02-23T13:13:38.453 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:13:38.408045+0000", 2026-02-23T13:13:38.453 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 3.4070000000000002e-06 2026-02-23T13:13:38.453 INFO:teuthology.orchestra.run.trial068.stdout: } 2026-02-23T13:13:38.453 INFO:teuthology.orchestra.run.trial068.stdout: ] 2026-02-23T13:13:38.453 INFO:teuthology.orchestra.run.trial068.stdout: } 2026-02-23T13:13:38.453 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:13:38.453 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:13:38.453 INFO:teuthology.orchestra.run.trial068.stdout: "description": "osd_op(client.14774.0:96262 5.6 5.412d8096 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e176)", 2026-02-23T13:13:38.453 INFO:teuthology.orchestra.run.trial068.stdout: "initiated_at": "2026-02-23T13:13:38.408025+0000", 2026-02-23T13:13:38.453 INFO:teuthology.orchestra.run.trial068.stdout: "age": 0.037873761999999998, 2026-02-23T13:13:38.453 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 0.037922233, 2026-02-23T13:13:38.453 INFO:teuthology.orchestra.run.trial068.stdout: "continuous": false, 2026-02-23T13:13:38.453 INFO:teuthology.orchestra.run.trial068.stdout: "type_data": { 2026-02-23T13:13:38.453 INFO:teuthology.orchestra.run.trial068.stdout: "flag_point": "waiting for peered", 2026-02-23T13:13:38.453 INFO:teuthology.orchestra.run.trial068.stdout: "client_info": { 2026-02-23T13:13:38.454 INFO:teuthology.orchestra.run.trial068.stdout: "client": "client.14774", 2026-02-23T13:13:38.454 INFO:teuthology.orchestra.run.trial068.stdout: "client_addr": "10.20.193.68:0/3517244806", 2026-02-23T13:13:38.454 INFO:teuthology.orchestra.run.trial068.stdout: "tid": 96262 2026-02-23T13:13:38.454 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:13:38.454 INFO:teuthology.orchestra.run.trial068.stdout: "events": [ 2026-02-23T13:13:38.454 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:13:38.454 INFO:teuthology.orchestra.run.trial068.stdout: "event": "initiated", 2026-02-23T13:13:38.454 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:13:38.408025+0000", 2026-02-23T13:13:38.454 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 0 2026-02-23T13:13:38.454 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:13:38.454 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:13:38.454 INFO:teuthology.orchestra.run.trial068.stdout: "event": "header_read", 2026-02-23T13:13:38.454 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:13:38.408025+0000", 2026-02-23T13:13:38.454 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 0 2026-02-23T13:13:38.454 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:13:38.454 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:13:38.454 INFO:teuthology.orchestra.run.trial068.stdout: "event": "throttled", 2026-02-23T13:13:38.454 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:13:38.408032+0000", 2026-02-23T13:13:38.454 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 7.9249999999999995e-06 2026-02-23T13:13:38.454 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:13:38.455 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:13:38.455 INFO:teuthology.orchestra.run.trial068.stdout: "event": "all_read", 2026-02-23T13:13:38.455 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:13:38.408060+0000", 2026-02-23T13:13:38.455 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 2.7962000000000001e-05 2026-02-23T13:13:38.455 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:13:38.455 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:13:38.455 INFO:teuthology.orchestra.run.trial068.stdout: "event": "dispatched", 2026-02-23T13:13:38.455 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:13:38.408064+0000", 2026-02-23T13:13:38.455 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 3.7670000000000001e-06 2026-02-23T13:13:38.455 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:13:38.455 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:13:38.455 INFO:teuthology.orchestra.run.trial068.stdout: "event": "queued_for_pg", 2026-02-23T13:13:38.455 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:13:38.408066+0000", 2026-02-23T13:13:38.455 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 1.9439999999999999e-06 2026-02-23T13:13:38.455 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:13:38.455 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:13:38.455 INFO:teuthology.orchestra.run.trial068.stdout: "event": "reached_pg", 2026-02-23T13:13:38.455 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:13:38.408091+0000", 2026-02-23T13:13:38.455 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 2.5057000000000001e-05 2026-02-23T13:13:38.455 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:13:38.455 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:13:38.456 INFO:teuthology.orchestra.run.trial068.stdout: "event": "waiting for peered", 2026-02-23T13:13:38.456 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:13:38.408095+0000", 2026-02-23T13:13:38.456 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 3.907e-06 2026-02-23T13:13:38.456 INFO:teuthology.orchestra.run.trial068.stdout: } 2026-02-23T13:13:38.456 INFO:teuthology.orchestra.run.trial068.stdout: ] 2026-02-23T13:13:38.456 INFO:teuthology.orchestra.run.trial068.stdout: } 2026-02-23T13:13:38.456 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:13:38.456 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:13:38.456 INFO:teuthology.orchestra.run.trial068.stdout: "description": "osd_op(client.14774.0:96263 5.6 5.412d8096 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e176)", 2026-02-23T13:13:38.456 INFO:teuthology.orchestra.run.trial068.stdout: "initiated_at": "2026-02-23T13:13:38.408076+0000", 2026-02-23T13:13:38.456 INFO:teuthology.orchestra.run.trial068.stdout: "age": 0.037822556, 2026-02-23T13:13:38.456 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 0.037893199000000002, 2026-02-23T13:13:38.456 INFO:teuthology.orchestra.run.trial068.stdout: "continuous": false, 2026-02-23T13:13:38.456 INFO:teuthology.orchestra.run.trial068.stdout: "type_data": { 2026-02-23T13:13:38.456 INFO:teuthology.orchestra.run.trial068.stdout: "flag_point": "waiting for peered", 2026-02-23T13:13:38.456 INFO:teuthology.orchestra.run.trial068.stdout: "client_info": { 2026-02-23T13:13:38.456 INFO:teuthology.orchestra.run.trial068.stdout: "client": "client.14774", 2026-02-23T13:13:38.456 INFO:teuthology.orchestra.run.trial068.stdout: "client_addr": "10.20.193.68:0/3517244806", 2026-02-23T13:13:38.456 INFO:teuthology.orchestra.run.trial068.stdout: "tid": 96263 2026-02-23T13:13:38.456 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:13:38.456 INFO:teuthology.orchestra.run.trial068.stdout: "events": [ 2026-02-23T13:13:38.457 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:13:38.457 INFO:teuthology.orchestra.run.trial068.stdout: "event": "initiated", 2026-02-23T13:13:38.457 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:13:38.408076+0000", 2026-02-23T13:13:38.457 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 0 2026-02-23T13:13:38.457 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:13:38.457 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:13:38.457 INFO:teuthology.orchestra.run.trial068.stdout: "event": "header_read", 2026-02-23T13:13:38.457 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:13:38.408076+0000", 2026-02-23T13:13:38.457 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 0 2026-02-23T13:13:38.457 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:13:38.457 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:13:38.457 INFO:teuthology.orchestra.run.trial068.stdout: "event": "throttled", 2026-02-23T13:13:38.457 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:13:38.408081+0000", 2026-02-23T13:13:38.457 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 4.7489999999999998e-06 2026-02-23T13:13:38.457 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:13:38.457 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:13:38.457 INFO:teuthology.orchestra.run.trial068.stdout: "event": "all_read", 2026-02-23T13:13:38.457 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:13:38.408099+0000", 2026-02-23T13:13:38.457 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 1.8274000000000002e-05 2026-02-23T13:13:38.457 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:13:38.458 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:13:38.458 INFO:teuthology.orchestra.run.trial068.stdout: "event": "dispatched", 2026-02-23T13:13:38.458 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:13:38.408104+0000", 2026-02-23T13:13:38.458 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 4.9799999999999998e-06 2026-02-23T13:13:38.458 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:13:38.458 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:13:38.458 INFO:teuthology.orchestra.run.trial068.stdout: "event": "queued_for_pg", 2026-02-23T13:13:38.458 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:13:38.408106+0000", 2026-02-23T13:13:38.458 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 1.8929999999999999e-06 2026-02-23T13:13:38.458 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:13:38.458 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:13:38.458 INFO:teuthology.orchestra.run.trial068.stdout: "event": "reached_pg", 2026-02-23T13:13:38.458 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:13:38.408130+0000", 2026-02-23T13:13:38.458 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 2.4807000000000002e-05 2026-02-23T13:13:38.458 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:13:38.458 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:13:38.458 INFO:teuthology.orchestra.run.trial068.stdout: "event": "waiting for peered", 2026-02-23T13:13:38.458 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:13:38.408133+0000", 2026-02-23T13:13:38.458 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 2.1440000000000001e-06 2026-02-23T13:13:38.458 INFO:teuthology.orchestra.run.trial068.stdout: } 2026-02-23T13:13:38.459 INFO:teuthology.orchestra.run.trial068.stdout: ] 2026-02-23T13:13:38.459 INFO:teuthology.orchestra.run.trial068.stdout: } 2026-02-23T13:13:38.459 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:13:38.459 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:13:38.459 INFO:teuthology.orchestra.run.trial068.stdout: "description": "osd_op(client.14774.0:96264 5.6 5.412d8096 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e176)", 2026-02-23T13:13:38.459 INFO:teuthology.orchestra.run.trial068.stdout: "initiated_at": "2026-02-23T13:13:38.408119+0000", 2026-02-23T13:13:38.459 INFO:teuthology.orchestra.run.trial068.stdout: "age": 0.037779365000000002, 2026-02-23T13:13:38.459 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 0.037869234000000002, 2026-02-23T13:13:38.459 INFO:teuthology.orchestra.run.trial068.stdout: "continuous": false, 2026-02-23T13:13:38.459 INFO:teuthology.orchestra.run.trial068.stdout: "type_data": { 2026-02-23T13:13:38.459 INFO:teuthology.orchestra.run.trial068.stdout: "flag_point": "waiting for peered", 2026-02-23T13:13:38.459 INFO:teuthology.orchestra.run.trial068.stdout: "client_info": { 2026-02-23T13:13:38.459 INFO:teuthology.orchestra.run.trial068.stdout: "client": "client.14774", 2026-02-23T13:13:38.459 INFO:teuthology.orchestra.run.trial068.stdout: "client_addr": "10.20.193.68:0/3517244806", 2026-02-23T13:13:38.459 INFO:teuthology.orchestra.run.trial068.stdout: "tid": 96264 2026-02-23T13:13:38.459 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:13:38.459 INFO:teuthology.orchestra.run.trial068.stdout: "events": [ 2026-02-23T13:13:38.459 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:13:38.459 INFO:teuthology.orchestra.run.trial068.stdout: "event": "initiated", 2026-02-23T13:13:38.459 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:13:38.408119+0000", 2026-02-23T13:13:38.460 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 0 2026-02-23T13:13:38.460 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:13:38.460 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:13:38.460 INFO:teuthology.orchestra.run.trial068.stdout: "event": "header_read", 2026-02-23T13:13:38.460 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:13:38.408119+0000", 2026-02-23T13:13:38.460 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 0 2026-02-23T13:13:38.460 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:13:38.460 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:13:38.460 INFO:teuthology.orchestra.run.trial068.stdout: "event": "throttled", 2026-02-23T13:13:38.460 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:13:38.408127+0000", 2026-02-23T13:13:38.460 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 8.4460000000000001e-06 2026-02-23T13:13:38.460 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:13:38.460 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:13:38.460 INFO:teuthology.orchestra.run.trial068.stdout: "event": "all_read", 2026-02-23T13:13:38.460 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:13:38.408159+0000", 2026-02-23T13:13:38.460 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 3.184e-05 2026-02-23T13:13:38.460 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:13:38.460 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:13:38.460 INFO:teuthology.orchestra.run.trial068.stdout: "event": "dispatched", 2026-02-23T13:13:38.461 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:13:38.408164+0000", 2026-02-23T13:13:38.461 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 4.2980000000000002e-06 2026-02-23T13:13:38.461 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:13:38.461 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:13:38.461 INFO:teuthology.orchestra.run.trial068.stdout: "event": "queued_for_pg", 2026-02-23T13:13:38.461 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:13:38.408167+0000", 2026-02-23T13:13:38.461 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 3.9269999999999998e-06 2026-02-23T13:13:38.461 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:13:38.461 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:13:38.461 INFO:teuthology.orchestra.run.trial068.stdout: "event": "reached_pg", 2026-02-23T13:13:38.461 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:13:38.408186+0000", 2026-02-23T13:13:38.461 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 1.8445e-05 2026-02-23T13:13:38.461 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:13:38.461 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:13:38.461 INFO:teuthology.orchestra.run.trial068.stdout: "event": "waiting for peered", 2026-02-23T13:13:38.461 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:13:38.408189+0000", 2026-02-23T13:13:38.461 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 3.3960000000000002e-06 2026-02-23T13:13:38.461 INFO:teuthology.orchestra.run.trial068.stdout: } 2026-02-23T13:13:38.461 INFO:teuthology.orchestra.run.trial068.stdout: ] 2026-02-23T13:13:38.462 INFO:teuthology.orchestra.run.trial068.stdout: } 2026-02-23T13:13:38.462 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:13:38.462 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:13:38.462 INFO:teuthology.orchestra.run.trial068.stdout: "description": "osd_op(client.14774.0:96274 5.c 5.52a5149c (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e176)", 2026-02-23T13:13:38.462 INFO:teuthology.orchestra.run.trial068.stdout: "initiated_at": "2026-02-23T13:13:38.408179+0000", 2026-02-23T13:13:38.462 INFO:teuthology.orchestra.run.trial068.stdout: "age": 0.037719552000000003, 2026-02-23T13:13:38.462 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 0.037835550000000003, 2026-02-23T13:13:38.462 INFO:teuthology.orchestra.run.trial068.stdout: "continuous": false, 2026-02-23T13:13:38.462 INFO:teuthology.orchestra.run.trial068.stdout: "type_data": { 2026-02-23T13:13:38.462 INFO:teuthology.orchestra.run.trial068.stdout: "flag_point": "waiting for peered", 2026-02-23T13:13:38.462 INFO:teuthology.orchestra.run.trial068.stdout: "client_info": { 2026-02-23T13:13:38.462 INFO:teuthology.orchestra.run.trial068.stdout: "client": "client.14774", 2026-02-23T13:13:38.462 INFO:teuthology.orchestra.run.trial068.stdout: "client_addr": "10.20.193.68:0/3517244806", 2026-02-23T13:13:38.462 INFO:teuthology.orchestra.run.trial068.stdout: "tid": 96274 2026-02-23T13:13:38.462 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:13:38.462 INFO:teuthology.orchestra.run.trial068.stdout: "events": [ 2026-02-23T13:13:38.462 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:13:38.462 INFO:teuthology.orchestra.run.trial068.stdout: "event": "initiated", 2026-02-23T13:13:38.462 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:13:38.408179+0000", 2026-02-23T13:13:38.463 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 0 2026-02-23T13:13:38.463 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:13:38.463 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:13:38.463 INFO:teuthology.orchestra.run.trial068.stdout: "event": "header_read", 2026-02-23T13:13:38.463 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:13:38.408179+0000", 2026-02-23T13:13:38.463 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 0 2026-02-23T13:13:38.463 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:13:38.463 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:13:38.463 INFO:teuthology.orchestra.run.trial068.stdout: "event": "throttled", 2026-02-23T13:13:38.463 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:13:38.408188+0000", 2026-02-23T13:13:38.463 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 9.5070000000000006e-06 2026-02-23T13:13:38.463 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:13:38.463 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:13:38.463 INFO:teuthology.orchestra.run.trial068.stdout: "event": "all_read", 2026-02-23T13:13:38.463 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:13:38.408213+0000", 2026-02-23T13:13:38.463 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 2.4837e-05 2026-02-23T13:13:38.463 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:13:38.463 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:13:38.463 INFO:teuthology.orchestra.run.trial068.stdout: "event": "dispatched", 2026-02-23T13:13:38.463 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:13:38.408217+0000", 2026-02-23T13:13:38.463 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 4.048e-06 2026-02-23T13:13:38.463 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:13:38.463 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:13:38.463 INFO:teuthology.orchestra.run.trial068.stdout: "event": "queued_for_pg", 2026-02-23T13:13:38.463 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:13:38.408219+0000", 2026-02-23T13:13:38.464 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 1.432e-06 2026-02-23T13:13:38.464 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:13:38.464 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:13:38.464 INFO:teuthology.orchestra.run.trial068.stdout: "event": "reached_pg", 2026-02-23T13:13:38.464 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:13:38.408272+0000", 2026-02-23T13:13:38.464 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 5.3610999999999999e-05 2026-02-23T13:13:38.464 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:13:38.464 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:13:38.464 INFO:teuthology.orchestra.run.trial068.stdout: "event": "waiting for peered", 2026-02-23T13:13:38.464 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:13:38.408277+0000", 2026-02-23T13:13:38.464 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 4.639e-06 2026-02-23T13:13:38.464 INFO:teuthology.orchestra.run.trial068.stdout: } 2026-02-23T13:13:38.464 INFO:teuthology.orchestra.run.trial068.stdout: ] 2026-02-23T13:13:38.464 INFO:teuthology.orchestra.run.trial068.stdout: } 2026-02-23T13:13:38.464 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:13:38.464 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:13:38.464 INFO:teuthology.orchestra.run.trial068.stdout: "description": "osd_op(client.14774.0:96275 5.c 5.52a5149c (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e176)", 2026-02-23T13:13:38.464 INFO:teuthology.orchestra.run.trial068.stdout: "initiated_at": "2026-02-23T13:13:38.408237+0000", 2026-02-23T13:13:38.464 INFO:teuthology.orchestra.run.trial068.stdout: "age": 0.037661412999999998, 2026-02-23T13:13:38.464 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 0.037811895999999998, 2026-02-23T13:13:38.464 INFO:teuthology.orchestra.run.trial068.stdout: "continuous": false, 2026-02-23T13:13:38.464 INFO:teuthology.orchestra.run.trial068.stdout: "type_data": { 2026-02-23T13:13:38.464 INFO:teuthology.orchestra.run.trial068.stdout: "flag_point": "waiting for peered", 2026-02-23T13:13:38.464 INFO:teuthology.orchestra.run.trial068.stdout: "client_info": { 2026-02-23T13:13:38.464 INFO:teuthology.orchestra.run.trial068.stdout: "client": "client.14774", 2026-02-23T13:13:38.464 INFO:teuthology.orchestra.run.trial068.stdout: "client_addr": "10.20.193.68:0/3517244806", 2026-02-23T13:13:38.464 INFO:teuthology.orchestra.run.trial068.stdout: "tid": 96275 2026-02-23T13:13:38.464 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:13:38.464 INFO:teuthology.orchestra.run.trial068.stdout: "events": [ 2026-02-23T13:13:38.464 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:13:38.465 INFO:teuthology.orchestra.run.trial068.stdout: "event": "initiated", 2026-02-23T13:13:38.465 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:13:38.408237+0000", 2026-02-23T13:13:38.465 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 0 2026-02-23T13:13:38.465 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:13:38.465 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:13:38.465 INFO:teuthology.orchestra.run.trial068.stdout: "event": "header_read", 2026-02-23T13:13:38.465 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:13:38.408237+0000", 2026-02-23T13:13:38.465 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 0 2026-02-23T13:13:38.465 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:13:38.465 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:13:38.465 INFO:teuthology.orchestra.run.trial068.stdout: "event": "throttled", 2026-02-23T13:13:38.465 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:13:38.408245+0000", 2026-02-23T13:13:38.465 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 8.0250000000000002e-06 2026-02-23T13:13:38.465 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:13:38.465 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:13:38.465 INFO:teuthology.orchestra.run.trial068.stdout: "event": "all_read", 2026-02-23T13:13:38.465 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:13:38.408273+0000", 2026-02-23T13:13:38.465 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 2.8223000000000001e-05 2026-02-23T13:13:38.465 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:13:38.465 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:13:38.465 INFO:teuthology.orchestra.run.trial068.stdout: "event": "dispatched", 2026-02-23T13:13:38.465 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:13:38.408278+0000", 2026-02-23T13:13:38.465 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 4.7790000000000003e-06 2026-02-23T13:13:38.465 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:13:38.465 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:13:38.465 INFO:teuthology.orchestra.run.trial068.stdout: "event": "queued_for_pg", 2026-02-23T13:13:38.465 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:13:38.408280+0000", 2026-02-23T13:13:38.465 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 2.3240000000000001e-06 2026-02-23T13:13:38.465 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:13:38.465 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:13:38.466 INFO:teuthology.orchestra.run.trial068.stdout: "event": "reached_pg", 2026-02-23T13:13:38.466 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:13:38.408315+0000", 2026-02-23T13:13:38.466 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 3.4705000000000001e-05 2026-02-23T13:13:38.466 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:13:38.466 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:13:38.466 INFO:teuthology.orchestra.run.trial068.stdout: "event": "waiting for peered", 2026-02-23T13:13:38.466 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:13:38.408318+0000", 2026-02-23T13:13:38.466 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 3.427e-06 2026-02-23T13:13:38.466 INFO:teuthology.orchestra.run.trial068.stdout: } 2026-02-23T13:13:38.466 INFO:teuthology.orchestra.run.trial068.stdout: ] 2026-02-23T13:13:38.466 INFO:teuthology.orchestra.run.trial068.stdout: } 2026-02-23T13:13:38.466 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:13:38.466 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:13:38.466 INFO:teuthology.orchestra.run.trial068.stdout: "description": "osd_op(client.14774.0:96276 5.c 5.52a5149c (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e176)", 2026-02-23T13:13:38.466 INFO:teuthology.orchestra.run.trial068.stdout: "initiated_at": "2026-02-23T13:13:38.408297+0000", 2026-02-23T13:13:38.466 INFO:teuthology.orchestra.run.trial068.stdout: "age": 0.037601550999999997, 2026-02-23T13:13:38.466 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 0.037772563000000002, 2026-02-23T13:13:38.466 INFO:teuthology.orchestra.run.trial068.stdout: "continuous": false, 2026-02-23T13:13:38.466 INFO:teuthology.orchestra.run.trial068.stdout: "type_data": { 2026-02-23T13:13:38.466 INFO:teuthology.orchestra.run.trial068.stdout: "flag_point": "waiting for peered", 2026-02-23T13:13:38.466 INFO:teuthology.orchestra.run.trial068.stdout: "client_info": { 2026-02-23T13:13:38.466 INFO:teuthology.orchestra.run.trial068.stdout: "client": "client.14774", 2026-02-23T13:13:38.466 INFO:teuthology.orchestra.run.trial068.stdout: "client_addr": "10.20.193.68:0/3517244806", 2026-02-23T13:13:38.466 INFO:teuthology.orchestra.run.trial068.stdout: "tid": 96276 2026-02-23T13:13:38.466 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:13:38.466 INFO:teuthology.orchestra.run.trial068.stdout: "events": [ 2026-02-23T13:13:38.466 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:13:38.466 INFO:teuthology.orchestra.run.trial068.stdout: "event": "initiated", 2026-02-23T13:13:38.466 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:13:38.408297+0000", 2026-02-23T13:13:38.466 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 0 2026-02-23T13:13:38.466 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:13:38.467 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:13:38.467 INFO:teuthology.orchestra.run.trial068.stdout: "event": "header_read", 2026-02-23T13:13:38.467 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:13:38.408297+0000", 2026-02-23T13:13:38.467 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 0 2026-02-23T13:13:38.467 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:13:38.467 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:13:38.467 INFO:teuthology.orchestra.run.trial068.stdout: "event": "throttled", 2026-02-23T13:13:38.467 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:13:38.408313+0000", 2026-02-23T13:13:38.467 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 1.6201e-05 2026-02-23T13:13:38.467 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:13:38.467 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:13:38.467 INFO:teuthology.orchestra.run.trial068.stdout: "event": "all_read", 2026-02-23T13:13:38.467 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:13:38.408358+0000", 2026-02-23T13:13:38.467 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 4.5515e-05 2026-02-23T13:13:38.467 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:13:38.467 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:13:38.467 INFO:teuthology.orchestra.run.trial068.stdout: "event": "dispatched", 2026-02-23T13:13:38.467 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:13:38.408368+0000", 2026-02-23T13:13:38.467 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 9.3780000000000007e-06 2026-02-23T13:13:38.467 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:13:38.467 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:13:38.467 INFO:teuthology.orchestra.run.trial068.stdout: "event": "queued_for_pg", 2026-02-23T13:13:38.467 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:13:38.408370+0000", 2026-02-23T13:13:38.467 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 2.2639999999999998e-06 2026-02-23T13:13:38.467 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:13:38.467 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:13:38.467 INFO:teuthology.orchestra.run.trial068.stdout: "event": "reached_pg", 2026-02-23T13:13:38.467 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:13:38.408412+0000", 2026-02-23T13:13:38.467 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 4.1578000000000001e-05 2026-02-23T13:13:38.467 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:13:38.467 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:13:38.468 INFO:teuthology.orchestra.run.trial068.stdout: "event": "waiting for peered", 2026-02-23T13:13:38.468 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:13:38.408416+0000", 2026-02-23T13:13:38.468 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 4.7489999999999998e-06 2026-02-23T13:13:38.468 INFO:teuthology.orchestra.run.trial068.stdout: } 2026-02-23T13:13:38.468 INFO:teuthology.orchestra.run.trial068.stdout: ] 2026-02-23T13:13:38.468 INFO:teuthology.orchestra.run.trial068.stdout: } 2026-02-23T13:13:38.468 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:13:38.468 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:13:38.468 INFO:teuthology.orchestra.run.trial068.stdout: "description": "osd_op(client.14774.0:96277 5.c 5.52a5149c (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e176)", 2026-02-23T13:13:38.468 INFO:teuthology.orchestra.run.trial068.stdout: "initiated_at": "2026-02-23T13:13:38.408440+0000", 2026-02-23T13:13:38.468 INFO:teuthology.orchestra.run.trial068.stdout: "age": 0.037458492000000003, 2026-02-23T13:13:38.468 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 0.037648068, 2026-02-23T13:13:38.468 INFO:teuthology.orchestra.run.trial068.stdout: "continuous": false, 2026-02-23T13:13:38.468 INFO:teuthology.orchestra.run.trial068.stdout: "type_data": { 2026-02-23T13:13:38.468 INFO:teuthology.orchestra.run.trial068.stdout: "flag_point": "waiting for peered", 2026-02-23T13:13:38.468 INFO:teuthology.orchestra.run.trial068.stdout: "client_info": { 2026-02-23T13:13:38.468 INFO:teuthology.orchestra.run.trial068.stdout: "client": "client.14774", 2026-02-23T13:13:38.468 INFO:teuthology.orchestra.run.trial068.stdout: "client_addr": "10.20.193.68:0/3517244806", 2026-02-23T13:13:38.468 INFO:teuthology.orchestra.run.trial068.stdout: "tid": 96277 2026-02-23T13:13:38.468 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:13:38.468 INFO:teuthology.orchestra.run.trial068.stdout: "events": [ 2026-02-23T13:13:38.468 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:13:38.468 INFO:teuthology.orchestra.run.trial068.stdout: "event": "initiated", 2026-02-23T13:13:38.468 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:13:38.408440+0000", 2026-02-23T13:13:38.468 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 0 2026-02-23T13:13:38.468 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:13:38.468 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:13:38.468 INFO:teuthology.orchestra.run.trial068.stdout: "event": "header_read", 2026-02-23T13:13:38.468 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:13:38.408440+0000", 2026-02-23T13:13:38.469 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 0 2026-02-23T13:13:38.469 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:13:38.469 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:13:38.469 INFO:teuthology.orchestra.run.trial068.stdout: "event": "throttled", 2026-02-23T13:13:38.469 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:13:38.408466+0000", 2026-02-23T13:13:38.469 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 2.6159000000000001e-05 2026-02-23T13:13:38.469 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:13:38.469 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:13:38.469 INFO:teuthology.orchestra.run.trial068.stdout: "event": "all_read", 2026-02-23T13:13:38.469 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:13:38.408792+0000", 2026-02-23T13:13:38.469 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 0.00032581199999999998 2026-02-23T13:13:38.469 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:13:38.469 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:13:38.469 INFO:teuthology.orchestra.run.trial068.stdout: "event": "dispatched", 2026-02-23T13:13:38.469 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:13:38.408795+0000", 2026-02-23T13:13:38.469 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 3.5370000000000002e-06 2026-02-23T13:13:38.469 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:13:38.469 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:13:38.469 INFO:teuthology.orchestra.run.trial068.stdout: "event": "queued_for_pg", 2026-02-23T13:13:38.469 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:13:38.408798+0000", 2026-02-23T13:13:38.469 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 2.3039999999999999e-06 2026-02-23T13:13:38.469 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:13:38.469 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:13:38.469 INFO:teuthology.orchestra.run.trial068.stdout: "event": "reached_pg", 2026-02-23T13:13:38.469 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:13:38.408820+0000", 2026-02-23T13:13:38.469 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 2.2773e-05 2026-02-23T13:13:38.469 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:13:38.469 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:13:38.469 INFO:teuthology.orchestra.run.trial068.stdout: "event": "waiting for peered", 2026-02-23T13:13:38.469 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:13:38.408822+0000", 2026-02-23T13:13:38.470 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 2.0040000000000002e-06 2026-02-23T13:13:38.470 INFO:teuthology.orchestra.run.trial068.stdout: } 2026-02-23T13:13:38.470 INFO:teuthology.orchestra.run.trial068.stdout: ] 2026-02-23T13:13:38.470 INFO:teuthology.orchestra.run.trial068.stdout: } 2026-02-23T13:13:38.470 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:13:38.470 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:13:38.470 INFO:teuthology.orchestra.run.trial068.stdout: "description": "osd_op(client.14774.0:96278 5.c 5.52a5149c (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e176)", 2026-02-23T13:13:38.470 INFO:teuthology.orchestra.run.trial068.stdout: "initiated_at": "2026-02-23T13:13:38.408813+0000", 2026-02-23T13:13:38.470 INFO:teuthology.orchestra.run.trial068.stdout: "age": 0.037085571999999997, 2026-02-23T13:13:38.470 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 0.037291538999999999, 2026-02-23T13:13:38.470 INFO:teuthology.orchestra.run.trial068.stdout: "continuous": false, 2026-02-23T13:13:38.470 INFO:teuthology.orchestra.run.trial068.stdout: "type_data": { 2026-02-23T13:13:38.470 INFO:teuthology.orchestra.run.trial068.stdout: "flag_point": "waiting for peered", 2026-02-23T13:13:38.470 INFO:teuthology.orchestra.run.trial068.stdout: "client_info": { 2026-02-23T13:13:38.470 INFO:teuthology.orchestra.run.trial068.stdout: "client": "client.14774", 2026-02-23T13:13:38.470 INFO:teuthology.orchestra.run.trial068.stdout: "client_addr": "10.20.193.68:0/3517244806", 2026-02-23T13:13:38.470 INFO:teuthology.orchestra.run.trial068.stdout: "tid": 96278 2026-02-23T13:13:38.470 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:13:38.470 INFO:teuthology.orchestra.run.trial068.stdout: "events": [ 2026-02-23T13:13:38.470 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:13:38.470 INFO:teuthology.orchestra.run.trial068.stdout: "event": "initiated", 2026-02-23T13:13:38.470 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:13:38.408813+0000", 2026-02-23T13:13:38.470 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 0 2026-02-23T13:13:38.470 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:13:38.470 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:13:38.470 INFO:teuthology.orchestra.run.trial068.stdout: "event": "header_read", 2026-02-23T13:13:38.470 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:13:38.408813+0000", 2026-02-23T13:13:38.470 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 0 2026-02-23T13:13:38.470 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:13:38.470 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:13:38.470 INFO:teuthology.orchestra.run.trial068.stdout: "event": "throttled", 2026-02-23T13:13:38.471 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:13:38.408828+0000", 2026-02-23T13:13:38.471 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 1.5169e-05 2026-02-23T13:13:38.471 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:13:38.471 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:13:38.471 INFO:teuthology.orchestra.run.trial068.stdout: "event": "all_read", 2026-02-23T13:13:38.471 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:13:38.408857+0000", 2026-02-23T13:13:38.471 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 2.8804000000000001e-05 2026-02-23T13:13:38.471 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:13:38.471 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:13:38.471 INFO:teuthology.orchestra.run.trial068.stdout: "event": "dispatched", 2026-02-23T13:13:38.471 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:13:38.408862+0000", 2026-02-23T13:13:38.471 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 5.1800000000000004e-06 2026-02-23T13:13:38.471 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:13:38.471 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:13:38.471 INFO:teuthology.orchestra.run.trial068.stdout: "event": "queued_for_pg", 2026-02-23T13:13:38.471 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:13:38.408864+0000", 2026-02-23T13:13:38.471 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 1.663e-06 2026-02-23T13:13:38.471 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:13:38.471 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:13:38.471 INFO:teuthology.orchestra.run.trial068.stdout: "event": "reached_pg", 2026-02-23T13:13:38.471 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:13:38.408892+0000", 2026-02-23T13:13:38.471 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 2.7982000000000001e-05 2026-02-23T13:13:38.471 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:13:38.471 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:13:38.471 INFO:teuthology.orchestra.run.trial068.stdout: "event": "waiting for peered", 2026-02-23T13:13:38.471 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:13:38.408893+0000", 2026-02-23T13:13:38.471 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 1.934e-06 2026-02-23T13:13:38.471 INFO:teuthology.orchestra.run.trial068.stdout: } 2026-02-23T13:13:38.471 INFO:teuthology.orchestra.run.trial068.stdout: ] 2026-02-23T13:13:38.471 INFO:teuthology.orchestra.run.trial068.stdout: } 2026-02-23T13:13:38.471 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:13:38.472 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:13:38.472 INFO:teuthology.orchestra.run.trial068.stdout: "description": "osd_op(client.14774.0:96279 5.c 5.52a5149c (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e176)", 2026-02-23T13:13:38.472 INFO:teuthology.orchestra.run.trial068.stdout: "initiated_at": "2026-02-23T13:13:38.408883+0000", 2026-02-23T13:13:38.472 INFO:teuthology.orchestra.run.trial068.stdout: "age": 0.037015068999999998, 2026-02-23T13:13:38.472 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 0.037236805999999997, 2026-02-23T13:13:38.472 INFO:teuthology.orchestra.run.trial068.stdout: "continuous": false, 2026-02-23T13:13:38.472 INFO:teuthology.orchestra.run.trial068.stdout: "type_data": { 2026-02-23T13:13:38.472 INFO:teuthology.orchestra.run.trial068.stdout: "flag_point": "waiting for peered", 2026-02-23T13:13:38.472 INFO:teuthology.orchestra.run.trial068.stdout: "client_info": { 2026-02-23T13:13:38.472 INFO:teuthology.orchestra.run.trial068.stdout: "client": "client.14774", 2026-02-23T13:13:38.472 INFO:teuthology.orchestra.run.trial068.stdout: "client_addr": "10.20.193.68:0/3517244806", 2026-02-23T13:13:38.472 INFO:teuthology.orchestra.run.trial068.stdout: "tid": 96279 2026-02-23T13:13:38.472 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:13:38.472 INFO:teuthology.orchestra.run.trial068.stdout: "events": [ 2026-02-23T13:13:38.472 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:13:38.472 INFO:teuthology.orchestra.run.trial068.stdout: "event": "initiated", 2026-02-23T13:13:38.472 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:13:38.408883+0000", 2026-02-23T13:13:38.472 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 0 2026-02-23T13:13:38.472 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:13:38.472 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:13:38.472 INFO:teuthology.orchestra.run.trial068.stdout: "event": "header_read", 2026-02-23T13:13:38.472 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:13:38.408883+0000", 2026-02-23T13:13:38.472 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 0 2026-02-23T13:13:38.472 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:13:38.472 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:13:38.472 INFO:teuthology.orchestra.run.trial068.stdout: "event": "throttled", 2026-02-23T13:13:38.472 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:13:38.408898+0000", 2026-02-23T13:13:38.472 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 1.5078e-05 2026-02-23T13:13:38.472 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:13:38.472 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:13:38.473 INFO:teuthology.orchestra.run.trial068.stdout: "event": "all_read", 2026-02-23T13:13:38.473 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:13:38.408933+0000", 2026-02-23T13:13:38.473 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 3.4615e-05 2026-02-23T13:13:38.473 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:13:38.473 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:13:38.473 INFO:teuthology.orchestra.run.trial068.stdout: "event": "dispatched", 2026-02-23T13:13:38.473 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:13:38.408938+0000", 2026-02-23T13:13:38.473 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 4.5990000000000004e-06 2026-02-23T13:13:38.473 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:13:38.473 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:13:38.473 INFO:teuthology.orchestra.run.trial068.stdout: "event": "queued_for_pg", 2026-02-23T13:13:38.473 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:13:38.408939+0000", 2026-02-23T13:13:38.473 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 1.8729999999999999e-06 2026-02-23T13:13:38.473 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:13:38.473 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:13:38.473 INFO:teuthology.orchestra.run.trial068.stdout: "event": "reached_pg", 2026-02-23T13:13:38.473 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:13:38.408979+0000", 2026-02-23T13:13:38.473 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 3.9334e-05 2026-02-23T13:13:38.473 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:13:38.473 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:13:38.473 INFO:teuthology.orchestra.run.trial068.stdout: "event": "waiting for peered", 2026-02-23T13:13:38.473 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:13:38.408983+0000", 2026-02-23T13:13:38.473 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 4.2880000000000003e-06 2026-02-23T13:13:38.473 INFO:teuthology.orchestra.run.trial068.stdout: } 2026-02-23T13:13:38.473 INFO:teuthology.orchestra.run.trial068.stdout: ] 2026-02-23T13:13:38.473 INFO:teuthology.orchestra.run.trial068.stdout: } 2026-02-23T13:13:38.473 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:13:38.473 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:13:38.473 INFO:teuthology.orchestra.run.trial068.stdout: "description": "osd_op(client.14774.0:96280 5.c 5.52a5149c (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e176)", 2026-02-23T13:13:38.473 INFO:teuthology.orchestra.run.trial068.stdout: "initiated_at": "2026-02-23T13:13:38.408960+0000", 2026-02-23T13:13:38.474 INFO:teuthology.orchestra.run.trial068.stdout: "age": 0.036938024, 2026-02-23T13:13:38.474 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 0.037175570999999998, 2026-02-23T13:13:38.474 INFO:teuthology.orchestra.run.trial068.stdout: "continuous": false, 2026-02-23T13:13:38.474 INFO:teuthology.orchestra.run.trial068.stdout: "type_data": { 2026-02-23T13:13:38.474 INFO:teuthology.orchestra.run.trial068.stdout: "flag_point": "waiting for peered", 2026-02-23T13:13:38.474 INFO:teuthology.orchestra.run.trial068.stdout: "client_info": { 2026-02-23T13:13:38.474 INFO:teuthology.orchestra.run.trial068.stdout: "client": "client.14774", 2026-02-23T13:13:38.474 INFO:teuthology.orchestra.run.trial068.stdout: "client_addr": "10.20.193.68:0/3517244806", 2026-02-23T13:13:38.474 INFO:teuthology.orchestra.run.trial068.stdout: "tid": 96280 2026-02-23T13:13:38.474 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:13:38.474 INFO:teuthology.orchestra.run.trial068.stdout: "events": [ 2026-02-23T13:13:38.474 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:13:38.474 INFO:teuthology.orchestra.run.trial068.stdout: "event": "initiated", 2026-02-23T13:13:38.474 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:13:38.408960+0000", 2026-02-23T13:13:38.474 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 0 2026-02-23T13:13:38.474 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:13:38.474 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:13:38.475 INFO:teuthology.orchestra.run.trial068.stdout: "event": "header_read", 2026-02-23T13:13:38.475 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:13:38.408960+0000", 2026-02-23T13:13:38.475 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 0 2026-02-23T13:13:38.475 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:13:38.475 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:13:38.475 INFO:teuthology.orchestra.run.trial068.stdout: "event": "throttled", 2026-02-23T13:13:38.475 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:13:38.408971+0000", 2026-02-23T13:13:38.475 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 1.0549e-05 2026-02-23T13:13:38.475 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:13:38.475 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:13:38.475 INFO:teuthology.orchestra.run.trial068.stdout: "event": "all_read", 2026-02-23T13:13:38.475 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:13:38.409023+0000", 2026-02-23T13:13:38.475 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 5.2308999999999997e-05 2026-02-23T13:13:38.475 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:13:38.475 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:13:38.475 INFO:teuthology.orchestra.run.trial068.stdout: "event": "dispatched", 2026-02-23T13:13:38.475 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:13:38.409029+0000", 2026-02-23T13:13:38.475 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 5.75e-06 2026-02-23T13:13:38.475 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:13:38.475 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:13:38.475 INFO:teuthology.orchestra.run.trial068.stdout: "event": "queued_for_pg", 2026-02-23T13:13:38.475 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:13:38.409031+0000", 2026-02-23T13:13:38.475 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 1.9640000000000002e-06 2026-02-23T13:13:38.475 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:13:38.475 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:13:38.475 INFO:teuthology.orchestra.run.trial068.stdout: "event": "reached_pg", 2026-02-23T13:13:38.475 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:13:38.409315+0000", 2026-02-23T13:13:38.476 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 0.000284505 2026-02-23T13:13:38.476 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:13:38.476 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:13:38.476 INFO:teuthology.orchestra.run.trial068.stdout: "event": "waiting for peered", 2026-02-23T13:13:38.476 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:13:38.409320+0000", 2026-02-23T13:13:38.476 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 4.318e-06 2026-02-23T13:13:38.476 INFO:teuthology.orchestra.run.trial068.stdout: } 2026-02-23T13:13:38.476 INFO:teuthology.orchestra.run.trial068.stdout: ] 2026-02-23T13:13:38.476 INFO:teuthology.orchestra.run.trial068.stdout: } 2026-02-23T13:13:38.476 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:13:38.476 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:13:38.476 INFO:teuthology.orchestra.run.trial068.stdout: "description": "osd_op(client.14774.0:96353 5.c 5.5301429c (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e176)", 2026-02-23T13:13:38.476 INFO:teuthology.orchestra.run.trial068.stdout: "initiated_at": "2026-02-23T13:13:38.409045+0000", 2026-02-23T13:13:38.476 INFO:teuthology.orchestra.run.trial068.stdout: "age": 0.036853735999999998, 2026-02-23T13:13:38.476 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 0.037107282999999998, 2026-02-23T13:13:38.476 INFO:teuthology.orchestra.run.trial068.stdout: "continuous": false, 2026-02-23T13:13:38.476 INFO:teuthology.orchestra.run.trial068.stdout: "type_data": { 2026-02-23T13:13:38.476 INFO:teuthology.orchestra.run.trial068.stdout: "flag_point": "waiting for peered", 2026-02-23T13:13:38.476 INFO:teuthology.orchestra.run.trial068.stdout: "client_info": { 2026-02-23T13:13:38.476 INFO:teuthology.orchestra.run.trial068.stdout: "client": "client.14774", 2026-02-23T13:13:38.476 INFO:teuthology.orchestra.run.trial068.stdout: "client_addr": "10.20.193.68:0/3517244806", 2026-02-23T13:13:38.476 INFO:teuthology.orchestra.run.trial068.stdout: "tid": 96353 2026-02-23T13:13:38.476 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:13:38.476 INFO:teuthology.orchestra.run.trial068.stdout: "events": [ 2026-02-23T13:13:38.476 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:13:38.476 INFO:teuthology.orchestra.run.trial068.stdout: "event": "initiated", 2026-02-23T13:13:38.476 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:13:38.409045+0000", 2026-02-23T13:13:38.476 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 0 2026-02-23T13:13:38.476 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:13:38.477 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:13:38.477 INFO:teuthology.orchestra.run.trial068.stdout: "event": "header_read", 2026-02-23T13:13:38.477 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:13:38.409045+0000", 2026-02-23T13:13:38.477 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 0 2026-02-23T13:13:38.477 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:13:38.477 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:13:38.477 INFO:teuthology.orchestra.run.trial068.stdout: "event": "throttled", 2026-02-23T13:13:38.477 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:13:38.409065+0000", 2026-02-23T13:13:38.477 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 2.0548e-05 2026-02-23T13:13:38.477 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:13:38.477 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:13:38.477 INFO:teuthology.orchestra.run.trial068.stdout: "event": "all_read", 2026-02-23T13:13:38.477 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:13:38.409108+0000", 2026-02-23T13:13:38.477 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 4.2429999999999999e-05 2026-02-23T13:13:38.477 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:13:38.477 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:13:38.477 INFO:teuthology.orchestra.run.trial068.stdout: "event": "dispatched", 2026-02-23T13:13:38.477 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:13:38.409111+0000", 2026-02-23T13:13:38.477 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 3.7969999999999998e-06 2026-02-23T13:13:38.477 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:13:38.477 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:13:38.477 INFO:teuthology.orchestra.run.trial068.stdout: "event": "queued_for_pg", 2026-02-23T13:13:38.477 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:13:38.409116+0000", 2026-02-23T13:13:38.477 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 4.9400000000000001e-06 2026-02-23T13:13:38.477 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:13:38.477 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:13:38.477 INFO:teuthology.orchestra.run.trial068.stdout: "event": "reached_pg", 2026-02-23T13:13:38.477 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:13:38.409348+0000", 2026-02-23T13:13:38.477 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 0.00023162500000000001 2026-02-23T13:13:38.477 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:13:38.477 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:13:38.478 INFO:teuthology.orchestra.run.trial068.stdout: "event": "waiting for peered", 2026-02-23T13:13:38.478 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:13:38.409352+0000", 2026-02-23T13:13:38.478 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 3.6370000000000001e-06 2026-02-23T13:13:38.478 INFO:teuthology.orchestra.run.trial068.stdout: } 2026-02-23T13:13:38.478 INFO:teuthology.orchestra.run.trial068.stdout: ] 2026-02-23T13:13:38.478 INFO:teuthology.orchestra.run.trial068.stdout: } 2026-02-23T13:13:38.478 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:13:38.478 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:13:38.478 INFO:teuthology.orchestra.run.trial068.stdout: "description": "osd_op(client.14774.0:96354 5.c 5.5301429c (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e176)", 2026-02-23T13:13:38.478 INFO:teuthology.orchestra.run.trial068.stdout: "initiated_at": "2026-02-23T13:13:38.409127+0000", 2026-02-23T13:13:38.478 INFO:teuthology.orchestra.run.trial068.stdout: "age": 0.036771250999999998, 2026-02-23T13:13:38.478 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 0.037041349000000001, 2026-02-23T13:13:38.478 INFO:teuthology.orchestra.run.trial068.stdout: "continuous": false, 2026-02-23T13:13:38.478 INFO:teuthology.orchestra.run.trial068.stdout: "type_data": { 2026-02-23T13:13:38.478 INFO:teuthology.orchestra.run.trial068.stdout: "flag_point": "waiting for peered", 2026-02-23T13:13:38.478 INFO:teuthology.orchestra.run.trial068.stdout: "client_info": { 2026-02-23T13:13:38.478 INFO:teuthology.orchestra.run.trial068.stdout: "client": "client.14774", 2026-02-23T13:13:38.478 INFO:teuthology.orchestra.run.trial068.stdout: "client_addr": "10.20.193.68:0/3517244806", 2026-02-23T13:13:38.478 INFO:teuthology.orchestra.run.trial068.stdout: "tid": 96354 2026-02-23T13:13:38.478 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:13:38.478 INFO:teuthology.orchestra.run.trial068.stdout: "events": [ 2026-02-23T13:13:38.478 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:13:38.478 INFO:teuthology.orchestra.run.trial068.stdout: "event": "initiated", 2026-02-23T13:13:38.478 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:13:38.409127+0000", 2026-02-23T13:13:38.478 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 0 2026-02-23T13:13:38.478 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:13:38.478 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:13:38.478 INFO:teuthology.orchestra.run.trial068.stdout: "event": "header_read", 2026-02-23T13:13:38.478 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:13:38.409127+0000", 2026-02-23T13:13:38.479 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 0 2026-02-23T13:13:38.479 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:13:38.479 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:13:38.479 INFO:teuthology.orchestra.run.trial068.stdout: "event": "throttled", 2026-02-23T13:13:38.479 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:13:38.409143+0000", 2026-02-23T13:13:38.479 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 1.6030000000000001e-05 2026-02-23T13:13:38.479 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:13:38.479 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:13:38.479 INFO:teuthology.orchestra.run.trial068.stdout: "event": "all_read", 2026-02-23T13:13:38.479 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:13:38.409199+0000", 2026-02-23T13:13:38.479 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 5.5964999999999997e-05 2026-02-23T13:13:38.479 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:13:38.479 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:13:38.479 INFO:teuthology.orchestra.run.trial068.stdout: "event": "dispatched", 2026-02-23T13:13:38.479 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:13:38.409207+0000", 2026-02-23T13:13:38.479 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 7.8250000000000005e-06 2026-02-23T13:13:38.479 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:13:38.479 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:13:38.479 INFO:teuthology.orchestra.run.trial068.stdout: "event": "queued_for_pg", 2026-02-23T13:13:38.479 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:13:38.409209+0000", 2026-02-23T13:13:38.479 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 2.605e-06 2026-02-23T13:13:38.479 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:13:38.479 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:13:38.479 INFO:teuthology.orchestra.run.trial068.stdout: "event": "reached_pg", 2026-02-23T13:13:38.479 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:13:38.409388+0000", 2026-02-23T13:13:38.479 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 0.00017888600000000001 2026-02-23T13:13:38.479 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:13:38.479 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:13:38.479 INFO:teuthology.orchestra.run.trial068.stdout: "event": "waiting for peered", 2026-02-23T13:13:38.479 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:13:38.409394+0000", 2026-02-23T13:13:38.480 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 5.6200000000000004e-06 2026-02-23T13:13:38.480 INFO:teuthology.orchestra.run.trial068.stdout: } 2026-02-23T13:13:38.480 INFO:teuthology.orchestra.run.trial068.stdout: ] 2026-02-23T13:13:38.480 INFO:teuthology.orchestra.run.trial068.stdout: } 2026-02-23T13:13:38.480 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:13:38.480 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:13:38.480 INFO:teuthology.orchestra.run.trial068.stdout: "description": "osd_op(client.14774.0:96355 5.c 5.5301429c (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e176)", 2026-02-23T13:13:38.480 INFO:teuthology.orchestra.run.trial068.stdout: "initiated_at": "2026-02-23T13:13:38.409232+0000", 2026-02-23T13:13:38.480 INFO:teuthology.orchestra.run.trial068.stdout: "age": 0.036666053999999997, 2026-02-23T13:13:38.480 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 0.036951471, 2026-02-23T13:13:38.480 INFO:teuthology.orchestra.run.trial068.stdout: "continuous": false, 2026-02-23T13:13:38.480 INFO:teuthology.orchestra.run.trial068.stdout: "type_data": { 2026-02-23T13:13:38.480 INFO:teuthology.orchestra.run.trial068.stdout: "flag_point": "waiting for peered", 2026-02-23T13:13:38.480 INFO:teuthology.orchestra.run.trial068.stdout: "client_info": { 2026-02-23T13:13:38.480 INFO:teuthology.orchestra.run.trial068.stdout: "client": "client.14774", 2026-02-23T13:13:38.480 INFO:teuthology.orchestra.run.trial068.stdout: "client_addr": "10.20.193.68:0/3517244806", 2026-02-23T13:13:38.480 INFO:teuthology.orchestra.run.trial068.stdout: "tid": 96355 2026-02-23T13:13:38.480 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:13:38.480 INFO:teuthology.orchestra.run.trial068.stdout: "events": [ 2026-02-23T13:13:38.480 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:13:38.480 INFO:teuthology.orchestra.run.trial068.stdout: "event": "initiated", 2026-02-23T13:13:38.480 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:13:38.409232+0000", 2026-02-23T13:13:38.480 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 0 2026-02-23T13:13:38.480 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:13:38.480 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:13:38.480 INFO:teuthology.orchestra.run.trial068.stdout: "event": "header_read", 2026-02-23T13:13:38.480 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:13:38.409232+0000", 2026-02-23T13:13:38.480 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 0 2026-02-23T13:13:38.480 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:13:38.480 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:13:38.481 INFO:teuthology.orchestra.run.trial068.stdout: "event": "throttled", 2026-02-23T13:13:38.481 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:13:38.409291+0000", 2026-02-23T13:13:38.481 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 5.8900999999999997e-05 2026-02-23T13:13:38.481 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:13:38.481 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:13:38.481 INFO:teuthology.orchestra.run.trial068.stdout: "event": "all_read", 2026-02-23T13:13:38.481 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:13:38.409362+0000", 2026-02-23T13:13:38.481 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 7.0593000000000007e-05 2026-02-23T13:13:38.481 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:13:38.481 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:13:38.481 INFO:teuthology.orchestra.run.trial068.stdout: "event": "dispatched", 2026-02-23T13:13:38.481 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:13:38.409367+0000", 2026-02-23T13:13:38.481 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 5.6400000000000002e-06 2026-02-23T13:13:38.481 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:13:38.481 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:13:38.481 INFO:teuthology.orchestra.run.trial068.stdout: "event": "queued_for_pg", 2026-02-23T13:13:38.481 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:13:38.409372+0000", 2026-02-23T13:13:38.481 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 4.1579999999999998e-06 2026-02-23T13:13:38.481 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:13:38.481 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:13:38.481 INFO:teuthology.orchestra.run.trial068.stdout: "event": "reached_pg", 2026-02-23T13:13:38.481 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:13:38.409428+0000", 2026-02-23T13:13:38.481 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 5.6366000000000002e-05 2026-02-23T13:13:38.481 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:13:38.481 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:13:38.481 INFO:teuthology.orchestra.run.trial068.stdout: "event": "waiting for peered", 2026-02-23T13:13:38.481 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:13:38.409433+0000", 2026-02-23T13:13:38.481 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 4.6890000000000003e-06 2026-02-23T13:13:38.481 INFO:teuthology.orchestra.run.trial068.stdout: } 2026-02-23T13:13:38.481 INFO:teuthology.orchestra.run.trial068.stdout: ] 2026-02-23T13:13:38.482 INFO:teuthology.orchestra.run.trial068.stdout: } 2026-02-23T13:13:38.482 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:13:38.482 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:13:38.482 INFO:teuthology.orchestra.run.trial068.stdout: "description": "osd_op(client.14774.0:96356 5.c 5.5301429c (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e176)", 2026-02-23T13:13:38.482 INFO:teuthology.orchestra.run.trial068.stdout: "initiated_at": "2026-02-23T13:13:38.409397+0000", 2026-02-23T13:13:38.482 INFO:teuthology.orchestra.run.trial068.stdout: "age": 0.036501705000000002, 2026-02-23T13:13:38.482 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 0.036804164, 2026-02-23T13:13:38.482 INFO:teuthology.orchestra.run.trial068.stdout: "continuous": false, 2026-02-23T13:13:38.482 INFO:teuthology.orchestra.run.trial068.stdout: "type_data": { 2026-02-23T13:13:38.482 INFO:teuthology.orchestra.run.trial068.stdout: "flag_point": "waiting for peered", 2026-02-23T13:13:38.482 INFO:teuthology.orchestra.run.trial068.stdout: "client_info": { 2026-02-23T13:13:38.482 INFO:teuthology.orchestra.run.trial068.stdout: "client": "client.14774", 2026-02-23T13:13:38.482 INFO:teuthology.orchestra.run.trial068.stdout: "client_addr": "10.20.193.68:0/3517244806", 2026-02-23T13:13:38.482 INFO:teuthology.orchestra.run.trial068.stdout: "tid": 96356 2026-02-23T13:13:38.482 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:13:38.482 INFO:teuthology.orchestra.run.trial068.stdout: "events": [ 2026-02-23T13:13:38.482 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:13:38.482 INFO:teuthology.orchestra.run.trial068.stdout: "event": "initiated", 2026-02-23T13:13:38.482 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:13:38.409397+0000", 2026-02-23T13:13:38.482 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 0 2026-02-23T13:13:38.482 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:13:38.482 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:13:38.482 INFO:teuthology.orchestra.run.trial068.stdout: "event": "header_read", 2026-02-23T13:13:38.482 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:13:38.409397+0000", 2026-02-23T13:13:38.482 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 0 2026-02-23T13:13:38.482 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:13:38.482 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:13:38.482 INFO:teuthology.orchestra.run.trial068.stdout: "event": "throttled", 2026-02-23T13:13:38.482 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:13:38.409426+0000", 2026-02-23T13:13:38.482 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 2.9195000000000001e-05 2026-02-23T13:13:38.482 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:13:38.483 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:13:38.483 INFO:teuthology.orchestra.run.trial068.stdout: "event": "all_read", 2026-02-23T13:13:38.483 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:13:38.409489+0000", 2026-02-23T13:13:38.483 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 6.3639999999999994e-05 2026-02-23T13:13:38.483 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:13:38.483 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:13:38.483 INFO:teuthology.orchestra.run.trial068.stdout: "event": "dispatched", 2026-02-23T13:13:38.483 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:13:38.409504+0000", 2026-02-23T13:13:38.483 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 1.4898e-05 2026-02-23T13:13:38.483 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:13:38.483 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:13:38.483 INFO:teuthology.orchestra.run.trial068.stdout: "event": "queued_for_pg", 2026-02-23T13:13:38.483 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:13:38.409514+0000", 2026-02-23T13:13:38.483 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 9.9580000000000002e-06 2026-02-23T13:13:38.483 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:13:38.483 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:13:38.483 INFO:teuthology.orchestra.run.trial068.stdout: "event": "reached_pg", 2026-02-23T13:13:38.483 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:13:38.409570+0000", 2026-02-23T13:13:38.483 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 5.6045999999999998e-05 2026-02-23T13:13:38.483 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:13:38.483 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:13:38.483 INFO:teuthology.orchestra.run.trial068.stdout: "event": "waiting for peered", 2026-02-23T13:13:38.483 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:13:38.409575+0000", 2026-02-23T13:13:38.483 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 4.2479999999999998e-06 2026-02-23T13:13:38.483 INFO:teuthology.orchestra.run.trial068.stdout: } 2026-02-23T13:13:38.483 INFO:teuthology.orchestra.run.trial068.stdout: ] 2026-02-23T13:13:38.483 INFO:teuthology.orchestra.run.trial068.stdout: } 2026-02-23T13:13:38.483 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:13:38.483 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:13:38.483 INFO:teuthology.orchestra.run.trial068.stdout: "description": "osd_op(client.14774.0:96357 5.c 5.5301429c (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e176)", 2026-02-23T13:13:38.483 INFO:teuthology.orchestra.run.trial068.stdout: "initiated_at": "2026-02-23T13:13:38.409530+0000", 2026-02-23T13:13:38.484 INFO:teuthology.orchestra.run.trial068.stdout: "age": 0.036368124000000002, 2026-02-23T13:13:38.484 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 0.036686202000000001, 2026-02-23T13:13:38.484 INFO:teuthology.orchestra.run.trial068.stdout: "continuous": false, 2026-02-23T13:13:38.484 INFO:teuthology.orchestra.run.trial068.stdout: "type_data": { 2026-02-23T13:13:38.484 INFO:teuthology.orchestra.run.trial068.stdout: "flag_point": "waiting for peered", 2026-02-23T13:13:38.484 INFO:teuthology.orchestra.run.trial068.stdout: "client_info": { 2026-02-23T13:13:38.484 INFO:teuthology.orchestra.run.trial068.stdout: "client": "client.14774", 2026-02-23T13:13:38.484 INFO:teuthology.orchestra.run.trial068.stdout: "client_addr": "10.20.193.68:0/3517244806", 2026-02-23T13:13:38.484 INFO:teuthology.orchestra.run.trial068.stdout: "tid": 96357 2026-02-23T13:13:38.484 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:13:38.484 INFO:teuthology.orchestra.run.trial068.stdout: "events": [ 2026-02-23T13:13:38.484 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:13:38.484 INFO:teuthology.orchestra.run.trial068.stdout: "event": "initiated", 2026-02-23T13:13:38.484 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:13:38.409530+0000", 2026-02-23T13:13:38.484 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 0 2026-02-23T13:13:38.484 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:13:38.484 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:13:38.484 INFO:teuthology.orchestra.run.trial068.stdout: "event": "header_read", 2026-02-23T13:13:38.484 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:13:38.409530+0000", 2026-02-23T13:13:38.484 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 0 2026-02-23T13:13:38.484 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:13:38.484 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:13:38.484 INFO:teuthology.orchestra.run.trial068.stdout: "event": "throttled", 2026-02-23T13:13:38.484 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:13:38.409580+0000", 2026-02-23T13:13:38.484 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 5.0294999999999999e-05 2026-02-23T13:13:38.484 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:13:38.484 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:13:38.484 INFO:teuthology.orchestra.run.trial068.stdout: "event": "all_read", 2026-02-23T13:13:38.484 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:13:38.409632+0000", 2026-02-23T13:13:38.484 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 5.1146000000000001e-05 2026-02-23T13:13:38.485 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:13:38.485 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:13:38.485 INFO:teuthology.orchestra.run.trial068.stdout: "event": "dispatched", 2026-02-23T13:13:38.485 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:13:38.409638+0000", 2026-02-23T13:13:38.485 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 6.3419999999999996e-06 2026-02-23T13:13:38.485 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:13:38.485 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:13:38.485 INFO:teuthology.orchestra.run.trial068.stdout: "event": "queued_for_pg", 2026-02-23T13:13:38.485 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:13:38.409640+0000", 2026-02-23T13:13:38.485 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 2.1739999999999999e-06 2026-02-23T13:13:38.485 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:13:38.485 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:13:38.485 INFO:teuthology.orchestra.run.trial068.stdout: "event": "reached_pg", 2026-02-23T13:13:38.485 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:13:38.409663+0000", 2026-02-23T13:13:38.485 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 2.2972999999999998e-05 2026-02-23T13:13:38.485 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:13:38.485 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:13:38.485 INFO:teuthology.orchestra.run.trial068.stdout: "event": "waiting for peered", 2026-02-23T13:13:38.485 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:13:38.409668+0000", 2026-02-23T13:13:38.485 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 4.9490000000000004e-06 2026-02-23T13:13:38.485 INFO:teuthology.orchestra.run.trial068.stdout: } 2026-02-23T13:13:38.485 INFO:teuthology.orchestra.run.trial068.stdout: ] 2026-02-23T13:13:38.485 INFO:teuthology.orchestra.run.trial068.stdout: } 2026-02-23T13:13:38.485 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:13:38.485 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:13:38.485 INFO:teuthology.orchestra.run.trial068.stdout: "description": "osd_op(client.14774.0:96358 5.c 5.5301429c (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e176)", 2026-02-23T13:13:38.485 INFO:teuthology.orchestra.run.trial068.stdout: "initiated_at": "2026-02-23T13:13:38.409652+0000", 2026-02-23T13:13:38.485 INFO:teuthology.orchestra.run.trial068.stdout: "age": 0.036246275000000001, 2026-02-23T13:13:38.485 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 0.036579320999999998, 2026-02-23T13:13:38.485 INFO:teuthology.orchestra.run.trial068.stdout: "continuous": false, 2026-02-23T13:13:38.485 INFO:teuthology.orchestra.run.trial068.stdout: "type_data": { 2026-02-23T13:13:38.486 INFO:teuthology.orchestra.run.trial068.stdout: "flag_point": "waiting for peered", 2026-02-23T13:13:38.486 INFO:teuthology.orchestra.run.trial068.stdout: "client_info": { 2026-02-23T13:13:38.486 INFO:teuthology.orchestra.run.trial068.stdout: "client": "client.14774", 2026-02-23T13:13:38.486 INFO:teuthology.orchestra.run.trial068.stdout: "client_addr": "10.20.193.68:0/3517244806", 2026-02-23T13:13:38.486 INFO:teuthology.orchestra.run.trial068.stdout: "tid": 96358 2026-02-23T13:13:38.486 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:13:38.486 INFO:teuthology.orchestra.run.trial068.stdout: "events": [ 2026-02-23T13:13:38.486 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:13:38.486 INFO:teuthology.orchestra.run.trial068.stdout: "event": "initiated", 2026-02-23T13:13:38.486 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:13:38.409652+0000", 2026-02-23T13:13:38.486 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 0 2026-02-23T13:13:38.486 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:13:38.486 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:13:38.486 INFO:teuthology.orchestra.run.trial068.stdout: "event": "header_read", 2026-02-23T13:13:38.486 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:13:38.409652+0000", 2026-02-23T13:13:38.486 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 0 2026-02-23T13:13:38.486 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:13:38.486 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:13:38.486 INFO:teuthology.orchestra.run.trial068.stdout: "event": "throttled", 2026-02-23T13:13:38.486 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:13:38.409668+0000", 2026-02-23T13:13:38.486 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 1.5559000000000001e-05 2026-02-23T13:13:38.486 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:13:38.486 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:13:38.486 INFO:teuthology.orchestra.run.trial068.stdout: "event": "all_read", 2026-02-23T13:13:38.486 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:13:38.409719+0000", 2026-02-23T13:13:38.486 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 5.1085999999999998e-05 2026-02-23T13:13:38.486 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:13:38.486 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:13:38.486 INFO:teuthology.orchestra.run.trial068.stdout: "event": "dispatched", 2026-02-23T13:13:38.486 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:13:38.409724+0000", 2026-02-23T13:13:38.486 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 5.4709999999999998e-06 2026-02-23T13:13:38.487 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:13:38.487 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:13:38.487 INFO:teuthology.orchestra.run.trial068.stdout: "event": "queued_for_pg", 2026-02-23T13:13:38.487 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:13:38.409727+0000", 2026-02-23T13:13:38.487 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 2.9050000000000001e-06 2026-02-23T13:13:38.487 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:13:38.487 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:13:38.487 INFO:teuthology.orchestra.run.trial068.stdout: "event": "reached_pg", 2026-02-23T13:13:38.487 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:13:38.409814+0000", 2026-02-23T13:13:38.487 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 8.7124000000000001e-05 2026-02-23T13:13:38.487 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:13:38.487 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:13:38.487 INFO:teuthology.orchestra.run.trial068.stdout: "event": "waiting for peered", 2026-02-23T13:13:38.487 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:13:38.409831+0000", 2026-02-23T13:13:38.487 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 1.6381000000000001e-05 2026-02-23T13:13:38.487 INFO:teuthology.orchestra.run.trial068.stdout: } 2026-02-23T13:13:38.487 INFO:teuthology.orchestra.run.trial068.stdout: ] 2026-02-23T13:13:38.487 INFO:teuthology.orchestra.run.trial068.stdout: } 2026-02-23T13:13:38.487 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:13:38.487 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:13:38.487 INFO:teuthology.orchestra.run.trial068.stdout: "description": "osd_op(client.14774.0:96359 5.c 5.5301429c (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e176)", 2026-02-23T13:13:38.487 INFO:teuthology.orchestra.run.trial068.stdout: "initiated_at": "2026-02-23T13:13:38.409757+0000", 2026-02-23T13:13:38.487 INFO:teuthology.orchestra.run.trial068.stdout: "age": 0.036141767999999998, 2026-02-23T13:13:38.487 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 0.036489752, 2026-02-23T13:13:38.487 INFO:teuthology.orchestra.run.trial068.stdout: "continuous": false, 2026-02-23T13:13:38.487 INFO:teuthology.orchestra.run.trial068.stdout: "type_data": { 2026-02-23T13:13:38.487 INFO:teuthology.orchestra.run.trial068.stdout: "flag_point": "waiting for peered", 2026-02-23T13:13:38.487 INFO:teuthology.orchestra.run.trial068.stdout: "client_info": { 2026-02-23T13:13:38.487 INFO:teuthology.orchestra.run.trial068.stdout: "client": "client.14774", 2026-02-23T13:13:38.487 INFO:teuthology.orchestra.run.trial068.stdout: "client_addr": "10.20.193.68:0/3517244806", 2026-02-23T13:13:38.488 INFO:teuthology.orchestra.run.trial068.stdout: "tid": 96359 2026-02-23T13:13:38.488 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:13:38.488 INFO:teuthology.orchestra.run.trial068.stdout: "events": [ 2026-02-23T13:13:38.488 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:13:38 trial068 ceph-mon[38805]: from='osd.1 [v2:10.20.193.68:6810/3818277133,v1:10.20.193.68:6811/3818277133]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2026-02-23T13:13:38.488 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:13:38 trial068 ceph-mon[38805]: osdmap e175: 4 total, 3 up, 4 in 2026-02-23T13:13:38.488 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:13:38 trial068 ceph-mon[38805]: from='osd.1 [v2:10.20.193.68:6810/3818277133,v1:10.20.193.68:6811/3818277133]' entity='osd.1' cmd={"prefix": "osd crush create-or-move", "id": 1, "weight":0.6836, "args": ["host=trial068", "root=default"]} : dispatch 2026-02-23T13:13:38.488 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:13:38 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2122184750' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:13:38.488 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:13:38.488 INFO:teuthology.orchestra.run.trial068.stdout: "event": "initiated", 2026-02-23T13:13:38.488 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:13:38.409757+0000", 2026-02-23T13:13:38.488 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 0 2026-02-23T13:13:38.488 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:13:38.488 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:13:38.488 INFO:teuthology.orchestra.run.trial068.stdout: "event": "header_read", 2026-02-23T13:13:38.489 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:13:38.409757+0000", 2026-02-23T13:13:38.489 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 0 2026-02-23T13:13:38.489 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:13:38.489 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:13:38.489 INFO:teuthology.orchestra.run.trial068.stdout: "event": "throttled", 2026-02-23T13:13:38.489 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:13:38.409799+0000", 2026-02-23T13:13:38.489 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 4.2840000000000003e-05 2026-02-23T13:13:38.489 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:13:38.489 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:13:38.489 INFO:teuthology.orchestra.run.trial068.stdout: "event": "all_read", 2026-02-23T13:13:38.489 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:13:38.409853+0000", 2026-02-23T13:13:38.489 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 5.3449999999999998e-05 2026-02-23T13:13:38.489 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:13:38.489 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:13:38.489 INFO:teuthology.orchestra.run.trial068.stdout: "event": "dispatched", 2026-02-23T13:13:38.489 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:13:38.409861+0000", 2026-02-23T13:13:38.489 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 8.2460000000000003e-06 2026-02-23T13:13:38.489 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:13:38.489 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:13:38.489 INFO:teuthology.orchestra.run.trial068.stdout: "event": "queued_for_pg", 2026-02-23T13:13:38.489 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:13:38.409865+0000", 2026-02-23T13:13:38.489 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 3.4970000000000001e-06 2026-02-23T13:13:38.489 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:13:38.489 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:13:38.489 INFO:teuthology.orchestra.run.trial068.stdout: "event": "reached_pg", 2026-02-23T13:13:38.489 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:13:38.409891+0000", 2026-02-23T13:13:38.489 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 2.6438999999999999e-05 2026-02-23T13:13:38.489 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:13:38.489 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:13:38.489 INFO:teuthology.orchestra.run.trial068.stdout: "event": "waiting for peered", 2026-02-23T13:13:38.489 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:13:38.409894+0000", 2026-02-23T13:13:38.490 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 2.976e-06 2026-02-23T13:13:38.490 INFO:teuthology.orchestra.run.trial068.stdout: } 2026-02-23T13:13:38.490 INFO:teuthology.orchestra.run.trial068.stdout: ] 2026-02-23T13:13:38.490 INFO:teuthology.orchestra.run.trial068.stdout: } 2026-02-23T13:13:38.490 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:13:38.490 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:13:38.490 INFO:teuthology.orchestra.run.trial068.stdout: "description": "osd_op(client.14774.0:96360 5.c 5.5301429c (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e176)", 2026-02-23T13:13:38.490 INFO:teuthology.orchestra.run.trial068.stdout: "initiated_at": "2026-02-23T13:13:38.409887+0000", 2026-02-23T13:13:38.490 INFO:teuthology.orchestra.run.trial068.stdout: "age": 0.036011703999999999, 2026-02-23T13:13:38.490 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 0.036374396000000003, 2026-02-23T13:13:38.490 INFO:teuthology.orchestra.run.trial068.stdout: "continuous": false, 2026-02-23T13:13:38.490 INFO:teuthology.orchestra.run.trial068.stdout: "type_data": { 2026-02-23T13:13:38.490 INFO:teuthology.orchestra.run.trial068.stdout: "flag_point": "waiting for peered", 2026-02-23T13:13:38.490 INFO:teuthology.orchestra.run.trial068.stdout: "client_info": { 2026-02-23T13:13:38.490 INFO:teuthology.orchestra.run.trial068.stdout: "client": "client.14774", 2026-02-23T13:13:38.490 INFO:teuthology.orchestra.run.trial068.stdout: "client_addr": "10.20.193.68:0/3517244806", 2026-02-23T13:13:38.490 INFO:teuthology.orchestra.run.trial068.stdout: "tid": 96360 2026-02-23T13:13:38.490 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:13:38.490 INFO:teuthology.orchestra.run.trial068.stdout: "events": [ 2026-02-23T13:13:38.490 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:13:38.490 INFO:teuthology.orchestra.run.trial068.stdout: "event": "initiated", 2026-02-23T13:13:38.490 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:13:38.409887+0000", 2026-02-23T13:13:38.490 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 0 2026-02-23T13:13:38.490 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:13:38.490 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:13:38.490 INFO:teuthology.orchestra.run.trial068.stdout: "event": "header_read", 2026-02-23T13:13:38.490 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:13:38.409887+0000", 2026-02-23T13:13:38.490 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 0 2026-02-23T13:13:38.490 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:13:38.490 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:13:38.490 INFO:teuthology.orchestra.run.trial068.stdout: "event": "throttled", 2026-02-23T13:13:38.491 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:13:38.409899+0000", 2026-02-23T13:13:38.491 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 1.2614e-05 2026-02-23T13:13:38.491 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:13:38.491 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:13:38.491 INFO:teuthology.orchestra.run.trial068.stdout: "event": "all_read", 2026-02-23T13:13:38.491 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:13:38.409926+0000", 2026-02-23T13:13:38.491 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 2.6339e-05 2026-02-23T13:13:38.491 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:13:38.491 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:13:38.491 INFO:teuthology.orchestra.run.trial068.stdout: "event": "dispatched", 2026-02-23T13:13:38.491 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:13:38.409929+0000", 2026-02-23T13:13:38.491 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 3.4869999999999998e-06 2026-02-23T13:13:38.491 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:13:38.491 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:13:38.491 INFO:teuthology.orchestra.run.trial068.stdout: "event": "queued_for_pg", 2026-02-23T13:13:38.491 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:13:38.409930+0000", 2026-02-23T13:13:38.491 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 1.2920000000000001e-06 2026-02-23T13:13:38.491 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:13:38.491 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:13:38.491 INFO:teuthology.orchestra.run.trial068.stdout: "event": "reached_pg", 2026-02-23T13:13:38.491 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:13:38.409945+0000", 2026-02-23T13:13:38.491 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 1.4657999999999999e-05 2026-02-23T13:13:38.491 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:13:38.491 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:13:38.491 INFO:teuthology.orchestra.run.trial068.stdout: "event": "waiting for peered", 2026-02-23T13:13:38.491 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:13:38.409947+0000", 2026-02-23T13:13:38.491 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 2.0329999999999998e-06 2026-02-23T13:13:38.491 INFO:teuthology.orchestra.run.trial068.stdout: } 2026-02-23T13:13:38.491 INFO:teuthology.orchestra.run.trial068.stdout: ] 2026-02-23T13:13:38.491 INFO:teuthology.orchestra.run.trial068.stdout: } 2026-02-23T13:13:38.491 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:13:38.491 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:13:38.491 INFO:teuthology.orchestra.run.trial068.stdout: "description": "osd_op(client.14774.0:96385 5.c 5.6fd305cc (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e176)", 2026-02-23T13:13:38.492 INFO:teuthology.orchestra.run.trial068.stdout: "initiated_at": "2026-02-23T13:13:38.411572+0000", 2026-02-23T13:13:38.492 INFO:teuthology.orchestra.run.trial068.stdout: "age": 0.034326247999999997, 2026-02-23T13:13:38.492 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 0.034704098000000003, 2026-02-23T13:13:38.492 INFO:teuthology.orchestra.run.trial068.stdout: "continuous": false, 2026-02-23T13:13:38.492 INFO:teuthology.orchestra.run.trial068.stdout: "type_data": { 2026-02-23T13:13:38.492 INFO:teuthology.orchestra.run.trial068.stdout: "flag_point": "waiting for peered", 2026-02-23T13:13:38.492 INFO:teuthology.orchestra.run.trial068.stdout: "client_info": { 2026-02-23T13:13:38.492 INFO:teuthology.orchestra.run.trial068.stdout: "client": "client.14774", 2026-02-23T13:13:38.492 INFO:teuthology.orchestra.run.trial068.stdout: "client_addr": "10.20.193.68:0/3517244806", 2026-02-23T13:13:38.492 INFO:teuthology.orchestra.run.trial068.stdout: "tid": 96385 2026-02-23T13:13:38.492 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:13:38.492 INFO:teuthology.orchestra.run.trial068.stdout: "events": [ 2026-02-23T13:13:38.492 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:13:38.492 INFO:teuthology.orchestra.run.trial068.stdout: "event": "initiated", 2026-02-23T13:13:38.492 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:13:38.411572+0000", 2026-02-23T13:13:38.492 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 0 2026-02-23T13:13:38.492 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:13:38.492 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:13:38.492 INFO:teuthology.orchestra.run.trial068.stdout: "event": "header_read", 2026-02-23T13:13:38.492 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:13:38.411572+0000", 2026-02-23T13:13:38.492 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 0 2026-02-23T13:13:38.492 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:13:38.492 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:13:38.492 INFO:teuthology.orchestra.run.trial068.stdout: "event": "throttled", 2026-02-23T13:13:38.492 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:13:38.411579+0000", 2026-02-23T13:13:38.492 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 6.5529999999999998e-06 2026-02-23T13:13:38.492 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:13:38.492 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:13:38.492 INFO:teuthology.orchestra.run.trial068.stdout: "event": "all_read", 2026-02-23T13:13:38.492 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:13:38.411598+0000", 2026-02-23T13:13:38.492 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 1.9485999999999999e-05 2026-02-23T13:13:38.492 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:13:38.493 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:13:38.493 INFO:teuthology.orchestra.run.trial068.stdout: "event": "dispatched", 2026-02-23T13:13:38.493 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:13:38.411603+0000", 2026-02-23T13:13:38.493 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 4.3980000000000001e-06 2026-02-23T13:13:38.493 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:13:38.493 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:13:38.493 INFO:teuthology.orchestra.run.trial068.stdout: "event": "queued_for_pg", 2026-02-23T13:13:38.493 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:13:38.411606+0000", 2026-02-23T13:13:38.493 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 3.6969999999999999e-06 2026-02-23T13:13:38.493 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:13:38.493 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:13:38.493 INFO:teuthology.orchestra.run.trial068.stdout: "event": "reached_pg", 2026-02-23T13:13:38.493 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:13:38.411631+0000", 2026-02-23T13:13:38.493 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 2.5117999999999999e-05 2026-02-23T13:13:38.493 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:13:38.493 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:13:38.493 INFO:teuthology.orchestra.run.trial068.stdout: "event": "waiting for peered", 2026-02-23T13:13:38.493 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:13:38.411634+0000", 2026-02-23T13:13:38.493 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 2.8449999999999999e-06 2026-02-23T13:13:38.493 INFO:teuthology.orchestra.run.trial068.stdout: } 2026-02-23T13:13:38.493 INFO:teuthology.orchestra.run.trial068.stdout: ] 2026-02-23T13:13:38.493 INFO:teuthology.orchestra.run.trial068.stdout: } 2026-02-23T13:13:38.493 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:13:38.493 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:13:38.493 INFO:teuthology.orchestra.run.trial068.stdout: "description": "osd_op(client.14774.0:96386 5.c 5.6fd305cc (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e176)", 2026-02-23T13:13:38.493 INFO:teuthology.orchestra.run.trial068.stdout: "initiated_at": "2026-02-23T13:13:38.411852+0000", 2026-02-23T13:13:38.493 INFO:teuthology.orchestra.run.trial068.stdout: "age": 0.034046402000000003, 2026-02-23T13:13:38.493 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 0.034438839999999998, 2026-02-23T13:13:38.493 INFO:teuthology.orchestra.run.trial068.stdout: "continuous": false, 2026-02-23T13:13:38.493 INFO:teuthology.orchestra.run.trial068.stdout: "type_data": { 2026-02-23T13:13:38.493 INFO:teuthology.orchestra.run.trial068.stdout: "flag_point": "waiting for peered", 2026-02-23T13:13:38.493 INFO:teuthology.orchestra.run.trial068.stdout: "client_info": { 2026-02-23T13:13:38.493 INFO:teuthology.orchestra.run.trial068.stdout: "client": "client.14774", 2026-02-23T13:13:38.494 INFO:teuthology.orchestra.run.trial068.stdout: "client_addr": "10.20.193.68:0/3517244806", 2026-02-23T13:13:38.494 INFO:teuthology.orchestra.run.trial068.stdout: "tid": 96386 2026-02-23T13:13:38.494 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:13:38.494 INFO:teuthology.orchestra.run.trial068.stdout: "events": [ 2026-02-23T13:13:38.494 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:13:38.494 INFO:teuthology.orchestra.run.trial068.stdout: "event": "initiated", 2026-02-23T13:13:38.494 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:13:38.411852+0000", 2026-02-23T13:13:38.494 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 0 2026-02-23T13:13:38.494 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:13:38.494 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:13:38.494 INFO:teuthology.orchestra.run.trial068.stdout: "event": "header_read", 2026-02-23T13:13:38.494 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:13:38.411852+0000", 2026-02-23T13:13:38.494 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 0 2026-02-23T13:13:38.494 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:13:38.494 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:13:38.494 INFO:teuthology.orchestra.run.trial068.stdout: "event": "throttled", 2026-02-23T13:13:38.494 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:13:38.411858+0000", 2026-02-23T13:13:38.494 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 6.562e-06 2026-02-23T13:13:38.494 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:13:38.494 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:13:38.494 INFO:teuthology.orchestra.run.trial068.stdout: "event": "all_read", 2026-02-23T13:13:38.494 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:13:38.411880+0000", 2026-02-23T13:13:38.494 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 2.2002e-05 2026-02-23T13:13:38.494 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:13:38.494 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:13:38.494 INFO:teuthology.orchestra.run.trial068.stdout: "event": "dispatched", 2026-02-23T13:13:38.494 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:13:38.411884+0000", 2026-02-23T13:13:38.494 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 3.326e-06 2026-02-23T13:13:38.494 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:13:38.494 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:13:38.494 INFO:teuthology.orchestra.run.trial068.stdout: "event": "queued_for_pg", 2026-02-23T13:13:38.494 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:13:38.411886+0000", 2026-02-23T13:13:38.494 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 1.8029999999999999e-06 2026-02-23T13:13:38.494 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:13:38.495 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:13:38.495 INFO:teuthology.orchestra.run.trial068.stdout: "event": "reached_pg", 2026-02-23T13:13:38.495 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:13:38.411904+0000", 2026-02-23T13:13:38.495 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 1.7934000000000001e-05 2026-02-23T13:13:38.495 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:13:38.495 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:13:38.495 INFO:teuthology.orchestra.run.trial068.stdout: "event": "waiting for peered", 2026-02-23T13:13:38.495 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:13:38.411906+0000", 2026-02-23T13:13:38.495 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 2.5150000000000001e-06 2026-02-23T13:13:38.495 INFO:teuthology.orchestra.run.trial068.stdout: } 2026-02-23T13:13:38.495 INFO:teuthology.orchestra.run.trial068.stdout: ] 2026-02-23T13:13:38.495 INFO:teuthology.orchestra.run.trial068.stdout: } 2026-02-23T13:13:38.495 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:13:38.495 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:13:38.495 INFO:teuthology.orchestra.run.trial068.stdout: "description": "osd_op(client.14774.0:96387 5.c 5.6fd305cc (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e176)", 2026-02-23T13:13:38.495 INFO:teuthology.orchestra.run.trial068.stdout: "initiated_at": "2026-02-23T13:13:38.413173+0000", 2026-02-23T13:13:38.495 INFO:teuthology.orchestra.run.trial068.stdout: "age": 0.032725771000000001, 2026-02-23T13:13:38.495 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 0.033133045999999999, 2026-02-23T13:13:38.495 INFO:teuthology.orchestra.run.trial068.stdout: "continuous": false, 2026-02-23T13:13:38.495 INFO:teuthology.orchestra.run.trial068.stdout: "type_data": { 2026-02-23T13:13:38.495 INFO:teuthology.orchestra.run.trial068.stdout: "flag_point": "waiting for peered", 2026-02-23T13:13:38.495 INFO:teuthology.orchestra.run.trial068.stdout: "client_info": { 2026-02-23T13:13:38.495 INFO:teuthology.orchestra.run.trial068.stdout: "client": "client.14774", 2026-02-23T13:13:38.495 INFO:teuthology.orchestra.run.trial068.stdout: "client_addr": "10.20.193.68:0/3517244806", 2026-02-23T13:13:38.495 INFO:teuthology.orchestra.run.trial068.stdout: "tid": 96387 2026-02-23T13:13:38.495 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:13:38.495 INFO:teuthology.orchestra.run.trial068.stdout: "events": [ 2026-02-23T13:13:38.495 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:13:38.495 INFO:teuthology.orchestra.run.trial068.stdout: "event": "initiated", 2026-02-23T13:13:38.495 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:13:38.413173+0000", 2026-02-23T13:13:38.495 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 0 2026-02-23T13:13:38.495 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:13:38.495 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:13:38.495 INFO:teuthology.orchestra.run.trial068.stdout: "event": "header_read", 2026-02-23T13:13:38.496 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:13:38.413173+0000", 2026-02-23T13:13:38.496 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 0 2026-02-23T13:13:38.496 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:13:38.496 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:13:38.496 INFO:teuthology.orchestra.run.trial068.stdout: "event": "throttled", 2026-02-23T13:13:38.496 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:13:38.413182+0000", 2026-02-23T13:13:38.496 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 9.0070000000000003e-06 2026-02-23T13:13:38.496 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:13:38.496 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:13:38.496 INFO:teuthology.orchestra.run.trial068.stdout: "event": "all_read", 2026-02-23T13:13:38.496 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:13:38.413210+0000", 2026-02-23T13:13:38.496 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 2.8102999999999998e-05 2026-02-23T13:13:38.496 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:13:38.496 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:13:38.496 INFO:teuthology.orchestra.run.trial068.stdout: "event": "dispatched", 2026-02-23T13:13:38.496 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:13:38.413214+0000", 2026-02-23T13:13:38.496 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 4.318e-06 2026-02-23T13:13:38.496 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:13:38.496 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:13:38.496 INFO:teuthology.orchestra.run.trial068.stdout: "event": "queued_for_pg", 2026-02-23T13:13:38.496 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:13:38.413216+0000", 2026-02-23T13:13:38.496 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 2.2740000000000002e-06 2026-02-23T13:13:38.496 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:13:38.496 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:13:38.496 INFO:teuthology.orchestra.run.trial068.stdout: "event": "reached_pg", 2026-02-23T13:13:38.496 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:13:38.413246+0000", 2026-02-23T13:13:38.496 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 2.9825999999999999e-05 2026-02-23T13:13:38.496 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:13:38.496 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:13:38.496 INFO:teuthology.orchestra.run.trial068.stdout: "event": "waiting for peered", 2026-02-23T13:13:38.496 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:13:38.413250+0000", 2026-02-23T13:13:38.496 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 3.9079999999999997e-06 2026-02-23T13:13:38.496 INFO:teuthology.orchestra.run.trial068.stdout: } 2026-02-23T13:13:38.496 INFO:teuthology.orchestra.run.trial068.stdout: ] 2026-02-23T13:13:38.496 INFO:teuthology.orchestra.run.trial068.stdout: } 2026-02-23T13:13:38.497 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:13:38.497 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:13:38.497 INFO:teuthology.orchestra.run.trial068.stdout: "description": "osd_op(client.14774.0:96388 5.c 5.6fd305cc (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e176)", 2026-02-23T13:13:38.497 INFO:teuthology.orchestra.run.trial068.stdout: "initiated_at": "2026-02-23T13:13:38.413235+0000", 2026-02-23T13:13:38.497 INFO:teuthology.orchestra.run.trial068.stdout: "age": 0.032663123000000002, 2026-02-23T13:13:38.497 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 0.033085306000000002, 2026-02-23T13:13:38.497 INFO:teuthology.orchestra.run.trial068.stdout: "continuous": false, 2026-02-23T13:13:38.497 INFO:teuthology.orchestra.run.trial068.stdout: "type_data": { 2026-02-23T13:13:38.497 INFO:teuthology.orchestra.run.trial068.stdout: "flag_point": "waiting for peered", 2026-02-23T13:13:38.497 INFO:teuthology.orchestra.run.trial068.stdout: "client_info": { 2026-02-23T13:13:38.497 INFO:teuthology.orchestra.run.trial068.stdout: "client": "client.14774", 2026-02-23T13:13:38.497 INFO:teuthology.orchestra.run.trial068.stdout: "client_addr": "10.20.193.68:0/3517244806", 2026-02-23T13:13:38.497 INFO:teuthology.orchestra.run.trial068.stdout: "tid": 96388 2026-02-23T13:13:38.497 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:13:38.497 INFO:teuthology.orchestra.run.trial068.stdout: "events": [ 2026-02-23T13:13:38.497 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:13:38.497 INFO:teuthology.orchestra.run.trial068.stdout: "event": "initiated", 2026-02-23T13:13:38.497 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:13:38.413235+0000", 2026-02-23T13:13:38.497 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 0 2026-02-23T13:13:38.497 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:13:38.497 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:13:38.497 INFO:teuthology.orchestra.run.trial068.stdout: "event": "header_read", 2026-02-23T13:13:38.497 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:13:38.413235+0000", 2026-02-23T13:13:38.497 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 0 2026-02-23T13:13:38.497 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:13:38.497 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:13:38.497 INFO:teuthology.orchestra.run.trial068.stdout: "event": "throttled", 2026-02-23T13:13:38.497 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:13:38.413247+0000", 2026-02-23T13:13:38.497 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 1.2272999999999999e-05 2026-02-23T13:13:38.497 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:13:38.497 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:13:38.497 INFO:teuthology.orchestra.run.trial068.stdout: "event": "all_read", 2026-02-23T13:13:38.497 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:13:38.413276+0000", 2026-02-23T13:13:38.497 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 2.8793999999999999e-05 2026-02-23T13:13:38.497 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:13:38.497 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:13:38.498 INFO:teuthology.orchestra.run.trial068.stdout: "event": "dispatched", 2026-02-23T13:13:38.498 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:13:38.413279+0000", 2026-02-23T13:13:38.498 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 3.0460000000000001e-06 2026-02-23T13:13:38.498 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:13:38.498 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:13:38.498 INFO:teuthology.orchestra.run.trial068.stdout: "event": "queued_for_pg", 2026-02-23T13:13:38.498 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:13:38.413281+0000", 2026-02-23T13:13:38.498 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 1.6029999999999999e-06 2026-02-23T13:13:38.498 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:13:38.498 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:13:38.498 INFO:teuthology.orchestra.run.trial068.stdout: "event": "reached_pg", 2026-02-23T13:13:38.498 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:13:38.413310+0000", 2026-02-23T13:13:38.498 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 2.8724000000000001e-05 2026-02-23T13:13:38.498 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:13:38.498 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:13:38.498 INFO:teuthology.orchestra.run.trial068.stdout: "event": "waiting for peered", 2026-02-23T13:13:38.498 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:13:38.413315+0000", 2026-02-23T13:13:38.498 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 5.0189999999999997e-06 2026-02-23T13:13:38.498 INFO:teuthology.orchestra.run.trial068.stdout: } 2026-02-23T13:13:38.498 INFO:teuthology.orchestra.run.trial068.stdout: ] 2026-02-23T13:13:38.498 INFO:teuthology.orchestra.run.trial068.stdout: } 2026-02-23T13:13:38.498 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:13:38.498 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:13:38.498 INFO:teuthology.orchestra.run.trial068.stdout: "description": "osd_op(client.14774.0:96389 5.c 5.6fd305cc (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e176)", 2026-02-23T13:13:38.498 INFO:teuthology.orchestra.run.trial068.stdout: "initiated_at": "2026-02-23T13:13:38.413291+0000", 2026-02-23T13:13:38.498 INFO:teuthology.orchestra.run.trial068.stdout: "age": 0.032606857000000003, 2026-02-23T13:13:38.498 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 0.033043747999999998, 2026-02-23T13:13:38.498 INFO:teuthology.orchestra.run.trial068.stdout: "continuous": false, 2026-02-23T13:13:38.498 INFO:teuthology.orchestra.run.trial068.stdout: "type_data": { 2026-02-23T13:13:38.498 INFO:teuthology.orchestra.run.trial068.stdout: "flag_point": "waiting for peered", 2026-02-23T13:13:38.498 INFO:teuthology.orchestra.run.trial068.stdout: "client_info": { 2026-02-23T13:13:38.498 INFO:teuthology.orchestra.run.trial068.stdout: "client": "client.14774", 2026-02-23T13:13:38.498 INFO:teuthology.orchestra.run.trial068.stdout: "client_addr": "10.20.193.68:0/3517244806", 2026-02-23T13:13:38.498 INFO:teuthology.orchestra.run.trial068.stdout: "tid": 96389 2026-02-23T13:13:38.498 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:13:38.498 INFO:teuthology.orchestra.run.trial068.stdout: "events": [ 2026-02-23T13:13:38.498 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:13:38.498 INFO:teuthology.orchestra.run.trial068.stdout: "event": "initiated", 2026-02-23T13:13:38.499 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:13:38.413291+0000", 2026-02-23T13:13:38.499 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 0 2026-02-23T13:13:38.499 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:13:38.499 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:13:38.499 INFO:teuthology.orchestra.run.trial068.stdout: "event": "header_read", 2026-02-23T13:13:38.499 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:13:38.413291+0000", 2026-02-23T13:13:38.499 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 0 2026-02-23T13:13:38.499 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:13:38.499 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:13:38.499 INFO:teuthology.orchestra.run.trial068.stdout: "event": "throttled", 2026-02-23T13:13:38.499 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:13:38.413309+0000", 2026-02-23T13:13:38.499 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 1.7742999999999999e-05 2026-02-23T13:13:38.499 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:13:38.499 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:13:38.499 INFO:teuthology.orchestra.run.trial068.stdout: "event": "all_read", 2026-02-23T13:13:38.499 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:13:38.413343+0000", 2026-02-23T13:13:38.499 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 3.3924000000000002e-05 2026-02-23T13:13:38.499 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:13:38.499 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:13:38.499 INFO:teuthology.orchestra.run.trial068.stdout: "event": "dispatched", 2026-02-23T13:13:38.499 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:13:38.413349+0000", 2026-02-23T13:13:38.499 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 5.801e-06 2026-02-23T13:13:38.499 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:13:38.499 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:13:38.499 INFO:teuthology.orchestra.run.trial068.stdout: "event": "queued_for_pg", 2026-02-23T13:13:38.499 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:13:38.413350+0000", 2026-02-23T13:13:38.499 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 1.372e-06 2026-02-23T13:13:38.499 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:13:38.499 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:13:38.499 INFO:teuthology.orchestra.run.trial068.stdout: "event": "reached_pg", 2026-02-23T13:13:38.499 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:13:38.413372+0000", 2026-02-23T13:13:38.499 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 2.2062e-05 2026-02-23T13:13:38.499 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:13:38.499 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:13:38.499 INFO:teuthology.orchestra.run.trial068.stdout: "event": "waiting for peered", 2026-02-23T13:13:38.499 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:13:38.413375+0000", 2026-02-23T13:13:38.499 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 2.3939999999999999e-06 2026-02-23T13:13:38.500 INFO:teuthology.orchestra.run.trial068.stdout: } 2026-02-23T13:13:38.500 INFO:teuthology.orchestra.run.trial068.stdout: ] 2026-02-23T13:13:38.500 INFO:teuthology.orchestra.run.trial068.stdout: } 2026-02-23T13:13:38.500 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:13:38.500 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:13:38.500 INFO:teuthology.orchestra.run.trial068.stdout: "description": "osd_op(client.14774.0:96390 5.c 5.6fd305cc (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e176)", 2026-02-23T13:13:38.500 INFO:teuthology.orchestra.run.trial068.stdout: "initiated_at": "2026-02-23T13:13:38.413360+0000", 2026-02-23T13:13:38.500 INFO:teuthology.orchestra.run.trial068.stdout: "age": 0.032537818000000003, 2026-02-23T13:13:38.500 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 0.032990147999999997, 2026-02-23T13:13:38.500 INFO:teuthology.orchestra.run.trial068.stdout: "continuous": false, 2026-02-23T13:13:38.500 INFO:teuthology.orchestra.run.trial068.stdout: "type_data": { 2026-02-23T13:13:38.500 INFO:teuthology.orchestra.run.trial068.stdout: "flag_point": "waiting for peered", 2026-02-23T13:13:38.500 INFO:teuthology.orchestra.run.trial068.stdout: "client_info": { 2026-02-23T13:13:38.500 INFO:teuthology.orchestra.run.trial068.stdout: "client": "client.14774", 2026-02-23T13:13:38.500 INFO:teuthology.orchestra.run.trial068.stdout: "client_addr": "10.20.193.68:0/3517244806", 2026-02-23T13:13:38.500 INFO:teuthology.orchestra.run.trial068.stdout: "tid": 96390 2026-02-23T13:13:38.500 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:13:38.500 INFO:teuthology.orchestra.run.trial068.stdout: "events": [ 2026-02-23T13:13:38.500 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:13:38.500 INFO:teuthology.orchestra.run.trial068.stdout: "event": "initiated", 2026-02-23T13:13:38.500 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:13:38.413360+0000", 2026-02-23T13:13:38.500 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 0 2026-02-23T13:13:38.500 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:13:38.500 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:13:38.500 INFO:teuthology.orchestra.run.trial068.stdout: "event": "header_read", 2026-02-23T13:13:38.500 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:13:38.413360+0000", 2026-02-23T13:13:38.500 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 0 2026-02-23T13:13:38.500 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:13:38.500 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:13:38.500 INFO:teuthology.orchestra.run.trial068.stdout: "event": "throttled", 2026-02-23T13:13:38.500 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:13:38.413368+0000", 2026-02-23T13:13:38.500 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 7.6249999999999998e-06 2026-02-23T13:13:38.500 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:13:38.500 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:13:38.500 INFO:teuthology.orchestra.run.trial068.stdout: "event": "all_read", 2026-02-23T13:13:38.500 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:13:38.413404+0000", 2026-02-23T13:13:38.500 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 3.5976999999999998e-05 2026-02-23T13:13:38.501 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:13:38.501 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:13:38.501 INFO:teuthology.orchestra.run.trial068.stdout: "event": "dispatched", 2026-02-23T13:13:38.501 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:13:38.413408+0000", 2026-02-23T13:13:38.501 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 3.4869999999999998e-06 2026-02-23T13:13:38.501 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:13:38.501 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:13:38.501 INFO:teuthology.orchestra.run.trial068.stdout: "event": "queued_for_pg", 2026-02-23T13:13:38.501 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:13:38.413409+0000", 2026-02-23T13:13:38.501 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 1.2419999999999999e-06 2026-02-23T13:13:38.501 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:13:38.501 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:13:38.501 INFO:teuthology.orchestra.run.trial068.stdout: "event": "reached_pg", 2026-02-23T13:13:38.501 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:13:38.413428+0000", 2026-02-23T13:13:38.501 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 1.9046e-05 2026-02-23T13:13:38.501 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:13:38.501 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:13:38.501 INFO:teuthology.orchestra.run.trial068.stdout: "event": "waiting for peered", 2026-02-23T13:13:38.501 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:13:38.413430+0000", 2026-02-23T13:13:38.501 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 2.3240000000000001e-06 2026-02-23T13:13:38.501 INFO:teuthology.orchestra.run.trial068.stdout: } 2026-02-23T13:13:38.501 INFO:teuthology.orchestra.run.trial068.stdout: ] 2026-02-23T13:13:38.501 INFO:teuthology.orchestra.run.trial068.stdout: } 2026-02-23T13:13:38.501 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:13:38.501 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:13:38.501 INFO:teuthology.orchestra.run.trial068.stdout: "description": "osd_op(client.14774.0:96391 5.c 5.6fd305cc (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e176)", 2026-02-23T13:13:38.501 INFO:teuthology.orchestra.run.trial068.stdout: "initiated_at": "2026-02-23T13:13:38.413515+0000", 2026-02-23T13:13:38.501 INFO:teuthology.orchestra.run.trial068.stdout: "age": 0.032383317000000002, 2026-02-23T13:13:38.501 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 0.032850474999999997, 2026-02-23T13:13:38.501 INFO:teuthology.orchestra.run.trial068.stdout: "continuous": false, 2026-02-23T13:13:38.501 INFO:teuthology.orchestra.run.trial068.stdout: "type_data": { 2026-02-23T13:13:38.501 INFO:teuthology.orchestra.run.trial068.stdout: "flag_point": "waiting for peered", 2026-02-23T13:13:38.501 INFO:teuthology.orchestra.run.trial068.stdout: "client_info": { 2026-02-23T13:13:38.501 INFO:teuthology.orchestra.run.trial068.stdout: "client": "client.14774", 2026-02-23T13:13:38.501 INFO:teuthology.orchestra.run.trial068.stdout: "client_addr": "10.20.193.68:0/3517244806", 2026-02-23T13:13:38.501 INFO:teuthology.orchestra.run.trial068.stdout: "tid": 96391 2026-02-23T13:13:38.501 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:13:38.502 INFO:teuthology.orchestra.run.trial068.stdout: "events": [ 2026-02-23T13:13:38.502 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:13:38.502 INFO:teuthology.orchestra.run.trial068.stdout: "event": "initiated", 2026-02-23T13:13:38.502 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:13:38.413515+0000", 2026-02-23T13:13:38.502 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 0 2026-02-23T13:13:38.502 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:13:38.502 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:13:38.502 INFO:teuthology.orchestra.run.trial068.stdout: "event": "header_read", 2026-02-23T13:13:38.502 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:13:38.413515+0000", 2026-02-23T13:13:38.502 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 0 2026-02-23T13:13:38.502 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:13:38.502 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:13:38.502 INFO:teuthology.orchestra.run.trial068.stdout: "event": "throttled", 2026-02-23T13:13:38.502 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:13:38.413523+0000", 2026-02-23T13:13:38.502 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 7.8639999999999996e-06 2026-02-23T13:13:38.502 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:13:38.502 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:13:38.502 INFO:teuthology.orchestra.run.trial068.stdout: "event": "all_read", 2026-02-23T13:13:38.502 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:13:38.413544+0000", 2026-02-23T13:13:38.502 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 2.1591e-05 2026-02-23T13:13:38.502 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:13:38.502 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:13:38.502 INFO:teuthology.orchestra.run.trial068.stdout: "event": "dispatched", 2026-02-23T13:13:38.502 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:13:38.413548+0000", 2026-02-23T13:13:38.502 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 3.5769999999999998e-06 2026-02-23T13:13:38.502 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:13:38.502 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:13:38.502 INFO:teuthology.orchestra.run.trial068.stdout: "event": "queued_for_pg", 2026-02-23T13:13:38.502 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:13:38.413549+0000", 2026-02-23T13:13:38.502 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 1.142e-06 2026-02-23T13:13:38.502 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:13:38.502 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:13:38.502 INFO:teuthology.orchestra.run.trial068.stdout: "event": "reached_pg", 2026-02-23T13:13:38.502 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:13:38.413567+0000", 2026-02-23T13:13:38.502 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 1.7663e-05 2026-02-23T13:13:38.502 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:13:38.502 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:13:38.502 INFO:teuthology.orchestra.run.trial068.stdout: "event": "waiting for peered", 2026-02-23T13:13:38.502 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:13:38.413569+0000", 2026-02-23T13:13:38.503 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 2.4650000000000001e-06 2026-02-23T13:13:38.503 INFO:teuthology.orchestra.run.trial068.stdout: } 2026-02-23T13:13:38.503 INFO:teuthology.orchestra.run.trial068.stdout: ] 2026-02-23T13:13:38.503 INFO:teuthology.orchestra.run.trial068.stdout: } 2026-02-23T13:13:38.503 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:13:38.503 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:13:38.503 INFO:teuthology.orchestra.run.trial068.stdout: "description": "osd_op(client.14774.0:96392 5.c 5.6fd305cc (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e176)", 2026-02-23T13:13:38.503 INFO:teuthology.orchestra.run.trial068.stdout: "initiated_at": "2026-02-23T13:13:38.414443+0000", 2026-02-23T13:13:38.503 INFO:teuthology.orchestra.run.trial068.stdout: "age": 0.031454923000000003, 2026-02-23T13:13:38.503 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 0.03193754, 2026-02-23T13:13:38.503 INFO:teuthology.orchestra.run.trial068.stdout: "continuous": false, 2026-02-23T13:13:38.503 INFO:teuthology.orchestra.run.trial068.stdout: "type_data": { 2026-02-23T13:13:38.503 INFO:teuthology.orchestra.run.trial068.stdout: "flag_point": "waiting for peered", 2026-02-23T13:13:38.503 INFO:teuthology.orchestra.run.trial068.stdout: "client_info": { 2026-02-23T13:13:38.503 INFO:teuthology.orchestra.run.trial068.stdout: "client": "client.14774", 2026-02-23T13:13:38.503 INFO:teuthology.orchestra.run.trial068.stdout: "client_addr": "10.20.193.68:0/3517244806", 2026-02-23T13:13:38.503 INFO:teuthology.orchestra.run.trial068.stdout: "tid": 96392 2026-02-23T13:13:38.503 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:13:38.503 INFO:teuthology.orchestra.run.trial068.stdout: "events": [ 2026-02-23T13:13:38.503 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:13:38.503 INFO:teuthology.orchestra.run.trial068.stdout: "event": "initiated", 2026-02-23T13:13:38.503 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:13:38.414443+0000", 2026-02-23T13:13:38.503 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 0 2026-02-23T13:13:38.503 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:13:38.503 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:13:38.503 INFO:teuthology.orchestra.run.trial068.stdout: "event": "header_read", 2026-02-23T13:13:38.503 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:13:38.414443+0000", 2026-02-23T13:13:38.503 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 0 2026-02-23T13:13:38.503 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:13:38.503 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:13:38.503 INFO:teuthology.orchestra.run.trial068.stdout: "event": "throttled", 2026-02-23T13:13:38.503 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:13:38.414452+0000", 2026-02-23T13:13:38.503 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 8.5560000000000008e-06 2026-02-23T13:13:38.503 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:13:38.503 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:13:38.503 INFO:teuthology.orchestra.run.trial068.stdout: "event": "all_read", 2026-02-23T13:13:38.503 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:13:38.414478+0000", 2026-02-23T13:13:38.503 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 2.5837999999999999e-05 2026-02-23T13:13:38.503 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:13:38.504 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:13:38.504 INFO:teuthology.orchestra.run.trial068.stdout: "event": "dispatched", 2026-02-23T13:13:38.504 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:13:38.414483+0000", 2026-02-23T13:13:38.504 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 5.1699999999999996e-06 2026-02-23T13:13:38.504 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:13:38.504 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:13:38.504 INFO:teuthology.orchestra.run.trial068.stdout: "event": "queued_for_pg", 2026-02-23T13:13:38.504 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:13:38.414485+0000", 2026-02-23T13:13:38.504 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 1.883e-06 2026-02-23T13:13:38.504 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:13:38.504 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:13:38.504 INFO:teuthology.orchestra.run.trial068.stdout: "event": "reached_pg", 2026-02-23T13:13:38.504 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:13:38.414505+0000", 2026-02-23T13:13:38.504 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 2.0177999999999999e-05 2026-02-23T13:13:38.504 INFO:teuthology.orchestra.run.trial068.stdout: }, 2026-02-23T13:13:38.504 INFO:teuthology.orchestra.run.trial068.stdout: { 2026-02-23T13:13:38.504 INFO:teuthology.orchestra.run.trial068.stdout: "event": "waiting for peered", 2026-02-23T13:13:38.504 INFO:teuthology.orchestra.run.trial068.stdout: "time": "2026-02-23T13:13:38.414507+0000", 2026-02-23T13:13:38.504 INFO:teuthology.orchestra.run.trial068.stdout: "duration": 2.3350000000000001e-06 2026-02-23T13:13:38.504 INFO:teuthology.orchestra.run.trial068.stdout: } 2026-02-23T13:13:38.504 INFO:teuthology.orchestra.run.trial068.stdout: ] 2026-02-23T13:13:38.504 INFO:teuthology.orchestra.run.trial068.stdout: } 2026-02-23T13:13:38.504 INFO:teuthology.orchestra.run.trial068.stdout: } 2026-02-23T13:13:38.504 INFO:teuthology.orchestra.run.trial068.stdout: ], 2026-02-23T13:13:38.504 INFO:teuthology.orchestra.run.trial068.stdout: "num_ops": 27 2026-02-23T13:13:38.504 INFO:teuthology.orchestra.run.trial068.stdout:} 2026-02-23T13:13:39.099 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 13 128 96449 96321 57.8811 15.8984 0.0115076 0.0163458 2026-02-23T13:13:39.789 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:13:39.814 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:13:39 trial068 ceph-mon[38805]: pgmap v337: 17 pgs: 10 active+undersized+degraded, 7 active+clean; 586 MiB data, 3.2 GiB used, 2.7 TiB / 2.7 TiB avail; 61 MiB/s wr, 7.76k op/s; 5744/28125 objects degraded (20.423%) 2026-02-23T13:13:39.814 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:13:39 trial068 ceph-mon[38805]: Health check cleared: OSD_DOWN (was: 1 osds down) 2026-02-23T13:13:39.815 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:13:39 trial068 ceph-mon[38805]: osd.1 [v2:10.20.193.68:6810/3818277133,v1:10.20.193.68:6811/3818277133] boot 2026-02-23T13:13:39.815 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:13:39 trial068 ceph-mon[38805]: osdmap e176: 4 total, 4 up, 4 in 2026-02-23T13:13:39.815 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:13:39 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2026-02-23T13:13:39.815 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:13:39 trial068 ceph-mon[38805]: osdmap e177: 4 total, 4 up, 4 in 2026-02-23T13:13:39.906 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:13:40.005 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:13:40.004+0000 7fe53a046640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:13:40.006 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:13:40.004+0000 7fe53a046640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:13:40.021 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:13:40.021 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:13:40.050 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:13:40.050 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_delete_sleep = '' osd_delete_sleep_hdd = '' osd_delete_sleep_hybrid = '' osd_delete_sleep_ssd = '' osd_enable_op_tracker = '' osd_recovery_max_active = '' osd_recovery_max_active_hdd = '' osd_recovery_max_active_ssd = '' osd_recovery_sleep = '' osd_recovery_sleep_degraded = '' osd_recovery_sleep_degraded_hdd = '' osd_recovery_sleep_degraded_hybrid = '' osd_recovery_sleep_degraded_ssd = '' osd_recovery_sleep_hdd = '' osd_recovery_sleep_hybrid = '' osd_recovery_sleep_ssd = '' osd_scrub_sleep = '' osd_snap_trim_sleep = '' osd_snap_trim_sleep_hdd = '' osd_snap_trim_sleep_hybrid = '' osd_snap_trim_sleep_ssd = '' 2026-02-23T13:13:40.061 INFO:teuthology.orchestra.run.trial068.stdout:osd.2: {} 2026-02-23T13:13:40.061 INFO:teuthology.orchestra.run.trial068.stderr:osd.2: osd_enable_op_tracker = '' 2026-02-23T13:13:40.072 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:13:40.072 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:13:40.099 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 14 128 96449 96321 53.7467 0 - 0.0163458 2026-02-23T13:13:40.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:13:40 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T13:13:40.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:13:40 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T13:13:40.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:13:40 trial068 ceph-mon[38805]: osdmap e178: 4 total, 4 up, 4 in 2026-02-23T13:13:40.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:13:40 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1805292975' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:13:41.099 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 15 128 98401 98273 51.1801 7.625 0.0437396 0.0195129 2026-02-23T13:13:41.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:13:41 trial068 ceph-mon[38805]: pgmap v341: 17 pgs: 10 active+undersized+degraded, 7 active+clean; 658 MiB data, 3.4 GiB used, 2.7 TiB / 2.7 TiB avail; 41 MiB/s wr, 5.28k op/s; 6587/31566 objects degraded (20.867%) 2026-02-23T13:13:42.099 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 16 128 103349 103221 50.3972 38.6562 0.0138462 0.0198216 2026-02-23T13:13:42.107 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:13:42.272 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:13:42.652 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:13:42 trial068 ceph-mon[38805]: Health check update: Degraded data redundancy: 7399/36741 objects degraded (20.138%), 10 pgs degraded (PG_DEGRADED) 2026-02-23T13:13:42.660 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:13:42.660 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:13:42.660 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:13:42.610+0000 7f4be675e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:13:42.660 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:13:42.610+0000 7f4be675e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:13:42.660 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:13:42.660 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:13:42.666 INFO:teuthology.orchestra.run.trial068.stdout:osd.2: {} 2026-02-23T13:13:42.666 INFO:teuthology.orchestra.run.trial068.stderr:osd.2: osd_enable_op_tracker = '' 2026-02-23T13:13:42.682 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:13:42.682 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:13:43.099 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 17 128 107105 106977 49.1586 29.3438 0.038533 0.02032 2026-02-23T13:13:43.506 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [3, 0, 2, 1] 2026-02-23T13:13:43.506 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2026-02-23T13:13:43.506 INFO:tasks.thrashosds.thrasher:Killing osd 2, live_osds are [3, 0, 2, 1] 2026-02-23T13:13:43.506 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2026-02-23T13:13:43.507 DEBUG:teuthology.orchestra.run.trial185:> sudo systemctl stop ceph-5f823084-10b8-11f1-be80-d404e6e7d460@osd.2 2026-02-23T13:13:43.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:13:43 trial068 ceph-mon[38805]: pgmap v342: 17 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 4 activating+undersized+degraded+remapped, 3 active+undersized+degraded, 7 active+clean; 766 MiB data, 3.7 GiB used, 2.7 TiB / 2.7 TiB avail; 37 MiB/s wr, 4.69k op/s; 7399/36741 objects degraded (20.138%); 722 KiB/s, 11 objects/s recovering 2026-02-23T13:13:43.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:13:43 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1663305531' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:13:43.857 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:13:43 trial185 systemd[1]: Stopping Ceph osd.2 for 5f823084-10b8-11f1-be80-d404e6e7d460... 2026-02-23T13:13:43.857 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:13:43 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-2[76501]: 2026-02-23T13:13:43.729+0000 7f3f010d2640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --osd-objectstore=bluestore (PID: 1) UID: 0 2026-02-23T13:13:43.857 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:13:43 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-2[76501]: 2026-02-23T13:13:43.729+0000 7f3f010d2640 -1 osd.2 179 *** Got signal Terminated *** 2026-02-23T13:13:43.857 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:13:43 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-2[76501]: 2026-02-23T13:13:43.729+0000 7f3f010d2640 -1 osd.2 179 *** Immediate shutdown (osd_fast_shutdown=true) *** 2026-02-23T13:13:44.099 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 18 128 110108 109980 47.7309 23.4609 0.00849252 0.0204977 2026-02-23T13:13:44.270 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:13:43 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:13:43.972Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:13:44.271 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:13:44 trial185 podman[77342]: 2026-02-23 13:13:44.135514071 +0000 UTC m=+0.524282431 container died 1b2663ad5b4f5fe41e99da7641a26011278c89ab49abbf0d9def63f68ec27ced (image=quay.ceph.io/ceph-ci/ceph@sha256:49621faaa845c39b2aae72e4fa24123ee9b2ac3fb8ae834753907068ee2972ea, name=ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-2, CEPH_SHA1=e371433fcd1390a0b82cfb18effc422c1f5d8c1d, FROM_IMAGE=quay.io/centos/centos:stream9, org.label-schema.build-date=20260216, CEPH_GIT_REPO=https://github.com/ceph/ceph-ci.git, org.opencontainers.image.authors=Ceph Release Team , io.buildah.version=1.41.3, OSD_FLAVOR=default, ceph=True, org.label-schema.license=GPLv2, org.opencontainers.image.documentation=https://docs.ceph.com/, GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfsganesha-5/, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, CEPH_REF=wip-shraddhaag-enable-cephadm-tests) 2026-02-23T13:13:44.271 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:13:44 trial185 podman[77342]: 2026-02-23 13:13:44.145523842 +0000 UTC m=+0.534292212 container remove 1b2663ad5b4f5fe41e99da7641a26011278c89ab49abbf0d9def63f68ec27ced (image=quay.ceph.io/ceph-ci/ceph@sha256:49621faaa845c39b2aae72e4fa24123ee9b2ac3fb8ae834753907068ee2972ea, name=ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-2, io.buildah.version=1.41.3, ceph=True, CEPH_SHA1=e371433fcd1390a0b82cfb18effc422c1f5d8c1d, org.label-schema.license=GPLv2, org.label-schema.build-date=20260216, FROM_IMAGE=quay.io/centos/centos:stream9, org.opencontainers.image.documentation=https://docs.ceph.com/, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfsganesha-5/, org.label-schema.vendor=CentOS, org.opencontainers.image.authors=Ceph Release Team , CEPH_GIT_REPO=https://github.com/ceph/ceph-ci.git, CEPH_REF=wip-shraddhaag-enable-cephadm-tests, OSD_FLAVOR=default) 2026-02-23T13:13:44.271 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:13:44 trial185 bash[77342]: ceph-5f823084-10b8-11f1-be80-d404e6e7d460-osd-2 2026-02-23T13:13:44.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:13:43 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:13:43] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:13:44.478 DEBUG:teuthology.orchestra.run.trial185:> sudo pkill -f 'journalctl -f -n 0 -u ceph-5f823084-10b8-11f1-be80-d404e6e7d460@osd.2.service' 2026-02-23T13:13:44.516 DEBUG:teuthology.orchestra.run:got remote process result: None 2026-02-23T13:13:44.516 INFO:tasks.cephadm.osd.2:Stopped osd.2 2026-02-23T13:13:44.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:13:44 trial068 ceph-mon[38805]: osdmap e179: 4 total, 4 up, 4 in 2026-02-23T13:13:44.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:13:44 trial068 ceph-mon[38805]: osd.2 marked itself down and dead 2026-02-23T13:13:44.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:13:44 trial068 ceph-mon[38805]: Health check failed: 1 osds down (OSD_DOWN) 2026-02-23T13:13:44.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:13:44 trial068 ceph-mon[38805]: osdmap e180: 4 total, 3 up, 4 in 2026-02-23T13:13:45.099 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 19 128 110299 110171 45.2973 1.49219 0.00680246 0.0215819 2026-02-23T13:13:45.358 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:13:45.478 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:13:45.593 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:13:45.591+0000 7fadf5eb7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:13:45.593 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:13:45.591+0000 7fadf5eb7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:13:45.606 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:13:45.606 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:13:45.617 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:13:45.617 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:13:45.618 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:13:45.629 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:13:45.630 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:13:45.632 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:13:45 trial068 ceph-mon[38805]: pgmap v345: 17 pgs: 1 stale+active+clean, 1 stale+active+undersized+remapped, 2 stale+active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 4 activating+undersized+degraded+remapped, 6 active+clean; 790 MiB data, 3.9 GiB used, 2.7 TiB / 2.7 TiB avail; 25 MiB/s wr, 3.20k op/s; 6677/37890 objects degraded (17.622%); 9.3 MiB/s, 148 objects/s recovering 2026-02-23T13:13:45.632 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:13:45 trial068 ceph-mon[38805]: osdmap e181: 4 total, 3 up, 4 in 2026-02-23T13:13:45.661 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:13:45.661 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:13:46.099 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T13:13:46.099317+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0228145 2026-02-23T13:13:46.100 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T13:13:46.100 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 20 128 110775 110647 43.2184 3.71875 0.056394 0.0228145 2026-02-23T13:13:46.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:13:46 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1312416057' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:13:46.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:13:46 trial068 ceph-mon[38805]: osdmap e182: 4 total, 3 up, 4 in 2026-02-23T13:13:47.099 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 21 128 111193 111065 41.3159 3.26562 0.0438139 0.0239117 2026-02-23T13:13:47.662 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:13:47.744 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:13:47 trial068 ceph-mon[38805]: pgmap v348: 17 pgs: 1 stale+active+clean, 2 peering, 1 stale+active+undersized+remapped, 1 active+undersized+remapped, 2 stale+active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+undersized+degraded, 3 active+clean; 836 MiB data, 4.0 GiB used, 2.7 TiB / 2.7 TiB avail; 18 MiB/s wr, 2.24k op/s; 6604/40104 objects degraded (16.467%); 42 MiB/s, 679 objects/s recovering 2026-02-23T13:13:47.745 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:13:47 trial068 ceph-mon[38805]: osdmap e183: 4 total, 3 up, 4 in 2026-02-23T13:13:47.781 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:13:47.887 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:13:47.885+0000 7fe1bd535640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:13:47.888 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:13:47.886+0000 7fe1bd535640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:13:47.900 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:13:47.900 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:13:47.911 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:13:47.912 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:13:47.912 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:13:47.922 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:13:47.923 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:13:47.954 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:13:47.954 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:13:48.099 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 22 128 112458 112330 39.887 9.88281 0.0157263 0.0243859 2026-02-23T13:13:48.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:13:48 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1286290371' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:13:48.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:13:48 trial068 ceph-mon[38805]: osdmap e184: 4 total, 3 up, 4 in 2026-02-23T13:13:49.099 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 23 128 117717 117589 39.939 41.0859 0.0081235 0.0246391 2026-02-23T13:13:49.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:13:49 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mon-a[38796]: 2026-02-23T13:13:49.008+0000 7f083cc96640 -1 log_channel(cluster) log [ERR] : Health check failed: Possible data damage: 1 pg recovery_unfound (PG_DAMAGED) 2026-02-23T13:13:49.517 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [2] live_osds: [3, 0, 1] 2026-02-23T13:13:49.518 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2026-02-23T13:13:49.518 INFO:tasks.thrashosds.thrasher:Reviving osd 2 2026-02-23T13:13:49.518 INFO:tasks.cephadm.osd.2:Restarting osd.2 (starting--it wasn't running)... 2026-02-23T13:13:49.518 DEBUG:teuthology.orchestra.run.trial185:osd.2> sudo journalctl -f -n 0 -u ceph-5f823084-10b8-11f1-be80-d404e6e7d460@osd.2.service 2026-02-23T13:13:49.521 DEBUG:teuthology.orchestra.run.trial185:> sudo systemctl start ceph-5f823084-10b8-11f1-be80-d404e6e7d460@osd.2 2026-02-23T13:13:49.590 INFO:teuthology.orchestra.run.trial185.stderr:Job for ceph-5f823084-10b8-11f1-be80-d404e6e7d460@osd.2.service failed. 2026-02-23T13:13:49.590 INFO:teuthology.orchestra.run.trial185.stderr:See "systemctl status ceph-5f823084-10b8-11f1-be80-d404e6e7d460@osd.2.service" and "journalctl -xeu ceph-5f823084-10b8-11f1-be80-d404e6e7d460@osd.2.service" for details. 2026-02-23T13:13:49.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-23T13:13:49.593 INFO:tasks.thrashosds.thrasher:Traceback (most recent call last): File "/home/teuthworker/src/github.com_shraddhaag_ceph_a78040ddef3ba5a489697ca2fe95df59a7d8bd08/qa/tasks/ceph_manager.py", line 192, in wrapper return func(self) ^^^^^^^^^^ File "/home/teuthworker/src/github.com_shraddhaag_ceph_a78040ddef3ba5a489697ca2fe95df59a7d8bd08/qa/tasks/ceph_manager.py", line 1513, in _do_thrash self.choose_action()() File "/home/teuthworker/src/github.com_shraddhaag_ceph_a78040ddef3ba5a489697ca2fe95df59a7d8bd08/qa/tasks/ceph_manager.py", line 562, in revive_osd self.ceph_manager.revive_osd( File "/home/teuthworker/src/github.com_shraddhaag_ceph_a78040ddef3ba5a489697ca2fe95df59a7d8bd08/qa/tasks/ceph_manager.py", line 3229, in revive_osd self.ctx.daemons.get_daemon('osd', osd, self.cluster).restart() File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/teuthology/orchestra/daemon/cephadmunit.py", line 95, in restart self.remote.sh(self.start_cmd) File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/teuthology/orchestra/remote.py", line 97, in sh proc = self.run(**kwargs) ^^^^^^^^^^^^^^^^^^ File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/teuthology/orchestra/remote.py", line 596, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/teuthology/orchestra/run.py", line 461, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on trial185 with status 1: 'sudo systemctl start ceph-5f823084-10b8-11f1-be80-d404e6e7d460@osd.2' 2026-02-23T13:13:49.593 ERROR:tasks.thrashosds.thrasher:exception: Traceback (most recent call last): File "/home/teuthworker/src/github.com_shraddhaag_ceph_a78040ddef3ba5a489697ca2fe95df59a7d8bd08/qa/tasks/ceph_manager.py", line 1384, in do_thrash self._do_thrash() File "/home/teuthworker/src/github.com_shraddhaag_ceph_a78040ddef3ba5a489697ca2fe95df59a7d8bd08/qa/tasks/ceph_manager.py", line 192, in wrapper return func(self) ^^^^^^^^^^ File "/home/teuthworker/src/github.com_shraddhaag_ceph_a78040ddef3ba5a489697ca2fe95df59a7d8bd08/qa/tasks/ceph_manager.py", line 1513, in _do_thrash self.choose_action()() File "/home/teuthworker/src/github.com_shraddhaag_ceph_a78040ddef3ba5a489697ca2fe95df59a7d8bd08/qa/tasks/ceph_manager.py", line 562, in revive_osd self.ceph_manager.revive_osd( File "/home/teuthworker/src/github.com_shraddhaag_ceph_a78040ddef3ba5a489697ca2fe95df59a7d8bd08/qa/tasks/ceph_manager.py", line 3229, in revive_osd self.ctx.daemons.get_daemon('osd', osd, self.cluster).restart() File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/teuthology/orchestra/daemon/cephadmunit.py", line 95, in restart self.remote.sh(self.start_cmd) File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/teuthology/orchestra/remote.py", line 97, in sh proc = self.run(**kwargs) ^^^^^^^^^^^^^^^^^^ File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/teuthology/orchestra/remote.py", line 596, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/teuthology/orchestra/run.py", line 461, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on trial185 with status 1: 'sudo systemctl start ceph-5f823084-10b8-11f1-be80-d404e6e7d460@osd.2' 2026-02-23T13:13:49.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:13:49 trial068 ceph-mon[38805]: pgmap v351: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 2 active+recovering+undersized+degraded, 4 peering, 1 active+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 8 active+undersized+degraded; 866 MiB data, 4.5 GiB used, 2.7 TiB / 2.7 TiB avail; 19 MiB/s wr, 2.44k op/s; 9984/41553 objects degraded (24.027%); 1/13851 objects unfound (0.007%); 74 MiB/s, 1.19k objects/s recovering 2026-02-23T13:13:49.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:13:49 trial068 ceph-mon[38805]: Health check update: Degraded data redundancy: 6604/40104 objects degraded (16.467%), 9 pgs degraded (PG_DEGRADED) 2026-02-23T13:13:49.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:13:49 trial068 ceph-mon[38805]: Health check failed: 1/13851 objects unfound (0.007%) (OBJECT_UNFOUND) 2026-02-23T13:13:49.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:13:49 trial068 ceph-mon[38805]: Health check failed: Possible data damage: 1 pg recovery_unfound (PG_DAMAGED) 2026-02-23T13:13:49.856 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:13:49 trial185 systemd[1]: ceph-5f823084-10b8-11f1-be80-d404e6e7d460@osd.2.service: Start request repeated too quickly. 2026-02-23T13:13:49.856 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:13:49 trial185 systemd[1]: ceph-5f823084-10b8-11f1-be80-d404e6e7d460@osd.2.service: Failed with result 'start-limit-hit'. 2026-02-23T13:13:49.856 INFO:journalctl@ceph.osd.2.trial185.stdout:Feb 23 13:13:49 trial185 systemd[1]: Failed to start Ceph osd.2 for 5f823084-10b8-11f1-be80-d404e6e7d460. 2026-02-23T13:13:49.955 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:13:50.100 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 24 128 122984 122856 39.9893 41.1484 0.00475776 0.0245513 2026-02-23T13:13:50.102 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:13:50.238 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:13:50.236+0000 7f889ad09640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:13:50.238 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:13:50.236+0000 7f889ad09640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:13:50.257 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:13:50.257 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:13:50.272 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:13:50.272 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:13:50.272 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:13:50.286 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:13:50.286 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:13:50.327 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:13:50.328 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:13:50.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:13:50 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1693543147' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:13:51.100 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 25 128 127721 127593 39.8699 37.0078 0.0130936 0.0245806 2026-02-23T13:13:51.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:13:51 trial068 ceph-mon[38805]: pgmap v352: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 2 active+recovering+undersized+degraded, 4 peering, 1 active+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 8 active+undersized+degraded; 866 MiB data, 4.5 GiB used, 2.7 TiB / 2.7 TiB avail; 14 MiB/s wr, 1.85k op/s; 9984/41553 objects degraded (24.027%); 1/13851 objects unfound (0.007%); 56 MiB/s, 899 objects/s recovering 2026-02-23T13:13:52.100 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 26 128 132441 132313 39.7546 36.875 0.0206651 0.0246143 2026-02-23T13:13:52.329 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:13:52.481 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:13:52.598 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:13:52.596+0000 7fa0821ff640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:13:52.598 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:13:52.597+0000 7fa0821ff640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:13:52.612 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:13:52.612 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:13:52.628 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:13:52.628 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:13:52.628 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:13:52.643 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:13:52.643 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:13:52.683 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:13:52.683 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:13:53.100 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 27 128 136906 136778 39.5741 34.8828 0.0626251 0.0246848 2026-02-23T13:13:53.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:13:53 trial068 ceph-mon[38805]: pgmap v353: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 2 active+recovering+undersized+degraded, 5 active+clean, 9 active+undersized+degraded; 974 MiB data, 4.7 GiB used, 2.7 TiB / 2.7 TiB avail; 22 MiB/s wr, 2.81k op/s; 11478/46725 objects degraded (24.565%); 1/15575 objects unfound (0.006%); 38 MiB/s, 609 objects/s recovering 2026-02-23T13:13:53.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:13:53 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/819904438' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:13:54.100 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 28 128 141377 141249 39.4081 34.9297 0.0176787 0.0247578 2026-02-23T13:13:54.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:13:53 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:13:53] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:13:54.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:13:53 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:13:53.972Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:13:54.685 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:13:54.710 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:13:54 trial068 ceph-mon[38805]: Health check update: Degraded data redundancy: 11478/46725 objects degraded (24.565%), 12 pgs degraded (PG_DEGRADED) 2026-02-23T13:13:54.829 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:13:54.954 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:13:54.952+0000 7fdd2cc1f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:13:54.955 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:13:54.952+0000 7fdd2cc1f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:13:54.972 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:13:54.973 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:13:54.988 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:13:54.988 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:13:54.988 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:13:55.002 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:13:55.003 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:13:55.045 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:13:55.046 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:13:55.100 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 29 128 145775 145647 39.2339 34.3594 0.0156884 0.0248294 2026-02-23T13:13:55.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:13:55 trial068 ceph-mon[38805]: pgmap v354: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 1008 MiB data, 4.9 GiB used, 2.7 TiB / 2.7 TiB avail; 21 MiB/s wr, 2.75k op/s; 11168/48369 objects degraded (23.089%); 1/16123 objects unfound (0.006%); 37 MiB/s, 585 objects/s recovering 2026-02-23T13:13:55.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:13:55 trial068 ceph-mon[38805]: Health check update: 1/16123 objects unfound (0.006%) (OBJECT_UNFOUND) 2026-02-23T13:13:55.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:13:55 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2656337028' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:13:56.100 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 30 128 149989 149861 39.0235 32.9219 0.0144889 0.024932 2026-02-23T13:13:57.047 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:13:57.100 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 31 128 154000 153872 38.7754 31.3359 0.031291 0.0250622 2026-02-23T13:13:57.193 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:13:57.307 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:13:57.305+0000 7f6d8a889640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:13:57.307 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:13:57.306+0000 7f6d8a889640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:13:57.322 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:13:57.322 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:13:57.335 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:13:57.335 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:13:57.335 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:13:57.349 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:13:57.349 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:13:57.388 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:13:57.389 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:13:57.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:13:57 trial068 ceph-mon[38805]: pgmap v355: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 1.1 GiB data, 5.1 GiB used, 2.7 TiB / 2.7 TiB avail; 24 MiB/s wr, 3.10k op/s; 11871/52086 objects degraded (22.791%); 1/17362 objects unfound (0.006%); 17 MiB/s, 276 objects/s recovering 2026-02-23T13:13:57.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:13:57 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3728037645' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:13:58.100 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 32 128 157782 157654 38.4869 29.5469 0.0324188 0.0252188 2026-02-23T13:13:59.100 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 33 128 161662 161534 38.2392 30.3125 0.0107078 0.0253592 2026-02-23T13:13:59.390 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:13:59.537 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:13:59.668 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:13:59.666+0000 7f4d98222640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:13:59.668 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:13:59.666+0000 7f4d98222640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:13:59.668 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:13:59 trial068 ceph-mon[38805]: pgmap v356: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 1.2 GiB data, 5.4 GiB used, 2.7 TiB / 2.7 TiB avail; 31 MiB/s wr, 3.92k op/s; 13033/56673 objects degraded (22.997%); 1/18891 objects unfound (0.005%); 12 MiB/s, 187 objects/s recovering 2026-02-23T13:13:59.669 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:13:59 trial068 ceph-mon[38805]: Health check update: Degraded data redundancy: 13033/56673 objects degraded (22.997%), 12 pgs degraded (PG_DEGRADED) 2026-02-23T13:13:59.684 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:13:59.684 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:13:59.698 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:13:59.699 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:13:59.699 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:13:59.713 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:13:59.713 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:13:59.753 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:13:59.754 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:14:00.100 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 34 128 165325 165197 37.9561 28.6172 0.0709656 0.0255139 2026-02-23T13:14:00.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:14:00 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2644279845' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:14:01.100 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 35 128 169153 169025 37.726 29.9062 0.0133997 0.0256579 2026-02-23T13:14:01.755 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:14:01.782 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:14:01 trial068 ceph-mon[38805]: pgmap v357: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 1.2 GiB data, 5.4 GiB used, 2.7 TiB / 2.7 TiB avail; 26 MiB/s wr, 3.36k op/s; 13033/56673 objects degraded (22.997%); 1/18891 objects unfound (0.005%); 10 MiB/s, 160 objects/s recovering 2026-02-23T13:14:01.900 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:14:02.018 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:14:02.015+0000 7f9ff5f3e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:14:02.018 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:14:02.016+0000 7f9ff5f3e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:14:02.033 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:14:02.033 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:14:02.049 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:14:02.049 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:14:02.049 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:14:02.062 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:14:02.062 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:14:02.100 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 36 128 172839 172711 37.4779 28.7969 0.0538107 0.0257969 2026-02-23T13:14:02.339 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:14:02.340 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:14:02.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:14:02 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/4010440204' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:14:03.100 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 37 127 176427 176300 37.2228 28.0391 0.103068 0.0259469 2026-02-23T13:14:03.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:14:03 trial068 ceph-mon[38805]: pgmap v358: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 1.3 GiB data, 5.8 GiB used, 2.7 TiB / 2.7 TiB avail; 36 MiB/s wr, 4.66k op/s; 14222/62529 objects degraded (22.745%); 1/20843 objects unfound (0.005%); 10 MiB/s, 160 objects/s recovering 2026-02-23T13:14:03.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:14:03 trial068 ceph-mon[38805]: Health check update: 1/20843 objects unfound (0.005%) (OBJECT_UNFOUND) 2026-02-23T13:14:04.100 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 38 128 180150 180022 37.0084 29.0781 0.0696014 0.0260859 2026-02-23T13:14:04.264 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:14:03 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:14:03.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:14:04.341 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:14:04.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:14:03 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:14:03] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:14:04.481 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:14:04.612 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:14:04.610+0000 7f7cf0c38640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:14:04.613 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:14:04.610+0000 7f7cf0c38640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:14:04.629 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:14:04.629 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:14:04.630 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:14:04 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T13:14:04.630 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:14:04 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T13:14:04.630 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:14:04 trial068 ceph-mon[38805]: Health check update: Degraded data redundancy: 14222/62529 objects degraded (22.745%), 12 pgs degraded (PG_DEGRADED) 2026-02-23T13:14:04.642 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:14:04.643 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:14:04.643 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:14:04.655 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:14:04.656 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:14:04.692 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:14:04.693 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:14:05.100 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 39 128 183913 183785 36.8132 29.3984 0.0195919 0.0262075 2026-02-23T13:14:05.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:14:05 trial068 ceph-mon[38805]: pgmap v359: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 1.3 GiB data, 5.9 GiB used, 2.7 TiB / 2.7 TiB avail; 30 MiB/s wr, 3.81k op/s; 14669/63870 objects degraded (22.967%); 1/21290 objects unfound (0.005%); 4.5 MiB/s, 71 objects/s recovering 2026-02-23T13:14:05.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:14:05 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/263339948' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:14:06.101 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T13:14:06.100839+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.026314 2026-02-23T13:14:06.101 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T13:14:06.101 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 40 128 187685 187557 36.6295 29.4688 0.031795 0.026314 2026-02-23T13:14:06.694 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:14:06.840 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:14:06.965 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:14:06.963+0000 7f6bc3c92640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:14:06.965 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:14:06.964+0000 7f6bc3c92640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:14:06.982 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:14:06.982 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:14:06.997 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:14:06.997 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:14:06.997 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:14:07.013 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:14:07.013 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:14:07.055 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:14:07.055 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:14:07.101 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 41 128 191378 191250 36.4398 28.8516 0.0236678 0.0264362 2026-02-23T13:14:07.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:14:07 trial068 ceph-mon[38805]: pgmap v360: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 1.4 GiB data, 6.0 GiB used, 2.7 TiB / 2.7 TiB avail; 32 MiB/s wr, 4.13k op/s; 15239/66945 objects degraded (22.763%); 1/22315 objects unfound (0.004%) 2026-02-23T13:14:07.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:14:07 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/287294605' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:14:08.101 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 42 127 195260 195133 36.2944 30.3359 0.0133768 0.0265256 2026-02-23T13:14:08.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:14:08 trial068 ceph-mon[38805]: pgmap v361: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 1.4 GiB data, 6.2 GiB used, 2.7 TiB / 2.7 TiB avail; 33 MiB/s wr, 4.16k op/s; 16261/70818 objects degraded (22.962%); 1/23606 objects unfound (0.004%) 2026-02-23T13:14:08.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:14:08 trial068 ceph-mon[38805]: Health check update: 1/23606 objects unfound (0.004%) (OBJECT_UNFOUND) 2026-02-23T13:14:09.056 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:14:09.101 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 43 128 198955 198827 36.1214 28.8594 0.0567316 0.0266326 2026-02-23T13:14:09.199 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:14:09.324 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:14:09.322+0000 7f9d873a1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:14:09.325 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:14:09.323+0000 7f9d873a1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:14:09.339 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:14:09.340 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:14:09.354 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:14:09.354 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:14:09.355 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:14:09.369 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:14:09.370 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:14:09.409 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:14:09.410 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:14:09.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:14:09 trial068 ceph-mon[38805]: Health check update: Degraded data redundancy: 16261/70818 objects degraded (22.962%), 12 pgs degraded (PG_DEGRADED) 2026-02-23T13:14:09.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:14:09 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1454092486' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:14:10.101 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 44 128 202761 202633 35.9762 29.7344 0.00693202 0.0267282 2026-02-23T13:14:10.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:14:10 trial068 ceph-mon[38805]: pgmap v362: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 1.4 GiB data, 6.2 GiB used, 2.7 TiB / 2.7 TiB avail; 25 MiB/s wr, 3.14k op/s; 16261/70818 objects degraded (22.962%); 1/23606 objects unfound (0.004%) 2026-02-23T13:14:11.101 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 45 128 206436 206308 35.8147 28.7109 0.0217223 0.0268311 2026-02-23T13:14:11.411 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:14:11.552 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:14:11.713 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:14:11.711+0000 7f0716784640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:14:11.713 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:14:11.711+0000 7f0716784640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:14:11.728 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:14:11.728 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:14:11.745 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:14:11.746 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:14:11.746 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:14:11.760 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:14:11.761 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:14:11.796 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:14:11.797 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:14:12.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:14:11 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1889931607' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:14:12.101 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 46 128 210034 209906 35.6471 28.1094 0.0691712 0.026947 2026-02-23T13:14:13.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:14:12 trial068 ceph-mon[38805]: pgmap v363: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 1.6 GiB data, 6.5 GiB used, 2.7 TiB / 2.7 TiB avail; 34 MiB/s wr, 4.42k op/s; 17480/76542 objects degraded (22.837%); 1/25514 objects unfound (0.004%) 2026-02-23T13:14:13.101 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 47 127 213803 213676 35.5153 29.4531 0.0215031 0.0270346 2026-02-23T13:14:13.798 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:14:13.954 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:14:14.066 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:14:14.064+0000 7f15d3ff0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:14:14.066 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:14:14.064+0000 7f15d3ff0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:14:14.079 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:14:14.080 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:14:14.080 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:14:14 trial068 ceph-mon[38805]: Health check update: 1/25514 objects unfound (0.004%) (OBJECT_UNFOUND) 2026-02-23T13:14:14.080 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:14:14 trial068 ceph-mon[38805]: Health check update: Degraded data redundancy: 17480/76542 objects degraded (22.837%), 12 pgs degraded (PG_DEGRADED) 2026-02-23T13:14:14.081 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:14:13 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:14:13] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:14:14.092 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:14:14.093 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:14:14.093 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:14:14.101 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 48 128 217488 217360 35.375 28.7812 0.0195637 0.027125 2026-02-23T13:14:14.105 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:14:14.106 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:14:14.145 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:14:14.145 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:14:14.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:14:13 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:14:13.972Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:14:15.102 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 49 128 221303 221175 35.2612 29.8047 0.0180355 0.0272005 2026-02-23T13:14:15.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:14:15 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/413001851' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:14:15.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:14:15 trial068 ceph-mon[38805]: pgmap v364: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 1.6 GiB data, 6.6 GiB used, 2.7 TiB / 2.7 TiB avail; 27 MiB/s wr, 3.39k op/s; 17897/77793 objects degraded (23.006%); 1/25931 objects unfound (0.004%) 2026-02-23T13:14:16.102 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 50 128 225147 225019 35.1566 30.0312 0.0661204 0.0272692 2026-02-23T13:14:16.146 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:14:16.300 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:14:16.428 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:14:16.426+0000 7f7340579640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:14:16.429 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:14:16.427+0000 7f7340579640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:14:16.444 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:14:16.444 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:14:16.456 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:14:16.457 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:14:16.457 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:14:16.470 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:14:16.470 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:14:16.508 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:14:16.508 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:14:17.102 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 51 128 228680 228552 35.0084 27.6016 0.0177749 0.0273719 2026-02-23T13:14:17.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:14:17 trial068 ceph-mon[38805]: pgmap v365: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 1.6 GiB data, 6.8 GiB used, 2.7 TiB / 2.7 TiB avail; 29 MiB/s wr, 3.76k op/s; 18467/80772 objects degraded (22.863%); 1/26924 objects unfound (0.004%) 2026-02-23T13:14:17.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:14:17 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2556006473' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:14:18.102 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 52 128 232185 232057 34.8617 27.3828 0.0650922 0.0274748 2026-02-23T13:14:18.509 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:14:18.656 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:14:18.777 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:14:18.775+0000 7fe9bf40b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:14:18.777 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:14:18.775+0000 7fe9bf40b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:14:18.792 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:14:18.792 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:14:18.806 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:14:18.807 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:14:18.807 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:14:18.820 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:14:18.820 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:14:18.859 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:14:18.860 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:14:19.102 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 53 128 235755 235627 34.7301 27.8906 0.0532708 0.0275689 2026-02-23T13:14:19.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:14:19 trial068 ceph-mon[38805]: pgmap v366: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 1.7 GiB data, 7.0 GiB used, 2.7 TiB / 2.7 TiB avail; 31 MiB/s wr, 3.96k op/s; 19481/84762 objects degraded (22.983%); 1/28254 objects unfound (0.004%) 2026-02-23T13:14:19.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:14:19 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/643530431' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:14:19.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:14:19 trial068 ceph-mon[38805]: Health check update: 1/28254 objects unfound (0.004%) (OBJECT_UNFOUND) 2026-02-23T13:14:19.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:14:19 trial068 ceph-mon[38805]: Health check update: Degraded data redundancy: 19481/84762 objects degraded (22.983%), 12 pgs degraded (PG_DEGRADED) 2026-02-23T13:14:20.102 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 54 128 239543 239415 34.635 29.5938 0.0152494 0.0276367 2026-02-23T13:14:20.861 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:14:21.003 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:14:21.102 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 55 128 243249 243121 34.5316 28.9531 0.0305397 0.0277044 2026-02-23T13:14:21.123 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:14:21.121+0000 7fdd7a47f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:14:21.123 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:14:21.122+0000 7fdd7a47f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:14:21.138 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:14:21.139 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:14:21.153 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:14:21.153 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:14:21.153 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:14:21.166 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:14:21.166 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:14:21.207 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:14:21.208 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:14:21.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:14:21 trial068 ceph-mon[38805]: pgmap v367: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 1.7 GiB data, 7.0 GiB used, 2.7 TiB / 2.7 TiB avail; 24 MiB/s wr, 3.10k op/s; 19481/84762 objects degraded (22.983%); 1/28254 objects unfound (0.004%) 2026-02-23T13:14:21.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:14:21 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2736913578' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:14:22.102 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 56 128 246974 246846 34.4346 29.1016 0.0460208 0.0277769 2026-02-23T13:14:23.102 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 57 128 250601 250473 34.3276 28.3359 0.0668955 0.0278521 2026-02-23T13:14:23.208 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:14:23.359 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:14:23.495 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:14:23.493+0000 7f86d7e7c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:14:23.495 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:14:23.493+0000 7f86d7e7c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:14:23.495 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:14:23 trial068 ceph-mon[38805]: pgmap v368: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 1.8 GiB data, 7.3 GiB used, 2.7 TiB / 2.7 TiB avail; 34 MiB/s wr, 4.35k op/s; 20643/90390 objects degraded (22.838%); 1/30130 objects unfound (0.003%) 2026-02-23T13:14:23.513 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:14:23.513 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:14:23.528 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:14:23.528 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:14:23.529 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:14:23.542 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:14:23.542 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:14:23.584 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:14:23.584 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:14:24.102 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 58 128 254226 254098 34.224 28.3203 0.0139041 0.0279255 2026-02-23T13:14:24.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:14:23 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:14:23] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:14:24.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:14:23 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:14:23.972Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:14:24.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:14:24 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1752178607' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:14:24.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:14:24 trial068 ceph-mon[38805]: Health check update: 1/30130 objects unfound (0.003%) (OBJECT_UNFOUND) 2026-02-23T13:14:24.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:14:24 trial068 ceph-mon[38805]: Health check update: Degraded data redundancy: 20643/90390 objects degraded (22.838%), 12 pgs degraded (PG_DEGRADED) 2026-02-23T13:14:25.102 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 59 128 257866 257738 34.1259 28.4375 0.024671 0.0279968 2026-02-23T13:14:25.585 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:14:25.610 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:14:25 trial068 ceph-mon[38805]: pgmap v369: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 1.9 GiB data, 7.5 GiB used, 2.7 TiB / 2.7 TiB avail; 26 MiB/s wr, 3.38k op/s; 21100/91761 objects degraded (22.995%); 1/30587 objects unfound (0.003%) 2026-02-23T13:14:25.735 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:14:25.857 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:14:25.855+0000 7faca3ef9640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:14:25.857 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:14:25.857+0000 7faca3ef9640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:14:25.874 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:14:25.874 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:14:25.886 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:14:25.886 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:14:25.886 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:14:25.900 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:14:25.900 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:14:25.940 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:14:25.940 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:14:26.102 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T13:14:26.102411+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0280658 2026-02-23T13:14:26.103 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T13:14:26.103 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 60 128 261528 261400 34.0339 28.6094 0.0256473 0.0280658 2026-02-23T13:14:26.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:14:26 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T13:14:26.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:14:26 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T13:14:26.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:14:26 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3312093909' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:14:27.103 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 61 128 265130 265002 33.9372 28.1406 0.0875523 0.0281331 2026-02-23T13:14:27.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:14:27 trial068 ceph-mon[38805]: pgmap v370: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 1.9 GiB data, 7.6 GiB used, 2.7 TiB / 2.7 TiB avail; 29 MiB/s wr, 3.75k op/s; 21664/94653 objects degraded (22.888%); 1/31551 objects unfound (0.003%) 2026-02-23T13:14:27.941 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:14:28.094 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:14:28.102 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 62 128 268622 268494 33.8299 27.2812 0.0249176 0.0282171 2026-02-23T13:14:28.219 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:14:28.217+0000 7fde95f3c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:14:28.219 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:14:28.217+0000 7fde95f3c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:14:28.234 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:14:28.234 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:14:28.247 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:14:28.247 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:14:28.248 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:14:28.262 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:14:28.262 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:14:28.302 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:14:28.303 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:14:28.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:14:28 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3258401782' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:14:29.102 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 63 128 272163 272035 33.732 27.6641 0.0307527 0.0282901 2026-02-23T13:14:29.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:14:29 trial068 ceph-mon[38805]: pgmap v371: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.0 GiB data, 7.9 GiB used, 2.7 TiB / 2.7 TiB avail; 31 MiB/s wr, 3.96k op/s; 22662/98577 objects degraded (22.989%); 1/32859 objects unfound (0.003%) 2026-02-23T13:14:29.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:14:29 trial068 ceph-mon[38805]: Health check update: 1/32859 objects unfound (0.003%) (OBJECT_UNFOUND) 2026-02-23T13:14:29.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:14:29 trial068 ceph-mon[38805]: Health check update: Degraded data redundancy: 22662/98577 objects degraded (22.989%), 12 pgs degraded (PG_DEGRADED) 2026-02-23T13:14:30.103 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 64 127 275910 275783 33.6624 29.2812 0.0151799 0.0283425 2026-02-23T13:14:30.304 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:14:30.451 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:14:30.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:14:30 trial068 ceph-mon[38805]: 5.e scrub starts 2026-02-23T13:14:30.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:14:30 trial068 ceph-mon[38805]: 5.e scrub ok 2026-02-23T13:14:30.805 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:14:30.803+0000 7f2523e3f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:14:30.805 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:14:30.803+0000 7f2523e3f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:14:30.820 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:14:30.820 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:14:30.833 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:14:30.834 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:14:30.834 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:14:30.848 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:14:30.848 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:14:30.887 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:14:30.887 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:14:31.103 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 65 127 279151 279024 33.534 25.3203 0.0345588 0.0284453 2026-02-23T13:14:31.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:14:31 trial068 ceph-mon[38805]: pgmap v372: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.0 GiB data, 7.9 GiB used, 2.7 TiB / 2.7 TiB avail; 24 MiB/s wr, 3.07k op/s; 22662/98577 objects degraded (22.989%); 1/32859 objects unfound (0.003%) 2026-02-23T13:14:31.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:14:31 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/4157144798' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:14:32.103 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 66 128 282864 282736 33.4653 29 0.0101593 0.0284881 2026-02-23T13:14:32.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:14:32 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2026-02-23T13:14:32.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:14:32 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-23T13:14:32.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:14:32 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2026-02-23T13:14:32.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:14:32 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T13:14:32.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:14:32 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2026-02-23T13:14:32.889 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:14:33.034 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:14:33.103 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 67 128 286502 286374 33.39 28.4219 0.0619774 0.0285508 2026-02-23T13:14:33.165 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:14:33.161+0000 7f31d49fc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:14:33.165 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:14:33.163+0000 7f31d49fc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:14:33.183 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:14:33.183 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:14:33.197 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:14:33.198 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:14:33.198 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:14:33.210 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:14:33.210 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:14:33.247 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:14:33.248 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:14:33.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:14:33 trial068 ceph-mon[38805]: pgmap v373: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.1 GiB data, 8.2 GiB used, 2.7 TiB / 2.7 TiB avail; 33 MiB/s wr, 4.28k op/s; 23782/104025 objects degraded (22.862%); 1/34675 objects unfound (0.003%) 2026-02-23T13:14:33.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:14:33 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2622524462' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:14:34.103 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 68 128 290053 289925 33.3067 27.7422 0.0381275 0.0286141 2026-02-23T13:14:34.265 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:14:33 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:14:33.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:14:34.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:14:33 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:14:33] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:14:34.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:14:34 trial068 ceph-mon[38805]: Health check update: 1/34675 objects unfound (0.003%) (OBJECT_UNFOUND) 2026-02-23T13:14:34.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:14:34 trial068 ceph-mon[38805]: Health check update: Degraded data redundancy: 23782/104025 objects degraded (22.862%), 12 pgs degraded (PG_DEGRADED) 2026-02-23T13:14:35.103 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 69 128 293826 293698 33.2512 29.4766 0.0355109 0.0286586 2026-02-23T13:14:35.249 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:14:35.395 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:14:35.522 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:14:35.521+0000 7efdb3665640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:14:35.522 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:14:35.521+0000 7efdb3665640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:14:35.538 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:14:35.538 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:14:35.538 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:14:35 trial068 ceph-mon[38805]: pgmap v374: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.1 GiB data, 8.3 GiB used, 2.7 TiB / 2.7 TiB avail; 26 MiB/s wr, 3.31k op/s; 24197/105270 objects degraded (22.986%); 1/35090 objects unfound (0.003%) 2026-02-23T13:14:35.551 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:14:35.551 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:14:35.552 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:14:35.566 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:14:35.566 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:14:35.604 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:14:35.604 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:14:36.103 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 70 128 297144 297016 33.1465 25.9219 0.0240758 0.0287341 2026-02-23T13:14:36.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:14:36 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3696497126' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:14:37.104 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 71 128 300633 300505 33.0635 27.2578 0.0162917 0.0288051 2026-02-23T13:14:37.605 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:14:37.628 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:14:37 trial068 ceph-mon[38805]: pgmap v375: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.2 GiB data, 8.4 GiB used, 2.7 TiB / 2.7 TiB avail; 28 MiB/s wr, 3.63k op/s; 24754/108129 objects degraded (22.893%); 1/36043 objects unfound (0.003%) 2026-02-23T13:14:37.748 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:14:37.885 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:14:37.883+0000 7fead72dc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:14:37.885 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:14:37.883+0000 7fead72dc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:14:37.901 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:14:37.902 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:14:37.916 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:14:37.916 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:14:37.917 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:14:37.931 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:14:37.931 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:14:38.104 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 72 128 303769 303641 32.9445 24.5 0.0161965 0.0288946 2026-02-23T13:14:38.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:14:38 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2783568355' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:14:38.648 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:14:38.648 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:14:39.104 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 73 128 306776 306648 32.8149 23.4922 0.0186004 0.0290109 2026-02-23T13:14:39.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:14:39 trial068 ceph-mon[38805]: pgmap v376: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.3 GiB data, 8.7 GiB used, 2.7 TiB / 2.7 TiB avail; 30 MiB/s wr, 3.83k op/s; 25724/111909 objects degraded (22.987%); 1/37303 objects unfound (0.003%) 2026-02-23T13:14:39.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:14:39 trial068 ceph-mon[38805]: Health check update: 1/37303 objects unfound (0.003%) (OBJECT_UNFOUND) 2026-02-23T13:14:39.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:14:39 trial068 ceph-mon[38805]: Health check update: Degraded data redundancy: 25724/111909 objects degraded (22.987%), 12 pgs degraded (PG_DEGRADED) 2026-02-23T13:14:40.104 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 74 127 310402 310275 32.7543 28.3359 0.0105278 0.0290572 2026-02-23T13:14:40.649 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:14:40.673 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:14:40 trial068 ceph-mon[38805]: 5.a scrub starts 2026-02-23T13:14:40.788 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:14:40.907 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:14:40.903+0000 7fa59df73640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:14:40.907 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:14:40.904+0000 7fa59df73640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:14:40.921 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:14:40.921 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:14:40.935 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:14:40.935 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:14:40.937 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:14:40.948 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:14:40.948 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:14:40.989 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:14:40.989 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:14:41.104 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 75 128 313707 313579 32.6617 25.8125 0.0613639 0.0291373 2026-02-23T13:14:41.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:14:41 trial068 ceph-mon[38805]: 5.a scrub ok 2026-02-23T13:14:41.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:14:41 trial068 ceph-mon[38805]: pgmap v377: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.3 GiB data, 8.7 GiB used, 2.7 TiB / 2.7 TiB avail; 23 MiB/s wr, 2.96k op/s; 25724/111909 objects degraded (22.987%); 1/37303 objects unfound (0.003%) 2026-02-23T13:14:41.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:14:41 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/578580531' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:14:42.104 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 76 128 317187 317059 32.5897 27.1875 0.0284199 0.0291941 2026-02-23T13:14:42.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:14:42 trial068 ceph-mon[38805]: 5.7 scrub starts 2026-02-23T13:14:42.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:14:42 trial068 ceph-mon[38805]: 5.7 scrub ok 2026-02-23T13:14:42.990 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:14:43.104 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 77 128 320652 320524 32.518 27.0703 0.0360201 0.0292506 2026-02-23T13:14:43.140 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:14:43.266 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:14:43.264+0000 7fb9882d5640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:14:43.266 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:14:43.264+0000 7fb9882d5640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:14:43.285 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:14:43.286 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:14:43.297 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:14:43.297 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:14:43.297 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:14:43.310 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:14:43.310 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:14:43.347 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:14:43.347 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:14:43.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:14:43 trial068 ceph-mon[38805]: pgmap v378: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.4 GiB data, 9.0 GiB used, 2.7 TiB / 2.7 TiB avail; 32 MiB/s wr, 4.09k op/s; 26773/117012 objects degraded (22.881%); 1/39004 objects unfound (0.003%) 2026-02-23T13:14:43.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:14:43 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2930286170' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:14:44.104 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 78 128 324036 323908 32.44 26.4375 0.0407922 0.0293124 2026-02-23T13:14:44.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:14:43 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:14:43] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:14:44.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:14:43 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:14:43.972Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:14:44.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:14:44 trial068 ceph-mon[38805]: Health check update: 1/39004 objects unfound (0.003%) (OBJECT_UNFOUND) 2026-02-23T13:14:44.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:14:44 trial068 ceph-mon[38805]: Health check update: Degraded data redundancy: 26773/117012 objects degraded (22.881%), 12 pgs degraded (PG_DEGRADED) 2026-02-23T13:14:45.105 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 79 128 327432 327304 32.3652 26.5312 0.0689714 0.0293822 2026-02-23T13:14:45.348 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:14:45.494 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:14:45.612 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:14:45.610+0000 7fac03223640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:14:45.612 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:14:45.611+0000 7fac03223640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:14:45.626 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:14:45.627 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:14:45.627 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:14:45 trial068 ceph-mon[38805]: pgmap v379: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.4 GiB data, 9.1 GiB used, 2.7 TiB / 2.7 TiB avail; 25 MiB/s wr, 3.14k op/s; 27161/118176 objects degraded (22.984%); 1/39392 objects unfound (0.003%) 2026-02-23T13:14:45.644 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:14:45.645 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:14:45.645 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:14:45.657 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:14:45.657 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:14:45.692 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:14:45.693 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:14:46.105 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T13:14:46.104549+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0294382 2026-02-23T13:14:46.105 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T13:14:46.105 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 80 128 330852 330724 32.2946 26.7188 0.0100377 0.0294382 2026-02-23T13:14:46.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:14:46 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1838561034' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:14:47.105 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 81 128 334413 334285 32.2393 27.8203 0.0734043 0.029487 2026-02-23T13:14:47.694 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:14:47.718 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:14:47 trial068 ceph-mon[38805]: pgmap v380: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.5 GiB data, 9.2 GiB used, 2.7 TiB / 2.7 TiB avail; 27 MiB/s wr, 3.49k op/s; 27696/120963 objects degraded (22.896%); 1/40321 objects unfound (0.002%) 2026-02-23T13:14:47.834 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:14:47.955 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:14:47.953+0000 7fc829540640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:14:47.955 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:14:47.953+0000 7fc829540640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:14:47.970 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:14:47.971 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:14:47.982 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:14:47.983 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:14:47.983 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:14:47.995 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:14:47.995 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:14:48.034 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:14:48.034 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:14:48.105 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 82 128 337629 337501 32.1525 25.125 0.0206315 0.0295591 2026-02-23T13:14:48.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:14:48 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/49602649' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:14:49.105 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 83 128 341109 340981 32.0927 27.1875 0.0451452 0.0296107 2026-02-23T13:14:49.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:14:49 trial068 ceph-mon[38805]: pgmap v381: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.5 GiB data, 9.5 GiB used, 2.7 TiB / 2.7 TiB avail; 29 MiB/s wr, 3.65k op/s; 28640/124578 objects degraded (22.990%); 1/41526 objects unfound (0.002%) 2026-02-23T13:14:49.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:14:49 trial068 ceph-mon[38805]: Health check update: 1/41526 objects unfound (0.002%) (OBJECT_UNFOUND) 2026-02-23T13:14:49.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:14:49 trial068 ceph-mon[38805]: Health check update: Degraded data redundancy: 28640/124578 objects degraded (22.990%), 12 pgs degraded, 12 pgs undersized (PG_DEGRADED) 2026-02-23T13:14:50.036 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:14:50.105 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 84 128 344727 344599 32.0471 28.2656 0.0625714 0.0296489 2026-02-23T13:14:50.188 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:14:50.428 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:14:50.426+0000 7f31f1773640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:14:50.428 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:14:50.426+0000 7f31f1773640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:14:50.442 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:14:50.443 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:14:50.454 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:14:50.455 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:14:50.455 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:14:50.468 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:14:50.468 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:14:50.508 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:14:50.509 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:14:51.105 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 85 128 348137 348009 31.9835 26.6406 0.0500594 0.0297019 2026-02-23T13:14:51.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:14:51 trial068 ceph-mon[38805]: pgmap v382: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.5 GiB data, 9.5 GiB used, 2.7 TiB / 2.7 TiB avail; 22 MiB/s wr, 2.81k op/s; 28640/124578 objects degraded (22.990%); 1/41526 objects unfound (0.002%) 2026-02-23T13:14:51.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:14:51 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2725864812' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:14:52.105 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 86 128 351720 351592 31.937 27.9922 0.0275629 0.0297411 2026-02-23T13:14:52.510 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:14:52.658 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:14:52.796 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:14:52.794+0000 7f408fdcc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:14:52.796 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:14:52.794+0000 7f408fdcc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:14:52.815 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:14:52.816 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:14:52.830 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:14:52.830 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:14:52.830 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:14:52.844 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:14:52.844 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:14:52.887 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:14:52.887 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:14:53.105 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 87 128 355200 355072 31.8824 27.1875 0.00816981 0.0297833 2026-02-23T13:14:53.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:14:53 trial068 ceph-mon[38805]: 5.8 scrub starts 2026-02-23T13:14:53.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:14:53 trial068 ceph-mon[38805]: pgmap v383: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.6 GiB data, 9.7 GiB used, 2.7 TiB / 2.7 TiB avail; 31 MiB/s wr, 4.01k op/s; 29762/129963 objects degraded (22.900%); 1/43321 objects unfound (0.002%) 2026-02-23T13:14:53.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:14:53 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1834034198' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:14:54.105 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 88 128 358633 358505 31.8249 26.8203 0.0322543 0.0298311 2026-02-23T13:14:54.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:14:53 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:14:53] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:14:54.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:14:53 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:14:53.972Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:14:54.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:14:54 trial068 ceph-mon[38805]: 5.8 scrub ok 2026-02-23T13:14:54.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:14:54 trial068 ceph-mon[38805]: Health check update: 1/43321 objects unfound (0.002%) (OBJECT_UNFOUND) 2026-02-23T13:14:54.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:14:54 trial068 ceph-mon[38805]: Health check update: Degraded data redundancy: 29762/129963 objects degraded (22.900%), 12 pgs degraded, 12 pgs undersized (PG_DEGRADED) 2026-02-23T13:14:54.888 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:14:55.044 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:14:55.106 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 89 128 361974 361846 31.7605 26.1016 0.0279187 0.029888 2026-02-23T13:14:55.167 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:14:55.166+0000 7fe82fe7a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:14:55.168 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:14:55.166+0000 7fe82fe7a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:14:55.181 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:14:55.182 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:14:55.195 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:14:55.195 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:14:55.195 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:14:55.208 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:14:55.209 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:14:55.245 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:14:55.246 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:14:55.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:14:55 trial068 ceph-mon[38805]: 5.5 scrub starts 2026-02-23T13:14:55.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:14:55 trial068 ceph-mon[38805]: pgmap v384: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.7 GiB data, 9.8 GiB used, 2.7 TiB / 2.7 TiB avail; 24 MiB/s wr, 3.13k op/s; 30148/131121 objects degraded (22.993%); 1/43707 objects unfound (0.002%) 2026-02-23T13:14:55.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:14:55 trial068 ceph-mon[38805]: osdmap e185: 4 total, 3 up, 4 in 2026-02-23T13:14:55.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:14:55 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2045280654' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:14:56.105 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 90 102 365281 365179 31.6968 26.0391 0.0208599 0.0299478 2026-02-23T13:14:57.106 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 91 8 365281 365273 31.3566 0.734375 0.059933 0.0299513 2026-02-23T13:14:57.247 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:14:57.365 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:14:57.529 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:14:57 trial068 ceph-mon[38805]: 5.5 scrub ok 2026-02-23T13:14:57.529 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:14:57 trial068 ceph-mon[38805]: pgmap v386: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.7 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 27 MiB/s wr, 3.48k op/s; 30703/134031 objects degraded (22.907%); 1/44677 objects unfound (0.002%) 2026-02-23T13:14:57.786 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:14:57.728+0000 7fcdb6eb7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:14:57.786 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:14:57.728+0000 7fcdb6eb7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:14:57.786 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:14:57.786 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:14:57.787 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:14:57.787 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:14:57.787 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:14:57.787 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:14:57.787 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:14:58.106 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 92 8 365281 365273 31.0158 0 - 0.0299513 2026-02-23T13:14:58.484 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:14:58.484 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:14:58.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:14:58 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/441683201' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:14:59.106 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 93 8 365281 365273 30.6823 0 - 0.0299513 2026-02-23T13:14:59.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:14:59 trial068 ceph-mon[38805]: pgmap v387: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 26 MiB/s wr, 3.27k op/s; 31432/136851 objects degraded (22.968%); 1/45617 objects unfound (0.002%) 2026-02-23T13:14:59.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:14:59 trial068 ceph-mon[38805]: Health check update: 1/45617 objects unfound (0.002%) (OBJECT_UNFOUND) 2026-02-23T13:14:59.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:14:59 trial068 ceph-mon[38805]: Health check update: Degraded data redundancy: 31432/136851 objects degraded (22.968%), 12 pgs degraded, 12 pgs undersized (PG_DEGRADED) 2026-02-23T13:14:59.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:14:59 trial068 ceph-mon[38805]: osdmap e186: 4 total, 3 up, 4 in 2026-02-23T13:15:00.106 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 94 8 365281 365273 30.3559 0 - 0.0299513 2026-02-23T13:15:00.486 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:15:00.603 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:15:00.708 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:15:00.706+0000 7f2c04971640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:15:00.708 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:15:00.707+0000 7f2c04971640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:15:00.720 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:15:00.721 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:15:00.731 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:15:00.732 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:15:00.732 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:15:00.743 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:15:00.744 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:15:00.774 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:15:00.774 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:15:01.106 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 95 8 365281 365273 30.0363 0 - 0.0299513 2026-02-23T13:15:01.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:15:01 trial068 ceph-mon[38805]: pgmap v389: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 18 MiB/s wr, 2.29k op/s; 31432/136851 objects degraded (22.968%); 1/45617 objects unfound (0.002%) 2026-02-23T13:15:01.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:15:01 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/343704075' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:15:02.106 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 96 8 365281 365273 29.7235 0 - 0.0299513 2026-02-23T13:15:02.775 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:15:02.893 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:15:02.993 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:15:02.991+0000 7feb348ba640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:15:02.993 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:15:02.991+0000 7feb348ba640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:15:03.006 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:15:03.007 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:15:03.016 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:15:03.017 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:15:03.017 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:15:03.027 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:15:03.028 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:15:03.058 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:15:03.058 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:15:03.106 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 97 8 365281 365273 29.417 0 - 0.0299513 2026-02-23T13:15:03.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:15:03 trial068 ceph-mon[38805]: pgmap v390: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 15 MiB/s wr, 1.96k op/s; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:15:03.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:15:03 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2544885731' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:15:04.106 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 98 8 365281 365273 29.1169 0 - 0.0299513 2026-02-23T13:15:04.265 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:15:03 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:15:03.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:15:04.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:15:03 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:15:03] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:15:04.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:15:04 trial068 ceph-mon[38805]: Health check update: 1/45663 objects unfound (0.002%) (OBJECT_UNFOUND) 2026-02-23T13:15:04.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:15:04 trial068 ceph-mon[38805]: Health check update: Degraded data redundancy: 31459/136989 objects degraded (22.965%), 12 pgs degraded, 12 pgs undersized (PG_DEGRADED) 2026-02-23T13:15:05.059 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:15:05.106 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 99 8 365281 365273 28.8228 0 - 0.0299513 2026-02-23T13:15:05.176 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:15:05.275 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:15:05.273+0000 7f9625e9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:15:05.275 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:15:05.273+0000 7f9625e9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:15:05.287 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:15:05.287 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:15:05.297 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:15:05.298 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:15:05.298 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:15:05.309 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:15:05.309 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:15:05.336 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:15:05.337 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:15:05.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:15:05 trial068 ceph-mon[38805]: pgmap v391: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 12 MiB/s wr, 1.58k op/s; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:15:05.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:15:05 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1079445756' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:15:06.106 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T13:15:06.106324+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T13:15:06.106 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T13:15:06.106 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 100 8 365281 365273 28.5345 0 - 0.0299513 2026-02-23T13:15:07.106 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 101 8 365281 365273 28.252 0 - 0.0299513 2026-02-23T13:15:07.338 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:15:07.455 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:15:07.552 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:15:07.550+0000 7ff8f8ad8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:15:07.552 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:15:07.550+0000 7ff8f8ad8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:15:07.563 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:15:07.564 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:15:07.574 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:15:07.574 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:15:07.574 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:15:07.584 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:15:07.585 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:15:07.614 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:15:07 trial068 ceph-mon[38805]: pgmap v392: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 6.2 MiB/s wr, 788 op/s; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:15:07.615 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:15:07.615 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:15:08.107 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 102 8 365281 365273 27.9751 0 - 0.0299513 2026-02-23T13:15:08.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:15:08 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1182073854' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:15:09.107 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 103 8 365281 365273 27.7035 0 - 0.0299513 2026-02-23T13:15:09.616 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:15:09.640 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:15:09 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T13:15:09.641 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:15:09 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T13:15:09.641 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:15:09 trial068 ceph-mon[38805]: pgmap v393: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 298 KiB/s wr, 37 op/s; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:15:09.733 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:15:09.834 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:15:09.832+0000 7fab1a673640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:15:09.834 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:15:09.832+0000 7fab1a673640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:15:09.846 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:15:09.846 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:15:09.856 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:15:09.857 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:15:09.857 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:15:09.867 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:15:09.867 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:15:09.898 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:15:09.899 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:15:10.107 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 104 8 365281 365273 27.4371 0 - 0.0299513 2026-02-23T13:15:10.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:15:10 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3317959779' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:15:11.107 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 105 8 365281 365273 27.1758 0 - 0.0299513 2026-02-23T13:15:11.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:15:11 trial068 ceph-mon[38805]: pgmap v394: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 264 KiB/s wr, 33 op/s; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:15:11.900 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:15:12.020 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:15:12.107 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 106 8 365281 365273 26.9194 0 - 0.0299513 2026-02-23T13:15:12.119 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:15:12.117+0000 7f2769644640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:15:12.119 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:15:12.117+0000 7f2769644640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:15:12.131 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:15:12.132 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:15:12.142 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:15:12.142 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:15:12.142 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:15:12.153 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:15:12.153 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:15:12.181 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:15:12.182 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:15:12.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:15:12 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/706096831' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:15:13.107 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 107 8 365281 365273 26.6678 0 - 0.0299513 2026-02-23T13:15:13.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:15:13 trial068 ceph-mon[38805]: pgmap v395: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 249 KiB/s wr, 31 op/s; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:15:14.107 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 108 8 365281 365273 26.4209 0 - 0.0299513 2026-02-23T13:15:14.183 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:15:14.299 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:15:14.322 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:15:13 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:15:13] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:15:14.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:15:13 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:15:13.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:15:14.398 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:15:14.396+0000 7fec414e3640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:15:14.398 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:15:14.396+0000 7fec414e3640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:15:14.410 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:15:14.410 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:15:14.420 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:15:14.421 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:15:14.421 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:15:14.431 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:15:14.431 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:15:14.461 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:15:14.462 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:15:15.107 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 109 8 365281 365273 26.1785 0 - 0.0299513 2026-02-23T13:15:15.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:15:15 trial068 ceph-mon[38805]: pgmap v396: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:15:15.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:15:15 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2176619595' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:15:16.107 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 110 8 365281 365273 25.9405 0 - 0.0299513 2026-02-23T13:15:16.463 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:15:16.581 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:15:16.678 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:15:16.676+0000 7f37914a4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:15:16.678 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:15:16.676+0000 7f37914a4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:15:16.690 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:15:16.690 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:15:16.700 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:15:16.701 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:15:16.701 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:15:16.711 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:15:16.711 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:15:16.742 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:15:16.742 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:15:17.107 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 111 8 365281 365273 25.7068 0 - 0.0299513 2026-02-23T13:15:17.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:15:17 trial068 ceph-mon[38805]: pgmap v397: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:15:17.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:15:17 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3349700116' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:15:18.107 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 112 8 365281 365273 25.4773 0 - 0.0299513 2026-02-23T13:15:18.743 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:15:18.859 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:15:18.955 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:15:18.954+0000 7f81b1532640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:15:18.956 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:15:18.954+0000 7f81b1532640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:15:18.967 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:15:18.968 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:15:18.978 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:15:18.978 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:15:18.978 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:15:18.989 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:15:18.989 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:15:19.020 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:15:19.020 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:15:19.107 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 113 8 365281 365273 25.2518 0 - 0.0299513 2026-02-23T13:15:19.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:15:19 trial068 ceph-mon[38805]: pgmap v398: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:15:19.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:15:19 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3207947863' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:15:20.107 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 114 8 365281 365273 25.0303 0 - 0.0299513 2026-02-23T13:15:20.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:15:20 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T13:15:20.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:15:20 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T13:15:21.021 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:15:21.107 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 115 8 365281 365273 24.8127 0 - 0.0299513 2026-02-23T13:15:21.140 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:15:21.238 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:15:21.236+0000 7f4ab63cf640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:15:21.238 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:15:21.236+0000 7f4ab63cf640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:15:21.250 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:15:21.251 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:15:21.261 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:15:21.262 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:15:21.262 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:15:21.272 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:15:21.273 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:15:21.303 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:15:21.303 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:15:21.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:15:21 trial068 ceph-mon[38805]: pgmap v399: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:15:21.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:15:21 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1117778391' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:15:22.108 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 116 8 365281 365273 24.5988 0 - 0.0299513 2026-02-23T13:15:23.108 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 117 8 365281 365273 24.3885 0 - 0.0299513 2026-02-23T13:15:23.304 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:15:23.424 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:15:23.523 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:15:23.521+0000 7fb6b6db4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:15:23.523 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:15:23.521+0000 7fb6b6db4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:15:23.535 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:15:23.535 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:15:23.545 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:15:23.546 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:15:23.546 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:15:23.556 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:15:23.556 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:15:23.586 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:15:23 trial068 ceph-mon[38805]: pgmap v400: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:15:23.587 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:15:23.587 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:15:24.108 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 118 8 365281 365273 24.1819 0 - 0.0299513 2026-02-23T13:15:24.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:15:23 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:15:23] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:15:24.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:15:23 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:15:23.974Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:15:24.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:15:24 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1038796606' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:15:25.108 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 119 8 365281 365273 23.9787 0 - 0.0299513 2026-02-23T13:15:25.589 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:15:25.707 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:15:25.728 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:15:25 trial068 ceph-mon[38805]: pgmap v401: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:15:25.805 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:15:25.803+0000 7f313d9d8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:15:25.806 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:15:25.804+0000 7f313d9d8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:15:25.817 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:15:25.818 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:15:25.827 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:15:25.828 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:15:25.828 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:15:25.838 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:15:25.838 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:15:25.867 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:15:25.867 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:15:26.108 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T13:15:26.107788+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T13:15:26.108 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T13:15:26.108 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 120 8 365281 365273 23.7788 0 - 0.0299513 2026-02-23T13:15:26.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:15:26 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1642137730' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:15:27.108 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 121 8 365281 365273 23.5823 0 - 0.0299513 2026-02-23T13:15:27.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:15:27 trial068 ceph-mon[38805]: pgmap v402: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:15:27.868 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:15:27.990 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:15:28.084 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:15:28.082+0000 7fe7842db640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:15:28.085 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:15:28.082+0000 7fe7842db640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:15:28.096 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:15:28.097 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:15:28.106 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:15:28.107 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:15:28.108 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:15:28.109 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 122 8 365281 365273 23.389 0 - 0.0299513 2026-02-23T13:15:28.117 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:15:28.117 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:15:28.146 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:15:28.147 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:15:28.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:15:28 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1876001587' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:15:29.108 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 123 8 365281 365273 23.1989 0 - 0.0299513 2026-02-23T13:15:29.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:15:29 trial068 ceph-mon[38805]: pgmap v403: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:15:30.108 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 124 8 365281 365273 23.0118 0 - 0.0299513 2026-02-23T13:15:30.148 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:15:30.266 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:15:30.361 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:15:30.359+0000 7f558847a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:15:30.361 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:15:30.360+0000 7f558847a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:15:30.373 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:15:30.373 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:15:30.383 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:15:30.383 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:15:30.383 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:15:30.394 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:15:30.394 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:15:30.424 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:15:30 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1113770803' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:15:30.425 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:15:30.425 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:15:31.108 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 125 8 365281 365273 22.8277 0 - 0.0299513 2026-02-23T13:15:31.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:15:31 trial068 ceph-mon[38805]: pgmap v404: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:15:32.108 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 126 8 365281 365273 22.6465 0 - 0.0299513 2026-02-23T13:15:32.426 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:15:32.541 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:15:32.640 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:15:32.638+0000 7fb6285f4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:15:32.640 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:15:32.638+0000 7fb6285f4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:15:32.652 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:15:32.652 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:15:32.663 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:15:32.663 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:15:32.663 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:15:32.674 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:15:32.674 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:15:32.675 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:15:32 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2026-02-23T13:15:32.675 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:15:32 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-23T13:15:32.675 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:15:32 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2026-02-23T13:15:32.675 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:15:32 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T13:15:32.675 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:15:32 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2026-02-23T13:15:32.702 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:15:32.703 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:15:33.108 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 127 8 365281 365273 22.4682 0 - 0.0299513 2026-02-23T13:15:33.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:15:33 trial068 ceph-mon[38805]: pgmap v405: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:15:33.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:15:33 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/181860912' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:15:34.108 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 128 8 365281 365273 22.2927 0 - 0.0299513 2026-02-23T13:15:34.266 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:15:33 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:15:33.974Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:15:34.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:15:33 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:15:33] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:15:34.704 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:15:34.821 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:15:34.919 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:15:34.917+0000 7fdaf4452640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:15:34.919 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:15:34.918+0000 7fdaf4452640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:15:34.931 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:15:34.931 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:15:34.941 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:15:34.941 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:15:34.941 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:15:34.952 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:15:34.952 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:15:34.982 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:15:34.983 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:15:35.109 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 129 8 365281 365273 22.1199 0 - 0.0299513 2026-02-23T13:15:35.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:15:35 trial068 ceph-mon[38805]: pgmap v406: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:15:35.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:15:35 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1656527656' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:15:36.109 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 130 8 365281 365273 21.9497 0 - 0.0299513 2026-02-23T13:15:36.984 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:15:37.102 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:15:37.108 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 131 8 365281 365273 21.7822 0 - 0.0299513 2026-02-23T13:15:37.197 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:15:37.195+0000 7f43be645640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:15:37.197 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:15:37.196+0000 7f43be645640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:15:37.209 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:15:37.209 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:15:37.219 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:15:37.219 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:15:37.220 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:15:37.230 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:15:37.230 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:15:37.258 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:15:37.259 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:15:37.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:15:37 trial068 ceph-mon[38805]: pgmap v407: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:15:37.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:15:37 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2399870453' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:15:38.109 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 132 8 365281 365273 21.6171 0 - 0.0299513 2026-02-23T13:15:39.109 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 133 8 365281 365273 21.4546 0 - 0.0299513 2026-02-23T13:15:39.260 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:15:39.377 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:15:39.475 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:15:39.473+0000 7ff855e88640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:15:39.475 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:15:39.473+0000 7ff855e88640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:15:39.487 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:15:39.487 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:15:39.497 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:15:39.498 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:15:39.498 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:15:39.507 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:15:39.508 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:15:39.537 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:15:39 trial068 ceph-mon[38805]: pgmap v408: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:15:39.537 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:15:39 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T13:15:39.537 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:15:39 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T13:15:39.538 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:15:39.538 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:15:40.109 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 134 8 365281 365273 21.2945 0 - 0.0299513 2026-02-23T13:15:40.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:15:40 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3911421161' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:15:41.109 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 135 8 365281 365273 21.1368 0 - 0.0299513 2026-02-23T13:15:41.539 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:15:41.657 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:15:41.680 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:15:41 trial068 ceph-mon[38805]: pgmap v409: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:15:41.762 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:15:41.760+0000 7f5131ee5640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:15:41.762 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:15:41.760+0000 7f5131ee5640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:15:41.773 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:15:41.773 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:15:41.783 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:15:41.784 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:15:41.784 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:15:41.794 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:15:41.794 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:15:41.826 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:15:41.826 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:15:42.109 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 136 8 365281 365273 20.9813 0 - 0.0299513 2026-02-23T13:15:42.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:15:42 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3081970613' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:15:43.109 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 137 8 365281 365273 20.8282 0 - 0.0299513 2026-02-23T13:15:43.828 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:15:43.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:15:43 trial068 ceph-mon[38805]: pgmap v410: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:15:43.946 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:15:44.053 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:15:44.051+0000 7fc257b73640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:15:44.053 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:15:44.051+0000 7fc257b73640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:15:44.064 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:15:44.065 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:15:44.075 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:15:44.075 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:15:44.075 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:15:44.086 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:15:44.086 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:15:44.103 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:15:43 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:15:43] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:15:44.109 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 138 8 365281 365273 20.6773 0 - 0.0299513 2026-02-23T13:15:44.117 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:15:44.117 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:15:44.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:15:43 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:15:43.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:15:45.109 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 139 8 365281 365273 20.5285 0 - 0.0299513 2026-02-23T13:15:45.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:15:45 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2653396415' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:15:45.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:15:45 trial068 ceph-mon[38805]: pgmap v411: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:15:46.109 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T13:15:46.109241+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T13:15:46.109 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T13:15:46.109 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 140 8 365281 365273 20.3819 0 - 0.0299513 2026-02-23T13:15:46.118 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:15:46.237 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:15:46.336 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:15:46.334+0000 7fd745812640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:15:46.336 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:15:46.335+0000 7fd745812640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:15:46.348 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:15:46.348 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:15:46.359 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:15:46.359 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:15:46.359 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:15:46.370 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:15:46.370 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:15:46.400 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:15:46 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3569899034' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:15:46.401 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:15:46.402 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:15:47.109 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 141 8 365281 365273 20.2373 0 - 0.0299513 2026-02-23T13:15:47.744 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:15:47 trial068 ceph-mon[38805]: pgmap v412: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:15:48.109 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 142 8 365281 365273 20.0948 0 - 0.0299513 2026-02-23T13:15:48.402 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:15:48.520 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:15:48.625 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:15:48.623+0000 7f8b48505640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:15:48.625 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:15:48.624+0000 7f8b48505640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:15:48.637 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:15:48.637 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:15:48.647 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:15:48.648 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:15:48.648 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:15:48 trial068 ceph-mon[38805]: 5.e scrub starts 2026-02-23T13:15:48.648 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:15:48 trial068 ceph-mon[38805]: 5.e scrub ok 2026-02-23T13:15:48.649 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:15:48.658 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:15:48.658 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:15:48.708 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:15:48.708 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:15:49.109 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 143 8 365281 365273 19.9543 0 - 0.0299513 2026-02-23T13:15:49.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:15:49 trial068 ceph-mon[38805]: pgmap v413: 17 pgs: 1 active+clean+scrubbing, 1 active+recovery_unfound+undersized+degraded, 4 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:15:49.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:15:49 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3837023103' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:15:50.110 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 144 8 365281 365273 19.8157 0 - 0.0299513 2026-02-23T13:15:50.710 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:15:50.828 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:15:50.927 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:15:50.926+0000 7eff9afcb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:15:50.927 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:15:50.926+0000 7eff9afcb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:15:50.939 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:15:50.939 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:15:50.949 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:15:50.949 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:15:50.949 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:15:50.960 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:15:50.960 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:15:50.990 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:15:50.991 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:15:51.110 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 145 8 365281 365273 19.6791 0 - 0.0299513 2026-02-23T13:15:51.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:15:51 trial068 ceph-mon[38805]: pgmap v414: 17 pgs: 1 active+clean+scrubbing, 1 active+recovery_unfound+undersized+degraded, 4 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:15:51.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:15:51 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/823188666' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:15:52.110 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 146 8 365281 365273 19.5443 0 - 0.0299513 2026-02-23T13:15:52.992 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:15:53.110 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 147 8 365281 365273 19.4113 0 - 0.0299513 2026-02-23T13:15:53.113 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:15:53.212 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:15:53.210+0000 7ffb6891c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:15:53.212 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:15:53.211+0000 7ffb6891c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:15:53.225 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:15:53.225 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:15:53.235 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:15:53.235 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:15:53.235 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:15:53.246 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:15:53.246 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:15:53.277 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:15:53.277 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:15:53.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:15:53 trial068 ceph-mon[38805]: pgmap v415: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:15:53.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:15:53 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1148685229' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:15:54.110 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 148 8 365281 365273 19.2802 0 - 0.0299513 2026-02-23T13:15:54.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:15:53 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:15:53] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:15:54.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:15:53 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:15:53.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:15:55.110 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 149 8 365281 365273 19.1508 0 - 0.0299513 2026-02-23T13:15:55.278 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:15:55.397 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:15:55.495 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:15:55.493+0000 7f276ad29640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:15:55.495 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:15:55.494+0000 7f276ad29640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:15:55.506 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:15:55.507 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:15:55.517 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:15:55.517 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:15:55.517 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:15:55.527 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:15:55.527 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:15:55.556 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:15:55 trial068 ceph-mon[38805]: pgmap v416: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:15:55.557 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:15:55.557 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:15:56.110 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 150 8 365281 365273 19.0231 0 - 0.0299513 2026-02-23T13:15:56.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:15:56 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3866560357' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:15:57.110 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 151 8 365281 365273 18.8971 0 - 0.0299513 2026-02-23T13:15:57.558 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:15:57.676 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:15:57.698 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:15:57 trial068 ceph-mon[38805]: pgmap v417: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:15:57.776 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:15:57.774+0000 7f6b4ce49640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:15:57.776 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:15:57.774+0000 7f6b4ce49640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:15:57.788 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:15:57.788 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:15:57.798 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:15:57.799 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:15:57.799 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:15:57.809 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:15:57.809 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:15:57.840 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:15:57.840 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:15:58.110 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 152 8 365281 365273 18.7728 0 - 0.0299513 2026-02-23T13:15:58.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:15:58 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/735062308' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:15:59.110 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 153 8 365281 365273 18.6501 0 - 0.0299513 2026-02-23T13:15:59.841 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:15:59.845 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:15:59 trial068 ceph-mon[38805]: pgmap v418: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:15:59.959 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:16:00.062 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:16:00.060+0000 7f58cc789640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:16:00.062 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:16:00.060+0000 7f58cc789640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:16:00.073 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:16:00.073 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:16:00.083 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:16:00.083 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:16:00.083 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:16:00.094 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:16:00.095 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:16:00.110 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 154 8 365281 365273 18.529 0 - 0.0299513 2026-02-23T13:16:00.126 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:16:00.126 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:16:00.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:16:00 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3726332402' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:16:01.110 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 155 8 365281 365273 18.4095 0 - 0.0299513 2026-02-23T13:16:01.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:16:01 trial068 ceph-mon[38805]: pgmap v419: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:16:02.110 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 156 8 365281 365273 18.2915 0 - 0.0299513 2026-02-23T13:16:02.127 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:16:02.245 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:16:02.347 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:16:02.345+0000 7f47da775640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:16:02.347 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:16:02.345+0000 7f47da775640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:16:02.358 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:16:02.359 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:16:02.369 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:16:02.369 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:16:02.369 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:16:02.381 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:16:02.381 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:16:02.413 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:16:02.414 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:16:02.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:16:02 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/234046221' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:16:03.110 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 157 8 365281 365273 18.1749 0 - 0.0299513 2026-02-23T13:16:03.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:16:03 trial068 ceph-mon[38805]: pgmap v420: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:16:04.110 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 158 8 365281 365273 18.0599 0 - 0.0299513 2026-02-23T13:16:04.265 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:16:03 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:16:03.974Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:16:04.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:16:03 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:16:03] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:16:04.414 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:16:04.532 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:16:04.631 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:16:04.629+0000 7f0f98b0d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:16:04.631 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:16:04.630+0000 7f0f98b0d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:16:04.644 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:16:04.644 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:16:04.653 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:16:04.654 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:16:04.654 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:16:04.664 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:16:04.664 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:16:04.664 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:16:04 trial068 ceph-mon[38805]: osdmap e187: 4 total, 3 up, 4 in 2026-02-23T13:16:04.665 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:16:04 trial068 ceph-mon[38805]: osdmap e188: 4 total, 3 up, 4 in 2026-02-23T13:16:04.694 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:16:04.694 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:16:05.111 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 159 8 365281 365273 17.9463 0 - 0.0299513 2026-02-23T13:16:05.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:16:05 trial068 ceph-mon[38805]: pgmap v423: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:16:05.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:16:05 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/930253347' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:16:06.111 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T13:16:06.110693+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T13:16:06.111 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T13:16:06.111 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 160 8 365281 365273 17.8342 0 - 0.0299513 2026-02-23T13:16:06.695 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:16:06.718 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:16:06 trial068 ceph-mon[38805]: 5.7 scrub starts 2026-02-23T13:16:06.810 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:16:06.912 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:16:06.910+0000 7fb268c6c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:16:06.912 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:16:06.910+0000 7fb268c6c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:16:06.923 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:16:06.924 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:16:06.934 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:16:06.934 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:16:06.934 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:16:06.944 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:16:06.944 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:16:06.976 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:16:06.976 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:16:07.111 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 161 8 365281 365273 17.7234 0 - 0.0299513 2026-02-23T13:16:07.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:16:07 trial068 ceph-mon[38805]: 5.7 scrub ok 2026-02-23T13:16:07.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:16:07 trial068 ceph-mon[38805]: pgmap v424: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:16:07.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:16:07 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2621012943' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:16:08.111 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 162 8 365281 365273 17.614 0 - 0.0299513 2026-02-23T13:16:08.978 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:16:09.097 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:16:09.111 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 163 8 365281 365273 17.5059 0 - 0.0299513 2026-02-23T13:16:09.197 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:16:09.195+0000 7f8719c36640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:16:09.197 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:16:09.196+0000 7f8719c36640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:16:09.209 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:16:09.209 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:16:09.219 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:16:09.219 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:16:09.219 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:16:09.230 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:16:09.231 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:16:09.258 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:16:09.259 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:16:09.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:16:09 trial068 ceph-mon[38805]: pgmap v425: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:16:09.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:16:09 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3119835318' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:16:10.111 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 164 8 365281 365273 17.3992 0 - 0.0299513 2026-02-23T13:16:11.111 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 165 8 365281 365273 17.2937 0 - 0.0299513 2026-02-23T13:16:11.260 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:16:11.375 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:16:11.473 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:16:11.472+0000 7f7e85867640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:16:11.474 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:16:11.472+0000 7f7e85867640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:16:11.485 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:16:11.485 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:16:11.495 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:16:11.496 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:16:11.496 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:16:11.506 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:16:11.506 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:16:11.535 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:16:11 trial068 ceph-mon[38805]: pgmap v426: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:16:11.536 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:16:11.537 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:16:12.111 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 166 8 365281 365273 17.1896 0 - 0.0299513 2026-02-23T13:16:12.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:16:12 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1238420739' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:16:13.111 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 167 8 365281 365273 17.0866 0 - 0.0299513 2026-02-23T13:16:13.538 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:16:13.656 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:16:13.678 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:16:13 trial068 ceph-mon[38805]: pgmap v427: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:16:13.757 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:16:13.755+0000 7f2bd6967640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:16:13.758 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:16:13.756+0000 7f2bd6967640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:16:13.769 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:16:13.769 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:16:13.780 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:16:13.780 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:16:13.780 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:16:13.790 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:16:13.791 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:16:13.820 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:16:13.821 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:16:14.111 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 168 8 365281 365273 16.9849 0 - 0.0299513 2026-02-23T13:16:14.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:16:13 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:16:13] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:16:14.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:16:13 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:16:13.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:16:14.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:16:14 trial068 ceph-mon[38805]: 5.a scrub starts 2026-02-23T13:16:14.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:16:14 trial068 ceph-mon[38805]: 5.a scrub ok 2026-02-23T13:16:14.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:16:14 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/4178617712' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:16:15.111 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 169 8 365281 365273 16.8844 0 - 0.0299513 2026-02-23T13:16:15.822 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:16:15.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:16:15 trial068 ceph-mon[38805]: pgmap v428: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:16:15.938 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:16:16.044 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:16:16.042+0000 7f9e3ba99640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:16:16.044 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:16:16.042+0000 7f9e3ba99640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:16:16.056 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:16:16.056 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:16:16.066 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:16:16.067 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:16:16.067 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:16:16.076 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:16:16.077 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:16:16.107 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:16:16.107 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:16:16.111 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 170 8 365281 365273 16.7851 0 - 0.0299513 2026-02-23T13:16:16.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:16:16 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/613642160' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:16:17.111 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 171 8 365281 365273 16.6869 0 - 0.0299513 2026-02-23T13:16:17.744 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:16:17 trial068 ceph-mon[38805]: pgmap v429: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:16:18.108 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:16:18.112 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 172 8 365281 365273 16.5899 0 - 0.0299513 2026-02-23T13:16:18.223 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:16:18.335 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:16:18.334+0000 7fead68c7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:16:18.336 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:16:18.334+0000 7fead68c7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:16:18.347 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:16:18.348 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:16:18.357 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:16:18.358 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:16:18.358 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:16:18.368 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:16:18.368 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:16:18.395 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:16:18.396 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:16:18.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:16:18 trial068 ceph-mon[38805]: 5.5 scrub starts 2026-02-23T13:16:18.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:16:18 trial068 ceph-mon[38805]: 5.5 scrub ok 2026-02-23T13:16:18.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:16:18 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2512541740' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:16:19.112 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 173 8 365281 365273 16.494 0 - 0.0299513 2026-02-23T13:16:19.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:16:19 trial068 ceph-mon[38805]: pgmap v430: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:16:20.112 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 174 8 365281 365273 16.3992 0 - 0.0299513 2026-02-23T13:16:20.397 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:16:20.513 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:16:20.610 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:16:20.608+0000 7ff7e3ea0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:16:20.610 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:16:20.609+0000 7ff7e3ea0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:16:20.622 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:16:20.622 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:16:20.632 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:16:20.632 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:16:20.632 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:16:20.642 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:16:20.642 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:16:20.672 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:16:20.672 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:16:21.112 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 175 8 365281 365273 16.3055 0 - 0.0299513 2026-02-23T13:16:21.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:16:21 trial068 ceph-mon[38805]: pgmap v431: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:16:21.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:16:21 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/551654247' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:16:22.112 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 176 8 365281 365273 16.2129 0 - 0.0299513 2026-02-23T13:16:22.673 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:16:22.696 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:16:22 trial068 ceph-mon[38805]: 5.8 scrub starts 2026-02-23T13:16:22.696 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:16:22 trial068 ceph-mon[38805]: 5.8 scrub ok 2026-02-23T13:16:22.787 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:16:22.888 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:16:22.886+0000 7effc7bfa640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:16:22.889 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:16:22.887+0000 7effc7bfa640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:16:22.900 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:16:22.900 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:16:22.910 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:16:22.910 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:16:22.910 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:16:22.920 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:16:22.921 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:16:22.964 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:16:22.964 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:16:23.112 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 177 8 365281 365273 16.1213 0 - 0.0299513 2026-02-23T13:16:23.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:16:23 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T13:16:23.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:16:23 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T13:16:23.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:16:23 trial068 ceph-mon[38805]: pgmap v432: 17 pgs: 1 active+clean+scrubbing, 1 active+recovery_unfound+undersized+degraded, 4 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:16:23.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:16:23 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/4047279256' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:16:24.112 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 178 8 365281 365273 16.0307 0 - 0.0299513 2026-02-23T13:16:24.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:16:23 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:16:23] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:16:24.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:16:23 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:16:23.974Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:16:24.965 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:16:25.083 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:16:25.112 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 179 8 365281 365273 15.9412 0 - 0.0299513 2026-02-23T13:16:25.184 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:16:25.182+0000 7f3a57759640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:16:25.184 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:16:25.182+0000 7f3a57759640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:16:25.195 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:16:25.196 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:16:25.205 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:16:25.206 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:16:25.206 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:16:25.216 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:16:25.216 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:16:25.243 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:16:25 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T13:16:25.243 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:16:25 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T13:16:25.243 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:16:25 trial068 ceph-mon[38805]: pgmap v433: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 1 active+clean+scrubbing, 4 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:16:25.244 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:16:25.245 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:16:26.112 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T13:16:26.112251+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T13:16:26.112 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T13:16:26.113 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 180 8 365281 365273 15.8526 0 - 0.0299513 2026-02-23T13:16:26.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:16:26 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/222604842' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:16:27.112 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 181 8 365281 365273 15.765 0 - 0.0299513 2026-02-23T13:16:27.245 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:16:27.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:16:27 trial068 ceph-mon[38805]: pgmap v434: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:16:27.363 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:16:27.470 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:16:27.469+0000 7fadec7c5640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:16:27.471 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:16:27.469+0000 7fadec7c5640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:16:27.482 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:16:27.482 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:16:27.492 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:16:27.492 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:16:27.492 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:16:27.503 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:16:27.503 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:16:27.533 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:16:27.534 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:16:28.112 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 182 8 365281 365273 15.6784 0 - 0.0299513 2026-02-23T13:16:28.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:16:28 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1727885034' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:16:29.112 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 183 8 365281 365273 15.5927 0 - 0.0299513 2026-02-23T13:16:29.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:16:29 trial068 ceph-mon[38805]: pgmap v435: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:16:29.535 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:16:29.650 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:16:29.748 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:16:29.746+0000 7f8d90b64640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:16:29.748 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:16:29.747+0000 7f8d90b64640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:16:29.760 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:16:29.761 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:16:29.770 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:16:29.771 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:16:29.771 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:16:29.780 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:16:29.781 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:16:29.812 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:16:29.813 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:16:30.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:16:30 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2605464987' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:16:30.112 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 184 8 365281 365273 15.508 0 - 0.0299513 2026-02-23T13:16:31.112 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 185 8 365281 365273 15.4242 0 - 0.0299513 2026-02-23T13:16:31.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:16:31 trial068 ceph-mon[38805]: pgmap v436: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:16:31.814 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:16:31.932 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:16:32.027 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:16:32.026+0000 7fe547bdd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:16:32.028 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:16:32.026+0000 7fe547bdd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:16:32.040 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:16:32.040 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:16:32.050 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:16:32.050 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:16:32.050 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:16:32.061 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:16:32.061 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:16:32.090 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:16:32 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3387397818' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:16:32.091 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:16:32.092 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:16:32.112 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 186 8 365281 365273 15.3412 0 - 0.0299513 2026-02-23T13:16:33.113 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 187 8 365281 365273 15.2592 0 - 0.0299513 2026-02-23T13:16:33.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:16:33 trial068 ceph-mon[38805]: pgmap v437: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:16:33.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:16:33 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2026-02-23T13:16:33.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:16:33 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T13:16:33.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:16:33 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T13:16:33.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:16:33 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T13:16:33.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:16:33 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T13:16:33.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:16:33 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-23T13:16:33.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:16:33 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2026-02-23T13:16:33.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:16:33 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T13:16:33.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:16:33 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2026-02-23T13:16:34.092 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:16:34.113 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 188 8 365281 365273 15.178 0 - 0.0299513 2026-02-23T13:16:34.211 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:16:34.234 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:16:33 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:16:33] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:16:34.265 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:16:33 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:16:33.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:16:34.313 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:16:34.311+0000 7f7fbd83a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:16:34.313 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:16:34.312+0000 7f7fbd83a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:16:34.325 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:16:34.326 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:16:34.336 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:16:34.336 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:16:34.336 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:16:34.348 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:16:34.348 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:16:34.380 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:16:34.381 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:16:35.113 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 189 8 365281 365273 15.0977 0 - 0.0299513 2026-02-23T13:16:35.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:16:35 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1267613429' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:16:35.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:16:35 trial068 ceph-mon[38805]: pgmap v438: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:16:36.113 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 190 8 365281 365273 15.0183 0 - 0.0299513 2026-02-23T13:16:36.382 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:16:36.500 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:16:36.597 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:16:36.595+0000 7fe9913a1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:16:36.597 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:16:36.596+0000 7fe9913a1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:16:36.609 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:16:36.610 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:16:36.620 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:16:36.620 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:16:36.620 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:16:36.630 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:16:36.631 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:16:36.660 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:16:36.661 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:16:37.113 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 191 8 365281 365273 14.9396 0 - 0.0299513 2026-02-23T13:16:37.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:16:37 trial068 ceph-mon[38805]: pgmap v439: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:16:37.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:16:37 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/4270929134' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:16:38.113 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 192 8 365281 365273 14.8618 0 - 0.0299513 2026-02-23T13:16:38.661 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:16:38.778 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:16:38.875 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:16:38.873+0000 7f91282e9640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:16:38.875 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:16:38.873+0000 7f91282e9640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:16:38.886 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:16:38.887 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:16:38.896 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:16:38.897 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:16:38.897 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:16:38.907 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:16:38.907 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:16:38.938 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:16:38.939 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:16:39.113 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 193 8 365281 365273 14.7848 0 - 0.0299513 2026-02-23T13:16:39.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:16:39 trial068 ceph-mon[38805]: pgmap v440: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:16:39.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:16:39 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2916530007' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:16:40.113 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 194 8 365281 365273 14.7086 0 - 0.0299513 2026-02-23T13:16:40.940 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:16:41.062 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:16:41.113 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 195 8 365281 365273 14.6332 0 - 0.0299513 2026-02-23T13:16:41.160 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:16:41.158+0000 7f7053934640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:16:41.161 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:16:41.159+0000 7f7053934640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:16:41.172 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:16:41.173 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:16:41.183 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:16:41.183 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:16:41.183 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:16:41.193 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:16:41.193 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:16:41.225 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:16:41.225 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:16:41.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:16:41 trial068 ceph-mon[38805]: pgmap v441: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:16:41.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:16:41 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3469009504' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:16:42.113 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 196 8 365281 365273 14.5585 0 - 0.0299513 2026-02-23T13:16:43.113 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 197 8 365281 365273 14.4846 0 - 0.0299513 2026-02-23T13:16:43.227 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:16:43.347 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:16:43.446 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:16:43.444+0000 7f1893e5e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:16:43.446 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:16:43.444+0000 7f1893e5e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:16:43.458 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:16:43.458 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:16:43.468 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:16:43.469 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:16:43.469 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:16:43.478 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:16:43.479 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:16:43.509 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:16:43 trial068 ceph-mon[38805]: pgmap v442: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:16:43.510 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:16:43.510 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:16:44.114 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 198 8 365281 365273 14.4115 0 - 0.0299513 2026-02-23T13:16:44.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:16:43 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:16:43] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:16:44.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:16:43 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:16:43.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:16:44.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:16:44 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1485056637' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:16:45.114 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 199 8 365281 365273 14.339 0 - 0.0299513 2026-02-23T13:16:45.512 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:16:45.631 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:16:45.653 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:16:45 trial068 ceph-mon[38805]: pgmap v443: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:16:45.730 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:16:45.728+0000 7f839a509640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:16:45.730 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:16:45.728+0000 7f839a509640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:16:45.742 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:16:45.742 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:16:45.752 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:16:45.753 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:16:45.753 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:16:45.763 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:16:45.764 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:16:45.793 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:16:45.793 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:16:46.113 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T13:16:46.113737+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T13:16:46.114 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T13:16:46.114 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 200 8 365281 365273 14.2673 0 - 0.0299513 2026-02-23T13:16:46.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:16:46 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3576499372' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:16:47.114 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 201 8 365281 365273 14.1964 0 - 0.0299513 2026-02-23T13:16:47.744 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:16:47 trial068 ceph-mon[38805]: pgmap v444: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:16:47.794 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:16:47.911 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:16:48.010 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:16:48.008+0000 7f204110f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:16:48.010 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:16:48.009+0000 7f204110f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:16:48.023 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:16:48.023 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:16:48.034 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:16:48.034 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:16:48.034 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:16:48.044 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:16:48.044 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:16:48.074 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:16:48.074 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:16:48.113 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 202 8 365281 365273 14.1261 0 - 0.0299513 2026-02-23T13:16:48.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:16:48 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3888247321' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:16:49.114 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 203 8 365281 365273 14.0565 0 - 0.0299513 2026-02-23T13:16:49.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:16:49 trial068 ceph-mon[38805]: pgmap v445: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:16:50.075 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:16:50.114 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 204 8 365281 365273 13.9876 0 - 0.0299513 2026-02-23T13:16:50.192 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:16:50.292 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:16:50.291+0000 7f294dd64640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:16:50.293 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:16:50.291+0000 7f294dd64640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:16:50.304 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:16:50.304 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:16:50.314 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:16:50.314 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:16:50.315 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:16:50.324 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:16:50.324 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:16:50.354 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:16:50.354 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:16:50.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:16:50 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2495513788' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:16:51.114 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 205 8 365281 365273 13.9194 0 - 0.0299513 2026-02-23T13:16:51.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:16:51 trial068 ceph-mon[38805]: pgmap v446: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:16:52.114 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 206 8 365281 365273 13.8518 0 - 0.0299513 2026-02-23T13:16:52.356 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:16:52.476 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:16:52.573 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:16:52.571+0000 7f72418e1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:16:52.573 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:16:52.571+0000 7f72418e1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:16:52.585 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:16:52.585 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:16:52.596 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:16:52.596 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:16:52.596 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:16:52.607 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:16:52.607 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:16:52.637 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:16:52.638 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:16:53.114 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 207 8 365281 365273 13.7849 0 - 0.0299513 2026-02-23T13:16:53.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:16:53 trial068 ceph-mon[38805]: pgmap v447: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:16:53.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:16:53 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3718622254' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:16:54.114 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 208 8 365281 365273 13.7186 0 - 0.0299513 2026-02-23T13:16:54.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:16:53 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:16:53] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:16:54.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:16:53 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:16:53.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:16:54.639 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:16:54.758 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:16:54.859 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:16:54.857+0000 7f55d5f5c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:16:54.859 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:16:54.857+0000 7f55d5f5c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:16:54.871 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:16:54.871 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:16:54.881 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:16:54.881 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:16:54.881 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:16:54.891 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:16:54.891 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:16:54.921 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:16:54.921 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:16:55.114 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 209 8 365281 365273 13.653 0 - 0.0299513 2026-02-23T13:16:55.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:16:55 trial068 ceph-mon[38805]: pgmap v448: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:16:55.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:16:55 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2686230501' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:16:56.114 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 210 8 365281 365273 13.588 0 - 0.0299513 2026-02-23T13:16:56.921 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:16:57.039 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:16:57.114 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 211 8 365281 365273 13.5236 0 - 0.0299513 2026-02-23T13:16:57.139 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:16:57.138+0000 7fe4eba20640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:16:57.140 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:16:57.138+0000 7fe4eba20640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:16:57.152 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:16:57.152 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:16:57.162 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:16:57.162 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:16:57.162 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:16:57.172 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:16:57.172 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:16:57.203 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:16:57.204 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:16:57.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:16:57 trial068 ceph-mon[38805]: pgmap v449: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:16:57.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:16:57 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/927290562' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:16:58.115 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 212 8 365281 365273 13.4598 0 - 0.0299513 2026-02-23T13:16:59.114 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 213 8 365281 365273 13.3966 0 - 0.0299513 2026-02-23T13:16:59.204 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:16:59.322 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:16:59.421 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:16:59.419+0000 7fce1c0e1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:16:59.421 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:16:59.419+0000 7fce1c0e1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:16:59.433 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:16:59.433 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:16:59.443 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:16:59.443 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:16:59.443 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:16:59.453 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:16:59.453 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:16:59.482 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:16:59 trial068 ceph-mon[38805]: pgmap v450: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:16:59.483 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:16:59.483 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:17:00.115 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 214 8 365281 365273 13.334 0 - 0.0299513 2026-02-23T13:17:00.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:17:00 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1771734891' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:17:00.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:17:00 trial068 ceph-mon[38805]: 5.e scrub starts 2026-02-23T13:17:00.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:17:00 trial068 ceph-mon[38805]: 5.e scrub ok 2026-02-23T13:17:01.115 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 215 8 365281 365273 13.272 0 - 0.0299513 2026-02-23T13:17:01.485 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:17:01.604 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:17:01.645 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:17:01 trial068 ceph-mon[38805]: pgmap v451: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:17:01.699 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:17:01.698+0000 7f2507032640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:17:01.699 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:17:01.698+0000 7f2507032640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:17:01.711 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:17:01.711 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:17:01.722 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:17:01.722 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:17:01.722 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:17:01.734 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:17:01.734 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:17:01.766 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:17:01.767 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:17:02.115 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 216 8 365281 365273 13.2105 0 - 0.0299513 2026-02-23T13:17:02.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:17:02 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/393693069' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:17:03.115 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 217 8 365281 365273 13.1496 0 - 0.0299513 2026-02-23T13:17:03.767 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:17:03.792 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:17:03 trial068 ceph-mon[38805]: pgmap v452: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:17:03.886 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:17:03.982 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:17:03.980+0000 7f7b5149b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:17:03.982 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:17:03.981+0000 7f7b5149b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:17:03.993 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:17:03.994 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:17:04.003 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:17:04.003 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:17:04.003 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:17:04.013 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:17:04.013 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:17:04.042 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:17:03 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:17:03] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:17:04.044 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:17:04.044 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:17:04.115 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 218 8 365281 365273 13.0893 0 - 0.0299513 2026-02-23T13:17:04.266 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:17:03 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:17:03.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:17:04.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:17:04 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T13:17:04.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:17:04 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T13:17:04.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:17:04 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2416926267' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:17:05.115 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 219 8 365281 365273 13.0295 0 - 0.0299513 2026-02-23T13:17:05.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:17:05 trial068 ceph-mon[38805]: pgmap v453: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:17:06.045 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:17:06.115 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T13:17:06.115311+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T13:17:06.115 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T13:17:06.116 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 220 8 365281 365273 12.9703 0 - 0.0299513 2026-02-23T13:17:06.163 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:17:06.263 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:17:06.262+0000 7efcbf577640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:17:06.264 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:17:06.262+0000 7efcbf577640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:17:06.275 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:17:06.275 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:17:06.285 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:17:06.286 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:17:06.286 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:17:06.295 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:17:06.295 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:17:06.325 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:17:06.326 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:17:06.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:17:06 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/901704661' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:17:07.115 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 221 8 365281 365273 12.9116 0 - 0.0299513 2026-02-23T13:17:07.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:17:07 trial068 ceph-mon[38805]: pgmap v454: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:17:08.115 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 222 8 365281 365273 12.8535 0 - 0.0299513 2026-02-23T13:17:08.327 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:17:08.443 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:17:08.542 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:17:08.541+0000 7f36baf62640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:17:08.543 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:17:08.541+0000 7f36baf62640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:17:08.555 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:17:08.555 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:17:08.565 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:17:08.565 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:17:08.565 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:17:08.575 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:17:08.576 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:17:08.603 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:17:08.604 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:17:09.116 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 223 8 365281 365273 12.7958 0 - 0.0299513 2026-02-23T13:17:09.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:17:09 trial068 ceph-mon[38805]: pgmap v455: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:17:09.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:17:09 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2781772080' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:17:10.115 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 224 8 365281 365273 12.7387 0 - 0.0299513 2026-02-23T13:17:10.605 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:17:10.723 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:17:10.825 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:17:10.824+0000 7f2ebc79b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:17:10.825 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:17:10.824+0000 7f2ebc79b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:17:10.837 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:17:10.837 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:17:10.847 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:17:10.847 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:17:10.847 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:17:10.857 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:17:10.857 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:17:10.885 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:17:10.886 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:17:11.115 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 225 8 365281 365273 12.6821 0 - 0.0299513 2026-02-23T13:17:11.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:17:11 trial068 ceph-mon[38805]: pgmap v456: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:17:11.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:17:11 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2078212826' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:17:12.116 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 226 8 365281 365273 12.626 0 - 0.0299513 2026-02-23T13:17:12.886 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:17:13.004 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:17:13.108 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:17:13.106+0000 7fb84bf78640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:17:13.108 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:17:13.106+0000 7fb84bf78640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:17:13.116 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 227 8 365281 365273 12.5704 0 - 0.0299513 2026-02-23T13:17:13.119 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:17:13.120 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:17:13.130 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:17:13.130 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:17:13.130 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:17:13.140 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:17:13.140 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:17:13.171 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:17:13.172 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:17:13.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:17:13 trial068 ceph-mon[38805]: pgmap v457: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:17:13.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:17:13 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2310709804' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:17:14.116 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 228 8 365281 365273 12.5152 0 - 0.0299513 2026-02-23T13:17:14.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:17:13 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:17:13] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:17:14.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:17:13 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:17:13.974Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:17:15.116 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 229 8 365281 365273 12.4606 0 - 0.0299513 2026-02-23T13:17:15.173 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:17:15.293 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:17:15.315 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:17:15 trial068 ceph-mon[38805]: pgmap v458: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:17:15.395 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:17:15.394+0000 7fc1e247d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:17:15.396 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:17:15.394+0000 7fc1e247d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:17:15.407 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:17:15.407 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:17:15.417 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:17:15.418 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:17:15.418 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:17:15.428 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:17:15.428 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:17:15.459 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:17:15.459 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:17:16.116 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 230 8 365281 365273 12.4064 0 - 0.0299513 2026-02-23T13:17:16.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:17:16 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2173935305' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:17:17.116 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 231 8 365281 365273 12.3527 0 - 0.0299513 2026-02-23T13:17:17.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:17:17 trial068 ceph-mon[38805]: pgmap v459: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:17:17.460 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:17:17.577 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:17:17.673 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:17:17.671+0000 7f026a300640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:17:17.673 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:17:17.672+0000 7f026a300640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:17:17.685 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:17:17.686 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:17:17.695 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:17:17.696 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:17:17.696 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:17:17.705 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:17:17.706 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:17:17.733 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:17:17.734 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:17:18.116 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 232 8 365281 365273 12.2994 0 - 0.0299513 2026-02-23T13:17:18.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:17:18 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3402995892' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:17:19.116 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 233 8 365281 365273 12.2467 0 - 0.0299513 2026-02-23T13:17:19.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:17:19 trial068 ceph-mon[38805]: pgmap v460: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:17:19.735 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:17:19.852 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:17:19.950 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:17:19.948+0000 7fb6438ae640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:17:19.950 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:17:19.948+0000 7fb6438ae640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:17:19.961 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:17:19.962 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:17:19.971 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:17:19.972 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:17:19.972 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:17:19.982 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:17:19.982 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:17:20.011 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:17:20.012 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:17:20.116 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 234 8 365281 365273 12.1943 0 - 0.0299513 2026-02-23T13:17:20.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:17:20 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/932975466' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:17:21.116 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 235 8 365281 365273 12.1424 0 - 0.0299513 2026-02-23T13:17:21.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:17:21 trial068 ceph-mon[38805]: pgmap v461: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:17:22.013 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:17:22.116 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 236 8 365281 365273 12.091 0 - 0.0299513 2026-02-23T13:17:22.132 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:17:22.229 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:17:22.227+0000 7f57a8c94640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:17:22.229 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:17:22.228+0000 7f57a8c94640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:17:22.241 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:17:22.241 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:17:22.251 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:17:22.251 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:17:22.251 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:17:22.261 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:17:22.261 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:17:22.289 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:17:22 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/4152266268' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:17:22.290 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:17:22.291 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:17:23.116 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 237 8 365281 365273 12.04 0 - 0.0299513 2026-02-23T13:17:23.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:17:23 trial068 ceph-mon[38805]: pgmap v462: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:17:24.116 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 238 8 365281 365273 11.9894 0 - 0.0299513 2026-02-23T13:17:24.291 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:17:24.315 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:17:23 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:17:23] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:17:24.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:17:23 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:17:23.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:17:24.408 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:17:24.507 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:17:24.505+0000 7fa07ae81640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:17:24.507 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:17:24.505+0000 7fa07ae81640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:17:24.519 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:17:24.519 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:17:24.529 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:17:24.529 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:17:24.529 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:17:24.539 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:17:24.539 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:17:24.567 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:17:24.568 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:17:25.117 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 239 8 365281 365273 11.9392 0 - 0.0299513 2026-02-23T13:17:25.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:17:25 trial068 ceph-mon[38805]: pgmap v463: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:17:25.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:17:25 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3791493560' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:17:26.117 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T13:17:26.116855+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T13:17:26.117 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T13:17:26.117 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 240 8 365281 365273 11.8895 0 - 0.0299513 2026-02-23T13:17:26.569 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:17:26.687 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:17:26.799 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:17:26.797+0000 7f7020f3a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:17:26.799 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:17:26.797+0000 7f7020f3a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:17:26.811 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:17:26.811 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:17:26.821 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:17:26.822 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:17:26.822 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:17:26.832 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:17:26.832 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:17:26.863 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:17:26.864 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:17:27.117 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 241 8 365281 365273 11.8401 0 - 0.0299513 2026-02-23T13:17:27.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:17:27 trial068 ceph-mon[38805]: pgmap v464: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:17:27.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:17:27 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1287058221' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:17:28.117 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 242 8 365281 365273 11.7912 0 - 0.0299513 2026-02-23T13:17:28.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:17:28 trial068 ceph-mon[38805]: 5.5 scrub starts 2026-02-23T13:17:28.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:17:28 trial068 ceph-mon[38805]: 5.5 scrub ok 2026-02-23T13:17:28.865 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:17:28.983 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:17:29.092 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:17:29.090+0000 7f262ce0e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:17:29.092 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:17:29.091+0000 7f262ce0e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:17:29.103 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:17:29.104 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:17:29.114 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:17:29.114 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:17:29.114 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:17:29.117 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 243 8 365281 365273 11.7427 0 - 0.0299513 2026-02-23T13:17:29.124 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:17:29.124 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:17:29.154 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:17:29.154 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:17:29.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:17:29 trial068 ceph-mon[38805]: pgmap v465: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:17:29.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:17:29 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3974883893' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:17:30.117 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 244 8 365281 365273 11.6945 0 - 0.0299513 2026-02-23T13:17:30.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:17:30 trial068 ceph-mon[38805]: 5.a scrub starts 2026-02-23T13:17:30.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:17:30 trial068 ceph-mon[38805]: 5.a scrub ok 2026-02-23T13:17:30.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:17:30 trial068 ceph-mon[38805]: 5.7 scrub starts 2026-02-23T13:17:30.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:17:30 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T13:17:30.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:17:30 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T13:17:30.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:17:30 trial068 ceph-mon[38805]: 5.7 scrub ok 2026-02-23T13:17:31.117 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 245 8 365281 365273 11.6468 0 - 0.0299513 2026-02-23T13:17:31.155 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:17:31.275 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:17:31.376 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:17:31.374+0000 7fb58ae2c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:17:31.377 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:17:31.375+0000 7fb58ae2c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:17:31.389 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:17:31.389 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:17:31.399 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:17:31.399 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:17:31.400 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:17:31.410 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:17:31.411 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:17:31.439 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:17:31 trial068 ceph-mon[38805]: pgmap v466: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:17:31.440 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:17:31 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/703123827' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:17:31.440 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:17:31.441 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:17:32.117 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 246 8 365281 365273 11.5995 0 - 0.0299513 2026-02-23T13:17:33.117 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 247 8 365281 365273 11.5525 0 - 0.0299513 2026-02-23T13:17:33.443 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:17:33.562 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:17:33.658 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:17:33.656+0000 7f7eb8e1b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:17:33.658 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:17:33.657+0000 7f7eb8e1b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:17:33.659 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:17:33 trial068 ceph-mon[38805]: 5.8 scrub starts 2026-02-23T13:17:33.659 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:17:33 trial068 ceph-mon[38805]: 5.8 scrub ok 2026-02-23T13:17:33.659 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:17:33 trial068 ceph-mon[38805]: pgmap v467: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:17:33.659 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:17:33 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2026-02-23T13:17:33.659 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:17:33 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T13:17:33.659 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:17:33 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T13:17:33.670 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:17:33.671 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:17:33.681 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:17:33.681 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:17:33.681 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:17:33.692 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:17:33.692 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:17:33.723 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:17:33.723 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:17:34.117 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 248 8 365281 365273 11.5059 0 - 0.0299513 2026-02-23T13:17:34.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:17:33 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:17:33] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:17:34.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:17:33 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:17:33.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:17:34.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:17:34 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T13:17:34.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:17:34 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T13:17:34.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:17:34 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2026-02-23T13:17:34.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:17:34 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2026-02-23T13:17:34.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:17:34 trial068 ceph-mon[38805]: Adjusting osd_memory_target on trial068 to 41511M 2026-02-23T13:17:34.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:17:34 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T13:17:34.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:17:34 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"} : dispatch 2026-02-23T13:17:34.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:17:34 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"} : dispatch 2026-02-23T13:17:34.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:17:34 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T13:17:34.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:17:34 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-23T13:17:34.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:17:34 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2026-02-23T13:17:34.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:17:34 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T13:17:34.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:17:34 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2026-02-23T13:17:34.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:17:34 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/615773036' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:17:35.117 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 249 8 365281 365273 11.4597 0 - 0.0299513 2026-02-23T13:17:35.725 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:17:35.749 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:17:35 trial068 ceph-mon[38805]: Adjusting osd_memory_target on trial185 to 43559M 2026-02-23T13:17:35.750 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:17:35 trial068 ceph-mon[38805]: pgmap v468: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:17:35.750 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:17:35 trial068 ceph-mon[38805]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2026-02-23T13:17:35.842 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:17:35.939 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:17:35.937+0000 7f363663b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:17:35.939 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:17:35.938+0000 7f363663b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:17:35.952 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:17:35.952 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:17:35.962 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:17:35.962 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:17:35.962 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:17:35.973 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:17:35.973 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:17:36.003 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:17:36.004 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:17:36.117 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 250 8 365281 365273 11.4139 0 - 0.0299513 2026-02-23T13:17:36.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:17:36 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2671001916' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:17:37.117 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 251 8 365281 365273 11.3684 0 - 0.0299513 2026-02-23T13:17:37.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:17:37 trial068 ceph-mon[38805]: pgmap v469: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:17:38.004 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:17:38.118 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 252 8 365281 365273 11.3233 0 - 0.0299513 2026-02-23T13:17:38.122 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:17:38.220 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:17:38.218+0000 7fcfaf809640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:17:38.220 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:17:38.219+0000 7fcfaf809640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:17:38.233 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:17:38.234 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:17:38.244 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:17:38.244 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:17:38.245 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:17:38.255 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:17:38.256 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:17:38.285 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:17:38.285 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:17:38.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:17:38 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1853323853' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:17:39.118 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 253 8 365281 365273 11.2785 0 - 0.0299513 2026-02-23T13:17:39.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:17:39 trial068 ceph-mon[38805]: pgmap v470: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:17:40.118 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 254 8 365281 365273 11.2341 0 - 0.0299513 2026-02-23T13:17:40.286 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:17:40.404 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:17:40.504 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:17:40.502+0000 7f7bb9515640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:17:40.504 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:17:40.503+0000 7f7bb9515640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:17:40.517 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:17:40.517 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:17:40.527 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:17:40.528 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:17:40.528 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:17:40.538 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:17:40.538 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:17:40.566 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:17:40.566 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:17:41.118 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 255 8 365281 365273 11.1901 0 - 0.0299513 2026-02-23T13:17:41.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:17:41 trial068 ceph-mon[38805]: pgmap v471: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:17:41.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:17:41 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2434833494' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:17:42.118 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 256 8 365281 365273 11.1464 0 - 0.0299513 2026-02-23T13:17:42.567 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:17:42.684 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:17:42.780 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:17:42.779+0000 7f597c333640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:17:42.780 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:17:42.779+0000 7f597c333640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:17:42.792 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:17:42.792 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:17:42.802 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:17:42.803 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:17:42.803 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:17:42.812 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:17:42.813 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:17:42.843 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:17:42.844 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:17:43.118 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 257 8 365281 365273 11.103 0 - 0.0299513 2026-02-23T13:17:43.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:17:43 trial068 ceph-mon[38805]: pgmap v472: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:17:43.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:17:43 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1575177226' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:17:44.118 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 258 8 365281 365273 11.06 0 - 0.0299513 2026-02-23T13:17:44.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:17:43 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:17:43] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:17:44.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:17:43 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:17:43.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:17:44.845 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:17:44.964 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:17:45.063 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:17:45.061+0000 7f1116e9c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:17:45.063 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:17:45.062+0000 7f1116e9c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:17:45.075 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:17:45.075 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:17:45.085 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:17:45.086 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:17:45.086 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:17:45.096 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:17:45.096 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:17:45.118 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 259 8 365281 365273 11.0173 0 - 0.0299513 2026-02-23T13:17:45.125 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:17:45.125 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:17:45.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:17:45 trial068 ceph-mon[38805]: pgmap v473: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:17:45.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:17:45 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3062263403' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:17:46.118 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T13:17:46.118437+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T13:17:46.118 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T13:17:46.119 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 260 8 365281 365273 10.9749 0 - 0.0299513 2026-02-23T13:17:47.118 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 261 8 365281 365273 10.9328 0 - 0.0299513 2026-02-23T13:17:47.126 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:17:47.246 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:17:47.347 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:17:47.345+0000 7f6ad61e5640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:17:47.347 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:17:47.346+0000 7f6ad61e5640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:17:47.359 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:17:47.359 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:17:47.369 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:17:47.369 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:17:47.370 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:17:47.379 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:17:47.379 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:17:47.408 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:17:47 trial068 ceph-mon[38805]: pgmap v474: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:17:47.408 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:17:47 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2878140638' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:17:47.409 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:17:47.410 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:17:48.118 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 262 8 365281 365273 10.8911 0 - 0.0299513 2026-02-23T13:17:49.118 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 263 8 365281 365273 10.8497 0 - 0.0299513 2026-02-23T13:17:49.410 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:17:49.528 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:17:49.625 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:17:49.624+0000 7ff5142f6640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:17:49.625 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:17:49.624+0000 7ff5142f6640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:17:49.637 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:17:49.638 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:17:49.648 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:17:49.648 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:17:49.648 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:17:49 trial068 ceph-mon[38805]: pgmap v475: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:17:49.648 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:17:49 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T13:17:49.649 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:17:49 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T13:17:49.649 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:17:49.658 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:17:49.658 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:17:49.688 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:17:49.688 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:17:50.119 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 264 8 365281 365273 10.8086 0 - 0.0299513 2026-02-23T13:17:50.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:17:50 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/4113314510' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:17:51.119 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 265 8 365281 365273 10.7678 0 - 0.0299513 2026-02-23T13:17:51.689 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:17:51.714 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:17:51 trial068 ceph-mon[38805]: pgmap v476: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:17:51.806 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:17:51.905 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:17:51.903+0000 7f3ab4281640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:17:51.905 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:17:51.904+0000 7f3ab4281640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:17:51.917 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:17:51.918 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:17:51.928 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:17:51.928 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:17:51.928 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:17:51.938 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:17:51.939 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:17:51.968 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:17:51.969 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:17:52.119 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 266 8 365281 365273 10.7273 0 - 0.0299513 2026-02-23T13:17:52.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:17:52 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1059423498' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:17:53.119 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 267 8 365281 365273 10.6872 0 - 0.0299513 2026-02-23T13:17:53.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:17:53 trial068 ceph-mon[38805]: pgmap v477: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:17:53.970 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:17:54.087 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:17:54.119 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 268 8 365281 365273 10.6473 0 - 0.0299513 2026-02-23T13:17:54.191 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:17:54.189+0000 7fdbcf6ca640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:17:54.191 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:17:54.190+0000 7fdbcf6ca640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:17:54.203 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:17:54.203 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:17:54.213 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:17:54.213 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:17:54.214 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:17:54.223 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:17:54.223 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:17:54.224 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:17:53 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:17:53] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:17:54.251 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:17:54.252 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:17:54.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:17:53 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:17:53.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:17:54.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:17:54 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/385721182' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:17:55.119 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 269 8 365281 365273 10.6077 0 - 0.0299513 2026-02-23T13:17:55.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:17:55 trial068 ceph-mon[38805]: pgmap v478: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:17:56.119 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 270 8 365281 365273 10.5684 0 - 0.0299513 2026-02-23T13:17:56.253 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:17:56.369 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:17:56.468 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:17:56.467+0000 7f47c2cc2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:17:56.468 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:17:56.467+0000 7f47c2cc2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:17:56.480 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:17:56.481 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:17:56.491 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:17:56.491 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:17:56.491 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:17:56.500 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:17:56.500 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:17:56.530 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:17:56.531 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:17:57.119 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 271 8 365281 365273 10.5294 0 - 0.0299513 2026-02-23T13:17:57.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:17:57 trial068 ceph-mon[38805]: pgmap v479: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:17:57.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:17:57 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/4248127805' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:17:58.119 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 272 8 365281 365273 10.4907 0 - 0.0299513 2026-02-23T13:17:58.531 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:17:58.649 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:17:58.747 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:17:58.745+0000 7fc902ef9640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:17:58.747 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:17:58.745+0000 7fc902ef9640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:17:58.758 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:17:58.759 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:17:58.769 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:17:58.769 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:17:58.769 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:17:58.779 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:17:58.779 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:17:58.809 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:17:58.810 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:17:59.119 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 273 8 365281 365273 10.4523 0 - 0.0299513 2026-02-23T13:17:59.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:17:59 trial068 ceph-mon[38805]: pgmap v480: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:17:59.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:17:59 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1940467713' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:18:00.119 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 274 8 365281 365273 10.4141 0 - 0.0299513 2026-02-23T13:18:00.810 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:18:00.928 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:18:01.028 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:18:01.026+0000 7f98ea966640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:18:01.028 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:18:01.027+0000 7f98ea966640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:18:01.040 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:18:01.040 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:18:01.050 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:18:01.050 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:18:01.050 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:18:01.060 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:18:01.061 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:18:01.092 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:18:01.093 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:18:01.119 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 275 8 365281 365273 10.3763 0 - 0.0299513 2026-02-23T13:18:01.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:18:01 trial068 ceph-mon[38805]: pgmap v481: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:18:01.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:18:01 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3526759189' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:18:02.120 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 276 8 365281 365273 10.3387 0 - 0.0299513 2026-02-23T13:18:03.094 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:18:03.120 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 277 8 365281 365273 10.3013 0 - 0.0299513 2026-02-23T13:18:03.211 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:18:03.317 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:18:03.316+0000 7f9bafa42640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:18:03.318 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:18:03.316+0000 7f9bafa42640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:18:03.329 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:18:03.330 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:18:03.339 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:18:03.339 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:18:03.339 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:18:03.350 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:18:03.350 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:18:03.377 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:18:03.377 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:18:03.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:18:03 trial068 ceph-mon[38805]: pgmap v482: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:18:03.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:18:03 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/4229878342' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:18:04.120 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 278 8 365281 365273 10.2643 0 - 0.0299513 2026-02-23T13:18:04.266 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:18:03 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:18:03.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:18:04.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:18:03 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:18:03] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:18:05.120 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 279 8 365281 365273 10.2275 0 - 0.0299513 2026-02-23T13:18:05.379 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:18:05.497 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:18:05.595 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:18:05.593+0000 7fb2adc69640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:18:05.595 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:18:05.594+0000 7fb2adc69640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:18:05.607 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:18:05.607 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:18:05.617 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:18:05.617 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:18:05.617 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:18:05.627 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:18:05.627 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:18:05.656 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:18:05 trial068 ceph-mon[38805]: pgmap v483: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:18:05.657 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:18:05.657 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:18:06.120 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T13:18:06.119924+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T13:18:06.120 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T13:18:06.120 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 280 8 365281 365273 10.191 0 - 0.0299513 2026-02-23T13:18:06.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:18:06 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3521254575' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:18:07.120 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 281 8 365281 365273 10.1547 0 - 0.0299513 2026-02-23T13:18:07.658 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:18:07.682 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:18:07 trial068 ceph-mon[38805]: pgmap v484: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:18:07.683 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:18:07 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T13:18:07.683 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:18:07 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T13:18:07.776 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:18:07.871 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:18:07.870+0000 7f5c8a52f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:18:07.871 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:18:07.870+0000 7f5c8a52f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:18:07.883 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:18:07.883 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:18:07.894 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:18:07.894 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:18:07.895 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:18:07.904 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:18:07.904 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:18:07.935 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:18:07.936 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:18:08.120 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 282 8 365281 365273 10.1187 0 - 0.0299513 2026-02-23T13:18:08.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:18:08 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3591703671' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:18:09.120 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 283 8 365281 365273 10.0829 0 - 0.0299513 2026-02-23T13:18:09.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:18:09 trial068 ceph-mon[38805]: pgmap v485: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:18:09.937 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:18:10.055 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:18:10.120 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 284 8 365281 365273 10.0474 0 - 0.0299513 2026-02-23T13:18:10.153 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:18:10.152+0000 7fdd95808640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:18:10.153 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:18:10.152+0000 7fdd95808640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:18:10.165 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:18:10.165 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:18:10.175 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:18:10.175 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:18:10.175 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:18:10.185 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:18:10.185 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:18:10.216 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:18:10.216 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:18:10.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:18:10 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/4155825163' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:18:11.120 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 285 8 365281 365273 10.0122 0 - 0.0299513 2026-02-23T13:18:11.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:18:11 trial068 ceph-mon[38805]: pgmap v486: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:18:11.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:18:11 trial068 ceph-mon[38805]: 5.e scrub starts 2026-02-23T13:18:11.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:18:11 trial068 ceph-mon[38805]: 5.e scrub ok 2026-02-23T13:18:12.120 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 286 8 365281 365273 9.97717 0 - 0.0299513 2026-02-23T13:18:12.217 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:18:12.336 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:18:12.437 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:18:12.435+0000 7f39e070b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:18:12.437 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:18:12.436+0000 7f39e070b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:18:12.449 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:18:12.449 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:18:12.459 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:18:12.459 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:18:12.459 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:18:12.470 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:18:12.471 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:18:12.502 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:18:12.503 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:18:13.120 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 287 8 365281 365273 9.94241 0 - 0.0299513 2026-02-23T13:18:13.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:18:13 trial068 ceph-mon[38805]: pgmap v487: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:18:13.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:18:13 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/450128622' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:18:14.120 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 288 8 365281 365273 9.90789 0 - 0.0299513 2026-02-23T13:18:14.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:18:13 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:18:13] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:18:14.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:18:13 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:18:13.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:18:14.503 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:18:14.621 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:18:14.719 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:18:14.718+0000 7f9d0b98d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:18:14.719 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:18:14.718+0000 7f9d0b98d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:18:14.731 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:18:14.732 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:18:14.741 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:18:14.742 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:18:14.742 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:18:14.752 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:18:14.752 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:18:14.782 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:18:14.782 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:18:15.120 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 289 8 365281 365273 9.8736 0 - 0.0299513 2026-02-23T13:18:15.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:18:15 trial068 ceph-mon[38805]: pgmap v488: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:18:15.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:18:15 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2357411336' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:18:16.120 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 290 8 365281 365273 9.83955 0 - 0.0299513 2026-02-23T13:18:16.783 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:18:16.899 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:18:16.994 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:18:16.993+0000 7f7a81f94640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:18:16.995 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:18:16.993+0000 7f7a81f94640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:18:17.007 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:18:17.007 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:18:17.018 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:18:17.018 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:18:17.018 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:18:17.028 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:18:17.028 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:18:17.057 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:18:17.058 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:18:17.120 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 291 8 365281 365273 9.80574 0 - 0.0299513 2026-02-23T13:18:17.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:18:17 trial068 ceph-mon[38805]: pgmap v489: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:18:17.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:18:17 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2619604832' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:18:18.121 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 292 8 365281 365273 9.77216 0 - 0.0299513 2026-02-23T13:18:19.058 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:18:19.121 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 293 8 365281 365273 9.73881 0 - 0.0299513 2026-02-23T13:18:19.174 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:18:19.273 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:18:19.272+0000 7ff89e161640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:18:19.273 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:18:19.272+0000 7ff89e161640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:18:19.285 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:18:19.285 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:18:19.296 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:18:19.296 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:18:19.296 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:18:19.306 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:18:19.307 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:18:19.340 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:18:19.340 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:18:19.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:18:19 trial068 ceph-mon[38805]: pgmap v490: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:18:19.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:18:19 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/698374093' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:18:20.121 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 294 8 365281 365273 9.70568 0 - 0.0299513 2026-02-23T13:18:21.121 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 295 8 365281 365273 9.67278 0 - 0.0299513 2026-02-23T13:18:21.341 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:18:21.459 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:18:21.556 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:18:21.555+0000 7f64d5ff2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:18:21.557 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:18:21.555+0000 7f64d5ff2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:18:21.568 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:18:21.568 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:18:21.579 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:18:21.579 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:18:21.579 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:18:21.589 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:18:21.589 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:18:21.618 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:18:21 trial068 ceph-mon[38805]: pgmap v491: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:18:21.619 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:18:21.620 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:18:22.121 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 296 8 365281 365273 9.64011 0 - 0.0299513 2026-02-23T13:18:22.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:18:22 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2316751312' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:18:23.121 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 297 8 365281 365273 9.60765 0 - 0.0299513 2026-02-23T13:18:23.620 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:18:23.738 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:18:23.760 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:18:23 trial068 ceph-mon[38805]: pgmap v492: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:18:23.835 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:18:23.834+0000 7f2e48c4a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:18:23.835 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:18:23.834+0000 7f2e48c4a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:18:23.847 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:18:23.848 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:18:23.858 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:18:23.858 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:18:23.858 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:18:23.869 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:18:23.869 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:18:23.902 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:18:23.902 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:18:24.093 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:18:23 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:18:23] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:18:24.121 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 298 8 365281 365273 9.57541 0 - 0.0299513 2026-02-23T13:18:24.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:18:23 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:18:23.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:18:24.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:18:24 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2760271516' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:18:25.121 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 299 8 365281 365273 9.54338 0 - 0.0299513 2026-02-23T13:18:25.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:18:25 trial068 ceph-mon[38805]: pgmap v493: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:18:25.902 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:18:26.021 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:18:26.119 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:18:26.118+0000 7fda9f69d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:18:26.119 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:18:26.118+0000 7fda9f69d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:18:26.121 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T13:18:26.121469+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T13:18:26.121 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T13:18:26.121 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 300 8 365281 365273 9.51157 0 - 0.0299513 2026-02-23T13:18:26.132 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:18:26.132 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:18:26.143 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:18:26.143 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:18:26.143 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:18:26.154 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:18:26.154 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:18:26.184 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:18:26.184 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:18:26.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:18:26 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2462249708' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:18:27.121 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 301 8 365281 365273 9.47997 0 - 0.0299513 2026-02-23T13:18:27.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:18:27 trial068 ceph-mon[38805]: pgmap v494: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:18:28.121 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 302 8 365281 365273 9.44858 0 - 0.0299513 2026-02-23T13:18:28.185 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:18:28.304 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:18:28.398 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:18:28.396+0000 7f8fab807640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:18:28.398 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:18:28.396+0000 7f8fab807640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:18:28.410 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:18:28.410 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:18:28.420 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:18:28.420 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:18:28.420 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:18:28.430 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:18:28.430 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:18:28.460 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:18:28 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/593678163' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:18:28.461 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:18:28.461 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:18:29.121 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 303 8 365281 365273 9.4174 0 - 0.0299513 2026-02-23T13:18:29.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:18:29 trial068 ceph-mon[38805]: pgmap v495: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:18:30.121 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 304 8 365281 365273 9.38642 0 - 0.0299513 2026-02-23T13:18:30.463 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:18:30.580 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:18:30.679 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:18:30.678+0000 7fd28a444640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:18:30.679 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:18:30.678+0000 7fd28a444640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:18:30.691 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:18:30.691 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:18:30.701 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:18:30.701 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:18:30.701 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:18:30.711 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:18:30.711 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:18:30.740 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:18:30.740 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:18:31.121 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 305 8 365281 365273 9.35564 0 - 0.0299513 2026-02-23T13:18:31.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:18:31 trial068 ceph-mon[38805]: pgmap v496: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:18:31.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:18:31 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1861153999' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:18:32.121 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 306 8 365281 365273 9.32507 0 - 0.0299513 2026-02-23T13:18:32.742 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:18:32.859 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:18:32.959 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:18:32.958+0000 7f6d494d8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:18:32.959 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:18:32.958+0000 7f6d494d8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:18:32.971 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:18:32.971 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:18:32.981 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:18:32.981 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:18:32.982 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:18:32.991 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:18:32.992 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:18:33.021 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:18:33.021 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:18:33.122 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 307 8 365281 365273 9.29469 0 - 0.0299513 2026-02-23T13:18:33.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:18:33 trial068 ceph-mon[38805]: 5.5 scrub starts 2026-02-23T13:18:33.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:18:33 trial068 ceph-mon[38805]: pgmap v497: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:18:33.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:18:33 trial068 ceph-mon[38805]: 5.5 scrub ok 2026-02-23T13:18:33.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:18:33 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2002683859' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:18:34.122 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 308 8 365281 365273 9.26452 0 - 0.0299513 2026-02-23T13:18:34.265 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:18:33 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:18:33.972Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:18:34.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:18:33 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:18:33] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:18:34.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:18:34 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2026-02-23T13:18:34.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:18:34 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-23T13:18:34.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:18:34 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2026-02-23T13:18:34.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:18:34 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T13:18:34.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:18:34 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2026-02-23T13:18:35.023 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:18:35.122 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 309 8 365281 365273 9.23453 0 - 0.0299513 2026-02-23T13:18:35.140 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:18:35.240 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:18:35.239+0000 7fb9cd147640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:18:35.241 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:18:35.239+0000 7fb9cd147640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:18:35.252 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:18:35.253 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:18:35.263 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:18:35.263 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:18:35.263 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:18:35.275 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:18:35.275 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:18:35.305 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:18:35.305 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:18:35.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:18:35 trial068 ceph-mon[38805]: pgmap v498: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:18:35.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:18:35 trial068 ceph-mon[38805]: pgmap v499: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:18:35.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:18:35 trial068 ceph-mon[38805]: pgmap v500: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:18:35.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:18:35 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2169682797' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:18:36.122 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 310 8 365281 365273 9.20474 0 - 0.0299513 2026-02-23T13:18:37.122 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 311 8 365281 365273 9.17515 0 - 0.0299513 2026-02-23T13:18:37.306 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:18:37.423 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:18:37.520 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:18:37.519+0000 7feabff46640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:18:37.520 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:18:37.519+0000 7feabff46640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:18:37.533 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:18:37.533 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:18:37.543 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:18:37.543 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:18:37.543 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:18:37.554 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:18:37.554 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:18:37.584 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:18:37 trial068 ceph-mon[38805]: pgmap v501: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:18:37.585 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:18:37.586 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:18:38.122 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 312 8 365281 365273 9.14574 0 - 0.0299513 2026-02-23T13:18:38.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:18:38 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/867670213' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:18:39.122 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 313 8 365281 365273 9.11652 0 - 0.0299513 2026-02-23T13:18:39.587 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:18:39.704 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:18:39.726 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:18:39 trial068 ceph-mon[38805]: pgmap v502: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:18:39.804 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:18:39.802+0000 7f35fcf2a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:18:39.804 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:18:39.803+0000 7f35fcf2a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:18:39.816 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:18:39.816 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:18:39.826 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:18:39.826 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:18:39.826 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:18:39.836 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:18:39.836 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:18:39.866 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:18:39.867 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:18:40.122 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 314 8 365281 365273 9.08749 0 - 0.0299513 2026-02-23T13:18:40.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:18:40 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2728669881' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:18:41.122 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 315 8 365281 365273 9.05864 0 - 0.0299513 2026-02-23T13:18:41.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:18:41 trial068 ceph-mon[38805]: pgmap v503: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:18:41.868 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:18:41.986 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:18:42.081 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:18:42.079+0000 7feaffeb5640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:18:42.082 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:18:42.079+0000 7feaffeb5640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:18:42.093 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:18:42.093 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:18:42.103 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:18:42.103 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:18:42.103 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:18:42.113 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:18:42.113 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:18:42.122 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 316 8 365281 365273 9.02997 0 - 0.0299513 2026-02-23T13:18:42.144 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:18:42.145 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:18:42.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:18:42 trial068 ceph-mon[38805]: osdmap e189: 4 total, 3 up, 4 in 2026-02-23T13:18:42.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:18:42 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/250599562' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:18:43.123 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 317 8 365281 365273 9.00149 0 - 0.0299513 2026-02-23T13:18:43.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:18:43 trial068 ceph-mon[38805]: pgmap v505: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:18:44.123 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 318 8 365281 365273 8.97318 0 - 0.0299513 2026-02-23T13:18:44.146 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:18:44.264 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:18:44.287 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:18:43 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:18:43] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:18:44.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:18:43 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:18:43.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:18:44.363 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:18:44.362+0000 7f8c5148e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:18:44.364 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:18:44.362+0000 7f8c5148e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:18:44.375 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:18:44.376 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:18:44.386 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:18:44.386 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:18:44.386 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:18:44.397 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:18:44.397 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:18:44.427 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:18:44.427 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:18:45.123 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 319 8 365281 365273 8.94505 0 - 0.0299513 2026-02-23T13:18:45.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:18:45 trial068 ceph-mon[38805]: pgmap v506: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:18:45.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:18:45 trial068 ceph-mon[38805]: osdmap e190: 4 total, 3 up, 4 in 2026-02-23T13:18:45.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:18:45 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/4008101472' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:18:46.123 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T13:18:46.123099+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T13:18:46.123 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T13:18:46.123 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 320 8 365281 365273 8.9171 0 - 0.0299513 2026-02-23T13:18:46.429 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:18:46.548 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:18:46.647 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:18:46.645+0000 7f3b2c20a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:18:46.647 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:18:46.646+0000 7f3b2c20a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:18:46.659 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:18:46.659 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:18:46.669 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:18:46.669 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:18:46.669 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:18:46.679 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:18:46.679 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:18:46.709 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:18:46.710 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:18:47.123 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 321 8 365281 365273 8.88932 0 - 0.0299513 2026-02-23T13:18:47.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:18:47 trial068 ceph-mon[38805]: pgmap v508: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:18:47.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:18:47 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T13:18:47.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:18:47 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T13:18:47.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:18:47 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/4034151044' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:18:48.123 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 322 8 365281 365273 8.86171 0 - 0.0299513 2026-02-23T13:18:48.712 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:18:48.832 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:18:48.933 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:18:48.932+0000 7fbba00bd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:18:48.934 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:18:48.932+0000 7fbba00bd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:18:48.945 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:18:48.946 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:18:48.955 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:18:48.956 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:18:48.956 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:18:48.967 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:18:48.967 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:18:48.997 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:18:48.998 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:18:49.123 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 323 8 365281 365273 8.83427 0 - 0.0299513 2026-02-23T13:18:49.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:18:49 trial068 ceph-mon[38805]: 5.a scrub starts 2026-02-23T13:18:49.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:18:49 trial068 ceph-mon[38805]: pgmap v509: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:18:49.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:18:49 trial068 ceph-mon[38805]: 5.a scrub ok 2026-02-23T13:18:49.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:18:49 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/987574883' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:18:50.123 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 324 8 365281 365273 8.80701 0 - 0.0299513 2026-02-23T13:18:50.999 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:18:51.115 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:18:51.123 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 325 8 365281 365273 8.77991 0 - 0.0299513 2026-02-23T13:18:51.213 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:18:51.212+0000 7f7a4e94e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:18:51.213 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:18:51.212+0000 7f7a4e94e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:18:51.225 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:18:51.226 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:18:51.235 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:18:51.236 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:18:51.236 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:18:51.246 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:18:51.246 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:18:51.275 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:18:51 trial068 ceph-mon[38805]: pgmap v510: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:18:51.276 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:18:51.276 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:18:52.124 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 326 8 365281 365273 8.75298 0 - 0.0299513 2026-02-23T13:18:52.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:18:52 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/767922542' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:18:53.124 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 327 8 365281 365273 8.7262 0 - 0.0299513 2026-02-23T13:18:53.277 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:18:53.302 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:18:53 trial068 ceph-mon[38805]: pgmap v511: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:18:53.395 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:18:53.491 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:18:53.489+0000 7f9bf0d62640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:18:53.491 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:18:53.489+0000 7f9bf0d62640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:18:53.503 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:18:53.503 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:18:53.513 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:18:53.513 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:18:53.513 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:18:53.523 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:18:53.523 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:18:53.553 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:18:53.553 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:18:54.124 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 328 8 365281 365273 8.6996 0 - 0.0299513 2026-02-23T13:18:54.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:18:54 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/162892553' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:18:54.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:18:53 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:18:53] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:18:54.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:18:53 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:18:53.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:18:55.124 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 329 8 365281 365273 8.67316 0 - 0.0299513 2026-02-23T13:18:55.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:18:55 trial068 ceph-mon[38805]: pgmap v512: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:18:55.554 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:18:55.670 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:18:55.769 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:18:55.768+0000 7f71ef476640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:18:55.769 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:18:55.768+0000 7f71ef476640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:18:55.781 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:18:55.781 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:18:55.791 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:18:55.791 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:18:55.792 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:18:55.802 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:18:55.803 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:18:55.833 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:18:55.833 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:18:56.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:18:56 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/4184457538' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:18:56.124 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 330 8 365281 365273 8.64687 0 - 0.0299513 2026-02-23T13:18:57.124 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 331 8 365281 365273 8.62075 0 - 0.0299513 2026-02-23T13:18:57.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:18:57 trial068 ceph-mon[38805]: pgmap v513: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:18:57.835 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:18:57.956 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:18:58.065 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:18:58.063+0000 7fa1d1d03640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:18:58.065 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:18:58.063+0000 7fa1d1d03640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:18:58.077 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:18:58.077 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:18:58.088 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:18:58.088 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:18:58.088 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:18:58.100 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:18:58.100 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:18:58.124 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 332 8 365281 365273 8.59479 0 - 0.0299513 2026-02-23T13:18:58.131 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:18:58.131 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:18:59.124 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 333 8 365281 365273 8.56898 0 - 0.0299513 2026-02-23T13:18:59.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:18:59 trial068 ceph-mon[38805]: pgmap v514: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:18:59.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:18:59 trial068 ceph-mon[38805]: 5.7 scrub starts 2026-02-23T13:18:59.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:18:59 trial068 ceph-mon[38805]: 5.7 scrub ok 2026-02-23T13:18:59.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:18:59 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2828595827' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:19:00.124 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 334 8 365281 365273 8.54332 0 - 0.0299513 2026-02-23T13:19:00.132 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:19:00.252 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:19:00.357 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:19:00.356+0000 7f579870f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:19:00.357 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:19:00.356+0000 7f579870f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:19:00.369 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:19:00.369 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:19:00.379 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:19:00.380 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:19:00.380 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:19:00.389 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:19:00.389 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:19:00.420 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:19:00.420 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:19:01.124 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 335 8 365281 365273 8.51782 0 - 0.0299513 2026-02-23T13:19:01.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:19:01 trial068 ceph-mon[38805]: pgmap v515: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:19:01.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:19:01 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1127523768' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:19:02.124 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 336 8 365281 365273 8.49247 0 - 0.0299513 2026-02-23T13:19:02.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:19:02 trial068 ceph-mon[38805]: 5.8 scrub starts 2026-02-23T13:19:02.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:19:02 trial068 ceph-mon[38805]: 5.8 scrub ok 2026-02-23T13:19:02.421 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:19:02.538 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:19:02.635 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:19:02.634+0000 7f3b9565a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:19:02.635 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:19:02.634+0000 7f3b9565a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:19:02.647 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:19:02.648 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:19:02.658 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:19:02.658 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:19:02.658 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:19:02.669 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:19:02.669 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:19:02.699 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:19:02.700 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:19:03.124 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 337 8 365281 365273 8.46727 0 - 0.0299513 2026-02-23T13:19:03.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:19:03 trial068 ceph-mon[38805]: pgmap v516: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:19:03.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:19:03 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1053376892' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:19:04.124 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 338 8 365281 365273 8.44222 0 - 0.0299513 2026-02-23T13:19:04.266 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:19:03 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:19:03.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:19:04.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:19:03 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:19:03] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:19:04.701 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:19:04.817 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:19:04.912 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:19:04.910+0000 7f6b93c4d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:19:04.912 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:19:04.910+0000 7f6b93c4d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:19:04.923 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:19:04.924 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:19:04.933 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:19:04.934 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:19:04.934 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:19:04.944 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:19:04.944 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:19:04.975 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:19:04.975 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:19:05.125 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 339 8 365281 365273 8.41731 0 - 0.0299513 2026-02-23T13:19:05.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:19:05 trial068 ceph-mon[38805]: pgmap v517: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:19:05.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:19:05 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3396683167' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:19:06.125 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T13:19:06.124863+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T13:19:06.125 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T13:19:06.125 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 340 8 365281 365273 8.39256 0 - 0.0299513 2026-02-23T13:19:06.977 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:19:07.095 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:19:07.125 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 341 8 365281 365273 8.36794 0 - 0.0299513 2026-02-23T13:19:07.193 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:19:07.191+0000 7fedab0b5640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:19:07.193 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:19:07.192+0000 7fedab0b5640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:19:07.205 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:19:07.205 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:19:07.215 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:19:07.215 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:19:07.215 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:19:07.225 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:19:07.226 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:19:07.255 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:19:07 trial068 ceph-mon[38805]: pgmap v518: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:19:07.256 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:19:07.256 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:19:08.125 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 342 8 365281 365273 8.34348 0 - 0.0299513 2026-02-23T13:19:08.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:19:08 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2983682799' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:19:09.125 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 343 8 365281 365273 8.31915 0 - 0.0299513 2026-02-23T13:19:09.257 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:19:09.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:19:09 trial068 ceph-mon[38805]: pgmap v519: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:19:09.373 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:19:09.475 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:19:09.474+0000 7f4bfc967640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:19:09.476 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:19:09.474+0000 7f4bfc967640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:19:09.488 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:19:09.488 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:19:09.498 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:19:09.498 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:19:09.498 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:19:09.509 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:19:09.509 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:19:09.538 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:19:09.539 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:19:10.125 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 344 8 365281 365273 8.29497 0 - 0.0299513 2026-02-23T13:19:10.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:19:10 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/374541605' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:19:11.125 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 345 8 365281 365273 8.27093 0 - 0.0299513 2026-02-23T13:19:11.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:19:11 trial068 ceph-mon[38805]: pgmap v520: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:19:11.539 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:19:11.657 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:19:11.755 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:19:11.754+0000 7fa9457e3640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:19:11.755 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:19:11.754+0000 7fa9457e3640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:19:11.767 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:19:11.767 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:19:11.777 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:19:11.777 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:19:11.778 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:19:11.787 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:19:11.788 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:19:11.817 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:19:11.818 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:19:12.125 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 346 8 365281 365273 8.24702 0 - 0.0299513 2026-02-23T13:19:12.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:19:12 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1254579590' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:19:13.125 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 347 8 365281 365273 8.22325 0 - 0.0299513 2026-02-23T13:19:13.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:19:13 trial068 ceph-mon[38805]: pgmap v521: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:19:13.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:19:13 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T13:19:13.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:19:13 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T13:19:13.819 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:19:13.939 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:19:14.037 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:19:14.035+0000 7fad2cc6e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:19:14.037 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:19:14.035+0000 7fad2cc6e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:19:14.048 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:19:14.048 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:19:14.059 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:19:14.059 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:19:14.059 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:19:14.069 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:19:14.069 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:19:14.098 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:19:14 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/208742604' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:19:14.098 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:19:13 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:19:13] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:19:14.099 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:19:14.099 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:19:14.125 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 348 8 365281 365273 8.19963 0 - 0.0299513 2026-02-23T13:19:14.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:19:13 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:19:13.972Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:19:15.125 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 349 8 365281 365273 8.17613 0 - 0.0299513 2026-02-23T13:19:15.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:19:15 trial068 ceph-mon[38805]: pgmap v522: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:19:16.101 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:19:16.125 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 350 8 365281 365273 8.15277 0 - 0.0299513 2026-02-23T13:19:16.217 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:19:16.314 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:19:16.313+0000 7fea7cadf640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:19:16.314 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:19:16.313+0000 7fea7cadf640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:19:16.326 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:19:16.326 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:19:16.336 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:19:16.337 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:19:16.337 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:19:16.347 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:19:16.347 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:19:16.377 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:19:16.378 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:19:17.125 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 351 8 365281 365273 8.12954 0 - 0.0299513 2026-02-23T13:19:17.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:19:17 trial068 ceph-mon[38805]: pgmap v523: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:19:17.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:19:17 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2120204523' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:19:18.126 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 352 8 365281 365273 8.10645 0 - 0.0299513 2026-02-23T13:19:18.380 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:19:18.499 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:19:18.596 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:19:18.594+0000 7fe4bc204640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:19:18.596 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:19:18.595+0000 7fe4bc204640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:19:18.608 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:19:18.608 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:19:18.618 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:19:18.619 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:19:18.619 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:19:18.628 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:19:18.628 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:19:18.662 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:19:18.663 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:19:19.126 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 353 8 365281 365273 8.08348 0 - 0.0299513 2026-02-23T13:19:19.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:19:19 trial068 ceph-mon[38805]: pgmap v524: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:19:19.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:19:19 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T13:19:19.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:19:19 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T13:19:19.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:19:19 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3877166567' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:19:20.125 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 354 8 365281 365273 8.06065 0 - 0.0299513 2026-02-23T13:19:20.664 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:19:20.781 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:19:20.883 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:19:20.882+0000 7fab58418640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:19:20.884 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:19:20.882+0000 7fab58418640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:19:20.895 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:19:20.896 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:19:20.905 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:19:20.905 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:19:20.906 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:19:20.915 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:19:20.915 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:19:20.942 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:19:20.943 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:19:21.126 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 355 8 365281 365273 8.03794 0 - 0.0299513 2026-02-23T13:19:21.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:19:21 trial068 ceph-mon[38805]: pgmap v525: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:19:21.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:19:21 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1160891767' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:19:22.126 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 356 8 365281 365273 8.01536 0 - 0.0299513 2026-02-23T13:19:22.944 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:19:23.062 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:19:23.126 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 357 8 365281 365273 7.99291 0 - 0.0299513 2026-02-23T13:19:23.157 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:19:23.156+0000 7f7b3b33c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:19:23.158 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:19:23.156+0000 7f7b3b33c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:19:23.170 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:19:23.170 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:19:23.180 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:19:23.180 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:19:23.180 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:19:23.191 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:19:23.191 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:19:23.220 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:19:23 trial068 ceph-mon[38805]: pgmap v526: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:19:23.221 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:19:23.221 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:19:24.126 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 358 8 365281 365273 7.97058 0 - 0.0299513 2026-02-23T13:19:24.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:19:24 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3294116149' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:19:24.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:19:23 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:19:23] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:19:24.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:19:23 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:19:23.974Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:19:25.126 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 359 8 365281 365273 7.94838 0 - 0.0299513 2026-02-23T13:19:25.222 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:19:25.340 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:19:25.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:19:25 trial068 ceph-mon[38805]: pgmap v527: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:19:25.438 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:19:25.437+0000 7f36a3b8f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:19:25.438 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:19:25.437+0000 7f36a3b8f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:19:25.450 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:19:25.450 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:19:25.460 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:19:25.461 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:19:25.461 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:19:25.470 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:19:25.470 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:19:25.501 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:19:25.502 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:19:26.126 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T13:19:26.126445+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T13:19:26.126 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T13:19:26.127 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 360 8 365281 365273 7.9263 0 - 0.0299513 2026-02-23T13:19:26.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:19:26 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/711982479' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:19:27.126 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 361 8 365281 365273 7.90435 0 - 0.0299513 2026-02-23T13:19:27.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:19:27 trial068 ceph-mon[38805]: pgmap v528: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:19:27.503 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:19:27.621 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:19:27.720 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:19:27.718+0000 7fb7d8b75640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:19:27.720 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:19:27.718+0000 7fb7d8b75640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:19:27.732 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:19:27.733 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:19:27.742 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:19:27.743 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:19:27.743 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:19:27.753 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:19:27.753 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:19:27.783 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:19:27.784 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:19:28.126 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 362 8 365281 365273 7.88251 0 - 0.0299513 2026-02-23T13:19:28.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:19:28 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2283566397' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:19:29.126 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 363 8 365281 365273 7.8608 0 - 0.0299513 2026-02-23T13:19:29.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:19:29 trial068 ceph-mon[38805]: pgmap v529: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:19:29.785 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:19:29.902 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:19:30.002 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:19:30.001+0000 7f9c86ec0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:19:30.002 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:19:30.001+0000 7f9c86ec0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:19:30.015 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:19:30.015 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:19:30.025 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:19:30.025 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:19:30.025 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:19:30.035 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:19:30.035 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:19:30.066 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:19:30.066 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:19:30.127 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 364 8 365281 365273 7.8392 0 - 0.0299513 2026-02-23T13:19:30.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:19:30 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3583970079' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:19:31.127 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 365 8 365281 365273 7.81772 0 - 0.0299513 2026-02-23T13:19:31.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:19:31 trial068 ceph-mon[38805]: pgmap v530: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:19:32.068 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:19:32.127 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 366 8 365281 365273 7.79636 0 - 0.0299513 2026-02-23T13:19:32.187 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:19:32.291 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:19:32.289+0000 7ff025857640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:19:32.291 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:19:32.289+0000 7ff025857640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:19:32.302 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:19:32.303 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:19:32.313 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:19:32.314 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:19:32.314 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:19:32.324 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:19:32.324 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:19:32.351 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:19:32.352 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:19:33.127 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 367 8 365281 365273 7.77512 0 - 0.0299513 2026-02-23T13:19:33.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:19:33 trial068 ceph-mon[38805]: pgmap v531: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:19:33.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:19:33 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1935637210' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:19:34.127 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 368 8 365281 365273 7.75399 0 - 0.0299513 2026-02-23T13:19:34.265 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:19:33 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:19:33.972Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:19:34.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:19:34 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2026-02-23T13:19:34.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:19:33 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:19:33] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:19:34.353 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:19:34.467 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:19:34.563 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:19:34.561+0000 7f0ac62c0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:19:34.563 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:19:34.561+0000 7f0ac62c0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:19:34.574 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:19:34.575 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:19:34.584 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:19:34.584 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:19:34.585 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:19:34.596 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:19:34.596 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:19:34.630 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:19:34.630 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:19:35.127 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 369 8 365281 365273 7.73298 0 - 0.0299513 2026-02-23T13:19:35.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:19:35 trial068 ceph-mon[38805]: pgmap v532: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:19:35.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:19:35 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-23T13:19:35.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:19:35 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2026-02-23T13:19:35.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:19:35 trial068 ceph-mon[38805]: pgmap v533: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:19:35.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:19:35 trial068 ceph-mon[38805]: pgmap v534: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:19:35.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:19:35 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T13:19:35.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:19:35 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2026-02-23T13:19:35.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:19:35 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1119156794' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:19:36.127 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 370 8 365281 365273 7.71208 0 - 0.0299513 2026-02-23T13:19:36.631 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:19:36.748 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:19:36.851 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:19:36.849+0000 7f481eedc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:19:36.851 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:19:36.850+0000 7f481eedc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:19:36.863 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:19:36.863 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:19:36.873 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:19:36.874 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:19:36.874 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:19:36.883 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:19:36.884 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:19:36.914 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:19:36.915 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:19:37.127 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 371 8 365281 365273 7.69129 0 - 0.0299513 2026-02-23T13:19:37.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:19:37 trial068 ceph-mon[38805]: pgmap v535: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:19:37.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:19:37 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2216107294' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:19:38.127 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 372 8 365281 365273 7.67062 0 - 0.0299513 2026-02-23T13:19:38.916 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:19:39.034 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:19:39.127 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 373 8 365281 365273 7.65005 0 - 0.0299513 2026-02-23T13:19:39.133 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:19:39.132+0000 7f05f9503640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:19:39.134 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:19:39.132+0000 7f05f9503640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:19:39.146 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:19:39.146 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:19:39.156 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:19:39.156 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:19:39.156 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:19:39.166 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:19:39.166 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:19:39.196 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:19:39 trial068 ceph-mon[38805]: pgmap v536: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:19:39.196 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:19:39 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2955160091' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:19:39.197 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:19:39.197 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:19:40.127 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 374 8 365281 365273 7.6296 0 - 0.0299513 2026-02-23T13:19:41.127 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 375 8 365281 365273 7.60925 0 - 0.0299513 2026-02-23T13:19:41.198 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:19:41.314 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:19:41.413 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:19:41.412+0000 7fa66d108640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:19:41.414 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:19:41.412+0000 7fa66d108640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:19:41.425 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:19:41.425 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:19:41.435 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:19:41.435 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:19:41.435 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:19:41.446 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:19:41.446 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:19:41.446 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:19:41 trial068 ceph-mon[38805]: pgmap v537: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:19:41.477 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:19:41.478 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:19:42.127 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 376 8 365281 365273 7.58902 0 - 0.0299513 2026-02-23T13:19:42.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:19:42 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/4150395673' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:19:43.127 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 377 8 365281 365273 7.56889 0 - 0.0299513 2026-02-23T13:19:43.479 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:19:43.505 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:19:43 trial068 ceph-mon[38805]: pgmap v538: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:19:43.599 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:19:43.700 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:19:43.698+0000 7f3c48019640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:19:43.700 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:19:43.698+0000 7f3c48019640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:19:43.712 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:19:43.712 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:19:43.722 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:19:43.722 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:19:43.722 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:19:43.732 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:19:43.732 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:19:43.761 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:19:43.762 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:19:44.093 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:19:43 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:19:43] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:19:44.127 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 378 8 365281 365273 7.54886 0 - 0.0299513 2026-02-23T13:19:44.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:19:43 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:19:43.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:19:44.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:19:44 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2059920537' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:19:45.128 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 379 8 365281 365273 7.52894 0 - 0.0299513 2026-02-23T13:19:45.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:19:45 trial068 ceph-mon[38805]: pgmap v539: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:19:45.763 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:19:45.880 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:19:45.980 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:19:45.979+0000 7f65fc605640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:19:45.981 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:19:45.979+0000 7f65fc605640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:19:45.993 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:19:45.993 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:19:46.003 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:19:46.003 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:19:46.003 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:19:46.014 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:19:46.014 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:19:46.047 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:19:46.047 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:19:46.128 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T13:19:46.127933+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T13:19:46.128 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T13:19:46.128 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 380 8 365281 365273 7.50913 0 - 0.0299513 2026-02-23T13:19:46.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:19:46 trial068 ceph-mon[38805]: 5.e scrub starts 2026-02-23T13:19:46.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:19:46 trial068 ceph-mon[38805]: 5.e scrub ok 2026-02-23T13:19:46.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:19:46 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2083289879' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:19:47.128 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 381 8 365281 365273 7.48942 0 - 0.0299513 2026-02-23T13:19:47.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:19:47 trial068 ceph-mon[38805]: pgmap v540: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:19:48.049 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:19:48.128 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 382 8 365281 365273 7.46982 0 - 0.0299513 2026-02-23T13:19:48.167 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:19:48.264 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:19:48.262+0000 7f1727c63640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:19:48.265 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:19:48.263+0000 7f1727c63640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:19:48.276 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:19:48.276 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:19:48.286 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:19:48.286 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:19:48.286 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:19:48.296 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:19:48.296 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:19:48.325 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:19:48.326 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:19:49.128 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 383 8 365281 365273 7.45031 0 - 0.0299513 2026-02-23T13:19:49.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:19:49 trial068 ceph-mon[38805]: pgmap v541: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:19:49.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:19:49 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3213086293' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:19:50.128 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 384 8 365281 365273 7.43091 0 - 0.0299513 2026-02-23T13:19:50.327 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:19:50.446 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:19:50.545 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:19:50.544+0000 7f9d374b4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:19:50.546 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:19:50.544+0000 7f9d374b4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:19:50.557 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:19:50.558 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:19:50.567 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:19:50.567 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:19:50.568 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:19:50.577 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:19:50.578 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:19:50.608 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:19:50.608 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:19:51.128 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 385 8 365281 365273 7.41161 0 - 0.0299513 2026-02-23T13:19:51.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:19:51 trial068 ceph-mon[38805]: pgmap v542: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:19:51.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:19:51 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1850567926' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:19:52.128 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 386 8 365281 365273 7.39241 0 - 0.0299513 2026-02-23T13:19:52.609 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:19:52.729 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:19:52.825 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:19:52.823+0000 7f76da241640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:19:52.825 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:19:52.823+0000 7f76da241640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:19:52.837 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:19:52.837 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:19:52.847 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:19:52.847 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:19:52.848 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:19:52.858 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:19:52.858 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:19:52.887 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:19:52.887 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:19:53.128 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 387 8 365281 365273 7.37331 0 - 0.0299513 2026-02-23T13:19:53.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:19:53 trial068 ceph-mon[38805]: pgmap v543: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:19:53.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:19:53 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/340757005' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:19:54.128 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 388 8 365281 365273 7.3543 0 - 0.0299513 2026-02-23T13:19:54.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:19:53 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:19:53] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:19:54.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:19:53 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:19:53.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:19:54.889 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:19:55.009 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:19:55.109 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:19:55.108+0000 7ff53c406640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:19:55.110 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:19:55.108+0000 7ff53c406640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:19:55.121 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:19:55.122 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:19:55.128 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 389 8 365281 365273 7.3354 0 - 0.0299513 2026-02-23T13:19:55.131 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:19:55.131 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:19:55.132 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:19:55.142 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:19:55.142 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:19:55.170 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:19:55.170 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:19:55.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:19:55 trial068 ceph-mon[38805]: pgmap v544: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:19:55.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:19:55 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/395663267' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:19:56.128 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 390 8 365281 365273 7.31659 0 - 0.0299513 2026-02-23T13:19:57.129 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 391 8 365281 365273 7.29788 0 - 0.0299513 2026-02-23T13:19:57.171 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:19:57.291 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:19:57.393 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:19:57.392+0000 7fb6829a4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:19:57.393 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:19:57.392+0000 7fb6829a4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:19:57.406 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:19:57.407 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:19:57.417 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:19:57.418 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:19:57.418 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:19:57.428 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:19:57.428 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:19:57.459 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:19:57 trial068 ceph-mon[38805]: 5.5 scrub starts 2026-02-23T13:19:57.459 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:19:57 trial068 ceph-mon[38805]: 5.5 scrub ok 2026-02-23T13:19:57.459 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:19:57 trial068 ceph-mon[38805]: pgmap v545: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:19:57.459 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:19:57 trial068 ceph-mon[38805]: osdmap e191: 4 total, 3 up, 4 in 2026-02-23T13:19:57.460 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:19:57.460 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:19:58.129 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 392 8 365281 365273 7.27926 0 - 0.0299513 2026-02-23T13:19:58.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:19:58 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/84504888' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:19:59.129 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 393 8 365281 365273 7.26074 0 - 0.0299513 2026-02-23T13:19:59.461 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:19:59.486 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:19:59 trial068 ceph-mon[38805]: pgmap v547: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:19:59.578 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:19:59.676 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:19:59.674+0000 7fc914e0c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:19:59.676 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:19:59.675+0000 7fc914e0c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:19:59.688 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:19:59.688 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:19:59.698 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:19:59.698 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:19:59.698 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:19:59.709 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:19:59.709 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:19:59.739 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:19:59.740 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:20:00.129 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 394 8 365281 365273 7.24231 0 - 0.0299513 2026-02-23T13:20:00.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:20:00 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/817030802' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:20:01.129 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 395 8 365281 365273 7.22397 0 - 0.0299513 2026-02-23T13:20:01.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:20:01 trial068 ceph-mon[38805]: 5.a scrub starts 2026-02-23T13:20:01.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:20:01 trial068 ceph-mon[38805]: 5.a scrub ok 2026-02-23T13:20:01.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:20:01 trial068 ceph-mon[38805]: pgmap v548: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:20:01.741 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:20:01.862 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:20:01.960 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:20:01.959+0000 7f2b94949640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:20:01.960 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:20:01.959+0000 7f2b94949640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:20:01.973 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:20:01.974 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:20:01.983 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:20:01.984 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:20:01.984 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:20:01.994 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:20:01.994 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:20:02.025 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:20:02.025 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:20:02.129 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 396 8 365281 365273 7.20573 0 - 0.0299513 2026-02-23T13:20:02.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:20:02 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1164536954' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:20:03.129 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 397 8 365281 365273 7.18758 0 - 0.0299513 2026-02-23T13:20:03.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:20:03 trial068 ceph-mon[38805]: pgmap v549: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:20:04.027 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:20:04.129 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 398 8 365281 365273 7.16952 0 - 0.0299513 2026-02-23T13:20:04.146 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:20:04.244 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:20:04.243+0000 7ff5fdebb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:20:04.244 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:20:03 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:20:03] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:20:04.245 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:20:04.243+0000 7ff5fdebb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:20:04.256 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:20:04.257 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:20:04.265 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:20:03 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:20:03.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:20:04.266 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:20:04.266 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:20:04.267 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:20:04.276 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:20:04.276 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:20:04.308 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:20:04.309 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:20:05.130 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 399 8 365281 365273 7.15155 0 - 0.0299513 2026-02-23T13:20:05.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:20:05 trial068 ceph-mon[38805]: pgmap v550: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:20:05.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:20:05 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1876703921' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:20:06.130 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T13:20:06.129811+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T13:20:06.130 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T13:20:06.130 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 400 8 365281 365273 7.13367 0 - 0.0299513 2026-02-23T13:20:06.309 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:20:06.427 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:20:06.526 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:20:06.524+0000 7f8be89b8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:20:06.526 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:20:06.524+0000 7f8be89b8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:20:06.538 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:20:06.538 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:20:06.548 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:20:06.549 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:20:06.549 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:20:06.559 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:20:06.559 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:20:06.590 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:20:06.590 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:20:07.130 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 401 8 365281 365273 7.11588 0 - 0.0299513 2026-02-23T13:20:07.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:20:07 trial068 ceph-mon[38805]: pgmap v551: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:20:07.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:20:07 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/944455917' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:20:08.130 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 402 8 365281 365273 7.09818 0 - 0.0299513 2026-02-23T13:20:08.592 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:20:08.710 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:20:08.814 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:20:08.813+0000 7f99857e9640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:20:08.814 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:20:08.813+0000 7f99857e9640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:20:08.826 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:20:08.827 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:20:08.837 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:20:08.837 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:20:08.837 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:20:08.847 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:20:08.847 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:20:08.878 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:20:08.879 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:20:09.130 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 403 8 365281 365273 7.08056 0 - 0.0299513 2026-02-23T13:20:09.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:20:09 trial068 ceph-mon[38805]: pgmap v552: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:20:09.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:20:09 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3453122863' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:20:10.130 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 404 8 365281 365273 7.06304 0 - 0.0299513 2026-02-23T13:20:10.880 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:20:10.999 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:20:11.094 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:20:11.093+0000 7f84b8eb7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:20:11.094 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:20:11.093+0000 7f84b8eb7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:20:11.106 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:20:11.107 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:20:11.117 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:20:11.118 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:20:11.118 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:20:11.127 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:20:11.128 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:20:11.130 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 405 8 365281 365273 7.0456 0 - 0.0299513 2026-02-23T13:20:11.157 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:20:11.158 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:20:11.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:20:11 trial068 ceph-mon[38805]: pgmap v553: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:20:11.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:20:11 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1697811039' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:20:12.130 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 406 8 365281 365273 7.02825 0 - 0.0299513 2026-02-23T13:20:12.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:20:12 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T13:20:12.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:20:12 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T13:20:13.130 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 407 8 365281 365273 7.01098 0 - 0.0299513 2026-02-23T13:20:13.159 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:20:13.276 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:20:13.379 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:20:13.377+0000 7f03698ff640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:20:13.379 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:20:13.377+0000 7f03698ff640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:20:13.391 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:20:13.391 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:20:13.401 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:20:13.402 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:20:13.402 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:20:13.411 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:20:13.412 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:20:13.441 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:20:13 trial068 ceph-mon[38805]: pgmap v554: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:20:13.442 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:20:13.443 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:20:14.130 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 408 8 365281 365273 6.99379 0 - 0.0299513 2026-02-23T13:20:14.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:20:14 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2276490161' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:20:14.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:20:13 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:20:13] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:20:14.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:20:13 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:20:13.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:20:15.130 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 409 8 365281 365273 6.97669 0 - 0.0299513 2026-02-23T13:20:15.443 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:20:15.467 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:20:15 trial068 ceph-mon[38805]: pgmap v555: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:20:15.560 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:20:15.660 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:20:15.659+0000 7fc147b98640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:20:15.661 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:20:15.659+0000 7fc147b98640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:20:15.672 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:20:15.672 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:20:15.683 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:20:15.683 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:20:15.683 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:20:15.693 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:20:15.693 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:20:15.721 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:20:15.722 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:20:16.130 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 410 8 365281 365273 6.95968 0 - 0.0299513 2026-02-23T13:20:16.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:20:16 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1249109055' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:20:17.130 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 411 8 365281 365273 6.94274 0 - 0.0299513 2026-02-23T13:20:17.494 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:20:17 trial068 ceph-mon[38805]: pgmap v556: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:20:17.723 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:20:17.843 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:20:17.945 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:20:17.944+0000 7f16560bf640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:20:17.946 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:20:17.944+0000 7f16560bf640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:20:17.957 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:20:17.957 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:20:17.967 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:20:17.967 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:20:17.967 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:20:17.977 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:20:17.977 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:20:18.010 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:20:18.011 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:20:18.131 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 412 8 365281 365273 6.92589 0 - 0.0299513 2026-02-23T13:20:18.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:20:18 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2120005023' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:20:19.130 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 413 8 365281 365273 6.90912 0 - 0.0299513 2026-02-23T13:20:19.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:20:19 trial068 ceph-mon[38805]: pgmap v557: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:20:20.011 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:20:20.126 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:20:20.130 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 414 8 365281 365273 6.89244 0 - 0.0299513 2026-02-23T13:20:20.227 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:20:20.226+0000 7f0bdd288640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:20:20.227 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:20:20.226+0000 7f0bdd288640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:20:20.239 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:20:20.240 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:20:20.249 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:20:20.250 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:20:20.250 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:20:20.259 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:20:20.259 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:20:20.289 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:20:20.290 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:20:21.131 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 415 8 365281 365273 6.87583 0 - 0.0299513 2026-02-23T13:20:21.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:20:21 trial068 ceph-mon[38805]: 5.8 scrub starts 2026-02-23T13:20:21.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:20:21 trial068 ceph-mon[38805]: 5.8 scrub ok 2026-02-23T13:20:21.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:20:21 trial068 ceph-mon[38805]: pgmap v558: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:20:21.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:20:21 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2434620942' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:20:22.131 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 416 8 365281 365273 6.8593 0 - 0.0299513 2026-02-23T13:20:22.291 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:20:22.411 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:20:22.464 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:20:22 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T13:20:22.464 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:20:22 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T13:20:22.509 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:20:22.508+0000 7f62dbd8a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:20:22.510 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:20:22.508+0000 7f62dbd8a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:20:22.522 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:20:22.522 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:20:22.532 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:20:22.532 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:20:22.532 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:20:22.542 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:20:22.542 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:20:22.573 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:20:22.574 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:20:23.131 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 417 8 365281 365273 6.84285 0 - 0.0299513 2026-02-23T13:20:23.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:20:23 trial068 ceph-mon[38805]: pgmap v559: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:20:23.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:20:23 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3044097313' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:20:24.131 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 418 8 365281 365273 6.82648 0 - 0.0299513 2026-02-23T13:20:24.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:20:23 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:20:23] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:20:24.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:20:23 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:20:23.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:20:24.574 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:20:24.692 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:20:24.796 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:20:24.794+0000 7f454a134640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:20:24.796 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:20:24.795+0000 7f454a134640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:20:24.808 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:20:24.808 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:20:24.818 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:20:24.819 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:20:24.819 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:20:24.828 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:20:24.829 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:20:24.856 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:20:24.857 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:20:25.131 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 419 8 365281 365273 6.81019 0 - 0.0299513 2026-02-23T13:20:25.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:20:25 trial068 ceph-mon[38805]: pgmap v560: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:20:25.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:20:25 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2004247888' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:20:26.131 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T13:20:26.131298+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T13:20:26.131 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T13:20:26.131 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 420 8 365281 365273 6.79397 0 - 0.0299513 2026-02-23T13:20:26.859 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:20:26.978 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:20:27.075 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:20:27.073+0000 7f1a89faf640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:20:27.075 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:20:27.073+0000 7f1a89faf640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:20:27.087 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:20:27.087 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:20:27.097 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:20:27.097 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:20:27.098 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:20:27.107 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:20:27.107 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:20:27.131 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 421 8 365281 365273 6.77783 0 - 0.0299513 2026-02-23T13:20:27.136 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:20:27.136 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:20:27.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:20:27 trial068 ceph-mon[38805]: pgmap v561: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:20:27.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:20:27 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3828797385' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:20:28.131 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 422 8 365281 365273 6.76177 0 - 0.0299513 2026-02-23T13:20:29.131 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 423 8 365281 365273 6.74579 0 - 0.0299513 2026-02-23T13:20:29.138 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:20:29.255 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:20:29.357 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:20:29.355+0000 7f798cda8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:20:29.357 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:20:29.355+0000 7f798cda8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:20:29.368 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:20:29.369 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:20:29.379 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:20:29.379 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:20:29.379 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:20:29.388 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:20:29.389 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:20:29.417 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:20:29 trial068 ceph-mon[38805]: pgmap v562: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:20:29.419 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:20:29.419 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:20:30.131 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 424 8 365281 365273 6.72988 0 - 0.0299513 2026-02-23T13:20:30.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:20:30 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1226581694' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:20:31.131 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 425 8 365281 365273 6.71404 0 - 0.0299513 2026-02-23T13:20:31.420 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:20:31.539 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:20:31.560 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:20:31 trial068 ceph-mon[38805]: 5.7 scrub starts 2026-02-23T13:20:31.560 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:20:31 trial068 ceph-mon[38805]: pgmap v563: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:20:31.560 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:20:31 trial068 ceph-mon[38805]: 5.7 scrub ok 2026-02-23T13:20:31.638 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:20:31.637+0000 7f3d68d64640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:20:31.638 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:20:31.637+0000 7f3d68d64640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:20:31.650 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:20:31.650 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:20:31.661 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:20:31.661 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:20:31.661 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:20:31.672 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:20:31.672 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:20:31.703 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:20:31.703 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:20:32.132 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 426 8 365281 365273 6.69828 0 - 0.0299513 2026-02-23T13:20:32.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:20:32 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2484945701' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:20:33.131 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 427 8 365281 365273 6.6826 0 - 0.0299513 2026-02-23T13:20:33.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:20:33 trial068 ceph-mon[38805]: pgmap v564: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:20:33.704 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:20:33.822 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:20:33.918 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:20:33.917+0000 7f38335e6640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:20:33.918 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:20:33.917+0000 7f38335e6640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:20:33.931 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:20:33.931 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:20:33.941 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:20:33.942 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:20:33.942 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:20:33.952 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:20:33.952 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:20:33.981 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:20:33 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:20:33] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:20:33.983 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:20:33.983 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:20:34.132 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 428 8 365281 365273 6.66698 0 - 0.0299513 2026-02-23T13:20:34.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:20:34 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2712917518' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:20:34.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:20:34 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2026-02-23T13:20:34.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:20:33 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:20:33.972Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:20:35.132 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 429 8 365281 365273 6.65144 0 - 0.0299513 2026-02-23T13:20:35.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:20:35 trial068 ceph-mon[38805]: pgmap v565: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:20:35.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:20:35 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-23T13:20:35.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:20:35 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2026-02-23T13:20:35.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:20:35 trial068 ceph-mon[38805]: pgmap v566: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:20:35.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:20:35 trial068 ceph-mon[38805]: pgmap v567: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:20:35.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:20:35 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T13:20:35.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:20:35 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2026-02-23T13:20:35.984 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:20:36.103 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:20:36.131 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 430 8 365281 365273 6.63597 0 - 0.0299513 2026-02-23T13:20:36.211 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:20:36.209+0000 7fd142d4f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:20:36.211 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:20:36.210+0000 7fd142d4f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:20:36.222 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:20:36.223 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:20:36.233 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:20:36.233 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:20:36.233 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:20:36.244 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:20:36.244 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:20:36.290 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:20:36.290 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:20:36.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:20:36 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1728160926' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:20:37.132 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 431 8 365281 365273 6.62058 0 - 0.0299513 2026-02-23T13:20:37.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:20:37 trial068 ceph-mon[38805]: pgmap v568: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:20:38.132 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 432 8 365281 365273 6.60525 0 - 0.0299513 2026-02-23T13:20:38.291 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:20:38.408 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:20:38.506 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:20:38.505+0000 7f6903c6a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:20:38.506 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:20:38.505+0000 7f6903c6a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:20:38.518 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:20:38.518 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:20:38.528 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:20:38.528 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:20:38.528 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:20:38.538 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:20:38.538 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:20:38.569 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:20:38.569 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:20:39.132 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 433 8 365281 365273 6.59 0 - 0.0299513 2026-02-23T13:20:39.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:20:39 trial068 ceph-mon[38805]: pgmap v569: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:20:39.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:20:39 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1886183898' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:20:39.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:20:39 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T13:20:39.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:20:39 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T13:20:40.132 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 434 8 365281 365273 6.57481 0 - 0.0299513 2026-02-23T13:20:40.570 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:20:40.685 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:20:40.784 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:20:40.783+0000 7fe0c59b1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:20:40.784 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:20:40.783+0000 7fe0c59b1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:20:40.796 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:20:40.797 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:20:40.807 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:20:40.808 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:20:40.808 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:20:40.818 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:20:40.818 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:20:40.848 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:20:40.848 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:20:41.132 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 435 8 365281 365273 6.5597 0 - 0.0299513 2026-02-23T13:20:41.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:20:41 trial068 ceph-mon[38805]: pgmap v570: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:20:41.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:20:41 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/575559314' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:20:42.132 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 436 8 365281 365273 6.54466 0 - 0.0299513 2026-02-23T13:20:42.850 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:20:42.968 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:20:43.071 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:20:43.069+0000 7f05c4be9640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:20:43.071 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:20:43.070+0000 7f05c4be9640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:20:43.083 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:20:43.083 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:20:43.092 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:20:43.093 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:20:43.093 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:20:43.103 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:20:43.103 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:20:43.132 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 437 8 365281 365273 6.52968 0 - 0.0299513 2026-02-23T13:20:43.133 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:20:43.134 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:20:43.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:20:43 trial068 ceph-mon[38805]: pgmap v571: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:20:43.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:20:43 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1898691553' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:20:44.132 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 438 8 365281 365273 6.51477 0 - 0.0299513 2026-02-23T13:20:44.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:20:43 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:20:43] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:20:44.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:20:43 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:20:43.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:20:45.132 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 439 8 365281 365273 6.49993 0 - 0.0299513 2026-02-23T13:20:45.135 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:20:45.256 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:20:45.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:20:45 trial068 ceph-mon[38805]: pgmap v572: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:20:45.354 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:20:45.353+0000 7f4441a00640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:20:45.354 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:20:45.353+0000 7f4441a00640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:20:45.367 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:20:45.367 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:20:45.377 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:20:45.377 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:20:45.378 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:20:45.386 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:20:45.387 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:20:45.416 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:20:45.417 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:20:46.132 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T13:20:46.132676+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T13:20:46.133 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T13:20:46.133 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 440 8 365281 365273 6.48516 0 - 0.0299513 2026-02-23T13:20:46.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:20:46 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3079428632' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:20:47.132 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 441 8 365281 365273 6.47045 0 - 0.0299513 2026-02-23T13:20:47.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:20:47 trial068 ceph-mon[38805]: pgmap v573: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:20:47.418 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:20:47.537 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:20:47.644 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:20:47.642+0000 7f5ffc6bc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:20:47.644 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:20:47.643+0000 7f5ffc6bc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:20:47.656 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:20:47.656 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:20:47.665 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:20:47.666 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:20:47.666 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:20:47.676 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:20:47.676 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:20:47.706 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:20:47.706 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:20:48.132 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 442 8 365281 365273 6.45581 0 - 0.0299513 2026-02-23T13:20:48.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:20:48 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3439471299' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:20:49.133 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 443 8 365281 365273 6.44124 0 - 0.0299513 2026-02-23T13:20:49.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:20:49 trial068 ceph-mon[38805]: pgmap v574: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:20:49.707 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:20:49.823 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:20:49.925 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:20:49.923+0000 7f21c9af3640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:20:49.925 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:20:49.923+0000 7f21c9af3640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:20:49.937 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:20:49.937 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:20:49.947 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:20:49.947 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:20:49.947 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:20:49.956 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:20:49.957 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:20:49.989 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:20:49.989 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:20:50.133 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 444 8 365281 365273 6.42673 0 - 0.0299513 2026-02-23T13:20:50.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:20:50 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1497844603' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:20:51.133 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 445 8 365281 365273 6.41229 0 - 0.0299513 2026-02-23T13:20:51.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:20:51 trial068 ceph-mon[38805]: pgmap v575: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:20:51.991 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:20:52.113 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:20:52.133 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 446 8 365281 365273 6.39791 0 - 0.0299513 2026-02-23T13:20:52.218 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:20:52.217+0000 7f71b5928640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:20:52.219 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:20:52.217+0000 7f71b5928640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:20:52.230 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:20:52.231 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:20:52.241 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:20:52.242 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:20:52.242 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:20:52.253 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:20:52.254 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:20:52.284 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:20:52.284 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:20:52.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:20:52 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3334088418' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:20:53.133 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 447 8 365281 365273 6.3836 0 - 0.0299513 2026-02-23T13:20:53.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:20:53 trial068 ceph-mon[38805]: 5.e scrub starts 2026-02-23T13:20:53.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:20:53 trial068 ceph-mon[38805]: 5.e scrub ok 2026-02-23T13:20:53.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:20:53 trial068 ceph-mon[38805]: pgmap v576: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:20:54.133 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 448 8 365281 365273 6.36935 0 - 0.0299513 2026-02-23T13:20:54.285 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:20:54.311 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:20:53 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:20:53] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:20:54.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:20:53 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:20:53.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:20:54.404 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:20:54.503 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:20:54.502+0000 7f4600923640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:20:54.503 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:20:54.502+0000 7f4600923640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:20:54.516 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:20:54.516 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:20:54.526 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:20:54.526 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:20:54.526 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:20:54.537 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:20:54.537 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:20:54.567 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:20:54.567 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:20:55.133 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 449 8 365281 365273 6.35517 0 - 0.0299513 2026-02-23T13:20:55.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:20:55 trial068 ceph-mon[38805]: pgmap v577: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:20:55.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:20:55 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1546259585' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:20:56.133 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 450 8 365281 365273 6.34104 0 - 0.0299513 2026-02-23T13:20:56.567 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:20:56.685 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:20:56.786 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:20:56.785+0000 7f1ec202a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:20:56.787 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:20:56.785+0000 7f1ec202a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:20:56.798 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:20:56.799 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:20:56.809 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:20:56.810 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:20:56.810 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:20:56.819 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:20:56.820 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:20:56.848 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:20:56.849 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:20:57.133 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 451 8 365281 365273 6.32698 0 - 0.0299513 2026-02-23T13:20:57.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:20:57 trial068 ceph-mon[38805]: pgmap v578: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:20:57.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:20:57 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1548641730' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:20:58.133 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 452 8 365281 365273 6.31299 0 - 0.0299513 2026-02-23T13:20:58.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:20:58 trial068 ceph-mon[38805]: 5.5 scrub starts 2026-02-23T13:20:58.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:20:58 trial068 ceph-mon[38805]: 5.5 scrub ok 2026-02-23T13:20:58.850 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:20:58.966 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:20:59.063 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:20:59.062+0000 7fb33edff640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:20:59.063 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:20:59.062+0000 7fb33edff640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:20:59.078 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:20:59.078 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:20:59.088 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:20:59.088 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:20:59.088 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:20:59.099 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:20:59.099 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:20:59.129 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:20:59.130 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:20:59.133 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 453 8 365281 365273 6.29905 0 - 0.0299513 2026-02-23T13:20:59.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:20:59 trial068 ceph-mon[38805]: pgmap v579: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:20:59.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:20:59 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/4039195942' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:21:00.133 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 454 8 365281 365273 6.28518 0 - 0.0299513 2026-02-23T13:21:01.132 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:21:01.135 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 455 8 365281 365273 6.27136 0 - 0.0299513 2026-02-23T13:21:01.249 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:21:01.350 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:21:01.349+0000 7fc1c4769640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:21:01.350 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:21:01.349+0000 7fc1c4769640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:21:01.362 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:21:01.362 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:21:01.372 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:21:01.372 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:21:01.372 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:21:01.383 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:21:01.384 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:21:01.415 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:21:01.415 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:21:01.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:21:01 trial068 ceph-mon[38805]: pgmap v580: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:21:01.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:21:01 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2293893477' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:21:02.134 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 456 8 365281 365273 6.25761 0 - 0.0299513 2026-02-23T13:21:03.134 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 457 8 365281 365273 6.24392 0 - 0.0299513 2026-02-23T13:21:03.416 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:21:03.536 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:21:03.637 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:21:03.636+0000 7f80e5396640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:21:03.637 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:21:03.636+0000 7f80e5396640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:21:03.650 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:21:03.650 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:21:03.660 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:21:03.661 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:21:03.661 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:21:03.671 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:21:03.671 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:21:03.698 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:21:03 trial068 ceph-mon[38805]: pgmap v581: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:21:03.699 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:21:03.700 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:21:04.134 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 458 8 365281 365273 6.23028 0 - 0.0299513 2026-02-23T13:21:04.308 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:21:03 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:21:03.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:21:04.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:21:03 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:21:03] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:21:04.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:21:04 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1725596124' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:21:05.134 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 459 8 365281 365273 6.21671 0 - 0.0299513 2026-02-23T13:21:05.700 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:21:05.725 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:21:05 trial068 ceph-mon[38805]: pgmap v582: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:21:05.821 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:21:05.930 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:21:05.928+0000 7f25425cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:21:05.930 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:21:05.929+0000 7f25425cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:21:05.942 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:21:05.942 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:21:05.952 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:21:05.952 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:21:05.952 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:21:05.962 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:21:05.963 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:21:05.993 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:21:05.994 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:21:06.134 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T13:21:06.134176+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T13:21:06.134 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T13:21:06.134 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 460 8 365281 365273 6.2032 0 - 0.0299513 2026-02-23T13:21:06.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:21:06 trial068 ceph-mon[38805]: osdmap e192: 4 total, 3 up, 4 in 2026-02-23T13:21:06.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:21:06 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/4009122771' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:21:07.134 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 461 8 365281 365273 6.18974 0 - 0.0299513 2026-02-23T13:21:07.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:21:07 trial068 ceph-mon[38805]: 5.a scrub starts 2026-02-23T13:21:07.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:21:07 trial068 ceph-mon[38805]: 5.a scrub ok 2026-02-23T13:21:07.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:21:07 trial068 ceph-mon[38805]: pgmap v584: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:21:07.995 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:21:08.114 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:21:08.134 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 462 8 365281 365273 6.17634 0 - 0.0299513 2026-02-23T13:21:08.214 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:21:08.212+0000 7ff75a388640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:21:08.214 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:21:08.212+0000 7ff75a388640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:21:08.226 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:21:08.226 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:21:08.236 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:21:08.236 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:21:08.236 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:21:08.246 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:21:08.246 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:21:08.275 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:21:08.276 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:21:08.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:21:08 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1903268634' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:21:09.134 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 463 8 365281 365273 6.163 0 - 0.0299513 2026-02-23T13:21:09.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:21:09 trial068 ceph-mon[38805]: pgmap v585: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:21:10.134 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 464 8 365281 365273 6.14972 0 - 0.0299513 2026-02-23T13:21:10.277 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:21:10.394 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:21:10.487 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:21:10.485+0000 7f6907a4d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:21:10.487 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:21:10.486+0000 7f6907a4d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:21:10.499 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:21:10.500 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:21:10.510 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:21:10.511 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:21:10.511 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:21:10.521 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:21:10.521 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:21:10.550 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:21:10.551 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:21:11.134 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 465 8 365281 365273 6.1365 0 - 0.0299513 2026-02-23T13:21:11.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:21:11 trial068 ceph-mon[38805]: pgmap v586: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:21:11.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:21:11 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3795339815' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:21:12.134 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 466 8 365281 365273 6.12333 0 - 0.0299513 2026-02-23T13:21:12.551 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:21:12.666 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:21:12.764 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:21:12.763+0000 7fa8d757a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:21:12.764 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:21:12.763+0000 7fa8d757a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:21:12.776 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:21:12.776 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:21:12.787 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:21:12.787 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:21:12.787 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:21:12.797 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:21:12.798 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:21:12.828 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:21:12.829 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:21:13.134 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 467 8 365281 365273 6.11021 0 - 0.0299513 2026-02-23T13:21:13.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:21:13 trial068 ceph-mon[38805]: pgmap v587: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:21:13.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:21:13 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1102250578' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:21:14.135 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 468 8 365281 365273 6.09716 0 - 0.0299513 2026-02-23T13:21:14.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:21:13 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:21:13] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:21:14.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:21:13 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:21:13.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:21:14.831 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:21:14.949 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:21:15.052 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:21:15.050+0000 7ffb0dbd0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:21:15.052 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:21:15.050+0000 7ffb0dbd0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:21:15.063 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:21:15.064 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:21:15.074 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:21:15.075 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:21:15.075 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:21:15.085 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:21:15.085 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:21:15.108 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:21:15 trial068 ceph-mon[38805]: pgmap v588: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:21:15.108 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:21:15 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2890782504' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:21:15.130 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:21:15.131 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:21:15.134 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 469 8 365281 365273 6.08416 0 - 0.0299513 2026-02-23T13:21:16.134 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 470 8 365281 365273 6.07121 0 - 0.0299513 2026-02-23T13:21:17.132 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:21:17.135 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 471 8 365281 365273 6.05832 0 - 0.0299513 2026-02-23T13:21:17.249 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:21:17.350 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:21:17.348+0000 7fbd6ae14640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:21:17.350 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:21:17.348+0000 7fbd6ae14640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:21:17.362 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:21:17.362 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:21:17.372 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:21:17.373 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:21:17.373 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:21:17.382 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:21:17.382 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:21:17.410 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:21:17.411 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:21:17.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:21:17 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T13:21:17.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:21:17 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T13:21:17.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:21:17 trial068 ceph-mon[38805]: pgmap v589: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:21:17.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:21:17 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2846256339' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:21:18.135 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 472 8 365281 365273 6.04549 0 - 0.0299513 2026-02-23T13:21:19.135 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 473 8 365281 365273 6.03271 0 - 0.0299513 2026-02-23T13:21:19.412 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:21:19.528 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:21:19.624 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:21:19.623+0000 7fef348d3640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:21:19.625 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:21:19.623+0000 7fef348d3640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:21:19.636 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:21:19.636 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:21:19.646 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:21:19.646 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:21:19.646 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:21:19.656 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:21:19.656 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:21:19.686 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:21:19.687 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:21:19.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:21:19 trial068 ceph-mon[38805]: pgmap v590: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:21:20.135 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 474 8 365281 365273 6.01998 0 - 0.0299513 2026-02-23T13:21:20.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:21:20 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1233291280' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:21:21.135 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 475 8 365281 365273 6.00731 0 - 0.0299513 2026-02-23T13:21:21.688 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:21:21.805 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:21:21.827 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:21:21 trial068 ceph-mon[38805]: pgmap v591: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:21:21.902 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:21:21.900+0000 7f0995e8f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:21:21.902 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:21:21.901+0000 7f0995e8f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:21:21.914 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:21:21.914 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:21:21.923 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:21:21.924 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:21:21.924 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:21:21.934 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:21:21.934 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:21:21.966 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:21:21.967 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:21:22.135 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 476 8 365281 365273 5.99469 0 - 0.0299513 2026-02-23T13:21:22.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:21:22 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3072881652' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:21:23.135 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 477 8 365281 365273 5.98212 0 - 0.0299513 2026-02-23T13:21:23.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:21:23 trial068 ceph-mon[38805]: pgmap v592: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:21:23.968 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:21:24.086 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:21:24.135 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 478 8 365281 365273 5.9696 0 - 0.0299513 2026-02-23T13:21:24.187 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:21:24.186+0000 7fc02bbf5640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:21:24.187 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:21:24.186+0000 7fc02bbf5640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:21:24.200 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:21:24.200 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:21:24.210 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:21:24.210 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:21:24.210 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:21:24.220 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:21:24.220 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:21:24.229 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:21:23 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:21:23] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:21:24.250 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:21:24.251 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:21:24.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:21:23 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:21:23.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:21:25.135 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 479 8 365281 365273 5.95714 0 - 0.0299513 2026-02-23T13:21:25.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:21:25 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/602187393' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:21:25.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:21:25 trial068 ceph-mon[38805]: pgmap v593: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:21:26.135 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T13:21:26.135701+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T13:21:26.136 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T13:21:26.136 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 480 8 365281 365273 5.94473 0 - 0.0299513 2026-02-23T13:21:26.252 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:21:26.370 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:21:26.475 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:21:26.474+0000 7fccfb53b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:21:26.475 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:21:26.474+0000 7fccfb53b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:21:26.487 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:21:26.487 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:21:26.497 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:21:26.498 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:21:26.498 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:21:26.508 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:21:26.508 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:21:26.537 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:21:26.538 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:21:27.135 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 481 8 365281 365273 5.93237 0 - 0.0299513 2026-02-23T13:21:27.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:21:27 trial068 ceph-mon[38805]: pgmap v594: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:21:27.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:21:27 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1547202605' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:21:28.135 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 482 8 365281 365273 5.92006 0 - 0.0299513 2026-02-23T13:21:28.538 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:21:28.657 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:21:28.758 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:21:28.756+0000 7f39e31a5640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:21:28.758 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:21:28.757+0000 7f39e31a5640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:21:28.770 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:21:28.770 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:21:28.780 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:21:28.780 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:21:28.780 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:21:28.790 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:21:28.790 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:21:28.821 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:21:28.821 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:21:29.136 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 483 8 365281 365273 5.90781 0 - 0.0299513 2026-02-23T13:21:29.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:21:29 trial068 ceph-mon[38805]: pgmap v595: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:21:29.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:21:29 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/281496411' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:21:30.136 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 484 8 365281 365273 5.8956 0 - 0.0299513 2026-02-23T13:21:30.823 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:21:30.938 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:21:31.035 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:21:31.034+0000 7f425fe5f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:21:31.035 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:21:31.034+0000 7f425fe5f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:21:31.047 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:21:31.048 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:21:31.058 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:21:31.058 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:21:31.058 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:21:31.068 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:21:31.069 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:21:31.099 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:21:31.099 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:21:31.136 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 485 8 365281 365273 5.88344 0 - 0.0299513 2026-02-23T13:21:31.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:21:31 trial068 ceph-mon[38805]: pgmap v596: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:21:31.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:21:31 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/4154464105' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:21:32.136 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 486 8 365281 365273 5.87134 0 - 0.0299513 2026-02-23T13:21:33.101 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:21:33.136 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 487 8 365281 365273 5.85928 0 - 0.0299513 2026-02-23T13:21:33.217 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:21:33.317 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:21:33.316+0000 7f3869b92640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:21:33.317 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:21:33.316+0000 7f3869b92640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:21:33.330 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:21:33.330 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:21:33.340 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:21:33.341 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:21:33.341 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:21:33.351 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:21:33.351 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:21:33.379 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:21:33.379 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:21:33.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:21:33 trial068 ceph-mon[38805]: pgmap v597: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:21:33.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:21:33 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2585991933' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:21:34.136 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 488 8 365281 365273 5.84728 0 - 0.0299513 2026-02-23T13:21:34.265 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:21:33 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:21:33.974Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:21:34.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:21:33 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:21:33] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:21:35.136 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 489 8 365281 365273 5.83532 0 - 0.0299513 2026-02-23T13:21:35.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:21:35 trial068 ceph-mon[38805]: pgmap v598: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:21:35.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:21:35 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2026-02-23T13:21:35.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:21:35 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-23T13:21:35.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:21:35 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2026-02-23T13:21:35.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:21:35 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T13:21:35.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:21:35 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2026-02-23T13:21:35.380 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:21:35.499 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:21:35.596 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:21:35.595+0000 7f0f582c0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:21:35.597 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:21:35.595+0000 7f0f582c0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:21:35.608 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:21:35.608 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:21:35.617 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:21:35.618 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:21:35.618 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:21:35.630 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:21:35.630 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:21:35.660 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:21:35.661 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:21:36.136 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 490 8 365281 365273 5.82341 0 - 0.0299513 2026-02-23T13:21:36.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:21:36 trial068 ceph-mon[38805]: pgmap v599: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:21:36.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:21:36 trial068 ceph-mon[38805]: pgmap v600: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:21:36.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:21:36 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3909252141' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:21:37.136 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 491 8 365281 365273 5.81155 0 - 0.0299513 2026-02-23T13:21:37.661 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:21:37.779 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:21:37.884 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:21:37.882+0000 7fd44382a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:21:37.884 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:21:37.882+0000 7fd44382a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:21:37.896 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:21:37.896 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:21:37.906 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:21:37.906 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:21:37.907 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:21:37.917 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:21:37.918 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:21:37.948 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:21:37.949 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:21:38.136 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 492 8 365281 365273 5.79974 0 - 0.0299513 2026-02-23T13:21:38.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:21:38 trial068 ceph-mon[38805]: pgmap v601: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:21:38.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:21:38 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3159119075' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:21:39.136 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 493 8 365281 365273 5.78797 0 - 0.0299513 2026-02-23T13:21:39.950 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:21:40.069 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:21:40.136 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 494 8 365281 365273 5.77626 0 - 0.0299513 2026-02-23T13:21:40.167 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:21:40.165+0000 7fabd12d7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:21:40.167 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:21:40.166+0000 7fabd12d7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:21:40.179 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:21:40.179 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:21:40.189 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:21:40.189 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:21:40.189 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:21:40.199 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:21:40.200 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:21:40.227 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:21:40 trial068 ceph-mon[38805]: pgmap v602: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:21:40.228 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:21:40.228 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:21:41.136 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 495 8 365281 365273 5.76459 0 - 0.0299513 2026-02-23T13:21:41.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:21:41 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2168279066' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:21:41.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:21:41 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T13:21:41.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:21:41 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T13:21:42.136 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 496 8 365281 365273 5.75297 0 - 0.0299513 2026-02-23T13:21:42.229 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:21:42.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:21:42 trial068 ceph-mon[38805]: pgmap v603: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:21:42.344 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:21:42.442 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:21:42.441+0000 7f5111b46640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:21:42.443 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:21:42.441+0000 7f5111b46640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:21:42.454 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:21:42.455 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:21:42.465 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:21:42.465 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:21:42.465 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:21:42.476 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:21:42.477 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:21:42.507 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:21:42.508 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:21:43.137 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 497 8 365281 365273 5.74139 0 - 0.0299513 2026-02-23T13:21:43.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:21:43 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2942794356' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:21:44.137 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 498 8 365281 365273 5.72986 0 - 0.0299513 2026-02-23T13:21:44.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:21:44 trial068 ceph-mon[38805]: pgmap v604: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:21:44.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:21:44 trial068 ceph-mon[38805]: 5.7 scrub starts 2026-02-23T13:21:44.345 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:21:44 trial068 ceph-mon[38805]: 5.7 scrub ok 2026-02-23T13:21:44.345 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:21:43 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:21:43] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:21:44.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:21:43 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:21:43.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:21:44.508 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:21:44.627 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:21:44.727 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:21:44.725+0000 7fd6cc66a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:21:44.727 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:21:44.726+0000 7fd6cc66a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:21:44.739 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:21:44.739 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:21:44.749 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:21:44.749 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:21:44.749 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:21:44.760 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:21:44.761 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:21:44.792 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:21:44.793 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:21:45.137 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 499 8 365281 365273 5.71838 0 - 0.0299513 2026-02-23T13:21:45.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:21:45 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/381874335' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:21:46.137 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T13:21:46.137229+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T13:21:46.137 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T13:21:46.137 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 500 8 365281 365273 5.70694 0 - 0.0299513 2026-02-23T13:21:46.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:21:46 trial068 ceph-mon[38805]: pgmap v605: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:21:46.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:21:46 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T13:21:46.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:21:46 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T13:21:46.793 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:21:46.912 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:21:47.006 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:21:47.004+0000 7fb34de37640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:21:47.006 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:21:47.005+0000 7fb34de37640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:21:47.017 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:21:47.018 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:21:47.027 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:21:47.028 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:21:47.028 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:21:47.038 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:21:47.038 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:21:47.068 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:21:47.069 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:21:47.137 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 501 8 365281 365273 5.69555 0 - 0.0299513 2026-02-23T13:21:47.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:21:47 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/777861226' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:21:48.137 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 502 8 365281 365273 5.68421 0 - 0.0299513 2026-02-23T13:21:48.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:21:48 trial068 ceph-mon[38805]: pgmap v606: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:21:49.070 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:21:49.137 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 503 8 365281 365273 5.6729 0 - 0.0299513 2026-02-23T13:21:49.187 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:21:49.285 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:21:49.284+0000 7fbb10ee8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:21:49.285 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:21:49.284+0000 7fbb10ee8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:21:49.298 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:21:49.298 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:21:49.307 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:21:49.308 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:21:49.308 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:21:49.318 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:21:49.318 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:21:49.347 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:21:49.348 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:21:50.137 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 504 8 365281 365273 5.66165 0 - 0.0299513 2026-02-23T13:21:50.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:21:50 trial068 ceph-mon[38805]: pgmap v607: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:21:50.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:21:50 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2153645401' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:21:51.137 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 505 8 365281 365273 5.65044 0 - 0.0299513 2026-02-23T13:21:51.349 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:21:51.466 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:21:51.565 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:21:51.563+0000 7f93bf577640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:21:51.565 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:21:51.563+0000 7f93bf577640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:21:51.576 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:21:51.577 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:21:51.586 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:21:51.587 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:21:51.587 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:21:51.598 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:21:51.598 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:21:51.627 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:21:51.628 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:21:52.137 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 506 8 365281 365273 5.63927 0 - 0.0299513 2026-02-23T13:21:52.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:21:52 trial068 ceph-mon[38805]: pgmap v608: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:21:52.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:21:52 trial068 ceph-mon[38805]: 5.8 scrub starts 2026-02-23T13:21:52.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:21:52 trial068 ceph-mon[38805]: 5.8 scrub ok 2026-02-23T13:21:52.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:21:52 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/181253528' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:21:53.137 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 507 8 365281 365273 5.62815 0 - 0.0299513 2026-02-23T13:21:53.629 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:21:53.749 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:21:53.847 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:21:53.845+0000 7fb94fd80640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:21:53.847 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:21:53.846+0000 7fb94fd80640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:21:53.860 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:21:53.860 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:21:53.870 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:21:53.871 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:21:53.871 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:21:53.881 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:21:53.881 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:21:53.910 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:21:53.911 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:21:54.137 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 508 8 365281 365273 5.61707 0 - 0.0299513 2026-02-23T13:21:54.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:21:54 trial068 ceph-mon[38805]: pgmap v609: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:21:54.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:21:54 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/409735139' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:21:54.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:21:53 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:21:53] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:21:54.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:21:53 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:21:53.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:21:55.138 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 509 8 365281 365273 5.60603 0 - 0.0299513 2026-02-23T13:21:55.912 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:21:56.029 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:21:56.134 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:21:56.132+0000 7f977a252640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:21:56.134 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:21:56.132+0000 7f977a252640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:21:56.137 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 510 8 365281 365273 5.59504 0 - 0.0299513 2026-02-23T13:21:56.146 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:21:56.146 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:21:56.155 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:21:56.156 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:21:56.156 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:21:56.166 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:21:56.166 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:21:56.194 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:21:56 trial068 ceph-mon[38805]: pgmap v610: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:21:56.195 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:21:56.196 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:21:57.138 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 511 8 365281 365273 5.58409 0 - 0.0299513 2026-02-23T13:21:57.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:21:57 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3953171982' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:21:58.138 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 512 8 365281 365273 5.57319 0 - 0.0299513 2026-02-23T13:21:58.197 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:21:58.312 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:21:58.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:21:58 trial068 ceph-mon[38805]: pgmap v611: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:21:58.407 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:21:58.406+0000 7f7fef480640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:21:58.408 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:21:58.406+0000 7f7fef480640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:21:58.419 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:21:58.420 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:21:58.430 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:21:58.430 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:21:58.430 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:21:58.440 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:21:58.440 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:21:58.469 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:21:58.470 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:21:59.138 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 513 8 365281 365273 5.56232 0 - 0.0299513 2026-02-23T13:21:59.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:21:59 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2203813779' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:22:00.138 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 514 8 365281 365273 5.5515 0 - 0.0299513 2026-02-23T13:22:00.471 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:22:00.495 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:22:00 trial068 ceph-mon[38805]: pgmap v612: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:22:00.588 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:22:00.689 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:22:00.688+0000 7f522db18640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:22:00.689 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:22:00.688+0000 7f522db18640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:22:00.701 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:22:00.701 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:22:00.711 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:22:00.711 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:22:00.711 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:22:00.721 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:22:00.722 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:22:00.755 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:22:00.756 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:22:01.138 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 515 8 365281 365273 5.54072 0 - 0.0299513 2026-02-23T13:22:01.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:22:01 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/932603780' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:22:02.138 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 516 8 365281 365273 5.52998 0 - 0.0299513 2026-02-23T13:22:02.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:22:02 trial068 ceph-mon[38805]: pgmap v613: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:22:02.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:22:02 trial068 ceph-mon[38805]: 5.e scrub starts 2026-02-23T13:22:02.757 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:22:02.875 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:22:02.979 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:22:02.977+0000 7f07f64ed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:22:02.979 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:22:02.978+0000 7f07f64ed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:22:02.991 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:22:02.991 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:22:03.001 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:22:03.001 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:22:03.001 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:22:03.012 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:22:03.012 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:22:03.040 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:22:03.041 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:22:03.138 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 517 8 365281 365273 5.51929 0 - 0.0299513 2026-02-23T13:22:03.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:22:03 trial068 ceph-mon[38805]: 5.e scrub ok 2026-02-23T13:22:03.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:22:03 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1396778588' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:22:04.138 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 518 8 365281 365273 5.50863 0 - 0.0299513 2026-02-23T13:22:04.264 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:22:03 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:22:03.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:22:04.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:22:04 trial068 ceph-mon[38805]: pgmap v614: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:22:04.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:22:03 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:22:03] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:22:05.042 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:22:05.138 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 519 8 365281 365273 5.49802 0 - 0.0299513 2026-02-23T13:22:05.162 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:22:05.261 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:22:05.259+0000 7f07fa7e3640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:22:05.261 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:22:05.260+0000 7f07fa7e3640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:22:05.273 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:22:05.273 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:22:05.283 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:22:05.284 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:22:05.284 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:22:05.294 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:22:05.294 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:22:05.324 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:22:05.324 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:22:06.138 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T13:22:06.138653+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T13:22:06.139 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T13:22:06.139 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 520 8 365281 365273 5.48745 0 - 0.0299513 2026-02-23T13:22:06.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:22:06 trial068 ceph-mon[38805]: pgmap v615: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:22:06.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:22:06 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2668049962' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:22:07.139 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 521 8 365281 365273 5.47691 0 - 0.0299513 2026-02-23T13:22:07.325 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:22:07.443 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:22:07.541 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:22:07.540+0000 7facebb62640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:22:07.541 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:22:07.540+0000 7facebb62640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:22:07.553 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:22:07.553 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:22:07.563 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:22:07.564 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:22:07.564 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:22:07.573 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:22:07.573 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:22:07.602 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:22:07.603 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:22:08.139 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 522 8 365281 365273 5.46642 0 - 0.0299513 2026-02-23T13:22:08.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:22:08 trial068 ceph-mon[38805]: pgmap v616: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:22:08.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:22:08 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2543363657' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:22:09.139 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 523 8 365281 365273 5.45597 0 - 0.0299513 2026-02-23T13:22:09.604 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:22:09.721 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:22:09.818 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:22:09.817+0000 7f6655965640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:22:09.818 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:22:09.817+0000 7f6655965640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:22:09.830 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:22:09.830 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:22:09.840 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:22:09.841 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:22:09.841 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:22:09.851 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:22:09.851 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:22:09.883 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:22:09.883 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:22:10.139 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 524 8 365281 365273 5.44556 0 - 0.0299513 2026-02-23T13:22:10.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:22:10 trial068 ceph-mon[38805]: pgmap v617: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:22:10.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:22:10 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/30272097' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:22:11.139 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 525 8 365281 365273 5.43518 0 - 0.0299513 2026-02-23T13:22:11.884 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:22:12.004 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:22:12.102 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:22:12.100+0000 7f7733c6a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:22:12.102 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:22:12.101+0000 7f7733c6a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:22:12.114 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:22:12.114 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:22:12.126 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:22:12.127 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:22:12.127 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:22:12.137 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:22:12.138 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:22:12.138 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 526 8 365281 365273 5.42485 0 - 0.0299513 2026-02-23T13:22:12.167 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:22:12 trial068 ceph-mon[38805]: pgmap v618: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:22:12.167 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:22:12 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3259993331' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:22:12.168 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:22:12.168 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:22:13.139 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 527 8 365281 365273 5.41456 0 - 0.0299513 2026-02-23T13:22:13.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:22:13 trial068 ceph-mon[38805]: 5.5 scrub starts 2026-02-23T13:22:13.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:22:13 trial068 ceph-mon[38805]: 5.5 scrub ok 2026-02-23T13:22:14.139 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 528 8 365281 365273 5.4043 0 - 0.0299513 2026-02-23T13:22:14.170 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:22:14.289 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:22:14.311 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:22:14 trial068 ceph-mon[38805]: pgmap v619: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:22:14.311 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:22:13 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:22:13] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:22:14.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:22:13 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:22:13.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:22:14.386 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:22:14.385+0000 7f709a072640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:22:14.387 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:22:14.385+0000 7f709a072640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:22:14.398 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:22:14.399 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:22:14.408 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:22:14.409 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:22:14.409 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:22:14.420 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:22:14.420 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:22:14.450 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:22:14.451 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:22:15.139 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 529 8 365281 365273 5.39409 0 - 0.0299513 2026-02-23T13:22:15.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:22:15 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/4013942293' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:22:16.139 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 530 8 365281 365273 5.38391 0 - 0.0299513 2026-02-23T13:22:16.452 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:22:16.477 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:22:16 trial068 ceph-mon[38805]: pgmap v620: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:22:16.569 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:22:16.669 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:22:16.667+0000 7f1a04821640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:22:16.670 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:22:16.668+0000 7f1a04821640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:22:16.681 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:22:16.681 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:22:16.691 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:22:16.691 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:22:16.691 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:22:16.701 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:22:16.702 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:22:16.733 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:22:16.733 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:22:17.139 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 531 8 365281 365273 5.37377 0 - 0.0299513 2026-02-23T13:22:17.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:22:17 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/287505783' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:22:18.139 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 532 8 365281 365273 5.36367 0 - 0.0299513 2026-02-23T13:22:18.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:22:18 trial068 ceph-mon[38805]: pgmap v621: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:22:18.734 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:22:18.850 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:22:18.948 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:22:18.946+0000 7f08b7e98640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:22:18.948 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:22:18.947+0000 7f08b7e98640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:22:18.959 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:22:18.960 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:22:18.970 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:22:18.970 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:22:18.970 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:22:18.980 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:22:18.980 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:22:19.010 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:22:19.011 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:22:19.139 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 533 8 365281 365273 5.35361 0 - 0.0299513 2026-02-23T13:22:19.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:22:19 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1477759677' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:22:20.139 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 534 8 365281 365273 5.34358 0 - 0.0299513 2026-02-23T13:22:20.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:22:20 trial068 ceph-mon[38805]: pgmap v622: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:22:21.012 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:22:21.129 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:22:21.139 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 535 8 365281 365273 5.33359 0 - 0.0299513 2026-02-23T13:22:21.225 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:22:21.224+0000 7fc690f37640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:22:21.225 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:22:21.224+0000 7fc690f37640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:22:21.237 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:22:21.237 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:22:21.247 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:22:21.248 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:22:21.248 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:22:21.258 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:22:21.258 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:22:21.287 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:22:21.288 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:22:22.139 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 536 8 365281 365273 5.32364 0 - 0.0299513 2026-02-23T13:22:22.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:22:22 trial068 ceph-mon[38805]: pgmap v623: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:22:22.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:22:22 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2500161317' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:22:23.139 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 537 8 365281 365273 5.31373 0 - 0.0299513 2026-02-23T13:22:23.289 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:22:23.409 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:22:23.507 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:22:23.505+0000 7f3833864640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:22:23.507 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:22:23.506+0000 7f3833864640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:22:23.519 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:22:23.519 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:22:23.530 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:22:23.530 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:22:23.530 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:22:23.540 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:22:23.540 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:22:23.569 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:22:23.570 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:22:24.139 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 538 8 365281 365273 5.30385 0 - 0.0299513 2026-02-23T13:22:24.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:22:24 trial068 ceph-mon[38805]: pgmap v624: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:22:24.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:22:24 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/887923787' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:22:24.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:22:23 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:22:23] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:22:24.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:22:23 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:22:23.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:22:25.140 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 539 8 365281 365273 5.29401 0 - 0.0299513 2026-02-23T13:22:25.571 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:22:25.688 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:22:25.791 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:22:25.789+0000 7fc9a5e72640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:22:25.791 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:22:25.789+0000 7fc9a5e72640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:22:25.802 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:22:25.803 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:22:25.814 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:22:25.814 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:22:25.814 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:22:25.824 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:22:25.825 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:22:25.854 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:22:25.855 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:22:26.140 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T13:22:26.140085+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T13:22:26.140 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T13:22:26.140 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 540 8 365281 365273 5.28421 0 - 0.0299513 2026-02-23T13:22:26.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:22:26 trial068 ceph-mon[38805]: pgmap v625: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:22:26.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:22:26 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3445181548' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:22:27.140 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 541 8 365281 365273 5.27444 0 - 0.0299513 2026-02-23T13:22:27.856 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:22:27.975 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:22:28.081 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:22:28.079+0000 7f41b9e3a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:22:28.081 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:22:28.079+0000 7f41b9e3a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:22:28.093 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:22:28.094 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:22:28.104 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:22:28.104 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:22:28.104 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:22:28.114 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:22:28.114 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:22:28.132 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:22:28 trial068 ceph-mon[38805]: pgmap v626: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:22:28.132 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:22:28 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2214892525' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:22:28.140 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 542 8 365281 365273 5.26471 0 - 0.0299513 2026-02-23T13:22:28.145 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:22:28.146 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:22:29.140 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 543 8 365281 365273 5.25501 0 - 0.0299513 2026-02-23T13:22:29.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:22:29 trial068 ceph-mon[38805]: 5.a scrub starts 2026-02-23T13:22:29.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:22:29 trial068 ceph-mon[38805]: 5.a scrub ok 2026-02-23T13:22:30.140 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 544 8 365281 365273 5.24535 0 - 0.0299513 2026-02-23T13:22:30.147 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:22:30.265 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:22:30.366 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:22:30.365+0000 7f5db1566640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:22:30.367 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:22:30.366+0000 7f5db1566640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:22:30.379 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:22:30 trial068 ceph-mon[38805]: pgmap v627: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:22:30.379 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:22:30.380 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:22:30.389 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:22:30.389 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:22:30.389 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:22:30.399 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:22:30.399 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:22:30.428 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:22:30.428 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:22:31.140 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 545 8 365281 365273 5.23573 0 - 0.0299513 2026-02-23T13:22:31.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:22:31 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1996825476' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:22:32.140 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 546 8 365281 365273 5.22614 0 - 0.0299513 2026-02-23T13:22:32.429 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:22:32.454 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:22:32 trial068 ceph-mon[38805]: pgmap v628: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:22:32.548 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:22:32.646 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:22:32.645+0000 7fba4df3d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:22:32.647 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:22:32.645+0000 7fba4df3d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:22:32.658 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:22:32.659 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:22:32.668 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:22:32.669 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:22:32.669 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:22:32.679 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:22:32.679 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:22:32.710 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:22:32.710 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:22:33.140 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 547 8 365281 365273 5.21659 0 - 0.0299513 2026-02-23T13:22:33.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:22:33 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/4038682618' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:22:34.140 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 548 8 365281 365273 5.20707 0 - 0.0299513 2026-02-23T13:22:34.265 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:22:33 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:22:33.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:22:34.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:22:34 trial068 ceph-mon[38805]: pgmap v629: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:22:34.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:22:33 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:22:33] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:22:34.712 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:22:34.828 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:22:34.936 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:22:34.934+0000 7f86d317d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:22:34.936 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:22:34.934+0000 7f86d317d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:22:34.948 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:22:34.948 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:22:34.958 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:22:34.958 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:22:34.959 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:22:34.969 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:22:34.969 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:22:35.001 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:22:35.002 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:22:35.140 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 549 8 365281 365273 5.19758 0 - 0.0299513 2026-02-23T13:22:35.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:22:35 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T13:22:35.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:22:35 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T13:22:35.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:22:35 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2026-02-23T13:22:35.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:22:35 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-23T13:22:35.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:22:35 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2026-02-23T13:22:35.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:22:35 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T13:22:35.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:22:35 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2026-02-23T13:22:35.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:22:35 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1734640612' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:22:36.140 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 550 8 365281 365273 5.18813 0 - 0.0299513 2026-02-23T13:22:36.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:22:36 trial068 ceph-mon[38805]: pgmap v630: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:22:36.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:22:36 trial068 ceph-mon[38805]: pgmap v631: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:22:36.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:22:36 trial068 ceph-mon[38805]: pgmap v632: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:22:37.002 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:22:37.121 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:22:37.140 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 551 8 365281 365273 5.17872 0 - 0.0299513 2026-02-23T13:22:37.232 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:22:37.231+0000 7f996891e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:22:37.232 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:22:37.231+0000 7f996891e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:22:37.244 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:22:37.245 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:22:37.254 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:22:37.254 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:22:37.254 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:22:37.265 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:22:37.265 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:22:37.296 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:22:37.296 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:22:38.140 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 552 8 365281 365273 5.16933 0 - 0.0299513 2026-02-23T13:22:38.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:22:38 trial068 ceph-mon[38805]: pgmap v633: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:22:38.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:22:38 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3127965641' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:22:39.141 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 553 8 365281 365273 5.15999 0 - 0.0299513 2026-02-23T13:22:39.297 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:22:39.415 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:22:39.518 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:22:39.517+0000 7f16e13de640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:22:39.519 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:22:39.517+0000 7f16e13de640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:22:39.530 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:22:39.531 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:22:39.540 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:22:39.541 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:22:39.541 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:22:39.551 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:22:39.551 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:22:39.581 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:22:39.582 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:22:40.140 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 554 8 365281 365273 5.15067 0 - 0.0299513 2026-02-23T13:22:40.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:22:40 trial068 ceph-mon[38805]: pgmap v634: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:22:40.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:22:40 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/4070953837' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:22:41.141 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 555 8 365281 365273 5.14139 0 - 0.0299513 2026-02-23T13:22:41.583 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:22:41.700 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:22:41.799 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:22:41.797+0000 7fbd427ff640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:22:41.799 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:22:41.797+0000 7fbd427ff640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:22:41.811 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:22:41.811 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:22:41.822 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:22:41.822 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:22:41.823 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:22:41.833 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:22:41.833 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:22:41.861 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:22:41.861 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:22:42.141 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 556 8 365281 365273 5.13215 0 - 0.0299513 2026-02-23T13:22:42.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:22:42 trial068 ceph-mon[38805]: pgmap v635: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:22:42.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:22:42 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3319178692' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:22:43.141 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 557 8 365281 365273 5.12293 0 - 0.0299513 2026-02-23T13:22:43.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:22:43 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T13:22:43.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:22:43 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T13:22:43.862 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:22:43.980 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:22:44.083 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:22:44.082+0000 7f9f178b2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:22:44.083 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:22:44.082+0000 7f9f178b2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:22:44.096 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:22:44.096 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:22:44.106 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:22:44.106 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:22:44.106 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:22:44.116 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:22:44.116 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:22:44.140 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:22:43 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:22:43] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:22:44.141 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 558 8 365281 365273 5.11375 0 - 0.0299513 2026-02-23T13:22:44.144 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:22:44.145 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:22:44.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:22:43 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:22:43.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:22:44.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:22:44 trial068 ceph-mon[38805]: pgmap v636: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:22:44.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:22:44 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/775989149' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:22:45.141 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 559 8 365281 365273 5.1046 0 - 0.0299513 2026-02-23T13:22:46.141 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T13:22:46.141583+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T13:22:46.141 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T13:22:46.141 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 560 8 365281 365273 5.09549 0 - 0.0299513 2026-02-23T13:22:46.146 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:22:46.265 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:22:46.369 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:22:46.368+0000 7f72a0b2e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:22:46.369 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:22:46.369+0000 7f72a0b2e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:22:46.381 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:22:46.381 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:22:46.391 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:22:46.391 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:22:46.391 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:22:46 trial068 ceph-mon[38805]: pgmap v637: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:22:46.392 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:22:46.402 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:22:46.402 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:22:46.433 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:22:46.433 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:22:47.141 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 561 8 365281 365273 5.0864 0 - 0.0299513 2026-02-23T13:22:47.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:22:47 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T13:22:47.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:22:47 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T13:22:47.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:22:47 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1907316458' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:22:48.141 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 562 8 365281 365273 5.07735 0 - 0.0299513 2026-02-23T13:22:48.434 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:22:48.458 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:22:48 trial068 ceph-mon[38805]: pgmap v638: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:22:48.552 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:22:48.660 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:22:48.659+0000 7fbf377ec640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:22:48.660 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:22:48.659+0000 7fbf377ec640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:22:48.673 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:22:48.673 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:22:48.683 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:22:48.683 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:22:48.683 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:22:48.694 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:22:48.694 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:22:48.725 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:22:48.726 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:22:49.141 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 563 8 365281 365273 5.06833 0 - 0.0299513 2026-02-23T13:22:49.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:22:49 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2326383677' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:22:50.142 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 564 8 365281 365273 5.05935 0 - 0.0299513 2026-02-23T13:22:50.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:22:50 trial068 ceph-mon[38805]: pgmap v639: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:22:50.727 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:22:50.846 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:22:50.945 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:22:50.943+0000 7f771725f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:22:50.945 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:22:50.943+0000 7f771725f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:22:50.957 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:22:50.957 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:22:50.966 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:22:50.967 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:22:50.967 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:22:50.977 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:22:50.977 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:22:51.006 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:22:51.007 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:22:51.142 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 565 8 365281 365273 5.05039 0 - 0.0299513 2026-02-23T13:22:51.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:22:51 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/714494036' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:22:52.141 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 566 8 365281 365273 5.04147 0 - 0.0299513 2026-02-23T13:22:52.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:22:52 trial068 ceph-mon[38805]: pgmap v640: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:22:53.008 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:22:53.127 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:22:53.142 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 567 8 365281 365273 5.03258 0 - 0.0299513 2026-02-23T13:22:53.227 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:22:53.225+0000 7f35f05fa640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:22:53.227 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:22:53.226+0000 7f35f05fa640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:22:53.239 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:22:53.239 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:22:53.249 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:22:53.249 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:22:53.249 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:22:53.259 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:22:53.260 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:22:53.291 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:22:53.292 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:22:54.142 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 568 8 365281 365273 5.02372 0 - 0.0299513 2026-02-23T13:22:54.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:22:54 trial068 ceph-mon[38805]: pgmap v641: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:22:54.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:22:54 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/4251490841' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:22:54.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:22:53 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:22:53] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:22:54.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:22:53 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:22:53.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:22:55.142 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 569 8 365281 365273 5.01489 0 - 0.0299513 2026-02-23T13:22:55.293 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:22:55.412 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:22:55.513 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:22:55.512+0000 7fc82d959640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:22:55.513 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:22:55.512+0000 7fc82d959640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:22:55.525 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:22:55.525 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:22:55.535 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:22:55.535 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:22:55.536 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:22:55.545 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:22:55.545 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:22:55.573 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:22:55.574 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:22:56.142 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 570 8 365281 365273 5.00609 0 - 0.0299513 2026-02-23T13:22:56.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:22:56 trial068 ceph-mon[38805]: pgmap v642: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:22:56.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:22:56 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1860735937' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:22:57.142 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 571 8 365281 365273 4.99732 0 - 0.0299513 2026-02-23T13:22:57.575 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:22:57.692 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:22:57.790 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:22:57.788+0000 7f47a092c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:22:57.790 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:22:57.789+0000 7f47a092c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:22:57.802 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:22:57.802 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:22:57.812 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:22:57.813 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:22:57.813 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:22:57.822 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:22:57.823 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:22:57.853 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:22:57.853 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:22:58.142 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 572 8 365281 365273 4.98859 0 - 0.0299513 2026-02-23T13:22:58.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:22:58 trial068 ceph-mon[38805]: pgmap v643: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:22:58.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:22:58 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2457875594' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:22:59.142 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 573 8 365281 365273 4.97988 0 - 0.0299513 2026-02-23T13:22:59.853 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:22:59.972 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:23:00.071 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:23:00.070+0000 7fb89a737640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:23:00.071 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:23:00.070+0000 7fb89a737640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:23:00.082 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:23:00.083 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:23:00.093 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:23:00.093 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:23:00.093 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:23:00.103 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:23:00.103 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:23:00.134 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:23:00.135 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:23:00.142 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 574 8 365281 365273 4.97121 0 - 0.0299513 2026-02-23T13:23:00.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:23:00 trial068 ceph-mon[38805]: pgmap v644: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:23:00.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:23:00 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3415923655' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:23:01.142 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 575 8 365281 365273 4.96256 0 - 0.0299513 2026-02-23T13:23:02.136 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:23:02.142 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 576 8 365281 365273 4.95395 0 - 0.0299513 2026-02-23T13:23:02.250 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:23:02.353 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:23:02.352+0000 7f5d29b91640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:23:02.353 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:23:02.352+0000 7f5d29b91640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:23:02.365 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:23:02.365 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:23:02.375 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:23:02.375 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:23:02.375 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:23:02.385 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:23:02.386 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:23:02.408 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:23:02 trial068 ceph-mon[38805]: pgmap v645: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:23:02.414 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:23:02.415 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:23:03.143 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 577 8 365281 365273 4.94536 0 - 0.0299513 2026-02-23T13:23:03.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:23:03 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1047629408' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:23:04.142 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 578 8 365281 365273 4.9368 0 - 0.0299513 2026-02-23T13:23:04.265 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:23:03 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:23:03.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:23:04.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:23:04 trial068 ceph-mon[38805]: pgmap v646: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:23:04.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:23:03 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:23:03] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:23:04.415 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:23:04.534 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:23:04.638 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:23:04.637+0000 7f0177843640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:23:04.638 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:23:04.637+0000 7f0177843640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:23:04.650 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:23:04.650 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:23:04.660 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:23:04.661 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:23:04.661 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:23:04.670 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:23:04.671 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:23:04.700 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:23:04.700 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:23:05.143 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 579 8 365281 365273 4.92828 0 - 0.0299513 2026-02-23T13:23:05.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:23:05 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2593701216' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:23:06.143 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T13:23:06.143193+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T13:23:06.143 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T13:23:06.143 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 580 8 365281 365273 4.91978 0 - 0.0299513 2026-02-23T13:23:06.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:23:06 trial068 ceph-mon[38805]: pgmap v647: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:23:06.701 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:23:06.821 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:23:06.921 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:23:06.920+0000 7f9f52584640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:23:06.922 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:23:06.920+0000 7f9f52584640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:23:06.933 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:23:06.934 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:23:06.944 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:23:06.944 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:23:06.945 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:23:06.955 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:23:06.955 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:23:06.984 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:23:06.984 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:23:07.143 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 581 8 365281 365273 4.91131 0 - 0.0299513 2026-02-23T13:23:07.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:23:07 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1598561819' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:23:08.143 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 582 8 365281 365273 4.90287 0 - 0.0299513 2026-02-23T13:23:08.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:23:08 trial068 ceph-mon[38805]: pgmap v648: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:23:08.985 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:23:09.105 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:23:09.143 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 583 8 365281 365273 4.89446 0 - 0.0299513 2026-02-23T13:23:09.208 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:23:09.206+0000 7f97f5728640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:23:09.208 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:23:09.207+0000 7f97f5728640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:23:09.219 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:23:09.220 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:23:09.229 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:23:09.230 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:23:09.230 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:23:09.241 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:23:09.241 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:23:09.270 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:23:09 trial068 ceph-mon[38805]: 5.7 scrub starts 2026-02-23T13:23:09.270 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:23:09 trial068 ceph-mon[38805]: 5.7 scrub ok 2026-02-23T13:23:09.271 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:23:09.271 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:23:10.143 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 584 8 365281 365273 4.88608 0 - 0.0299513 2026-02-23T13:23:10.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:23:10 trial068 ceph-mon[38805]: pgmap v649: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:23:10.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:23:10 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3124895429' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:23:11.143 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 585 8 365281 365273 4.87773 0 - 0.0299513 2026-02-23T13:23:11.272 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:23:11.389 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:23:11.489 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:23:11.488+0000 7fa93bd4a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:23:11.490 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:23:11.488+0000 7fa93bd4a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:23:11.501 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:23:11.502 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:23:11.512 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:23:11.512 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:23:11.513 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:23:11.523 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:23:11.523 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:23:11.553 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:23:11.554 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:23:12.143 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 586 8 365281 365273 4.86941 0 - 0.0299513 2026-02-23T13:23:12.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:23:12 trial068 ceph-mon[38805]: pgmap v650: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:23:12.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:23:12 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/669411899' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:23:13.143 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 587 8 365281 365273 4.86111 0 - 0.0299513 2026-02-23T13:23:13.555 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:23:13.673 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:23:13.773 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:23:13.772+0000 7fc09f1d0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:23:13.774 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:23:13.772+0000 7fc09f1d0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:23:13.785 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:23:13.786 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:23:13.796 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:23:13.796 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:23:13.797 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:23:13.806 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:23:13.807 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:23:13.851 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:23:13.852 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:23:14.093 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:23:13 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:23:13] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:23:14.143 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 588 8 365281 365273 4.85284 0 - 0.0299513 2026-02-23T13:23:14.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:23:13 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:23:13.972Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:23:14.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:23:14 trial068 ceph-mon[38805]: pgmap v651: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:23:14.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:23:14 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/526969076' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:23:15.144 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 589 8 365281 365273 4.8446 0 - 0.0299513 2026-02-23T13:23:15.852 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:23:15.970 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:23:16.069 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:23:16.068+0000 7f2619b63640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:23:16.070 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:23:16.068+0000 7f2619b63640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:23:16.081 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:23:16.081 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:23:16.092 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:23:16.093 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:23:16.093 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:23:16.103 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:23:16.103 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:23:16.134 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:23:16.135 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:23:16.144 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 590 8 365281 365273 4.83639 0 - 0.0299513 2026-02-23T13:23:16.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:23:16 trial068 ceph-mon[38805]: 5.8 scrub starts 2026-02-23T13:23:16.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:23:16 trial068 ceph-mon[38805]: 5.8 scrub ok 2026-02-23T13:23:16.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:23:16 trial068 ceph-mon[38805]: pgmap v652: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:23:16.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:23:16 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2612355942' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:23:17.144 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 591 8 365281 365273 4.82821 0 - 0.0299513 2026-02-23T13:23:18.136 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:23:18.143 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 592 8 365281 365273 4.82005 0 - 0.0299513 2026-02-23T13:23:18.254 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:23:18.349 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:23:18.348+0000 7f9c27d47640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:23:18.349 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:23:18.348+0000 7f9c27d47640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:23:18.361 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:23:18.361 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:23:18.372 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:23:18.372 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:23:18.372 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:23:18.382 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:23:18.383 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:23:18.411 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:23:18.412 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:23:18.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:23:18 trial068 ceph-mon[38805]: pgmap v653: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:23:19.144 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 593 8 365281 365273 4.81193 0 - 0.0299513 2026-02-23T13:23:19.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:23:19 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/433746306' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:23:20.144 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 594 8 365281 365273 4.80383 0 - 0.0299513 2026-02-23T13:23:20.412 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:23:20.438 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:23:20 trial068 ceph-mon[38805]: pgmap v654: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:23:20.533 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:23:20.631 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:23:20.629+0000 7f4e82594640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:23:20.631 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:23:20.630+0000 7f4e82594640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:23:20.642 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:23:20.642 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:23:20.653 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:23:20.653 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:23:20.653 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:23:20.663 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:23:20.663 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:23:20.693 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:23:20.694 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:23:21.144 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 595 8 365281 365273 4.79575 0 - 0.0299513 2026-02-23T13:23:21.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:23:21 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3798687042' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:23:22.144 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 596 8 365281 365273 4.78771 0 - 0.0299513 2026-02-23T13:23:22.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:23:22 trial068 ceph-mon[38805]: pgmap v655: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:23:22.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:23:22 trial068 ceph-mon[38805]: 5.e scrub starts 2026-02-23T13:23:22.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:23:22 trial068 ceph-mon[38805]: 5.e scrub ok 2026-02-23T13:23:22.694 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:23:22.810 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:23:22.909 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:23:22.907+0000 7fb72ea64640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:23:22.909 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:23:22.908+0000 7fb72ea64640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:23:22.921 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:23:22.922 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:23:22.932 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:23:22.932 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:23:22.932 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:23:22.943 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:23:22.943 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:23:22.975 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:23:22.976 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:23:23.144 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 597 8 365281 365273 4.77969 0 - 0.0299513 2026-02-23T13:23:23.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:23:23 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1478323674' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:23:24.144 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 598 8 365281 365273 4.77169 0 - 0.0299513 2026-02-23T13:23:24.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:23:24 trial068 ceph-mon[38805]: pgmap v656: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:23:24.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:23:23 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:23:23] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:23:24.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:23:23 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:23:23.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:23:24.977 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:23:25.096 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:23:25.144 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 599 8 365281 365273 4.76373 0 - 0.0299513 2026-02-23T13:23:25.196 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:23:25.194+0000 7f6b973a1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:23:25.196 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:23:25.194+0000 7f6b973a1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:23:25.207 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:23:25.208 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:23:25.218 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:23:25.219 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:23:25.219 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:23:25.228 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:23:25.229 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:23:25.258 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:23:25.259 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:23:26.144 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T13:23:26.144751+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T13:23:26.144 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T13:23:26.144 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 600 8 365281 365273 4.75579 0 - 0.0299513 2026-02-23T13:23:26.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:23:26 trial068 ceph-mon[38805]: pgmap v657: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:23:26.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:23:26 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3642768355' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:23:27.145 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 601 8 365281 365273 4.74787 0 - 0.0299513 2026-02-23T13:23:27.260 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:23:27.377 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:23:27.485 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:23:27.484+0000 7fafa4a77640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:23:27.485 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:23:27.484+0000 7fafa4a77640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:23:27.497 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:23:27.498 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:23:27.507 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:23:27.508 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:23:27.508 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:23:27.518 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:23:27.518 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:23:27.548 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:23:27.549 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:23:28.145 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 602 8 365281 365273 4.73999 0 - 0.0299513 2026-02-23T13:23:28.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:23:28 trial068 ceph-mon[38805]: pgmap v658: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:23:28.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:23:28 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1376570771' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:23:29.144 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 603 8 365281 365273 4.73213 0 - 0.0299513 2026-02-23T13:23:29.550 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:23:29.667 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:23:29.764 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:23:29.763+0000 7f11a5a7e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:23:29.765 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:23:29.763+0000 7f11a5a7e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:23:29.777 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:23:29.777 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:23:29.787 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:23:29.787 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:23:29.788 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:23:29.797 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:23:29.797 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:23:29.825 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:23:29.826 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:23:30.144 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 604 8 365281 365273 4.72429 0 - 0.0299513 2026-02-23T13:23:30.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:23:30 trial068 ceph-mon[38805]: pgmap v659: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:23:30.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:23:30 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2653405931' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:23:31.145 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 605 8 365281 365273 4.71648 0 - 0.0299513 2026-02-23T13:23:31.827 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:23:31.947 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:23:32.047 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:23:32.045+0000 7ff1b790c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:23:32.047 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:23:32.046+0000 7ff1b790c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:23:32.059 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:23:32.059 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:23:32.070 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:23:32.071 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:23:32.071 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:23:32.080 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:23:32.081 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:23:32.110 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:23:32.111 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:23:32.145 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 606 8 365281 365273 4.7087 0 - 0.0299513 2026-02-23T13:23:32.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:23:32 trial068 ceph-mon[38805]: pgmap v660: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:23:32.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:23:32 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/4272692641' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:23:33.145 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 607 8 365281 365273 4.70094 0 - 0.0299513 2026-02-23T13:23:34.112 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:23:34.145 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 608 8 365281 365273 4.69321 0 - 0.0299513 2026-02-23T13:23:34.230 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:23:34.252 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:23:34 trial068 ceph-mon[38805]: pgmap v661: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:23:34.252 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:23:33 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:23:33] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:23:34.265 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:23:33 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:23:33.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:23:34.326 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:23:34.325+0000 7fb6335b3640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:23:34.326 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:23:34.325+0000 7fb6335b3640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:23:34.338 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:23:34.339 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:23:34.349 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:23:34.349 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:23:34.349 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:23:34.359 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:23:34.359 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:23:34.388 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:23:34.388 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:23:35.145 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 609 8 365281 365273 4.6855 0 - 0.0299513 2026-02-23T13:23:35.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:23:35 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1790222336' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:23:35.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:23:35 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2026-02-23T13:23:36.145 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 610 8 365281 365273 4.67782 0 - 0.0299513 2026-02-23T13:23:36.390 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:23:36.505 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:23:36.527 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:23:36 trial068 ceph-mon[38805]: pgmap v662: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:23:36.527 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:23:36 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-23T13:23:36.527 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:23:36 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2026-02-23T13:23:36.527 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:23:36 trial068 ceph-mon[38805]: pgmap v663: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:23:36.527 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:23:36 trial068 ceph-mon[38805]: pgmap v664: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:23:36.527 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:23:36 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T13:23:36.527 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:23:36 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2026-02-23T13:23:36.610 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:23:36.608+0000 7fde27df5640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:23:36.610 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:23:36.609+0000 7fde27df5640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:23:36.621 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:23:36.622 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:23:36.631 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:23:36.632 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:23:36.632 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:23:36.643 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:23:36.643 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:23:36.673 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:23:36.673 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:23:37.146 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 611 8 365281 365273 4.67017 0 - 0.0299513 2026-02-23T13:23:37.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:23:37 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/931012459' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:23:38.145 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 612 8 365281 365273 4.66254 0 - 0.0299513 2026-02-23T13:23:38.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:23:38 trial068 ceph-mon[38805]: pgmap v665: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:23:38.674 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:23:38.788 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:23:38.887 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:23:38.885+0000 7fa1890ab640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:23:38.887 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:23:38.886+0000 7fa1890ab640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:23:38.899 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:23:38.899 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:23:38.909 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:23:38.910 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:23:38.910 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:23:38.920 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:23:38.920 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:23:38.950 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:23:38.950 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:23:39.145 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 613 8 365281 365273 4.65493 0 - 0.0299513 2026-02-23T13:23:39.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:23:39 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1436344350' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:23:40.145 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 614 8 365281 365273 4.64735 0 - 0.0299513 2026-02-23T13:23:40.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:23:40 trial068 ceph-mon[38805]: pgmap v666: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:23:40.951 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:23:41.068 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:23:41.145 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 615 8 365281 365273 4.63979 0 - 0.0299513 2026-02-23T13:23:41.164 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:23:41.163+0000 7efedd025640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:23:41.165 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:23:41.164+0000 7efedd025640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:23:41.176 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:23:41.177 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:23:41.187 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:23:41.188 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:23:41.188 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:23:41.198 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:23:41.198 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:23:41.228 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:23:41.229 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:23:41.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:23:41 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2889668224' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:23:42.146 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 616 8 365281 365273 4.63226 0 - 0.0299513 2026-02-23T13:23:42.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:23:42 trial068 ceph-mon[38805]: pgmap v667: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:23:43.146 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 617 8 365281 365273 4.62475 0 - 0.0299513 2026-02-23T13:23:43.230 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:23:43.348 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:23:43.446 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:23:43.444+0000 7fbeb2a58640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:23:43.446 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:23:43.445+0000 7fbeb2a58640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:23:43.457 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:23:43.457 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:23:43.467 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:23:43.468 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:23:43.468 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:23:43.478 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:23:43.478 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:23:43.508 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:23:43 trial068 ceph-mon[38805]: osdmap e193: 4 total, 3 up, 4 in 2026-02-23T13:23:43.509 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:23:43.510 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:23:44.146 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 618 8 365281 365273 4.61727 0 - 0.0299513 2026-02-23T13:23:44.256 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:23:43 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:23:43] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:23:44.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:23:43 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:23:43.974Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:23:44.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:23:44 trial068 ceph-mon[38805]: pgmap v669: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:23:44.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:23:44 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/159328349' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:23:45.146 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 619 8 365281 365273 4.60981 0 - 0.0299513 2026-02-23T13:23:45.511 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:23:45.630 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:23:45.728 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:23:45.727+0000 7f5bf8f9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:23:45.728 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:23:45.727+0000 7f5bf8f9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:23:45.740 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:23:45.740 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:23:45.750 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:23:45.750 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:23:45.750 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:23:45.760 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:23:45.760 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:23:45.788 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:23:45.789 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:23:46.146 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T13:23:46.146471+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T13:23:46.146 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T13:23:46.147 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 620 8 365281 365273 4.60237 0 - 0.0299513 2026-02-23T13:23:46.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:23:46 trial068 ceph-mon[38805]: pgmap v670: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:23:46.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:23:46 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/440807183' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:23:47.146 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 621 8 365281 365273 4.59496 0 - 0.0299513 2026-02-23T13:23:47.790 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:23:47.907 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:23:48.009 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:23:48.008+0000 7f5294809640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:23:48.009 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:23:48.008+0000 7f5294809640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:23:48.021 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:23:48.021 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:23:48.032 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:23:48.032 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:23:48.032 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:23:48.042 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:23:48.042 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:23:48.072 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:23:48.072 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:23:48.146 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 622 8 365281 365273 4.58758 0 - 0.0299513 2026-02-23T13:23:48.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:23:48 trial068 ceph-mon[38805]: pgmap v671: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:23:48.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:23:48 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/4114668232' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:23:49.146 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 623 8 365281 365273 4.58021 0 - 0.0299513 2026-02-23T13:23:50.074 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:23:50.147 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 624 8 365281 365273 4.57287 0 - 0.0299513 2026-02-23T13:23:50.191 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:23:50.298 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:23:50.296+0000 7fe8a5632640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:23:50.298 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:23:50.297+0000 7fe8a5632640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:23:50.310 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:23:50.310 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:23:50.320 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:23:50.321 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:23:50.321 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:23:50.331 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:23:50.331 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:23:50.359 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:23:50 trial068 ceph-mon[38805]: pgmap v672: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:23:50.360 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:23:50.360 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:23:51.146 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 625 8 365281 365273 4.56555 0 - 0.0299513 2026-02-23T13:23:51.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:23:51 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T13:23:51.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:23:51 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T13:23:51.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:23:51 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1677492070' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:23:52.146 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 626 8 365281 365273 4.55826 0 - 0.0299513 2026-02-23T13:23:52.362 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:23:52.479 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:23:52.521 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:23:52 trial068 ceph-mon[38805]: 5.5 scrub starts 2026-02-23T13:23:52.521 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:23:52 trial068 ceph-mon[38805]: 5.5 scrub ok 2026-02-23T13:23:52.521 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:23:52 trial068 ceph-mon[38805]: pgmap v673: 17 pgs: 1 active+clean+scrubbing, 1 active+recovery_unfound+undersized+degraded, 4 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:23:52.576 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:23:52.575+0000 7fc0493ea640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:23:52.576 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:23:52.575+0000 7fc0493ea640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:23:52.588 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:23:52.588 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:23:52.598 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:23:52.598 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:23:52.598 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:23:52.609 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:23:52.609 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:23:52.637 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:23:52.637 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:23:53.147 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 627 8 365281 365273 4.55099 0 - 0.0299513 2026-02-23T13:23:53.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:23:53 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2031858989' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:23:54.147 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 628 8 365281 365273 4.54374 0 - 0.0299513 2026-02-23T13:23:54.266 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:23:53 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:23:53] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:23:54.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:23:53 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:23:53.974Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:23:54.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:23:54 trial068 ceph-mon[38805]: pgmap v674: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 1 active+clean+scrubbing, 4 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:23:54.638 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:23:54.756 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:23:54.851 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:23:54.850+0000 7f76c610c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:23:54.851 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:23:54.850+0000 7f76c610c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:23:54.864 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:23:54.864 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:23:54.874 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:23:54.875 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:23:54.875 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:23:54.886 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:23:54.886 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:23:54.917 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:23:54.918 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:23:55.147 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 629 8 365281 365273 4.53652 0 - 0.0299513 2026-02-23T13:23:55.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:23:55 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1525544726' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:23:56.147 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 630 8 365281 365273 4.52932 0 - 0.0299513 2026-02-23T13:23:56.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:23:56 trial068 ceph-mon[38805]: pgmap v675: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 1 active+clean+scrubbing, 4 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:23:56.919 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:23:57.037 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:23:57.139 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:23:57.138+0000 7feca12bd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:23:57.140 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:23:57.138+0000 7feca12bd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:23:57.147 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 631 8 365281 365273 4.52214 0 - 0.0299513 2026-02-23T13:23:57.151 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:23:57.152 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:23:57.163 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:23:57.164 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:23:57.164 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:23:57.173 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:23:57.174 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:23:57.204 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:23:57.205 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:23:57.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:23:57 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T13:23:57.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:23:57 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T13:23:57.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:23:57 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1160657787' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:23:58.147 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 632 8 365281 365273 4.51499 0 - 0.0299513 2026-02-23T13:23:58.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:23:58 trial068 ceph-mon[38805]: pgmap v676: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:23:59.147 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 633 8 365281 365273 4.50785 0 - 0.0299513 2026-02-23T13:23:59.206 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:23:59.325 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:23:59.424 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:23:59.423+0000 7f0d8b23c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:23:59.424 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:23:59.423+0000 7f0d8b23c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:23:59.436 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:23:59.436 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:23:59.446 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:23:59.446 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:23:59.446 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:23:59.456 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:23:59.456 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:23:59.485 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:23:59.486 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:24:00.147 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 634 8 365281 365273 4.50074 0 - 0.0299513 2026-02-23T13:24:00.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:24:00 trial068 ceph-mon[38805]: 5.a scrub starts 2026-02-23T13:24:00.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:24:00 trial068 ceph-mon[38805]: 5.a scrub ok 2026-02-23T13:24:00.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:24:00 trial068 ceph-mon[38805]: pgmap v677: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:24:00.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:24:00 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1024653527' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:24:01.147 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 635 8 365281 365273 4.49366 0 - 0.0299513 2026-02-23T13:24:01.487 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:24:01.605 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:24:01.701 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:24:01.700+0000 7f80e25d4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:24:01.702 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:24:01.700+0000 7f80e25d4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:24:01.713 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:24:01.713 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:24:01.723 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:24:01.723 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:24:01.723 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:24:01.734 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:24:01.734 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:24:01.765 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:24:01.766 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:24:02.147 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 636 8 365281 365273 4.48659 0 - 0.0299513 2026-02-23T13:24:02.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:24:02 trial068 ceph-mon[38805]: pgmap v678: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:24:02.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:24:02 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3837400671' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:24:03.148 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 637 8 365281 365273 4.47955 0 - 0.0299513 2026-02-23T13:24:03.768 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:24:03.885 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:24:03.990 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:24:03.989+0000 7fd312c77640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:24:03.990 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:24:03.989+0000 7fd312c77640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:24:04.002 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:24:04.002 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:24:04.012 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:24:04.013 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:24:04.013 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:24:04.023 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:24:04.023 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:24:04.052 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:24:03 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:24:03] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:24:04.052 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:24:04.053 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:24:04.148 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 638 8 365281 365273 4.47253 0 - 0.0299513 2026-02-23T13:24:04.264 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:24:03 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:24:03.972Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:24:04.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:24:04 trial068 ceph-mon[38805]: pgmap v679: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:24:04.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:24:04 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2801132545' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:24:05.147 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 639 8 365281 365273 4.46553 0 - 0.0299513 2026-02-23T13:24:06.054 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:24:06.148 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T13:24:06.148067+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T13:24:06.148 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T13:24:06.148 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 640 8 365281 365273 4.45855 0 - 0.0299513 2026-02-23T13:24:06.173 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:24:06.272 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:24:06.271+0000 7fac20d46640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:24:06.273 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:24:06.271+0000 7fac20d46640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:24:06.285 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:24:06.285 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:24:06.295 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:24:06.295 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:24:06.295 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:24:06.305 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:24:06.306 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:24:06.336 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:24:06 trial068 ceph-mon[38805]: pgmap v680: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:24:06.337 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:24:06.338 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:24:07.148 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 641 8 365281 365273 4.45159 0 - 0.0299513 2026-02-23T13:24:07.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:24:07 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1367366600' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:24:08.148 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 642 8 365281 365273 4.44466 0 - 0.0299513 2026-02-23T13:24:08.338 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:24:08.465 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:24:08.508 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:24:08 trial068 ceph-mon[38805]: pgmap v681: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:24:08.562 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:24:08.561+0000 7fdaa4602640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:24:08.563 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:24:08.562+0000 7fdaa4602640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:24:08.576 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:24:08.576 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:24:08.586 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:24:08.586 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:24:08.586 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:24:08.596 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:24:08.597 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:24:08.627 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:24:08.627 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:24:09.148 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 643 8 365281 365273 4.43775 0 - 0.0299513 2026-02-23T13:24:09.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:24:09 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/248717074' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:24:10.148 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 644 8 365281 365273 4.43086 0 - 0.0299513 2026-02-23T13:24:10.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:24:10 trial068 ceph-mon[38805]: pgmap v682: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:24:10.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:24:10 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T13:24:10.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:24:10 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T13:24:10.628 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:24:10.744 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:24:10.841 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:24:10.840+0000 7f53fe3a7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:24:10.842 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:24:10.840+0000 7f53fe3a7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:24:10.853 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:24:10.854 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:24:10.863 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:24:10.864 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:24:10.864 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:24:10.874 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:24:10.874 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:24:10.904 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:24:10.905 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:24:11.148 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 645 8 365281 365273 4.42399 0 - 0.0299513 2026-02-23T13:24:11.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:24:11 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/512984454' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:24:12.148 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 646 8 365281 365273 4.41714 0 - 0.0299513 2026-02-23T13:24:12.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:24:12 trial068 ceph-mon[38805]: pgmap v683: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:24:12.905 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:24:13.020 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:24:13.117 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:24:13.115+0000 7f0a7c1b6640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:24:13.117 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:24:13.116+0000 7f0a7c1b6640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:24:13.129 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:24:13.129 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:24:13.139 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:24:13.140 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:24:13.140 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:24:13.148 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 647 8 365281 365273 4.41031 0 - 0.0299513 2026-02-23T13:24:13.150 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:24:13.150 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:24:13.180 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:24:13.180 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:24:13.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:24:13 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3137347336' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:24:14.148 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 648 8 365281 365273 4.40351 0 - 0.0299513 2026-02-23T13:24:14.268 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:24:13 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:24:13] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:24:14.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:24:13 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:24:13.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:24:14.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:24:14 trial068 ceph-mon[38805]: pgmap v684: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:24:15.148 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 649 8 365281 365273 4.39672 0 - 0.0299513 2026-02-23T13:24:15.181 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:24:15.301 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:24:15.402 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:24:15.400+0000 7f991c0ef640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:24:15.402 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:24:15.400+0000 7f991c0ef640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:24:15.414 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:24:15.414 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:24:15.424 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:24:15.424 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:24:15.424 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:24:15.435 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:24:15.435 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:24:15.465 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:24:15.466 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:24:16.149 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 650 8 365281 365273 4.38996 0 - 0.0299513 2026-02-23T13:24:16.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:24:16 trial068 ceph-mon[38805]: pgmap v685: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:24:16.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:24:16 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3352671977' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:24:17.149 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 651 8 365281 365273 4.38321 0 - 0.0299513 2026-02-23T13:24:17.467 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:24:17.586 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:24:17.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:24:17 trial068 ceph-mon[38805]: 5.8 scrub starts 2026-02-23T13:24:17.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:24:17 trial068 ceph-mon[38805]: 5.8 scrub ok 2026-02-23T13:24:17.683 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:24:17.682+0000 7fcf12c3c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:24:17.683 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:24:17.682+0000 7fcf12c3c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:24:17.695 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:24:17.696 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:24:17.706 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:24:17.707 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:24:17.707 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:24:17.717 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:24:17.717 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:24:17.747 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:24:17.748 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:24:18.149 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 652 8 365281 365273 4.37649 0 - 0.0299513 2026-02-23T13:24:18.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:24:18 trial068 ceph-mon[38805]: pgmap v686: 17 pgs: 1 active+clean+scrubbing, 1 active+recovery_unfound+undersized+degraded, 4 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:24:18.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:24:18 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3594353880' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:24:19.149 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 653 8 365281 365273 4.36979 0 - 0.0299513 2026-02-23T13:24:19.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:24:19 trial068 ceph-mon[38805]: 5.7 scrub starts 2026-02-23T13:24:19.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:24:19 trial068 ceph-mon[38805]: 5.7 scrub ok 2026-02-23T13:24:19.750 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:24:19.868 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:24:19.977 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:24:19.976+0000 7fde26777640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:24:19.977 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:24:19.976+0000 7fde26777640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:24:19.989 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:24:19.989 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:24:19.999 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:24:20.000 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:24:20.000 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:24:20.010 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:24:20.010 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:24:20.041 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:24:20.041 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:24:20.149 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 654 8 365281 365273 4.36311 0 - 0.0299513 2026-02-23T13:24:20.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:24:20 trial068 ceph-mon[38805]: pgmap v687: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 1 active+clean+scrubbing, 4 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:24:20.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:24:20 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3364632987' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:24:21.148 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 655 8 365281 365273 4.35645 0 - 0.0299513 2026-02-23T13:24:22.042 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:24:22.149 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 656 8 365281 365273 4.3498 0 - 0.0299513 2026-02-23T13:24:22.159 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:24:22.265 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:24:22.263+0000 7fd4c716d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:24:22.265 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:24:22.263+0000 7fd4c716d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:24:22.277 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:24:22.277 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:24:22.287 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:24:22.287 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:24:22.287 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:24:22.298 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:24:22.299 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:24:22.327 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:24:22 trial068 ceph-mon[38805]: pgmap v688: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:24:22.328 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:24:22 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2640012319' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:24:22.328 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:24:22.329 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:24:23.149 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 657 8 365281 365273 4.34318 0 - 0.0299513 2026-02-23T13:24:24.149 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 658 8 365281 365273 4.33658 0 - 0.0299513 2026-02-23T13:24:24.275 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:24:23 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:24:23] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:24:24.330 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:24:24.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:24:23 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:24:23.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:24:24.445 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:24:24.544 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:24:24.543+0000 7fa55e60f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:24:24.545 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:24:24.543+0000 7fa55e60f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:24:24.545 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:24:24 trial068 ceph-mon[38805]: pgmap v689: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:24:24.556 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:24:24.556 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:24:24.566 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:24:24.567 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:24:24.567 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:24:24.577 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:24:24.577 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:24:24.608 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:24:24.608 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:24:25.149 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 659 8 365281 365273 4.33 0 - 0.0299513 2026-02-23T13:24:25.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:24:25 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3133341568' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:24:26.149 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T13:24:26.149588+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T13:24:26.149 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T13:24:26.149 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 660 8 365281 365273 4.32344 0 - 0.0299513 2026-02-23T13:24:26.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:24:26 trial068 ceph-mon[38805]: pgmap v690: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:24:26.609 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:24:26.726 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:24:26.820 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:24:26.819+0000 7f05fb567640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:24:26.821 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:24:26.819+0000 7f05fb567640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:24:26.832 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:24:26.833 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:24:26.842 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:24:26.843 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:24:26.843 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:24:26.853 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:24:26.853 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:24:26.882 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:24:26.882 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:24:27.149 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 661 8 365281 365273 4.3169 0 - 0.0299513 2026-02-23T13:24:27.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:24:27 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/166090970' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:24:28.149 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 662 8 365281 365273 4.31038 0 - 0.0299513 2026-02-23T13:24:28.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:24:28 trial068 ceph-mon[38805]: pgmap v691: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:24:28.883 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:24:28.999 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:24:29.094 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:24:29.093+0000 7f6564ed3640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:24:29.095 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:24:29.094+0000 7f6564ed3640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:24:29.108 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:24:29.108 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:24:29.118 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:24:29.119 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:24:29.119 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:24:29.129 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:24:29.130 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:24:29.149 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 663 8 365281 365273 4.30388 0 - 0.0299513 2026-02-23T13:24:29.159 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:24:29.159 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:24:30.150 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 664 8 365281 365273 4.2974 0 - 0.0299513 2026-02-23T13:24:30.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:24:30 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/738556270' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:24:30.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:24:30 trial068 ceph-mon[38805]: pgmap v692: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:24:31.149 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 665 8 365281 365273 4.29094 0 - 0.0299513 2026-02-23T13:24:31.160 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:24:31.275 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:24:31.374 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:24:31.373+0000 7f6f351a0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:24:31.374 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:24:31.373+0000 7f6f351a0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:24:31.386 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:24:31.386 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:24:31.396 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:24:31.396 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:24:31.396 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:24:31.407 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:24:31.407 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:24:31.437 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:24:31.437 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:24:32.149 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 666 8 365281 365273 4.28449 0 - 0.0299513 2026-02-23T13:24:32.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:24:32 trial068 ceph-mon[38805]: pgmap v693: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:24:32.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:24:32 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2122227145' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:24:33.149 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 667 8 365281 365273 4.27807 0 - 0.0299513 2026-02-23T13:24:33.438 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:24:33.554 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:24:33.650 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:24:33.648+0000 7f83fa7b6640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:24:33.650 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:24:33.649+0000 7f83fa7b6640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:24:33.661 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:24:33.661 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:24:33.671 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:24:33.672 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:24:33.672 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:24:33.681 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:24:33.682 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:24:33.712 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:24:33.712 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:24:34.150 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 668 8 365281 365273 4.27166 0 - 0.0299513 2026-02-23T13:24:34.257 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:24:33 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:24:33] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:24:34.264 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:24:33 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:24:33.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:24:34.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:24:34 trial068 ceph-mon[38805]: pgmap v694: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:24:34.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:24:34 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3817665012' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:24:35.150 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 669 8 365281 365273 4.26528 0 - 0.0299513 2026-02-23T13:24:35.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:24:35 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2026-02-23T13:24:35.713 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:24:35.830 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:24:35.933 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:24:35.931+0000 7f3dc8237640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:24:35.933 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:24:35.932+0000 7f3dc8237640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:24:35.945 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:24:35.945 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:24:35.955 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:24:35.955 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:24:35.955 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:24:35.966 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:24:35.967 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:24:35.999 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:24:36.000 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:24:36.150 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 670 8 365281 365273 4.25891 0 - 0.0299513 2026-02-23T13:24:36.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:24:36 trial068 ceph-mon[38805]: pgmap v695: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:24:36.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:24:36 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-23T13:24:36.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:24:36 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2026-02-23T13:24:36.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:24:36 trial068 ceph-mon[38805]: pgmap v696: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:24:36.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:24:36 trial068 ceph-mon[38805]: pgmap v697: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:24:36.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:24:36 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T13:24:36.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:24:36 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2026-02-23T13:24:36.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:24:36 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3407663155' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:24:37.150 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 671 8 365281 365273 4.25257 0 - 0.0299513 2026-02-23T13:24:37.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:24:37 trial068 ceph-mon[38805]: 5.e scrub starts 2026-02-23T13:24:37.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:24:37 trial068 ceph-mon[38805]: 5.e scrub ok 2026-02-23T13:24:38.001 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:24:38.121 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:24:38.150 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 672 8 365281 365273 4.24624 0 - 0.0299513 2026-02-23T13:24:38.221 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:24:38.220+0000 7f06f844b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:24:38.221 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:24:38.220+0000 7f06f844b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:24:38.234 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:24:38.234 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:24:38.244 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:24:38.244 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:24:38.244 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:24:38.255 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:24:38.255 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:24:38.284 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:24:38 trial068 ceph-mon[38805]: pgmap v698: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:24:38.284 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:24:38 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/144299578' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:24:38.285 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:24:38.286 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:24:39.150 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 673 8 365281 365273 4.23993 0 - 0.0299513 2026-02-23T13:24:40.150 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 674 8 365281 365273 4.23364 0 - 0.0299513 2026-02-23T13:24:40.287 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:24:40.406 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:24:40.507 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:24:40.505+0000 7fda17b58640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:24:40.507 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:24:40.505+0000 7fda17b58640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:24:40.519 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:24:40.519 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:24:40.529 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:24:40.529 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:24:40.529 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:24:40.539 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:24:40.539 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:24:40.567 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:24:40 trial068 ceph-mon[38805]: pgmap v699: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:24:40.568 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:24:40 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2217811647' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:24:40.569 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:24:40.569 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:24:41.150 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 675 8 365281 365273 4.22737 0 - 0.0299513 2026-02-23T13:24:42.150 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 676 8 365281 365273 4.22111 0 - 0.0299513 2026-02-23T13:24:42.571 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:24:42.688 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:24:42.789 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:24:42.787+0000 7f1233271640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:24:42.789 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:24:42 trial068 ceph-mon[38805]: pgmap v700: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:24:42.790 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:24:42.788+0000 7f1233271640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:24:42.802 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:24:42.802 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:24:42.813 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:24:42.813 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:24:42.814 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:24:42.824 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:24:42.824 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:24:42.855 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:24:42.855 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:24:43.151 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 677 8 365281 365273 4.21488 0 - 0.0299513 2026-02-23T13:24:43.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:24:43 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/61516583' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:24:44.151 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 678 8 365281 365273 4.20866 0 - 0.0299513 2026-02-23T13:24:44.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:24:43 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:24:43] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:24:44.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:24:43 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:24:43.974Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:24:44.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:24:44 trial068 ceph-mon[38805]: pgmap v701: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:24:44.856 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:24:44.972 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:24:45.073 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:24:45.072+0000 7f1db4c97640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:24:45.074 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:24:45.072+0000 7f1db4c97640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:24:45.085 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:24:45.085 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:24:45.095 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:24:45.095 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:24:45.095 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:24:45.106 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:24:45.106 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:24:45.136 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:24:45.136 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:24:45.150 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 679 8 365281 365273 4.20246 0 - 0.0299513 2026-02-23T13:24:45.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:24:45 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2215731289' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:24:46.151 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T13:24:46.151079+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T13:24:46.151 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T13:24:46.151 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 680 8 365281 365273 4.19628 0 - 0.0299513 2026-02-23T13:24:46.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:24:46 trial068 ceph-mon[38805]: pgmap v702: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:24:47.137 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:24:47.151 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 681 8 365281 365273 4.19012 0 - 0.0299513 2026-02-23T13:24:47.253 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:24:47.352 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:24:47.350+0000 7f5934a12640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:24:47.352 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:24:47.351+0000 7f5934a12640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:24:47.364 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:24:47.364 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:24:47.375 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:24:47.376 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:24:47.376 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:24:47.385 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:24:47.386 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:24:47.416 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:24:47.416 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:24:47.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:24:47 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2893032739' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:24:48.151 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 682 8 365281 365273 4.18398 0 - 0.0299513 2026-02-23T13:24:48.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:24:48 trial068 ceph-mon[38805]: pgmap v703: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:24:49.151 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 683 8 365281 365273 4.17785 0 - 0.0299513 2026-02-23T13:24:49.417 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:24:49.536 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:24:49.633 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:24:49.632+0000 7f33dbb96640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:24:49.634 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:24:49.632+0000 7f33dbb96640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:24:49.645 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:24:49.646 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:24:49.656 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:24:49.657 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:24:49.657 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:24:49.666 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:24:49.667 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:24:49.696 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:24:49.697 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:24:50.151 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 684 8 365281 365273 4.17174 0 - 0.0299513 2026-02-23T13:24:50.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:24:50 trial068 ceph-mon[38805]: pgmap v704: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:24:50.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:24:50 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3133138895' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:24:51.151 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 685 8 365281 365273 4.16565 0 - 0.0299513 2026-02-23T13:24:51.698 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:24:51.814 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:24:51.915 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:24:51.913+0000 7fba00460640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:24:51.915 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:24:51.913+0000 7fba00460640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:24:51.926 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:24:51.927 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:24:51.937 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:24:51.937 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:24:51.937 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:24:51.947 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:24:51.947 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:24:51.978 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:24:51.978 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:24:52.151 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 686 8 365281 365273 4.15958 0 - 0.0299513 2026-02-23T13:24:52.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:24:52 trial068 ceph-mon[38805]: pgmap v705: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:24:52.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:24:52 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1764560150' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:24:53.151 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 687 8 365281 365273 4.15353 0 - 0.0299513 2026-02-23T13:24:53.979 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:24:54.095 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:24:54.151 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 688 8 365281 365273 4.14749 0 - 0.0299513 2026-02-23T13:24:54.195 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:24:54.194+0000 7f0f896ce640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:24:54.196 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:24:54.194+0000 7f0f896ce640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:24:54.208 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:24:54.208 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:24:54.218 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:24:54.218 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:24:54.218 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:24:54.228 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:24:54.228 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:24:54.228 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:24:53 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:24:53] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:24:54.259 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:24:54.259 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:24:54.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:24:53 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:24:53.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:24:54.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:24:54 trial068 ceph-mon[38805]: pgmap v706: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:24:54.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:24:54 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3464114384' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:24:55.151 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 689 8 365281 365273 4.14147 0 - 0.0299513 2026-02-23T13:24:56.151 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 690 8 365281 365273 4.13547 0 - 0.0299513 2026-02-23T13:24:56.260 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:24:56.378 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:24:56.482 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:24:56.481+0000 7f2081ea2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:24:56.483 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:24:56.481+0000 7f2081ea2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:24:56.494 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:24:56.494 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:24:56.504 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:24:56.505 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:24:56.505 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:24:56.515 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:24:56.515 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:24:56.543 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:24:56 trial068 ceph-mon[38805]: pgmap v707: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:24:56.543 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:24:56 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2948020894' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:24:56.544 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:24:56.545 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:24:57.151 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 691 8 365281 365273 4.12948 0 - 0.0299513 2026-02-23T13:24:58.152 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 692 8 365281 365273 4.12351 0 - 0.0299513 2026-02-23T13:24:58.546 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:24:58.662 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:24:58.756 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:24:58.755+0000 7f5e5de0c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:24:58.757 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:24:58.756+0000 7f5e5de0c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:24:58.769 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:24:58.769 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:24:58.779 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:24:58.780 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:24:58.780 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:24:58 trial068 ceph-mon[38805]: pgmap v708: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:24:58.780 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:24:58 trial068 ceph-mon[38805]: osdmap e194: 4 total, 3 up, 4 in 2026-02-23T13:24:58.780 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:24:58.790 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:24:58.790 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:24:58.820 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:24:58.821 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:24:59.152 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 693 8 365281 365273 4.11756 0 - 0.0299513 2026-02-23T13:24:59.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:24:59 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3239465704' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:25:00.152 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 694 8 365281 365273 4.11163 0 - 0.0299513 2026-02-23T13:25:00.822 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:25:00.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:25:00 trial068 ceph-mon[38805]: pgmap v710: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:25:00.941 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:25:01.040 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:25:01.039+0000 7fa3b7176640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:25:01.040 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:25:01.039+0000 7fa3b7176640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:25:01.052 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:25:01.052 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:25:01.062 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:25:01.062 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:25:01.062 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:25:01.072 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:25:01.073 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:25:01.103 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:25:01.103 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:25:01.152 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 695 8 365281 365273 4.10572 0 - 0.0299513 2026-02-23T13:25:01.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:25:01 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2655979875' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:25:02.152 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 696 8 365281 365273 4.09982 0 - 0.0299513 2026-02-23T13:25:02.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:25:02 trial068 ceph-mon[38805]: pgmap v711: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:25:02.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:25:02 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T13:25:02.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:25:02 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T13:25:03.104 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:25:03.152 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 697 8 365281 365273 4.09393 0 - 0.0299513 2026-02-23T13:25:03.223 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:25:03.320 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:25:03.318+0000 7fe7aac91640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:25:03.320 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:25:03.318+0000 7fe7aac91640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:25:03.331 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:25:03.331 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:25:03.341 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:25:03.341 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:25:03.341 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:25:03.351 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:25:03.351 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:25:03.381 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:25:03.382 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:25:03.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:25:03 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2692736929' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:25:04.152 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 698 8 365281 365273 4.08807 0 - 0.0299513 2026-02-23T13:25:04.264 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:25:03 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:25:03.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:25:04.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:25:03 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:25:03] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:25:04.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:25:04 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T13:25:04.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:25:04 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T13:25:04.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:25:04 trial068 ceph-mon[38805]: pgmap v712: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:25:05.152 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 699 8 365281 365273 4.08222 0 - 0.0299513 2026-02-23T13:25:05.383 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:25:05.502 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:25:05.600 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:25:05.599+0000 7f407ae0d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:25:05.601 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:25:05.599+0000 7f407ae0d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:25:05.613 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:25:05.613 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:25:05.623 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:25:05.623 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:25:05.624 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:25:05.633 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:25:05.633 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:25:05.662 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:25:05.663 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:25:06.152 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T13:25:06.152590+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T13:25:06.152 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T13:25:06.152 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 700 8 365281 365273 4.07639 0 - 0.0299513 2026-02-23T13:25:06.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:25:06 trial068 ceph-mon[38805]: pgmap v713: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:25:06.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:25:06 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1435094591' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:25:07.152 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 701 8 365281 365273 4.07057 0 - 0.0299513 2026-02-23T13:25:07.664 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:25:07.781 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:25:07.877 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:25:07.875+0000 7f8a1188c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:25:07.877 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:25:07.876+0000 7f8a1188c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:25:07.889 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:25:07.890 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:25:07.899 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:25:07.900 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:25:07.900 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:25:07.910 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:25:07.910 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:25:07.939 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:25:07.940 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:25:08.152 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 702 8 365281 365273 4.06478 0 - 0.0299513 2026-02-23T13:25:08.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:25:08 trial068 ceph-mon[38805]: pgmap v714: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:25:08.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:25:08 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/61820570' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:25:09.152 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 703 8 365281 365273 4.05899 0 - 0.0299513 2026-02-23T13:25:09.941 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:25:10.061 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:25:10.153 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 704 8 365281 365273 4.05323 0 - 0.0299513 2026-02-23T13:25:10.161 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:25:10.159+0000 7fcafeb78640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:25:10.161 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:25:10.160+0000 7fcafeb78640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:25:10.172 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:25:10.173 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:25:10.182 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:25:10.183 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:25:10.183 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:25:10.193 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:25:10.193 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:25:10.225 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:25:10.225 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:25:10.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:25:10 trial068 ceph-mon[38805]: pgmap v715: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:25:10.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:25:10 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/124353121' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:25:11.153 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 705 8 365281 365273 4.04748 0 - 0.0299513 2026-02-23T13:25:12.153 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 706 8 365281 365273 4.04174 0 - 0.0299513 2026-02-23T13:25:12.227 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:25:12.346 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:25:12.446 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:25:12.445+0000 7fda524f6640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:25:12.446 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:25:12.445+0000 7fda524f6640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:25:12.458 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:25:12.458 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:25:12.468 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:25:12.469 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:25:12.469 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:25:12.479 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:25:12.479 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:25:12.507 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:25:12.507 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:25:12.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:25:12 trial068 ceph-mon[38805]: 5.a scrub starts 2026-02-23T13:25:12.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:25:12 trial068 ceph-mon[38805]: 5.a scrub ok 2026-02-23T13:25:12.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:25:12 trial068 ceph-mon[38805]: pgmap v716: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:25:12.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:25:12 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1548644776' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:25:13.153 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 707 8 365281 365273 4.03603 0 - 0.0299513 2026-02-23T13:25:14.153 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 708 8 365281 365273 4.03033 0 - 0.0299513 2026-02-23T13:25:14.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:25:13 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:25:13] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:25:14.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:25:13 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:25:13.974Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:25:14.507 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:25:14.625 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:25:14.732 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:25:14.730+0000 7f82d57f7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:25:14.732 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:25:14.731+0000 7f82d57f7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:25:14.744 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:25:14.744 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:25:14.754 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:25:14.755 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:25:14.755 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:25:14.766 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:25:14.766 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:25:14.783 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:25:14 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T13:25:14.783 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:25:14 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T13:25:14.783 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:25:14 trial068 ceph-mon[38805]: 5.5 scrub starts 2026-02-23T13:25:14.783 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:25:14 trial068 ceph-mon[38805]: pgmap v717: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:25:14.784 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:25:14 trial068 ceph-mon[38805]: 5.5 scrub ok 2026-02-23T13:25:14.796 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:25:14.796 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:25:15.153 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 709 8 365281 365273 4.02464 0 - 0.0299513 2026-02-23T13:25:15.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:25:15 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3167388309' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:25:16.153 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 710 8 365281 365273 4.01897 0 - 0.0299513 2026-02-23T13:25:16.797 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:25:16.822 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:25:16 trial068 ceph-mon[38805]: pgmap v718: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:25:16.914 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:25:17.013 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:25:17.011+0000 7fc6374fb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:25:17.013 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:25:17.011+0000 7fc6374fb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:25:17.024 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:25:17.025 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:25:17.034 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:25:17.034 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:25:17.034 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:25:17.044 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:25:17.044 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:25:17.087 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:25:17.088 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:25:17.153 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 711 8 365281 365273 4.01332 0 - 0.0299513 2026-02-23T13:25:17.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:25:17 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3351976312' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:25:18.153 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 712 8 365281 365273 4.00768 0 - 0.0299513 2026-02-23T13:25:18.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:25:18 trial068 ceph-mon[38805]: pgmap v719: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:25:19.089 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:25:19.153 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 713 8 365281 365273 4.00206 0 - 0.0299513 2026-02-23T13:25:19.205 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:25:19.302 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:25:19.301+0000 7fd84646d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:25:19.302 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:25:19.301+0000 7fd84646d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:25:19.313 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:25:19.313 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:25:19.324 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:25:19.324 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:25:19.324 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:25:19.334 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:25:19.334 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:25:19.364 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:25:19.365 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:25:19.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:25:19 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3689079963' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:25:20.154 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 714 8 365281 365273 3.99646 0 - 0.0299513 2026-02-23T13:25:20.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:25:20 trial068 ceph-mon[38805]: pgmap v720: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:25:21.154 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 715 8 365281 365273 3.99087 0 - 0.0299513 2026-02-23T13:25:21.365 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:25:21.483 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:25:21.580 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:25:21.579+0000 7fee6ddd8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:25:21.580 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:25:21.579+0000 7fee6ddd8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:25:21.593 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:25:21.593 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:25:21.603 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:25:21.604 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:25:21.604 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:25:21.614 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:25:21.614 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:25:21.647 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:25:21.648 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:25:22.153 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 716 8 365281 365273 3.9853 0 - 0.0299513 2026-02-23T13:25:22.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:25:22 trial068 ceph-mon[38805]: pgmap v721: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:25:22.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:25:22 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2987589281' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:25:23.154 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 717 8 365281 365273 3.97974 0 - 0.0299513 2026-02-23T13:25:23.649 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:25:23.766 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:25:23.864 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:25:23.862+0000 7f0e80fe9640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:25:23.864 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:25:23.862+0000 7f0e80fe9640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:25:23.875 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:25:23.876 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:25:23.885 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:25:23.886 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:25:23.887 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:25:23.896 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:25:23.896 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:25:23.924 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:25:23.925 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:25:24.154 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 718 8 365281 365273 3.97419 0 - 0.0299513 2026-02-23T13:25:24.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:25:23 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:25:23] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:25:24.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:25:23 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:25:23.972Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:25:24.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:25:24 trial068 ceph-mon[38805]: pgmap v722: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:25:24.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:25:24 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1932004756' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:25:25.154 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 719 8 365281 365273 3.96867 0 - 0.0299513 2026-02-23T13:25:25.925 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:25:26.041 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:25:26.139 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:25:26.137+0000 7efd8d886640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:25:26.139 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:25:26.138+0000 7efd8d886640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:25:26.150 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:25:26.151 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:25:26.154 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T13:25:26.154390+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T13:25:26.154 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T13:25:26.154 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 720 8 365281 365273 3.96315 0 - 0.0299513 2026-02-23T13:25:26.160 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:25:26.161 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:25:26.161 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:25:26.171 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:25:26.171 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:25:26.201 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:25:26.201 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:25:26.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:25:26 trial068 ceph-mon[38805]: pgmap v723: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:25:26.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:25:26 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/688446668' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:25:27.154 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 721 8 365281 365273 3.95766 0 - 0.0299513 2026-02-23T13:25:28.154 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 722 8 365281 365273 3.95218 0 - 0.0299513 2026-02-23T13:25:28.202 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:25:28.321 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:25:28.416 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:25:28.415+0000 7ff48cee8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:25:28.416 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:25:28.415+0000 7ff48cee8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:25:28.428 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:25:28.429 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:25:28.438 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:25:28.439 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:25:28.439 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:25:28.449 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:25:28.449 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:25:28.481 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:25:28.482 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:25:28.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:25:28 trial068 ceph-mon[38805]: pgmap v724: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:25:28.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:25:28 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1032932069' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:25:29.154 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 723 8 365281 365273 3.94671 0 - 0.0299513 2026-02-23T13:25:30.154 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 724 8 365281 365273 3.94126 0 - 0.0299513 2026-02-23T13:25:30.482 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:25:30.508 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:25:30 trial068 ceph-mon[38805]: pgmap v725: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:25:30.601 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:25:30.700 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:25:30.699+0000 7fc673e03640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:25:30.701 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:25:30.700+0000 7fc673e03640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:25:30.713 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:25:30.713 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:25:30.723 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:25:30.724 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:25:30.724 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:25:30.733 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:25:30.733 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:25:30.764 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:25:30.765 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:25:31.154 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 725 8 365281 365273 3.93582 0 - 0.0299513 2026-02-23T13:25:31.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:25:31 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1360851220' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:25:32.154 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 726 8 365281 365273 3.9304 0 - 0.0299513 2026-02-23T13:25:32.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:25:32 trial068 ceph-mon[38805]: pgmap v726: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:25:32.766 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:25:32.881 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:25:32.984 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:25:32.983+0000 7f0bbb378640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:25:32.985 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:25:32.983+0000 7f0bbb378640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:25:32.996 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:25:32.997 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:25:33.006 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:25:33.007 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:25:33.007 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:25:33.017 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:25:33.017 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:25:33.046 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:25:33.046 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:25:33.154 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 727 8 365281 365273 3.925 0 - 0.0299513 2026-02-23T13:25:33.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:25:33 trial068 ceph-mon[38805]: 5.8 scrub starts 2026-02-23T13:25:33.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:25:33 trial068 ceph-mon[38805]: 5.8 scrub ok 2026-02-23T13:25:33.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:25:33 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/618236527' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:25:34.155 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 728 8 365281 365273 3.9196 0 - 0.0299513 2026-02-23T13:25:34.265 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:25:33 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:25:33.974Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:25:34.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:25:34 trial068 ceph-mon[38805]: pgmap v727: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:25:34.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:25:33 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:25:33] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:25:35.048 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:25:35.154 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 729 8 365281 365273 3.91423 0 - 0.0299513 2026-02-23T13:25:35.167 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:25:35.270 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:25:35.269+0000 7f10438af640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:25:35.270 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:25:35.269+0000 7f10438af640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:25:35.283 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:25:35.283 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:25:35.293 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:25:35.294 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:25:35.294 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:25:35.304 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:25:35.305 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:25:35.323 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:25:35 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2006301691' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:25:35.350 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:25:35.351 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:25:36.154 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 730 8 365281 365273 3.90887 0 - 0.0299513 2026-02-23T13:25:36.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:25:36 trial068 ceph-mon[38805]: pgmap v728: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:25:36.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:25:36 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2026-02-23T13:25:36.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:25:36 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-23T13:25:36.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:25:36 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2026-02-23T13:25:36.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:25:36 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T13:25:36.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:25:36 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2026-02-23T13:25:37.154 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 731 8 365281 365273 3.90352 0 - 0.0299513 2026-02-23T13:25:37.352 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:25:37.471 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:25:37.572 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:25:37.570+0000 7f4913974640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:25:37.572 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:25:37.571+0000 7f4913974640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:25:37.584 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:25:37.585 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:25:37.585 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:25:37 trial068 ceph-mon[38805]: pgmap v729: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:25:37.585 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:25:37 trial068 ceph-mon[38805]: pgmap v730: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:25:37.594 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:25:37.594 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:25:37.594 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:25:37.605 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:25:37.606 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:25:37.635 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:25:37.636 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:25:38.155 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 732 8 365281 365273 3.89819 0 - 0.0299513 2026-02-23T13:25:38.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:25:38 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2270294640' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:25:39.155 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 733 8 365281 365273 3.89287 0 - 0.0299513 2026-02-23T13:25:39.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:25:39 trial068 ceph-mon[38805]: pgmap v731: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:25:39.637 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:25:39.752 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:25:39.853 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:25:39.851+0000 7ff74a1ed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:25:39.853 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:25:39.852+0000 7ff74a1ed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:25:39.865 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:25:39.865 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:25:39.876 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:25:39.877 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:25:39.877 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:25:39.886 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:25:39.886 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:25:39.917 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:25:39.918 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:25:40.155 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 734 8 365281 365273 3.88756 0 - 0.0299513 2026-02-23T13:25:40.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:25:40 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2888218336' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:25:41.155 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 735 8 365281 365273 3.88228 0 - 0.0299513 2026-02-23T13:25:41.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:25:41 trial068 ceph-mon[38805]: pgmap v732: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:25:41.919 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:25:42.037 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:25:42.134 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:25:42.133+0000 7f7470a70640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:25:42.135 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:25:42.133+0000 7f7470a70640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:25:42.147 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:25:42.147 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:25:42.155 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 736 8 365281 365273 3.877 0 - 0.0299513 2026-02-23T13:25:42.157 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:25:42.157 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:25:42.157 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:25:42.168 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:25:42.169 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:25:42.215 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:25:42.216 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:25:42.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:25:42 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1454762619' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:25:43.155 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 737 8 365281 365273 3.87174 0 - 0.0299513 2026-02-23T13:25:43.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:25:43 trial068 ceph-mon[38805]: pgmap v733: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:25:44.155 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 738 8 365281 365273 3.86649 0 - 0.0299513 2026-02-23T13:25:44.217 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:25:44.242 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:25:43 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:25:43] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:25:44.335 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:25:44.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:25:43 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:25:43.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:25:44.442 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:25:44.440+0000 7f0907177640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:25:44.442 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:25:44.441+0000 7f0907177640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:25:44.454 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:25:44.454 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:25:44.463 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:25:44.464 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:25:44.464 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:25:44.473 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:25:44.474 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:25:44.503 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:25:44.504 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:25:45.155 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 739 8 365281 365273 3.86126 0 - 0.0299513 2026-02-23T13:25:45.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:25:45 trial068 ceph-mon[38805]: pgmap v734: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:25:45.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:25:45 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1311294928' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:25:46.155 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T13:25:46.155725+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T13:25:46.156 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T13:25:46.156 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 740 8 365281 365273 3.85604 0 - 0.0299513 2026-02-23T13:25:46.505 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:25:46.621 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:25:46.719 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:25:46.718+0000 7f81699b6640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:25:46.719 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:25:46.718+0000 7f81699b6640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:25:46.731 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:25:46.731 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:25:46.741 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:25:46.741 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:25:46.741 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:25:46.751 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:25:46.751 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:25:46.781 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:25:46.781 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:25:47.155 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 741 8 365281 365273 3.85084 0 - 0.0299513 2026-02-23T13:25:47.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:25:47 trial068 ceph-mon[38805]: pgmap v735: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:25:47.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:25:47 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2727169274' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:25:48.155 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 742 8 365281 365273 3.84565 0 - 0.0299513 2026-02-23T13:25:48.783 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:25:48.902 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:25:49.001 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:25:48.999+0000 7f7eb4d85640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:25:49.001 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:25:49.000+0000 7f7eb4d85640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:25:49.013 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:25:49.013 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:25:49.023 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:25:49.024 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:25:49.024 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:25:49.033 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:25:49.034 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:25:49.064 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:25:49.065 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:25:49.155 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 743 8 365281 365273 3.84047 0 - 0.0299513 2026-02-23T13:25:49.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:25:49 trial068 ceph-mon[38805]: pgmap v736: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:25:49.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:25:49 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/936302550' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:25:50.156 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 744 8 365281 365273 3.83531 0 - 0.0299513 2026-02-23T13:25:51.067 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:25:51.156 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 745 8 365281 365273 3.83016 0 - 0.0299513 2026-02-23T13:25:51.185 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:25:51.284 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:25:51.282+0000 7f5b6c5be640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:25:51.284 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:25:51.283+0000 7f5b6c5be640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:25:51.296 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:25:51.296 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:25:51.306 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:25:51.306 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:25:51.307 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:25:51.316 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:25:51.316 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:25:51.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:25:51 trial068 ceph-mon[38805]: pgmap v737: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:25:51.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:25:51 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3084499342' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:25:51.345 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:25:51.345 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:25:52.156 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 746 8 365281 365273 3.82503 0 - 0.0299513 2026-02-23T13:25:53.156 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 747 8 365281 365273 3.81991 0 - 0.0299513 2026-02-23T13:25:53.346 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:25:53.465 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:25:53.570 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:25:53.568+0000 7f7aa0025640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:25:53.570 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:25:53.569+0000 7f7aa0025640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:25:53.582 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:25:53.582 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:25:53.591 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:25:53.592 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:25:53.592 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:25:53 trial068 ceph-mon[38805]: pgmap v738: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:25:53.592 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:25:53.602 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:25:53.602 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:25:53.632 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:25:53.633 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:25:54.156 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 748 8 365281 365273 3.8148 0 - 0.0299513 2026-02-23T13:25:54.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:25:53 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:25:53] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:25:54.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:25:53 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:25:53.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:25:54.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:25:54 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2432228619' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:25:55.156 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 749 8 365281 365273 3.80971 0 - 0.0299513 2026-02-23T13:25:55.634 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:25:55.659 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:25:55 trial068 ceph-mon[38805]: pgmap v739: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:25:55.753 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:25:55.854 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:25:55.853+0000 7f3b9d808640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:25:55.855 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:25:55.853+0000 7f3b9d808640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:25:55.865 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:25:55.866 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:25:55.876 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:25:55.876 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:25:55.876 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:25:55.886 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:25:55.886 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:25:55.917 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:25:55.918 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:25:56.156 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 750 8 365281 365273 3.80463 0 - 0.0299513 2026-02-23T13:25:56.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:25:56 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3331721318' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:25:57.156 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 751 8 365281 365273 3.79956 0 - 0.0299513 2026-02-23T13:25:57.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:25:57 trial068 ceph-mon[38805]: pgmap v740: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:25:57.919 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:25:58.035 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:25:58.130 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:25:58.129+0000 7f8e7ef65640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:25:58.131 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:25:58.129+0000 7f8e7ef65640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:25:58.142 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:25:58.142 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:25:58.152 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:25:58.153 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:25:58.153 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:25:58.156 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 752 8 365281 365273 3.79451 0 - 0.0299513 2026-02-23T13:25:58.163 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:25:58.163 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:25:58.193 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:25:58.193 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:25:58.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:25:58 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/175217839' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:25:59.156 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 753 8 365281 365273 3.78947 0 - 0.0299513 2026-02-23T13:25:59.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:25:59 trial068 ceph-mon[38805]: pgmap v741: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:25:59.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:25:59 trial068 ceph-mon[38805]: 5.7 scrub starts 2026-02-23T13:25:59.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:25:59 trial068 ceph-mon[38805]: 5.7 scrub ok 2026-02-23T13:26:00.156 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 754 8 365281 365273 3.78445 0 - 0.0299513 2026-02-23T13:26:00.195 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:26:00.311 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:26:00.408 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:26:00.407+0000 7f7e37276640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:26:00.408 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:26:00.407+0000 7f7e37276640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:26:00.421 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:26:00.421 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:26:00.431 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:26:00.431 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:26:00.432 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:26:00.443 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:26:00.443 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:26:00.475 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:26:00.476 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:26:01.156 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 755 8 365281 365273 3.77943 0 - 0.0299513 2026-02-23T13:26:01.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:26:01 trial068 ceph-mon[38805]: pgmap v742: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:26:01.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:26:01 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1980819754' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:26:02.156 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 756 8 365281 365273 3.77443 0 - 0.0299513 2026-02-23T13:26:02.477 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:26:02.594 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:26:02.691 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:26:02.689+0000 7f1f0a70e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:26:02.691 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:26:02.689+0000 7f1f0a70e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:26:02.702 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:26:02.702 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:26:02.712 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:26:02.713 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:26:02.713 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:26:02.723 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:26:02.724 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:26:02.751 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:26:02.752 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:26:03.156 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 757 8 365281 365273 3.76945 0 - 0.0299513 2026-02-23T13:26:03.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:26:03 trial068 ceph-mon[38805]: pgmap v743: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:26:03.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:26:03 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1891229922' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:26:04.156 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 758 8 365281 365273 3.76448 0 - 0.0299513 2026-02-23T13:26:04.265 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:26:03 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:26:03.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:26:04.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:26:03 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:26:03] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:26:04.753 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:26:04.872 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:26:04.972 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:26:04.971+0000 7fe44de1f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:26:04.972 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:26:04.971+0000 7fe44de1f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:26:04.984 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:26:04.984 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:26:04.994 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:26:04.995 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:26:04.995 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:26:05.005 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:26:05.006 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:26:05.036 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:26:05.037 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:26:05.157 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 759 8 365281 365273 3.75952 0 - 0.0299513 2026-02-23T13:26:05.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:26:05 trial068 ceph-mon[38805]: pgmap v744: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:26:05.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:26:05 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1869369941' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:26:06.156 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T13:26:06.157145+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T13:26:06.157 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T13:26:06.157 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 760 8 365281 365273 3.75457 0 - 0.0299513 2026-02-23T13:26:07.037 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:26:07.157 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 761 8 365281 365273 3.74964 0 - 0.0299513 2026-02-23T13:26:07.157 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:26:07.260 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:26:07.258+0000 7f040e150640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:26:07.260 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:26:07.259+0000 7f040e150640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:26:07.271 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:26:07.272 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:26:07.282 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:26:07.282 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:26:07.283 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:26:07.292 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:26:07.292 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:26:07.323 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:26:07.324 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:26:07.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:26:07 trial068 ceph-mon[38805]: pgmap v745: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:26:07.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:26:07 trial068 ceph-mon[38805]: osdmap e195: 4 total, 3 up, 4 in 2026-02-23T13:26:07.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:26:07 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3380411537' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:26:08.157 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 762 8 365281 365273 3.74472 0 - 0.0299513 2026-02-23T13:26:09.157 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 763 8 365281 365273 3.73981 0 - 0.0299513 2026-02-23T13:26:09.324 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:26:09.443 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:26:09.541 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:26:09.540+0000 7f48502b9640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:26:09.542 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:26:09.540+0000 7f48502b9640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:26:09.554 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:26:09.554 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:26:09.564 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:26:09.565 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:26:09.565 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:26:09.576 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:26:09.576 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:26:09.603 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:26:09 trial068 ceph-mon[38805]: pgmap v747: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:26:09.604 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:26:09 trial068 ceph-mon[38805]: 5.e scrub starts 2026-02-23T13:26:09.604 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:26:09 trial068 ceph-mon[38805]: 5.e scrub ok 2026-02-23T13:26:09.605 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:26:09.605 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:26:10.157 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 764 8 365281 365273 3.73491 0 - 0.0299513 2026-02-23T13:26:10.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:26:10 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3939343530' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:26:11.157 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 765 8 365281 365273 3.73003 0 - 0.0299513 2026-02-23T13:26:11.607 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:26:11.631 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:26:11 trial068 ceph-mon[38805]: pgmap v748: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:26:11.723 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:26:11.820 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:26:11.818+0000 7f5dbb78b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:26:11.820 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:26:11.818+0000 7f5dbb78b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:26:11.832 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:26:11.832 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:26:11.842 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:26:11.843 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:26:11.843 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:26:11.853 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:26:11.854 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:26:11.883 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:26:11.884 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:26:12.157 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 766 8 365281 365273 3.72516 0 - 0.0299513 2026-02-23T13:26:12.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:26:12 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3777233063' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:26:13.157 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 767 8 365281 365273 3.7203 0 - 0.0299513 2026-02-23T13:26:13.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:26:13 trial068 ceph-mon[38805]: pgmap v749: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:26:13.885 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:26:14.004 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:26:14.101 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:26:14.100+0000 7f7229075640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:26:14.101 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:26:14.100+0000 7f7229075640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:26:14.113 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:26:14.113 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:26:14.124 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:26:14.124 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:26:14.125 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:26:14.134 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:26:14.135 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:26:14.157 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 768 8 365281 365273 3.71546 0 - 0.0299513 2026-02-23T13:26:14.162 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:26:13 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:26:13] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:26:14.163 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:26:14.163 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:26:14.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:26:13 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:26:13.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:26:14.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:26:14 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/4092370834' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:26:15.157 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 769 8 365281 365273 3.71063 0 - 0.0299513 2026-02-23T13:26:15.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:26:15 trial068 ceph-mon[38805]: pgmap v750: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:26:16.157 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 770 8 365281 365273 3.70581 0 - 0.0299513 2026-02-23T13:26:16.164 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:26:16.282 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:26:16.379 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:26:16.377+0000 7f6eab0b4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:26:16.379 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:26:16.378+0000 7f6eab0b4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:26:16.392 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:26:16.392 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:26:16.402 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:26:16.402 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:26:16.402 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:26:16.412 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:26:16.413 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:26:16.443 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:26:16.444 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:26:17.158 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 771 8 365281 365273 3.701 0 - 0.0299513 2026-02-23T13:26:17.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:26:17 trial068 ceph-mon[38805]: pgmap v751: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:26:17.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:26:17 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3722024740' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:26:18.158 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 772 8 365281 365273 3.69621 0 - 0.0299513 2026-02-23T13:26:18.445 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:26:18.563 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:26:18.664 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:26:18.662+0000 7f4d0fed7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:26:18.664 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:26:18.663+0000 7f4d0fed7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:26:18.676 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:26:18.676 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:26:18.687 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:26:18.687 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:26:18.687 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:26:18.697 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:26:18.697 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:26:18.727 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:26:18.728 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:26:19.158 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 773 8 365281 365273 3.69143 0 - 0.0299513 2026-02-23T13:26:19.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:26:19 trial068 ceph-mon[38805]: pgmap v752: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:26:19.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:26:19 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2588660696' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:26:20.158 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 774 8 365281 365273 3.68666 0 - 0.0299513 2026-02-23T13:26:20.728 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:26:20.844 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:26:20.940 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:26:20.939+0000 7efe628f7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:26:20.940 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:26:20.939+0000 7efe628f7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:26:20.952 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:26:20.953 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:26:20.962 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:26:20.963 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:26:20.963 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:26:20.974 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:26:20.974 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:26:21.003 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:26:21.004 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:26:21.158 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 775 8 365281 365273 3.6819 0 - 0.0299513 2026-02-23T13:26:21.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:26:21 trial068 ceph-mon[38805]: pgmap v753: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:26:21.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:26:21 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1968618226' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:26:22.158 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 776 8 365281 365273 3.67716 0 - 0.0299513 2026-02-23T13:26:23.005 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:26:23.122 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:26:23.158 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 777 8 365281 365273 3.67242 0 - 0.0299513 2026-02-23T13:26:23.224 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:26:23.223+0000 7fe72b8d2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:26:23.224 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:26:23.223+0000 7fe72b8d2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:26:23.236 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:26:23.237 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:26:23.247 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:26:23.247 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:26:23.247 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:26:23.258 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:26:23.258 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:26:23.287 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:26:23.287 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:26:23.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:26:23 trial068 ceph-mon[38805]: pgmap v754: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:26:23.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:26:23 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1144337749' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:26:24.158 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 778 8 365281 365273 3.6677 0 - 0.0299513 2026-02-23T13:26:24.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:26:23 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:26:23] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:26:24.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:26:23 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:26:23.974Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:26:25.158 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 779 8 365281 365273 3.66299 0 - 0.0299513 2026-02-23T13:26:25.288 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:26:25.407 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:26:25.507 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:26:25.506+0000 7fbcde867640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:26:25.507 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:26:25.506+0000 7fbcde867640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:26:25.519 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:26:25.519 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:26:25.528 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:26:25.529 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:26:25.529 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:26:25.540 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:26:25.540 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:26:25.568 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:26:25 trial068 ceph-mon[38805]: pgmap v755: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:26:25.568 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:26:25 trial068 ceph-mon[38805]: 5.a scrub starts 2026-02-23T13:26:25.568 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:26:25 trial068 ceph-mon[38805]: 5.a scrub ok 2026-02-23T13:26:25.569 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:26:25.570 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:26:26.158 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T13:26:26.158744+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T13:26:26.159 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T13:26:26.159 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 780 8 365281 365273 3.6583 0 - 0.0299513 2026-02-23T13:26:26.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:26:26 trial068 ceph-mon[38805]: 5.5 scrub starts 2026-02-23T13:26:26.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:26:26 trial068 ceph-mon[38805]: 5.5 scrub ok 2026-02-23T13:26:26.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:26:26 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3338743775' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:26:27.158 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 781 8 365281 365273 3.65361 0 - 0.0299513 2026-02-23T13:26:27.571 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:26:27.688 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:26:27.710 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:26:27 trial068 ceph-mon[38805]: pgmap v756: 17 pgs: 1 active+clean+scrubbing, 1 active+recovery_unfound+undersized+degraded, 4 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:26:27.791 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:26:27.790+0000 7f1c6ffea640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:26:27.791 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:26:27.790+0000 7f1c6ffea640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:26:27.803 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:26:27.803 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:26:27.813 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:26:27.814 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:26:27.814 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:26:27.824 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:26:27.824 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:26:27.853 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:26:27.854 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:26:28.159 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 782 8 365281 365273 3.64894 0 - 0.0299513 2026-02-23T13:26:28.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:26:28 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3892584836' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:26:29.159 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 783 8 365281 365273 3.64428 0 - 0.0299513 2026-02-23T13:26:29.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:26:29 trial068 ceph-mon[38805]: pgmap v757: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 1 active+clean+scrubbing, 4 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:26:29.855 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:26:29.971 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:26:30.072 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:26:30.070+0000 7ffb6b04d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:26:30.072 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:26:30.071+0000 7ffb6b04d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:26:30.084 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:26:30.084 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:26:30.094 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:26:30.095 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:26:30.095 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:26:30.105 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:26:30.105 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:26:30.133 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:26:30.134 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:26:30.159 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 784 8 365281 365273 3.63963 0 - 0.0299513 2026-02-23T13:26:30.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:26:30 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T13:26:30.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:26:30 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T13:26:30.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:26:30 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/641602818' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:26:31.158 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 785 8 365281 365273 3.635 0 - 0.0299513 2026-02-23T13:26:31.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:26:31 trial068 ceph-mon[38805]: pgmap v758: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 1 active+clean+scrubbing, 4 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:26:31.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:26:31 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T13:26:31.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:26:31 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T13:26:32.135 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:26:32.159 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 786 8 365281 365273 3.63037 0 - 0.0299513 2026-02-23T13:26:32.252 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:26:32.358 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:26:32.356+0000 7efe40e3c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:26:32.358 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:26:32.356+0000 7efe40e3c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:26:32.369 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:26:32.370 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:26:32.379 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:26:32.380 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:26:32.380 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:26:32.390 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:26:32.390 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:26:32.411 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:26:32 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T13:26:32.412 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:26:32 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T13:26:32.412 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:26:32 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1888823234' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:26:32.440 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:26:32.440 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:26:33.159 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 787 8 365281 365273 3.62576 0 - 0.0299513 2026-02-23T13:26:33.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:26:33 trial068 ceph-mon[38805]: pgmap v759: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:26:34.159 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 788 8 365281 365273 3.62116 0 - 0.0299513 2026-02-23T13:26:34.264 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:26:33 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:26:33.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:26:34.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:26:33 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:26:33] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:26:34.442 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:26:34.560 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:26:34.660 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:26:34.658+0000 7f2ce9863640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:26:34.660 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:26:34.658+0000 7f2ce9863640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:26:34.671 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:26:34.672 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:26:34.682 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:26:34.683 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:26:34.683 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:26:34.692 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:26:34.692 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:26:34.722 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:26:34.723 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:26:35.159 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 789 8 365281 365273 3.61657 0 - 0.0299513 2026-02-23T13:26:35.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:26:35 trial068 ceph-mon[38805]: pgmap v760: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:26:35.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:26:35 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2622494886' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:26:36.159 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 790 8 365281 365273 3.61199 0 - 0.0299513 2026-02-23T13:26:36.723 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:26:36.751 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:26:36 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2026-02-23T13:26:36.751 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:26:36 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T13:26:36.751 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:26:36 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T13:26:36.751 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:26:36 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T13:26:36.751 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:26:36 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T13:26:36.751 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:26:36 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-23T13:26:36.751 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:26:36 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2026-02-23T13:26:36.751 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:26:36 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T13:26:36.751 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:26:36 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2026-02-23T13:26:36.843 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:26:36.945 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:26:36.944+0000 7f14a7db2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:26:36.945 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:26:36.944+0000 7f14a7db2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:26:36.957 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:26:36.957 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:26:36.967 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:26:36.967 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:26:36.967 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:26:36.978 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:26:36.979 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:26:37.011 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:26:37.012 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:26:37.159 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 791 8 365281 365273 3.60742 0 - 0.0299513 2026-02-23T13:26:37.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:26:37 trial068 ceph-mon[38805]: pgmap v761: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:26:37.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:26:37 trial068 ceph-mon[38805]: pgmap v762: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:26:37.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:26:37 trial068 ceph-mon[38805]: pgmap v763: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:26:37.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:26:37 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/755037088' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:26:38.159 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 792 8 365281 365273 3.60287 0 - 0.0299513 2026-02-23T13:26:39.013 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:26:39.131 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:26:39.159 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 793 8 365281 365273 3.59833 0 - 0.0299513 2026-02-23T13:26:39.230 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:26:39.228+0000 7f4a0db48640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:26:39.230 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:26:39.229+0000 7f4a0db48640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:26:39.242 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:26:39.242 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:26:39.252 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:26:39.252 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:26:39.253 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:26:39.263 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:26:39.263 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:26:39.292 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:26:39.293 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:26:39.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:26:39 trial068 ceph-mon[38805]: pgmap v764: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:26:39.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:26:39 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/70934941' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:26:40.159 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 794 8 365281 365273 3.59379 0 - 0.0299513 2026-02-23T13:26:41.160 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 795 8 365281 365273 3.58927 0 - 0.0299513 2026-02-23T13:26:41.294 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:26:41.413 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:26:41.515 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:26:41.513+0000 7f0f0bc28640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:26:41.515 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:26:41.514+0000 7f0f0bc28640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:26:41.527 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:26:41.528 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:26:41.538 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:26:41.538 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:26:41.538 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:26:41.548 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:26:41.548 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:26:41.577 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:26:41 trial068 ceph-mon[38805]: pgmap v765: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:26:41.578 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:26:41.578 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:26:42.160 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 796 8 365281 365273 3.58476 0 - 0.0299513 2026-02-23T13:26:42.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:26:42 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3636715742' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:26:43.160 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 797 8 365281 365273 3.58027 0 - 0.0299513 2026-02-23T13:26:43.579 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:26:43.698 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:26:43.720 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:26:43 trial068 ceph-mon[38805]: pgmap v766: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:26:43.796 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:26:43.795+0000 7f4b77d21640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:26:43.796 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:26:43.795+0000 7f4b77d21640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:26:43.808 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:26:43.808 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:26:43.818 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:26:43.818 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:26:43.818 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:26:43.828 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:26:43.828 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:26:43.858 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:26:43.859 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:26:44.160 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 798 8 365281 365273 3.57578 0 - 0.0299513 2026-02-23T13:26:44.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:26:43 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:26:43] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:26:44.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:26:43 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:26:43.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:26:44.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:26:44 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/4066518625' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:26:45.160 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 799 8 365281 365273 3.5713 0 - 0.0299513 2026-02-23T13:26:45.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:26:45 trial068 ceph-mon[38805]: pgmap v767: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:26:45.860 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:26:45.977 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:26:46.081 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:26:46.079+0000 7f15ff08a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:26:46.081 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:26:46.080+0000 7f15ff08a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:26:46.093 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:26:46.093 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:26:46.104 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:26:46.104 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:26:46.104 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:26:46.114 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:26:46.115 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:26:46.144 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:26:46.145 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:26:46.160 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T13:26:46.160330+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T13:26:46.160 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T13:26:46.160 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 800 8 365281 365273 3.56684 0 - 0.0299513 2026-02-23T13:26:46.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:26:46 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2954123921' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:26:47.160 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 801 8 365281 365273 3.56239 0 - 0.0299513 2026-02-23T13:26:47.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:26:47 trial068 ceph-mon[38805]: pgmap v768: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:26:48.146 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:26:48.160 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 802 8 365281 365273 3.55795 0 - 0.0299513 2026-02-23T13:26:48.263 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:26:48.361 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:26:48.359+0000 7fc783bca640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:26:48.361 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:26:48.360+0000 7fc783bca640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:26:48.373 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:26:48.373 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:26:48.383 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:26:48.384 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:26:48.384 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:26:48.394 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:26:48.394 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:26:48.424 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:26:48 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3643724363' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:26:48.425 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:26:48.426 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:26:49.160 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 803 8 365281 365273 3.55352 0 - 0.0299513 2026-02-23T13:26:49.744 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:26:49 trial068 ceph-mon[38805]: pgmap v769: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:26:50.160 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 804 8 365281 365273 3.5491 0 - 0.0299513 2026-02-23T13:26:50.427 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:26:50.546 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:26:50.640 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:26:50.638+0000 7f258cdd0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:26:50.640 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:26:50.638+0000 7f258cdd0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:26:50.651 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:26:50.652 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:26:50.662 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:26:50.662 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:26:50.662 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:26:50.672 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:26:50.672 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:26:50.702 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:26:50.702 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:26:51.160 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 805 8 365281 365273 3.54469 0 - 0.0299513 2026-02-23T13:26:51.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:26:51 trial068 ceph-mon[38805]: pgmap v770: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:26:51.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:26:51 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1507366747' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:26:52.160 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 806 8 365281 365273 3.54029 0 - 0.0299513 2026-02-23T13:26:52.704 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:26:52.822 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:26:52.922 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:26:52.921+0000 7fd5dcb65640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:26:52.922 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:26:52.921+0000 7fd5dcb65640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:26:52.934 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:26:52.935 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:26:52.945 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:26:52.945 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:26:52.945 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:26:52.955 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:26:52.955 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:26:52.988 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:26:52.988 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:26:53.160 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 807 8 365281 365273 3.5359 0 - 0.0299513 2026-02-23T13:26:53.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:26:53 trial068 ceph-mon[38805]: pgmap v771: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:26:53.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:26:53 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1335144483' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:26:54.160 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 808 8 365281 365273 3.53153 0 - 0.0299513 2026-02-23T13:26:54.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:26:53 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:26:53] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:26:54.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:26:53 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:26:53.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:26:54.989 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:26:55.109 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:26:55.161 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 809 8 365281 365273 3.52716 0 - 0.0299513 2026-02-23T13:26:55.211 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:26:55.210+0000 7f0718ae7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:26:55.212 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:26:55.210+0000 7f0718ae7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:26:55.224 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:26:55.224 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:26:55.235 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:26:55.235 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:26:55.236 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:26:55.246 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:26:55.246 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:26:55.277 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:26:55.278 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:26:55.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:26:55 trial068 ceph-mon[38805]: pgmap v772: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:26:55.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:26:55 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3369014267' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:26:56.161 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 810 8 365281 365273 3.52281 0 - 0.0299513 2026-02-23T13:26:57.161 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 811 8 365281 365273 3.51846 0 - 0.0299513 2026-02-23T13:26:57.279 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:26:57.402 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:26:57.498 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:26:57.497+0000 7f8f95dc6640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:26:57.498 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:26:57.497+0000 7f8f95dc6640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:26:57.510 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:26:57.510 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:26:57.520 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:26:57.521 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:26:57.521 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:26:57.530 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:26:57.530 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:26:57.560 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:26:57 trial068 ceph-mon[38805]: pgmap v773: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:26:57.561 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:26:57.562 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:26:58.161 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 812 8 365281 365273 3.51413 0 - 0.0299513 2026-02-23T13:26:58.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:26:58 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1117682513' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:26:59.161 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 813 8 365281 365273 3.50981 0 - 0.0299513 2026-02-23T13:26:59.563 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:26:59.681 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:26:59.704 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:26:59 trial068 ceph-mon[38805]: pgmap v774: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:26:59.779 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:26:59.777+0000 7f276ed08640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:26:59.779 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:26:59.777+0000 7f276ed08640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:26:59.791 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:26:59.791 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:26:59.802 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:26:59.803 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:26:59.803 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:26:59.813 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:26:59.813 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:26:59.843 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:26:59.844 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:27:00.161 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 814 8 365281 365273 3.5055 0 - 0.0299513 2026-02-23T13:27:00.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:27:00 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/365912426' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:27:01.161 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 815 8 365281 365273 3.50119 0 - 0.0299513 2026-02-23T13:27:01.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:27:01 trial068 ceph-mon[38805]: pgmap v775: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:27:01.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:27:01 trial068 ceph-mon[38805]: 5.8 scrub starts 2026-02-23T13:27:01.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:27:01 trial068 ceph-mon[38805]: 5.8 scrub ok 2026-02-23T13:27:01.845 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:27:01.963 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:27:02.063 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:27:02.061+0000 7f6727ffb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:27:02.063 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:27:02.061+0000 7f6727ffb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:27:02.075 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:27:02.075 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:27:02.085 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:27:02.085 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:27:02.086 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:27:02.096 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:27:02.096 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:27:02.124 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:27:02.125 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:27:02.161 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 816 8 365281 365273 3.4969 0 - 0.0299513 2026-02-23T13:27:02.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:27:02 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3917814787' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:27:03.161 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 817 8 365281 365273 3.49262 0 - 0.0299513 2026-02-23T13:27:03.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:27:03 trial068 ceph-mon[38805]: pgmap v776: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:27:04.126 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:27:04.161 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 818 8 365281 365273 3.48835 0 - 0.0299513 2026-02-23T13:27:04.243 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:27:04.265 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:27:03 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:27:03] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:27:04.266 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:27:03 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:27:03.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:27:04.342 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:27:04.341+0000 7fb444375640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:27:04.342 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:27:04.341+0000 7fb444375640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:27:04.354 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:27:04.354 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:27:04.364 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:27:04.364 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:27:04.365 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:27:04.374 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:27:04.374 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:27:04.405 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:27:04.406 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:27:04.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:27:04 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1728273262' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:27:05.161 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 819 8 365281 365273 3.48409 0 - 0.0299513 2026-02-23T13:27:05.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:27:05 trial068 ceph-mon[38805]: pgmap v777: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:27:06.161 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T13:27:06.161777+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T13:27:06.162 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T13:27:06.162 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 820 8 365281 365273 3.47985 0 - 0.0299513 2026-02-23T13:27:06.407 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:27:06.522 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:27:06.619 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:27:06.617+0000 7f21cd4d3640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:27:06.619 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:27:06.617+0000 7f21cd4d3640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:27:06.630 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:27:06.631 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:27:06.641 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:27:06.641 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:27:06.641 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:27:06.651 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:27:06.651 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:27:06.682 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:27:06.682 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:27:07.162 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 821 8 365281 365273 3.47561 0 - 0.0299513 2026-02-23T13:27:07.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:27:07 trial068 ceph-mon[38805]: pgmap v778: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:27:07.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:27:07 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1324281058' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:27:08.162 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 822 8 365281 365273 3.47138 0 - 0.0299513 2026-02-23T13:27:08.684 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:27:08.800 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:27:08.902 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:27:08.901+0000 7f643ba88640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:27:08.902 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:27:08.901+0000 7f643ba88640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:27:08.914 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:27:08.914 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:27:08.924 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:27:08.924 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:27:08.924 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:27:08.934 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:27:08.934 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:27:08.964 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:27:08.965 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:27:09.162 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 823 8 365281 365273 3.46716 0 - 0.0299513 2026-02-23T13:27:09.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:27:09 trial068 ceph-mon[38805]: pgmap v779: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:27:09.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:27:09 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1719175487' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:27:10.162 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 824 8 365281 365273 3.46295 0 - 0.0299513 2026-02-23T13:27:10.965 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:27:11.081 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:27:11.162 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 825 8 365281 365273 3.45876 0 - 0.0299513 2026-02-23T13:27:11.182 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:27:11.180+0000 7f6bfc804640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:27:11.182 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:27:11.181+0000 7f6bfc804640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:27:11.194 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:27:11.194 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:27:11.204 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:27:11.204 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:27:11.205 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:27:11.215 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:27:11.215 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:27:11.243 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:27:11.244 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:27:11.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:27:11 trial068 ceph-mon[38805]: pgmap v780: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:27:11.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:27:11 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2726708631' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:27:12.162 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 826 8 365281 365273 3.45457 0 - 0.0299513 2026-02-23T13:27:13.162 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 827 8 365281 365273 3.45039 0 - 0.0299513 2026-02-23T13:27:13.244 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:27:13.362 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:27:13.461 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:27:13.460+0000 7f9f3ea99640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:27:13.461 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:27:13.460+0000 7f9f3ea99640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:27:13.473 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:27:13.473 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:27:13.484 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:27:13.484 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:27:13.484 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:27:13.494 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:27:13.494 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:27:13.525 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:27:13 trial068 ceph-mon[38805]: pgmap v781: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:27:13.526 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:27:13.527 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:27:14.162 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 828 8 365281 365273 3.44622 0 - 0.0299513 2026-02-23T13:27:14.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:27:13 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:27:13] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:27:14.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:27:13 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:27:13.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:27:14.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:27:14 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2005269668' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:27:15.162 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 829 8 365281 365273 3.44207 0 - 0.0299513 2026-02-23T13:27:15.528 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:27:15.647 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:27:15.691 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:27:15 trial068 ceph-mon[38805]: pgmap v782: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:27:15.746 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:27:15.745+0000 7f54f09e7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:27:15.746 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:27:15.745+0000 7f54f09e7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:27:15.758 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:27:15.758 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:27:15.768 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:27:15.768 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:27:15.768 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:27:15.778 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:27:15.778 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:27:15.809 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:27:15.810 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:27:16.162 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 830 8 365281 365273 3.43792 0 - 0.0299513 2026-02-23T13:27:16.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:27:16 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1881821190' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:27:17.162 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 831 8 365281 365273 3.43378 0 - 0.0299513 2026-02-23T13:27:17.811 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:27:17.835 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:27:17 trial068 ceph-mon[38805]: pgmap v783: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:27:17.928 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:27:18.029 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:27:18.028+0000 7fc96a2d1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:27:18.029 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:27:18.028+0000 7fc96a2d1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:27:18.041 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:27:18.041 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:27:18.052 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:27:18.052 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:27:18.052 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:27:18.062 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:27:18.062 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:27:18.093 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:27:18.094 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:27:18.162 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 832 8 365281 365273 3.42966 0 - 0.0299513 2026-02-23T13:27:18.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:27:18 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3283032621' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:27:19.162 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 833 8 365281 365273 3.42554 0 - 0.0299513 2026-02-23T13:27:19.744 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:27:19 trial068 ceph-mon[38805]: pgmap v784: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:27:20.095 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:27:20.162 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 834 8 365281 365273 3.42143 0 - 0.0299513 2026-02-23T13:27:20.212 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:27:20.305 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:27:20.303+0000 7f5ea3cd0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:27:20.305 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:27:20.304+0000 7f5ea3cd0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:27:20.317 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:27:20.317 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:27:20.327 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:27:20.327 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:27:20.328 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:27:20.337 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:27:20.337 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:27:20.367 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:27:20.368 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:27:20.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:27:20 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1954631085' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:27:21.162 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 835 8 365281 365273 3.41733 0 - 0.0299513 2026-02-23T13:27:21.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:27:21 trial068 ceph-mon[38805]: pgmap v785: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:27:22.163 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 836 8 365281 365273 3.41325 0 - 0.0299513 2026-02-23T13:27:22.369 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:27:22.488 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:27:22.587 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:27:22.586+0000 7f81c2119640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:27:22.588 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:27:22.586+0000 7f81c2119640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:27:22.600 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:27:22.600 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:27:22.610 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:27:22.611 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:27:22.611 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:27:22.620 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:27:22.620 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:27:22.651 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:27:22.652 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:27:23.163 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 837 8 365281 365273 3.40917 0 - 0.0299513 2026-02-23T13:27:23.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:27:23 trial068 ceph-mon[38805]: pgmap v786: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:27:23.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:27:23 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1620876099' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:27:24.163 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 838 8 365281 365273 3.4051 0 - 0.0299513 2026-02-23T13:27:24.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:27:23 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:27:23] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:27:24.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:27:23 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:27:23.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:27:24.653 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:27:24.771 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:27:24.866 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:27:24.864+0000 7f64988b0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:27:24.866 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:27:24.865+0000 7f64988b0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:27:24.878 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:27:24.878 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:27:24.889 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:27:24.889 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:27:24.889 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:27:24.898 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:27:24.899 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:27:24.929 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:27:24.929 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:27:25.163 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 839 8 365281 365273 3.40104 0 - 0.0299513 2026-02-23T13:27:25.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:27:25 trial068 ceph-mon[38805]: pgmap v787: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:27:25.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:27:25 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3943222785' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:27:26.163 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T13:27:26.163267+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T13:27:26.163 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T13:27:26.163 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 840 8 365281 365273 3.39699 0 - 0.0299513 2026-02-23T13:27:26.930 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:27:27.048 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:27:27.145 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:27:27.143+0000 7fc36892c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:27:27.145 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:27:27.143+0000 7fc36892c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:27:27.157 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:27:27.157 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:27:27.163 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 841 8 365281 365273 3.39295 0 - 0.0299513 2026-02-23T13:27:27.166 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:27:27.167 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:27:27.167 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:27:27.176 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:27:27.177 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:27:27.208 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:27:27.209 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:27:27.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:27:27 trial068 ceph-mon[38805]: pgmap v788: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:27:27.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:27:27 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/181644827' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:27:28.163 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 842 8 365281 365273 3.38892 0 - 0.0299513 2026-02-23T13:27:29.163 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 843 8 365281 365273 3.3849 0 - 0.0299513 2026-02-23T13:27:29.210 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:27:29.326 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:27:29.426 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:27:29.424+0000 7ff4cc5da640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:27:29.426 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:27:29.425+0000 7ff4cc5da640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:27:29.438 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:27:29.438 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:27:29.450 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:27:29.450 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:27:29.450 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:27:29.461 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:27:29.461 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:27:29.489 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:27:29 trial068 ceph-mon[38805]: pgmap v789: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:27:29.489 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:27:29 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1107047085' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:27:29.489 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:27:29.490 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:27:30.163 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 844 8 365281 365273 3.38089 0 - 0.0299513 2026-02-23T13:27:30.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:27:30 trial068 ceph-mon[38805]: 5.e scrub starts 2026-02-23T13:27:30.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:27:30 trial068 ceph-mon[38805]: 5.e scrub ok 2026-02-23T13:27:31.163 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 845 8 365281 365273 3.37689 0 - 0.0299513 2026-02-23T13:27:31.491 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:27:31.610 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:27:31.707 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:27:31.706+0000 7f1026590640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:27:31.707 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:27:31.706+0000 7f1026590640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:27:31.708 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:27:31 trial068 ceph-mon[38805]: pgmap v790: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:27:31.719 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:27:31.719 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:27:31.729 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:27:31.730 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:27:31.730 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:27:31.740 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:27:31.740 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:27:31.769 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:27:31.770 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:27:32.164 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 846 8 365281 365273 3.3729 0 - 0.0299513 2026-02-23T13:27:32.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:27:32 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/989011215' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:27:33.164 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 847 8 365281 365273 3.36892 0 - 0.0299513 2026-02-23T13:27:33.771 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:27:33.797 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:27:33 trial068 ceph-mon[38805]: pgmap v791: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:27:33.890 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:27:33.987 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:27:33.985+0000 7ff67f934640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:27:33.987 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:27:33.985+0000 7ff67f934640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:27:33.998 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:27:33.999 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:27:34.008 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:27:34.009 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:27:34.009 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:27:34.020 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:27:34.020 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:27:34.050 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:27:33 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:27:33] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:27:34.051 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:27:34.052 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:27:34.164 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 848 8 365281 365273 3.36494 0 - 0.0299513 2026-02-23T13:27:34.265 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:27:33 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:27:33.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:27:34.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:27:34 trial068 ceph-mon[38805]: 5.7 scrub starts 2026-02-23T13:27:34.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:27:34 trial068 ceph-mon[38805]: 5.7 scrub ok 2026-02-23T13:27:34.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:27:34 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2868317045' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:27:35.164 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 849 8 365281 365273 3.36098 0 - 0.0299513 2026-02-23T13:27:35.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:27:35 trial068 ceph-mon[38805]: pgmap v792: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:27:36.052 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:27:36.164 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 850 8 365281 365273 3.35703 0 - 0.0299513 2026-02-23T13:27:36.169 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:27:36.276 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:27:36.275+0000 7f188b06b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:27:36.277 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:27:36.275+0000 7f188b06b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:27:36.289 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:27:36.290 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:27:36.300 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:27:36.300 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:27:36.300 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:27:36.310 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:27:36.311 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:27:36.341 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:27:36.342 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:27:36.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:27:36 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2026-02-23T13:27:36.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:27:36 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3438634183' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:27:37.164 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 851 8 365281 365273 3.35308 0 - 0.0299513 2026-02-23T13:27:37.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:27:37 trial068 ceph-mon[38805]: pgmap v793: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:27:37.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:27:37 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T13:27:37.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:27:37 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T13:27:37.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:27:37 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T13:27:37.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:27:37 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T13:27:37.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:27:37 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-23T13:27:37.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:27:37 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2026-02-23T13:27:37.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:27:37 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T13:27:37.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:27:37 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2026-02-23T13:27:38.164 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 852 8 365281 365273 3.34915 0 - 0.0299513 2026-02-23T13:27:38.343 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:27:38.460 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:27:38.573 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:27:38.571+0000 7f030d611640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:27:38.573 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:27:38.571+0000 7f030d611640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:27:38.585 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:27:38.585 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:27:38.595 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:27:38.596 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:27:38.596 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:27:38.607 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:27:38.608 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:27:38.620 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:27:38 trial068 ceph-mon[38805]: pgmap v794: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:27:38.620 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:27:38 trial068 ceph-mon[38805]: pgmap v795: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:27:38.638 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:27:38.639 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:27:39.164 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 853 8 365281 365273 3.34522 0 - 0.0299513 2026-02-23T13:27:39.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:27:39 trial068 ceph-mon[38805]: 5.5 scrub starts 2026-02-23T13:27:39.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:27:39 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/293624322' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:27:39.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:27:39 trial068 ceph-mon[38805]: 5.5 scrub ok 2026-02-23T13:27:40.164 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 854 8 365281 365273 3.3413 0 - 0.0299513 2026-02-23T13:27:40.640 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:27:40.756 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:27:40.778 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:27:40 trial068 ceph-mon[38805]: pgmap v796: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:27:40.853 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:27:40.852+0000 7f5b39e02640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:27:40.853 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:27:40.852+0000 7f5b39e02640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:27:40.865 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:27:40.866 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:27:40.876 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:27:40.876 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:27:40.877 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:27:40.886 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:27:40.886 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:27:40.916 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:27:40.916 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:27:41.164 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 855 8 365281 365273 3.3374 0 - 0.0299513 2026-02-23T13:27:41.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:27:41 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/935461230' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:27:42.164 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 856 8 365281 365273 3.3335 0 - 0.0299513 2026-02-23T13:27:42.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:27:42 trial068 ceph-mon[38805]: pgmap v797: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:27:42.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:27:42 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T13:27:42.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:27:42 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T13:27:42.917 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:27:43.036 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:27:43.140 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:27:43.138+0000 7fc1c01a4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:27:43.140 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:27:43.139+0000 7fc1c01a4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:27:43.152 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:27:43.152 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:27:43.162 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:27:43.163 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:27:43.163 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:27:43.164 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 857 8 365281 365273 3.32961 0 - 0.0299513 2026-02-23T13:27:43.173 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:27:43.174 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:27:43.202 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:27:43.203 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:27:43.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:27:43 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2707061589' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:27:44.164 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 858 8 365281 365273 3.32573 0 - 0.0299513 2026-02-23T13:27:44.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:27:43 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:27:43] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:27:44.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:27:43 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:27:43.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:27:44.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:27:44 trial068 ceph-mon[38805]: pgmap v798: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:27:44.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:27:44 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T13:27:44.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:27:44 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T13:27:45.164 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 859 8 365281 365273 3.32185 0 - 0.0299513 2026-02-23T13:27:45.204 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:27:45.321 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:27:45.419 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:27:45.417+0000 7f1866041640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:27:45.419 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:27:45.417+0000 7f1866041640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:27:45.430 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:27:45.431 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:27:45.441 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:27:45.441 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:27:45.441 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:27:45.451 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:27:45.451 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:27:45.479 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:27:45 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1454098598' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:27:45.480 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:27:45.481 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:27:46.165 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T13:27:46.164814+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T13:27:46.165 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T13:27:46.165 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 860 8 365281 365273 3.31799 0 - 0.0299513 2026-02-23T13:27:46.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:27:46 trial068 ceph-mon[38805]: pgmap v799: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:27:47.165 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 861 8 365281 365273 3.31414 0 - 0.0299513 2026-02-23T13:27:47.482 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:27:47.600 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:27:47.704 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:27:47.703+0000 7f1739bdd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:27:47.705 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:27:47.703+0000 7f1739bdd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:27:47.717 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:27:47.717 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:27:47.718 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:27:47 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T13:27:47.718 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:27:47 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T13:27:47.728 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:27:47.728 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:27:47.728 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:27:47.738 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:27:47.738 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:27:47.783 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:27:47.783 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:27:48.165 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 862 8 365281 365273 3.31029 0 - 0.0299513 2026-02-23T13:27:48.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:27:48 trial068 ceph-mon[38805]: pgmap v800: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:27:48.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:27:48 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1436669576' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:27:49.165 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 863 8 365281 365273 3.30646 0 - 0.0299513 2026-02-23T13:27:49.784 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:27:49.902 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:27:50.003 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:27:50.001+0000 7f103c0e2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:27:50.003 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:27:50.001+0000 7f103c0e2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:27:50.015 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:27:50.015 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:27:50.025 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:27:50.025 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:27:50.025 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:27:50.035 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:27:50.035 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:27:50.066 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:27:50.066 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:27:50.165 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 864 8 365281 365273 3.30263 0 - 0.0299513 2026-02-23T13:27:50.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:27:50 trial068 ceph-mon[38805]: pgmap v801: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:27:50.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:27:50 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1031540058' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:27:51.165 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 865 8 365281 365273 3.29881 0 - 0.0299513 2026-02-23T13:27:51.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:27:51 trial068 ceph-mon[38805]: 5.a scrub starts 2026-02-23T13:27:51.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:27:51 trial068 ceph-mon[38805]: 5.a scrub ok 2026-02-23T13:27:52.068 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:27:52.165 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 866 8 365281 365273 3.295 0 - 0.0299513 2026-02-23T13:27:52.187 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:27:52.282 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:27:52.281+0000 7f97d6d1c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:27:52.283 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:27:52.281+0000 7f97d6d1c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:27:52.294 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:27:52.294 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:27:52.304 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:27:52.305 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:27:52.305 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:27:52.315 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:27:52.315 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:27:52.345 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:27:52.345 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:27:52.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:27:52 trial068 ceph-mon[38805]: pgmap v802: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:27:52.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:27:52 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1583353569' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:27:53.165 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 867 8 365281 365273 3.2912 0 - 0.0299513 2026-02-23T13:27:54.165 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 868 8 365281 365273 3.28741 0 - 0.0299513 2026-02-23T13:27:54.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:27:53 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:27:53] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:27:54.347 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:27:54.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:27:53 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:27:53.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:27:54.462 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:27:54.563 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:27:54.561+0000 7f8a521aa640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:27:54.563 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:27:54.561+0000 7f8a521aa640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:27:54.575 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:27:54.575 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:27:54.585 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:27:54.586 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:27:54.586 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:27:54.596 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:27:54.597 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:27:54.627 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:27:54 trial068 ceph-mon[38805]: pgmap v803: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:27:54.628 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:27:54.629 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:27:55.165 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 869 8 365281 365273 3.28363 0 - 0.0299513 2026-02-23T13:27:55.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:27:55 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1595586001' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:27:56.165 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 870 8 365281 365273 3.27985 0 - 0.0299513 2026-02-23T13:27:56.630 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:27:56.748 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:27:56.770 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:27:56 trial068 ceph-mon[38805]: pgmap v804: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:27:56.849 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:27:56.847+0000 7f9f2a466640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:27:56.849 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:27:56.847+0000 7f9f2a466640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:27:56.861 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:27:56.861 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:27:56.871 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:27:56.872 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:27:56.872 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:27:56.882 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:27:56.883 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:27:56.913 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:27:56.913 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:27:57.165 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 871 8 365281 365273 3.27609 0 - 0.0299513 2026-02-23T13:27:57.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:27:57 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3639978836' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:27:58.165 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 872 8 365281 365273 3.27233 0 - 0.0299513 2026-02-23T13:27:58.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:27:58 trial068 ceph-mon[38805]: pgmap v805: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:27:58.915 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:27:59.035 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:27:59.137 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:27:59.135+0000 7f412b8e9640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:27:59.137 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:27:59.135+0000 7f412b8e9640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:27:59.149 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:27:59.150 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:27:59.159 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:27:59.159 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:27:59.160 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:27:59.165 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 873 8 365281 365273 3.26858 0 - 0.0299513 2026-02-23T13:27:59.169 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:27:59.169 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:27:59.200 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:27:59.201 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:27:59.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:27:59 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2880549309' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:28:00.165 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 874 8 365281 365273 3.26484 0 - 0.0299513 2026-02-23T13:28:00.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:28:00 trial068 ceph-mon[38805]: pgmap v806: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:28:01.166 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 875 8 365281 365273 3.26111 0 - 0.0299513 2026-02-23T13:28:01.201 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:28:01.320 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:28:01.425 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:28:01.424+0000 7f1cc0ec5640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:28:01.425 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:28:01.424+0000 7f1cc0ec5640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:28:01.437 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:28:01.438 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:28:01.448 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:28:01.448 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:28:01.449 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:28:01.458 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:28:01.458 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:28:01.486 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:28:01.487 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:28:01.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:28:01 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1145508089' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:28:02.166 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 876 8 365281 365273 3.25739 0 - 0.0299513 2026-02-23T13:28:02.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:28:02 trial068 ceph-mon[38805]: pgmap v807: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:28:03.166 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 877 8 365281 365273 3.25368 0 - 0.0299513 2026-02-23T13:28:03.488 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:28:03.607 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:28:03.707 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:28:03.705+0000 7fb8e5ea5640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:28:03.707 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:28:03.706+0000 7fb8e5ea5640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:28:03.719 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:28:03.719 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:28:03.729 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:28:03.730 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:28:03.730 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:28:03.739 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:28:03.739 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:28:03.770 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:28:03.771 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:28:04.093 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:28:03 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:28:03] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:28:04.166 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 878 8 365281 365273 3.24997 0 - 0.0299513 2026-02-23T13:28:04.265 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:28:03 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:28:03.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:28:04.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:28:04 trial068 ceph-mon[38805]: pgmap v808: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:28:04.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:28:04 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3453430765' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:28:05.166 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 879 8 365281 365273 3.24627 0 - 0.0299513 2026-02-23T13:28:05.772 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:28:05.895 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:28:06.000 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:28:05.998+0000 7fe9e76a8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:28:06.000 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:28:05.999+0000 7fe9e76a8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:28:06.012 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:28:06.013 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:28:06.022 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:28:06.023 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:28:06.023 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:28:06.033 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:28:06.033 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:28:06.063 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:28:06.063 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:28:06.166 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T13:28:06.166252+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T13:28:06.166 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T13:28:06.166 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 880 8 365281 365273 3.24258 0 - 0.0299513 2026-02-23T13:28:06.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:28:06 trial068 ceph-mon[38805]: pgmap v809: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:28:06.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:28:06 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/4199658624' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:28:07.166 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 881 8 365281 365273 3.2389 0 - 0.0299513 2026-02-23T13:28:08.064 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:28:08.166 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 882 8 365281 365273 3.23523 0 - 0.0299513 2026-02-23T13:28:08.184 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:28:08.280 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:28:08.279+0000 7f339abc4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:28:08.280 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:28:08.279+0000 7f339abc4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:28:08.292 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:28:08.292 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:28:08.303 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:28:08.303 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:28:08.304 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:28:08.313 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:28:08.313 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:28:08.342 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:28:08.343 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:28:08.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:28:08 trial068 ceph-mon[38805]: pgmap v810: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:28:08.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:28:08 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/4075323418' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:28:09.166 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 883 8 365281 365273 3.23157 0 - 0.0299513 2026-02-23T13:28:10.166 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 884 8 365281 365273 3.22791 0 - 0.0299513 2026-02-23T13:28:10.344 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:28:10.464 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:28:10.558 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:28:10.556+0000 7fbed2bc4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:28:10.558 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:28:10.557+0000 7fbed2bc4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:28:10.569 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:28:10.570 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:28:10.580 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:28:10.580 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:28:10.580 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:28:10.590 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:28:10.590 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:28:10.619 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:28:10 trial068 ceph-mon[38805]: pgmap v811: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:28:10.621 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:28:10.621 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:28:11.166 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 885 8 365281 365273 3.22426 0 - 0.0299513 2026-02-23T13:28:11.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:28:11 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3111850249' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:28:12.166 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 886 8 365281 365273 3.22063 0 - 0.0299513 2026-02-23T13:28:12.623 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:28:12.741 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:28:12.763 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:28:12 trial068 ceph-mon[38805]: pgmap v812: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:28:12.840 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:28:12.839+0000 7fcd3fddb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:28:12.840 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:28:12.839+0000 7fcd3fddb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:28:12.852 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:28:12.852 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:28:12.862 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:28:12.862 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:28:12.863 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:28:12.872 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:28:12.872 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:28:12.903 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:28:12.904 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:28:13.167 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 887 8 365281 365273 3.21699 0 - 0.0299513 2026-02-23T13:28:13.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:28:13 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/799351739' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:28:14.167 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 888 8 365281 365273 3.21337 0 - 0.0299513 2026-02-23T13:28:14.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:28:13 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:28:13] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:28:14.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:28:13 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:28:13.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:28:14.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:28:14 trial068 ceph-mon[38805]: pgmap v813: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:28:14.906 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:28:15.023 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:28:15.124 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:28:15.122+0000 7fd332b6a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:28:15.124 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:28:15.123+0000 7fd332b6a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:28:15.136 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:28:15.136 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:28:15.146 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:28:15.147 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:28:15.147 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:28:15.157 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:28:15.157 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:28:15.166 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 889 8 365281 365273 3.20976 0 - 0.0299513 2026-02-23T13:28:15.188 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:28:15.188 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:28:15.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:28:15 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3113063874' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:28:16.167 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 890 8 365281 365273 3.20615 0 - 0.0299513 2026-02-23T13:28:16.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:28:16 trial068 ceph-mon[38805]: pgmap v814: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:28:17.167 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 891 8 365281 365273 3.20255 0 - 0.0299513 2026-02-23T13:28:17.189 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:28:17.307 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:28:17.406 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:28:17.405+0000 7f83e6b2a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:28:17.406 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:28:17.405+0000 7f83e6b2a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:28:17.418 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:28:17.418 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:28:17.429 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:28:17.429 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:28:17.429 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:28:17.439 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:28:17.440 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:28:17.471 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:28:17.472 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:28:17.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:28:17 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1730265126' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:28:18.167 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 892 8 365281 365273 3.19896 0 - 0.0299513 2026-02-23T13:28:18.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:28:18 trial068 ceph-mon[38805]: pgmap v815: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:28:19.167 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 893 8 365281 365273 3.19538 0 - 0.0299513 2026-02-23T13:28:19.473 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:28:19.590 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:28:19.688 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:28:19.686+0000 7f577e27a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:28:19.688 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:28:19.687+0000 7f577e27a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:28:19.699 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:28:19.700 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:28:19.710 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:28:19.710 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:28:19.710 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:28:19.720 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:28:19.720 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:28:19.748 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:28:19.749 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:28:19.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:28:19 trial068 ceph-mon[38805]: 5.8 scrub starts 2026-02-23T13:28:19.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:28:19 trial068 ceph-mon[38805]: 5.8 scrub ok 2026-02-23T13:28:20.167 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 894 8 365281 365273 3.19181 0 - 0.0299513 2026-02-23T13:28:20.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:28:20 trial068 ceph-mon[38805]: pgmap v816: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:28:20.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:28:20 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2606052886' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:28:21.167 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 895 8 365281 365273 3.18824 0 - 0.0299513 2026-02-23T13:28:21.749 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:28:21.863 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:28:21.962 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:28:21.960+0000 7f886a601640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:28:21.962 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:28:21.961+0000 7f886a601640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:28:21.974 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:28:21.974 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:28:21.984 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:28:21.985 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:28:21.985 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:28:21.994 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:28:21.995 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:28:22.023 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:28:22.024 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:28:22.167 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 896 8 365281 365273 3.18468 0 - 0.0299513 2026-02-23T13:28:22.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:28:22 trial068 ceph-mon[38805]: pgmap v817: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:28:22.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:28:22 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1702153372' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:28:23.167 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 897 8 365281 365273 3.18113 0 - 0.0299513 2026-02-23T13:28:24.025 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:28:24.142 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:28:24.167 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 898 8 365281 365273 3.17759 0 - 0.0299513 2026-02-23T13:28:24.243 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:28:24.242+0000 7f4671df0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:28:24.243 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:28:23 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:28:23] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:28:24.243 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:28:24.242+0000 7f4671df0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:28:24.254 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:28:24.255 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:28:24.265 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:28:24.265 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:28:24.265 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:28:24.275 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:28:24.275 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:28:24.305 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:28:24.305 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:28:24.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:28:23 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:28:23.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:28:24.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:28:24 trial068 ceph-mon[38805]: pgmap v818: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:28:24.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:28:24 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/855487346' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:28:25.167 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 899 8 365281 365273 3.17405 0 - 0.0299513 2026-02-23T13:28:26.167 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T13:28:26.167664+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T13:28:26.168 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T13:28:26.168 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 900 8 365281 365273 3.17053 0 - 0.0299513 2026-02-23T13:28:26.306 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:28:26.424 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:28:26.522 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:28:26.521+0000 7fe27648d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:28:26.522 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:28:26.521+0000 7fe27648d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:28:26.534 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:28:26.534 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:28:26.544 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:28:26.545 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:28:26.545 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:28:26.554 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:28:26.554 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:28:26.583 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:28:26 trial068 ceph-mon[38805]: pgmap v819: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:28:26.584 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:28:26.584 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:28:27.168 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 901 8 365281 365273 3.16701 0 - 0.0299513 2026-02-23T13:28:27.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:28:27 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/73374373' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:28:28.168 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 902 8 365281 365273 3.1635 0 - 0.0299513 2026-02-23T13:28:28.585 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:28:28.705 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:28:28.762 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:28:28 trial068 ceph-mon[38805]: pgmap v820: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:28:28.805 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:28:28.804+0000 7f040c40e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:28:28.805 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:28:28.804+0000 7f040c40e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:28:28.818 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:28:28.818 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:28:28.828 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:28:28.828 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:28:28.829 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:28:28.839 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:28:28.839 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:28:28.869 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:28:28.870 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:28:29.167 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 903 8 365281 365273 3.15999 0 - 0.0299513 2026-02-23T13:28:29.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:28:29 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/495898280' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:28:30.167 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 904 8 365281 365273 3.1565 0 - 0.0299513 2026-02-23T13:28:30.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:28:30 trial068 ceph-mon[38805]: pgmap v821: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:28:30.870 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:28:30.989 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:28:31.087 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:28:31.085+0000 7faadf15d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:28:31.087 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:28:31.086+0000 7faadf15d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:28:31.099 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:28:31.099 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:28:31.110 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:28:31.110 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:28:31.110 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:28:31.120 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:28:31.120 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:28:31.148 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:28:31.149 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:28:31.168 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 905 8 365281 365273 3.15301 0 - 0.0299513 2026-02-23T13:28:31.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:28:31 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2757819684' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:28:32.168 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 906 8 365281 365273 3.14953 0 - 0.0299513 2026-02-23T13:28:32.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:28:32 trial068 ceph-mon[38805]: pgmap v822: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:28:33.150 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:28:33.168 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 907 8 365281 365273 3.14606 0 - 0.0299513 2026-02-23T13:28:33.267 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:28:33.366 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:28:33.364+0000 7f3e9c287640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:28:33.366 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:28:33.364+0000 7f3e9c287640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:28:33.378 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:28:33.378 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:28:33.388 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:28:33.388 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:28:33.389 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:28:33.399 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:28:33.399 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:28:33.429 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:28:33.430 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:28:33.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:28:33 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3207931428' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:28:34.168 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 908 8 365281 365273 3.14259 0 - 0.0299513 2026-02-23T13:28:34.265 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:28:33 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:28:33.974Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:28:34.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:28:33 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:28:33] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:28:34.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:28:34 trial068 ceph-mon[38805]: pgmap v823: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:28:35.168 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 909 8 365281 365273 3.13914 0 - 0.0299513 2026-02-23T13:28:35.431 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:28:35.551 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:28:35.652 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:28:35.651+0000 7f7cd7ffc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:28:35.653 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:28:35.651+0000 7f7cd7ffc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:28:35.664 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:28:35.665 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:28:35.675 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:28:35.675 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:28:35.676 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:28:35.686 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:28:35.686 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:28:35.717 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:28:35.718 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:28:36.168 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 910 8 365281 365273 3.13569 0 - 0.0299513 2026-02-23T13:28:36.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:28:36 trial068 ceph-mon[38805]: pgmap v824: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:28:36.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:28:36 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2486899546' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:28:37.168 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 911 8 365281 365273 3.13224 0 - 0.0299513 2026-02-23T13:28:37.719 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:28:37.837 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:28:37.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:28:37 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2026-02-23T13:28:37.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:28:37 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-23T13:28:37.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:28:37 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2026-02-23T13:28:37.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:28:37 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T13:28:37.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:28:37 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2026-02-23T13:28:37.933 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:28:37.931+0000 7f0bd9a3b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:28:37.933 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:28:37.932+0000 7f0bd9a3b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:28:37.945 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:28:37.945 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:28:37.955 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:28:37.956 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:28:37.956 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:28:37.967 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:28:37.967 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:28:37.995 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:28:37.996 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:28:38.168 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 912 8 365281 365273 3.12881 0 - 0.0299513 2026-02-23T13:28:38.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:28:38 trial068 ceph-mon[38805]: pgmap v825: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:28:38.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:28:38 trial068 ceph-mon[38805]: pgmap v826: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:28:38.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:28:38 trial068 ceph-mon[38805]: pgmap v827: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:28:38.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:28:38 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3654264929' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:28:39.168 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 913 8 365281 365273 3.12538 0 - 0.0299513 2026-02-23T13:28:39.997 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:28:40.115 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:28:40.168 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 914 8 365281 365273 3.12196 0 - 0.0299513 2026-02-23T13:28:40.215 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:28:40.213+0000 7faf99a39640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:28:40.215 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:28:40.213+0000 7faf99a39640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:28:40.226 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:28:40.227 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:28:40.237 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:28:40.237 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:28:40.237 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:28:40.247 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:28:40.247 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:28:40.295 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:28:40.295 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:28:40.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:28:40 trial068 ceph-mon[38805]: pgmap v828: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:28:40.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:28:40 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2176308117' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:28:41.168 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 915 8 365281 365273 3.11855 0 - 0.0299513 2026-02-23T13:28:42.168 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 916 8 365281 365273 3.11515 0 - 0.0299513 2026-02-23T13:28:42.295 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:28:42.413 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:28:42.511 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:28:42.510+0000 7f4e9cb53640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:28:42.512 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:28:42.510+0000 7f4e9cb53640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:28:42.523 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:28:42.523 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:28:42.533 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:28:42.534 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:28:42.534 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:28:42.544 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:28:42.544 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:28:42.576 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:28:42 trial068 ceph-mon[38805]: pgmap v829: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:28:42.576 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:28:42 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/104847741' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:28:42.577 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:28:42.577 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:28:43.169 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 917 8 365281 365273 3.11175 0 - 0.0299513 2026-02-23T13:28:43.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:28:43 trial068 ceph-mon[38805]: 5.7 scrub starts 2026-02-23T13:28:43.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:28:43 trial068 ceph-mon[38805]: 5.7 scrub ok 2026-02-23T13:28:44.169 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 918 8 365281 365273 3.10836 0 - 0.0299513 2026-02-23T13:28:44.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:28:43 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:28:43] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:28:44.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:28:43 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:28:43.974Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:28:44.579 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:28:44.695 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:28:44.802 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:28:44.800+0000 7ff9af3f6640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:28:44.802 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:28:44.800+0000 7ff9af3f6640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:28:44.803 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:28:44 trial068 ceph-mon[38805]: pgmap v830: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:28:44.803 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:28:44 trial068 ceph-mon[38805]: osdmap e196: 4 total, 3 up, 4 in 2026-02-23T13:28:44.814 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:28:44.814 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:28:44.824 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:28:44.824 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:28:44.824 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:28:44.835 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:28:44.835 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:28:44.880 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:28:44.880 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:28:45.169 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 919 8 365281 365273 3.10498 0 - 0.0299513 2026-02-23T13:28:45.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:28:45 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/394820952' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:28:46.169 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T13:28:46.169109+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T13:28:46.169 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T13:28:46.169 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 920 8 365281 365273 3.1016 0 - 0.0299513 2026-02-23T13:28:46.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:28:46 trial068 ceph-mon[38805]: pgmap v832: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:28:46.882 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:28:47.001 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:28:47.104 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:28:47.103+0000 7f040bea3640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:28:47.104 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:28:47.103+0000 7f040bea3640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:28:47.117 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:28:47.117 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:28:47.127 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:28:47.127 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:28:47.127 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:28:47.138 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:28:47.138 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:28:47.169 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:28:47.169 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:28:47.169 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 921 8 365281 365273 3.09823 0 - 0.0299513 2026-02-23T13:28:47.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:28:47 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2318360423' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:28:48.169 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 922 8 365281 365273 3.09487 0 - 0.0299513 2026-02-23T13:28:48.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:28:48 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T13:28:48.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:28:48 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T13:28:48.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:28:48 trial068 ceph-mon[38805]: pgmap v833: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:28:49.169 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 923 8 365281 365273 3.09152 0 - 0.0299513 2026-02-23T13:28:49.169 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:28:49.284 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:28:49.378 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:28:49.376+0000 7f5cebb24640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:28:49.378 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:28:49.376+0000 7f5cebb24640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:28:49.390 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:28:49.390 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:28:49.400 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:28:49.401 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:28:49.401 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:28:49.411 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:28:49.412 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:28:49.442 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:28:49.443 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:28:50.169 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 924 8 365281 365273 3.08818 0 - 0.0299513 2026-02-23T13:28:50.495 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:28:50 trial068 ceph-mon[38805]: pgmap v834: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:28:50.496 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:28:50 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/802019150' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:28:51.169 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 925 8 365281 365273 3.08484 0 - 0.0299513 2026-02-23T13:28:51.444 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:28:51.468 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:28:51 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T13:28:51.469 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:28:51 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T13:28:51.562 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:28:51.670 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:28:51.669+0000 7f843953b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:28:51.671 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:28:51.669+0000 7f843953b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:28:51.685 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:28:51.685 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:28:51.696 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:28:51.696 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:28:51.697 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:28:51.706 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:28:51.706 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:28:51.735 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:28:51.735 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:28:52.169 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 926 8 365281 365273 3.08151 0 - 0.0299513 2026-02-23T13:28:52.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:28:52 trial068 ceph-mon[38805]: pgmap v835: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:28:52.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:28:52 trial068 ceph-mon[38805]: 5.5 scrub starts 2026-02-23T13:28:52.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:28:52 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/760490108' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:28:53.169 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 927 8 365281 365273 3.07818 0 - 0.0299513 2026-02-23T13:28:53.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:28:53 trial068 ceph-mon[38805]: 5.5 scrub ok 2026-02-23T13:28:53.736 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:28:53.856 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:28:53.954 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:28:53.952+0000 7f01905e0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:28:53.954 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:28:53.953+0000 7f01905e0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:28:53.966 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:28:53.966 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:28:53.977 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:28:53.978 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:28:53.978 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:28:53.988 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:28:53.989 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:28:54.016 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:28:53 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:28:53] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:28:54.017 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:28:54.018 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:28:54.169 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 928 8 365281 365273 3.07486 0 - 0.0299513 2026-02-23T13:28:54.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:28:54 trial068 ceph-mon[38805]: pgmap v836: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:28:54.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:28:54 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/4126597587' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:28:54.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:28:53 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:28:53.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:28:55.169 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 929 8 365281 365273 3.07155 0 - 0.0299513 2026-02-23T13:28:56.019 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:28:56.140 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:28:56.169 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 930 8 365281 365273 3.06825 0 - 0.0299513 2026-02-23T13:28:56.234 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:28:56.232+0000 7feb27c68640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:28:56.234 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:28:56.232+0000 7feb27c68640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:28:56.245 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:28:56.246 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:28:56.255 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:28:56.256 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:28:56.256 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:28:56.266 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:28:56.267 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:28:56.297 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:28:56 trial068 ceph-mon[38805]: pgmap v837: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:28:56.297 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:28:56 trial068 ceph-mon[38805]: 5.e scrub starts 2026-02-23T13:28:56.297 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:28:56 trial068 ceph-mon[38805]: 5.e scrub ok 2026-02-23T13:28:56.297 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:28:56 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/306074618' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:28:56.298 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:28:56.298 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:28:57.170 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 931 8 365281 365273 3.06496 0 - 0.0299513 2026-02-23T13:28:58.170 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 932 8 365281 365273 3.06167 0 - 0.0299513 2026-02-23T13:28:58.299 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:28:58.416 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:28:58.514 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:28:58.513+0000 7f6d8e1a2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:28:58.514 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:28:58.513+0000 7f6d8e1a2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:28:58.515 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:28:58 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T13:28:58.515 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:28:58 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T13:28:58.515 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:28:58 trial068 ceph-mon[38805]: pgmap v838: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:28:58.526 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:28:58.527 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:28:58.536 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:28:58.537 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:28:58.537 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:28:58.547 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:28:58.547 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:28:58.577 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:28:58.577 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:28:59.170 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 933 8 365281 365273 3.05839 0 - 0.0299513 2026-02-23T13:28:59.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:28:59 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3704430302' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:29:00.170 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 934 8 365281 365273 3.05511 0 - 0.0299513 2026-02-23T13:29:00.578 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:29:00.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:29:00 trial068 ceph-mon[38805]: pgmap v839: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:29:00.694 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:29:00.800 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:29:00.798+0000 7f27ac42d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:29:00.800 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:29:00.798+0000 7f27ac42d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:29:00.812 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:29:00.812 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:29:00.822 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:29:00.822 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:29:00.822 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:29:00.831 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:29:00.832 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:29:00.861 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:29:00.862 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:29:01.170 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 935 8 365281 365273 3.05184 0 - 0.0299513 2026-02-23T13:29:01.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:29:01 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1748710071' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:29:02.170 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 936 8 365281 365273 3.04858 0 - 0.0299513 2026-02-23T13:29:02.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:29:02 trial068 ceph-mon[38805]: pgmap v840: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:29:02.863 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:29:02.979 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:29:03.078 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:29:03.076+0000 7f83b928b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:29:03.078 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:29:03.077+0000 7f83b928b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:29:03.090 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:29:03.090 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:29:03.101 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:29:03.101 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:29:03.101 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:29:03.111 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:29:03.112 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:29:03.142 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:29:03.143 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:29:03.170 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 937 8 365281 365273 3.04533 0 - 0.0299513 2026-02-23T13:29:03.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:29:03 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/439801604' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:29:04.170 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 938 8 365281 365273 3.04208 0 - 0.0299513 2026-02-23T13:29:04.261 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:29:03 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:29:03] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:29:04.265 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:29:03 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:29:03.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:29:04.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:29:04 trial068 ceph-mon[38805]: pgmap v841: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:29:05.144 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:29:05.170 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 939 8 365281 365273 3.03884 0 - 0.0299513 2026-02-23T13:29:05.262 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:29:05.364 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:29:05.363+0000 7f3c86890640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:29:05.364 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:29:05.363+0000 7f3c86890640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:29:05.376 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:29:05.376 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:29:05.386 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:29:05.387 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:29:05.387 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:29:05.396 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:29:05.397 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:29:05.427 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:29:05.427 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:29:06.170 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T13:29:06.170624+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T13:29:06.170 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T13:29:06.171 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 940 8 365281 365273 3.03561 0 - 0.0299513 2026-02-23T13:29:06.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:29:06 trial068 ceph-mon[38805]: pgmap v842: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:29:06.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:29:06 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/781113296' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:29:07.170 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 941 8 365281 365273 3.03239 0 - 0.0299513 2026-02-23T13:29:07.428 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:29:07.547 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:29:07.646 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:29:07.645+0000 7f6d75735640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:29:07.646 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:29:07.645+0000 7f6d75735640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:29:07.658 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:29:07.659 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:29:07.668 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:29:07.669 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:29:07.669 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:29:07.679 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:29:07.679 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:29:07.710 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:29:07.711 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:29:08.170 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 942 8 365281 365273 3.02917 0 - 0.0299513 2026-02-23T13:29:08.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:29:08 trial068 ceph-mon[38805]: pgmap v843: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:29:08.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:29:08 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1633942184' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:29:09.170 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 943 8 365281 365273 3.02595 0 - 0.0299513 2026-02-23T13:29:09.712 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:29:09.829 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:29:09.931 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:29:09.930+0000 7feec5994640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:29:09.931 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:29:09.930+0000 7feec5994640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:29:09.943 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:29:09.943 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:29:09.953 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:29:09.953 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:29:09.953 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:29:09.963 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:29:09.963 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:29:09.991 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:29:09.991 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:29:10.170 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 944 8 365281 365273 3.02275 0 - 0.0299513 2026-02-23T13:29:10.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:29:10 trial068 ceph-mon[38805]: pgmap v844: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:29:10.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:29:10 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3286236456' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:29:11.171 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 945 8 365281 365273 3.01955 0 - 0.0299513 2026-02-23T13:29:11.992 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:29:12.110 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:29:12.171 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 946 8 365281 365273 3.01636 0 - 0.0299513 2026-02-23T13:29:12.210 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:29:12.208+0000 7fa58afbf640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:29:12.210 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:29:12.209+0000 7fa58afbf640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:29:12.223 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:29:12.223 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:29:12.233 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:29:12.233 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:29:12.233 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:29:12.244 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:29:12.244 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:29:12.274 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:29:12 trial068 ceph-mon[38805]: pgmap v845: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:29:12.274 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:29:12 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2027763033' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:29:12.275 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:29:12.275 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:29:13.171 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 947 8 365281 365273 3.01317 0 - 0.0299513 2026-02-23T13:29:14.171 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 948 8 365281 365273 3.00999 0 - 0.0299513 2026-02-23T13:29:14.253 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:29:13 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:29:13] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:29:14.276 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:29:14.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:29:13 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:29:13.974Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:29:14.392 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:29:14.487 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:29:14.486+0000 7f7b990ee640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:29:14.487 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:29:14.486+0000 7f7b990ee640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:29:14.499 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:29:14.500 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:29:14.505 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:29:14 trial068 ceph-mon[38805]: pgmap v846: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:29:14.509 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:29:14.510 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:29:14.510 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:29:14.520 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:29:14.520 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:29:14.549 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:29:14.549 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:29:15.171 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 949 8 365281 365273 3.00682 0 - 0.0299513 2026-02-23T13:29:15.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:29:15 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/646697732' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:29:16.171 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 950 8 365281 365273 3.00366 0 - 0.0299513 2026-02-23T13:29:16.551 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:29:16.576 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:29:16 trial068 ceph-mon[38805]: pgmap v847: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:29:16.668 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:29:16.771 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:29:16.770+0000 7f0bc1bfa640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:29:16.772 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:29:16.770+0000 7f0bc1bfa640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:29:16.784 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:29:16.784 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:29:16.794 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:29:16.795 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:29:16.795 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:29:16.804 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:29:16.804 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:29:16.835 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:29:16.835 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:29:17.171 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 951 8 365281 365273 3.0005 0 - 0.0299513 2026-02-23T13:29:17.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:29:17 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2848441096' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:29:18.171 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 952 8 365281 365273 2.99735 0 - 0.0299513 2026-02-23T13:29:18.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:29:18 trial068 ceph-mon[38805]: pgmap v848: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:29:18.837 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:29:18.953 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:29:19.056 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:29:19.054+0000 7f0ebd8c4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:29:19.056 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:29:19.055+0000 7f0ebd8c4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:29:19.068 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:29:19.068 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:29:19.078 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:29:19.078 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:29:19.078 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:29:19.088 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:29:19.088 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:29:19.118 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:29:19.119 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:29:19.171 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 953 8 365281 365273 2.9942 0 - 0.0299513 2026-02-23T13:29:19.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:29:19 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1577513633' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:29:20.171 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 954 8 365281 365273 2.99106 0 - 0.0299513 2026-02-23T13:29:20.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:29:20 trial068 ceph-mon[38805]: pgmap v849: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:29:21.119 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:29:21.171 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 955 8 365281 365273 2.98793 0 - 0.0299513 2026-02-23T13:29:21.237 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:29:21.336 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:29:21.335+0000 7f6ebc584640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:29:21.337 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:29:21.335+0000 7f6ebc584640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:29:21.348 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:29:21.348 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:29:21.358 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:29:21.359 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:29:21.359 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:29:21.369 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:29:21.369 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:29:21.399 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:29:21.399 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:29:22.172 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 956 8 365281 365273 2.98481 0 - 0.0299513 2026-02-23T13:29:22.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:29:22 trial068 ceph-mon[38805]: pgmap v850: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:29:22.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:29:22 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2417567032' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:29:23.172 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 957 8 365281 365273 2.98169 0 - 0.0299513 2026-02-23T13:29:23.400 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:29:23.520 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:29:23.541 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:29:23 trial068 ceph-mon[38805]: 5.a scrub starts 2026-02-23T13:29:23.541 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:29:23 trial068 ceph-mon[38805]: 5.a scrub ok 2026-02-23T13:29:23.619 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:29:23.617+0000 7f2fa1794640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:29:23.619 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:29:23.617+0000 7f2fa1794640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:29:23.631 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:29:23.632 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:29:23.641 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:29:23.642 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:29:23.642 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:29:23.652 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:29:23.652 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:29:23.682 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:29:23.683 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:29:24.172 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 958 8 365281 365273 2.97857 0 - 0.0299513 2026-02-23T13:29:24.266 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:29:23 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:29:23] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:29:24.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:29:23 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:29:23.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:29:24.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:29:24 trial068 ceph-mon[38805]: pgmap v851: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:29:24.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:29:24 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1757894040' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:29:25.172 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 959 8 365281 365273 2.97547 0 - 0.0299513 2026-02-23T13:29:25.684 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:29:25.803 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:29:25.906 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:29:25.904+0000 7f3b73581640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:29:25.906 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:29:25.905+0000 7f3b73581640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:29:25.918 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:29:25.918 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:29:25.928 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:29:25.928 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:29:25.928 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:29:25.938 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:29:25.938 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:29:25.968 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:29:25.969 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:29:26.172 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T13:29:26.172392+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T13:29:26.172 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T13:29:26.173 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 960 8 365281 365273 2.97237 0 - 0.0299513 2026-02-23T13:29:26.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:29:26 trial068 ceph-mon[38805]: pgmap v852: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:29:26.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:29:26 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1223927131' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:29:27.172 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 961 8 365281 365273 2.96928 0 - 0.0299513 2026-02-23T13:29:27.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:29:27 trial068 ceph-mon[38805]: 5.8 scrub starts 2026-02-23T13:29:27.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:29:27 trial068 ceph-mon[38805]: 5.8 scrub ok 2026-02-23T13:29:27.970 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:29:28.087 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:29:28.172 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 962 8 365281 365273 2.96619 0 - 0.0299513 2026-02-23T13:29:28.181 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:29:28.180+0000 7f128e60c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:29:28.182 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:29:28.181+0000 7f128e60c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:29:28.193 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:29:28.194 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:29:28.203 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:29:28.204 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:29:28.204 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:29:28.213 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:29:28.213 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:29:28.241 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:29:28.242 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:29:28.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:29:28 trial068 ceph-mon[38805]: pgmap v853: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:29:28.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:29:28 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2973800050' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:29:29.172 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 963 8 365281 365273 2.96311 0 - 0.0299513 2026-02-23T13:29:30.173 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 964 8 365281 365273 2.96003 0 - 0.0299513 2026-02-23T13:29:30.243 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:29:30.363 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:29:30.385 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:29:30 trial068 ceph-mon[38805]: pgmap v854: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:29:30.460 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:29:30.458+0000 7f929c578640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:29:30.460 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:29:30.459+0000 7f929c578640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:29:30.472 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:29:30.472 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:29:30.482 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:29:30.482 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:29:30.483 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:29:30.492 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:29:30.493 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:29:30.523 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:29:30.523 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:29:31.173 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 965 8 365281 365273 2.95697 0 - 0.0299513 2026-02-23T13:29:31.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:29:31 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/4227038891' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:29:32.173 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 966 8 365281 365273 2.95391 0 - 0.0299513 2026-02-23T13:29:32.525 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:29:32.549 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:29:32 trial068 ceph-mon[38805]: pgmap v855: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:29:32.641 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:29:32.739 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:29:32.737+0000 7f5d8d0c6640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:29:32.739 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:29:32.737+0000 7f5d8d0c6640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:29:32.751 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:29:32.752 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:29:32.762 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:29:32.762 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:29:32.762 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:29:32.772 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:29:32.772 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:29:32.803 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:29:32.803 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:29:33.173 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 967 8 365281 365273 2.95085 0 - 0.0299513 2026-02-23T13:29:33.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:29:33 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/194241802' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:29:34.173 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 968 8 365281 365273 2.9478 0 - 0.0299513 2026-02-23T13:29:34.265 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:29:33 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:29:33.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:29:34.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:29:34 trial068 ceph-mon[38805]: pgmap v856: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:29:34.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:29:33 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:29:33] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:29:34.805 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:29:34.924 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:29:35.029 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:29:35.027+0000 7ff94d3a8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:29:35.029 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:29:35.028+0000 7ff94d3a8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:29:35.040 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:29:35.041 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:29:35.050 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:29:35.051 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:29:35.051 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:29:35.060 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:29:35.061 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:29:35.093 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:29:35.094 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:29:35.173 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 969 8 365281 365273 2.94476 0 - 0.0299513 2026-02-23T13:29:35.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:29:35 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3042303933' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:29:36.173 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 970 8 365281 365273 2.94172 0 - 0.0299513 2026-02-23T13:29:36.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:29:36 trial068 ceph-mon[38805]: pgmap v857: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:29:37.095 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:29:37.173 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 971 8 365281 365273 2.9387 0 - 0.0299513 2026-02-23T13:29:37.214 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:29:37.326 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:29:37.324+0000 7f8e0c5ce640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:29:37.326 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:29:37.325+0000 7f8e0c5ce640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:29:37.338 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:29:37.338 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:29:37.348 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:29:37.348 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:29:37.349 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:29:37.359 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:29:37.359 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:29:37.374 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:29:37 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2026-02-23T13:29:37.389 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:29:37.390 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:29:38.173 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 972 8 365281 365273 2.93567 0 - 0.0299513 2026-02-23T13:29:38.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:29:38 trial068 ceph-mon[38805]: pgmap v858: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:29:38.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:29:38 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1455596665' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:29:38.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:29:38 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-23T13:29:38.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:29:38 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2026-02-23T13:29:38.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:29:38 trial068 ceph-mon[38805]: pgmap v859: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:29:38.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:29:38 trial068 ceph-mon[38805]: pgmap v860: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:29:38.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:29:38 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T13:29:38.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:29:38 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2026-02-23T13:29:39.173 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 973 8 365281 365273 2.93265 0 - 0.0299513 2026-02-23T13:29:39.391 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:29:39.507 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:29:39.613 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:29:39.612+0000 7fda21993640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:29:39.613 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:29:39.612+0000 7fda21993640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:29:39.625 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:29:39.625 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:29:39.635 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:29:39.635 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:29:39.635 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:29:39.646 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:29:39.647 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:29:39.677 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:29:39.678 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:29:40.173 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 974 8 365281 365273 2.92964 0 - 0.0299513 2026-02-23T13:29:40.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:29:40 trial068 ceph-mon[38805]: pgmap v861: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:29:40.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:29:40 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2094713298' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:29:41.173 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 975 8 365281 365273 2.92664 0 - 0.0299513 2026-02-23T13:29:41.679 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:29:41.795 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:29:41.891 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:29:41.889+0000 7f2eb7b4e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:29:41.891 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:29:41.889+0000 7f2eb7b4e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:29:41.902 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:29:41.903 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:29:41.912 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:29:41.913 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:29:41.913 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:29:41.923 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:29:41.923 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:29:41.953 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:29:41.954 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:29:42.173 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 976 8 365281 365273 2.92364 0 - 0.0299513 2026-02-23T13:29:42.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:29:42 trial068 ceph-mon[38805]: pgmap v862: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:29:42.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:29:42 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1077972025' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:29:43.173 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 977 8 365281 365273 2.92065 0 - 0.0299513 2026-02-23T13:29:43.955 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:29:44.072 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:29:44.171 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:29:44.169+0000 7fa13d1e3640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:29:44.171 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:29:44.169+0000 7fa13d1e3640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:29:44.173 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 978 8 365281 365273 2.91766 0 - 0.0299513 2026-02-23T13:29:44.183 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:29:44.183 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:29:44.194 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:29:44.194 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:29:44.194 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:29:44.204 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:29:44.205 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:29:44.233 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:29:43 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:29:43] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:29:44.234 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:29:44.235 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:29:44.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:29:43 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:29:43.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:29:44.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:29:44 trial068 ceph-mon[38805]: pgmap v863: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:29:44.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:29:44 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/221362067' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:29:45.174 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 979 8 365281 365273 2.91468 0 - 0.0299513 2026-02-23T13:29:46.174 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T13:29:46.174169+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T13:29:46.174 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T13:29:46.174 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 980 8 365281 365273 2.91171 0 - 0.0299513 2026-02-23T13:29:46.236 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:29:46.353 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:29:46.446 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:29:46.444+0000 7f4bdab95640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:29:46.446 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:29:46.445+0000 7f4bdab95640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:29:46.458 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:29:46.459 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:29:46.468 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:29:46.469 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:29:46.469 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:29:46.478 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:29:46.479 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:29:46.509 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:29:46.509 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:29:46.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:29:46 trial068 ceph-mon[38805]: pgmap v864: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:29:46.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:29:46 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2788116687' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:29:47.174 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 981 8 365281 365273 2.90874 0 - 0.0299513 2026-02-23T13:29:48.174 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 982 8 365281 365273 2.90578 0 - 0.0299513 2026-02-23T13:29:48.510 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:29:48.630 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:29:48.737 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:29:48.736+0000 7f559576b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:29:48.737 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:29:48.737+0000 7f559576b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:29:48.750 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:29:48.750 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:29:48.760 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:29:48.760 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:29:48.760 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:29:48 trial068 ceph-mon[38805]: pgmap v865: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:29:48.760 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:29:48.770 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:29:48.770 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:29:48.815 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:29:48.815 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:29:49.174 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 983 8 365281 365273 2.90282 0 - 0.0299513 2026-02-23T13:29:49.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:29:49 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2555484414' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:29:50.174 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 984 8 365281 365273 2.89987 0 - 0.0299513 2026-02-23T13:29:50.816 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:29:50.840 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:29:50 trial068 ceph-mon[38805]: pgmap v866: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:29:50.933 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:29:51.034 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:29:51.032+0000 7ff9f55c7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:29:51.034 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:29:51.032+0000 7ff9f55c7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:29:51.045 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:29:51.046 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:29:51.055 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:29:51.056 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:29:51.056 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:29:51.065 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:29:51.065 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:29:51.093 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:29:51.094 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:29:51.174 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 985 8 365281 365273 2.89693 0 - 0.0299513 2026-02-23T13:29:51.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:29:51 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2149759229' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:29:52.174 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 986 8 365281 365273 2.89399 0 - 0.0299513 2026-02-23T13:29:52.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:29:52 trial068 ceph-mon[38805]: pgmap v867: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:29:53.095 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:29:53.174 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 987 8 365281 365273 2.89106 0 - 0.0299513 2026-02-23T13:29:53.214 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:29:53.312 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:29:53.310+0000 7f602fc35640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:29:53.312 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:29:53.310+0000 7f602fc35640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:29:53.324 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:29:53.324 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:29:53.334 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:29:53.334 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:29:53.334 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:29:53.344 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:29:53.344 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:29:53.375 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:29:53.375 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:29:53.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:29:53 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1536257819' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:29:54.174 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 988 8 365281 365273 2.88813 0 - 0.0299513 2026-02-23T13:29:54.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:29:53 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:29:53] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:29:54.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:29:53 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:29:53.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:29:54.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:29:54 trial068 ceph-mon[38805]: pgmap v868: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:29:55.174 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 989 8 365281 365273 2.88521 0 - 0.0299513 2026-02-23T13:29:55.376 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:29:55.492 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:29:55.595 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:29:55.593+0000 7efcfcf50640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:29:55.595 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:29:55.594+0000 7efcfcf50640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:29:55.607 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:29:55.607 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:29:55.617 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:29:55.618 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:29:55.618 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:29:55.628 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:29:55.628 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:29:55.658 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:29:55.658 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:29:56.174 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 990 8 365281 365273 2.8823 0 - 0.0299513 2026-02-23T13:29:56.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:29:56 trial068 ceph-mon[38805]: pgmap v869: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:29:56.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:29:56 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2007141938' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:29:57.174 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 991 8 365281 365273 2.87939 0 - 0.0299513 2026-02-23T13:29:57.659 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:29:57.779 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:29:57.874 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:29:57.872+0000 7f06430f4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:29:57.874 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:29:57.873+0000 7f06430f4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:29:57.886 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:29:57.887 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:29:57.896 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:29:57.897 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:29:57.897 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:29:57.906 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:29:57.907 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:29:57.935 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:29:57.936 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:29:58.175 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 992 8 365281 365273 2.87649 0 - 0.0299513 2026-02-23T13:29:58.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:29:58 trial068 ceph-mon[38805]: pgmap v870: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:29:58.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:29:58 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2849897992' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:29:59.175 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 993 8 365281 365273 2.87359 0 - 0.0299513 2026-02-23T13:29:59.936 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:30:00.054 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:30:00.153 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:30:00.151+0000 7f7f4087a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:30:00.153 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:30:00.152+0000 7f7f4087a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:30:00.165 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:30:00.166 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:30:00.175 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 994 8 365281 365273 2.8707 0 - 0.0299513 2026-02-23T13:30:00.176 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:30:00.176 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:30:00.176 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:30:00.186 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:30:00.186 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:30:00.218 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:30:00.218 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:30:00.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:30:00 trial068 ceph-mon[38805]: pgmap v871: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:30:00.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:30:00 trial068 ceph-mon[38805]: osdmap e197: 4 total, 3 up, 4 in 2026-02-23T13:30:00.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:30:00 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3416844335' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:30:01.175 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 995 8 365281 365273 2.86781 0 - 0.0299513 2026-02-23T13:30:02.175 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 996 8 365281 365273 2.86493 0 - 0.0299513 2026-02-23T13:30:02.219 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:30:02.337 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:30:02.436 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:30:02.435+0000 7f00676c9640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:30:02.436 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:30:02.435+0000 7f00676c9640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:30:02.449 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:30:02.449 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:30:02.459 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:30:02.459 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:30:02.459 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:30:02.469 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:30:02.469 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:30:02.499 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:30:02.500 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:30:02.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:30:02 trial068 ceph-mon[38805]: pgmap v873: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:30:02.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:30:02 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2510729111' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:30:03.175 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 997 8 365281 365273 2.86206 0 - 0.0299513 2026-02-23T13:30:04.175 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 998 8 365281 365273 2.85919 0 - 0.0299513 2026-02-23T13:30:04.265 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:30:03 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:30:03.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:30:04.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:30:03 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:30:03] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:30:04.501 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:30:04.620 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:30:04.729 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:30:04.728+0000 7f28d534f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:30:04.729 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:30:04.728+0000 7f28d534f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:30:04.741 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:30:04.742 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:30:04.752 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:30:04.752 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:30:04.752 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:30:04.762 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:30:04.762 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:30:04.777 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:30:04 trial068 ceph-mon[38805]: pgmap v874: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:30:04.792 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:30:04.792 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:30:05.175 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 999 8 365281 365273 2.85633 0 - 0.0299513 2026-02-23T13:30:05.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:30:05 trial068 ceph-mon[38805]: 5.5 scrub starts 2026-02-23T13:30:05.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:30:05 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2827689070' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:30:06.175 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T13:30:06.175687+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T13:30:06.176 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T13:30:06.176 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1000 8 365281 365273 2.85347 0 - 0.0299513 2026-02-23T13:30:06.794 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:30:06.819 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:30:06 trial068 ceph-mon[38805]: 5.5 scrub ok 2026-02-23T13:30:06.819 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:30:06 trial068 ceph-mon[38805]: pgmap v875: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:30:06.911 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:30:07.011 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:30:07.010+0000 7fe0e6676640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:30:07.011 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:30:07.010+0000 7fe0e6676640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:30:07.023 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:30:07.024 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:30:07.034 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:30:07.034 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:30:07.034 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:30:07.044 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:30:07.045 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:30:07.074 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:30:07.075 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:30:07.175 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1001 8 365281 365273 2.85062 0 - 0.0299513 2026-02-23T13:30:07.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:30:07 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1829101529' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:30:08.175 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1002 8 365281 365273 2.84778 0 - 0.0299513 2026-02-23T13:30:08.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:30:08 trial068 ceph-mon[38805]: pgmap v876: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:30:09.076 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:30:09.176 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1003 8 365281 365273 2.84494 0 - 0.0299513 2026-02-23T13:30:09.194 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:30:09.297 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:30:09.296+0000 7f75c7577640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:30:09.297 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:30:09.296+0000 7f75c7577640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:30:09.310 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:30:09.310 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:30:09.320 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:30:09.321 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:30:09.321 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:30:09.330 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:30:09.331 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:30:09.360 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:30:09.361 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:30:09.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:30:09 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T13:30:09.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:30:09 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T13:30:09.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:30:09 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/860520545' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:30:10.175 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1004 8 365281 365273 2.8421 0 - 0.0299513 2026-02-23T13:30:10.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:30:10 trial068 ceph-mon[38805]: pgmap v877: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:30:11.175 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1005 8 365281 365273 2.83928 0 - 0.0299513 2026-02-23T13:30:11.361 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:30:11.481 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:30:11.582 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:30:11.581+0000 7feb1c9cf640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:30:11.583 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:30:11.581+0000 7feb1c9cf640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:30:11.595 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:30:11.595 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:30:11.605 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:30:11.606 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:30:11.606 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:30:11.616 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:30:11.616 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:30:11.658 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:30:11.659 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:30:12.176 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1006 8 365281 365273 2.83645 0 - 0.0299513 2026-02-23T13:30:12.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:30:12 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T13:30:12.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:30:12 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T13:30:12.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:30:12 trial068 ceph-mon[38805]: pgmap v878: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:30:12.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:30:12 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1971769002' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:30:13.176 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1007 8 365281 365273 2.83364 0 - 0.0299513 2026-02-23T13:30:13.660 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:30:13.777 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:30:13.881 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:30:13.880+0000 7f3a94707640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:30:13.882 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:30:13.880+0000 7f3a94707640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:30:13.893 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:30:13.894 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:30:13.904 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:30:13.905 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:30:13.905 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:30:13.915 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:30:13.915 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:30:13.945 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:30:13.946 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:30:14.176 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1008 8 365281 365273 2.83083 0 - 0.0299513 2026-02-23T13:30:14.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:30:13 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:30:13] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:30:14.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:30:13 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:30:13.972Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:30:14.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:30:14 trial068 ceph-mon[38805]: pgmap v879: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:30:14.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:30:14 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1913438293' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:30:15.176 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1009 8 365281 365273 2.82802 0 - 0.0299513 2026-02-23T13:30:15.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:30:15 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T13:30:15.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:30:15 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T13:30:15.946 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:30:16.066 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:30:16.167 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:30:16.165+0000 7f188dbe3640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:30:16.167 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:30:16.165+0000 7f188dbe3640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:30:16.176 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1010 8 365281 365273 2.82522 0 - 0.0299513 2026-02-23T13:30:16.178 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:30:16.179 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:30:16.189 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:30:16.189 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:30:16.189 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:30:16.199 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:30:16.199 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:30:16.229 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:30:16.230 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:30:16.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:30:16 trial068 ceph-mon[38805]: pgmap v880: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:30:16.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:30:16 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2836580179' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:30:17.176 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1011 8 365281 365273 2.82243 0 - 0.0299513 2026-02-23T13:30:18.176 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1012 8 365281 365273 2.81964 0 - 0.0299513 2026-02-23T13:30:18.231 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:30:18.349 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:30:18.446 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:30:18.445+0000 7fa4e5efb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:30:18.447 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:30:18.445+0000 7fa4e5efb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:30:18.459 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:30:18.459 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:30:18.469 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:30:18.469 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:30:18.469 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:30:18.479 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:30:18.479 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:30:18.510 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:30:18.511 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:30:18.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:30:18 trial068 ceph-mon[38805]: pgmap v881: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:30:18.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:30:18 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/269416552' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:30:19.176 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1013 8 365281 365273 2.81685 0 - 0.0299513 2026-02-23T13:30:20.176 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1014 8 365281 365273 2.81408 0 - 0.0299513 2026-02-23T13:30:20.512 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:30:20.537 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:30:20 trial068 ceph-mon[38805]: pgmap v882: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:30:20.537 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:30:20 trial068 ceph-mon[38805]: 5.e scrub starts 2026-02-23T13:30:20.631 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:30:20.729 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:30:20.728+0000 7f4eb8770640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:30:20.730 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:30:20.728+0000 7f4eb8770640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:30:20.741 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:30:20.741 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:30:20.751 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:30:20.752 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:30:20.752 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:30:20.763 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:30:20.763 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:30:20.794 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:30:20.795 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:30:21.176 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1015 8 365281 365273 2.8113 0 - 0.0299513 2026-02-23T13:30:21.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:30:21 trial068 ceph-mon[38805]: 5.e scrub ok 2026-02-23T13:30:21.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:30:21 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/171879055' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:30:22.177 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1016 8 365281 365273 2.80854 0 - 0.0299513 2026-02-23T13:30:22.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:30:22 trial068 ceph-mon[38805]: pgmap v883: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:30:22.795 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:30:22.915 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:30:23.014 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:30:23.013+0000 7f4bb61b6640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:30:23.014 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:30:23.013+0000 7f4bb61b6640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:30:23.027 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:30:23.027 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:30:23.038 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:30:23.039 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:30:23.039 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:30:23.051 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:30:23.051 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:30:23.086 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:30:23.087 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:30:23.177 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1017 8 365281 365273 2.80578 0 - 0.0299513 2026-02-23T13:30:23.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:30:23 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1201691106' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:30:24.177 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1018 8 365281 365273 2.80302 0 - 0.0299513 2026-02-23T13:30:24.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:30:23 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:30:23] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:30:24.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:30:24 trial068 ceph-mon[38805]: 5.7 scrub starts 2026-02-23T13:30:24.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:30:24 trial068 ceph-mon[38805]: 5.7 scrub ok 2026-02-23T13:30:24.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:30:24 trial068 ceph-mon[38805]: pgmap v884: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:30:24.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:30:23 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:30:23.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:30:25.088 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:30:25.177 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1019 8 365281 365273 2.80027 0 - 0.0299513 2026-02-23T13:30:25.208 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:30:25.318 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:30:25.316+0000 7fe66fd1a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:30:25.318 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:30:25.317+0000 7fe66fd1a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:30:25.330 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:30:25.330 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:30:25.340 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:30:25.341 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:30:25.341 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:30:25.351 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:30:25.352 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:30:25.383 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:30:25.384 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:30:25.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:30:25 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2740894618' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:30:26.177 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T13:30:26.177221+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T13:30:26.177 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T13:30:26.177 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1020 8 365281 365273 2.79752 0 - 0.0299513 2026-02-23T13:30:26.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:30:26 trial068 ceph-mon[38805]: pgmap v885: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:30:27.177 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1021 8 365281 365273 2.79478 0 - 0.0299513 2026-02-23T13:30:27.385 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:30:27.503 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:30:27.607 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:30:27.606+0000 7efc4d6d8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:30:27.607 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:30:27.606+0000 7efc4d6d8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:30:27.619 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:30:27.620 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:30:27.630 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:30:27.630 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:30:27.630 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:30:27.641 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:30:27.641 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:30:27.671 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:30:27.671 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:30:28.177 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1022 8 365281 365273 2.79205 0 - 0.0299513 2026-02-23T13:30:28.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:30:28 trial068 ceph-mon[38805]: pgmap v886: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:30:28.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:30:28 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1116955872' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:30:29.177 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1023 8 365281 365273 2.78932 0 - 0.0299513 2026-02-23T13:30:29.673 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:30:29.792 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:30:29.889 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:30:29.888+0000 7f55068af640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:30:29.889 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:30:29.888+0000 7f55068af640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:30:29.901 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:30:29.901 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:30:29.911 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:30:29.912 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:30:29.912 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:30:29.922 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:30:29.923 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:30:29.952 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:30:29.953 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:30:30.177 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1024 8 365281 365273 2.7866 0 - 0.0299513 2026-02-23T13:30:30.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:30:30 trial068 ceph-mon[38805]: pgmap v887: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:30:30.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:30:30 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1102598104' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:30:31.177 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1025 8 365281 365273 2.78388 0 - 0.0299513 2026-02-23T13:30:31.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:30:31 trial068 ceph-mon[38805]: 5.8 scrub starts 2026-02-23T13:30:31.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:30:31 trial068 ceph-mon[38805]: 5.8 scrub ok 2026-02-23T13:30:31.954 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:30:32.073 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:30:32.176 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:30:32.175+0000 7fd3dcc26640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:30:32.176 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:30:32.175+0000 7fd3dcc26640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:30:32.177 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1026 8 365281 365273 2.78116 0 - 0.0299513 2026-02-23T13:30:32.188 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:30:32.188 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:30:32.198 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:30:32.199 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:30:32.199 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:30:32.209 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:30:32.209 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:30:32.237 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:30:32 trial068 ceph-mon[38805]: pgmap v888: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:30:32.238 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:30:32.238 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:30:33.177 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1027 8 365281 365273 2.77846 0 - 0.0299513 2026-02-23T13:30:33.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:30:33 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3393993628' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:30:34.177 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1028 8 365281 365273 2.77575 0 - 0.0299513 2026-02-23T13:30:34.239 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:30:34.262 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:30:34 trial068 ceph-mon[38805]: pgmap v889: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:30:34.262 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:30:33 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:30:33] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:30:34.265 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:30:33 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:30:33.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:30:34.356 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:30:34.453 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:30:34.452+0000 7ffa4de40640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:30:34.454 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:30:34.453+0000 7ffa4de40640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:30:34.465 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:30:34.466 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:30:34.475 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:30:34.476 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:30:34.476 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:30:34.485 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:30:34.486 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:30:34.516 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:30:34.517 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:30:35.177 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1029 8 365281 365273 2.77305 0 - 0.0299513 2026-02-23T13:30:35.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:30:35 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1525395810' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:30:36.178 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1030 8 365281 365273 2.77036 0 - 0.0299513 2026-02-23T13:30:36.518 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:30:36.543 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:30:36 trial068 ceph-mon[38805]: pgmap v890: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:30:36.635 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:30:36.732 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:30:36.730+0000 7f89b1a9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:30:36.732 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:30:36.731+0000 7f89b1a9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:30:36.744 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:30:36.744 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:30:36.754 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:30:36.754 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:30:36.754 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:30:36.763 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:30:36.764 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:30:36.792 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:30:36.793 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:30:37.178 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1031 8 365281 365273 2.76768 0 - 0.0299513 2026-02-23T13:30:37.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:30:37 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/4140627867' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:30:38.178 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1032 8 365281 365273 2.76499 0 - 0.0299513 2026-02-23T13:30:38.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:30:38 trial068 ceph-mon[38805]: pgmap v891: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:30:38.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:30:38 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2026-02-23T13:30:38.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:30:38 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-23T13:30:38.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:30:38 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2026-02-23T13:30:38.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:30:38 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T13:30:38.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:30:38 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2026-02-23T13:30:38.794 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:30:38.909 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:30:39.007 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:30:39.006+0000 7f16f3230640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:30:39.008 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:30:39.006+0000 7f16f3230640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:30:39.020 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:30:39.020 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:30:39.030 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:30:39.030 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:30:39.030 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:30:39.042 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:30:39.042 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:30:39.071 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:30:39.072 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:30:39.178 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1033 8 365281 365273 2.76232 0 - 0.0299513 2026-02-23T13:30:39.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:30:39 trial068 ceph-mon[38805]: pgmap v892: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:30:39.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:30:39 trial068 ceph-mon[38805]: pgmap v893: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:30:39.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:30:39 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/755133964' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:30:40.178 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1034 8 365281 365273 2.75965 0 - 0.0299513 2026-02-23T13:30:41.072 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:30:41.178 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1035 8 365281 365273 2.75698 0 - 0.0299513 2026-02-23T13:30:41.191 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:30:41.288 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:30:41.286+0000 7fa05b8bc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:30:41.288 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:30:41.287+0000 7fa05b8bc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:30:41.300 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:30:41.300 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:30:41.311 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:30:41.311 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:30:41.312 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:30:41.322 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:30:41.322 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:30:41.351 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:30:41 trial068 ceph-mon[38805]: pgmap v894: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:30:41.352 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:30:41.352 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:30:42.178 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1036 8 365281 365273 2.75432 0 - 0.0299513 2026-02-23T13:30:42.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:30:42 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1360964025' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:30:43.178 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1037 8 365281 365273 2.75166 0 - 0.0299513 2026-02-23T13:30:43.353 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:30:43.474 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:30:43.497 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:30:43 trial068 ceph-mon[38805]: pgmap v895: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:30:43.576 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:30:43.574+0000 7fd3ad918640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:30:43.576 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:30:43.575+0000 7fd3ad918640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:30:43.588 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:30:43.588 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:30:43.598 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:30:43.599 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:30:43.599 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:30:43.609 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:30:43.609 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:30:43.640 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:30:43.640 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:30:44.178 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1038 8 365281 365273 2.74901 0 - 0.0299513 2026-02-23T13:30:44.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:30:44 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3046949221' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:30:44.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:30:43 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:30:43] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:30:44.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:30:43 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:30:43.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:30:45.178 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1039 8 365281 365273 2.74637 0 - 0.0299513 2026-02-23T13:30:45.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:30:45 trial068 ceph-mon[38805]: pgmap v896: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:30:45.641 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:30:45.761 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:30:45.860 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:30:45.858+0000 7f130c359640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:30:45.860 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:30:45.858+0000 7f130c359640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:30:45.871 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:30:45.872 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:30:45.882 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:30:45.882 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:30:45.882 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:30:45.891 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:30:45.892 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:30:45.922 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:30:45.923 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:30:46.178 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T13:30:46.178696+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T13:30:46.178 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T13:30:46.179 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1040 8 365281 365273 2.74372 0 - 0.0299513 2026-02-23T13:30:46.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:30:46 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3147646743' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:30:47.178 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1041 8 365281 365273 2.74109 0 - 0.0299513 2026-02-23T13:30:47.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:30:47 trial068 ceph-mon[38805]: pgmap v897: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:30:47.924 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:30:48.042 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:30:48.138 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:30:48.137+0000 7ff8ef9b9640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:30:48.139 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:30:48.137+0000 7ff8ef9b9640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:30:48.150 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:30:48.150 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:30:48.160 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:30:48.161 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:30:48.161 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:30:48.170 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:30:48.171 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:30:48.178 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1042 8 365281 365273 2.73846 0 - 0.0299513 2026-02-23T13:30:48.200 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:30:48 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3412529143' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:30:48.201 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:30:48.201 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:30:49.178 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1043 8 365281 365273 2.73583 0 - 0.0299513 2026-02-23T13:30:49.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:30:49 trial068 ceph-mon[38805]: pgmap v898: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:30:50.179 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1044 8 365281 365273 2.73321 0 - 0.0299513 2026-02-23T13:30:50.202 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:30:50.319 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:30:50.426 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:30:50.425+0000 7fd6242d3640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:30:50.426 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:30:50.425+0000 7fd6242d3640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:30:50.438 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:30:50.438 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:30:50.448 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:30:50.449 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:30:50.449 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:30:50.460 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:30:50.460 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:30:50.508 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:30:50.509 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:30:51.179 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1045 8 365281 365273 2.7306 0 - 0.0299513 2026-02-23T13:30:51.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:30:51 trial068 ceph-mon[38805]: pgmap v899: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:30:51.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:30:51 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2461996514' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:30:52.179 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1046 8 365281 365273 2.72799 0 - 0.0299513 2026-02-23T13:30:52.510 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:30:52.534 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:30:52 trial068 ceph-mon[38805]: 5.a scrub starts 2026-02-23T13:30:52.534 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:30:52 trial068 ceph-mon[38805]: 5.a scrub ok 2026-02-23T13:30:52.628 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:30:52.731 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:30:52.729+0000 7f6ae0927640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:30:52.731 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:30:52.730+0000 7f6ae0927640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:30:52.742 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:30:52.743 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:30:52.752 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:30:52.753 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:30:52.753 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:30:52.763 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:30:52.764 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:30:52.793 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:30:52.793 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:30:53.179 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1047 8 365281 365273 2.72538 0 - 0.0299513 2026-02-23T13:30:53.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:30:53 trial068 ceph-mon[38805]: pgmap v900: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:30:53.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:30:53 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/4221842751' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:30:54.179 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1048 8 365281 365273 2.72278 0 - 0.0299513 2026-02-23T13:30:54.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:30:53 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:30:53] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:30:54.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:30:53 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:30:53.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:30:54.794 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:30:54.911 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:30:55.013 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:30:55.012+0000 7ffbb7f87640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:30:55.014 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:30:55.012+0000 7ffbb7f87640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:30:55.025 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:30:55.026 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:30:55.036 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:30:55.037 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:30:55.037 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:30:55.047 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:30:55.047 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:30:55.077 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:30:55.078 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:30:55.179 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1049 8 365281 365273 2.72018 0 - 0.0299513 2026-02-23T13:30:55.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:30:55 trial068 ceph-mon[38805]: pgmap v901: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:30:55.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:30:55 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/205746666' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:30:56.179 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1050 8 365281 365273 2.71759 0 - 0.0299513 2026-02-23T13:30:57.079 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:30:57.179 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1051 8 365281 365273 2.71501 0 - 0.0299513 2026-02-23T13:30:57.196 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:30:57.296 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:30:57.294+0000 7f8630b39640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:30:57.296 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:30:57.294+0000 7f8630b39640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:30:57.307 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:30:57.308 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:30:57.318 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:30:57.319 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:30:57.319 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:30:57.328 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:30:57.328 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:30:57.357 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:30:57 trial068 ceph-mon[38805]: pgmap v902: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:30:57.358 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:30:57.359 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:30:58.179 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1052 8 365281 365273 2.71243 0 - 0.0299513 2026-02-23T13:30:58.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:30:58 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/704837699' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:30:59.179 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1053 8 365281 365273 2.70985 0 - 0.0299513 2026-02-23T13:30:59.360 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:30:59.480 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:30:59.502 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:30:59 trial068 ceph-mon[38805]: pgmap v903: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:30:59.578 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:30:59.577+0000 7f2452175640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:30:59.578 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:30:59.577+0000 7f2452175640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:30:59.590 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:30:59.590 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:30:59.600 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:30:59.601 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:30:59.601 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:30:59.610 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:30:59.611 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:30:59.639 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:30:59.640 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:31:00.179 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1054 8 365281 365273 2.70728 0 - 0.0299513 2026-02-23T13:31:00.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:31:00 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3261720597' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:31:01.179 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1055 8 365281 365273 2.70471 0 - 0.0299513 2026-02-23T13:31:01.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:31:01 trial068 ceph-mon[38805]: pgmap v904: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:31:01.640 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:31:01.757 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:31:01.857 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:31:01.855+0000 7f290b0a7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:31:01.857 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:31:01.856+0000 7f290b0a7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:31:01.869 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:31:01.869 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:31:01.880 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:31:01.880 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:31:01.880 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:31:01.890 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:31:01.890 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:31:01.919 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:31:01.920 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:31:02.179 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1056 8 365281 365273 2.70215 0 - 0.0299513 2026-02-23T13:31:02.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:31:02 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1111402196' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:31:03.179 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1057 8 365281 365273 2.6996 0 - 0.0299513 2026-02-23T13:31:03.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:31:03 trial068 ceph-mon[38805]: pgmap v905: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:31:03.921 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:31:04.038 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:31:04.145 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:31:04.144+0000 7f3d59cbd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:31:04.146 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:31:04.144+0000 7f3d59cbd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:31:04.159 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:31:04.160 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:31:04.169 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:31:04.170 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:31:04.170 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:31:04.179 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1058 8 365281 365273 2.69705 0 - 0.0299513 2026-02-23T13:31:04.180 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:31:04.180 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:31:04.198 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:31:03 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:31:03] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:31:04.211 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:31:04.212 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:31:04.265 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:31:03 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:31:03.972Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:31:05.180 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1059 8 365281 365273 2.6945 0 - 0.0299513 2026-02-23T13:31:05.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:31:05 trial068 ceph-mon[38805]: pgmap v906: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:31:05.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:31:05 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1942392493' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:31:06.180 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T13:31:06.180391+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T13:31:06.180 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T13:31:06.180 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1060 8 365281 365273 2.69196 0 - 0.0299513 2026-02-23T13:31:06.213 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:31:06.332 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:31:06.429 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:31:06.428+0000 7f4545b4f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:31:06.430 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:31:06.428+0000 7f4545b4f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:31:06.442 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:31:06.443 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:31:06.453 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:31:06.453 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:31:06.453 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:31:06.464 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:31:06.464 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:31:06.495 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:31:06.495 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:31:07.180 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1061 8 365281 365273 2.68942 0 - 0.0299513 2026-02-23T13:31:07.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:31:07 trial068 ceph-mon[38805]: pgmap v907: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:31:07.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:31:07 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2603762939' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:31:08.180 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1062 8 365281 365273 2.68689 0 - 0.0299513 2026-02-23T13:31:08.496 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:31:08.614 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:31:08.721 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:31:08.720+0000 7f0c4ee38640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:31:08.721 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:31:08.720+0000 7f0c4ee38640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:31:08.733 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:31:08.733 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:31:08.743 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:31:08.743 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:31:08.743 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:31:08.753 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:31:08.754 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:31:08.786 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:31:08.787 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:31:09.180 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1063 8 365281 365273 2.68436 0 - 0.0299513 2026-02-23T13:31:09.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:31:09 trial068 ceph-mon[38805]: pgmap v908: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:31:09.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:31:09 trial068 ceph-mon[38805]: osdmap e198: 4 total, 3 up, 4 in 2026-02-23T13:31:09.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:31:09 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3807174247' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:31:10.180 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1064 8 365281 365273 2.68184 0 - 0.0299513 2026-02-23T13:31:10.788 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:31:10.905 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:31:11.002 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:31:11.000+0000 7fc0d58c4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:31:11.002 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:31:11.001+0000 7fc0d58c4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:31:11.014 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:31:11.015 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:31:11.025 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:31:11.025 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:31:11.025 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:31:11.034 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:31:11.035 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:31:11.064 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:31:11.064 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:31:11.180 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1065 8 365281 365273 2.67932 0 - 0.0299513 2026-02-23T13:31:11.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:31:11 trial068 ceph-mon[38805]: pgmap v910: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:31:11.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:31:11 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/711850046' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:31:12.180 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1066 8 365281 365273 2.6768 0 - 0.0299513 2026-02-23T13:31:13.065 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:31:13.180 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1067 8 365281 365273 2.6743 0 - 0.0299513 2026-02-23T13:31:13.184 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:31:13.280 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:31:13.279+0000 7fac98f5b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:31:13.280 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:31:13.279+0000 7fac98f5b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:31:13.293 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:31:13.293 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:31:13.303 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:31:13.303 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:31:13.303 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:31:13.313 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:31:13.314 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:31:13.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:31:13 trial068 ceph-mon[38805]: pgmap v911: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:31:13.345 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:31:13.345 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:31:14.181 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1068 8 365281 365273 2.67179 0 - 0.0299513 2026-02-23T13:31:14.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:31:14 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2104495679' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:31:14.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:31:13 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:31:13] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:31:14.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:31:13 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:31:13.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:31:15.181 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1069 8 365281 365273 2.66929 0 - 0.0299513 2026-02-23T13:31:15.346 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:31:15.461 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:31:15.483 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:31:15 trial068 ceph-mon[38805]: pgmap v912: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:31:15.561 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:31:15.560+0000 7f938cba2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:31:15.562 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:31:15.560+0000 7f938cba2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:31:15.573 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:31:15.573 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:31:15.583 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:31:15.583 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:31:15.583 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:31:15.593 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:31:15.593 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:31:15.620 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:31:15.621 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:31:16.181 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1070 8 365281 365273 2.6668 0 - 0.0299513 2026-02-23T13:31:16.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:31:16 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/49765803' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:31:17.181 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1071 8 365281 365273 2.66431 0 - 0.0299513 2026-02-23T13:31:17.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:31:17 trial068 ceph-mon[38805]: pgmap v913: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:31:17.621 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:31:17.739 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:31:17.848 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:31:17.847+0000 7faf3c45a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:31:17.848 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:31:17.847+0000 7faf3c45a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:31:17.860 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:31:17.860 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:31:17.870 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:31:17.870 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:31:17.871 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:31:17.881 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:31:17.881 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:31:17.911 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:31:17.911 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:31:18.181 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1072 8 365281 365273 2.66182 0 - 0.0299513 2026-02-23T13:31:18.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:31:18 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3661457080' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:31:19.181 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1073 8 365281 365273 2.65934 0 - 0.0299513 2026-02-23T13:31:19.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:31:19 trial068 ceph-mon[38805]: pgmap v914: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:31:19.913 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:31:20.031 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:31:20.134 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:31:20.133+0000 7fdb661c8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:31:20.134 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:31:20.133+0000 7fdb661c8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:31:20.146 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:31:20.146 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:31:20.156 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:31:20.156 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:31:20.157 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:31:20.166 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:31:20.166 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:31:20.181 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1074 8 365281 365273 2.65687 0 - 0.0299513 2026-02-23T13:31:20.196 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:31:20 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3461620517' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:31:20.197 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:31:20.197 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:31:21.181 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1075 8 365281 365273 2.65439 0 - 0.0299513 2026-02-23T13:31:21.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:31:21 trial068 ceph-mon[38805]: pgmap v915: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:31:22.181 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1076 8 365281 365273 2.65193 0 - 0.0299513 2026-02-23T13:31:22.198 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:31:22.315 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:31:22.415 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:31:22.413+0000 7f4f55d52640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:31:22.415 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:31:22.414+0000 7f4f55d52640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:31:22.426 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:31:22.427 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:31:22.436 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:31:22.437 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:31:22.437 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:31:22.446 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:31:22.447 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:31:22.477 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:31:22.478 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:31:23.181 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1077 8 365281 365273 2.64946 0 - 0.0299513 2026-02-23T13:31:23.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:31:23 trial068 ceph-mon[38805]: pgmap v916: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:31:23.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:31:23 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3274457683' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:31:24.181 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1078 8 365281 365273 2.64701 0 - 0.0299513 2026-02-23T13:31:24.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:31:24 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T13:31:24.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:31:24 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T13:31:24.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:31:23 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:31:23] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:31:24.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:31:23 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:31:23.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:31:24.478 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:31:24.596 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:31:24.693 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:31:24.691+0000 7f601ed8e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:31:24.693 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:31:24.692+0000 7f601ed8e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:31:24.705 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:31:24.705 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:31:24.715 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:31:24.715 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:31:24.716 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:31:24.725 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:31:24.725 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:31:24.754 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:31:24.755 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:31:25.181 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1079 8 365281 365273 2.64455 0 - 0.0299513 2026-02-23T13:31:25.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:31:25 trial068 ceph-mon[38805]: pgmap v917: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:31:25.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:31:25 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3694350065' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:31:26.182 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T13:31:26.181808+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T13:31:26.182 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T13:31:26.182 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1080 8 365281 365273 2.64211 0 - 0.0299513 2026-02-23T13:31:26.757 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:31:26.875 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:31:26.984 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:31:26.983+0000 7f0008b08640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:31:26.984 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:31:26.983+0000 7f0008b08640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:31:26.996 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:31:26.996 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:31:27.006 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:31:27.007 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:31:27.007 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:31:27.016 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:31:27.016 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:31:27.047 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:31:27.048 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:31:27.181 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1081 8 365281 365273 2.63966 0 - 0.0299513 2026-02-23T13:31:27.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:31:27 trial068 ceph-mon[38805]: pgmap v918: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:31:27.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:31:27 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1440833187' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:31:28.182 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1082 8 365281 365273 2.63722 0 - 0.0299513 2026-02-23T13:31:29.048 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:31:29.167 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:31:29.181 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1083 8 365281 365273 2.63479 0 - 0.0299513 2026-02-23T13:31:29.266 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:31:29.265+0000 7f28815c9640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:31:29.266 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:31:29.265+0000 7f28815c9640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:31:29.278 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:31:29.278 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:31:29.288 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:31:29.289 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:31:29.289 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:31:29.298 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:31:29.298 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:31:29.327 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:31:29 trial068 ceph-mon[38805]: pgmap v919: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:31:29.328 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:31:29.328 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:31:30.182 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1084 8 365281 365273 2.63236 0 - 0.0299513 2026-02-23T13:31:30.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:31:30 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/42470074' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:31:31.182 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1085 8 365281 365273 2.62993 0 - 0.0299513 2026-02-23T13:31:31.329 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:31:31.448 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:31:31.469 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:31:31 trial068 ceph-mon[38805]: pgmap v920: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:31:31.549 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:31:31.547+0000 7f72384a4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:31:31.549 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:31:31.548+0000 7f72384a4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:31:31.561 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:31:31.561 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:31:31.572 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:31:31.572 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:31:31.572 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:31:31.584 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:31:31.584 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:31:31.617 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:31:31.618 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:31:32.182 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1086 8 365281 365273 2.62751 0 - 0.0299513 2026-02-23T13:31:32.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:31:32 trial068 ceph-mon[38805]: 5.e scrub starts 2026-02-23T13:31:32.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:31:32 trial068 ceph-mon[38805]: 5.e scrub ok 2026-02-23T13:31:32.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:31:32 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/204079989' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:31:33.182 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1087 8 365281 365273 2.62509 0 - 0.0299513 2026-02-23T13:31:33.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:31:33 trial068 ceph-mon[38805]: pgmap v921: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:31:33.619 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:31:33.738 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:31:33.832 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:31:33.830+0000 7fd183ff8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:31:33.832 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:31:33.831+0000 7fd183ff8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:31:33.844 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:31:33.844 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:31:33.854 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:31:33.854 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:31:33.854 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:31:33.865 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:31:33.865 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:31:33.896 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:31:33.896 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:31:34.182 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1088 8 365281 365273 2.62268 0 - 0.0299513 2026-02-23T13:31:34.265 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:31:33 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:31:33.972Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:31:34.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:31:33 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:31:33] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:31:34.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:31:34 trial068 ceph-mon[38805]: 5.7 scrub starts 2026-02-23T13:31:34.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:31:34 trial068 ceph-mon[38805]: 5.7 scrub ok 2026-02-23T13:31:34.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:31:34 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2001518576' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:31:35.182 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1089 8 365281 365273 2.62027 0 - 0.0299513 2026-02-23T13:31:35.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:31:35 trial068 ceph-mon[38805]: pgmap v922: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:31:35.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:31:35 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T13:31:35.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:31:35 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T13:31:35.897 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:31:36.015 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:31:36.113 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:31:36.112+0000 7f187889e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:31:36.113 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:31:36.112+0000 7f187889e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:31:36.125 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:31:36.126 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:31:36.135 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:31:36.136 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:31:36.136 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:31:36.147 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:31:36.147 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:31:36.177 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:31:36.177 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:31:36.182 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1090 8 365281 365273 2.61787 0 - 0.0299513 2026-02-23T13:31:36.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:31:36 trial068 ceph-mon[38805]: 5.5 scrub starts 2026-02-23T13:31:36.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:31:36 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2639217980' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:31:37.182 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1091 8 365281 365273 2.61547 0 - 0.0299513 2026-02-23T13:31:37.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:31:37 trial068 ceph-mon[38805]: pgmap v923: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:31:37.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:31:37 trial068 ceph-mon[38805]: 5.5 scrub ok 2026-02-23T13:31:38.179 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:31:38.182 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1092 8 365281 365273 2.61307 0 - 0.0299513 2026-02-23T13:31:38.294 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:31:38.395 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:31:38.394+0000 7f90b4e21640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:31:38.395 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:31:38.394+0000 7f90b4e21640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:31:38.407 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:31:38.407 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:31:38.417 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:31:38.417 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:31:38.417 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:31:38.428 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:31:38.428 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:31:38.440 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:31:38 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2026-02-23T13:31:38.440 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:31:38 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-23T13:31:38.440 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:31:38 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2026-02-23T13:31:38.440 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:31:38 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T13:31:38.441 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:31:38 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2026-02-23T13:31:38.458 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:31:38.458 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:31:39.182 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1093 8 365281 365273 2.61068 0 - 0.0299513 2026-02-23T13:31:39.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:31:39 trial068 ceph-mon[38805]: pgmap v924: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:31:39.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:31:39 trial068 ceph-mon[38805]: pgmap v925: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:31:39.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:31:39 trial068 ceph-mon[38805]: pgmap v926: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:31:39.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:31:39 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2044579382' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:31:40.182 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1094 8 365281 365273 2.60829 0 - 0.0299513 2026-02-23T13:31:40.459 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:31:40.484 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:31:40 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T13:31:40.484 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:31:40 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T13:31:40.576 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:31:40.681 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:31:40.680+0000 7f6048c9a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:31:40.681 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:31:40.680+0000 7f6048c9a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:31:40.694 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:31:40.694 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:31:40.704 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:31:40.704 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:31:40.704 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:31:40.714 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:31:40.714 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:31:40.742 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:31:40.743 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:31:41.183 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1095 8 365281 365273 2.60591 0 - 0.0299513 2026-02-23T13:31:41.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:31:41 trial068 ceph-mon[38805]: pgmap v927: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:31:41.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:31:41 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1367567789' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:31:42.183 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1096 8 365281 365273 2.60353 0 - 0.0299513 2026-02-23T13:31:42.744 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:31:42.864 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:31:42.969 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:31:42.967+0000 7eff25247640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:31:42.969 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:31:42.968+0000 7eff25247640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:31:42.980 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:31:42.981 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:31:42.990 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:31:42.992 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:31:42.992 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:31:43.001 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:31:43.002 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:31:43.032 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:31:43.033 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:31:43.183 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1097 8 365281 365273 2.60116 0 - 0.0299513 2026-02-23T13:31:43.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:31:43 trial068 ceph-mon[38805]: pgmap v928: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:31:43.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:31:43 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/648456030' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:31:44.183 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1098 8 365281 365273 2.59879 0 - 0.0299513 2026-02-23T13:31:44.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:31:43 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:31:43] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:31:44.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:31:43 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:31:43.974Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:31:45.034 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:31:45.151 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:31:45.183 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1099 8 365281 365273 2.59643 0 - 0.0299513 2026-02-23T13:31:45.251 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:31:45.250+0000 7f3839ee4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:31:45.251 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:31:45.250+0000 7f3839ee4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:31:45.263 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:31:45.263 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:31:45.273 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:31:45.273 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:31:45.274 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:31:45.283 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:31:45.283 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:31:45.311 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:31:45 trial068 ceph-mon[38805]: pgmap v929: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:31:45.312 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:31:45.313 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:31:46.183 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T13:31:46.183273+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T13:31:46.183 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T13:31:46.183 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1100 8 365281 365273 2.59407 0 - 0.0299513 2026-02-23T13:31:46.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:31:46 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1782620272' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:31:47.183 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1101 8 365281 365273 2.59171 0 - 0.0299513 2026-02-23T13:31:47.314 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:31:47.433 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:31:47.455 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:31:47 trial068 ceph-mon[38805]: pgmap v930: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:31:47.534 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:31:47.532+0000 7f217fad9640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:31:47.534 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:31:47.533+0000 7f217fad9640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:31:47.546 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:31:47.547 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:31:47.557 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:31:47.558 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:31:47.558 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:31:47.568 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:31:47.568 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:31:47.599 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:31:47.600 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:31:48.183 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1102 8 365281 365273 2.58936 0 - 0.0299513 2026-02-23T13:31:48.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:31:48 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2502455370' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:31:49.183 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1103 8 365281 365273 2.58701 0 - 0.0299513 2026-02-23T13:31:49.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:31:49 trial068 ceph-mon[38805]: pgmap v931: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:31:49.601 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:31:49.717 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:31:49.812 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:31:49.811+0000 7ff1258a6640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:31:49.813 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:31:49.811+0000 7ff1258a6640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:31:49.824 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:31:49.825 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:31:49.834 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:31:49.835 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:31:49.835 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:31:49.844 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:31:49.845 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:31:49.875 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:31:49.876 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:31:50.183 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1104 8 365281 365273 2.58467 0 - 0.0299513 2026-02-23T13:31:50.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:31:50 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2087342614' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:31:51.183 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1105 8 365281 365273 2.58233 0 - 0.0299513 2026-02-23T13:31:51.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:31:51 trial068 ceph-mon[38805]: pgmap v932: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:31:51.877 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:31:51.996 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:31:52.099 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:31:52.098+0000 7f3469543640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:31:52.099 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:31:52.098+0000 7f3469543640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:31:52.111 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:31:52.111 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:31:52.121 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:31:52.121 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:31:52.121 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:31:52.131 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:31:52.131 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:31:52.160 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:31:52.160 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:31:52.183 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1106 8 365281 365273 2.57999 0 - 0.0299513 2026-02-23T13:31:52.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:31:52 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2303127289' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:31:53.183 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1107 8 365281 365273 2.57766 0 - 0.0299513 2026-02-23T13:31:53.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:31:53 trial068 ceph-mon[38805]: pgmap v933: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:31:54.161 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:31:54.183 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1108 8 365281 365273 2.57534 0 - 0.0299513 2026-02-23T13:31:54.279 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:31:54.302 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:31:53 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:31:53] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:31:54.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:31:53 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:31:53.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:31:54.381 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:31:54.379+0000 7f1ac13df640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:31:54.381 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:31:54.380+0000 7f1ac13df640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:31:54.393 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:31:54.393 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:31:54.403 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:31:54.404 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:31:54.404 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:31:54.413 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:31:54.414 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:31:54.444 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:31:54.445 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:31:55.184 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1109 8 365281 365273 2.57302 0 - 0.0299513 2026-02-23T13:31:55.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:31:55 trial068 ceph-mon[38805]: pgmap v934: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:31:55.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:31:55 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1315136335' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:31:56.184 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1110 8 365281 365273 2.5707 0 - 0.0299513 2026-02-23T13:31:56.445 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:31:56.564 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:31:56.663 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:31:56.662+0000 7fe0dd88c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:31:56.663 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:31:56.662+0000 7fe0dd88c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:31:56.675 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:31:56.675 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:31:56.685 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:31:56.686 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:31:56.686 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:31:56.695 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:31:56.695 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:31:56.727 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:31:56.728 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:31:57.184 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1111 8 365281 365273 2.56838 0 - 0.0299513 2026-02-23T13:31:57.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:31:57 trial068 ceph-mon[38805]: pgmap v935: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:31:57.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:31:57 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3076621261' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:31:58.184 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1112 8 365281 365273 2.56607 0 - 0.0299513 2026-02-23T13:31:58.729 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:31:58.851 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:31:58.951 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:31:58.949+0000 7f5e5a4e4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:31:58.951 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:31:58.949+0000 7f5e5a4e4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:31:58.962 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:31:58.963 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:31:58.972 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:31:58.973 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:31:58.973 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:31:58.984 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:31:58.984 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:31:59.014 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:31:59.015 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:31:59.184 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1113 8 365281 365273 2.56377 0 - 0.0299513 2026-02-23T13:31:59.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:31:59 trial068 ceph-mon[38805]: 5.a scrub starts 2026-02-23T13:31:59.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:31:59 trial068 ceph-mon[38805]: 5.a scrub ok 2026-02-23T13:31:59.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:31:59 trial068 ceph-mon[38805]: pgmap v936: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:31:59.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:31:59 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3570738749' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:32:00.184 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1114 8 365281 365273 2.56147 0 - 0.0299513 2026-02-23T13:32:00.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:32:00 trial068 ceph-mon[38805]: 5.8 scrub starts 2026-02-23T13:32:00.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:32:00 trial068 ceph-mon[38805]: 5.8 scrub ok 2026-02-23T13:32:01.016 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:32:01.133 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:32:01.184 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1115 8 365281 365273 2.55917 0 - 0.0299513 2026-02-23T13:32:01.227 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:32:01.226+0000 7f92b948f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:32:01.227 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:32:01.226+0000 7f92b948f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:32:01.239 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:32:01.239 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:32:01.249 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:32:01.249 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:32:01.249 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:32:01.259 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:32:01.260 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:32:01.289 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:32:01.290 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:32:01.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:32:01 trial068 ceph-mon[38805]: pgmap v937: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:32:02.184 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1116 8 365281 365273 2.55688 0 - 0.0299513 2026-02-23T13:32:02.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:32:02 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/87019000' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:32:03.184 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1117 8 365281 365273 2.55459 0 - 0.0299513 2026-02-23T13:32:03.291 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:32:03.409 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:32:03.510 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:32:03.509+0000 7f0a9be1a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:32:03.511 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:32:03.509+0000 7f0a9be1a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:32:03.511 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:32:03 trial068 ceph-mon[38805]: pgmap v938: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:32:03.522 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:32:03.523 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:32:03.532 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:32:03.533 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:32:03.533 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:32:03.542 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:32:03.543 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:32:03.572 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:32:03.573 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:32:04.185 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1118 8 365281 365273 2.5523 0 - 0.0299513 2026-02-23T13:32:04.265 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:32:03 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:32:03.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:32:04.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:32:04 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/989985438' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:32:04.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:32:03 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:32:03] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:32:05.185 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1119 8 365281 365273 2.55002 0 - 0.0299513 2026-02-23T13:32:05.573 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:32:05.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:32:05 trial068 ceph-mon[38805]: pgmap v939: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:32:05.691 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:32:05.789 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:32:05.788+0000 7f488fd2a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:32:05.790 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:32:05.788+0000 7f488fd2a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:32:05.801 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:32:05.801 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:32:05.811 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:32:05.811 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:32:05.811 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:32:05.821 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:32:05.822 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:32:05.851 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:32:05.852 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:32:06.185 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T13:32:06.184972+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T13:32:06.185 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T13:32:06.185 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1120 8 365281 365273 2.54774 0 - 0.0299513 2026-02-23T13:32:06.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:32:06 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1608587642' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:32:07.185 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1121 8 365281 365273 2.54547 0 - 0.0299513 2026-02-23T13:32:07.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:32:07 trial068 ceph-mon[38805]: pgmap v940: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:32:07.854 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:32:07.974 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:32:08.070 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:32:08.069+0000 7fb8628d4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:32:08.070 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:32:08.069+0000 7fb8628d4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:32:08.082 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:32:08.082 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:32:08.091 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:32:08.092 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:32:08.092 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:32:08.101 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:32:08.101 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:32:08.132 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:32:08.132 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:32:08.185 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1122 8 365281 365273 2.5432 0 - 0.0299513 2026-02-23T13:32:08.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:32:08 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/31523042' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:32:09.185 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1123 8 365281 365273 2.54094 0 - 0.0299513 2026-02-23T13:32:09.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:32:09 trial068 ceph-mon[38805]: pgmap v941: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:32:10.134 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:32:10.185 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1124 8 365281 365273 2.53868 0 - 0.0299513 2026-02-23T13:32:10.253 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:32:10.349 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:32:10.348+0000 7f0fbb61e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:32:10.349 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:32:10.348+0000 7f0fbb61e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:32:10.361 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:32:10.361 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:32:10.371 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:32:10.371 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:32:10.371 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:32:10.381 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:32:10.382 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:32:10.411 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:32:10.411 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:32:11.185 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1125 8 365281 365273 2.53642 0 - 0.0299513 2026-02-23T13:32:11.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:32:11 trial068 ceph-mon[38805]: pgmap v942: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:32:11.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:32:11 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3729320509' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:32:12.185 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1126 8 365281 365273 2.53417 0 - 0.0299513 2026-02-23T13:32:12.412 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:32:12.530 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:32:12.630 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:32:12.628+0000 7f6bed423640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:32:12.630 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:32:12.629+0000 7f6bed423640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:32:12.642 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:32:12.642 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:32:12.652 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:32:12.653 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:32:12.653 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:32:12.663 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:32:12.663 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:32:12.694 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:32:12.695 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:32:13.185 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1127 8 365281 365273 2.53192 0 - 0.0299513 2026-02-23T13:32:13.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:32:13 trial068 ceph-mon[38805]: pgmap v943: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:32:13.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:32:13 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/273969622' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:32:14.185 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1128 8 365281 365273 2.52968 0 - 0.0299513 2026-02-23T13:32:14.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:32:13 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:32:13] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:32:14.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:32:13 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:32:13.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:32:14.695 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:32:14.810 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:32:14.905 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:32:14.903+0000 7f53e7e12640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:32:14.905 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:32:14.904+0000 7f53e7e12640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:32:14.917 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:32:14.917 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:32:14.927 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:32:14.927 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:32:14.927 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:32:14.937 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:32:14.937 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:32:14.965 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:32:14.965 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:32:15.185 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1129 8 365281 365273 2.52743 0 - 0.0299513 2026-02-23T13:32:15.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:32:15 trial068 ceph-mon[38805]: pgmap v944: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:32:15.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:32:15 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2169238419' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:32:16.185 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1130 8 365281 365273 2.5252 0 - 0.0299513 2026-02-23T13:32:16.967 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:32:17.084 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:32:17.185 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1131 8 365281 365273 2.52297 0 - 0.0299513 2026-02-23T13:32:17.192 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:32:17.191+0000 7ff91655e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:32:17.192 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:32:17.191+0000 7ff91655e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:32:17.204 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:32:17.205 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:32:17.215 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:32:17.215 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:32:17.215 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:32:17.225 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:32:17.225 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:32:17.243 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:32:17 trial068 ceph-mon[38805]: pgmap v945: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:32:17.243 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:32:17 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/4077551697' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:32:17.256 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:32:17.256 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:32:18.185 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1132 8 365281 365273 2.52074 0 - 0.0299513 2026-02-23T13:32:19.186 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1133 8 365281 365273 2.51851 0 - 0.0299513 2026-02-23T13:32:19.257 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:32:19.375 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:32:19.472 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:32:19.471+0000 7f763cf43640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:32:19.472 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:32:19.471+0000 7f763cf43640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:32:19.484 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:32:19.485 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:32:19.485 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:32:19 trial068 ceph-mon[38805]: pgmap v946: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:32:19.494 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:32:19.494 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:32:19.495 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:32:19.504 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:32:19.504 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:32:19.534 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:32:19.535 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:32:20.186 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1134 8 365281 365273 2.51629 0 - 0.0299513 2026-02-23T13:32:20.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:32:20 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3044564516' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:32:21.186 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1135 8 365281 365273 2.51407 0 - 0.0299513 2026-02-23T13:32:21.536 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:32:21.561 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:32:21 trial068 ceph-mon[38805]: pgmap v947: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:32:21.653 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:32:21.760 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:32:21.759+0000 7f30062ba640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:32:21.760 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:32:21.759+0000 7f30062ba640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:32:21.772 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:32:21.773 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:32:21.783 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:32:21.783 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:32:21.783 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:32:21.793 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:32:21.794 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:32:21.824 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:32:21.824 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:32:22.186 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1136 8 365281 365273 2.51186 0 - 0.0299513 2026-02-23T13:32:22.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:32:22 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2207625275' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:32:23.186 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1137 8 365281 365273 2.50965 0 - 0.0299513 2026-02-23T13:32:23.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:32:23 trial068 ceph-mon[38805]: pgmap v948: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:32:23.825 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:32:23.942 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:32:24.044 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:32:24.042+0000 7f17c5722640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:32:24.044 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:32:24.043+0000 7f17c5722640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:32:24.056 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:32:24.057 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:32:24.067 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:32:24.067 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:32:24.068 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:32:24.077 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:32:24.077 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:32:24.106 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:32:23 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:32:23] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:32:24.107 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:32:24.108 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:32:24.186 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1138 8 365281 365273 2.50745 0 - 0.0299513 2026-02-23T13:32:24.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:32:23 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:32:23.972Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:32:24.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:32:24 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1862281073' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:32:25.186 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1139 8 365281 365273 2.50524 0 - 0.0299513 2026-02-23T13:32:25.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:32:25 trial068 ceph-mon[38805]: pgmap v949: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:32:26.109 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:32:26.186 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T13:32:26.186899+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T13:32:26.187 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T13:32:26.187 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1140 8 365281 365273 2.50305 0 - 0.0299513 2026-02-23T13:32:26.229 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:32:26.331 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:32:26.329+0000 7fa5c3365640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:32:26.331 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:32:26.330+0000 7fa5c3365640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:32:26.343 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:32:26.343 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:32:26.353 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:32:26.354 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:32:26.354 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:32:26.363 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:32:26.363 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:32:26.393 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:32:26.393 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:32:27.187 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1141 8 365281 365273 2.50085 0 - 0.0299513 2026-02-23T13:32:27.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:32:27 trial068 ceph-mon[38805]: pgmap v950: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:32:27.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:32:27 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/4213090127' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:32:28.187 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1142 8 365281 365273 2.49866 0 - 0.0299513 2026-02-23T13:32:28.394 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:32:28.515 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:32:28.620 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:32:28.618+0000 7f4986b24640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:32:28.620 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:32:28.619+0000 7f4986b24640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:32:28.632 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:32:28.632 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:32:28.642 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:32:28.642 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:32:28.643 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:32:28.652 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:32:28.652 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:32:28.683 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:32:28.684 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:32:29.187 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1143 8 365281 365273 2.49648 0 - 0.0299513 2026-02-23T13:32:29.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:32:29 trial068 ceph-mon[38805]: pgmap v951: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:32:29.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:32:29 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/113623467' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:32:30.187 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1144 8 365281 365273 2.49429 0 - 0.0299513 2026-02-23T13:32:30.685 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:32:30.801 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:32:30.902 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:32:30.900+0000 7f12d545b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:32:30.902 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:32:30.901+0000 7f12d545b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:32:30.914 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:32:30.914 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:32:30.924 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:32:30.925 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:32:30.925 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:32:30.935 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:32:30.935 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:32:30.965 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:32:30.966 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:32:31.187 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1145 8 365281 365273 2.49212 0 - 0.0299513 2026-02-23T13:32:31.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:32:31 trial068 ceph-mon[38805]: pgmap v952: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:32:31.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:32:31 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1402663542' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:32:32.187 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1146 8 365281 365273 2.48994 0 - 0.0299513 2026-02-23T13:32:32.967 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:32:33.087 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:32:33.187 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1147 8 365281 365273 2.48777 0 - 0.0299513 2026-02-23T13:32:33.189 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:32:33.188+0000 7f7ce294d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:32:33.189 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:32:33.188+0000 7f7ce294d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:32:33.201 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:32:33.201 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:32:33.211 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:32:33.212 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:32:33.212 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:32:33.222 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:32:33.223 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:32:33.255 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:32:33.255 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:32:33.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:32:33 trial068 ceph-mon[38805]: pgmap v953: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:32:33.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:32:33 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1433234128' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:32:34.187 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1148 8 365281 365273 2.4856 0 - 0.0299513 2026-02-23T13:32:34.265 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:32:33 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:32:33.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:32:34.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:32:33 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:32:33] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:32:35.187 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1149 8 365281 365273 2.48344 0 - 0.0299513 2026-02-23T13:32:35.256 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:32:35.375 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:32:35.477 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:32:35.476+0000 7fa4f0c29640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:32:35.477 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:32:35.476+0000 7fa4f0c29640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:32:35.489 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:32:35.490 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:32:35.499 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:32:35.500 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:32:35.500 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:32:35 trial068 ceph-mon[38805]: pgmap v954: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:32:35.500 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:32:35.510 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:32:35.510 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:32:35.538 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:32:35.538 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:32:36.187 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1150 8 365281 365273 2.48128 0 - 0.0299513 2026-02-23T13:32:36.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:32:36 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/753811684' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:32:37.187 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1151 8 365281 365273 2.47912 0 - 0.0299513 2026-02-23T13:32:37.539 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:32:37.565 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:32:37 trial068 ceph-mon[38805]: pgmap v955: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:32:37.658 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:32:37.757 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:32:37.755+0000 7f2b8e6c9640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:32:37.757 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:32:37.756+0000 7f2b8e6c9640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:32:37.769 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:32:37.769 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:32:37.779 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:32:37.780 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:32:37.780 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:32:37.790 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:32:37.790 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:32:37.821 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:32:37.821 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:32:38.188 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1152 8 365281 365273 2.47697 0 - 0.0299513 2026-02-23T13:32:38.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:32:38 trial068 ceph-mon[38805]: 5.e scrub starts 2026-02-23T13:32:38.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:32:38 trial068 ceph-mon[38805]: 5.e scrub ok 2026-02-23T13:32:38.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:32:38 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3979404572' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:32:38.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:32:38 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2026-02-23T13:32:38.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:32:38 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-23T13:32:38.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:32:38 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2026-02-23T13:32:38.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:32:38 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T13:32:38.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:32:38 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2026-02-23T13:32:39.188 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1153 8 365281 365273 2.47482 0 - 0.0299513 2026-02-23T13:32:39.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:32:39 trial068 ceph-mon[38805]: pgmap v956: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:32:39.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:32:39 trial068 ceph-mon[38805]: pgmap v957: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:32:39.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:32:39 trial068 ceph-mon[38805]: pgmap v958: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:32:39.822 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:32:39.939 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:32:40.039 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:32:40.037+0000 7f6747eff640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:32:40.039 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:32:40.037+0000 7f6747eff640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:32:40.050 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:32:40.051 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:32:40.061 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:32:40.061 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:32:40.061 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:32:40.072 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:32:40.072 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:32:40.102 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:32:40.103 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:32:40.188 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1154 8 365281 365273 2.47268 0 - 0.0299513 2026-02-23T13:32:40.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:32:40 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/85304494' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:32:41.188 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1155 8 365281 365273 2.47054 0 - 0.0299513 2026-02-23T13:32:41.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:32:41 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T13:32:41.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:32:41 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T13:32:41.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:32:41 trial068 ceph-mon[38805]: pgmap v959: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:32:42.104 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:32:42.188 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1156 8 365281 365273 2.4684 0 - 0.0299513 2026-02-23T13:32:42.225 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:32:42.326 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:32:42.324+0000 7feca9884640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:32:42.326 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:32:42.324+0000 7feca9884640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:32:42.339 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:32:42.339 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:32:42.349 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:32:42.349 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:32:42.349 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:32:42.361 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:32:42.361 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:32:42.390 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:32:42 trial068 ceph-mon[38805]: 5.7 scrub starts 2026-02-23T13:32:42.390 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:32:42 trial068 ceph-mon[38805]: 5.7 scrub ok 2026-02-23T13:32:42.391 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:32:42.391 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:32:43.188 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1157 8 365281 365273 2.46627 0 - 0.0299513 2026-02-23T13:32:43.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:32:43 trial068 ceph-mon[38805]: pgmap v960: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:32:43.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:32:43 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/417015675' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:32:44.188 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1158 8 365281 365273 2.46414 0 - 0.0299513 2026-02-23T13:32:44.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:32:43 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:32:43] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:32:44.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:32:43 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:32:43.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:32:44.393 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:32:44.513 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:32:44.608 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:32:44.607+0000 7f579388f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:32:44.609 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:32:44.607+0000 7f579388f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:32:44.622 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:32:44.622 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:32:44.632 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:32:44.633 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:32:44.633 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:32:44.643 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:32:44.644 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:32:44.674 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:32:44.674 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:32:45.188 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1159 8 365281 365273 2.46201 0 - 0.0299513 2026-02-23T13:32:45.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:32:45 trial068 ceph-mon[38805]: pgmap v961: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:32:45.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:32:45 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2142824079' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:32:46.188 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T13:32:46.188543+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T13:32:46.188 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T13:32:46.189 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1160 8 365281 365273 2.45989 0 - 0.0299513 2026-02-23T13:32:46.675 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:32:46.791 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:32:46.892 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:32:46.891+0000 7f9dced3a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:32:46.893 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:32:46.891+0000 7f9dced3a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:32:46.904 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:32:46.905 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:32:46.914 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:32:46.915 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:32:46.915 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:32:46.924 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:32:46.925 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:32:46.954 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:32:46.954 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:32:47.188 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1161 8 365281 365273 2.45777 0 - 0.0299513 2026-02-23T13:32:47.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:32:47 trial068 ceph-mon[38805]: pgmap v962: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:32:47.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:32:47 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3155037188' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:32:48.188 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1162 8 365281 365273 2.45566 0 - 0.0299513 2026-02-23T13:32:48.955 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:32:49.073 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:32:49.178 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:32:49.176+0000 7f57bd01e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:32:49.178 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:32:49.176+0000 7f57bd01e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:32:49.188 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1163 8 365281 365273 2.45354 0 - 0.0299513 2026-02-23T13:32:49.190 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:32:49.190 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:32:49.200 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:32:49.201 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:32:49.201 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:32:49.211 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:32:49.211 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:32:49.242 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:32:49.243 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:32:49.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:32:49 trial068 ceph-mon[38805]: pgmap v963: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:32:49.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:32:49 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/675761686' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:32:50.189 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1164 8 365281 365273 2.45144 0 - 0.0299513 2026-02-23T13:32:51.189 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1165 8 365281 365273 2.44933 0 - 0.0299513 2026-02-23T13:32:51.243 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:32:51.362 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:32:51.460 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:32:51.459+0000 7f592f178640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:32:51.461 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:32:51.459+0000 7f592f178640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:32:51.472 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:32:51.473 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:32:51.482 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:32:51.483 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:32:51.483 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:32:51.492 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:32:51.492 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:32:51.520 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:32:51 trial068 ceph-mon[38805]: pgmap v964: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:32:51.521 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:32:51.522 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:32:52.189 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1166 8 365281 365273 2.44723 0 - 0.0299513 2026-02-23T13:32:52.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:32:52 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3119778323' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:32:53.189 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1167 8 365281 365273 2.44513 0 - 0.0299513 2026-02-23T13:32:53.523 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:32:53.545 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:32:53 trial068 ceph-mon[38805]: pgmap v965: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:32:53.638 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:32:53.741 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:32:53.739+0000 7f93ded57640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:32:53.741 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:32:53.739+0000 7f93ded57640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:32:53.752 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:32:53.753 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:32:53.763 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:32:53.763 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:32:53.763 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:32:53.774 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:32:53.774 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:32:53.805 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:32:53.806 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:32:54.093 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:32:53 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:32:53] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:32:54.189 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1168 8 365281 365273 2.44304 0 - 0.0299513 2026-02-23T13:32:54.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:32:53 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:32:53.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:32:54.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:32:54 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3548915075' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:32:55.189 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1169 8 365281 365273 2.44095 0 - 0.0299513 2026-02-23T13:32:55.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:32:55 trial068 ceph-mon[38805]: pgmap v966: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:32:55.807 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:32:55.927 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:32:56.031 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:32:56.030+0000 7f476c215640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:32:56.031 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:32:56.030+0000 7f476c215640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:32:56.043 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:32:56.043 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:32:56.053 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:32:56.054 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:32:56.054 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:32:56.064 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:32:56.064 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:32:56.095 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:32:56.095 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:32:56.189 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1170 8 365281 365273 2.43887 0 - 0.0299513 2026-02-23T13:32:56.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:32:56 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/4222863203' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:32:57.189 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1171 8 365281 365273 2.43678 0 - 0.0299513 2026-02-23T13:32:57.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:32:57 trial068 ceph-mon[38805]: pgmap v967: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:32:57.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:32:57 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T13:32:57.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:32:57 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T13:32:58.096 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:32:58.189 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1172 8 365281 365273 2.4347 0 - 0.0299513 2026-02-23T13:32:58.214 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:32:58.313 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:32:58.312+0000 7f34fd8ec640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:32:58.313 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:32:58.312+0000 7f34fd8ec640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:32:58.325 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:32:58.325 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:32:58.335 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:32:58.336 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:32:58.336 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:32:58.345 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:32:58.346 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:32:58.375 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:32:58.375 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:32:59.189 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1173 8 365281 365273 2.43263 0 - 0.0299513 2026-02-23T13:32:59.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:32:59 trial068 ceph-mon[38805]: pgmap v968: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:32:59.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:32:59 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3536882114' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:33:00.189 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1174 8 365281 365273 2.43056 0 - 0.0299513 2026-02-23T13:33:00.376 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:33:00.494 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:33:00.592 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:33:00.590+0000 7fe91f4a1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:33:00.592 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:33:00.591+0000 7fe91f4a1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:33:00.604 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:33:00.604 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:33:00.614 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:33:00.614 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:33:00.614 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:33:00.623 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:33:00.624 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:33:00.653 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:33:00.654 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:33:01.189 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1175 8 365281 365273 2.42849 0 - 0.0299513 2026-02-23T13:33:01.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:33:01 trial068 ceph-mon[38805]: pgmap v969: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:33:01.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:33:01 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3630453894' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:33:02.190 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1176 8 365281 365273 2.42642 0 - 0.0299513 2026-02-23T13:33:02.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:33:02 trial068 ceph-mon[38805]: 5.5 scrub starts 2026-02-23T13:33:02.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:33:02 trial068 ceph-mon[38805]: 5.5 scrub ok 2026-02-23T13:33:02.655 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:33:02.771 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:33:02.868 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:33:02.867+0000 7f881ee46640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:33:02.868 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:33:02.867+0000 7f881ee46640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:33:02.880 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:33:02.881 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:33:02.891 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:33:02.891 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:33:02.891 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:33:02.901 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:33:02.902 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:33:02.933 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:33:02.933 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:33:03.190 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1177 8 365281 365273 2.42436 0 - 0.0299513 2026-02-23T13:33:03.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:33:03 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T13:33:03.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:33:03 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T13:33:03.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:33:03 trial068 ceph-mon[38805]: pgmap v970: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:33:03.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:33:03 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/4215300761' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:33:04.190 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1178 8 365281 365273 2.4223 0 - 0.0299513 2026-02-23T13:33:04.280 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:33:03 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:33:03] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:33:04.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:33:03 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:33:03.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:33:04.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:33:04 trial068 ceph-mon[38805]: 5.8 scrub starts 2026-02-23T13:33:04.935 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:33:05.050 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:33:05.146 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:33:05.144+0000 7f3ca0b88640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:33:05.146 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:33:05.144+0000 7f3ca0b88640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:33:05.158 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:33:05.159 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:33:05.168 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:33:05.169 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:33:05.169 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:33:05.180 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:33:05.180 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:33:05.189 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1179 8 365281 365273 2.42025 0 - 0.0299513 2026-02-23T13:33:05.210 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:33:05.211 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:33:05.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:33:05 trial068 ceph-mon[38805]: 5.8 scrub ok 2026-02-23T13:33:05.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:33:05 trial068 ceph-mon[38805]: pgmap v971: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:33:05.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:33:05 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2897950320' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:33:06.190 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T13:33:06.190055+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T13:33:06.190 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T13:33:06.190 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1180 8 365281 365273 2.4182 0 - 0.0299513 2026-02-23T13:33:07.190 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1181 8 365281 365273 2.41615 0 - 0.0299513 2026-02-23T13:33:07.212 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:33:07.329 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:33:07.431 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:33:07.429+0000 7f2f2c570640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:33:07.431 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:33:07.430+0000 7f2f2c570640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:33:07.442 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:33:07.443 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:33:07.452 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:33:07.452 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:33:07.453 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:33:07.463 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:33:07.463 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:33:07.493 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:33:07 trial068 ceph-mon[38805]: pgmap v972: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:33:07.493 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:33:07.494 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:33:08.190 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1182 8 365281 365273 2.41411 0 - 0.0299513 2026-02-23T13:33:08.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:33:08 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1523700151' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:33:09.190 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1183 8 365281 365273 2.41206 0 - 0.0299513 2026-02-23T13:33:09.495 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:33:09.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:33:09 trial068 ceph-mon[38805]: pgmap v973: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:33:09.613 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:33:09.710 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:33:09.709+0000 7f539fe67640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:33:09.710 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:33:09.709+0000 7f539fe67640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:33:09.723 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:33:09.723 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:33:09.732 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:33:09.733 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:33:09.733 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:33:09.743 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:33:09.743 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:33:09.773 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:33:09.773 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:33:10.190 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1184 8 365281 365273 2.41003 0 - 0.0299513 2026-02-23T13:33:10.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:33:10 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3759961676' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:33:11.190 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1185 8 365281 365273 2.40799 0 - 0.0299513 2026-02-23T13:33:11.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:33:11 trial068 ceph-mon[38805]: pgmap v974: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:33:11.774 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:33:11.891 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:33:11.985 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:33:11.984+0000 7fbd3f268640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:33:11.986 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:33:11.984+0000 7fbd3f268640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:33:11.997 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:33:11.997 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:33:12.007 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:33:12.007 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:33:12.007 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:33:12.017 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:33:12.017 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:33:12.047 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:33:12.048 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:33:12.190 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1186 8 365281 365273 2.40596 0 - 0.0299513 2026-02-23T13:33:12.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:33:12 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3212446605' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:33:13.190 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1187 8 365281 365273 2.40394 0 - 0.0299513 2026-02-23T13:33:13.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:33:13 trial068 ceph-mon[38805]: pgmap v975: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:33:14.049 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:33:14.164 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:33:14.190 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1188 8 365281 365273 2.40191 0 - 0.0299513 2026-02-23T13:33:14.265 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:33:14.263+0000 7f7facf6b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:33:14.265 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:33:14.264+0000 7f7facf6b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:33:14.265 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:33:13 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:33:13] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:33:14.276 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:33:14.277 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:33:14.287 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:33:14.287 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:33:14.287 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:33:14.297 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:33:14.297 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:33:14.327 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:33:14.328 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:33:14.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:33:13 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:33:13.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:33:14.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:33:14 trial068 ceph-mon[38805]: 5.a scrub starts 2026-02-23T13:33:14.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:33:14 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2895777999' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:33:15.190 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1189 8 365281 365273 2.39989 0 - 0.0299513 2026-02-23T13:33:15.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:33:15 trial068 ceph-mon[38805]: 5.a scrub ok 2026-02-23T13:33:15.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:33:15 trial068 ceph-mon[38805]: pgmap v976: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:33:16.190 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1190 8 365281 365273 2.39788 0 - 0.0299513 2026-02-23T13:33:16.329 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:33:16.448 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:33:16.547 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:33:16.545+0000 7f87a3c93640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:33:16.547 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:33:16.546+0000 7f87a3c93640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:33:16.559 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:33:16.559 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:33:16.569 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:33:16.569 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:33:16.569 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:33:16.579 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:33:16.579 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:33:16.608 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:33:16.609 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:33:17.190 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1191 8 365281 365273 2.39586 0 - 0.0299513 2026-02-23T13:33:17.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:33:17 trial068 ceph-mon[38805]: pgmap v977: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:33:17.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:33:17 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/4194573464' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:33:18.190 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1192 8 365281 365273 2.39385 0 - 0.0299513 2026-02-23T13:33:18.610 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:33:18.729 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:33:18.828 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:33:18.827+0000 7fadd2848640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:33:18.828 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:33:18.827+0000 7fadd2848640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:33:18.840 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:33:18.840 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:33:18.850 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:33:18.850 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:33:18.851 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:33:18.861 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:33:18.861 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:33:18.890 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:33:18.891 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:33:19.191 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1193 8 365281 365273 2.39185 0 - 0.0299513 2026-02-23T13:33:19.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:33:19 trial068 ceph-mon[38805]: pgmap v978: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:33:19.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:33:19 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/794236922' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:33:20.191 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1194 8 365281 365273 2.38984 0 - 0.0299513 2026-02-23T13:33:20.891 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:33:21.010 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:33:21.107 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:33:21.105+0000 7f8a15ddc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:33:21.107 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:33:21.106+0000 7f8a15ddc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:33:21.118 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:33:21.118 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:33:21.128 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:33:21.128 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:33:21.129 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:33:21.138 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:33:21.139 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:33:21.168 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:33:21.169 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:33:21.191 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1195 8 365281 365273 2.38784 0 - 0.0299513 2026-02-23T13:33:21.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:33:21 trial068 ceph-mon[38805]: pgmap v979: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:33:21.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:33:21 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1836183648' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:33:22.191 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1196 8 365281 365273 2.38585 0 - 0.0299513 2026-02-23T13:33:23.170 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:33:23.191 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1197 8 365281 365273 2.38385 0 - 0.0299513 2026-02-23T13:33:23.286 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:33:23.384 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:33:23.383+0000 7f89d7a6e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:33:23.385 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:33:23.383+0000 7f89d7a6e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:33:23.397 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:33:23.397 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:33:23.407 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:33:23.407 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:33:23.407 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:33:23.418 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:33:23.418 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:33:23.448 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:33:23 trial068 ceph-mon[38805]: pgmap v980: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:33:23.449 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:33:23.450 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:33:24.191 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1198 8 365281 365273 2.38186 0 - 0.0299513 2026-02-23T13:33:24.302 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:33:23 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:33:23] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:33:24.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:33:23 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:33:23.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:33:24.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:33:24 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2033024737' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:33:25.191 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1199 8 365281 365273 2.37988 0 - 0.0299513 2026-02-23T13:33:25.451 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:33:25.568 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:33:25.590 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:33:25 trial068 ceph-mon[38805]: pgmap v981: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:33:25.668 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:33:25.666+0000 7f9814c85640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:33:25.668 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:33:25.667+0000 7f9814c85640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:33:25.679 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:33:25.679 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:33:25.689 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:33:25.689 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:33:25.690 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:33:25.699 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:33:25.699 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:33:25.729 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:33:25.729 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:33:26.191 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T13:33:26.191717+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T13:33:26.192 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T13:33:26.192 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1200 8 365281 365273 2.37789 0 - 0.0299513 2026-02-23T13:33:26.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:33:26 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1391632821' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:33:27.191 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1201 8 365281 365273 2.37591 0 - 0.0299513 2026-02-23T13:33:27.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:33:27 trial068 ceph-mon[38805]: pgmap v982: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:33:27.730 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:33:27.845 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:33:27.951 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:33:27.949+0000 7fb1fd3b4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:33:27.951 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:33:27.950+0000 7fb1fd3b4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:33:27.963 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:33:27.963 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:33:27.973 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:33:27.973 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:33:27.974 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:33:27.983 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:33:27.984 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:33:28.013 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:33:28.013 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:33:28.191 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1202 8 365281 365273 2.37394 0 - 0.0299513 2026-02-23T13:33:28.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:33:28 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1095840897' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:33:29.192 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1203 8 365281 365273 2.37196 0 - 0.0299513 2026-02-23T13:33:29.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:33:29 trial068 ceph-mon[38805]: pgmap v983: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:33:30.014 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:33:30.131 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:33:30.191 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1204 8 365281 365273 2.36999 0 - 0.0299513 2026-02-23T13:33:30.235 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:33:30.234+0000 7efda983c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:33:30.236 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:33:30.234+0000 7efda983c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:33:30.247 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:33:30.248 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:33:30.257 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:33:30.258 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:33:30.258 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:33:30.267 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:33:30.268 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:33:30.298 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:33:30.298 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:33:30.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:33:30 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1032955974' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:33:31.192 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1205 8 365281 365273 2.36803 0 - 0.0299513 2026-02-23T13:33:31.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:33:31 trial068 ceph-mon[38805]: pgmap v984: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:33:32.192 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1206 8 365281 365273 2.36606 0 - 0.0299513 2026-02-23T13:33:32.299 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:33:32.416 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:33:32.516 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:33:32.515+0000 7fdd934dc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:33:32.516 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:33:32.515+0000 7fdd934dc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:33:32.528 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:33:32.528 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:33:32.538 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:33:32.538 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:33:32.538 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:33:32.548 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:33:32.548 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:33:32.578 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:33:32.578 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:33:33.192 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1207 8 365281 365273 2.3641 0 - 0.0299513 2026-02-23T13:33:33.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:33:33 trial068 ceph-mon[38805]: pgmap v985: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:33:33.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:33:33 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2419286575' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:33:34.192 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1208 8 365281 365273 2.36215 0 - 0.0299513 2026-02-23T13:33:34.265 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:33:33 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:33:33.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:33:34.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:33:33 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:33:33] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:33:34.579 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:33:34.697 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:33:34.795 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:33:34.794+0000 7f970f2c9640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:33:34.796 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:33:34.794+0000 7f970f2c9640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:33:34.808 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:33:34.808 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:33:34.818 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:33:34.818 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:33:34.818 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:33:34.827 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:33:34.828 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:33:34.858 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:33:34.858 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:33:35.192 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1209 8 365281 365273 2.36019 0 - 0.0299513 2026-02-23T13:33:35.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:33:35 trial068 ceph-mon[38805]: pgmap v986: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:33:35.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:33:35 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2236506125' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:33:36.192 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1210 8 365281 365273 2.35824 0 - 0.0299513 2026-02-23T13:33:36.859 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:33:36.979 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:33:37.082 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:33:37.080+0000 7f2320c29640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:33:37.082 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:33:37.081+0000 7f2320c29640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:33:37.094 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:33:37.094 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:33:37.103 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:33:37.104 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:33:37.104 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:33:37.114 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:33:37.114 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:33:37.145 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:33:37.145 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:33:37.192 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1211 8 365281 365273 2.35629 0 - 0.0299513 2026-02-23T13:33:37.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:33:37 trial068 ceph-mon[38805]: pgmap v987: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:33:37.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:33:37 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1770442017' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:33:38.192 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1212 8 365281 365273 2.35435 0 - 0.0299513 2026-02-23T13:33:38.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:33:38 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2026-02-23T13:33:39.147 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:33:39.192 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1213 8 365281 365273 2.35241 0 - 0.0299513 2026-02-23T13:33:39.264 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:33:39.358 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:33:39.356+0000 7f1c38223640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:33:39.358 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:33:39.357+0000 7f1c38223640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:33:39.370 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:33:39.370 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:33:39.380 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:33:39.380 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:33:39.381 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:33:39.391 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:33:39.392 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:33:39.422 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:33:39.422 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:33:39.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:33:39 trial068 ceph-mon[38805]: pgmap v988: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:33:39.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:33:39 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-23T13:33:39.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:33:39 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2026-02-23T13:33:39.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:33:39 trial068 ceph-mon[38805]: pgmap v989: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:33:39.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:33:39 trial068 ceph-mon[38805]: pgmap v990: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:33:39.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:33:39 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T13:33:39.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:33:39 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2026-02-23T13:33:40.192 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1214 8 365281 365273 2.35047 0 - 0.0299513 2026-02-23T13:33:40.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:33:40 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1569503366' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:33:41.192 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1215 8 365281 365273 2.34854 0 - 0.0299513 2026-02-23T13:33:41.423 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:33:41.540 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:33:41.562 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:33:41 trial068 ceph-mon[38805]: pgmap v991: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:33:41.635 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:33:41.634+0000 7f3248a17640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:33:41.635 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:33:41.634+0000 7f3248a17640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:33:41.647 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:33:41.647 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:33:41.657 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:33:41.657 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:33:41.657 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:33:41.668 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:33:41.668 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:33:41.698 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:33:41.698 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:33:42.192 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1216 8 365281 365273 2.34661 0 - 0.0299513 2026-02-23T13:33:42.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:33:42 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2949420329' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:33:43.192 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1217 8 365281 365273 2.34468 0 - 0.0299513 2026-02-23T13:33:43.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:33:43 trial068 ceph-mon[38805]: pgmap v992: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:33:43.699 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:33:43.816 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:33:43.915 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:33:43.914+0000 7f90f2dd0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:33:43.915 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:33:43.914+0000 7f90f2dd0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:33:43.927 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:33:43.927 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:33:43.937 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:33:43.937 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:33:43.937 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:33:43.947 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:33:43.947 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:33:43.975 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:33:43.975 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:33:44.093 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:33:43 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:33:43] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:33:44.192 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1218 8 365281 365273 2.34275 0 - 0.0299513 2026-02-23T13:33:44.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:33:43 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:33:43.972Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:33:44.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:33:44 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3061169671' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:33:45.193 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1219 8 365281 365273 2.34083 0 - 0.0299513 2026-02-23T13:33:45.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:33:45 trial068 ceph-mon[38805]: pgmap v993: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:33:45.976 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:33:46.092 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:33:46.187 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:33:46.185+0000 7ff95109c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:33:46.187 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:33:46.186+0000 7ff95109c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:33:46.193 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T13:33:46.193136+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T13:33:46.193 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T13:33:46.193 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1220 8 365281 365273 2.33891 0 - 0.0299513 2026-02-23T13:33:46.199 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:33:46.199 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:33:46.209 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:33:46.209 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:33:46.209 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:33:46.219 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:33:46.219 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:33:46.247 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:33:46.248 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:33:46.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:33:46 trial068 ceph-mon[38805]: osdmap e199: 4 total, 3 up, 4 in 2026-02-23T13:33:46.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:33:46 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2622353924' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:33:47.193 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1221 8 365281 365273 2.337 0 - 0.0299513 2026-02-23T13:33:47.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:33:47 trial068 ceph-mon[38805]: pgmap v995: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:33:48.193 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1222 8 365281 365273 2.33508 0 - 0.0299513 2026-02-23T13:33:48.248 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:33:48.368 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:33:48.465 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:33:48.463+0000 7ff33745c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:33:48.465 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:33:48.464+0000 7ff33745c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:33:48.476 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:33:48.477 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:33:48.486 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:33:48.487 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:33:48.487 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:33:48.497 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:33:48.497 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:33:48.524 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:33:48 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3399596850' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:33:48.525 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:33:48.526 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:33:49.193 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1223 8 365281 365273 2.33317 0 - 0.0299513 2026-02-23T13:33:49.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:33:49 trial068 ceph-mon[38805]: pgmap v996: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:33:50.193 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1224 8 365281 365273 2.33127 0 - 0.0299513 2026-02-23T13:33:50.527 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:33:50.646 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:33:50.742 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:33:50.741+0000 7fe17338b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:33:50.742 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:33:50.741+0000 7fe17338b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:33:50.754 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:33:50.754 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:33:50.764 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:33:50.765 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:33:50.765 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:33:50.774 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:33:50.775 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:33:50.805 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:33:50.806 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:33:51.193 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1225 8 365281 365273 2.32937 0 - 0.0299513 2026-02-23T13:33:51.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:33:51 trial068 ceph-mon[38805]: pgmap v997: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:33:51.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:33:51 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2876220387' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:33:52.193 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1226 8 365281 365273 2.32747 0 - 0.0299513 2026-02-23T13:33:52.807 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:33:52.926 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:33:53.027 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:33:53.026+0000 7f10faea5640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:33:53.027 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:33:53.026+0000 7f10faea5640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:33:53.039 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:33:53.040 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:33:53.050 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:33:53.050 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:33:53.050 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:33:53.060 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:33:53.060 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:33:53.092 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:33:53.093 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:33:53.193 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1227 8 365281 365273 2.32557 0 - 0.0299513 2026-02-23T13:33:53.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:33:53 trial068 ceph-mon[38805]: pgmap v998: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:33:53.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:33:53 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3182129972' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:33:54.193 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1228 8 365281 365273 2.32367 0 - 0.0299513 2026-02-23T13:33:54.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:33:53 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:33:53] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:33:54.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:33:53 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:33:53.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:33:54.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:33:54 trial068 ceph-mon[38805]: 5.e scrub starts 2026-02-23T13:33:54.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:33:54 trial068 ceph-mon[38805]: 5.e scrub ok 2026-02-23T13:33:55.094 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:33:55.193 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1229 8 365281 365273 2.32178 0 - 0.0299513 2026-02-23T13:33:55.213 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:33:55.318 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:33:55.317+0000 7fa7a6f78640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:33:55.318 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:33:55.317+0000 7fa7a6f78640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:33:55.330 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:33:55.331 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:33:55.341 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:33:55.341 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:33:55.341 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:33:55.352 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:33:55.352 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:33:55.381 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:33:55.382 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:33:55.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:33:55 trial068 ceph-mon[38805]: pgmap v999: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:33:55.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:33:55 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1161708999' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:33:56.193 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1230 8 365281 365273 2.3199 0 - 0.0299513 2026-02-23T13:33:57.194 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1231 8 365281 365273 2.31801 0 - 0.0299513 2026-02-23T13:33:57.383 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:33:57.499 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:33:57.601 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:33:57.600+0000 7fa17237c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:33:57.601 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:33:57.600+0000 7fa17237c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:33:57.613 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:33:57.614 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:33:57.623 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:33:57.623 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:33:57.624 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:33:57.634 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:33:57.635 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:33:57.663 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:33:57 trial068 ceph-mon[38805]: pgmap v1000: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:33:57.664 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:33:57.665 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:33:58.194 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1232 8 365281 365273 2.31613 0 - 0.0299513 2026-02-23T13:33:58.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:33:58 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2575204884' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:33:59.194 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1233 8 365281 365273 2.31425 0 - 0.0299513 2026-02-23T13:33:59.666 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:33:59.782 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:33:59.829 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:33:59 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T13:33:59.829 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:33:59 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T13:33:59.829 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:33:59 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T13:33:59.829 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:33:59 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T13:33:59.829 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:33:59 trial068 ceph-mon[38805]: pgmap v1001: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:33:59.886 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:33:59.885+0000 7f5729b6c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:33:59.886 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:33:59.885+0000 7f5729b6c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:33:59.898 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:33:59.899 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:33:59.908 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:33:59.909 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:33:59.909 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:33:59.919 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:33:59.919 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:33:59.949 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:33:59.950 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:34:00.194 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1234 8 365281 365273 2.31238 0 - 0.0299513 2026-02-23T13:34:00.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:34:00 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1041601061' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:34:01.194 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1235 8 365281 365273 2.3105 0 - 0.0299513 2026-02-23T13:34:01.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:34:01 trial068 ceph-mon[38805]: pgmap v1002: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:34:01.951 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:34:02.070 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:34:02.172 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:34:02.170+0000 7f3181a80640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:34:02.172 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:34:02.170+0000 7f3181a80640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:34:02.185 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:34:02.185 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:34:02.194 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1236 8 365281 365273 2.30863 0 - 0.0299513 2026-02-23T13:34:02.195 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:34:02.196 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:34:02.196 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:34:02.205 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:34:02.205 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:34:02.235 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:34:02.235 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:34:02.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:34:02 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/203952272' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:34:03.194 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1237 8 365281 365273 2.30677 0 - 0.0299513 2026-02-23T13:34:03.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:34:03 trial068 ceph-mon[38805]: pgmap v1003: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:34:04.194 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1238 8 365281 365273 2.30491 0 - 0.0299513 2026-02-23T13:34:04.237 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:34:04.260 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:34:03 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:34:03] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:34:04.265 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:34:03 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:34:03.974Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:34:04.354 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:34:04.452 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:34:04.451+0000 7fa4c7e42640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:34:04.452 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:34:04.451+0000 7fa4c7e42640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:34:04.464 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:34:04.464 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:34:04.474 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:34:04.475 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:34:04.475 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:34:04.484 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:34:04.485 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:34:04.514 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:34:04.515 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:34:05.194 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1239 8 365281 365273 2.30304 0 - 0.0299513 2026-02-23T13:34:05.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:34:05 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2376136934' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:34:05.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:34:05 trial068 ceph-mon[38805]: pgmap v1004: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:34:06.194 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T13:34:06.194674+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T13:34:06.194 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T13:34:06.195 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1240 8 365281 365273 2.30119 0 - 0.0299513 2026-02-23T13:34:06.516 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:34:06.635 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:34:06.741 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:34:06.739+0000 7f6dff089640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:34:06.741 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:34:06.740+0000 7f6dff089640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:34:06.752 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:34:06.753 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:34:06.763 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:34:06.763 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:34:06.763 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:34:06.773 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:34:06.773 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:34:06.804 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:34:06.805 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:34:07.194 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1241 8 365281 365273 2.29933 0 - 0.0299513 2026-02-23T13:34:07.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:34:07 trial068 ceph-mon[38805]: pgmap v1005: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:34:07.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:34:07 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1416740510' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:34:08.194 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1242 8 365281 365273 2.29748 0 - 0.0299513 2026-02-23T13:34:08.806 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:34:08.923 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:34:09.020 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:34:09.018+0000 7f286c5e8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:34:09.020 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:34:09.019+0000 7f286c5e8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:34:09.032 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:34:09.032 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:34:09.042 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:34:09.042 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:34:09.043 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:34:09.052 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:34:09.052 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:34:09.082 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:34:09.082 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:34:09.194 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1243 8 365281 365273 2.29563 0 - 0.0299513 2026-02-23T13:34:09.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:34:09 trial068 ceph-mon[38805]: pgmap v1006: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:34:09.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:34:09 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2268075036' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:34:10.195 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1244 8 365281 365273 2.29379 0 - 0.0299513 2026-02-23T13:34:11.084 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:34:11.195 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1245 8 365281 365273 2.29195 0 - 0.0299513 2026-02-23T13:34:11.201 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:34:11.297 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:34:11.296+0000 7fe8594ca640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:34:11.297 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:34:11.296+0000 7fe8594ca640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:34:11.309 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:34:11.310 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:34:11.320 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:34:11.320 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:34:11.320 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:34:11.331 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:34:11.332 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:34:11.362 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:34:11.362 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:34:11.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:34:11 trial068 ceph-mon[38805]: pgmap v1007: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:34:11.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:34:11 trial068 ceph-mon[38805]: 5.7 scrub starts 2026-02-23T13:34:11.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:34:11 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/4220110386' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:34:12.195 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1246 8 365281 365273 2.29011 0 - 0.0299513 2026-02-23T13:34:12.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:34:12 trial068 ceph-mon[38805]: 5.7 scrub ok 2026-02-23T13:34:13.195 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1247 8 365281 365273 2.28827 0 - 0.0299513 2026-02-23T13:34:13.363 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:34:13.483 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:34:13.584 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:34:13.582+0000 7fa97d22e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:34:13.584 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:34:13.582+0000 7fa97d22e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:34:13.595 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:34:13.596 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:34:13.605 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:34:13.606 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:34:13.606 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:34:13.616 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:34:13.616 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:34:13.644 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:34:13 trial068 ceph-mon[38805]: pgmap v1008: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:34:13.645 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:34:13.646 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:34:14.195 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1248 8 365281 365273 2.28644 0 - 0.0299513 2026-02-23T13:34:14.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:34:13 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:34:13] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:34:14.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:34:13 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:34:13.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:34:14.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:34:14 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2238693677' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:34:15.195 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1249 8 365281 365273 2.28461 0 - 0.0299513 2026-02-23T13:34:15.647 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:34:15.765 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:34:15.823 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:34:15 trial068 ceph-mon[38805]: pgmap v1009: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:34:15.866 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:34:15.865+0000 7f84e59f8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:34:15.866 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:34:15.865+0000 7f84e59f8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:34:15.878 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:34:15.878 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:34:15.888 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:34:15.888 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:34:15.889 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:34:15.898 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:34:15.898 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:34:15.925 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:34:15.926 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:34:16.195 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1250 8 365281 365273 2.28278 0 - 0.0299513 2026-02-23T13:34:16.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:34:16 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2091324452' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:34:17.195 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1251 8 365281 365273 2.28095 0 - 0.0299513 2026-02-23T13:34:17.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:34:17 trial068 ceph-mon[38805]: pgmap v1010: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:34:17.928 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:34:18.047 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:34:18.148 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:34:18.146+0000 7f2f8398a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:34:18.148 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:34:18.147+0000 7f2f8398a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:34:18.160 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:34:18.160 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:34:18.170 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:34:18.170 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:34:18.170 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:34:18.180 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:34:18.180 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:34:18.195 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1252 8 365281 365273 2.27913 0 - 0.0299513 2026-02-23T13:34:18.208 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:34:18.209 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:34:18.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:34:18 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2397376414' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:34:19.196 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1253 8 365281 365273 2.27731 0 - 0.0299513 2026-02-23T13:34:19.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:34:19 trial068 ceph-mon[38805]: pgmap v1011: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:34:20.196 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1254 8 365281 365273 2.2755 0 - 0.0299513 2026-02-23T13:34:20.210 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:34:20.327 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:34:20.424 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:34:20.422+0000 7ff31fa87640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:34:20.424 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:34:20.423+0000 7ff31fa87640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:34:20.435 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:34:20.436 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:34:20.446 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:34:20.446 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:34:20.446 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:34:20.457 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:34:20.457 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:34:20.489 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:34:20.489 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:34:20.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:34:20 trial068 ceph-mon[38805]: 5.a scrub starts 2026-02-23T13:34:20.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:34:20 trial068 ceph-mon[38805]: 5.a scrub ok 2026-02-23T13:34:20.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:34:20 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2794413461' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:34:21.196 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1255 8 365281 365273 2.27368 0 - 0.0299513 2026-02-23T13:34:21.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:34:21 trial068 ceph-mon[38805]: pgmap v1012: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:34:22.196 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1256 8 365281 365273 2.27187 0 - 0.0299513 2026-02-23T13:34:22.491 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:34:22.609 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:34:22.706 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:34:22.705+0000 7fe3c48ff640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:34:22.706 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:34:22.705+0000 7fe3c48ff640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:34:22.718 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:34:22.718 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:34:22.728 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:34:22.729 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:34:22.729 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:34:22.739 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:34:22.739 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:34:22.784 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:34:22.785 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:34:23.196 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1257 8 365281 365273 2.27007 0 - 0.0299513 2026-02-23T13:34:23.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:34:23 trial068 ceph-mon[38805]: pgmap v1013: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:34:23.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:34:23 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2660318361' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:34:24.196 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1258 8 365281 365273 2.26826 0 - 0.0299513 2026-02-23T13:34:24.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:34:23 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:34:23] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:34:24.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:34:23 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:34:23.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:34:24.786 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:34:24.904 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:34:25.004 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:34:25.002+0000 7f395ee75640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:34:25.004 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:34:25.003+0000 7f395ee75640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:34:25.016 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:34:25.016 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:34:25.026 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:34:25.026 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:34:25.027 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:34:25.036 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:34:25.036 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:34:25.064 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:34:25.065 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:34:25.196 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1259 8 365281 365273 2.26646 0 - 0.0299513 2026-02-23T13:34:25.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:34:25 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T13:34:25.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:34:25 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T13:34:25.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:34:25 trial068 ceph-mon[38805]: pgmap v1014: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:34:25.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:34:25 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1492685505' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:34:26.196 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T13:34:26.196442+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T13:34:26.196 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T13:34:26.196 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1260 8 365281 365273 2.26466 0 - 0.0299513 2026-02-23T13:34:27.066 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:34:27.185 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:34:27.196 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1261 8 365281 365273 2.26286 0 - 0.0299513 2026-02-23T13:34:27.288 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:34:27.287+0000 7f14fcef2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:34:27.289 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:34:27.287+0000 7f14fcef2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:34:27.300 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:34:27.301 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:34:27.310 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:34:27.311 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:34:27.311 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:34:27.321 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:34:27.321 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:34:27.350 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:34:27.350 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:34:27.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:34:27 trial068 ceph-mon[38805]: pgmap v1015: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:34:27.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:34:27 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3361557735' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:34:28.196 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1262 8 365281 365273 2.26107 0 - 0.0299513 2026-02-23T13:34:28.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:34:28 trial068 ceph-mon[38805]: 5.5 scrub starts 2026-02-23T13:34:28.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:34:28 trial068 ceph-mon[38805]: 5.5 scrub ok 2026-02-23T13:34:29.196 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1263 8 365281 365273 2.25928 0 - 0.0299513 2026-02-23T13:34:29.351 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:34:29.470 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:34:29.571 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:34:29.570+0000 7f2f5ac65640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:34:29.572 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:34:29.571+0000 7f2f5ac65640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:34:29.584 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:34:29.584 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:34:29.594 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:34:29.594 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:34:29.595 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:34:29.604 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:34:29.605 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:34:29.631 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:34:29 trial068 ceph-mon[38805]: 5.8 scrub starts 2026-02-23T13:34:29.631 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:34:29 trial068 ceph-mon[38805]: 5.8 scrub ok 2026-02-23T13:34:29.632 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:34:29 trial068 ceph-mon[38805]: pgmap v1016: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:34:29.632 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:34:29 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3009612288' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:34:29.632 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:34:29.632 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:34:30.196 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1264 8 365281 365273 2.25749 0 - 0.0299513 2026-02-23T13:34:31.196 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1265 8 365281 365273 2.25571 0 - 0.0299513 2026-02-23T13:34:31.633 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:34:31.748 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:34:31.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:34:31 trial068 ceph-mon[38805]: pgmap v1017: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:34:31.844 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:34:31.843+0000 7f674f558640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:34:31.845 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:34:31.843+0000 7f674f558640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:34:31.856 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:34:31.856 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:34:31.866 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:34:31.867 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:34:31.867 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:34:31.877 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:34:31.877 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:34:31.906 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:34:31.907 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:34:32.196 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1266 8 365281 365273 2.25393 0 - 0.0299513 2026-02-23T13:34:32.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:34:32 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1369069112' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:34:33.196 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1267 8 365281 365273 2.25215 0 - 0.0299513 2026-02-23T13:34:33.908 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:34:33.933 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:34:33 trial068 ceph-mon[38805]: pgmap v1018: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:34:34.026 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:34:34.123 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:34:34.122+0000 7fd3e4cc3640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:34:34.123 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:34:34.123+0000 7fd3e4cc3640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:34:34.136 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:34:34.136 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:34:34.146 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:34:34.146 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:34:34.146 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:34:34.156 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:34:34.156 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:34:34.185 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:34:33 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:34:33] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:34:34.186 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:34:34.186 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:34:34.196 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1268 8 365281 365273 2.25037 0 - 0.0299513 2026-02-23T13:34:34.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:34:33 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:34:33.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:34:35.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:34:34 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1289339828' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:34:35.196 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1269 8 365281 365273 2.2486 0 - 0.0299513 2026-02-23T13:34:36.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:34:35 trial068 ceph-mon[38805]: pgmap v1019: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:34:36.187 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:34:36.196 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1270 8 365281 365273 2.24683 0 - 0.0299513 2026-02-23T13:34:36.305 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:34:36.412 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:34:36.411+0000 7f01d69b4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:34:36.412 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:34:36.411+0000 7f01d69b4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:34:36.424 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:34:36.424 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:34:36.434 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:34:36.434 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:34:36.434 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:34:36.444 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:34:36.444 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:34:36.475 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:34:36.475 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:34:36.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:34:36 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3600370697' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:34:37.196 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1271 8 365281 365273 2.24506 0 - 0.0299513 2026-02-23T13:34:38.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:34:37 trial068 ceph-mon[38805]: pgmap v1020: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:34:38.197 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1272 8 365281 365273 2.2433 0 - 0.0299513 2026-02-23T13:34:38.476 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:34:38.601 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:34:38.699 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:34:38.698+0000 7ff05d207640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:34:38.699 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:34:38.699+0000 7ff05d207640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:34:38.712 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:34:38.712 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:34:38.722 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:34:38.722 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:34:38.722 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:34:38.733 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:34:38.733 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:34:38.763 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:34:38 trial068 ceph-mon[38805]: pgmap v1021: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:34:38.763 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:34:38 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2026-02-23T13:34:38.764 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:34:38.765 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:34:39.197 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1273 8 365281 365273 2.24153 0 - 0.0299513 2026-02-23T13:34:40.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:34:39 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/690438508' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:34:40.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:34:39 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-23T13:34:40.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:34:39 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2026-02-23T13:34:40.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:34:39 trial068 ceph-mon[38805]: pgmap v1022: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:34:40.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:34:39 trial068 ceph-mon[38805]: pgmap v1023: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:34:40.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:34:39 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T13:34:40.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:34:39 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2026-02-23T13:34:40.197 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1274 8 365281 365273 2.23977 0 - 0.0299513 2026-02-23T13:34:40.766 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:34:40.884 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:34:40.985 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:34:40.984+0000 7f89736fe640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:34:40.985 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:34:40.984+0000 7f89736fe640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:34:40.998 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:34:40.998 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:34:41.008 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:34:41.009 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:34:41.009 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:34:41.019 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:34:41.020 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:34:41.051 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:34:41.051 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:34:41.197 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1275 8 365281 365273 2.23802 0 - 0.0299513 2026-02-23T13:34:42.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:34:41 trial068 ceph-mon[38805]: pgmap v1024: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:34:42.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:34:41 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2709234652' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:34:42.197 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1276 8 365281 365273 2.23626 0 - 0.0299513 2026-02-23T13:34:43.052 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:34:43.171 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:34:43.197 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1277 8 365281 365273 2.23451 0 - 0.0299513 2026-02-23T13:34:43.265 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:34:43.263+0000 7f23ac381640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:34:43.265 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:34:43.264+0000 7f23ac381640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:34:43.278 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:34:43.278 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:34:43.288 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:34:43.289 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:34:43.289 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:34:43.300 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:34:43.300 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:34:43.331 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:34:43.331 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:34:44.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:34:43 trial068 ceph-mon[38805]: pgmap v1025: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:34:44.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:34:43 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/4083466608' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:34:44.094 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:34:43 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:34:43] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:34:44.197 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1278 8 365281 365273 2.23276 0 - 0.0299513 2026-02-23T13:34:44.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:34:43 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:34:43.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:34:45.198 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1279 8 365281 365273 2.23102 0 - 0.0299513 2026-02-23T13:34:45.332 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:34:45.449 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:34:45.551 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:34:45.550+0000 7fc3d1fd9640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:34:45.551 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:34:45.550+0000 7fc3d1fd9640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:34:45.562 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:34:45.563 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:34:45.573 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:34:45.573 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:34:45.573 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:34:45.583 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:34:45.583 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:34:45.613 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:34:45.614 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:34:46.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:34:45 trial068 ceph-mon[38805]: pgmap v1026: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:34:46.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:34:45 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1034313309' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:34:46.198 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T13:34:46.198152+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T13:34:46.198 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T13:34:46.198 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1280 8 365281 365273 2.22927 0 - 0.0299513 2026-02-23T13:34:47.198 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1281 8 365281 365273 2.22753 0 - 0.0299513 2026-02-23T13:34:47.616 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:34:47.734 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:34:47.834 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:34:47.833+0000 7f1b3916a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:34:47.834 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:34:47.833+0000 7f1b3916a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:34:47.847 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:34:47.847 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:34:47.857 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:34:47.858 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:34:47.858 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:34:47.868 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:34:47.868 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:34:47.895 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:34:47 trial068 ceph-mon[38805]: pgmap v1027: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:34:47.896 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:34:47.897 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:34:48.198 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1282 8 365281 365273 2.2258 0 - 0.0299513 2026-02-23T13:34:49.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:34:48 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/4249657309' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:34:49.198 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1283 8 365281 365273 2.22406 0 - 0.0299513 2026-02-23T13:34:49.898 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:34:50.019 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:34:50.114 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:34:50.112+0000 7fc25f1bc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:34:50.114 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:34:50.112+0000 7fc25f1bc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:34:50.126 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:34:50.126 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:34:50.136 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:34:50.136 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:34:50.136 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:34:50.146 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:34:50.146 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:34:50.176 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:34:50.177 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:34:50.198 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1284 8 365281 365273 2.22233 0 - 0.0299513 2026-02-23T13:34:50.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:34:50 trial068 ceph-mon[38805]: pgmap v1028: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:34:50.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:34:50 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2719110860' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:34:51.198 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1285 8 365281 365273 2.2206 0 - 0.0299513 2026-02-23T13:34:52.178 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:34:52.198 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1286 8 365281 365273 2.21887 0 - 0.0299513 2026-02-23T13:34:52.296 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:34:52.404 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:34:52.403+0000 7f16e20b2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:34:52.404 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:34:52.403+0000 7f16e20b2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:34:52.416 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:34:52.416 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:34:52.426 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:34:52.426 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:34:52.427 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:34:52.436 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:34:52.437 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:34:52.437 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:34:52 trial068 ceph-mon[38805]: pgmap v1029: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:34:52.467 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:34:52.468 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:34:53.198 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1287 8 365281 365273 2.21715 0 - 0.0299513 2026-02-23T13:34:53.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:34:53 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1185458106' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:34:54.198 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1288 8 365281 365273 2.21543 0 - 0.0299513 2026-02-23T13:34:54.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:34:54 trial068 ceph-mon[38805]: pgmap v1030: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:34:54.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:34:53 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:34:53] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:34:54.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:34:53 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:34:53.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:34:54.469 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:34:54.587 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:34:54.687 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:34:54.685+0000 7f8b88465640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:34:54.687 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:34:54.685+0000 7f8b88465640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:34:54.698 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:34:54.699 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:34:54.709 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:34:54.709 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:34:54.709 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:34:54.719 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:34:54.719 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:34:54.746 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:34:54.747 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:34:55.198 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1289 8 365281 365273 2.21371 0 - 0.0299513 2026-02-23T13:34:55.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:34:55 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3622074229' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:34:56.198 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1290 8 365281 365273 2.21199 0 - 0.0299513 2026-02-23T13:34:56.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:34:56 trial068 ceph-mon[38805]: pgmap v1031: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:34:56.749 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:34:56.866 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:34:56.963 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:34:56.961+0000 7f387bcfd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:34:56.963 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:34:56.961+0000 7f387bcfd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:34:56.975 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:34:56.976 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:34:56.986 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:34:56.986 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:34:56.986 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:34:56.996 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:34:56.996 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:34:57.029 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:34:57.029 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:34:57.199 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1291 8 365281 365273 2.21028 0 - 0.0299513 2026-02-23T13:34:57.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:34:57 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3492549867' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:34:58.199 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1292 8 365281 365273 2.20857 0 - 0.0299513 2026-02-23T13:34:58.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:34:58 trial068 ceph-mon[38805]: pgmap v1032: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:34:59.031 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:34:59.151 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:34:59.199 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1293 8 365281 365273 2.20686 0 - 0.0299513 2026-02-23T13:34:59.248 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:34:59.247+0000 7f43836ef640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:34:59.249 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:34:59.247+0000 7f43836ef640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:34:59.260 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:34:59.260 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:34:59.270 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:34:59.270 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:34:59.271 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:34:59.280 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:34:59.281 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:34:59.310 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:34:59.311 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:35:00.199 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1294 8 365281 365273 2.20516 0 - 0.0299513 2026-02-23T13:35:00.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:35:00 trial068 ceph-mon[38805]: pgmap v1033: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:35:00.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:35:00 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/682035020' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:35:01.199 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1295 8 365281 365273 2.20345 0 - 0.0299513 2026-02-23T13:35:01.312 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:35:01.432 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:35:01.536 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:35:01.535+0000 7f6ed2e39640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:35:01.537 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:35:01.535+0000 7f6ed2e39640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:35:01.549 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:35:01.550 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:35:01.559 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:35:01.560 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:35:01.560 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:35:01.570 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:35:01.571 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:35:01.598 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:35:01.598 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:35:02.199 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1296 8 365281 365273 2.20175 0 - 0.0299513 2026-02-23T13:35:02.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:35:02 trial068 ceph-mon[38805]: pgmap v1034: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:35:02.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:35:02 trial068 ceph-mon[38805]: osdmap e200: 4 total, 3 up, 4 in 2026-02-23T13:35:02.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:35:02 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/194524041' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:35:03.199 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1297 8 365281 365273 2.20006 0 - 0.0299513 2026-02-23T13:35:03.599 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:35:03.718 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:35:03.817 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:35:03.815+0000 7f9c766d1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:35:03.817 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:35:03.816+0000 7f9c766d1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:35:03.831 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:35:03.831 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:35:03.841 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:35:03.841 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:35:03.841 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:35:03.852 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:35:03.852 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:35:03.882 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:35:03.883 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:35:04.199 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1298 8 365281 365273 2.19836 0 - 0.0299513 2026-02-23T13:35:04.200 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:35:03 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:35:03] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:35:04.265 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:35:03 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:35:03.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:35:04.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:35:04 trial068 ceph-mon[38805]: pgmap v1036: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:35:04.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:35:04 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/884523351' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:35:05.199 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1299 8 365281 365273 2.19667 0 - 0.0299513 2026-02-23T13:35:05.884 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:35:06.003 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:35:06.103 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:35:06.102+0000 7fd8dddca640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:35:06.103 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:35:06.102+0000 7fd8dddca640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:35:06.116 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:35:06.116 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:35:06.126 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:35:06.126 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:35:06.127 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:35:06.136 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:35:06.136 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:35:06.166 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:35:06.167 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:35:06.199 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T13:35:06.199657+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T13:35:06.200 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T13:35:06.200 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1300 8 365281 365273 2.19498 0 - 0.0299513 2026-02-23T13:35:06.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:35:06 trial068 ceph-mon[38805]: pgmap v1037: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:35:06.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:35:06 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2870311294' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:35:07.199 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1301 8 365281 365273 2.19329 0 - 0.0299513 2026-02-23T13:35:08.168 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:35:08.200 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1302 8 365281 365273 2.19161 0 - 0.0299513 2026-02-23T13:35:08.285 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:35:08.385 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:35:08.383+0000 7f23fe23d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:35:08.385 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:35:08.383+0000 7f23fe23d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:35:08.397 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:35:08.397 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:35:08.407 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:35:08.407 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:35:08.407 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:35:08.417 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:35:08.417 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:35:08.445 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:35:08 trial068 ceph-mon[38805]: pgmap v1038: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:35:08.446 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:35:08.447 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:35:09.200 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1303 8 365281 365273 2.18992 0 - 0.0299513 2026-02-23T13:35:09.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:35:09 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3065628043' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:35:10.200 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1304 8 365281 365273 2.18824 0 - 0.0299513 2026-02-23T13:35:10.448 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:35:10.473 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:35:10 trial068 ceph-mon[38805]: pgmap v1039: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:35:10.566 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:35:10.662 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:35:10.661+0000 7f4fe52c8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:35:10.662 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:35:10.661+0000 7f4fe52c8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:35:10.674 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:35:10.674 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:35:10.684 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:35:10.684 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:35:10.684 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:35:10.694 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:35:10.694 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:35:10.725 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:35:10.726 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:35:11.200 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1305 8 365281 365273 2.18657 0 - 0.0299513 2026-02-23T13:35:11.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:35:11 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1740447394' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:35:12.200 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1306 8 365281 365273 2.18489 0 - 0.0299513 2026-02-23T13:35:12.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:35:12 trial068 ceph-mon[38805]: pgmap v1040: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:35:12.727 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:35:12.847 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:35:12.951 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:35:12.949+0000 7f0202a08640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:35:12.951 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:35:12.950+0000 7f0202a08640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:35:12.963 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:35:12.963 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:35:12.973 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:35:12.973 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:35:12.973 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:35:12.983 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:35:12.983 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:35:13.013 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:35:13.014 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:35:13.200 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1307 8 365281 365273 2.18322 0 - 0.0299513 2026-02-23T13:35:13.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:35:13 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/224610051' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:35:14.200 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1308 8 365281 365273 2.18155 0 - 0.0299513 2026-02-23T13:35:14.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:35:14 trial068 ceph-mon[38805]: pgmap v1041: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:35:14.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:35:13 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:35:13] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:35:14.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:35:13 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:35:13.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:35:15.015 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:35:15.132 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:35:15.201 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1309 8 365281 365273 2.17989 0 - 0.0299513 2026-02-23T13:35:15.235 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:35:15.233+0000 7fc3ae94d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:35:15.235 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:35:15.233+0000 7fc3ae94d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:35:15.247 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:35:15.247 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:35:15.256 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:35:15.257 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:35:15.257 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:35:15.267 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:35:15.267 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:35:15.297 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:35:15.298 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:35:16.201 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1310 8 365281 365273 2.17822 0 - 0.0299513 2026-02-23T13:35:16.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:35:16 trial068 ceph-mon[38805]: pgmap v1042: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:35:16.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:35:16 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T13:35:16.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:35:16 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T13:35:16.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:35:16 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3752847926' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:35:17.201 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1311 8 365281 365273 2.17656 0 - 0.0299513 2026-02-23T13:35:17.299 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:35:17.416 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:35:17.516 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:35:17.514+0000 7fbaf89ce640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:35:17.516 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:35:17.515+0000 7fbaf89ce640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:35:17.528 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:35:17.528 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:35:17.538 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:35:17.538 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:35:17.538 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:35:17.548 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:35:17.548 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:35:17.576 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:35:17.577 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:35:18.201 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1312 8 365281 365273 2.1749 0 - 0.0299513 2026-02-23T13:35:18.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:35:18 trial068 ceph-mon[38805]: pgmap v1043: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:35:18.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:35:18 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3742256315' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:35:19.201 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1313 8 365281 365273 2.17325 0 - 0.0299513 2026-02-23T13:35:19.578 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:35:19.694 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:35:19.794 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:35:19.793+0000 7fde0bc03640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:35:19.794 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:35:19.793+0000 7fde0bc03640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:35:19.806 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:35:19.806 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:35:19.816 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:35:19.816 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:35:19.816 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:35:19.826 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:35:19.826 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:35:19.855 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:35:19.856 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:35:20.201 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1314 8 365281 365273 2.17159 0 - 0.0299513 2026-02-23T13:35:20.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:35:20 trial068 ceph-mon[38805]: pgmap v1044: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:35:20.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:35:20 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/219753822' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:35:21.201 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1315 8 365281 365273 2.16994 0 - 0.0299513 2026-02-23T13:35:21.857 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:35:21.973 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:35:22.069 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:35:22.067+0000 7f46cc442640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:35:22.069 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:35:22.067+0000 7f46cc442640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:35:22.081 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:35:22.081 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:35:22.091 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:35:22.092 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:35:22.092 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:35:22.102 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:35:22.102 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:35:22.131 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:35:22.132 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:35:22.201 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1316 8 365281 365273 2.16829 0 - 0.0299513 2026-02-23T13:35:22.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:35:22 trial068 ceph-mon[38805]: pgmap v1045: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:35:22.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:35:22 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/4015848283' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:35:23.201 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1317 8 365281 365273 2.16664 0 - 0.0299513 2026-02-23T13:35:24.133 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:35:24.201 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1318 8 365281 365273 2.165 0 - 0.0299513 2026-02-23T13:35:24.252 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:35:24.274 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:35:23 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:35:23] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:35:24.274 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:35:24 trial068 ceph-mon[38805]: pgmap v1046: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:35:24.274 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:35:24 trial068 ceph-mon[38805]: 5.e scrub starts 2026-02-23T13:35:24.274 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:35:24 trial068 ceph-mon[38805]: 5.e scrub ok 2026-02-23T13:35:24.350 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:35:24.349+0000 7f1cd4668640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:35:24.351 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:35:24.349+0000 7f1cd4668640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:35:24.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:35:23 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:35:23.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:35:24.362 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:35:24.362 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:35:24.372 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:35:24.373 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:35:24.373 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:35:24.383 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:35:24.383 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:35:24.428 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:35:24.429 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:35:25.202 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1319 8 365281 365273 2.16336 0 - 0.0299513 2026-02-23T13:35:25.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:35:25 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3648950445' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:35:26.201 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T13:35:26.201716+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T13:35:26.201 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T13:35:26.202 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1320 8 365281 365273 2.16172 0 - 0.0299513 2026-02-23T13:35:26.430 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:35:26.547 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:35:26.570 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:35:26 trial068 ceph-mon[38805]: pgmap v1047: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:35:26.646 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:35:26.644+0000 7f60569f8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:35:26.646 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:35:26.644+0000 7f60569f8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:35:26.658 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:35:26.658 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:35:26.668 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:35:26.669 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:35:26.669 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:35:26.679 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:35:26.679 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:35:26.709 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:35:26.710 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:35:27.202 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1321 8 365281 365273 2.16008 0 - 0.0299513 2026-02-23T13:35:27.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:35:27 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/805127212' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:35:28.202 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1322 8 365281 365273 2.15845 0 - 0.0299513 2026-02-23T13:35:28.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:35:28 trial068 ceph-mon[38805]: pgmap v1048: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:35:28.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:35:28 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T13:35:28.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:35:28 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T13:35:28.711 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:35:28.830 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:35:28.928 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:35:28.927+0000 7f0aa56b9640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:35:28.929 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:35:28.927+0000 7f0aa56b9640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:35:28.941 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:35:28.941 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:35:28.951 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:35:28.952 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:35:28.952 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:35:28.961 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:35:28.962 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:35:28.991 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:35:28.992 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:35:29.202 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1323 8 365281 365273 2.15682 0 - 0.0299513 2026-02-23T13:35:29.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:35:29 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2178607965' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:35:30.202 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1324 8 365281 365273 2.15519 0 - 0.0299513 2026-02-23T13:35:30.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:35:30 trial068 ceph-mon[38805]: pgmap v1049: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:35:30.993 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:35:31.111 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:35:31.202 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1325 8 365281 365273 2.15356 0 - 0.0299513 2026-02-23T13:35:31.208 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:35:31.206+0000 7ff32c09a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:35:31.208 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:35:31.206+0000 7ff32c09a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:35:31.220 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:35:31.220 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:35:31.230 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:35:31.231 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:35:31.231 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:35:31.241 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:35:31.242 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:35:31.269 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:35:31 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1171417249' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:35:31.270 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:35:31.270 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:35:32.202 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1326 8 365281 365273 2.15194 0 - 0.0299513 2026-02-23T13:35:32.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:35:32 trial068 ceph-mon[38805]: pgmap v1050: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:35:32.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:35:32 trial068 ceph-mon[38805]: 5.a scrub starts 2026-02-23T13:35:32.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:35:32 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T13:35:32.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:35:32 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T13:35:33.202 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1327 8 365281 365273 2.15032 0 - 0.0299513 2026-02-23T13:35:33.271 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:35:33.391 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:35:33.500 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:35:33.498+0000 7fdd04bf1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:35:33.500 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:35:33.498+0000 7fdd04bf1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:35:33.500 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:35:33 trial068 ceph-mon[38805]: 5.a scrub ok 2026-02-23T13:35:33.512 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:35:33.512 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:35:33.523 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:35:33.523 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:35:33.524 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:35:33.533 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:35:33.534 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:35:33.564 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:35:33.564 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:35:34.203 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1328 8 365281 365273 2.1487 0 - 0.0299513 2026-02-23T13:35:34.234 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:35:33 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:35:33] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:35:34.265 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:35:33 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:35:33.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:35:34.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:35:34 trial068 ceph-mon[38805]: pgmap v1051: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:35:34.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:35:34 trial068 ceph-mon[38805]: 5.7 scrub starts 2026-02-23T13:35:34.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:35:34 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3112317032' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:35:34.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:35:34 trial068 ceph-mon[38805]: 5.7 scrub ok 2026-02-23T13:35:35.202 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1329 8 365281 365273 2.14708 0 - 0.0299513 2026-02-23T13:35:35.565 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:35:35.686 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:35:35.789 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:35:35.787+0000 7fc8b3087640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:35:35.789 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:35:35.787+0000 7fc8b3087640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:35:35.800 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:35:35.801 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:35:35.811 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:35:35.811 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:35:35.811 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:35:35.822 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:35:35.822 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:35:35.852 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:35:35.852 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:35:36.203 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1330 8 365281 365273 2.14547 0 - 0.0299513 2026-02-23T13:35:36.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:35:36 trial068 ceph-mon[38805]: pgmap v1052: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:35:36.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:35:36 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2612719020' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:35:37.203 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1331 8 365281 365273 2.14385 0 - 0.0299513 2026-02-23T13:35:37.854 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:35:37.971 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:35:38.073 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:35:38.071+0000 7fd2b70a7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:35:38.073 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:35:38.071+0000 7fd2b70a7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:35:38.085 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:35:38.086 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:35:38.096 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:35:38.096 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:35:38.096 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:35:38.105 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:35:38.106 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:35:38.136 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:35:38.136 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:35:38.203 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1332 8 365281 365273 2.14225 0 - 0.0299513 2026-02-23T13:35:38.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:35:38 trial068 ceph-mon[38805]: pgmap v1053: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:35:38.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:35:38 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1925028672' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:35:39.203 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1333 8 365281 365273 2.14064 0 - 0.0299513 2026-02-23T13:35:39.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:35:39 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2026-02-23T13:35:39.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:35:39 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-23T13:35:39.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:35:39 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2026-02-23T13:35:39.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:35:39 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T13:35:39.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:35:39 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2026-02-23T13:35:40.137 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:35:40.203 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1334 8 365281 365273 2.13903 0 - 0.0299513 2026-02-23T13:35:40.256 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:35:40.355 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:35:40.354+0000 7fdd28b5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:35:40.356 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:35:40.354+0000 7fdd28b5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:35:40.368 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:35:40.368 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:35:40.378 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:35:40.379 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:35:40.379 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:35:40.390 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:35:40.391 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:35:40.420 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:35:40 trial068 ceph-mon[38805]: pgmap v1054: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:35:40.420 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:35:40 trial068 ceph-mon[38805]: pgmap v1055: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:35:40.420 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:35:40 trial068 ceph-mon[38805]: pgmap v1056: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:35:40.421 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:35:40.421 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:35:41.203 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1335 8 365281 365273 2.13743 0 - 0.0299513 2026-02-23T13:35:41.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:35:41 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1491518021' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:35:42.203 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1336 8 365281 365273 2.13583 0 - 0.0299513 2026-02-23T13:35:42.422 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:35:42.543 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:35:42.565 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:35:42 trial068 ceph-mon[38805]: pgmap v1057: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:35:42.640 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:35:42.638+0000 7f2046392640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:35:42.640 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:35:42.638+0000 7f2046392640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:35:42.652 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:35:42.653 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:35:42.663 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:35:42.663 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:35:42.663 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:35:42.674 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:35:42.674 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:35:42.704 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:35:42.705 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:35:43.203 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1337 8 365281 365273 2.13423 0 - 0.0299513 2026-02-23T13:35:43.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:35:43 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/449029703' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:35:44.203 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1338 8 365281 365273 2.13264 0 - 0.0299513 2026-02-23T13:35:44.247 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:35:43 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:35:43] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:35:44.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:35:43 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:35:43.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:35:44.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:35:44 trial068 ceph-mon[38805]: pgmap v1058: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:35:44.706 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:35:44.825 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:35:44.922 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:35:44.920+0000 7fa9aa89e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:35:44.922 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:35:44.920+0000 7fa9aa89e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:35:44.933 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:35:44.934 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:35:44.943 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:35:44.944 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:35:44.944 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:35:44.954 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:35:44.954 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:35:44.982 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:35:44.982 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:35:45.203 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1339 8 365281 365273 2.13105 0 - 0.0299513 2026-02-23T13:35:45.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:35:45 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/687892511' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:35:46.203 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T13:35:46.203215+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T13:35:46.204 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T13:35:46.204 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1340 8 365281 365273 2.12946 0 - 0.0299513 2026-02-23T13:35:46.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:35:46 trial068 ceph-mon[38805]: pgmap v1059: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:35:46.984 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:35:47.102 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:35:47.199 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:35:47.197+0000 7f4788f8e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:35:47.199 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:35:47.197+0000 7f4788f8e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:35:47.203 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1341 8 365281 365273 2.12787 0 - 0.0299513 2026-02-23T13:35:47.211 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:35:47.211 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:35:47.221 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:35:47.222 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:35:47.222 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:35:47.232 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:35:47.232 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:35:47.263 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:35:47 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1186387775' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:35:47.264 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:35:47.264 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:35:48.203 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1342 8 365281 365273 2.12628 0 - 0.0299513 2026-02-23T13:35:48.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:35:48 trial068 ceph-mon[38805]: pgmap v1060: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:35:49.204 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1343 8 365281 365273 2.1247 0 - 0.0299513 2026-02-23T13:35:49.265 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:35:49.381 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:35:49.478 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:35:49.476+0000 7f8cb479c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:35:49.478 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:35:49.477+0000 7f8cb479c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:35:49.490 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:35:49.490 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:35:49.500 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:35:49.501 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:35:49.501 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:35:49.510 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:35:49.511 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:35:49.540 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:35:49.540 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:35:50.204 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1344 8 365281 365273 2.12312 0 - 0.0299513 2026-02-23T13:35:50.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:35:50 trial068 ceph-mon[38805]: pgmap v1061: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:35:50.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:35:50 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3933590819' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:35:51.204 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1345 8 365281 365273 2.12154 0 - 0.0299513 2026-02-23T13:35:51.542 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:35:51.661 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:35:51.770 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:35:51.768+0000 7f8b1f224640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:35:51.770 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:35:51.768+0000 7f8b1f224640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:35:51.782 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:35:51.782 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:35:51.792 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:35:51.792 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:35:51.793 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:35:51.803 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:35:51.803 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:35:51.847 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:35:51.847 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:35:52.204 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1346 8 365281 365273 2.11996 0 - 0.0299513 2026-02-23T13:35:52.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:35:52 trial068 ceph-mon[38805]: pgmap v1062: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:35:52.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:35:52 trial068 ceph-mon[38805]: 5.8 scrub starts 2026-02-23T13:35:52.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:35:52 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3037729308' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:35:53.204 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1347 8 365281 365273 2.11839 0 - 0.0299513 2026-02-23T13:35:53.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:35:53 trial068 ceph-mon[38805]: 5.8 scrub ok 2026-02-23T13:35:53.848 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:35:53.968 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:35:54.074 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:35:54.072+0000 7fd14d115640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:35:54.075 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:35:54.073+0000 7fd14d115640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:35:54.086 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:35:54.086 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:35:54.096 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:35:54.097 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:35:54.097 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:35:54.108 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:35:54.108 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:35:54.125 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:35:53 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:35:53] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:35:54.140 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:35:54.140 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:35:54.204 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1348 8 365281 365273 2.11682 0 - 0.0299513 2026-02-23T13:35:54.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:35:53 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:35:53.972Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:35:54.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:35:54 trial068 ceph-mon[38805]: pgmap v1063: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:35:54.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:35:54 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/215456316' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:35:55.204 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1349 8 365281 365273 2.11525 0 - 0.0299513 2026-02-23T13:35:56.141 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:35:56.204 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1350 8 365281 365273 2.11368 0 - 0.0299513 2026-02-23T13:35:56.261 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:35:56.361 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:35:56.360+0000 7fc6a73dc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:35:56.362 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:35:56.360+0000 7fc6a73dc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:35:56.373 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:35:56.373 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:35:56.383 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:35:56.383 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:35:56.384 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:35:56.394 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:35:56.394 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:35:56.421 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:35:56 trial068 ceph-mon[38805]: pgmap v1064: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:35:56.421 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:35:56.422 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:35:57.204 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1351 8 365281 365273 2.11212 0 - 0.0299513 2026-02-23T13:35:57.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:35:57 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3831298892' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:35:57.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:35:57 trial068 ceph-mon[38805]: 5.5 scrub starts 2026-02-23T13:35:58.204 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1352 8 365281 365273 2.11056 0 - 0.0299513 2026-02-23T13:35:58.423 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:35:58.543 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:35:58.566 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:35:58 trial068 ceph-mon[38805]: 5.5 scrub ok 2026-02-23T13:35:58.566 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:35:58 trial068 ceph-mon[38805]: pgmap v1065: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:35:58.640 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:35:58.639+0000 7f3111bc6640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:35:58.641 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:35:58.639+0000 7f3111bc6640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:35:58.652 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:35:58.653 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:35:58.663 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:35:58.663 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:35:58.663 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:35:58.673 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:35:58.674 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:35:58.704 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:35:58.704 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:35:59.204 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1353 8 365281 365273 2.109 0 - 0.0299513 2026-02-23T13:35:59.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:35:59 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2235827722' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:36:00.204 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1354 8 365281 365273 2.10744 0 - 0.0299513 2026-02-23T13:36:00.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:36:00 trial068 ceph-mon[38805]: pgmap v1066: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:36:00.706 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:36:00.824 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:36:00.926 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:36:00.924+0000 7fa47a2f5640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:36:00.926 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:36:00.924+0000 7fa47a2f5640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:36:00.938 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:36:00.938 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:36:00.948 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:36:00.948 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:36:00.948 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:36:00.958 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:36:00.958 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:36:00.988 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:36:00.988 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:36:01.204 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1355 8 365281 365273 2.10588 0 - 0.0299513 2026-02-23T13:36:01.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:36:01 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3477088882' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:36:02.204 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1356 8 365281 365273 2.10433 0 - 0.0299513 2026-02-23T13:36:02.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:36:02 trial068 ceph-mon[38805]: pgmap v1067: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:36:02.989 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:36:03.106 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:36:03.205 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:36:03.203+0000 7f5b5094a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:36:03.205 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:36:03.203+0000 7f5b5094a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:36:03.205 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1357 8 365281 365273 2.10278 0 - 0.0299513 2026-02-23T13:36:03.216 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:36:03.217 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:36:03.227 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:36:03.227 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:36:03.227 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:36:03.237 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:36:03.237 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:36:03.267 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:36:03.267 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:36:03.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:36:03 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3892841288' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:36:04.205 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1358 8 365281 365273 2.10123 0 - 0.0299513 2026-02-23T13:36:04.265 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:36:03 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:36:03.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:36:04.271 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:36:03 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:36:03] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:36:04.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:36:04 trial068 ceph-mon[38805]: pgmap v1068: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:36:05.205 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1359 8 365281 365273 2.09968 0 - 0.0299513 2026-02-23T13:36:05.268 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:36:05.386 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:36:05.491 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:36:05.489+0000 7f5fa38d9640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:36:05.491 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:36:05.489+0000 7f5fa38d9640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:36:05.502 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:36:05.503 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:36:05.513 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:36:05.513 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:36:05.513 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:36:05.523 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:36:05.523 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:36:05.553 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:36:05.553 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:36:06.205 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T13:36:06.204795+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T13:36:06.205 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T13:36:06.205 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1360 8 365281 365273 2.09814 0 - 0.0299513 2026-02-23T13:36:06.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:36:06 trial068 ceph-mon[38805]: pgmap v1069: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:36:06.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:36:06 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3217809138' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:36:07.205 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1361 8 365281 365273 2.0966 0 - 0.0299513 2026-02-23T13:36:07.555 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:36:07.674 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:36:07.771 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:36:07.768+0000 7fb46325b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:36:07.771 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:36:07.768+0000 7fb46325b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:36:07.783 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:36:07.783 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:36:07.793 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:36:07.793 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:36:07.794 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:36:07.804 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:36:07.804 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:36:07.835 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:36:07.835 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:36:08.205 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1362 8 365281 365273 2.09506 0 - 0.0299513 2026-02-23T13:36:08.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:36:08 trial068 ceph-mon[38805]: pgmap v1070: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:36:08.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:36:08 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1362735102' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:36:09.205 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1363 8 365281 365273 2.09352 0 - 0.0299513 2026-02-23T13:36:09.836 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:36:09.954 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:36:10.048 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:36:10.046+0000 7f4377860640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:36:10.048 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:36:10.046+0000 7f4377860640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:36:10.061 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:36:10.061 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:36:10.071 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:36:10.072 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:36:10.072 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:36:10.082 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:36:10.082 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:36:10.113 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:36:10.114 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:36:10.205 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1364 8 365281 365273 2.09199 0 - 0.0299513 2026-02-23T13:36:10.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:36:10 trial068 ceph-mon[38805]: pgmap v1071: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:36:10.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:36:10 trial068 ceph-mon[38805]: osdmap e201: 4 total, 3 up, 4 in 2026-02-23T13:36:10.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:36:10 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3763030536' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:36:11.205 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1365 8 365281 365273 2.09045 0 - 0.0299513 2026-02-23T13:36:12.114 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:36:12.205 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1366 8 365281 365273 2.08892 0 - 0.0299513 2026-02-23T13:36:12.235 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:36:12.342 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:36:12.340+0000 7fcb037c4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:36:12.342 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:36:12.340+0000 7fcb037c4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:36:12.353 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:36:12.354 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:36:12.364 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:36:12.364 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:36:12.364 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:36:12.374 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:36:12.374 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:36:12.390 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:36:12 trial068 ceph-mon[38805]: pgmap v1073: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:36:12.404 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:36:12.405 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:36:13.205 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1367 8 365281 365273 2.0874 0 - 0.0299513 2026-02-23T13:36:13.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:36:13 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/495708197' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:36:14.205 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1368 8 365281 365273 2.08587 0 - 0.0299513 2026-02-23T13:36:14.281 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:36:13 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:36:13] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:36:14.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:36:13 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:36:13.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:36:14.406 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:36:14.523 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:36:14.546 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:36:14 trial068 ceph-mon[38805]: pgmap v1074: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:36:14.628 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:36:14.626+0000 7fdc30570640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:36:14.628 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:36:14.626+0000 7fdc30570640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:36:14.640 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:36:14.640 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:36:14.650 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:36:14.650 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:36:14.650 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:36:14.659 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:36:14.660 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:36:14.705 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:36:14.705 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:36:15.206 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1369 8 365281 365273 2.08435 0 - 0.0299513 2026-02-23T13:36:15.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:36:15 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1631473592' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:36:16.206 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1370 8 365281 365273 2.08283 0 - 0.0299513 2026-02-23T13:36:16.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:36:16 trial068 ceph-mon[38805]: pgmap v1075: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:36:16.706 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:36:16.827 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:36:16.925 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:36:16.923+0000 7fb0fa21a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:36:16.925 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:36:16.923+0000 7fb0fa21a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:36:16.937 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:36:16.937 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:36:16.947 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:36:16.947 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:36:16.947 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:36:16.957 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:36:16.958 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:36:17.002 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:36:17.003 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:36:17.206 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1371 8 365281 365273 2.08131 0 - 0.0299513 2026-02-23T13:36:17.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:36:17 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/510463718' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:36:18.206 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1372 8 365281 365273 2.07979 0 - 0.0299513 2026-02-23T13:36:18.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:36:18 trial068 ceph-mon[38805]: pgmap v1076: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:36:19.004 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:36:19.121 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:36:19.206 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1373 8 365281 365273 2.07827 0 - 0.0299513 2026-02-23T13:36:19.221 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:36:19.219+0000 7f1738a62640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:36:19.221 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:36:19.219+0000 7f1738a62640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:36:19.232 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:36:19.233 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:36:19.242 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:36:19.243 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:36:19.243 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:36:19.252 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:36:19.253 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:36:19.282 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:36:19.282 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:36:20.206 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1374 8 365281 365273 2.07676 0 - 0.0299513 2026-02-23T13:36:20.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:36:20 trial068 ceph-mon[38805]: pgmap v1077: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:36:20.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:36:20 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3532343943' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:36:21.206 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1375 8 365281 365273 2.07525 0 - 0.0299513 2026-02-23T13:36:21.283 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:36:21.401 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:36:21.494 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:36:21.492+0000 7f109279f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:36:21.494 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:36:21.493+0000 7f109279f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:36:21.506 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:36:21.507 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:36:21.517 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:36:21.517 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:36:21.517 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:36:21.527 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:36:21.528 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:36:21.557 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:36:21.558 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:36:22.206 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1376 8 365281 365273 2.07374 0 - 0.0299513 2026-02-23T13:36:22.495 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:36:22 trial068 ceph-mon[38805]: pgmap v1078: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:36:22.496 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:36:22 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2333905' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:36:23.206 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1377 8 365281 365273 2.07224 0 - 0.0299513 2026-02-23T13:36:23.559 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:36:23.678 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:36:23.785 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:36:23.783+0000 7f4486bbc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:36:23.785 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:36:23.784+0000 7f4486bbc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:36:23.797 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:36:23.798 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:36:23.807 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:36:23.808 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:36:23.808 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:36:23.817 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:36:23.817 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:36:23.848 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:36:23.849 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:36:24.197 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:36:23 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:36:23] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:36:24.206 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1378 8 365281 365273 2.07073 0 - 0.0299513 2026-02-23T13:36:24.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:36:23 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:36:23.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:36:24.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:36:24 trial068 ceph-mon[38805]: pgmap v1079: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:36:24.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:36:24 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2471648980' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:36:25.207 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1379 8 365281 365273 2.06923 0 - 0.0299513 2026-02-23T13:36:25.850 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:36:25.970 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:36:26.072 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:36:26.070+0000 7f90a46fd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:36:26.072 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:36:26.070+0000 7f90a46fd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:36:26.083 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:36:26.084 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:36:26.095 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:36:26.095 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:36:26.095 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:36:26.106 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:36:26.107 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:36:26.138 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:36:26.138 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:36:26.207 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T13:36:26.206526+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T13:36:26.207 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T13:36:26.207 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1380 8 365281 365273 2.06773 0 - 0.0299513 2026-02-23T13:36:26.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:36:26 trial068 ceph-mon[38805]: pgmap v1080: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:36:26.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:36:26 trial068 ceph-mon[38805]: 5.e scrub starts 2026-02-23T13:36:26.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:36:26 trial068 ceph-mon[38805]: 5.e scrub ok 2026-02-23T13:36:26.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:36:26 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2380576294' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:36:27.207 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1381 8 365281 365273 2.06624 0 - 0.0299513 2026-02-23T13:36:28.139 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:36:28.207 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1382 8 365281 365273 2.06474 0 - 0.0299513 2026-02-23T13:36:28.257 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:36:28.358 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:36:28.356+0000 7fa3b876d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:36:28.358 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:36:28.356+0000 7fa3b876d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:36:28.370 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:36:28.370 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:36:28.380 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:36:28.380 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:36:28.381 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:36:28.390 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:36:28.391 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:36:28.422 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:36:28 trial068 ceph-mon[38805]: pgmap v1081: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:36:28.423 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:36:28.424 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:36:29.207 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1383 8 365281 365273 2.06325 0 - 0.0299513 2026-02-23T13:36:29.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:36:29 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/139000346' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:36:30.207 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1384 8 365281 365273 2.06176 0 - 0.0299513 2026-02-23T13:36:30.424 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:36:30.543 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:36:30.565 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:36:30 trial068 ceph-mon[38805]: pgmap v1082: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:36:30.648 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:36:30.646+0000 7f90ef6d0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:36:30.649 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:36:30.647+0000 7f90ef6d0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:36:30.660 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:36:30.661 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:36:30.670 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:36:30.671 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:36:30.671 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:36:30.680 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:36:30.681 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:36:30.710 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:36:30.711 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:36:31.207 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1385 8 365281 365273 2.06027 0 - 0.0299513 2026-02-23T13:36:31.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:36:31 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2850691819' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:36:32.207 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1386 8 365281 365273 2.05878 0 - 0.0299513 2026-02-23T13:36:32.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:36:32 trial068 ceph-mon[38805]: pgmap v1083: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:36:32.711 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:36:32.831 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:36:32.932 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:36:32.930+0000 7f6bd0ef7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:36:32.932 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:36:32.930+0000 7f6bd0ef7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:36:32.944 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:36:32.944 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:36:32.954 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:36:32.954 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:36:32.954 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:36:32.965 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:36:32.965 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:36:32.995 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:36:32.995 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:36:33.207 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1387 8 365281 365273 2.0573 0 - 0.0299513 2026-02-23T13:36:33.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:36:33 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1149853259' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:36:34.207 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1388 8 365281 365273 2.05581 0 - 0.0299513 2026-02-23T13:36:34.264 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:36:33 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:36:33.974Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:36:34.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:36:34 trial068 ceph-mon[38805]: pgmap v1084: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:36:34.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:36:33 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:36:33] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:36:34.995 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:36:35.113 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:36:35.207 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1389 8 365281 365273 2.05433 0 - 0.0299513 2026-02-23T13:36:35.209 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:36:35.207+0000 7f019b081640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:36:35.209 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:36:35.208+0000 7f019b081640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:36:35.221 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:36:35.221 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:36:35.232 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:36:35.232 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:36:35.232 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:36:35.242 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:36:35.242 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:36:35.273 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:36:35.273 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:36:36.207 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1390 8 365281 365273 2.05286 0 - 0.0299513 2026-02-23T13:36:36.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:36:36 trial068 ceph-mon[38805]: pgmap v1085: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:36:36.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:36:36 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/4094459631' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:36:37.207 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1391 8 365281 365273 2.05138 0 - 0.0299513 2026-02-23T13:36:37.274 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:36:37.391 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:36:37.492 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:36:37.490+0000 7fe2a0522640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:36:37.492 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:36:37.490+0000 7fe2a0522640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:36:37.504 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:36:37.504 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:36:37.514 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:36:37.514 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:36:37.514 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:36:37.524 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:36:37.525 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:36:37.554 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:36:37.554 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:36:38.208 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1392 8 365281 365273 2.04991 0 - 0.0299513 2026-02-23T13:36:38.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:36:38 trial068 ceph-mon[38805]: pgmap v1086: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:36:38.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:36:38 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3599959893' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:36:39.208 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1393 8 365281 365273 2.04844 0 - 0.0299513 2026-02-23T13:36:39.555 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:36:39.576 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:36:39 trial068 ceph-mon[38805]: 5.a scrub starts 2026-02-23T13:36:39.577 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:36:39 trial068 ceph-mon[38805]: 5.a scrub ok 2026-02-23T13:36:39.577 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:36:39 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2026-02-23T13:36:39.669 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:36:39.767 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:36:39.765+0000 7fdbebd54640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:36:39.767 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:36:39.765+0000 7fdbebd54640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:36:39.780 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:36:39.780 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:36:39.790 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:36:39.790 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:36:39.790 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:36:39.802 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:36:39.803 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:36:39.833 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:36:39.834 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:36:40.208 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1394 8 365281 365273 2.04697 0 - 0.0299513 2026-02-23T13:36:40.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:36:40 trial068 ceph-mon[38805]: pgmap v1087: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:36:40.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:36:40 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T13:36:40.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:36:40 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T13:36:40.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:36:40 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T13:36:40.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:36:40 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T13:36:40.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:36:40 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-23T13:36:40.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:36:40 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2026-02-23T13:36:40.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:36:40 trial068 ceph-mon[38805]: pgmap v1088: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:36:40.595 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:36:40 trial068 ceph-mon[38805]: pgmap v1089: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:36:40.595 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:36:40 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T13:36:40.595 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:36:40 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2026-02-23T13:36:40.595 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:36:40 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/25012344' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:36:41.208 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1395 8 365281 365273 2.0455 0 - 0.0299513 2026-02-23T13:36:41.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:36:41 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T13:36:41.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:36:41 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T13:36:41.835 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:36:41.953 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:36:42.055 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:36:42.054+0000 7fca03f52640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:36:42.056 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:36:42.054+0000 7fca03f52640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:36:42.067 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:36:42.067 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:36:42.077 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:36:42.077 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:36:42.077 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:36:42.088 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:36:42.089 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:36:42.118 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:36:42.119 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:36:42.208 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1396 8 365281 365273 2.04403 0 - 0.0299513 2026-02-23T13:36:42.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:36:42 trial068 ceph-mon[38805]: pgmap v1090: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:36:42.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:36:42 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/4212458922' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:36:43.208 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1397 8 365281 365273 2.04257 0 - 0.0299513 2026-02-23T13:36:44.120 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:36:44.208 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1398 8 365281 365273 2.04111 0 - 0.0299513 2026-02-23T13:36:44.236 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:36:44.258 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:36:43 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:36:43] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:36:44.337 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:36:44.335+0000 7f8a12465640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:36:44.337 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:36:44.335+0000 7f8a12465640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:36:44.349 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:36:44.350 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:36:44.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:36:43 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:36:43.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:36:44.360 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:36:44.360 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:36:44.361 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:36:44.370 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:36:44.370 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:36:44.400 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:36:44.400 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:36:44.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:36:44 trial068 ceph-mon[38805]: pgmap v1091: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:36:44.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:36:44 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3821192108' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:36:45.208 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1399 8 365281 365273 2.03965 0 - 0.0299513 2026-02-23T13:36:45.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:36:45 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T13:36:45.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:36:45 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T13:36:46.208 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T13:36:46.208237+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T13:36:46.209 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T13:36:46.209 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1400 8 365281 365273 2.03819 0 - 0.0299513 2026-02-23T13:36:46.401 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:36:46.521 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:36:46.619 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:36:46.617+0000 7fc1b9639640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:36:46.619 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:36:46.617+0000 7fc1b9639640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:36:46.631 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:36:46.631 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:36:46.641 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:36:46.641 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:36:46.641 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:36:46.651 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:36:46.652 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:36:46.681 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:36:46 trial068 ceph-mon[38805]: pgmap v1092: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:36:46.681 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:36:46.682 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:36:47.208 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1401 8 365281 365273 2.03674 0 - 0.0299513 2026-02-23T13:36:47.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:36:47 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2387486158' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:36:48.208 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1402 8 365281 365273 2.03529 0 - 0.0299513 2026-02-23T13:36:48.683 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:36:48.800 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:36:48.822 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:36:48 trial068 ceph-mon[38805]: pgmap v1093: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:36:48.898 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:36:48.897+0000 7f158ae5a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:36:48.899 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:36:48.897+0000 7f158ae5a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:36:48.911 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:36:48.911 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:36:48.921 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:36:48.922 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:36:48.922 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:36:48.932 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:36:48.932 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:36:48.963 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:36:48.964 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:36:49.209 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1403 8 365281 365273 2.03384 0 - 0.0299513 2026-02-23T13:36:49.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:36:49 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/925285347' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:36:50.209 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1404 8 365281 365273 2.03239 0 - 0.0299513 2026-02-23T13:36:50.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:36:50 trial068 ceph-mon[38805]: pgmap v1094: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:36:50.965 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:36:51.084 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:36:51.180 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:36:51.178+0000 7ffb723ae640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:36:51.180 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:36:51.178+0000 7ffb723ae640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:36:51.194 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:36:51.194 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:36:51.204 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:36:51.204 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:36:51.204 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:36:51.208 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1405 8 365281 365273 2.03094 0 - 0.0299513 2026-02-23T13:36:51.215 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:36:51.216 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:36:51.244 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:36:51.245 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:36:51.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:36:51 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3055823977' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:36:52.209 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1406 8 365281 365273 2.0295 0 - 0.0299513 2026-02-23T13:36:52.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:36:52 trial068 ceph-mon[38805]: pgmap v1095: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:36:53.209 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1407 8 365281 365273 2.02805 0 - 0.0299513 2026-02-23T13:36:53.245 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:36:53.366 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:36:53.464 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:36:53.463+0000 7fba8aca7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:36:53.465 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:36:53.463+0000 7fba8aca7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:36:53.477 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:36:53.477 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:36:53.488 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:36:53.488 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:36:53.488 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:36:53.498 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:36:53.499 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:36:53.529 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:36:53.529 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:36:53.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:36:53 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2321375273' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:36:54.209 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1408 8 365281 365273 2.02661 0 - 0.0299513 2026-02-23T13:36:54.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:36:53 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:36:53] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:36:54.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:36:53 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:36:53.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:36:54.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:36:54 trial068 ceph-mon[38805]: pgmap v1096: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:36:55.209 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1409 8 365281 365273 2.02517 0 - 0.0299513 2026-02-23T13:36:55.530 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:36:55.648 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:36:55.746 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:36:55.743+0000 7f8b89c5a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:36:55.746 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:36:55.744+0000 7f8b89c5a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:36:55.757 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:36:55.757 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:36:55.767 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:36:55.767 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:36:55.767 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:36:55.777 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:36:55.777 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:36:55.806 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:36:55.806 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:36:56.209 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1410 8 365281 365273 2.02374 0 - 0.0299513 2026-02-23T13:36:56.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:36:56 trial068 ceph-mon[38805]: pgmap v1097: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:36:56.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:36:56 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3844632443' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:36:57.209 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1411 8 365281 365273 2.0223 0 - 0.0299513 2026-02-23T13:36:57.807 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:36:57.925 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:36:58.031 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:36:58.029+0000 7f0c60e30640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:36:58.031 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:36:58.030+0000 7f0c60e30640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:36:58.044 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:36:58.044 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:36:58.055 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:36:58.055 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:36:58.055 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:36:58.065 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:36:58.066 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:36:58.097 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:36:58.098 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:36:58.209 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1412 8 365281 365273 2.02087 0 - 0.0299513 2026-02-23T13:36:58.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:36:58 trial068 ceph-mon[38805]: 5.7 scrub starts 2026-02-23T13:36:58.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:36:58 trial068 ceph-mon[38805]: 5.7 scrub ok 2026-02-23T13:36:58.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:36:58 trial068 ceph-mon[38805]: pgmap v1098: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:36:58.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:36:58 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T13:36:58.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:36:58 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T13:36:58.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:36:58 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/20638654' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:36:59.209 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1413 8 365281 365273 2.01944 0 - 0.0299513 2026-02-23T13:37:00.099 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:37:00.209 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1414 8 365281 365273 2.01801 0 - 0.0299513 2026-02-23T13:37:00.216 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:37:00.313 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:37:00.311+0000 7efda65fd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:37:00.313 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:37:00.311+0000 7efda65fd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:37:00.325 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:37:00.325 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:37:00.335 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:37:00.336 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:37:00.336 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:37:00.346 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:37:00.346 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:37:00.391 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:37:00.391 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:37:00.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:37:00 trial068 ceph-mon[38805]: pgmap v1099: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:37:00.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:37:00 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2280914284' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:37:01.209 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1415 8 365281 365273 2.01659 0 - 0.0299513 2026-02-23T13:37:01.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:37:01 trial068 ceph-mon[38805]: 5.8 scrub starts 2026-02-23T13:37:01.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:37:01 trial068 ceph-mon[38805]: 5.8 scrub ok 2026-02-23T13:37:02.209 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1416 8 365281 365273 2.01516 0 - 0.0299513 2026-02-23T13:37:02.393 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:37:02.511 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:37:02.610 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:37:02.608+0000 7f97fd81e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:37:02.610 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:37:02.608+0000 7f97fd81e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:37:02.622 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:37:02.623 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:37:02.633 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:37:02.633 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:37:02.633 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:37:02.644 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:37:02.644 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:37:02.671 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:37:02 trial068 ceph-mon[38805]: pgmap v1100: 17 pgs: 1 active+clean+scrubbing, 1 active+recovery_unfound+undersized+degraded, 4 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:37:02.672 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:37:02.673 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:37:03.210 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1417 8 365281 365273 2.01374 0 - 0.0299513 2026-02-23T13:37:03.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:37:03 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1632317438' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:37:04.209 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1418 8 365281 365273 2.01232 0 - 0.0299513 2026-02-23T13:37:04.264 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:37:03 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:37:03.974Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:37:04.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:37:03 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:37:03] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:37:04.674 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:37:04.793 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:37:04.816 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:37:04 trial068 ceph-mon[38805]: pgmap v1101: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 1 active+clean+scrubbing, 4 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:37:04.890 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:37:04.888+0000 7f2f7fbff640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:37:04.890 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:37:04.888+0000 7f2f7fbff640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:37:04.902 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:37:04.902 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:37:04.912 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:37:04.912 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:37:04.912 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:37:04.923 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:37:04.923 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:37:04.952 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:37:04.953 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:37:05.210 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1419 8 365281 365273 2.0109 0 - 0.0299513 2026-02-23T13:37:05.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:37:05 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1276369609' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:37:06.210 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T13:37:06.209654+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T13:37:06.210 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T13:37:06.210 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1420 8 365281 365273 2.00949 0 - 0.0299513 2026-02-23T13:37:06.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:37:06 trial068 ceph-mon[38805]: pgmap v1102: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:37:06.954 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:37:07.072 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:37:07.169 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:37:07.167+0000 7f04c4124640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:37:07.169 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:37:07.167+0000 7f04c4124640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:37:07.181 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:37:07.181 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:37:07.190 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:37:07.191 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:37:07.191 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:37:07.201 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:37:07.201 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:37:07.210 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1421 8 365281 365273 2.00807 0 - 0.0299513 2026-02-23T13:37:07.231 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:37:07.232 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:37:07.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:37:07 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1518921288' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:37:08.210 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1422 8 365281 365273 2.00666 0 - 0.0299513 2026-02-23T13:37:08.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:37:08 trial068 ceph-mon[38805]: pgmap v1103: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:37:09.210 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1423 8 365281 365273 2.00525 0 - 0.0299513 2026-02-23T13:37:09.234 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:37:09.350 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:37:09.451 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:37:09.449+0000 7f2a8a621640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:37:09.451 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:37:09.450+0000 7f2a8a621640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:37:09.463 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:37:09.463 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:37:09.473 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:37:09.474 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:37:09.474 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:37:09.484 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:37:09.484 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:37:09.530 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:37:09.531 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:37:09.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:37:09 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2089779552' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:37:10.210 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1424 8 365281 365273 2.00384 0 - 0.0299513 2026-02-23T13:37:10.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:37:10 trial068 ceph-mon[38805]: pgmap v1104: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:37:11.210 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1425 8 365281 365273 2.00244 0 - 0.0299513 2026-02-23T13:37:11.532 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:37:11.651 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:37:11.746 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:37:11.744+0000 7fb8ef689640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:37:11.747 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:37:11.745+0000 7fb8ef689640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:37:11.758 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:37:11.759 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:37:11.768 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:37:11.769 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:37:11.769 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:37:11.778 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:37:11.779 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:37:11.810 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:37:11.810 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:37:12.210 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1426 8 365281 365273 2.00103 0 - 0.0299513 2026-02-23T13:37:12.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:37:12 trial068 ceph-mon[38805]: pgmap v1105: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:37:12.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:37:12 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2863743496' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:37:13.210 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1427 8 365281 365273 1.99963 0 - 0.0299513 2026-02-23T13:37:13.812 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:37:13.931 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:37:14.027 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:37:14.025+0000 7f4d69ee5640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:37:14.027 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:37:14.025+0000 7f4d69ee5640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:37:14.039 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:37:14.039 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:37:14.049 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:37:14.049 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:37:14.049 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:37:14.059 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:37:14.059 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:37:14.091 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:37:13 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:37:13] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:37:14.091 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:37:14.092 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:37:14.211 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1428 8 365281 365273 1.99823 0 - 0.0299513 2026-02-23T13:37:14.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:37:13 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:37:13.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:37:14.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:37:14 trial068 ceph-mon[38805]: pgmap v1106: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:37:14.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:37:14 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2103669144' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:37:15.210 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1429 8 365281 365273 1.99683 0 - 0.0299513 2026-02-23T13:37:16.092 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:37:16.211 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1430 8 365281 365273 1.99543 0 - 0.0299513 2026-02-23T13:37:16.211 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:37:16.314 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:37:16.312+0000 7fd3306a3640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:37:16.314 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:37:16.312+0000 7fd3306a3640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:37:16.325 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:37:16.326 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:37:16.335 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:37:16.336 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:37:16.336 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:37:16.346 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:37:16.346 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:37:16.376 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:37:16.377 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:37:16.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:37:16 trial068 ceph-mon[38805]: pgmap v1107: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:37:16.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:37:16 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3193112755' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:37:17.211 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1431 8 365281 365273 1.99404 0 - 0.0299513 2026-02-23T13:37:18.211 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1432 8 365281 365273 1.99265 0 - 0.0299513 2026-02-23T13:37:18.378 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:37:18.495 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:37:18.593 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:37:18.591+0000 7f51c68e6640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:37:18.593 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:37:18.591+0000 7f51c68e6640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:37:18.604 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:37:18.605 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:37:18.615 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:37:18.616 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:37:18.616 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:37:18.625 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:37:18.626 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:37:18.654 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:37:18 trial068 ceph-mon[38805]: pgmap v1108: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:37:18.656 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:37:18.656 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:37:19.211 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1433 8 365281 365273 1.99126 0 - 0.0299513 2026-02-23T13:37:19.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:37:19 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3910151625' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:37:20.211 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1434 8 365281 365273 1.98987 0 - 0.0299513 2026-02-23T13:37:20.657 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:37:20.775 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:37:20.815 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:37:20 trial068 ceph-mon[38805]: pgmap v1109: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:37:20.872 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:37:20.870+0000 7f76eab20640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:37:20.872 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:37:20.870+0000 7f76eab20640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:37:20.884 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:37:20.884 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:37:20.894 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:37:20.895 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:37:20.895 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:37:20.905 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:37:20.905 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:37:20.935 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:37:20.935 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:37:21.211 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1435 8 365281 365273 1.98848 0 - 0.0299513 2026-02-23T13:37:21.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:37:21 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1987454026' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:37:22.211 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1436 8 365281 365273 1.9871 0 - 0.0299513 2026-02-23T13:37:22.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:37:22 trial068 ceph-mon[38805]: pgmap v1110: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:37:22.936 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:37:23.056 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:37:23.159 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:37:23.158+0000 7ff403c1a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:37:23.160 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:37:23.158+0000 7ff403c1a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:37:23.171 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:37:23.172 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:37:23.181 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:37:23.182 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:37:23.182 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:37:23.192 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:37:23.192 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:37:23.211 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1437 8 365281 365273 1.98571 0 - 0.0299513 2026-02-23T13:37:23.222 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:37:23.223 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:37:23.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:37:23 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2663230477' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:37:24.211 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1438 8 365281 365273 1.98433 0 - 0.0299513 2026-02-23T13:37:24.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:37:23 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:37:23] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:37:24.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:37:23 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:37:23.974Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:37:24.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:37:24 trial068 ceph-mon[38805]: pgmap v1111: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:37:25.211 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1439 8 365281 365273 1.98295 0 - 0.0299513 2026-02-23T13:37:25.224 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:37:25.343 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:37:25.443 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:37:25.441+0000 7f7d760e3640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:37:25.444 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:37:25.442+0000 7f7d760e3640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:37:25.455 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:37:25.455 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:37:25.465 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:37:25.465 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:37:25.465 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:37:25.475 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:37:25.476 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:37:25.506 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:37:25.506 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:37:25.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:37:25 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1810568807' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:37:26.211 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T13:37:26.211165+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T13:37:26.211 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T13:37:26.212 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1440 8 365281 365273 1.98158 0 - 0.0299513 2026-02-23T13:37:26.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:37:26 trial068 ceph-mon[38805]: pgmap v1112: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:37:27.211 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1441 8 365281 365273 1.9802 0 - 0.0299513 2026-02-23T13:37:27.507 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:37:27.627 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:37:27.728 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:37:27.726+0000 7f3802433640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:37:27.728 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:37:27.726+0000 7f3802433640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:37:27.740 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:37:27.740 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:37:27.750 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:37:27.751 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:37:27.751 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:37:27.761 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:37:27.761 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:37:27.791 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:37:27.792 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:37:28.211 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1442 8 365281 365273 1.97883 0 - 0.0299513 2026-02-23T13:37:28.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:37:28 trial068 ceph-mon[38805]: pgmap v1113: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:37:28.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:37:28 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2033093789' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:37:29.211 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1443 8 365281 365273 1.97746 0 - 0.0299513 2026-02-23T13:37:29.792 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:37:29.909 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:37:30.006 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:37:30.004+0000 7ff0d6786640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:37:30.006 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:37:30.005+0000 7ff0d6786640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:37:30.018 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:37:30.018 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:37:30.029 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:37:30.029 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:37:30.029 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:37:30.039 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:37:30.039 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:37:30.071 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:37:30.071 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:37:30.211 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1444 8 365281 365273 1.97609 0 - 0.0299513 2026-02-23T13:37:30.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:37:30 trial068 ceph-mon[38805]: pgmap v1114: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:37:30.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:37:30 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3448554459' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:37:31.212 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1445 8 365281 365273 1.97472 0 - 0.0299513 2026-02-23T13:37:32.072 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:37:32.190 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:37:32.212 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1446 8 365281 365273 1.97335 0 - 0.0299513 2026-02-23T13:37:32.294 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:37:32.292+0000 7efdc5920640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:37:32.295 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:37:32.293+0000 7efdc5920640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:37:32.306 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:37:32.306 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:37:32.316 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:37:32.317 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:37:32.317 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:37:32.326 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:37:32.327 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:37:32.360 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:37:32.361 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:37:32.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:37:32 trial068 ceph-mon[38805]: pgmap v1115: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:37:32.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:37:32 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2663738897' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:37:33.211 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1447 8 365281 365273 1.97199 0 - 0.0299513 2026-02-23T13:37:33.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:37:33 trial068 ceph-mon[38805]: 5.5 scrub starts 2026-02-23T13:37:33.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:37:33 trial068 ceph-mon[38805]: 5.5 scrub ok 2026-02-23T13:37:34.212 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1448 8 365281 365273 1.97063 0 - 0.0299513 2026-02-23T13:37:34.264 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:37:33 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:37:33.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:37:34.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:37:33 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:37:33] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:37:34.362 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:37:34.482 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:37:34.581 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:37:34.579+0000 7f74e877b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:37:34.582 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:37:34.580+0000 7f74e877b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:37:34.593 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:37:34.593 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:37:34.603 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:37:34.604 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:37:34.604 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:37:34.614 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:37:34.614 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:37:34.642 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:37:34 trial068 ceph-mon[38805]: pgmap v1116: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:37:34.643 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:37:34.644 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:37:35.212 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1449 8 365281 365273 1.96927 0 - 0.0299513 2026-02-23T13:37:35.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:37:35 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2225385534' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:37:36.212 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1450 8 365281 365273 1.96791 0 - 0.0299513 2026-02-23T13:37:36.645 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:37:36.762 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:37:36.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:37:36 trial068 ceph-mon[38805]: pgmap v1117: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:37:36.856 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:37:36.854+0000 7fb431121640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:37:36.856 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:37:36.855+0000 7fb431121640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:37:36.868 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:37:36.868 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:37:36.878 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:37:36.879 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:37:36.879 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:37:36.888 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:37:36.889 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:37:36.917 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:37:36.917 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:37:37.212 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1451 8 365281 365273 1.96655 0 - 0.0299513 2026-02-23T13:37:37.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:37:37 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1777167108' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:37:38.212 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1452 8 365281 365273 1.9652 0 - 0.0299513 2026-02-23T13:37:38.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:37:38 trial068 ceph-mon[38805]: pgmap v1118: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:37:38.918 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:37:39.041 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:37:39.145 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:37:39.143+0000 7ff1fd307640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:37:39.145 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:37:39.143+0000 7ff1fd307640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:37:39.157 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:37:39.157 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:37:39.168 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:37:39.168 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:37:39.168 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:37:39.179 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:37:39.179 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:37:39.210 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:37:39.211 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:37:39.212 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1453 8 365281 365273 1.96385 0 - 0.0299513 2026-02-23T13:37:39.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:37:39 trial068 ceph-mon[38805]: 5.e scrub starts 2026-02-23T13:37:39.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:37:39 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1951937462' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:37:39.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:37:39 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2026-02-23T13:37:40.212 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1454 8 365281 365273 1.9625 0 - 0.0299513 2026-02-23T13:37:40.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:37:40 trial068 ceph-mon[38805]: 5.e scrub ok 2026-02-23T13:37:40.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:37:40 trial068 ceph-mon[38805]: pgmap v1119: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:37:40.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:37:40 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T13:37:40.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:37:40 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T13:37:40.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:37:40 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T13:37:40.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:37:40 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T13:37:40.845 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:37:40 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T13:37:40.845 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:37:40 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T13:37:40.845 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:37:40 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T13:37:40.845 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:37:40 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T13:37:41.212 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:37:41.213 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1455 8 365281 365273 1.96115 0 - 0.0299513 2026-02-23T13:37:41.337 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:37:41.436 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:37:41.434+0000 7efe420d2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:37:41.436 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:37:41.434+0000 7efe420d2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:37:41.448 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:37:41.448 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:37:41.458 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:37:41.458 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:37:41.459 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:37:41.470 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:37:41.470 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:37:41.499 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:37:41.499 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:37:42.212 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1456 8 365281 365273 1.9598 0 - 0.0299513 2026-02-23T13:37:42.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:37:42 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T13:37:42.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:37:42 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T13:37:42.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:37:42 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T13:37:42.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:37:42 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T13:37:42.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:37:42 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-23T13:37:42.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:37:42 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2026-02-23T13:37:42.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:37:42 trial068 ceph-mon[38805]: pgmap v1120: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:37:42.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:37:42 trial068 ceph-mon[38805]: pgmap v1121: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:37:42.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:37:42 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T13:37:42.595 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:37:42 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2026-02-23T13:37:42.595 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:37:42 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2679121200' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:37:43.212 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1457 8 365281 365273 1.95846 0 - 0.0299513 2026-02-23T13:37:43.500 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:37:43.620 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:37:43.723 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:37:43.721+0000 7f08e480a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:37:43.723 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:37:43.722+0000 7f08e480a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:37:43.735 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:37:43.735 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:37:43.745 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:37:43.745 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:37:43.745 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:37:43.756 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:37:43.756 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:37:43.787 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:37:43.787 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:37:44.093 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:37:43 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:37:43] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:37:44.213 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1458 8 365281 365273 1.95711 0 - 0.0299513 2026-02-23T13:37:44.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:37:43 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:37:43.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:37:44.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:37:44 trial068 ceph-mon[38805]: pgmap v1122: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:37:44.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:37:44 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1314602657' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:37:45.213 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1459 8 365281 365273 1.95577 0 - 0.0299513 2026-02-23T13:37:45.789 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:37:45.908 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:37:46.009 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:37:46.007+0000 7f12e61ac640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:37:46.009 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:37:46.007+0000 7f12e61ac640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:37:46.020 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:37:46.020 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:37:46.030 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:37:46.030 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:37:46.030 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:37:46.040 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:37:46.041 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:37:46.071 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:37:46.071 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:37:46.213 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T13:37:46.212650+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T13:37:46.213 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T13:37:46.213 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1460 8 365281 365273 1.95443 0 - 0.0299513 2026-02-23T13:37:46.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:37:46 trial068 ceph-mon[38805]: pgmap v1123: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:37:46.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:37:46 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/4083095557' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:37:47.213 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1461 8 365281 365273 1.95309 0 - 0.0299513 2026-02-23T13:37:48.072 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:37:48.188 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:37:48.213 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1462 8 365281 365273 1.95176 0 - 0.0299513 2026-02-23T13:37:48.293 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:37:48.291+0000 7f4fbd7ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:37:48.293 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:37:48.291+0000 7f4fbd7ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:37:48.305 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:37:48.305 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:37:48.314 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:37:48.315 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:37:48.315 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:37:48.324 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:37:48.325 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:37:48.353 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:37:48.354 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:37:48.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:37:48 trial068 ceph-mon[38805]: pgmap v1124: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:37:48.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:37:48 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2462460462' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:37:49.213 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1463 8 365281 365273 1.95042 0 - 0.0299513 2026-02-23T13:37:50.213 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1464 8 365281 365273 1.94909 0 - 0.0299513 2026-02-23T13:37:50.355 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:37:50.473 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:37:50.494 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:37:50 trial068 ceph-mon[38805]: pgmap v1125: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:37:50.567 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:37:50.565+0000 7fb130473640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:37:50.567 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:37:50.565+0000 7fb130473640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:37:50.579 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:37:50.579 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:37:50.589 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:37:50.590 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:37:50.590 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:37:50.599 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:37:50.600 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:37:50.630 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:37:50.631 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:37:51.213 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1465 8 365281 365273 1.94776 0 - 0.0299513 2026-02-23T13:37:51.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:37:51 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3865803000' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:37:52.213 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1466 8 365281 365273 1.94643 0 - 0.0299513 2026-02-23T13:37:52.494 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:37:52 trial068 ceph-mon[38805]: pgmap v1126: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:37:52.632 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:37:52.750 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:37:52.852 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:37:52.850+0000 7fd69e1f6640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:37:52.852 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:37:52.850+0000 7fd69e1f6640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:37:52.864 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:37:52.864 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:37:52.875 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:37:52.876 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:37:52.876 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:37:52.886 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:37:52.887 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:37:52.917 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:37:52.918 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:37:53.213 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1467 8 365281 365273 1.94511 0 - 0.0299513 2026-02-23T13:37:53.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:37:53 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3488311492' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:37:54.213 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1468 8 365281 365273 1.94378 0 - 0.0299513 2026-02-23T13:37:54.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:37:54 trial068 ceph-mon[38805]: pgmap v1127: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:37:54.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:37:53 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:37:53] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:37:54.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:37:53 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:37:53.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:37:54.919 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:37:55.036 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:37:55.139 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:37:55.137+0000 7f703af66640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:37:55.139 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:37:55.138+0000 7f703af66640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:37:55.151 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:37:55.151 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:37:55.161 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:37:55.162 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:37:55.162 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:37:55.171 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:37:55.172 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:37:55.201 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:37:55.202 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:37:55.213 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1469 8 365281 365273 1.94246 0 - 0.0299513 2026-02-23T13:37:55.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:37:55 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1908600701' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:37:56.213 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1470 8 365281 365273 1.94114 0 - 0.0299513 2026-02-23T13:37:56.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:37:56 trial068 ceph-mon[38805]: pgmap v1128: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:37:57.203 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:37:57.213 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1471 8 365281 365273 1.93982 0 - 0.0299513 2026-02-23T13:37:57.320 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:37:57.426 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:37:57.424+0000 7fb57df1d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:37:57.426 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:37:57.424+0000 7fb57df1d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:37:57.438 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:37:57.438 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:37:57.448 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:37:57.448 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:37:57.448 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:37:57.458 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:37:57.458 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:37:57.491 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:37:57.491 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:37:58.214 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1472 8 365281 365273 1.9385 0 - 0.0299513 2026-02-23T13:37:58.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:37:58 trial068 ceph-mon[38805]: pgmap v1129: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:37:58.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:37:58 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/639252028' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:37:59.214 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1473 8 365281 365273 1.93718 0 - 0.0299513 2026-02-23T13:37:59.492 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:37:59.612 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:37:59.634 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:37:59 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T13:37:59.634 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:37:59 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T13:37:59.709 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:37:59.707+0000 7f27a5da8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:37:59.709 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:37:59.708+0000 7f27a5da8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:37:59.721 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:37:59.722 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:37:59.731 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:37:59.732 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:37:59.732 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:37:59.741 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:37:59.742 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:37:59.773 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:37:59.773 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:38:00.214 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1474 8 365281 365273 1.93587 0 - 0.0299513 2026-02-23T13:38:00.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:38:00 trial068 ceph-mon[38805]: pgmap v1130: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:38:00.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:38:00 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3407200522' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:38:01.214 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1475 8 365281 365273 1.93456 0 - 0.0299513 2026-02-23T13:38:01.774 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:38:01.891 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:38:01.989 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:38:01.987+0000 7f8b224ac640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:38:01.989 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:38:01.987+0000 7f8b224ac640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:38:02.000 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:38:02.001 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:38:02.011 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:38:02.011 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:38:02.011 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:38:02.021 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:38:02.021 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:38:02.051 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:38:02.051 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:38:02.214 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1476 8 365281 365273 1.93325 0 - 0.0299513 2026-02-23T13:38:02.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:38:02 trial068 ceph-mon[38805]: pgmap v1131: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:38:02.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:38:02 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2068468586' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:38:03.214 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1477 8 365281 365273 1.93194 0 - 0.0299513 2026-02-23T13:38:03.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:38:03 trial068 ceph-mon[38805]: 5.8 scrub starts 2026-02-23T13:38:03.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:38:03 trial068 ceph-mon[38805]: 5.8 scrub ok 2026-02-23T13:38:04.052 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:38:04.172 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:38:04.214 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1478 8 365281 365273 1.93063 0 - 0.0299513 2026-02-23T13:38:04.224 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:38:03 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:38:03] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:38:04.264 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:38:03 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:38:03.974Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:38:04.272 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:38:04.270+0000 7fca10b29640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:38:04.272 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:38:04.270+0000 7fca10b29640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:38:04.284 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:38:04.284 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:38:04.294 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:38:04.294 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:38:04.294 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:38:04.305 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:38:04.305 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:38:04.333 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:38:04.333 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:38:04.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:38:04 trial068 ceph-mon[38805]: pgmap v1132: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:38:04.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:38:04 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/576453784' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:38:05.214 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1479 8 365281 365273 1.92932 0 - 0.0299513 2026-02-23T13:38:06.214 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T13:38:06.214179+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T13:38:06.214 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T13:38:06.215 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1480 8 365281 365273 1.92802 0 - 0.0299513 2026-02-23T13:38:06.335 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:38:06.454 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:38:06.557 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:38:06.555+0000 7fc3ca3d8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:38:06.558 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:38:06.555+0000 7fc3ca3d8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:38:06.569 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:38:06.570 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:38:06.579 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:38:06.580 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:38:06.580 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:38:06.590 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:38:06.590 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:38:06.613 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:38:06 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T13:38:06.613 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:38:06 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T13:38:06.614 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:38:06 trial068 ceph-mon[38805]: pgmap v1133: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:38:06.636 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:38:06.637 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:38:07.215 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1481 8 365281 365273 1.92672 0 - 0.0299513 2026-02-23T13:38:07.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:38:07 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/879910289' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:38:08.214 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1482 8 365281 365273 1.92542 0 - 0.0299513 2026-02-23T13:38:08.638 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:38:08.663 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:38:08 trial068 ceph-mon[38805]: pgmap v1134: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:38:08.756 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:38:08.856 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:38:08.854+0000 7ff784609640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:38:08.856 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:38:08.854+0000 7ff784609640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:38:08.868 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:38:08.868 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:38:08.878 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:38:08.878 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:38:08.879 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:38:08.889 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:38:08.889 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:38:08.919 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:38:08.919 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:38:09.214 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1483 8 365281 365273 1.92412 0 - 0.0299513 2026-02-23T13:38:09.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:38:09 trial068 ceph-mon[38805]: 5.a scrub starts 2026-02-23T13:38:09.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:38:09 trial068 ceph-mon[38805]: 5.a scrub ok 2026-02-23T13:38:09.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:38:09 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/833161428' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:38:10.214 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1484 8 365281 365273 1.92282 0 - 0.0299513 2026-02-23T13:38:10.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:38:10 trial068 ceph-mon[38805]: pgmap v1135: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:38:10.920 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:38:11.041 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:38:11.141 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:38:11.139+0000 7f44f0b84640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:38:11.141 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:38:11.140+0000 7f44f0b84640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:38:11.153 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:38:11.153 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:38:11.163 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:38:11.163 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:38:11.163 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:38:11.173 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:38:11.173 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:38:11.203 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:38:11.204 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:38:11.214 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1485 8 365281 365273 1.92153 0 - 0.0299513 2026-02-23T13:38:11.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:38:11 trial068 ceph-mon[38805]: 5.7 scrub starts 2026-02-23T13:38:11.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:38:11 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/4293441537' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:38:12.215 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1486 8 365281 365273 1.92024 0 - 0.0299513 2026-02-23T13:38:12.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:38:12 trial068 ceph-mon[38805]: 5.7 scrub ok 2026-02-23T13:38:12.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:38:12 trial068 ceph-mon[38805]: pgmap v1136: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:38:13.205 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:38:13.215 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1487 8 365281 365273 1.91894 0 - 0.0299513 2026-02-23T13:38:13.323 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:38:13.418 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:38:13.416+0000 7fb08144d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:38:13.419 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:38:13.417+0000 7fb08144d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:38:13.430 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:38:13.430 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:38:13.440 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:38:13.441 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:38:13.441 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:38:13.452 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:38:13.452 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:38:13.483 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:38:13.484 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:38:14.215 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1488 8 365281 365273 1.91766 0 - 0.0299513 2026-02-23T13:38:14.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:38:13 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:38:13] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:38:14.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:38:13 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:38:13.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:38:14.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:38:14 trial068 ceph-mon[38805]: pgmap v1137: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:38:14.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:38:14 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3642288774' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:38:15.215 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1489 8 365281 365273 1.91637 0 - 0.0299513 2026-02-23T13:38:15.484 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:38:15.605 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:38:15.649 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:38:15 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T13:38:15.649 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:38:15 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T13:38:15.706 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:38:15.704+0000 7f46aa905640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:38:15.706 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:38:15.705+0000 7f46aa905640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:38:15.718 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:38:15.718 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:38:15.728 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:38:15.729 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:38:15.729 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:38:15.738 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:38:15.738 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:38:15.766 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:38:15.767 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:38:16.215 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1490 8 365281 365273 1.91508 0 - 0.0299513 2026-02-23T13:38:16.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:38:16 trial068 ceph-mon[38805]: pgmap v1138: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:38:16.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:38:16 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1430258097' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:38:17.215 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1491 8 365281 365273 1.9138 0 - 0.0299513 2026-02-23T13:38:17.767 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:38:17.883 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:38:17.982 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:38:17.980+0000 7f4016999640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:38:17.982 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:38:17.980+0000 7f4016999640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:38:17.994 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:38:17.995 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:38:18.004 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:38:18.005 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:38:18.005 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:38:18.015 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:38:18.016 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:38:18.043 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:38:18.044 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:38:18.215 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1492 8 365281 365273 1.91251 0 - 0.0299513 2026-02-23T13:38:18.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:38:18 trial068 ceph-mon[38805]: pgmap v1139: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:38:18.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:38:18 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/4088360622' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:38:19.215 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1493 8 365281 365273 1.91123 0 - 0.0299513 2026-02-23T13:38:20.045 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:38:20.164 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:38:20.216 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1494 8 365281 365273 1.90995 0 - 0.0299513 2026-02-23T13:38:20.260 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:38:20.258+0000 7f7c11a9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:38:20.261 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:38:20.259+0000 7f7c11a9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:38:20.272 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:38:20.272 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:38:20.282 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:38:20.282 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:38:20.282 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:38:20.292 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:38:20.292 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:38:20.321 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:38:20 trial068 ceph-mon[38805]: pgmap v1140: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:38:20.323 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:38:20.323 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:38:21.215 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1495 8 365281 365273 1.90868 0 - 0.0299513 2026-02-23T13:38:21.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:38:21 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1177347304' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:38:22.216 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1496 8 365281 365273 1.9074 0 - 0.0299513 2026-02-23T13:38:22.324 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:38:22.443 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:38:22.485 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:38:22 trial068 ceph-mon[38805]: pgmap v1141: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:38:22.542 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:38:22.540+0000 7fb6b7e95640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:38:22.542 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:38:22.540+0000 7fb6b7e95640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:38:22.554 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:38:22.554 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:38:22.564 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:38:22.564 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:38:22.564 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:38:22.574 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:38:22.575 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:38:22.602 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:38:22.603 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:38:23.216 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1497 8 365281 365273 1.90613 0 - 0.0299513 2026-02-23T13:38:23.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:38:23 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2162004995' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:38:24.216 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1498 8 365281 365273 1.90485 0 - 0.0299513 2026-02-23T13:38:24.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:38:24 trial068 ceph-mon[38805]: pgmap v1142: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:38:24.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:38:23 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:38:23] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:38:24.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:38:23 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:38:23.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:38:24.603 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:38:24.722 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:38:24.824 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:38:24.822+0000 7fb4fde2f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:38:24.824 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:38:24.822+0000 7fb4fde2f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:38:24.835 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:38:24.835 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:38:24.845 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:38:24.846 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:38:24.846 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:38:24.856 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:38:24.856 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:38:24.884 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:38:24.885 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:38:25.216 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1499 8 365281 365273 1.90358 0 - 0.0299513 2026-02-23T13:38:25.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:38:25 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/817907903' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:38:26.216 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T13:38:26.215707+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T13:38:26.216 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T13:38:26.216 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1500 8 365281 365273 1.90231 0 - 0.0299513 2026-02-23T13:38:26.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:38:26 trial068 ceph-mon[38805]: pgmap v1143: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:38:26.886 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:38:27.003 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:38:27.103 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:38:27.101+0000 7fd51b2c3640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:38:27.103 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:38:27.101+0000 7fd51b2c3640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:38:27.115 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:38:27.115 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:38:27.125 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:38:27.125 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:38:27.125 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:38:27.135 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:38:27.135 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:38:27.165 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:38:27.165 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:38:27.216 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1501 8 365281 365273 1.90105 0 - 0.0299513 2026-02-23T13:38:27.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:38:27 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1062504334' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:38:28.216 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1502 8 365281 365273 1.89978 0 - 0.0299513 2026-02-23T13:38:28.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:38:28 trial068 ceph-mon[38805]: pgmap v1144: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:38:29.166 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:38:29.216 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1503 8 365281 365273 1.89852 0 - 0.0299513 2026-02-23T13:38:29.284 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:38:29.385 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:38:29.383+0000 7fa1a64fe640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:38:29.385 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:38:29.384+0000 7fa1a64fe640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:38:29.397 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:38:29.398 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:38:29.408 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:38:29.409 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:38:29.409 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:38:29.419 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:38:29.419 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:38:29.449 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:38:29.450 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:38:30.216 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1504 8 365281 365273 1.89725 0 - 0.0299513 2026-02-23T13:38:30.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:38:30 trial068 ceph-mon[38805]: pgmap v1145: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:38:30.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:38:30 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3850818131' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:38:31.216 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1505 8 365281 365273 1.89599 0 - 0.0299513 2026-02-23T13:38:31.450 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:38:31.569 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:38:31.667 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:38:31.665+0000 7ff383fa9640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:38:31.667 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:38:31.665+0000 7ff383fa9640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:38:31.679 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:38:31.679 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:38:31.689 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:38:31.690 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:38:31.690 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:38:31.700 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:38:31.700 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:38:31.730 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:38:31.731 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:38:32.216 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1506 8 365281 365273 1.89474 0 - 0.0299513 2026-02-23T13:38:32.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:38:32 trial068 ceph-mon[38805]: pgmap v1146: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:38:32.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:38:32 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3910159747' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:38:33.216 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1507 8 365281 365273 1.89348 0 - 0.0299513 2026-02-23T13:38:33.731 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:38:33.850 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:38:33.957 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:38:33.955+0000 7f82db7d9640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:38:33.957 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:38:33.955+0000 7f82db7d9640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:38:33.969 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:38:33.969 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:38:33.979 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:38:33.979 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:38:33.979 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:38:33.989 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:38:33.989 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:38:34.007 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:38:33 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:38:33] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:38:34.020 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:38:34.020 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:38:34.216 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1508 8 365281 365273 1.89222 0 - 0.0299513 2026-02-23T13:38:34.264 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:38:33 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:38:33.972Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:38:34.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:38:34 trial068 ceph-mon[38805]: pgmap v1147: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:38:34.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:38:34 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3978880542' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:38:35.216 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1509 8 365281 365273 1.89097 0 - 0.0299513 2026-02-23T13:38:36.021 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:38:36.141 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:38:36.217 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1510 8 365281 365273 1.88972 0 - 0.0299513 2026-02-23T13:38:36.244 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:38:36.241+0000 7fd51da93640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:38:36.244 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:38:36.242+0000 7fd51da93640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:38:36.255 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:38:36.255 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:38:36.265 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:38:36.265 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:38:36.265 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:38:36.275 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:38:36.275 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:38:36.306 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:38:36.307 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:38:36.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:38:36 trial068 ceph-mon[38805]: pgmap v1148: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:38:36.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:38:36 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1820022835' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:38:37.217 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1511 8 365281 365273 1.88847 0 - 0.0299513 2026-02-23T13:38:38.217 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1512 8 365281 365273 1.88722 0 - 0.0299513 2026-02-23T13:38:38.308 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:38:38.427 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:38:38.541 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:38:38.539+0000 7ff0275f2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:38:38.541 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:38:38.539+0000 7ff0275f2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:38:38.553 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:38:38.553 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:38:38.563 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:38:38.564 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:38:38.564 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:38:38.573 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:38:38.574 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:38:38.586 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:38:38 trial068 ceph-mon[38805]: pgmap v1149: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:38:38.604 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:38:38.605 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:38:39.217 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1513 8 365281 365273 1.88597 0 - 0.0299513 2026-02-23T13:38:39.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:38:39 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2913607503' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:38:40.217 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1514 8 365281 365273 1.88472 0 - 0.0299513 2026-02-23T13:38:40.606 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:38:40.723 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:38:40.745 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:38:40 trial068 ceph-mon[38805]: pgmap v1150: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:38:40.821 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:38:40.819+0000 7f3229853640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:38:40.821 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:38:40.819+0000 7f3229853640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:38:40.833 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:38:40.833 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:38:40.843 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:38:40.844 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:38:40.844 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:38:40.853 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:38:40.854 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:38:40.883 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:38:40.884 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:38:41.217 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1515 8 365281 365273 1.88348 0 - 0.0299513 2026-02-23T13:38:41.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:38:41 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2137350141' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:38:41.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:38:41 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2026-02-23T13:38:42.217 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1516 8 365281 365273 1.88224 0 - 0.0299513 2026-02-23T13:38:42.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:38:42 trial068 ceph-mon[38805]: pgmap v1151: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:38:42.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:38:42 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-23T13:38:42.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:38:42 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2026-02-23T13:38:42.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:38:42 trial068 ceph-mon[38805]: pgmap v1152: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:38:42.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:38:42 trial068 ceph-mon[38805]: pgmap v1153: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:38:42.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:38:42 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T13:38:42.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:38:42 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2026-02-23T13:38:42.885 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:38:43.001 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:38:43.101 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:38:43.099+0000 7f88fb292640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:38:43.101 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:38:43.099+0000 7f88fb292640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:38:43.112 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:38:43.113 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:38:43.123 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:38:43.123 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:38:43.123 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:38:43.135 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:38:43.135 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:38:43.163 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:38:43.164 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:38:43.217 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1517 8 365281 365273 1.881 0 - 0.0299513 2026-02-23T13:38:43.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:38:43 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1633367760' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:38:44.217 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1518 8 365281 365273 1.87976 0 - 0.0299513 2026-02-23T13:38:44.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:38:43 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:38:43] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:38:44.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:38:43 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:38:43.974Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:38:44.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:38:44 trial068 ceph-mon[38805]: 5.5 scrub starts 2026-02-23T13:38:44.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:38:44 trial068 ceph-mon[38805]: pgmap v1154: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:38:44.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:38:44 trial068 ceph-mon[38805]: 5.5 scrub ok 2026-02-23T13:38:45.165 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:38:45.217 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1519 8 365281 365273 1.87852 0 - 0.0299513 2026-02-23T13:38:45.286 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:38:45.383 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:38:45.381+0000 7fd33d124640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:38:45.384 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:38:45.382+0000 7fd33d124640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:38:45.396 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:38:45.396 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:38:45.406 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:38:45.407 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:38:45.407 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:38:45.417 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:38:45.418 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:38:45.445 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:38:45 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2744409316' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:38:45.446 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:38:45.447 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:38:46.217 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T13:38:46.217252+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T13:38:46.218 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T13:38:46.218 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1520 8 365281 365273 1.87728 0 - 0.0299513 2026-02-23T13:38:46.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:38:46 trial068 ceph-mon[38805]: pgmap v1155: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:38:47.217 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1521 8 365281 365273 1.87605 0 - 0.0299513 2026-02-23T13:38:47.448 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:38:47.568 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:38:47.674 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:38:47 trial068 ceph-mon[38805]: osdmap e202: 4 total, 3 up, 4 in 2026-02-23T13:38:47.675 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:38:47.672+0000 7f52943bc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:38:47.675 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:38:47.672+0000 7f52943bc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:38:47.686 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:38:47.687 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:38:47.696 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:38:47.697 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:38:47.697 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:38:47.707 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:38:47.707 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:38:47.737 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:38:47.738 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:38:48.217 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1522 8 365281 365273 1.87482 0 - 0.0299513 2026-02-23T13:38:48.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:38:48 trial068 ceph-mon[38805]: pgmap v1157: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:38:48.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:38:48 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/132561638' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:38:49.218 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1523 8 365281 365273 1.87359 0 - 0.0299513 2026-02-23T13:38:49.739 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:38:49.856 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:38:49.958 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:38:49.956+0000 7fc8de789640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:38:49.958 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:38:49.956+0000 7fc8de789640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:38:49.970 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:38:49.971 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:38:49.981 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:38:49.981 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:38:49.981 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:38:49.992 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:38:49.993 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:38:50.022 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:38:50.023 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:38:50.218 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1524 8 365281 365273 1.87236 0 - 0.0299513 2026-02-23T13:38:50.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:38:50 trial068 ceph-mon[38805]: pgmap v1158: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:38:50.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:38:50 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3104520988' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:38:51.218 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1525 8 365281 365273 1.87113 0 - 0.0299513 2026-02-23T13:38:52.024 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:38:52.050 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:38:51 trial068 ceph-mon[38805]: pgmap v1159: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:38:52.143 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:38:52.218 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1526 8 365281 365273 1.8699 0 - 0.0299513 2026-02-23T13:38:52.250 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:38:52.249+0000 7f9219142640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:38:52.251 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:38:52.249+0000 7f9219142640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:38:52.262 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:38:52.263 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:38:52.272 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:38:52.273 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:38:52.273 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:38:52.283 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:38:52.283 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:38:52.312 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:38:52.312 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:38:53.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:38:52 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1664992013' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:38:53.218 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1527 8 365281 365273 1.86868 0 - 0.0299513 2026-02-23T13:38:53.973 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:38:53 trial068 ceph-mon[38805]: pgmap v1160: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:38:54.218 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1528 8 365281 365273 1.86746 0 - 0.0299513 2026-02-23T13:38:54.313 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:38:54.338 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:38:53 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:38:53] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:38:54.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:38:53 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:38:53.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:38:54.433 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:38:54.533 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:38:54.530+0000 7fc7ffcc2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:38:54.533 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:38:54.531+0000 7fc7ffcc2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:38:54.545 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:38:54.545 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:38:54.556 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:38:54.556 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:38:54.556 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:38:54.567 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:38:54.567 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:38:54.596 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:38:54.596 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:38:55.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:38:54 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2278748816' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:38:55.218 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1529 8 365281 365273 1.86623 0 - 0.0299513 2026-02-23T13:38:56.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:38:55 trial068 ceph-mon[38805]: pgmap v1161: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:38:56.218 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1530 8 365281 365273 1.86501 0 - 0.0299513 2026-02-23T13:38:56.598 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:38:56.719 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:38:56.822 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:38:56.821+0000 7f9b01e59640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:38:56.823 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:38:56.821+0000 7f9b01e59640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:38:56.836 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:38:56.836 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:38:56.847 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:38:56.847 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:38:56.847 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:38:56.858 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:38:56.858 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:38:56.881 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:38:56 trial068 ceph-mon[38805]: 5.e scrub starts 2026-02-23T13:38:56.909 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:38:56.909 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:38:57.218 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1531 8 365281 365273 1.8638 0 - 0.0299513 2026-02-23T13:38:58.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:38:57 trial068 ceph-mon[38805]: 5.e scrub ok 2026-02-23T13:38:58.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:38:57 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2186583878' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:38:58.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:38:57 trial068 ceph-mon[38805]: pgmap v1162: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:38:58.218 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1532 8 365281 365273 1.86258 0 - 0.0299513 2026-02-23T13:38:58.910 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:38:59.030 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:38:59.135 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:38:59.133+0000 7f3e36ff3640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:38:59.135 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:38:59.134+0000 7f3e36ff3640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:38:59.147 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:38:59.148 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:38:59.157 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:38:59.158 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:38:59.158 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:38:59.168 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:38:59.168 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:38:59.198 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:38:59.199 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:38:59.218 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1533 8 365281 365273 1.86136 0 - 0.0299513 2026-02-23T13:38:59.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:38:59 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/157670048' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:39:00.218 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1534 8 365281 365273 1.86015 0 - 0.0299513 2026-02-23T13:39:00.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:39:00 trial068 ceph-mon[38805]: pgmap v1163: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:39:01.199 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:39:01.218 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1535 8 365281 365273 1.85894 0 - 0.0299513 2026-02-23T13:39:01.319 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:39:01.416 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:39:01.415+0000 7f4f5b84f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:39:01.417 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:39:01.415+0000 7f4f5b84f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:39:01.428 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:39:01.429 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:39:01.439 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:39:01.439 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:39:01.439 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:39:01.449 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:39:01.449 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:39:01.478 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:39:01 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3859235326' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:39:01.479 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:39:01.479 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:39:02.218 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1536 8 365281 365273 1.85773 0 - 0.0299513 2026-02-23T13:39:02.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:39:02 trial068 ceph-mon[38805]: pgmap v1164: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:39:03.218 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1537 8 365281 365273 1.85652 0 - 0.0299513 2026-02-23T13:39:03.480 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:39:03.598 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:39:03.692 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:39:03.690+0000 7f8be5b7a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:39:03.693 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:39:03.690+0000 7f8be5b7a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:39:03.704 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:39:03.705 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:39:03.715 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:39:03.715 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:39:03.716 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:39:03.725 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:39:03.725 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:39:03.756 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:39:03.756 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:39:03.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:39:03 trial068 ceph-mon[38805]: pgmap v1165: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:39:04.219 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1538 8 365281 365273 1.85531 0 - 0.0299513 2026-02-23T13:39:04.265 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:39:03 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:39:03.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:39:04.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:39:03 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:39:03] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:39:05.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:39:04 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3643571809' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:39:05.219 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1539 8 365281 365273 1.85411 0 - 0.0299513 2026-02-23T13:39:05.757 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:39:05.874 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:39:05.922 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:39:05 trial068 ceph-mon[38805]: pgmap v1166: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:39:05.979 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:39:05.978+0000 7fe95ed58640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:39:05.980 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:39:05.978+0000 7fe95ed58640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:39:05.991 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:39:05.992 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:39:06.001 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:39:06.002 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:39:06.002 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:39:06.011 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:39:06.012 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:39:06.041 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:39:06.042 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:39:06.219 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T13:39:06.218696+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T13:39:06.219 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T13:39:06.219 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1540 8 365281 365273 1.8529 0 - 0.0299513 2026-02-23T13:39:07.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:39:06 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/4151681185' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:39:07.219 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1541 8 365281 365273 1.8517 0 - 0.0299513 2026-02-23T13:39:08.044 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:39:08.071 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:39:07 trial068 ceph-mon[38805]: pgmap v1167: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:39:08.162 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:39:08.219 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1542 8 365281 365273 1.8505 0 - 0.0299513 2026-02-23T13:39:08.263 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:39:08.261+0000 7f38bf26e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:39:08.263 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:39:08.261+0000 7f38bf26e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:39:08.275 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:39:08.275 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:39:08.285 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:39:08.285 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:39:08.286 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:39:08.296 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:39:08.296 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:39:08.326 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:39:08.327 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:39:09.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:39:08 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/269845424' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:39:09.219 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1543 8 365281 365273 1.8493 0 - 0.0299513 2026-02-23T13:39:10.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:39:09 trial068 ceph-mon[38805]: pgmap v1168: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:39:10.219 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1544 8 365281 365273 1.8481 0 - 0.0299513 2026-02-23T13:39:10.328 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:39:10.446 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:39:10.544 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:39:10.543+0000 7fbf3fd77640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:39:10.544 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:39:10.543+0000 7fbf3fd77640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:39:10.557 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:39:10.557 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:39:10.568 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:39:10.568 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:39:10.568 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:39:10.578 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:39:10.578 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:39:10.608 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:39:10.608 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:39:11.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:39:10 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/699736319' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:39:11.219 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1545 8 365281 365273 1.84691 0 - 0.0299513 2026-02-23T13:39:12.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:39:11 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T13:39:12.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:39:11 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T13:39:12.095 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:39:11 trial068 ceph-mon[38805]: pgmap v1169: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:39:12.219 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1546 8 365281 365273 1.84571 0 - 0.0299513 2026-02-23T13:39:12.610 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:39:12.728 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:39:12.827 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:39:12.826+0000 7f853932d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:39:12.827 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:39:12.826+0000 7f853932d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:39:12.839 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:39:12.839 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:39:12.850 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:39:12.850 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:39:12.850 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:39:12.860 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:39:12.860 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:39:12.890 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:39:12 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T13:39:12.890 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:39:12 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T13:39:12.891 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:39:12.891 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:39:13.219 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1547 8 365281 365273 1.84452 0 - 0.0299513 2026-02-23T13:39:13.974 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:39:13 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/423730469' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:39:13.974 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:39:13 trial068 ceph-mon[38805]: 5.8 scrub starts 2026-02-23T13:39:13.974 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:39:13 trial068 ceph-mon[38805]: pgmap v1170: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:39:13.974 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:39:13 trial068 ceph-mon[38805]: 5.8 scrub ok 2026-02-23T13:39:14.220 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1548 8 365281 365273 1.84333 0 - 0.0299513 2026-02-23T13:39:14.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:39:13 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:39:13] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:39:14.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:39:13 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:39:13.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:39:14.891 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:39:15.013 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:39:15.119 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:39:15.116+0000 7fa6ab23e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:39:15.119 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:39:15.117+0000 7fa6ab23e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:39:15.131 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:39:15.131 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:39:15.141 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:39:15.142 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:39:15.142 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:39:15.151 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:39:15.152 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:39:15.181 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:39:15.182 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:39:15.220 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1549 8 365281 365273 1.84214 0 - 0.0299513 2026-02-23T13:39:15.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:39:15 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/21151279' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:39:16.220 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1550 8 365281 365273 1.84095 0 - 0.0299513 2026-02-23T13:39:16.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:39:16 trial068 ceph-mon[38805]: pgmap v1171: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:39:17.184 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:39:17.220 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1551 8 365281 365273 1.83976 0 - 0.0299513 2026-02-23T13:39:17.305 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:39:17.402 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:39:17.400+0000 7fbe615c6640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:39:17.402 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:39:17.400+0000 7fbe615c6640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:39:17.414 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:39:17.414 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:39:17.424 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:39:17.424 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:39:17.424 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:39:17.434 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:39:17.435 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:39:17.461 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:39:17 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2869467241' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:39:17.462 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:39:17.463 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:39:18.220 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1552 8 365281 365273 1.83858 0 - 0.0299513 2026-02-23T13:39:18.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:39:18 trial068 ceph-mon[38805]: pgmap v1172: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:39:19.220 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1553 8 365281 365273 1.83739 0 - 0.0299513 2026-02-23T13:39:19.464 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:39:19.581 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:39:19.678 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:39:19.676+0000 7f83eb5da640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:39:19.678 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:39:19.676+0000 7f83eb5da640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:39:19.690 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:39:19.690 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:39:19.700 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:39:19.701 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:39:19.701 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:39:19.711 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:39:19.712 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:39:19.743 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:39:19.743 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:39:20.220 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1554 8 365281 365273 1.83621 0 - 0.0299513 2026-02-23T13:39:20.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:39:20 trial068 ceph-mon[38805]: pgmap v1173: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:39:20.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:39:20 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/891795936' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:39:21.220 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1555 8 365281 365273 1.83503 0 - 0.0299513 2026-02-23T13:39:21.744 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:39:21.863 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:39:21.920 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:39:21 trial068 ceph-mon[38805]: pgmap v1174: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:39:21.965 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:39:21.963+0000 7f1e2537f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:39:21.966 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:39:21.964+0000 7f1e2537f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:39:21.977 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:39:21.977 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:39:21.987 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:39:21.987 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:39:21.988 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:39:21.997 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:39:21.997 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:39:22.028 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:39:22.029 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:39:22.220 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1556 8 365281 365273 1.83385 0 - 0.0299513 2026-02-23T13:39:23.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:39:22 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3064771277' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:39:23.220 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1557 8 365281 365273 1.83267 0 - 0.0299513 2026-02-23T13:39:23.974 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:39:23 trial068 ceph-mon[38805]: 5.7 scrub starts 2026-02-23T13:39:23.974 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:39:23 trial068 ceph-mon[38805]: 5.7 scrub ok 2026-02-23T13:39:23.974 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:39:23 trial068 ceph-mon[38805]: pgmap v1175: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:39:24.030 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:39:24.147 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:39:24.220 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1558 8 365281 365273 1.8315 0 - 0.0299513 2026-02-23T13:39:24.249 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:39:24.248+0000 7fd29d589640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:39:24.250 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:39:23 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:39:23] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:39:24.250 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:39:24.248+0000 7fd29d589640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:39:24.262 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:39:24.262 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:39:24.272 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:39:24.272 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:39:24.273 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:39:24.283 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:39:24.283 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:39:24.316 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:39:24.317 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:39:24.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:39:23 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:39:23.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:39:25.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:39:24 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/4131511997' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:39:25.220 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1559 8 365281 365273 1.83032 0 - 0.0299513 2026-02-23T13:39:26.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:39:25 trial068 ceph-mon[38805]: pgmap v1176: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:39:26.220 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T13:39:26.220229+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T13:39:26.220 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T13:39:26.220 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1560 8 365281 365273 1.82915 0 - 0.0299513 2026-02-23T13:39:26.318 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:39:26.434 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:39:26.533 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:39:26.531+0000 7f7d65e45640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:39:26.533 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:39:26.531+0000 7f7d65e45640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:39:26.544 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:39:26.545 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:39:26.554 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:39:26.555 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:39:26.555 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:39:26.565 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:39:26.566 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:39:26.596 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:39:26.596 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:39:27.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:39:26 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1768661647' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:39:27.220 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1561 8 365281 365273 1.82798 0 - 0.0299513 2026-02-23T13:39:28.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:39:27 trial068 ceph-mon[38805]: pgmap v1177: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:39:28.221 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1562 8 365281 365273 1.82681 0 - 0.0299513 2026-02-23T13:39:28.597 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:39:28.714 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:39:28.817 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:39:28.815+0000 7f074e2e8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:39:28.817 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:39:28.815+0000 7f074e2e8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:39:28.828 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:39:28.829 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:39:28.839 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:39:28.839 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:39:28.840 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:39:28.849 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:39:28.850 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:39:28.880 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:39:28 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3601189513' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:39:28.881 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:39:28.882 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:39:29.221 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1563 8 365281 365273 1.82564 0 - 0.0299513 2026-02-23T13:39:30.221 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1564 8 365281 365273 1.82447 0 - 0.0299513 2026-02-23T13:39:30.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:39:30 trial068 ceph-mon[38805]: pgmap v1178: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:39:30.883 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:39:31.002 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:39:31.099 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:39:31.097+0000 7f7c13ca5640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:39:31.099 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:39:31.097+0000 7f7c13ca5640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:39:31.111 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:39:31.111 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:39:31.121 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:39:31.121 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:39:31.121 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:39:31.131 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:39:31.131 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:39:31.162 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:39:31.163 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:39:31.221 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1565 8 365281 365273 1.8233 0 - 0.0299513 2026-02-23T13:39:31.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:39:31 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2820595463' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:39:32.221 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1566 8 365281 365273 1.82214 0 - 0.0299513 2026-02-23T13:39:32.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:39:32 trial068 ceph-mon[38805]: pgmap v1179: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:39:33.164 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:39:33.221 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1567 8 365281 365273 1.82098 0 - 0.0299513 2026-02-23T13:39:33.284 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:39:33.380 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:39:33.378+0000 7f72334b3640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:39:33.381 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:39:33.378+0000 7f72334b3640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:39:33.392 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:39:33.392 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:39:33.402 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:39:33.402 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:39:33.403 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:39:33.412 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:39:33.412 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:39:33.441 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:39:33.442 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:39:33.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:39:33 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/775364903' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:39:34.221 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1568 8 365281 365273 1.81982 0 - 0.0299513 2026-02-23T13:39:34.265 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:39:33 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:39:33.974Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:39:34.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:39:33 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:39:33] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:39:34.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:39:34 trial068 ceph-mon[38805]: pgmap v1180: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:39:35.221 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1569 8 365281 365273 1.81866 0 - 0.0299513 2026-02-23T13:39:35.443 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:39:35.562 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:39:35.667 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:39:35.665+0000 7f7a3e863640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:39:35.667 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:39:35.665+0000 7f7a3e863640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:39:35.680 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:39:35.681 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:39:35.690 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:39:35.691 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:39:35.691 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:39:35.700 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:39:35.701 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:39:35.730 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:39:35 trial068 ceph-mon[38805]: pgmap v1181: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:39:35.730 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:39:35 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/238492938' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:39:35.731 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:39:35.732 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:39:36.221 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1570 8 365281 365273 1.8175 0 - 0.0299513 2026-02-23T13:39:37.221 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1571 8 365281 365273 1.81634 0 - 0.0299513 2026-02-23T13:39:37.732 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:39:37.849 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:39:37.949 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:39:37 trial068 ceph-mon[38805]: pgmap v1182: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:39:37.949 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:39:37.947+0000 7fcc2ce14640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:39:37.949 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:39:37.947+0000 7fcc2ce14640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:39:37.960 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:39:37.961 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:39:37.971 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:39:37.971 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:39:37.971 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:39:37.981 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:39:37.981 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:39:38.012 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:39:38.012 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:39:38.222 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1572 8 365281 365273 1.81519 0 - 0.0299513 2026-02-23T13:39:39.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:39:38 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2182538463' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:39:39.221 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1573 8 365281 365273 1.81403 0 - 0.0299513 2026-02-23T13:39:40.013 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:39:40.130 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:39:40.222 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1574 8 365281 365273 1.81288 0 - 0.0299513 2026-02-23T13:39:40.225 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:39:40.223+0000 7f843d381640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:39:40.225 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:39:40.224+0000 7f843d381640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:39:40.237 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:39:40.237 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:39:40.248 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:39:40.248 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:39:40.248 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:39:40.258 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:39:40.258 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:39:40.287 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:39:40.288 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:39:40.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:39:40 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T13:39:40.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:39:40 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T13:39:40.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:39:40 trial068 ceph-mon[38805]: pgmap v1183: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:39:41.222 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1575 8 365281 365273 1.81173 0 - 0.0299513 2026-02-23T13:39:41.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:39:41 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2088666335' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:39:42.222 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1576 8 365281 365273 1.81058 0 - 0.0299513 2026-02-23T13:39:42.289 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:39:42.406 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:39:42.508 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:39:42.506+0000 7f7b11b9c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:39:42.508 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:39:42.507+0000 7f7b11b9c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:39:42.509 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:39:42 trial068 ceph-mon[38805]: 5.a scrub starts 2026-02-23T13:39:42.509 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:39:42 trial068 ceph-mon[38805]: 5.a scrub ok 2026-02-23T13:39:42.509 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:39:42 trial068 ceph-mon[38805]: pgmap v1184: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:39:42.509 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:39:42 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2026-02-23T13:39:42.509 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:39:42 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-23T13:39:42.509 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:39:42 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2026-02-23T13:39:42.509 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:39:42 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T13:39:42.509 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:39:42 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2026-02-23T13:39:42.520 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:39:42.521 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:39:42.530 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:39:42.531 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:39:42.531 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:39:42.543 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:39:42.543 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:39:42.572 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:39:42.573 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:39:43.222 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1577 8 365281 365273 1.80943 0 - 0.0299513 2026-02-23T13:39:43.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:39:43 trial068 ceph-mon[38805]: pgmap v1185: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:39:43.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:39:43 trial068 ceph-mon[38805]: pgmap v1186: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:39:43.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:39:43 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3032629508' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:39:44.222 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1578 8 365281 365273 1.80828 0 - 0.0299513 2026-02-23T13:39:44.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:39:43 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:39:43] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:39:44.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:39:43 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:39:43.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:39:44.573 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:39:44.692 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:39:44.794 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:39:44.792+0000 7f096d50d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:39:44.795 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:39:44.793+0000 7f096d50d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:39:44.807 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:39:44.807 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:39:44.817 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:39:44.818 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:39:44.818 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:39:44.828 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:39:44.828 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:39:44.859 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:39:44.860 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:39:45.222 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1579 8 365281 365273 1.80714 0 - 0.0299513 2026-02-23T13:39:45.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:39:45 trial068 ceph-mon[38805]: pgmap v1187: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:39:45.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:39:45 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2589046347' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:39:46.222 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T13:39:46.222090+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T13:39:46.223 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T13:39:46.223 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1580 8 365281 365273 1.80599 0 - 0.0299513 2026-02-23T13:39:46.861 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:39:46.979 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:39:47.085 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:39:47.083+0000 7fe1cd1ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:39:47.085 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:39:47.084+0000 7fe1cd1ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:39:47.097 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:39:47.097 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:39:47.107 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:39:47.108 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:39:47.108 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:39:47.117 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:39:47.117 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:39:47.147 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:39:47.148 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:39:47.222 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1581 8 365281 365273 1.80485 0 - 0.0299513 2026-02-23T13:39:47.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:39:47 trial068 ceph-mon[38805]: pgmap v1188: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:39:47.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:39:47 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2898042732' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:39:48.222 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1582 8 365281 365273 1.80371 0 - 0.0299513 2026-02-23T13:39:49.149 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:39:49.222 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1583 8 365281 365273 1.80257 0 - 0.0299513 2026-02-23T13:39:49.268 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:39:49.373 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:39:49.371+0000 7fbf9d063640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:39:49.373 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:39:49.371+0000 7fbf9d063640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:39:49.384 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:39:49.385 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:39:49.394 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:39:49.395 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:39:49.395 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:39:49.404 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:39:49.405 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:39:49.434 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:39:49 trial068 ceph-mon[38805]: pgmap v1189: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:39:49.435 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:39:49.435 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:39:50.223 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1584 8 365281 365273 1.80143 0 - 0.0299513 2026-02-23T13:39:50.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:39:50 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1608808404' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:39:51.222 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1585 8 365281 365273 1.8003 0 - 0.0299513 2026-02-23T13:39:51.436 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:39:51.558 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:39:51.580 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:39:51 trial068 ceph-mon[38805]: pgmap v1190: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:39:51.658 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:39:51.656+0000 7fe8c00ce640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:39:51.658 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:39:51.656+0000 7fe8c00ce640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:39:51.670 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:39:51.670 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:39:51.681 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:39:51.681 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:39:51.682 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:39:51.691 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:39:51.691 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:39:51.721 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:39:51.722 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:39:52.223 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1586 8 365281 365273 1.79916 0 - 0.0299513 2026-02-23T13:39:52.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:39:52 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2145897997' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:39:53.223 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1587 8 365281 365273 1.79803 0 - 0.0299513 2026-02-23T13:39:53.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:39:53 trial068 ceph-mon[38805]: pgmap v1191: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:39:53.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:39:53 trial068 ceph-mon[38805]: 5.5 scrub starts 2026-02-23T13:39:53.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:39:53 trial068 ceph-mon[38805]: 5.5 scrub ok 2026-02-23T13:39:53.723 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:39:53.842 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:39:53.943 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:39:53.941+0000 7f97dd7a3640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:39:53.944 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:39:53.942+0000 7f97dd7a3640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:39:53.955 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:39:53.956 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:39:53.965 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:39:53.966 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:39:53.966 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:39:53.976 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:39:53.976 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:39:54.005 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:39:53 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:39:53] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:39:54.006 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:39:54.006 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:39:54.223 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1588 8 365281 365273 1.7969 0 - 0.0299513 2026-02-23T13:39:54.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:39:54 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3661303384' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:39:54.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:39:53 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:39:53.972Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:39:55.223 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1589 8 365281 365273 1.79577 0 - 0.0299513 2026-02-23T13:39:55.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:39:55 trial068 ceph-mon[38805]: pgmap v1192: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:39:56.007 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:39:56.124 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:39:56.223 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1590 8 365281 365273 1.79464 0 - 0.0299513 2026-02-23T13:39:56.225 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:39:56.223+0000 7f7118090640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:39:56.225 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:39:56.224+0000 7f7118090640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:39:56.237 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:39:56.237 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:39:56.247 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:39:56.247 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:39:56.248 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:39:56.257 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:39:56.258 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:39:56.287 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:39:56 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/4191444604' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:39:56.289 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:39:56.289 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:39:57.223 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1591 8 365281 365273 1.79351 0 - 0.0299513 2026-02-23T13:39:57.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:39:57 trial068 ceph-mon[38805]: pgmap v1193: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:39:58.223 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1592 8 365281 365273 1.79238 0 - 0.0299513 2026-02-23T13:39:58.290 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:39:58.409 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:39:58.505 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:39:58.504+0000 7fba7207e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:39:58.506 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:39:58.504+0000 7fba7207e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:39:58.518 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:39:58.518 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:39:58.529 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:39:58.529 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:39:58.529 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:39:58.539 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:39:58.540 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:39:58.569 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:39:58.570 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:39:59.223 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1593 8 365281 365273 1.79126 0 - 0.0299513 2026-02-23T13:39:59.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:39:59 trial068 ceph-mon[38805]: pgmap v1194: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:39:59.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:39:59 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2533248727' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:40:00.223 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1594 8 365281 365273 1.79013 0 - 0.0299513 2026-02-23T13:40:00.571 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:40:00.691 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:40:00.792 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:40:00.789+0000 7f1c9f212640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:40:00.792 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:40:00.790+0000 7f1c9f212640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:40:00.803 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:40:00.804 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:40:00.813 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:40:00.814 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:40:00.814 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:40:00.824 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:40:00.824 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:40:00.855 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:40:00.855 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:40:01.224 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1595 8 365281 365273 1.78901 0 - 0.0299513 2026-02-23T13:40:01.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:40:01 trial068 ceph-mon[38805]: pgmap v1195: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:40:01.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:40:01 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1119640863' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:40:02.224 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1596 8 365281 365273 1.78789 0 - 0.0299513 2026-02-23T13:40:02.856 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:40:02.976 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:40:03.077 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:40:03.075+0000 7fe4056b2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:40:03.077 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:40:03.075+0000 7fe4056b2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:40:03.089 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:40:03.089 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:40:03.099 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:40:03.099 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:40:03.100 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:40:03.110 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:40:03.110 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:40:03.140 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:40:03.141 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:40:03.224 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1597 8 365281 365273 1.78677 0 - 0.0299513 2026-02-23T13:40:03.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:40:03 trial068 ceph-mon[38805]: pgmap v1196: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:40:03.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:40:03 trial068 ceph-mon[38805]: osdmap e203: 4 total, 3 up, 4 in 2026-02-23T13:40:03.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:40:03 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1339001376' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:40:04.224 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1598 8 365281 365273 1.78565 0 - 0.0299513 2026-02-23T13:40:04.255 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:40:03 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:40:03] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:40:04.265 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:40:03 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:40:03.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:40:04.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:40:04 trial068 ceph-mon[38805]: 5.e scrub starts 2026-02-23T13:40:04.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:40:04 trial068 ceph-mon[38805]: 5.e scrub ok 2026-02-23T13:40:05.141 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:40:05.224 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1599 8 365281 365273 1.78453 0 - 0.0299513 2026-02-23T13:40:05.262 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:40:05.367 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:40:05.365+0000 7fd131722640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:40:05.367 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:40:05.365+0000 7fd131722640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:40:05.379 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:40:05.380 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:40:05.389 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:40:05.390 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:40:05.390 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:40:05.401 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:40:05.401 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:40:05.428 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:40:05 trial068 ceph-mon[38805]: pgmap v1198: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:40:05.429 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:40:05.429 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:40:06.224 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T13:40:06.223790+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T13:40:06.224 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T13:40:06.224 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1600 8 365281 365273 1.78342 0 - 0.0299513 2026-02-23T13:40:06.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:40:06 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2154012437' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:40:07.224 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1601 8 365281 365273 1.78231 0 - 0.0299513 2026-02-23T13:40:07.431 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:40:07.548 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:40:07.571 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:40:07 trial068 ceph-mon[38805]: pgmap v1199: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:40:07.648 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:40:07.646+0000 7fb725172640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:40:07.648 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:40:07.646+0000 7fb725172640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:40:07.660 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:40:07.660 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:40:07.670 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:40:07.670 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:40:07.670 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:40:07.681 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:40:07.681 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:40:07.711 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:40:07.712 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:40:08.224 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1602 8 365281 365273 1.78119 0 - 0.0299513 2026-02-23T13:40:08.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:40:08 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1607780888' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:40:09.224 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1603 8 365281 365273 1.78008 0 - 0.0299513 2026-02-23T13:40:09.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:40:09 trial068 ceph-mon[38805]: pgmap v1200: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:40:09.712 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:40:09.830 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:40:09.928 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:40:09.926+0000 7f63c990a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:40:09.928 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:40:09.927+0000 7f63c990a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:40:09.941 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:40:09.941 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:40:09.951 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:40:09.951 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:40:09.952 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:40:09.962 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:40:09.962 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:40:09.992 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:40:09.992 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:40:10.224 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1604 8 365281 365273 1.77897 0 - 0.0299513 2026-02-23T13:40:10.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:40:10 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/788015592' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:40:11.224 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1605 8 365281 365273 1.77786 0 - 0.0299513 2026-02-23T13:40:11.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:40:11 trial068 ceph-mon[38805]: pgmap v1201: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:40:11.992 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:40:12.111 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:40:12.216 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:40:12.214+0000 7f9258a5c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:40:12.217 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:40:12.215+0000 7f9258a5c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:40:12.224 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1606 8 365281 365273 1.77676 0 - 0.0299513 2026-02-23T13:40:12.228 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:40:12.228 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:40:12.238 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:40:12.239 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:40:12.239 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:40:12.249 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:40:12.249 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:40:12.282 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:40:12.283 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:40:12.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:40:12 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1154956637' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:40:13.224 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1607 8 365281 365273 1.77565 0 - 0.0299513 2026-02-23T13:40:13.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:40:13 trial068 ceph-mon[38805]: pgmap v1202: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:40:14.224 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1608 8 365281 365273 1.77455 0 - 0.0299513 2026-02-23T13:40:14.284 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:40:14.306 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:40:13 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:40:13] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:40:14.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:40:13 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:40:13.974Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:40:14.401 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:40:14.496 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:40:14.494+0000 7fd155a57640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:40:14.496 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:40:14.494+0000 7fd155a57640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:40:14.507 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:40:14.508 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:40:14.518 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:40:14.518 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:40:14.518 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:40:14.528 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:40:14.528 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:40:14.561 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:40:14.561 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:40:15.225 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1609 8 365281 365273 1.77344 0 - 0.0299513 2026-02-23T13:40:15.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:40:15 trial068 ceph-mon[38805]: pgmap v1203: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:40:15.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:40:15 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1863812755' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:40:16.225 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1610 8 365281 365273 1.77234 0 - 0.0299513 2026-02-23T13:40:16.563 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:40:16.687 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:40:16.785 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:40:16.783+0000 7fdff596e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:40:16.785 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:40:16.783+0000 7fdff596e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:40:16.797 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:40:16.797 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:40:16.807 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:40:16.808 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:40:16.808 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:40:16.818 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:40:16.818 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:40:16.849 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:40:16.850 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:40:17.225 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1611 8 365281 365273 1.77124 0 - 0.0299513 2026-02-23T13:40:17.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:40:17 trial068 ceph-mon[38805]: pgmap v1204: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:40:17.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:40:17 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1075350456' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:40:18.225 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1612 8 365281 365273 1.77014 0 - 0.0299513 2026-02-23T13:40:18.851 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:40:18.985 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:40:19.081 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:40:19.079+0000 7f532763b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:40:19.082 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:40:19.080+0000 7f532763b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:40:19.093 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:40:19.093 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:40:19.103 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:40:19.103 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:40:19.103 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:40:19.113 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:40:19.113 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:40:19.145 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:40:19.146 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:40:19.225 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1613 8 365281 365273 1.76905 0 - 0.0299513 2026-02-23T13:40:19.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:40:19 trial068 ceph-mon[38805]: pgmap v1205: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:40:19.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:40:19 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2115550158' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:40:20.225 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1614 8 365281 365273 1.76795 0 - 0.0299513 2026-02-23T13:40:21.146 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:40:21.225 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1615 8 365281 365273 1.76685 0 - 0.0299513 2026-02-23T13:40:21.263 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:40:21.358 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:40:21.356+0000 7f6efb8ec640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:40:21.358 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:40:21.356+0000 7f6efb8ec640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:40:21.371 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:40:21.371 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:40:21.381 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:40:21.382 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:40:21.382 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:40:21.391 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:40:21.392 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:40:21.421 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:40:21.421 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:40:21.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:40:21 trial068 ceph-mon[38805]: pgmap v1206: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:40:22.225 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1616 8 365281 365273 1.76576 0 - 0.0299513 2026-02-23T13:40:22.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:40:22 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2824113969' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:40:23.225 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1617 8 365281 365273 1.76467 0 - 0.0299513 2026-02-23T13:40:23.423 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:40:23.550 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:40:23.572 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:40:23 trial068 ceph-mon[38805]: pgmap v1207: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:40:23.647 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:40:23.645+0000 7fe2843d5640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:40:23.647 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:40:23.645+0000 7fe2843d5640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:40:23.659 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:40:23.659 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:40:23.669 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:40:23.669 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:40:23.669 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:40:23.680 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:40:23.680 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:40:23.710 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:40:23.711 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:40:24.225 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1618 8 365281 365273 1.76358 0 - 0.0299513 2026-02-23T13:40:24.279 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:40:23 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:40:23] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:40:24.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:40:23 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:40:23.974Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:40:24.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:40:24 trial068 ceph-mon[38805]: 5.7 scrub starts 2026-02-23T13:40:24.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:40:24 trial068 ceph-mon[38805]: 5.7 scrub ok 2026-02-23T13:40:24.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:40:24 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/999020320' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:40:25.225 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1619 8 365281 365273 1.76249 0 - 0.0299513 2026-02-23T13:40:25.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:40:25 trial068 ceph-mon[38805]: pgmap v1208: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:40:25.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:40:25 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T13:40:25.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:40:25 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T13:40:25.712 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:40:25.828 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:40:25.926 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:40:25.923+0000 7fd61eeb0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:40:25.926 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:40:25.924+0000 7fd61eeb0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:40:25.937 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:40:25.937 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:40:25.947 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:40:25.948 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:40:25.948 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:40:25.957 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:40:25.958 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:40:25.986 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:40:25.986 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:40:26.225 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T13:40:26.225343+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T13:40:26.225 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T13:40:26.225 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1620 8 365281 365273 1.7614 0 - 0.0299513 2026-02-23T13:40:26.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:40:26 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1183940641' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:40:27.226 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1621 8 365281 365273 1.76032 0 - 0.0299513 2026-02-23T13:40:27.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:40:27 trial068 ceph-mon[38805]: pgmap v1209: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:40:27.988 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:40:28.105 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:40:28.203 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:40:28.201+0000 7f85dd603640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:40:28.204 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:40:28.201+0000 7f85dd603640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:40:28.215 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:40:28.215 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:40:28.225 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:40:28.225 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:40:28.226 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:40:28.226 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1622 8 365281 365273 1.75923 0 - 0.0299513 2026-02-23T13:40:28.235 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:40:28.236 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:40:28.265 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:40:28.266 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:40:28.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:40:28 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1779580817' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:40:29.226 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1623 8 365281 365273 1.75815 0 - 0.0299513 2026-02-23T13:40:29.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:40:29 trial068 ceph-mon[38805]: pgmap v1210: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:40:30.226 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1624 8 365281 365273 1.75706 0 - 0.0299513 2026-02-23T13:40:30.267 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:40:30.385 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:40:30.482 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:40:30.480+0000 7f313df16640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:40:30.482 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:40:30.480+0000 7f313df16640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:40:30.493 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:40:30.494 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:40:30.503 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:40:30.504 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:40:30.504 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:40:30.514 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:40:30.514 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:40:30.543 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:40:30.544 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:40:31.226 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1625 8 365281 365273 1.75598 0 - 0.0299513 2026-02-23T13:40:31.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:40:31 trial068 ceph-mon[38805]: pgmap v1211: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:40:31.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:40:31 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3332813381' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:40:32.226 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1626 8 365281 365273 1.7549 0 - 0.0299513 2026-02-23T13:40:32.545 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:40:32.569 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:40:32 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T13:40:32.570 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:40:32 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T13:40:32.664 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:40:32.767 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:40:32.765+0000 7f7731c00640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:40:32.767 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:40:32.765+0000 7f7731c00640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:40:32.779 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:40:32.779 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:40:32.789 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:40:32.790 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:40:32.790 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:40:32.800 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:40:32.800 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:40:32.830 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:40:32.830 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:40:33.226 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1627 8 365281 365273 1.75382 0 - 0.0299513 2026-02-23T13:40:33.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:40:33 trial068 ceph-mon[38805]: pgmap v1212: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:40:33.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:40:33 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/4260040877' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:40:34.226 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1628 8 365281 365273 1.75275 0 - 0.0299513 2026-02-23T13:40:34.264 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:40:33 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:40:33.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:40:34.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:40:33 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:40:33] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:40:34.831 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:40:34.950 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:40:35.049 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:40:35.048+0000 7ff9febf5640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:40:35.050 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:40:35.048+0000 7ff9febf5640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:40:35.062 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:40:35.062 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:40:35.072 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:40:35.073 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:40:35.073 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:40:35.083 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:40:35.083 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:40:35.115 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:40:35.115 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:40:35.226 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1629 8 365281 365273 1.75167 0 - 0.0299513 2026-02-23T13:40:35.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:40:35 trial068 ceph-mon[38805]: pgmap v1213: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:40:35.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:40:35 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1723627702' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:40:36.226 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1630 8 365281 365273 1.7506 0 - 0.0299513 2026-02-23T13:40:37.116 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:40:37.226 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1631 8 365281 365273 1.74952 0 - 0.0299513 2026-02-23T13:40:37.233 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:40:37.335 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:40:37.333+0000 7f51392b9640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:40:37.335 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:40:37.334+0000 7f51392b9640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:40:37.347 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:40:37.347 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:40:37.356 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:40:37.357 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:40:37.357 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:40:37.368 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:40:37.368 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:40:37.396 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:40:37 trial068 ceph-mon[38805]: pgmap v1214: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:40:37.397 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:40:37.397 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:40:38.226 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1632 8 365281 365273 1.74845 0 - 0.0299513 2026-02-23T13:40:38.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:40:38 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/564626694' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:40:39.226 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1633 8 365281 365273 1.74738 0 - 0.0299513 2026-02-23T13:40:39.398 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:40:39.515 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:40:39.555 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:40:39 trial068 ceph-mon[38805]: pgmap v1215: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:40:39.612 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:40:39.610+0000 7f8b3b486640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:40:39.612 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:40:39.610+0000 7f8b3b486640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:40:39.624 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:40:39.624 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:40:39.634 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:40:39.634 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:40:39.634 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:40:39.643 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:40:39.657 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:40:39.672 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:40:39.673 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:40:40.227 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1634 8 365281 365273 1.74631 0 - 0.0299513 2026-02-23T13:40:40.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:40:40 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3036251555' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:40:41.227 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1635 8 365281 365273 1.74524 0 - 0.0299513 2026-02-23T13:40:41.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:40:41 trial068 ceph-mon[38805]: pgmap v1216: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:40:41.674 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:40:41.792 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:40:41.895 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:40:41.893+0000 7f8d5b098640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:40:41.895 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:40:41.893+0000 7f8d5b098640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:40:41.907 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:40:41.908 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:40:41.917 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:40:41.918 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:40:41.918 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:40:41.928 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:40:41.929 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:40:41.960 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:40:41.961 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:40:42.227 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1636 8 365281 365273 1.74418 0 - 0.0299513 2026-02-23T13:40:42.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:40:42 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3403640571' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:40:42.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:40:42 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2026-02-23T13:40:42.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:40:42 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-23T13:40:42.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:40:42 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2026-02-23T13:40:42.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:40:42 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T13:40:42.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:40:42 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2026-02-23T13:40:43.227 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1637 8 365281 365273 1.74311 0 - 0.0299513 2026-02-23T13:40:43.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:40:43 trial068 ceph-mon[38805]: pgmap v1217: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:40:43.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:40:43 trial068 ceph-mon[38805]: pgmap v1218: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:40:43.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:40:43 trial068 ceph-mon[38805]: pgmap v1219: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:40:43.963 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:40:44.083 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:40:44.185 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:40:44.183+0000 7fd26d0e3640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:40:44.185 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:40:44.183+0000 7fd26d0e3640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:40:44.197 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:40:44.197 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:40:44.207 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:40:44.207 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:40:44.208 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:40:44.219 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:40:44.219 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:40:44.226 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:40:43 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:40:43] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:40:44.227 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1638 8 365281 365273 1.74205 0 - 0.0299513 2026-02-23T13:40:44.247 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:40:44.247 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:40:44.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:40:43 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:40:43.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:40:44.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:40:44 trial068 ceph-mon[38805]: 5.8 scrub starts 2026-02-23T13:40:44.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:40:44 trial068 ceph-mon[38805]: 5.8 scrub ok 2026-02-23T13:40:44.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:40:44 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/506027154' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:40:45.233 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1639 8 365281 365273 1.74098 0 - 0.0299513 2026-02-23T13:40:45.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:40:45 trial068 ceph-mon[38805]: pgmap v1220: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:40:46.227 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T13:40:46.226811+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T13:40:46.227 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T13:40:46.227 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1640 8 365281 365273 1.73992 0 - 0.0299513 2026-02-23T13:40:46.248 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:40:46.366 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:40:46.470 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:40:46.468+0000 7f27d6f52640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:40:46.470 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:40:46.468+0000 7f27d6f52640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:40:46.482 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:40:46.482 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:40:46.491 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:40:46.492 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:40:46.492 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:40:46.502 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:40:46.502 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:40:46.531 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:40:46.531 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:40:47.227 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1641 8 365281 365273 1.73886 0 - 0.0299513 2026-02-23T13:40:47.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:40:47 trial068 ceph-mon[38805]: pgmap v1221: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:40:47.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:40:47 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/773452395' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:40:48.227 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1642 8 365281 365273 1.7378 0 - 0.0299513 2026-02-23T13:40:48.532 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:40:48.557 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:40:48 trial068 ceph-mon[38805]: 5.a scrub starts 2026-02-23T13:40:48.557 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:40:48 trial068 ceph-mon[38805]: 5.a scrub ok 2026-02-23T13:40:48.650 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:40:48.745 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:40:48.743+0000 7f9baf667640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:40:48.745 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:40:48.743+0000 7f9baf667640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:40:48.757 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:40:48.757 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:40:48.767 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:40:48.767 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:40:48.767 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:40:48.777 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:40:48.777 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:40:48.807 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:40:48.808 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:40:49.227 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1643 8 365281 365273 1.73674 0 - 0.0299513 2026-02-23T13:40:49.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:40:49 trial068 ceph-mon[38805]: pgmap v1222: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:40:49.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:40:49 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1219375712' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:40:50.227 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1644 8 365281 365273 1.73569 0 - 0.0299513 2026-02-23T13:40:50.809 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:40:50.926 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:40:51.028 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:40:51.027+0000 7f2642fba640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:40:51.029 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:40:51.027+0000 7f2642fba640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:40:51.040 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:40:51.041 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:40:51.051 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:40:51.051 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:40:51.051 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:40:51.061 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:40:51.061 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:40:51.092 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:40:51.093 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:40:51.227 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1645 8 365281 365273 1.73463 0 - 0.0299513 2026-02-23T13:40:51.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:40:51 trial068 ceph-mon[38805]: pgmap v1223: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:40:51.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:40:51 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/838984436' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:40:52.228 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1646 8 365281 365273 1.73358 0 - 0.0299513 2026-02-23T13:40:53.094 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:40:53.211 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:40:53.228 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1647 8 365281 365273 1.73253 0 - 0.0299513 2026-02-23T13:40:53.309 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:40:53.307+0000 7f8f83177640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:40:53.309 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:40:53.308+0000 7f8f83177640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:40:53.322 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:40:53.323 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:40:53.333 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:40:53.333 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:40:53.333 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:40:53.343 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:40:53.343 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:40:53.372 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:40:53 trial068 ceph-mon[38805]: pgmap v1224: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:40:53.374 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:40:53.374 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:40:54.228 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1648 8 365281 365273 1.73147 0 - 0.0299513 2026-02-23T13:40:54.310 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:40:53 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:40:53] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:40:54.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:40:53 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:40:53.974Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:40:54.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:40:54 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1428766085' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:40:55.228 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1649 8 365281 365273 1.73042 0 - 0.0299513 2026-02-23T13:40:55.375 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:40:55.490 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:40:55.587 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:40:55.586+0000 7fb733ed3640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:40:55.587 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:40:55 trial068 ceph-mon[38805]: pgmap v1225: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:40:55.587 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:40:55.586+0000 7fb733ed3640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:40:55.599 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:40:55.599 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:40:55.610 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:40:55.610 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:40:55.610 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:40:55.622 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:40:55.622 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:40:55.651 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:40:55.652 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:40:56.228 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1650 8 365281 365273 1.72938 0 - 0.0299513 2026-02-23T13:40:56.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:40:56 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3743925996' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:40:57.228 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1651 8 365281 365273 1.72833 0 - 0.0299513 2026-02-23T13:40:57.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:40:57 trial068 ceph-mon[38805]: pgmap v1226: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:40:57.653 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:40:57.772 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:40:57.866 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:40:57.864+0000 7fd73137a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:40:57.867 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:40:57.865+0000 7fd73137a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:40:57.878 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:40:57.879 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:40:57.889 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:40:57.890 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:40:57.890 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:40:57.899 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:40:57.899 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:40:57.930 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:40:57.931 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:40:58.228 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1652 8 365281 365273 1.72728 0 - 0.0299513 2026-02-23T13:40:58.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:40:58 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/4139294119' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:40:59.228 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1653 8 365281 365273 1.72624 0 - 0.0299513 2026-02-23T13:40:59.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:40:59 trial068 ceph-mon[38805]: pgmap v1227: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:40:59.933 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:41:00.051 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:41:00.144 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:41:00.142+0000 7fb86875e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:41:00.145 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:41:00.143+0000 7fb86875e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:41:00.156 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:41:00.156 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:41:00.166 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:41:00.166 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:41:00.166 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:41:00.176 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:41:00.177 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:41:00.206 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:41:00.207 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:41:00.228 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1654 8 365281 365273 1.72519 0 - 0.0299513 2026-02-23T13:41:00.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:41:00 trial068 ceph-mon[38805]: 5.5 scrub starts 2026-02-23T13:41:00.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:41:00 trial068 ceph-mon[38805]: 5.5 scrub ok 2026-02-23T13:41:00.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:41:00 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3043621140' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:41:01.228 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1655 8 365281 365273 1.72415 0 - 0.0299513 2026-02-23T13:41:01.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:41:01 trial068 ceph-mon[38805]: pgmap v1228: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:41:02.208 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:41:02.229 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1656 8 365281 365273 1.72311 0 - 0.0299513 2026-02-23T13:41:02.328 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:41:02.432 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:41:02.431+0000 7f7d1c2e8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:41:02.433 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:41:02.431+0000 7f7d1c2e8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:41:02.444 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:41:02.444 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:41:02.454 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:41:02.454 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:41:02.455 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:41:02.464 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:41:02.465 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:41:02.492 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:41:02.493 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:41:03.228 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1657 8 365281 365273 1.72207 0 - 0.0299513 2026-02-23T13:41:03.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:41:03 trial068 ceph-mon[38805]: pgmap v1229: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:41:03.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:41:03 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3910496801' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:41:04.229 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1658 8 365281 365273 1.72103 0 - 0.0299513 2026-02-23T13:41:04.265 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:41:03 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:41:03.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:41:04.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:41:03 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:41:03] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:41:04.494 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:41:04.614 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:41:04.715 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:41:04.713+0000 7fe54c33e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:41:04.715 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:41:04.713+0000 7fe54c33e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:41:04.726 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:41:04.727 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:41:04.736 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:41:04.737 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:41:04.737 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:41:04.746 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:41:04.747 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:41:04.777 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:41:04.777 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:41:05.229 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1659 8 365281 365273 1.71999 0 - 0.0299513 2026-02-23T13:41:05.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:41:05 trial068 ceph-mon[38805]: pgmap v1230: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:41:05.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:41:05 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3910676625' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:41:06.229 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T13:41:06.228657+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T13:41:06.229 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T13:41:06.229 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1660 8 365281 365273 1.71896 0 - 0.0299513 2026-02-23T13:41:06.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:41:06 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T13:41:06.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:41:06 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T13:41:06.778 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:41:06.904 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:41:07.001 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:41:07.000+0000 7f8a2a483640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:41:07.001 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:41:07.000+0000 7f8a2a483640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:41:07.013 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:41:07.013 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:41:07.023 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:41:07.023 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:41:07.023 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:41:07.033 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:41:07.034 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:41:07.064 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:41:07.064 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:41:07.229 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1661 8 365281 365273 1.71792 0 - 0.0299513 2026-02-23T13:41:07.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:41:07 trial068 ceph-mon[38805]: pgmap v1231: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:41:07.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:41:07 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3791357719' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:41:08.229 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1662 8 365281 365273 1.71689 0 - 0.0299513 2026-02-23T13:41:09.066 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:41:09.184 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:41:09.229 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1663 8 365281 365273 1.71586 0 - 0.0299513 2026-02-23T13:41:09.281 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:41:09.279+0000 7f392935f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:41:09.281 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:41:09.279+0000 7f392935f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:41:09.292 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:41:09.293 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:41:09.302 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:41:09.303 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:41:09.303 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:41:09.313 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:41:09.313 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:41:09.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:41:09 trial068 ceph-mon[38805]: pgmap v1232: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:41:09.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:41:09 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/4247196338' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:41:09.344 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:41:09.345 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:41:10.229 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1664 8 365281 365273 1.71483 0 - 0.0299513 2026-02-23T13:41:11.229 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1665 8 365281 365273 1.7138 0 - 0.0299513 2026-02-23T13:41:11.346 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:41:11.465 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:41:11.562 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:41:11.560+0000 7fbd2a6b1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:41:11.562 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:41:11.560+0000 7fbd2a6b1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:41:11.574 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:41:11.574 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:41:11.584 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:41:11.584 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:41:11.585 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:41:11 trial068 ceph-mon[38805]: pgmap v1233: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:41:11.585 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:41:11 trial068 ceph-mon[38805]: osdmap e204: 4 total, 3 up, 4 in 2026-02-23T13:41:11.585 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:41:11.594 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:41:11.595 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:41:11.625 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:41:11.625 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:41:12.230 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1666 8 365281 365273 1.71277 0 - 0.0299513 2026-02-23T13:41:12.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:41:12 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/160035903' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:41:13.229 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1667 8 365281 365273 1.71174 0 - 0.0299513 2026-02-23T13:41:13.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:41:13 trial068 ceph-mon[38805]: pgmap v1235: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:41:13.626 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:41:13.744 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:41:13.843 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:41:13.840+0000 7fa42598b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:41:13.843 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:41:13.841+0000 7fa42598b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:41:13.855 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:41:13.855 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:41:13.865 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:41:13.865 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:41:13.866 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:41:13.876 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:41:13.876 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:41:13.908 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:41:13.909 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:41:14.229 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1668 8 365281 365273 1.71071 0 - 0.0299513 2026-02-23T13:41:14.337 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:41:13 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:41:13] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:41:14.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:41:13 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:41:13.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:41:14.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:41:14 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/879137739' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:41:15.229 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1669 8 365281 365273 1.70969 0 - 0.0299513 2026-02-23T13:41:15.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:41:15 trial068 ceph-mon[38805]: pgmap v1236: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:41:15.909 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:41:16.026 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:41:16.122 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:41:16.120+0000 7f84bfcae640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:41:16.122 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:41:16.120+0000 7f84bfcae640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:41:16.134 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:41:16.134 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:41:16.144 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:41:16.145 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:41:16.145 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:41:16.155 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:41:16.156 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:41:16.184 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:41:16.184 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:41:16.229 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1670 8 365281 365273 1.70867 0 - 0.0299513 2026-02-23T13:41:16.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:41:16 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/4091940015' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:41:17.230 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1671 8 365281 365273 1.70764 0 - 0.0299513 2026-02-23T13:41:17.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:41:17 trial068 ceph-mon[38805]: pgmap v1237: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:41:17.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:41:17 trial068 ceph-mon[38805]: 5.e scrub starts 2026-02-23T13:41:17.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:41:17 trial068 ceph-mon[38805]: 5.e scrub ok 2026-02-23T13:41:18.185 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:41:18.230 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1672 8 365281 365273 1.70662 0 - 0.0299513 2026-02-23T13:41:18.301 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:41:18.402 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:41:18.400+0000 7fbf16640640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:41:18.402 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:41:18.401+0000 7fbf16640640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:41:18.414 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:41:18.414 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:41:18.424 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:41:18.425 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:41:18.425 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:41:18.436 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:41:18.436 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:41:18.466 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:41:18.466 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:41:19.230 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1673 8 365281 365273 1.7056 0 - 0.0299513 2026-02-23T13:41:19.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:41:19 trial068 ceph-mon[38805]: pgmap v1238: 17 pgs: 1 active+clean+scrubbing, 1 active+recovery_unfound+undersized+degraded, 4 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:41:19.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:41:19 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/4023288486' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:41:20.230 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1674 8 365281 365273 1.70458 0 - 0.0299513 2026-02-23T13:41:20.467 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:41:20.586 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:41:20.687 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:41:20.686+0000 7f819dd0f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:41:20.688 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:41:20.686+0000 7f819dd0f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:41:20.699 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:41:20.699 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:41:20.709 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:41:20.710 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:41:20.710 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:41:20.720 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:41:20.720 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:41:20.751 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:41:20.751 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:41:21.230 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1675 8 365281 365273 1.70356 0 - 0.0299513 2026-02-23T13:41:21.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:41:21 trial068 ceph-mon[38805]: pgmap v1239: 17 pgs: 1 active+clean+scrubbing, 1 active+recovery_unfound+undersized+degraded, 4 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:41:21.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:41:21 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1126230431' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:41:22.230 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1676 8 365281 365273 1.70255 0 - 0.0299513 2026-02-23T13:41:22.752 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:41:22.871 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:41:22.974 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:41:22.972+0000 7f025e9ab640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:41:22.975 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:41:22.973+0000 7f025e9ab640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:41:22.986 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:41:22.986 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:41:22.997 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:41:22.997 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:41:22.998 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:41:23.009 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:41:23.009 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:41:23.039 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:41:23.040 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:41:23.230 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1677 8 365281 365273 1.70153 0 - 0.0299513 2026-02-23T13:41:23.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:41:23 trial068 ceph-mon[38805]: pgmap v1240: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:41:23.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:41:23 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/6460836' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:41:24.230 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1678 8 365281 365273 1.70052 0 - 0.0299513 2026-02-23T13:41:24.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:41:23 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:41:23] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:41:24.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:41:23 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:41:23.974Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:41:25.041 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:41:25.160 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:41:25.230 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1679 8 365281 365273 1.69951 0 - 0.0299513 2026-02-23T13:41:25.256 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:41:25.254+0000 7f89ea019640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:41:25.256 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:41:25.255+0000 7f89ea019640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:41:25.268 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:41:25.268 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:41:25.278 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:41:25.278 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:41:25.278 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:41:25.289 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:41:25.289 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:41:25.320 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:41:25.320 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:41:25.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:41:25 trial068 ceph-mon[38805]: pgmap v1241: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:41:25.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:41:25 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1352281874' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:41:26.230 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T13:41:26.230067+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T13:41:26.231 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T13:41:26.231 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1680 8 365281 365273 1.69849 0 - 0.0299513 2026-02-23T13:41:27.230 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1681 8 365281 365273 1.69748 0 - 0.0299513 2026-02-23T13:41:27.321 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:41:27.438 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:41:27.544 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:41:27.543+0000 7fdc413f2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:41:27.544 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:41:27.543+0000 7fdc413f2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:41:27.557 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:41:27.557 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:41:27.568 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:41:27.568 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:41:27.568 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:41:27.578 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:41:27.578 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:41:27.607 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:41:27 trial068 ceph-mon[38805]: pgmap v1242: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:41:27.608 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:41:27.608 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:41:28.230 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1682 8 365281 365273 1.69647 0 - 0.0299513 2026-02-23T13:41:28.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:41:28 trial068 ceph-mon[38805]: 5.7 scrub starts 2026-02-23T13:41:28.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:41:28 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2943937125' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:41:28.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:41:28 trial068 ceph-mon[38805]: 5.7 scrub ok 2026-02-23T13:41:29.230 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1683 8 365281 365273 1.69547 0 - 0.0299513 2026-02-23T13:41:29.610 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:41:29.634 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:41:29 trial068 ceph-mon[38805]: pgmap v1243: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:41:29.727 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:41:29.829 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:41:29.827+0000 7f7cf777a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:41:29.829 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:41:29.827+0000 7f7cf777a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:41:29.842 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:41:29.842 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:41:29.852 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:41:29.853 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:41:29.853 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:41:29.862 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:41:29.863 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:41:29.896 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:41:29.897 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:41:30.230 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1684 8 365281 365273 1.69446 0 - 0.0299513 2026-02-23T13:41:30.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:41:30 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1042128795' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:41:31.231 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1685 8 365281 365273 1.69345 0 - 0.0299513 2026-02-23T13:41:31.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:41:31 trial068 ceph-mon[38805]: pgmap v1244: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:41:31.898 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:41:32.014 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:41:32.108 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:41:32.106+0000 7f1494c85640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:41:32.108 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:41:32.106+0000 7f1494c85640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:41:32.120 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:41:32.120 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:41:32.130 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:41:32.130 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:41:32.130 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:41:32.140 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:41:32.141 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:41:32.171 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:41:32.171 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:41:32.230 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1686 8 365281 365273 1.69245 0 - 0.0299513 2026-02-23T13:41:32.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:41:32 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2086941839' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:41:33.231 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1687 8 365281 365273 1.69145 0 - 0.0299513 2026-02-23T13:41:33.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:41:33 trial068 ceph-mon[38805]: pgmap v1245: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:41:34.172 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:41:34.231 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1688 8 365281 365273 1.69044 0 - 0.0299513 2026-02-23T13:41:34.265 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:41:33 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:41:33.974Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:41:34.291 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:41:34.314 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:41:33 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:41:33] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:41:34.391 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:41:34.389+0000 7f01db69a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:41:34.391 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:41:34.389+0000 7f01db69a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:41:34.402 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:41:34.403 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:41:34.413 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:41:34.413 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:41:34.413 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:41:34.423 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:41:34.423 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:41:34.455 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:41:34.455 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:41:35.231 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1689 8 365281 365273 1.68944 0 - 0.0299513 2026-02-23T13:41:35.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:41:35 trial068 ceph-mon[38805]: pgmap v1246: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:41:35.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:41:35 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/74942336' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:41:36.231 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1690 8 365281 365273 1.68844 0 - 0.0299513 2026-02-23T13:41:36.456 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:41:36.574 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:41:36.680 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:41:36.678+0000 7f47c0f52640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:41:36.680 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:41:36.678+0000 7f47c0f52640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:41:36.692 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:41:36.692 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:41:36.702 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:41:36.702 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:41:36.703 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:41:36.712 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:41:36.713 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:41:36.745 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:41:36.746 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:41:37.231 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1691 8 365281 365273 1.68745 0 - 0.0299513 2026-02-23T13:41:37.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:41:37 trial068 ceph-mon[38805]: pgmap v1247: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:41:37.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:41:37 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3208205945' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:41:38.231 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1692 8 365281 365273 1.68645 0 - 0.0299513 2026-02-23T13:41:38.747 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:41:38.864 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:41:38.962 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:41:38.959+0000 7f2ba7969640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:41:38.962 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:41:38.960+0000 7f2ba7969640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:41:38.973 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:41:38.974 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:41:38.984 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:41:38.984 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:41:38.984 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:41:38.994 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:41:38.994 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:41:39.026 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:41:39.026 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:41:39.231 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1693 8 365281 365273 1.68545 0 - 0.0299513 2026-02-23T13:41:39.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:41:39 trial068 ceph-mon[38805]: pgmap v1248: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:41:39.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:41:39 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/281653558' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:41:40.231 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1694 8 365281 365273 1.68446 0 - 0.0299513 2026-02-23T13:41:41.028 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:41:41.144 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:41:41.232 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1695 8 365281 365273 1.68346 0 - 0.0299513 2026-02-23T13:41:41.245 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:41:41.243+0000 7f5e49f69640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:41:41.245 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:41:41.243+0000 7f5e49f69640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:41:41.257 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:41:41.257 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:41:41.267 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:41:41.268 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:41:41.268 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:41:41.277 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:41:41.277 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:41:41.308 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:41:41.309 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:41:41.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:41:41 trial068 ceph-mon[38805]: pgmap v1249: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:41:41.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:41:41 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3809998673' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:41:42.231 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1696 8 365281 365273 1.68247 0 - 0.0299513 2026-02-23T13:41:42.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:41:42 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2026-02-23T13:41:43.231 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1697 8 365281 365273 1.68148 0 - 0.0299513 2026-02-23T13:41:43.310 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:41:43.429 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:41:43.524 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:41:43.522+0000 7f0f6cb43640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:41:43.524 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:41:43.522+0000 7f0f6cb43640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:41:43.536 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:41:43.536 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:41:43.546 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:41:43.546 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:41:43.546 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:41:43.558 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:41:43.558 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:41:43.586 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:41:43 trial068 ceph-mon[38805]: pgmap v1250: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:41:43.586 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:41:43 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-23T13:41:43.586 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:41:43 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2026-02-23T13:41:43.586 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:41:43 trial068 ceph-mon[38805]: pgmap v1251: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:41:43.586 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:41:43 trial068 ceph-mon[38805]: pgmap v1252: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:41:43.586 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:41:43 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T13:41:43.586 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:41:43 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2026-02-23T13:41:43.587 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:41:43.587 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:41:44.232 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1698 8 365281 365273 1.68049 0 - 0.0299513 2026-02-23T13:41:44.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:41:43 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:41:43] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:41:44.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:41:43 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:41:43.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:41:44.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:41:44 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3595899162' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:41:45.232 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1699 8 365281 365273 1.6795 0 - 0.0299513 2026-02-23T13:41:45.588 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:41:45.622 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:41:45 trial068 ceph-mon[38805]: pgmap v1253: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:41:45.715 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:41:45.813 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:41:45.811+0000 7f6107978640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:41:45.813 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:41:45.811+0000 7f6107978640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:41:45.825 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:41:45.825 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:41:45.836 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:41:45.836 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:41:45.836 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:41:45.847 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:41:45.847 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:41:45.899 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:41:45.899 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:41:46.232 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T13:41:46.231475+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T13:41:46.233 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T13:41:46.233 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1700 8 365281 365273 1.67851 0 - 0.0299513 2026-02-23T13:41:46.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:41:46 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2865909807' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:41:47.232 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1701 8 365281 365273 1.67753 0 - 0.0299513 2026-02-23T13:41:47.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:41:47 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T13:41:47.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:41:47 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T13:41:47.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:41:47 trial068 ceph-mon[38805]: pgmap v1254: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:41:47.900 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:41:48.016 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:41:48.120 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:41:48.118+0000 7f4f30d83640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:41:48.120 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:41:48.118+0000 7f4f30d83640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:41:48.132 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:41:48.132 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:41:48.142 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:41:48.142 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:41:48.143 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:41:48.153 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:41:48.153 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:41:48.189 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:41:48.190 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:41:48.232 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1702 8 365281 365273 1.67654 0 - 0.0299513 2026-02-23T13:41:48.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:41:48 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1374864639' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:41:49.232 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1703 8 365281 365273 1.67556 0 - 0.0299513 2026-02-23T13:41:49.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:41:49 trial068 ceph-mon[38805]: 5.8 scrub starts 2026-02-23T13:41:49.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:41:49 trial068 ceph-mon[38805]: 5.8 scrub ok 2026-02-23T13:41:49.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:41:49 trial068 ceph-mon[38805]: pgmap v1255: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:41:50.190 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:41:50.232 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1704 8 365281 365273 1.67457 0 - 0.0299513 2026-02-23T13:41:50.307 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:41:50.409 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:41:50.407+0000 7fcdf264d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:41:50.410 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:41:50.408+0000 7fcdf264d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:41:50.421 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:41:50.421 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:41:50.432 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:41:50.432 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:41:50.432 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:41:50.442 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:41:50.443 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:41:50.471 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:41:50 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/923931951' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:41:50.472 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:41:50.473 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:41:51.232 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1705 8 365281 365273 1.67359 0 - 0.0299513 2026-02-23T13:41:51.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:41:51 trial068 ceph-mon[38805]: pgmap v1256: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:41:52.232 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1706 8 365281 365273 1.67261 0 - 0.0299513 2026-02-23T13:41:52.474 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:41:52.593 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:41:52.689 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:41:52.688+0000 7f2b98a0c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:41:52.689 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:41:52.688+0000 7f2b98a0c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:41:52.702 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:41:52.702 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:41:52.712 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:41:52.712 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:41:52.712 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:41:52.723 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:41:52.723 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:41:52.755 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:41:52.755 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:41:53.232 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1707 8 365281 365273 1.67163 0 - 0.0299513 2026-02-23T13:41:53.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:41:53 trial068 ceph-mon[38805]: pgmap v1257: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:41:53.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:41:53 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2422036153' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:41:54.232 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1708 8 365281 365273 1.67065 0 - 0.0299513 2026-02-23T13:41:54.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:41:53 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:41:53] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:41:54.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:41:53 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:41:53.974Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:41:54.757 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:41:54.782 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:41:54 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T13:41:54.782 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:41:54 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T13:41:54.876 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:41:54.977 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:41:54.976+0000 7f30b5d2f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:41:54.978 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:41:54.976+0000 7f30b5d2f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:41:54.989 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:41:54.989 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:41:55.000 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:41:55.000 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:41:55.000 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:41:55.010 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:41:55.010 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:41:55.041 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:41:55.041 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:41:55.233 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1709 8 365281 365273 1.66967 0 - 0.0299513 2026-02-23T13:41:55.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:41:55 trial068 ceph-mon[38805]: pgmap v1258: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:41:55.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:41:55 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1517422312' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:41:56.232 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1710 8 365281 365273 1.6687 0 - 0.0299513 2026-02-23T13:41:57.042 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:41:57.160 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:41:57.233 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1711 8 365281 365273 1.66772 0 - 0.0299513 2026-02-23T13:41:57.255 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:41:57.254+0000 7f9827259640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:41:57.256 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:41:57.254+0000 7f9827259640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:41:57.267 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:41:57.268 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:41:57.277 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:41:57.278 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:41:57.278 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:41:57.287 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:41:57.287 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:41:57.318 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:41:57.318 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:41:57.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:41:57 trial068 ceph-mon[38805]: pgmap v1259: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:41:57.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:41:57 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2160845150' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:41:58.233 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1712 8 365281 365273 1.66675 0 - 0.0299513 2026-02-23T13:41:59.233 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1713 8 365281 365273 1.66577 0 - 0.0299513 2026-02-23T13:41:59.319 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:41:59.437 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:41:59.534 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:41:59.532+0000 7ff9bb7b4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:41:59.535 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:41:59.533+0000 7ff9bb7b4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:41:59.547 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:41:59.547 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:41:59.558 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:41:59.558 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:41:59.558 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:41:59.568 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:41:59.568 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:41:59.597 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:41:59 trial068 ceph-mon[38805]: pgmap v1260: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:41:59.598 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:41:59.599 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:42:00.233 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1714 8 365281 365273 1.6648 0 - 0.0299513 2026-02-23T13:42:00.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:42:00 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2872200428' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:42:01.233 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1715 8 365281 365273 1.66383 0 - 0.0299513 2026-02-23T13:42:01.600 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:42:01.716 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:42:01.739 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:42:01 trial068 ceph-mon[38805]: pgmap v1261: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:42:01.815 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:42:01.813+0000 7f2c44293640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:42:01.815 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:42:01.813+0000 7f2c44293640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:42:01.827 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:42:01.827 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:42:01.838 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:42:01.838 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:42:01.838 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:42:01.848 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:42:01.848 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:42:01.876 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:42:01.877 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:42:02.233 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1716 8 365281 365273 1.66286 0 - 0.0299513 2026-02-23T13:42:02.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:42:02 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2845123657' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:42:03.233 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1717 8 365281 365273 1.66189 0 - 0.0299513 2026-02-23T13:42:03.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:42:03 trial068 ceph-mon[38805]: pgmap v1262: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:42:03.878 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:42:03.994 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:42:04.095 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:42:04.093+0000 7f2084dc9640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:42:04.095 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:42:04.093+0000 7f2084dc9640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:42:04.108 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:42:04.108 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:42:04.118 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:42:04.118 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:42:04.118 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:42:04.128 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:42:04.129 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:42:04.158 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:42:03 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:42:03] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:42:04.159 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:42:04.159 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:42:04.233 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1718 8 365281 365273 1.66093 0 - 0.0299513 2026-02-23T13:42:04.265 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:42:03 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:42:03.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:42:04.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:42:04 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1017170102' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:42:05.234 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1719 8 365281 365273 1.65996 0 - 0.0299513 2026-02-23T13:42:05.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:42:05 trial068 ceph-mon[38805]: pgmap v1263: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:42:06.160 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:42:06.233 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T13:42:06.233491+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T13:42:06.234 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T13:42:06.234 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1720 8 365281 365273 1.65899 0 - 0.0299513 2026-02-23T13:42:06.278 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:42:06.381 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:42:06.380+0000 7f371d6b1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:42:06.382 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:42:06.380+0000 7f371d6b1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:42:06.393 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:42:06.393 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:42:06.403 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:42:06.404 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:42:06.404 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:42:06.414 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:42:06.414 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:42:06.442 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:42:06.442 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:42:06.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:42:06 trial068 ceph-mon[38805]: 5.a scrub starts 2026-02-23T13:42:06.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:42:06 trial068 ceph-mon[38805]: 5.a scrub ok 2026-02-23T13:42:06.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:42:06 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2593416609' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:42:07.233 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1721 8 365281 365273 1.65803 0 - 0.0299513 2026-02-23T13:42:07.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:42:07 trial068 ceph-mon[38805]: pgmap v1264: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:42:08.234 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1722 8 365281 365273 1.65707 0 - 0.0299513 2026-02-23T13:42:08.444 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:42:08.566 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:42:08.664 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:42:08.662+0000 7f34b7c2c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:42:08.664 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:42:08.662+0000 7f34b7c2c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:42:08.676 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:42:08.676 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:42:08.686 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:42:08.686 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:42:08.687 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:42:08.696 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:42:08.697 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:42:08.740 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:42:08.740 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:42:09.234 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1723 8 365281 365273 1.65611 0 - 0.0299513 2026-02-23T13:42:09.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:42:09 trial068 ceph-mon[38805]: 5.5 scrub starts 2026-02-23T13:42:09.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:42:09 trial068 ceph-mon[38805]: 5.5 scrub ok 2026-02-23T13:42:09.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:42:09 trial068 ceph-mon[38805]: pgmap v1265: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:42:09.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:42:09 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2611177585' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:42:10.234 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1724 8 365281 365273 1.65515 0 - 0.0299513 2026-02-23T13:42:10.741 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:42:10.858 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:42:10.954 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:42:10.952+0000 7f177cb0c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:42:10.954 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:42:10.952+0000 7f177cb0c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:42:10.966 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:42:10.966 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:42:10.976 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:42:10.976 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:42:10.976 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:42:10.987 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:42:10.987 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:42:11.018 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:42:11.019 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:42:11.234 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1725 8 365281 365273 1.65419 0 - 0.0299513 2026-02-23T13:42:11.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:42:11 trial068 ceph-mon[38805]: pgmap v1266: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:42:11.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:42:11 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2698730583' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:42:12.234 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1726 8 365281 365273 1.65323 0 - 0.0299513 2026-02-23T13:42:13.020 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:42:13.138 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:42:13.234 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1727 8 365281 365273 1.65227 0 - 0.0299513 2026-02-23T13:42:13.240 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:42:13.238+0000 7fd297a5c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:42:13.240 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:42:13.238+0000 7fd297a5c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:42:13.252 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:42:13.252 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:42:13.262 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:42:13.262 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:42:13.262 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:42:13.273 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:42:13.273 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:42:13.304 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:42:13.304 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:42:13.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:42:13 trial068 ceph-mon[38805]: pgmap v1267: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:42:13.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:42:13 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/4221921813' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:42:14.234 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1728 8 365281 365273 1.65131 0 - 0.0299513 2026-02-23T13:42:14.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:42:13 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:42:13] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:42:14.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:42:13 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:42:13.972Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:42:15.234 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1729 8 365281 365273 1.65036 0 - 0.0299513 2026-02-23T13:42:15.305 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:42:15.422 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:42:15.520 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:42:15.518+0000 7f9626d10640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:42:15.520 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:42:15.518+0000 7f9626d10640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:42:15.520 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:42:15 trial068 ceph-mon[38805]: pgmap v1268: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:42:15.532 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:42:15.532 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:42:15.542 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:42:15.542 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:42:15.542 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:42:15.552 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:42:15.552 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:42:15.580 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:42:15.580 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:42:16.234 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1730 8 365281 365273 1.6494 0 - 0.0299513 2026-02-23T13:42:16.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:42:16 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3386834249' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:42:17.234 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1731 8 365281 365273 1.64845 0 - 0.0299513 2026-02-23T13:42:17.581 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:42:17.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:42:17 trial068 ceph-mon[38805]: pgmap v1269: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:42:17.701 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:42:17.797 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:42:17.795+0000 7f1292f04640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:42:17.797 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:42:17.796+0000 7f1292f04640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:42:17.809 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:42:17.809 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:42:17.819 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:42:17.820 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:42:17.820 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:42:17.830 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:42:17.830 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:42:17.861 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:42:17.861 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:42:18.234 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1732 8 365281 365273 1.6475 0 - 0.0299513 2026-02-23T13:42:18.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:42:18 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3512408485' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:42:19.235 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1733 8 365281 365273 1.64655 0 - 0.0299513 2026-02-23T13:42:19.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:42:19 trial068 ceph-mon[38805]: pgmap v1270: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:42:19.862 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:42:19.978 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:42:20.077 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:42:20.075+0000 7f3e9d319640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:42:20.077 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:42:20.075+0000 7f3e9d319640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:42:20.088 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:42:20.089 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:42:20.099 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:42:20.099 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:42:20.099 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:42:20.109 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:42:20.109 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:42:20.138 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:42:20.139 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:42:20.235 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1734 8 365281 365273 1.6456 0 - 0.0299513 2026-02-23T13:42:20.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:42:20 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3329142486' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:42:21.235 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1735 8 365281 365273 1.64465 0 - 0.0299513 2026-02-23T13:42:21.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:42:21 trial068 ceph-mon[38805]: pgmap v1271: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:42:22.139 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:42:22.235 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1736 8 365281 365273 1.6437 0 - 0.0299513 2026-02-23T13:42:22.256 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:42:22.354 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:42:22.352+0000 7f455f391640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:42:22.354 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:42:22.352+0000 7f455f391640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:42:22.366 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:42:22.367 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:42:22.377 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:42:22.377 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:42:22.378 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:42:22.387 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:42:22.387 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:42:22.417 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:42:22 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1016782276' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:42:22.418 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:42:22.419 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:42:23.235 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1737 8 365281 365273 1.64276 0 - 0.0299513 2026-02-23T13:42:23.745 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:42:23 trial068 ceph-mon[38805]: pgmap v1272: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:42:24.094 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:42:23 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:42:23] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:42:24.235 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1738 8 365281 365273 1.64181 0 - 0.0299513 2026-02-23T13:42:24.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:42:23 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:42:23.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:42:24.420 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:42:24.539 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:42:24.634 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:42:24.632+0000 7f905ba7b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:42:24.635 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:42:24.633+0000 7f905ba7b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:42:24.647 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:42:24.647 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:42:24.657 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:42:24.657 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:42:24.657 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:42:24.667 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:42:24.668 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:42:24.700 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:42:24.700 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:42:25.235 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1739 8 365281 365273 1.64087 0 - 0.0299513 2026-02-23T13:42:25.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:42:25 trial068 ceph-mon[38805]: pgmap v1273: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:42:25.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:42:25 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3456027876' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:42:26.235 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T13:42:26.235017+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T13:42:26.236 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T13:42:26.237 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1740 8 365281 365273 1.63993 0 - 0.0299513 2026-02-23T13:42:26.701 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:42:26.819 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:42:26.925 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:42:26.922+0000 7fe79096f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:42:26.925 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:42:26.923+0000 7fe79096f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:42:26.936 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:42:26.936 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:42:26.946 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:42:26.947 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:42:26.947 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:42:26.956 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:42:26.956 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:42:26.986 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:42:26.987 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:42:27.235 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1741 8 365281 365273 1.63898 0 - 0.0299513 2026-02-23T13:42:27.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:42:27 trial068 ceph-mon[38805]: pgmap v1274: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:42:27.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:42:27 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/862315783' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:42:28.235 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1742 8 365281 365273 1.63804 0 - 0.0299513 2026-02-23T13:42:28.987 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:42:29.106 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:42:29.209 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:42:29.207+0000 7f4598e4c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:42:29.209 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:42:29.207+0000 7f4598e4c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:42:29.221 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:42:29.221 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:42:29.231 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:42:29.231 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:42:29.232 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:42:29.235 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1743 8 365281 365273 1.6371 0 - 0.0299513 2026-02-23T13:42:29.241 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:42:29.241 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:42:29.272 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:42:29.273 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:42:29.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:42:29 trial068 ceph-mon[38805]: pgmap v1275: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:42:29.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:42:29 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1991775397' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:42:30.236 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1744 8 365281 365273 1.63616 0 - 0.0299513 2026-02-23T13:42:30.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:42:30 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T13:42:30.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:42:30 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T13:42:31.235 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1745 8 365281 365273 1.63523 0 - 0.0299513 2026-02-23T13:42:31.274 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:42:31.388 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:42:31.485 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:42:31.483+0000 7f314c90a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:42:31.485 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:42:31.484+0000 7f314c90a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:42:31.498 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:42:31.498 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:42:31.508 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:42:31.508 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:42:31.508 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:42:31.518 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:42:31.518 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:42:31.550 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:42:31 trial068 ceph-mon[38805]: pgmap v1276: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:42:31.551 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:42:31.551 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:42:32.236 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1746 8 365281 365273 1.63429 0 - 0.0299513 2026-02-23T13:42:32.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:42:32 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3999121659' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:42:33.235 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1747 8 365281 365273 1.63335 0 - 0.0299513 2026-02-23T13:42:33.552 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:42:33.672 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:42:33.717 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:42:33 trial068 ceph-mon[38805]: pgmap v1277: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:42:33.774 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:42:33.772+0000 7f7ace387640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:42:33.774 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:42:33.772+0000 7f7ace387640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:42:33.785 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:42:33.786 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:42:33.795 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:42:33.796 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:42:33.796 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:42:33.807 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:42:33.807 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:42:33.838 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:42:33.839 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:42:34.236 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1748 8 365281 365273 1.63242 0 - 0.0299513 2026-02-23T13:42:34.265 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:42:33 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:42:33.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:42:34.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:42:33 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:42:33] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:42:34.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:42:34 trial068 ceph-mon[38805]: 5.e scrub starts 2026-02-23T13:42:34.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:42:34 trial068 ceph-mon[38805]: 5.e scrub ok 2026-02-23T13:42:34.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:42:34 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/515544984' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:42:35.236 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1749 8 365281 365273 1.63149 0 - 0.0299513 2026-02-23T13:42:35.839 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:42:35.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:42:35 trial068 ceph-mon[38805]: pgmap v1278: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:42:35.958 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:42:36.058 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:42:36.056+0000 7f06346ee640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:42:36.059 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:42:36.056+0000 7f06346ee640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:42:36.070 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:42:36.070 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:42:36.081 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:42:36.081 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:42:36.081 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:42:36.092 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:42:36.093 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:42:36.121 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:42:36.122 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:42:36.236 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1750 8 365281 365273 1.63055 0 - 0.0299513 2026-02-23T13:42:36.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:42:36 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2382242078' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:42:37.236 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1751 8 365281 365273 1.62962 0 - 0.0299513 2026-02-23T13:42:37.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:42:37 trial068 ceph-mon[38805]: pgmap v1279: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:42:38.123 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:42:38.236 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1752 8 365281 365273 1.62869 0 - 0.0299513 2026-02-23T13:42:38.243 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:42:38.344 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:42:38.342+0000 7f2fd123c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:42:38.344 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:42:38.342+0000 7f2fd123c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:42:38.356 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:42:38.356 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:42:38.366 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:42:38.366 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:42:38.366 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:42:38.376 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:42:38.377 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:42:38.407 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:42:38.407 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:42:38.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:42:38 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3227270815' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:42:39.236 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1753 8 365281 365273 1.62776 0 - 0.0299513 2026-02-23T13:42:39.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:42:39 trial068 ceph-mon[38805]: pgmap v1280: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:42:40.236 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1754 8 365281 365273 1.62684 0 - 0.0299513 2026-02-23T13:42:40.408 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:42:40.528 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:42:40.627 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:42:40.625+0000 7f924c281640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:42:40.627 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:42:40.625+0000 7f924c281640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:42:40.638 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:42:40.639 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:42:40.648 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:42:40.649 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:42:40.649 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:42:40.660 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:42:40.660 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:42:40.689 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:42:40.689 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:42:41.236 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1755 8 365281 365273 1.62591 0 - 0.0299513 2026-02-23T13:42:41.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:42:41 trial068 ceph-mon[38805]: pgmap v1281: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:42:41.890 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:42:41 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1452133733' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:42:42.236 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1756 8 365281 365273 1.62498 0 - 0.0299513 2026-02-23T13:42:42.691 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:42:42.804 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:42:42.826 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:42:42 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2026-02-23T13:42:42.900 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:42:42.898+0000 7fd6b2230640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:42:42.900 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:42:42.898+0000 7fd6b2230640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:42:42.913 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:42:42.914 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:42:42.923 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:42:42.924 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:42:42.924 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:42:42.934 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:42:42.935 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:42:42.965 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:42:42.965 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:42:43.236 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1757 8 365281 365273 1.62406 0 - 0.0299513 2026-02-23T13:42:43.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:42:43 trial068 ceph-mon[38805]: pgmap v1282: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:42:43.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:42:43 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-23T13:42:43.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:42:43 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2026-02-23T13:42:43.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:42:43 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T13:42:43.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:42:43 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2026-02-23T13:42:43.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:42:43 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/4089909609' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:42:44.237 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1758 8 365281 365273 1.62313 0 - 0.0299513 2026-02-23T13:42:44.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:42:43 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:42:43] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:42:44.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:42:43 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:42:43.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:42:44.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:42:44 trial068 ceph-mon[38805]: pgmap v1283: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:42:44.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:42:44 trial068 ceph-mon[38805]: pgmap v1284: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:42:44.966 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:42:45.089 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:42:45.185 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:42:45.183+0000 7f3ceb0f7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:42:45.185 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:42:45.183+0000 7f3ceb0f7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:42:45.197 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:42:45.198 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:42:45.207 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:42:45.208 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:42:45.208 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:42:45.219 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:42:45.219 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:42:45.237 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1759 8 365281 365273 1.62221 0 - 0.0299513 2026-02-23T13:42:45.248 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:42:45.249 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:42:45.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:42:45 trial068 ceph-mon[38805]: 5.7 scrub starts 2026-02-23T13:42:45.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:42:45 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1168506485' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:42:46.237 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T13:42:46.236483+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T13:42:46.237 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T13:42:46.237 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1760 8 365281 365273 1.62129 0 - 0.0299513 2026-02-23T13:42:46.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:42:46 trial068 ceph-mon[38805]: pgmap v1285: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:42:46.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:42:46 trial068 ceph-mon[38805]: 5.7 scrub ok 2026-02-23T13:42:47.237 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1761 8 365281 365273 1.62037 0 - 0.0299513 2026-02-23T13:42:47.250 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:42:47.367 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:42:47.464 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:42:47.461+0000 7ff5aa148640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:42:47.464 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:42:47.462+0000 7ff5aa148640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:42:47.475 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:42:47.476 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:42:47.487 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:42:47.487 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:42:47.488 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:42:47.497 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:42:47.498 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:42:47.527 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:42:47 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/640615267' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:42:47.528 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:42:47.529 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:42:48.237 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1762 8 365281 365273 1.61945 0 - 0.0299513 2026-02-23T13:42:48.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:42:48 trial068 ceph-mon[38805]: pgmap v1286: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:42:49.237 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1763 8 365281 365273 1.61853 0 - 0.0299513 2026-02-23T13:42:49.530 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:42:49.650 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:42:49.754 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:42:49.752+0000 7fea6c57d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:42:49.754 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:42:49.752+0000 7fea6c57d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:42:49.766 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:42:49.766 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:42:49.776 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:42:49.776 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:42:49.777 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:42:49.786 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:42:49.787 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:42:49.818 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:42:49.819 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:42:50.237 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1764 8 365281 365273 1.61761 0 - 0.0299513 2026-02-23T13:42:50.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:42:50 trial068 ceph-mon[38805]: pgmap v1287: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:42:50.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:42:50 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1338206954' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:42:51.237 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1765 8 365281 365273 1.6167 0 - 0.0299513 2026-02-23T13:42:51.820 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:42:51.938 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:42:52.036 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:42:52.034+0000 7fb598075640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:42:52.036 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:42:52.034+0000 7fb598075640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:42:52.048 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:42:52.048 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:42:52.058 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:42:52.058 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:42:52.059 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:42:52.068 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:42:52.069 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:42:52.098 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:42:52.099 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:42:52.237 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1766 8 365281 365273 1.61578 0 - 0.0299513 2026-02-23T13:42:52.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:42:52 trial068 ceph-mon[38805]: pgmap v1288: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:42:52.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:42:52 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3369116979' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:42:53.237 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1767 8 365281 365273 1.61487 0 - 0.0299513 2026-02-23T13:42:54.101 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:42:54.217 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:42:54.237 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1768 8 365281 365273 1.61395 0 - 0.0299513 2026-02-23T13:42:54.240 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:42:53 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:42:53] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:42:54.320 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:42:54.318+0000 7f8df2a27640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:42:54.320 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:42:54.318+0000 7f8df2a27640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:42:54.332 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:42:54.332 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:42:54.343 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:42:54.343 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:42:54.343 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:42:54.354 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:42:54.354 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:42:54.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:42:53 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:42:53.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:42:54.386 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:42:54.387 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:42:54.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:42:54 trial068 ceph-mon[38805]: pgmap v1289: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:42:54.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:42:54 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2736599730' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:42:55.237 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1769 8 365281 365273 1.61304 0 - 0.0299513 2026-02-23T13:42:56.237 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1770 8 365281 365273 1.61213 0 - 0.0299513 2026-02-23T13:42:56.387 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:42:56.506 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:42:56.604 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:42:56.603+0000 7fb9e0678640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:42:56.605 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:42:56.603+0000 7fb9e0678640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:42:56.617 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:42:56.617 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:42:56.627 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:42:56.627 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:42:56.628 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:42:56.638 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:42:56.638 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:42:56.668 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:42:56 trial068 ceph-mon[38805]: pgmap v1290: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:42:56.668 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:42:56 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T13:42:56.668 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:42:56 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T13:42:56.669 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:42:56.670 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:42:57.237 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1771 8 365281 365273 1.61122 0 - 0.0299513 2026-02-23T13:42:57.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:42:57 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2201101604' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:42:58.237 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1772 8 365281 365273 1.61031 0 - 0.0299513 2026-02-23T13:42:58.670 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:42:58.786 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:42:58.808 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:42:58 trial068 ceph-mon[38805]: pgmap v1291: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:42:58.808 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:42:58 trial068 ceph-mon[38805]: 5.8 scrub starts 2026-02-23T13:42:58.808 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:42:58 trial068 ceph-mon[38805]: 5.8 scrub ok 2026-02-23T13:42:58.888 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:42:58.886+0000 7f039a393640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:42:58.888 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:42:58.886+0000 7f039a393640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:42:58.899 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:42:58.900 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:42:58.910 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:42:58.910 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:42:58.910 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:42:58.920 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:42:58.920 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:42:58.951 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:42:58.951 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:42:59.237 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1773 8 365281 365273 1.6094 0 - 0.0299513 2026-02-23T13:42:59.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:42:59 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3933896820' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:43:00.237 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1774 8 365281 365273 1.6085 0 - 0.0299513 2026-02-23T13:43:00.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:43:00 trial068 ceph-mon[38805]: pgmap v1292: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:43:00.953 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:43:01.073 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:43:01.171 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:43:01.168+0000 7ffa820c5640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:43:01.171 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:43:01.169+0000 7ffa820c5640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:43:01.182 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:43:01.183 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:43:01.193 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:43:01.194 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:43:01.194 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:43:01.203 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:43:01.203 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:43:01.230 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:43:01.231 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:43:01.237 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1775 8 365281 365273 1.60759 0 - 0.0299513 2026-02-23T13:43:01.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:43:01 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2112078597' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:43:02.238 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1776 8 365281 365273 1.60668 0 - 0.0299513 2026-02-23T13:43:02.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:43:02 trial068 ceph-mon[38805]: pgmap v1293: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:43:03.233 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:43:03.238 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1777 8 365281 365273 1.60578 0 - 0.0299513 2026-02-23T13:43:03.349 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:43:03.446 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:43:03.444+0000 7f9a945e0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:43:03.446 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:43:03.444+0000 7f9a945e0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:43:03.457 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:43:03.458 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:43:03.468 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:43:03.468 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:43:03.468 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:43:03.478 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:43:03.478 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:43:03.508 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:43:03 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/964679157' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:43:03.509 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:43:03.510 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:43:04.238 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1778 8 365281 365273 1.60488 0 - 0.0299513 2026-02-23T13:43:04.265 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:43:03 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:43:03.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:43:04.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:43:03 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:43:03] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:43:04.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:43:04 trial068 ceph-mon[38805]: pgmap v1294: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:43:05.238 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1779 8 365281 365273 1.60397 0 - 0.0299513 2026-02-23T13:43:05.511 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:43:05.631 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:43:05.727 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:43:05.724+0000 7f3549822640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:43:05.728 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:43:05.724+0000 7f3549822640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:43:05.738 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:43:05.738 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:43:05.749 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:43:05.749 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:43:05.749 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:43:05.760 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:43:05.760 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:43:05.788 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:43:05.789 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:43:06.238 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T13:43:06.237996+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T13:43:06.238 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T13:43:06.239 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1780 8 365281 365273 1.60307 0 - 0.0299513 2026-02-23T13:43:06.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:43:06 trial068 ceph-mon[38805]: pgmap v1295: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:43:06.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:43:06 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3160994240' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:43:07.238 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1781 8 365281 365273 1.60217 0 - 0.0299513 2026-02-23T13:43:07.789 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:43:07.907 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:43:08.004 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:43:08.002+0000 7f2046bbd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:43:08.004 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:43:08.002+0000 7f2046bbd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:43:08.016 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:43:08.016 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:43:08.026 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:43:08.026 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:43:08.026 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:43:08.036 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:43:08.036 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:43:08.068 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:43:08.069 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:43:08.238 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1782 8 365281 365273 1.60127 0 - 0.0299513 2026-02-23T13:43:08.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:43:08 trial068 ceph-mon[38805]: pgmap v1296: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:43:08.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:43:08 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3528791331' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:43:09.238 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1783 8 365281 365273 1.60038 0 - 0.0299513 2026-02-23T13:43:09.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:43:09 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T13:43:09.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:43:09 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T13:43:10.070 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:43:10.189 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:43:10.239 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1784 8 365281 365273 1.59948 0 - 0.0299513 2026-02-23T13:43:10.293 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:43:10.291+0000 7f14bfe11640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:43:10.293 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:43:10.291+0000 7f14bfe11640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:43:10.304 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:43:10.305 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:43:10.315 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:43:10.315 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:43:10.315 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:43:10.325 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:43:10.325 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:43:10.357 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:43:10.358 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:43:10.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:43:10 trial068 ceph-mon[38805]: pgmap v1297: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:43:10.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:43:10 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3114777135' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:43:11.239 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1785 8 365281 365273 1.59858 0 - 0.0299513 2026-02-23T13:43:12.239 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1786 8 365281 365273 1.59769 0 - 0.0299513 2026-02-23T13:43:12.359 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:43:12.484 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:43:12.585 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:43:12.582+0000 7f5fa4130640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:43:12.585 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:43:12.583+0000 7f5fa4130640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:43:12.596 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:43:12.596 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:43:12.606 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:43:12.607 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:43:12.607 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:43:12.618 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:43:12.618 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:43:12.647 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:43:12 trial068 ceph-mon[38805]: pgmap v1298: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:43:12.647 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:43:12.648 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:43:13.239 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1787 8 365281 365273 1.59679 0 - 0.0299513 2026-02-23T13:43:13.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:43:13 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2576224850' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:43:14.239 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1788 8 365281 365273 1.5959 0 - 0.0299513 2026-02-23T13:43:14.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:43:13 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:43:13] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:43:14.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:43:13 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:43:13.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:43:14.649 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:43:14.766 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:43:14.787 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:43:14 trial068 ceph-mon[38805]: pgmap v1299: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:43:14.862 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:43:14.860+0000 7fd4d3d17640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:43:14.863 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:43:14.861+0000 7fd4d3d17640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:43:14.874 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:43:14.875 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:43:14.885 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:43:14.886 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:43:14.886 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:43:14.897 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:43:14.897 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:43:14.928 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:43:14.929 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:43:15.239 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1789 8 365281 365273 1.59501 0 - 0.0299513 2026-02-23T13:43:15.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:43:15 trial068 ceph-mon[38805]: 5.a scrub starts 2026-02-23T13:43:15.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:43:15 trial068 ceph-mon[38805]: 5.a scrub ok 2026-02-23T13:43:15.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:43:15 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/377758113' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:43:16.239 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1790 8 365281 365273 1.59412 0 - 0.0299513 2026-02-23T13:43:16.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:43:16 trial068 ceph-mon[38805]: pgmap v1300: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:43:16.930 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:43:17.048 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:43:17.144 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:43:17.142+0000 7fa24b1e2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:43:17.145 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:43:17.142+0000 7fa24b1e2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:43:17.156 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:43:17.156 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:43:17.166 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:43:17.166 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:43:17.166 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:43:17.176 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:43:17.176 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:43:17.207 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:43:17.207 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:43:17.239 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1791 8 365281 365273 1.59323 0 - 0.0299513 2026-02-23T13:43:17.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:43:17 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2589709369' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:43:18.239 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1792 8 365281 365273 1.59234 0 - 0.0299513 2026-02-23T13:43:18.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:43:18 trial068 ceph-mon[38805]: pgmap v1301: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:43:19.208 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:43:19.239 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1793 8 365281 365273 1.59145 0 - 0.0299513 2026-02-23T13:43:19.325 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:43:19.423 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:43:19.421+0000 7f8e3df78640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:43:19.423 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:43:19.422+0000 7f8e3df78640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:43:19.435 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:43:19.435 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:43:19.445 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:43:19.446 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:43:19.446 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:43:19.456 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:43:19.456 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:43:19.487 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:43:19.487 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:43:19.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:43:19 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3403166663' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:43:20.239 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1794 8 365281 365273 1.59056 0 - 0.0299513 2026-02-23T13:43:20.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:43:20 trial068 ceph-mon[38805]: pgmap v1302: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:43:21.239 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1795 8 365281 365273 1.58968 0 - 0.0299513 2026-02-23T13:43:21.489 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:43:21.607 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:43:21.708 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:43:21.706+0000 7f3828b08640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:43:21.708 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:43:21.706+0000 7f3828b08640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:43:21.719 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:43:21.720 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:43:21.730 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:43:21.731 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:43:21.731 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:43:21.740 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:43:21.740 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:43:21.770 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:43:21.771 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:43:22.239 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1796 8 365281 365273 1.58879 0 - 0.0299513 2026-02-23T13:43:22.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:43:22 trial068 ceph-mon[38805]: pgmap v1303: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:43:22.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:43:22 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1471605838' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:43:23.239 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1797 8 365281 365273 1.58791 0 - 0.0299513 2026-02-23T13:43:23.771 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:43:23.891 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:43:23.990 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:43:23.988+0000 7fb7ddc53640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:43:23.990 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:43:23.988+0000 7fb7ddc53640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:43:24.002 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:43:24.003 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:43:24.013 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:43:24.014 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:43:24.014 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:43:24.024 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:43:24.024 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:43:24.054 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:43:23 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:43:23] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:43:24.055 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:43:24.055 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:43:24.239 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1798 8 365281 365273 1.58702 0 - 0.0299513 2026-02-23T13:43:24.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:43:23 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:43:23.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:43:24.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:43:24 trial068 ceph-mon[38805]: pgmap v1304: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:43:24.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:43:24 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/264321957' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:43:25.240 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1799 8 365281 365273 1.58614 0 - 0.0299513 2026-02-23T13:43:26.057 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:43:26.180 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:43:26.240 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T13:43:26.239467+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T13:43:26.240 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T13:43:26.240 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1800 8 365281 365273 1.58526 0 - 0.0299513 2026-02-23T13:43:26.284 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:43:26.283+0000 7f6c4f196640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:43:26.285 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:43:26.283+0000 7f6c4f196640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:43:26.296 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:43:26.296 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:43:26.306 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:43:26.306 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:43:26.307 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:43:26.316 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:43:26.317 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:43:26.347 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:43:26.348 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:43:26.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:43:26 trial068 ceph-mon[38805]: pgmap v1305: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:43:26.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:43:26 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/388043327' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:43:27.240 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1801 8 365281 365273 1.58438 0 - 0.0299513 2026-02-23T13:43:28.240 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1802 8 365281 365273 1.5835 0 - 0.0299513 2026-02-23T13:43:28.349 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:43:28.468 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:43:28.566 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:43:28.564+0000 7f8c98c3a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:43:28.566 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:43:28.564+0000 7f8c98c3a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:43:28.578 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:43:28.578 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:43:28.588 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:43:28.588 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:43:28.588 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:43:28.598 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:43:28.598 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:43:28.628 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:43:28 trial068 ceph-mon[38805]: pgmap v1306: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:43:28.628 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:43:28.629 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:43:29.240 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1803 8 365281 365273 1.58262 0 - 0.0299513 2026-02-23T13:43:29.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:43:29 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1646838275' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:43:30.240 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1804 8 365281 365273 1.58175 0 - 0.0299513 2026-02-23T13:43:30.630 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:43:30.748 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:43:30.794 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:43:30 trial068 ceph-mon[38805]: pgmap v1307: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:43:30.851 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:43:30.849+0000 7ff0ac48d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:43:30.851 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:43:30.850+0000 7ff0ac48d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:43:30.862 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:43:30.863 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:43:30.872 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:43:30.873 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:43:30.873 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:43:30.883 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:43:30.883 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:43:30.915 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:43:30.915 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:43:31.240 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1805 8 365281 365273 1.58087 0 - 0.0299513 2026-02-23T13:43:31.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:43:31 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1573790039' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:43:32.240 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1806 8 365281 365273 1.57999 0 - 0.0299513 2026-02-23T13:43:32.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:43:32 trial068 ceph-mon[38805]: pgmap v1308: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:43:32.916 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:43:33.035 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:43:33.131 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:43:33.129+0000 7f0afd5d1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:43:33.132 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:43:33.130+0000 7f0afd5d1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:43:33.143 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:43:33.144 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:43:33.153 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:43:33.154 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:43:33.154 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:43:33.164 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:43:33.164 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:43:33.194 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:43:33.195 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:43:33.240 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1807 8 365281 365273 1.57912 0 - 0.0299513 2026-02-23T13:43:33.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:43:33 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3099528069' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:43:34.240 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1808 8 365281 365273 1.57825 0 - 0.0299513 2026-02-23T13:43:34.265 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:43:33 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:43:33.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:43:34.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:43:33 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:43:33] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:43:34.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:43:34 trial068 ceph-mon[38805]: pgmap v1309: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:43:34.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:43:34 trial068 ceph-mon[38805]: 5.5 scrub starts 2026-02-23T13:43:34.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:43:34 trial068 ceph-mon[38805]: 5.5 scrub ok 2026-02-23T13:43:35.196 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:43:35.240 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1809 8 365281 365273 1.57737 0 - 0.0299513 2026-02-23T13:43:35.315 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:43:35.416 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:43:35.414+0000 7f19f87c7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:43:35.416 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:43:35.414+0000 7f19f87c7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:43:35.428 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:43:35.429 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:43:35.439 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:43:35.439 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:43:35.439 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:43:35.449 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:43:35.450 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:43:35.479 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:43:35.480 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:43:35.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:43:35 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3939576455' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:43:36.240 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1810 8 365281 365273 1.5765 0 - 0.0299513 2026-02-23T13:43:36.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:43:36 trial068 ceph-mon[38805]: pgmap v1310: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:43:37.240 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1811 8 365281 365273 1.57563 0 - 0.0299513 2026-02-23T13:43:37.481 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:43:37.600 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:43:37.701 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:43:37.699+0000 7fc785947640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:43:37.701 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:43:37.700+0000 7fc785947640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:43:37.713 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:43:37.714 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:43:37.724 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:43:37.724 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:43:37.725 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:43:37.734 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:43:37.734 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:43:37.764 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:43:37 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T13:43:37.764 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:43:37 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T13:43:37.765 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:43:37.765 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:43:38.240 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1812 8 365281 365273 1.57476 0 - 0.0299513 2026-02-23T13:43:38.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:43:38 trial068 ceph-mon[38805]: pgmap v1311: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:43:38.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:43:38 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3632969177' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:43:39.241 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1813 8 365281 365273 1.57389 0 - 0.0299513 2026-02-23T13:43:39.766 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:43:39.885 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:43:39.984 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:43:39.982+0000 7fd84ffcd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:43:39.984 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:43:39.982+0000 7fd84ffcd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:43:39.996 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:43:39.996 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:43:40.006 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:43:40.007 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:43:40.007 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:43:40.017 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:43:40.017 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:43:40.047 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:43:40.048 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:43:40.241 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1814 8 365281 365273 1.57303 0 - 0.0299513 2026-02-23T13:43:40.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:43:40 trial068 ceph-mon[38805]: pgmap v1312: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:43:40.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:43:40 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2817679286' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:43:41.241 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1815 8 365281 365273 1.57216 0 - 0.0299513 2026-02-23T13:43:42.049 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:43:42.166 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:43:42.241 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1816 8 365281 365273 1.57129 0 - 0.0299513 2026-02-23T13:43:42.263 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:43:42.261+0000 7fe360da8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:43:42.263 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:43:42.262+0000 7fe360da8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:43:42.276 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:43:42.276 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:43:42.286 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:43:42.286 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:43:42.286 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:43:42.296 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:43:42.296 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:43:42.327 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:43:42.327 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:43:42.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:43:42 trial068 ceph-mon[38805]: pgmap v1313: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:43:42.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:43:42 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1166231430' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:43:43.241 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1817 8 365281 365273 1.57043 0 - 0.0299513 2026-02-23T13:43:43.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:43:43 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2026-02-23T13:43:43.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:43:43 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-23T13:43:43.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:43:43 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2026-02-23T13:43:43.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:43:43 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T13:43:43.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:43:43 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2026-02-23T13:43:44.241 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1818 8 365281 365273 1.56957 0 - 0.0299513 2026-02-23T13:43:44.329 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:43:44.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:43:43 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:43:43] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:43:44.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:43:43 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:43:43.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:43:44.443 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:43:44.542 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:43:44.540+0000 7fb446b2b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:43:44.542 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:43:44.541+0000 7fb446b2b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:43:44.555 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:43:44.555 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:43:44.565 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:43:44.565 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:43:44.566 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:43:44.577 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:43:44.578 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:43:44.606 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:43:44 trial068 ceph-mon[38805]: pgmap v1314: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:43:44.606 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:43:44 trial068 ceph-mon[38805]: pgmap v1315: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:43:44.606 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:43:44 trial068 ceph-mon[38805]: pgmap v1316: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:43:44.606 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:43:44 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/339068309' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:43:44.607 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:43:44.607 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:43:45.241 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1819 8 365281 365273 1.5687 0 - 0.0299513 2026-02-23T13:43:46.241 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T13:43:46.241060+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T13:43:46.241 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T13:43:46.241 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1820 8 365281 365273 1.56784 0 - 0.0299513 2026-02-23T13:43:46.608 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:43:46.726 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:43:46.823 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:43:46.821+0000 7f669471b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:43:46.823 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:43:46 trial068 ceph-mon[38805]: pgmap v1317: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:43:46.824 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:43:46.821+0000 7f669471b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:43:46.835 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:43:46.835 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:43:46.845 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:43:46.846 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:43:46.846 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:43:46.856 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:43:46.856 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:43:46.887 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:43:46.888 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:43:47.241 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1821 8 365281 365273 1.56698 0 - 0.0299513 2026-02-23T13:43:47.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:43:47 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3384137962' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:43:48.241 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1822 8 365281 365273 1.56612 0 - 0.0299513 2026-02-23T13:43:48.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:43:48 trial068 ceph-mon[38805]: pgmap v1318: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:43:48.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:43:48 trial068 ceph-mon[38805]: osdmap e205: 4 total, 3 up, 4 in 2026-02-23T13:43:48.888 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:43:49.007 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:43:49.105 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:43:49.103+0000 7fdaabbe7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:43:49.105 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:43:49.104+0000 7fdaabbe7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:43:49.117 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:43:49.117 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:43:49.127 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:43:49.127 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:43:49.128 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:43:49.137 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:43:49.137 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:43:49.167 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:43:49.167 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:43:49.241 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1823 8 365281 365273 1.56526 0 - 0.0299513 2026-02-23T13:43:49.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:43:49 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1122514773' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:43:50.242 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1824 8 365281 365273 1.5644 0 - 0.0299513 2026-02-23T13:43:50.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:43:50 trial068 ceph-mon[38805]: pgmap v1320: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:43:51.169 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:43:51.242 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1825 8 365281 365273 1.56355 0 - 0.0299513 2026-02-23T13:43:51.286 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:43:51.385 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:43:51.382+0000 7f978e645640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:43:51.385 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:43:51.383+0000 7f978e645640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:43:51.396 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:43:51.397 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:43:51.407 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:43:51.408 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:43:51.408 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:43:51.417 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:43:51.418 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:43:51.447 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:43:51.448 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:43:51.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:43:51 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2866372827' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:43:52.242 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1826 8 365281 365273 1.56269 0 - 0.0299513 2026-02-23T13:43:52.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:43:52 trial068 ceph-mon[38805]: pgmap v1321: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:43:53.242 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1827 8 365281 365273 1.56183 0 - 0.0299513 2026-02-23T13:43:53.448 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:43:53.568 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:43:53.667 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:43:53.665+0000 7fe9bd91d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:43:53.667 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:43:53.665+0000 7fe9bd91d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:43:53.679 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:43:53.679 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:43:53.690 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:43:53.690 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:43:53.691 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:43:53.700 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:43:53.700 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:43:53.731 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:43:53.731 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:43:54.094 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:43:53 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:43:53] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:43:54.242 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1828 8 365281 365273 1.56098 0 - 0.0299513 2026-02-23T13:43:54.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:43:53 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:43:53.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:43:54.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:43:54 trial068 ceph-mon[38805]: pgmap v1322: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:43:54.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:43:54 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/109455709' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:43:55.242 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1829 8 365281 365273 1.56013 0 - 0.0299513 2026-02-23T13:43:55.732 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:43:55.851 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:43:55.958 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:43:55.956+0000 7fd49289d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:43:55.958 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:43:55.957+0000 7fd49289d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:43:55.971 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:43:55.971 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:43:55.982 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:43:55.982 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:43:55.982 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:43:55.993 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:43:55.993 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:43:56.024 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:43:56.025 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:43:56.242 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1830 8 365281 365273 1.55927 0 - 0.0299513 2026-02-23T13:43:56.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:43:56 trial068 ceph-mon[38805]: pgmap v1323: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:43:56.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:43:56 trial068 ceph-mon[38805]: 5.e scrub starts 2026-02-23T13:43:56.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:43:56 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1359939786' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:43:57.242 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1831 8 365281 365273 1.55842 0 - 0.0299513 2026-02-23T13:43:57.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:43:57 trial068 ceph-mon[38805]: 5.e scrub ok 2026-02-23T13:43:58.026 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:43:58.143 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:43:58.240 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:43:58.239+0000 7f8e3a69c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:43:58.241 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:43:58.239+0000 7f8e3a69c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:43:58.242 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1832 8 365281 365273 1.55757 0 - 0.0299513 2026-02-23T13:43:58.252 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:43:58.252 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:43:58.262 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:43:58.262 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:43:58.263 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:43:58.274 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:43:58.274 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:43:58.318 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:43:58.319 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:43:58.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:43:58 trial068 ceph-mon[38805]: pgmap v1324: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:43:58.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:43:58 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3577988615' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:43:59.242 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1833 8 365281 365273 1.55672 0 - 0.0299513 2026-02-23T13:44:00.242 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1834 8 365281 365273 1.55587 0 - 0.0299513 2026-02-23T13:44:00.320 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:44:00.437 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:44:00.540 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:44:00.538+0000 7f3d13a84640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:44:00.540 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:44:00.538+0000 7f3d13a84640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:44:00.551 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:44:00.552 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:44:00.562 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:44:00.562 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:44:00.562 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:44:00.572 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:44:00.572 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:44:00.604 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:44:00 trial068 ceph-mon[38805]: pgmap v1325: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:44:00.604 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:44:00 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/688444889' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:44:00.605 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:44:00.605 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:44:01.242 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1835 8 365281 365273 1.55503 0 - 0.0299513 2026-02-23T13:44:02.242 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1836 8 365281 365273 1.55418 0 - 0.0299513 2026-02-23T13:44:02.606 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:44:02.725 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:44:02.821 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:44:02.820+0000 7f0ee2bba640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:44:02.822 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:44:02 trial068 ceph-mon[38805]: pgmap v1326: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:44:02.822 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:44:02.820+0000 7f0ee2bba640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:44:02.834 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:44:02.834 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:44:02.845 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:44:02.845 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:44:02.845 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:44:02.855 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:44:02.856 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:44:02.886 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:44:02.887 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:44:03.242 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1837 8 365281 365273 1.55333 0 - 0.0299513 2026-02-23T13:44:03.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:44:03 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3040916727' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:44:04.242 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1838 8 365281 365273 1.55249 0 - 0.0299513 2026-02-23T13:44:04.265 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:44:03 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:44:03.974Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:44:04.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:44:03 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:44:03] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:44:04.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:44:04 trial068 ceph-mon[38805]: pgmap v1327: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:44:04.888 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:44:05.006 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:44:05.103 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:44:05.101+0000 7f817e851640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:44:05.104 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:44:05.102+0000 7f817e851640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:44:05.115 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:44:05.116 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:44:05.125 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:44:05.126 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:44:05.126 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:44:05.136 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:44:05.136 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:44:05.169 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:44:05.169 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:44:05.243 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1839 8 365281 365273 1.55164 0 - 0.0299513 2026-02-23T13:44:05.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:44:05 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3874664196' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:44:06.243 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T13:44:06.242534+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T13:44:06.243 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T13:44:06.243 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1840 8 365281 365273 1.5508 0 - 0.0299513 2026-02-23T13:44:06.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:44:06 trial068 ceph-mon[38805]: pgmap v1328: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:44:07.170 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:44:07.243 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1841 8 365281 365273 1.54996 0 - 0.0299513 2026-02-23T13:44:07.290 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:44:07.387 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:44:07.386+0000 7f674678e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:44:07.388 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:44:07.386+0000 7f674678e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:44:07.399 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:44:07.399 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:44:07.409 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:44:07.409 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:44:07.410 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:44:07.419 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:44:07.420 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:44:07.450 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:44:07.451 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:44:07.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:44:07 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2558633517' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:44:08.243 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1842 8 365281 365273 1.54912 0 - 0.0299513 2026-02-23T13:44:08.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:44:08 trial068 ceph-mon[38805]: pgmap v1329: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:44:09.243 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1843 8 365281 365273 1.54828 0 - 0.0299513 2026-02-23T13:44:09.452 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:44:09.570 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:44:09.666 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:44:09.665+0000 7f07ed899640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:44:09.666 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:44:09.665+0000 7f07ed899640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:44:09.679 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:44:09.679 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:44:09.688 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:44:09.688 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:44:09.689 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:44:09.698 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:44:09.698 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:44:09.727 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:44:09.727 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:44:10.243 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1844 8 365281 365273 1.54744 0 - 0.0299513 2026-02-23T13:44:10.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:44:10 trial068 ceph-mon[38805]: pgmap v1330: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:44:10.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:44:10 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/648223678' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:44:10.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:44:10 trial068 ceph-mon[38805]: 5.7 scrub starts 2026-02-23T13:44:11.243 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1845 8 365281 365273 1.5466 0 - 0.0299513 2026-02-23T13:44:11.728 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:44:11.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:44:11 trial068 ceph-mon[38805]: 5.7 scrub ok 2026-02-23T13:44:11.846 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:44:11.947 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:44:11.945+0000 7f1f09524640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:44:11.947 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:44:11.945+0000 7f1f09524640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:44:11.959 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:44:11.959 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:44:11.969 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:44:11.969 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:44:11.969 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:44:11.981 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:44:11.981 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:44:12.011 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:44:12.011 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:44:12.243 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1846 8 365281 365273 1.54576 0 - 0.0299513 2026-02-23T13:44:12.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:44:12 trial068 ceph-mon[38805]: pgmap v1331: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:44:12.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:44:12 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/694661947' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:44:13.243 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1847 8 365281 365273 1.54492 0 - 0.0299513 2026-02-23T13:44:14.012 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:44:14.130 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:44:14.231 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:44:14.229+0000 7fbb0eb86640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:44:14.231 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:44:14.230+0000 7fbb0eb86640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:44:14.231 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:44:13 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:44:13] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:44:14.243 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1848 8 365281 365273 1.54409 0 - 0.0299513 2026-02-23T13:44:14.243 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:44:14.243 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:44:14.253 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:44:14.253 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:44:14.253 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:44:14.264 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:44:14.264 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:44:14.294 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:44:14.294 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:44:14.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:44:13 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:44:13.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:44:14.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:44:14 trial068 ceph-mon[38805]: pgmap v1332: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:44:14.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:44:14 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T13:44:14.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:44:14 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T13:44:14.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:44:14 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/4192257283' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:44:15.243 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1849 8 365281 365273 1.54325 0 - 0.0299513 2026-02-23T13:44:15.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:44:15 trial068 ceph-mon[38805]: 5.8 scrub starts 2026-02-23T13:44:15.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:44:15 trial068 ceph-mon[38805]: 5.8 scrub ok 2026-02-23T13:44:16.243 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1850 8 365281 365273 1.54242 0 - 0.0299513 2026-02-23T13:44:16.295 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:44:16.411 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:44:16.511 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:44:16.509+0000 7f2db8408640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:44:16.511 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:44:16.510+0000 7f2db8408640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:44:16.523 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:44:16.523 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:44:16.533 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:44:16.533 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:44:16.533 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:44:16.544 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:44:16.545 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:44:16.574 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:44:16.575 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:44:16.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:44:16 trial068 ceph-mon[38805]: pgmap v1333: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:44:16.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:44:16 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3405806443' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:44:17.243 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1851 8 365281 365273 1.54158 0 - 0.0299513 2026-02-23T13:44:18.244 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1852 8 365281 365273 1.54075 0 - 0.0299513 2026-02-23T13:44:18.576 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:44:18.694 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:44:18.793 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:44:18.791+0000 7f5b65c8a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:44:18.793 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:44:18.791+0000 7f5b65c8a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:44:18.805 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:44:18.805 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:44:18.815 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:44:18.816 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:44:18.816 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:44:18.826 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:44:18.826 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:44:18.841 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:44:18 trial068 ceph-mon[38805]: pgmap v1334: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:44:18.855 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:44:18.856 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:44:19.244 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1853 8 365281 365273 1.53992 0 - 0.0299513 2026-02-23T13:44:19.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:44:19 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1305174531' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:44:20.244 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1854 8 365281 365273 1.53909 0 - 0.0299513 2026-02-23T13:44:20.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:44:20 trial068 ceph-mon[38805]: pgmap v1335: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:44:20.857 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:44:20.973 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:44:21.078 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:44:21.077+0000 7f616c75a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:44:21.079 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:44:21.077+0000 7f616c75a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:44:21.090 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:44:21.091 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:44:21.100 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:44:21.101 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:44:21.101 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:44:21.110 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:44:21.111 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:44:21.139 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:44:21.139 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:44:21.244 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1855 8 365281 365273 1.53826 0 - 0.0299513 2026-02-23T13:44:21.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:44:21 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2221809525' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:44:22.244 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1856 8 365281 365273 1.53743 0 - 0.0299513 2026-02-23T13:44:23.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:44:22 trial068 ceph-mon[38805]: pgmap v1336: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:44:23.140 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:44:23.244 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1857 8 365281 365273 1.5366 0 - 0.0299513 2026-02-23T13:44:23.260 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:44:23.365 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:44:23.363+0000 7fe10205e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:44:23.365 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:44:23.364+0000 7fe10205e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:44:23.377 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:44:23.378 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:44:23.387 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:44:23.388 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:44:23.388 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:44:23.397 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:44:23.398 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:44:23.427 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:44:23.427 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:44:23.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:44:23 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1052583002' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:44:24.244 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1858 8 365281 365273 1.53578 0 - 0.0299513 2026-02-23T13:44:24.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:44:23 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:44:23] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:44:24.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:44:23 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:44:23.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:44:25.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:44:24 trial068 ceph-mon[38805]: pgmap v1337: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:44:25.244 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1859 8 365281 365273 1.53495 0 - 0.0299513 2026-02-23T13:44:25.428 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:44:25.545 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:44:25.645 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:44:25.643+0000 7f71e8648640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:44:25.645 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:44:25.644+0000 7f71e8648640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:44:25.658 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:44:25.658 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:44:25.669 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:44:25.669 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:44:25.670 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:44:25.680 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:44:25.680 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:44:25.710 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:44:25.711 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:44:26.244 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T13:44:26.243974+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T13:44:26.245 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T13:44:26.245 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1860 8 365281 365273 1.53412 0 - 0.0299513 2026-02-23T13:44:27.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:44:26 trial068 ceph-mon[38805]: pgmap v1338: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:44:27.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:44:26 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1101467516' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:44:27.244 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1861 8 365281 365273 1.5333 0 - 0.0299513 2026-02-23T13:44:27.712 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:44:27.827 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:44:27.922 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:44:27.920+0000 7fd477511640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:44:27.923 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:44:27.921+0000 7fd477511640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:44:27.935 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:44:27.935 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:44:27.945 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:44:27.945 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:44:27.945 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:44:27.955 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:44:27.955 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:44:27.985 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:44:27.986 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:44:28.244 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1862 8 365281 365273 1.53248 0 - 0.0299513 2026-02-23T13:44:29.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:44:28 trial068 ceph-mon[38805]: pgmap v1339: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:44:29.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:44:28 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1313793386' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:44:29.244 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1863 8 365281 365273 1.53165 0 - 0.0299513 2026-02-23T13:44:29.987 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:44:30.106 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:44:30.211 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:44:30.209+0000 7fdbd827d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:44:30.211 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:44:30.209+0000 7fdbd827d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:44:30.223 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:44:30.223 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:44:30.233 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:44:30.233 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:44:30.233 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:44:30.243 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:44:30.243 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:44:30.244 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1864 8 365281 365273 1.53083 0 - 0.0299513 2026-02-23T13:44:30.274 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:44:30.274 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:44:31.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:44:30 trial068 ceph-mon[38805]: pgmap v1340: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:44:31.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:44:30 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/806818130' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:44:31.245 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1865 8 365281 365273 1.53001 0 - 0.0299513 2026-02-23T13:44:32.245 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1866 8 365281 365273 1.52919 0 - 0.0299513 2026-02-23T13:44:32.275 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:44:32.393 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:44:32.497 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:44:32.495+0000 7f69f5140640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:44:32.497 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:44:32.496+0000 7f69f5140640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:44:32.509 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:44:32.510 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:44:32.520 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:44:32.520 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:44:32.520 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:44:32.530 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:44:32.530 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:44:32.560 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:44:32.560 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:44:32.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:44:32 trial068 ceph-mon[38805]: pgmap v1341: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:44:32.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:44:32 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3777535214' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:44:33.245 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1867 8 365281 365273 1.52837 0 - 0.0299513 2026-02-23T13:44:33.973 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:44:33 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T13:44:33.974 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:44:33 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T13:44:34.245 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1868 8 365281 365273 1.52755 0 - 0.0299513 2026-02-23T13:44:34.265 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:44:33 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:44:33.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:44:34.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:44:33 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:44:33] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:44:34.561 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:44:34.681 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:44:34.780 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:44:34.778+0000 7fab0c7f9640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:44:34.780 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:44:34.778+0000 7fab0c7f9640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:44:34.792 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:44:34.792 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:44:34.802 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:44:34.802 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:44:34.802 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:44:34.812 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:44:34.812 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:44:34.842 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:44:34 trial068 ceph-mon[38805]: pgmap v1342: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:44:34.843 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:44:34.844 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:44:35.245 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1869 8 365281 365273 1.52674 0 - 0.0299513 2026-02-23T13:44:36.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:44:35 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1788405687' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:44:36.245 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1870 8 365281 365273 1.52592 0 - 0.0299513 2026-02-23T13:44:36.845 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:44:36.871 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:44:36 trial068 ceph-mon[38805]: pgmap v1343: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:44:36.871 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:44:36 trial068 ceph-mon[38805]: 5.a scrub starts 2026-02-23T13:44:36.965 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:44:37.063 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:44:37.061+0000 7ff069fe5640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:44:37.063 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:44:37.062+0000 7ff069fe5640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:44:37.075 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:44:37.075 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:44:37.085 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:44:37.086 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:44:37.086 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:44:37.097 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:44:37.097 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:44:37.128 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:44:37.128 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:44:37.245 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1871 8 365281 365273 1.5251 0 - 0.0299513 2026-02-23T13:44:38.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:44:37 trial068 ceph-mon[38805]: 5.a scrub ok 2026-02-23T13:44:38.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:44:37 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3119053823' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:44:38.245 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1872 8 365281 365273 1.52429 0 - 0.0299513 2026-02-23T13:44:39.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:44:38 trial068 ceph-mon[38805]: pgmap v1344: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:44:39.129 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:44:39.245 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1873 8 365281 365273 1.52348 0 - 0.0299513 2026-02-23T13:44:39.245 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:44:39.347 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:44:39.345+0000 7f4dfe028640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:44:39.347 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:44:39.345+0000 7f4dfe028640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:44:39.359 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:44:39.359 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:44:39.370 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:44:39.370 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:44:39.370 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:44:39.380 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:44:39.380 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:44:39.411 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:44:39.411 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:44:40.245 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1874 8 365281 365273 1.52266 0 - 0.0299513 2026-02-23T13:44:40.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:44:40 trial068 ceph-mon[38805]: pgmap v1345: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:44:40.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:44:40 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2277174782' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:44:41.245 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1875 8 365281 365273 1.52185 0 - 0.0299513 2026-02-23T13:44:41.412 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:44:41.529 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:44:41.625 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:44:41.623+0000 7f5a21064640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:44:41.626 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:44:41.624+0000 7f5a21064640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:44:41.637 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:44:41.637 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:44:41.647 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:44:41.647 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:44:41.647 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:44:41.657 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:44:41.658 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:44:41.687 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:44:41.688 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:44:42.245 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1876 8 365281 365273 1.52104 0 - 0.0299513 2026-02-23T13:44:42.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:44:42 trial068 ceph-mon[38805]: pgmap v1346: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:44:42.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:44:42 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1240701210' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:44:43.245 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1877 8 365281 365273 1.52023 0 - 0.0299513 2026-02-23T13:44:43.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:44:43 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2026-02-23T13:44:43.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:44:43 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-23T13:44:43.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:44:43 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2026-02-23T13:44:43.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:44:43 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T13:44:43.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:44:43 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2026-02-23T13:44:43.689 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:44:43.807 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:44:43.911 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:44:43.909+0000 7ffbb96e4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:44:43.911 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:44:43.909+0000 7ffbb96e4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:44:43.923 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:44:43.923 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:44:43.932 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:44:43.933 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:44:43.933 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:44:43.945 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:44:43.945 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:44:43.975 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:44:43.976 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:44:44.246 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1878 8 365281 365273 1.51942 0 - 0.0299513 2026-02-23T13:44:44.269 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:44:43 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:44:43] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:44:44.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:44:43 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:44:43.972Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:44:44.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:44:44 trial068 ceph-mon[38805]: pgmap v1347: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:44:44.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:44:44 trial068 ceph-mon[38805]: pgmap v1348: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:44:44.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:44:44 trial068 ceph-mon[38805]: pgmap v1349: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:44:44.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:44:44 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2945822404' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:44:45.245 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1879 8 365281 365273 1.51861 0 - 0.0299513 2026-02-23T13:44:45.977 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:44:46.093 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:44:46.188 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:44:46.186+0000 7f42d1d6b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:44:46.188 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:44:46.186+0000 7f42d1d6b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:44:46.199 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:44:46.199 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:44:46.209 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:44:46.209 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:44:46.209 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:44:46.220 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:44:46.220 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:44:46.245 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T13:44:46.245509+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T13:44:46.245 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T13:44:46.245 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1880 8 365281 365273 1.5178 0 - 0.0299513 2026-02-23T13:44:46.251 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:44:46.252 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:44:46.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:44:46 trial068 ceph-mon[38805]: pgmap v1350: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:44:46.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:44:46 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3849556378' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:44:47.245 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1881 8 365281 365273 1.517 0 - 0.0299513 2026-02-23T13:44:48.246 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1882 8 365281 365273 1.51619 0 - 0.0299513 2026-02-23T13:44:48.252 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:44:48.371 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:44:48.470 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:44:48.468+0000 7f38a0950640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:44:48.470 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:44:48.469+0000 7f38a0950640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:44:48.482 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:44:48.482 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:44:48.492 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:44:48.492 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:44:48.492 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:44:48.502 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:44:48.502 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:44:48.531 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:44:48 trial068 ceph-mon[38805]: pgmap v1351: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:44:48.532 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:44:48.533 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:44:49.246 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1883 8 365281 365273 1.51539 0 - 0.0299513 2026-02-23T13:44:49.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:44:49 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2027278375' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:44:50.246 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1884 8 365281 365273 1.51458 0 - 0.0299513 2026-02-23T13:44:50.533 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:44:50.558 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:44:50 trial068 ceph-mon[38805]: pgmap v1352: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:44:50.653 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:44:50.750 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:44:50.748+0000 7f647b477640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:44:50.750 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:44:50.748+0000 7f647b477640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:44:50.762 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:44:50.762 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:44:50.772 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:44:50.772 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:44:50.772 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:44:50.783 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:44:50.783 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:44:50.816 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:44:50.817 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:44:51.246 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1885 8 365281 365273 1.51378 0 - 0.0299513 2026-02-23T13:44:51.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:44:51 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/585548960' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:44:52.246 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1886 8 365281 365273 1.51298 0 - 0.0299513 2026-02-23T13:44:52.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:44:52 trial068 ceph-mon[38805]: pgmap v1353: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:44:52.818 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:44:52.936 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:44:53.035 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:44:53.034+0000 7f19c825a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:44:53.036 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:44:53.034+0000 7f19c825a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:44:53.048 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:44:53.048 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:44:53.058 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:44:53.059 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:44:53.059 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:44:53.068 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:44:53.069 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:44:53.099 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:44:53.100 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:44:53.246 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1887 8 365281 365273 1.51217 0 - 0.0299513 2026-02-23T13:44:53.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:44:53 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3557768487' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:44:54.246 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1888 8 365281 365273 1.51137 0 - 0.0299513 2026-02-23T13:44:54.281 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:44:53 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:44:53] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:44:54.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:44:53 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:44:53.972Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:44:54.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:44:54 trial068 ceph-mon[38805]: pgmap v1354: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:44:55.101 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:44:55.221 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:44:55.246 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1889 8 365281 365273 1.51057 0 - 0.0299513 2026-02-23T13:44:55.320 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:44:55.318+0000 7f98b6809640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:44:55.320 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:44:55.318+0000 7f98b6809640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:44:55.332 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:44:55.332 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:44:55.342 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:44:55.343 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:44:55.343 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:44:55.353 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:44:55.353 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:44:55.380 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:44:55 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T13:44:55.380 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:44:55 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T13:44:55.381 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:44:55.381 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:44:56.246 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1890 8 365281 365273 1.50977 0 - 0.0299513 2026-02-23T13:44:56.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:44:56 trial068 ceph-mon[38805]: pgmap v1355: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:44:56.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:44:56 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1067802678' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:44:57.246 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1891 8 365281 365273 1.50897 0 - 0.0299513 2026-02-23T13:44:57.383 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:44:57.500 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:44:57.592 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:44:57.591+0000 7ff50964e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:44:57.593 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:44:57.591+0000 7ff50964e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:44:57.605 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:44:57.605 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:44:57.616 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:44:57.616 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:44:57.616 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:44:57.627 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:44:57.627 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:44:57.657 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:44:57.658 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:44:58.246 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1892 8 365281 365273 1.50818 0 - 0.0299513 2026-02-23T13:44:58.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:44:58 trial068 ceph-mon[38805]: pgmap v1356: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:44:58.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:44:58 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2238521873' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:44:59.247 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1893 8 365281 365273 1.50738 0 - 0.0299513 2026-02-23T13:44:59.659 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:44:59.777 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:44:59.880 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:44:59.878+0000 7f97f442d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:44:59.880 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:44:59.878+0000 7f97f442d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:44:59.894 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:44:59.894 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:44:59.905 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:44:59.905 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:44:59.905 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:44:59.915 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:44:59.916 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:44:59.945 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:44:59.946 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:45:00.247 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1894 8 365281 365273 1.50658 0 - 0.0299513 2026-02-23T13:45:00.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:45:00 trial068 ceph-mon[38805]: pgmap v1357: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:45:00.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:45:00 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2370684872' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:45:01.247 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1895 8 365281 365273 1.50579 0 - 0.0299513 2026-02-23T13:45:01.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:45:01 trial068 ceph-mon[38805]: 5.5 scrub starts 2026-02-23T13:45:01.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:45:01 trial068 ceph-mon[38805]: 5.5 scrub ok 2026-02-23T13:45:01.947 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:45:02.064 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:45:02.163 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:45:02.161+0000 7fa3c8a3f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:45:02.163 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:45:02.161+0000 7fa3c8a3f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:45:02.175 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:45:02.175 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:45:02.185 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:45:02.186 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:45:02.186 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:45:02.196 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:45:02.196 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:45:02.224 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:45:02.225 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:45:02.247 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1896 8 365281 365273 1.505 0 - 0.0299513 2026-02-23T13:45:02.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:45:02 trial068 ceph-mon[38805]: pgmap v1358: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:45:02.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:45:02 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3049075889' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:45:03.247 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1897 8 365281 365273 1.5042 0 - 0.0299513 2026-02-23T13:45:04.226 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:45:04.247 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1898 8 365281 365273 1.50341 0 - 0.0299513 2026-02-23T13:45:04.250 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:45:03 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:45:03] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:45:04.266 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:45:03 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:45:03.974Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:45:04.345 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:45:04.444 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:45:04.442+0000 7fa69b0e9640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:45:04.445 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:45:04.443+0000 7fa69b0e9640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:45:04.456 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:45:04.457 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:45:04.467 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:45:04.467 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:45:04.467 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:45:04.478 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:45:04.478 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:45:04.505 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:45:04 trial068 ceph-mon[38805]: pgmap v1359: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:45:04.506 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:45:04.507 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:45:05.247 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1899 8 365281 365273 1.50262 0 - 0.0299513 2026-02-23T13:45:05.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:45:05 trial068 ceph-mon[38805]: osdmap e206: 4 total, 3 up, 4 in 2026-02-23T13:45:05.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:45:05 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1061857262' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:45:06.247 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T13:45:06.246861+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T13:45:06.247 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T13:45:06.247 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1900 8 365281 365273 1.50183 0 - 0.0299513 2026-02-23T13:45:06.508 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:45:06.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:45:06 trial068 ceph-mon[38805]: pgmap v1361: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:45:06.627 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:45:06.728 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:45:06.726+0000 7fec93b8a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:45:06.728 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:45:06.727+0000 7fec93b8a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:45:06.740 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:45:06.740 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:45:06.750 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:45:06.750 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:45:06.750 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:45:06.760 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:45:06.760 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:45:06.791 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:45:06.792 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:45:07.247 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1901 8 365281 365273 1.50104 0 - 0.0299513 2026-02-23T13:45:07.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:45:07 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3665024750' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:45:07.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:45:07 trial068 ceph-mon[38805]: 5.e scrub starts 2026-02-23T13:45:07.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:45:07 trial068 ceph-mon[38805]: 5.e scrub ok 2026-02-23T13:45:08.247 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1902 8 365281 365273 1.50025 0 - 0.0299513 2026-02-23T13:45:08.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:45:08 trial068 ceph-mon[38805]: pgmap v1362: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:45:08.793 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:45:08.913 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:45:09.014 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:45:09.012+0000 7fa78b81d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:45:09.014 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:45:09.012+0000 7fa78b81d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:45:09.026 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:45:09.026 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:45:09.036 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:45:09.036 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:45:09.036 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:45:09.047 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:45:09.047 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:45:09.077 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:45:09.077 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:45:09.247 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1903 8 365281 365273 1.49946 0 - 0.0299513 2026-02-23T13:45:09.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:45:09 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1593859496' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:45:10.247 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1904 8 365281 365273 1.49867 0 - 0.0299513 2026-02-23T13:45:10.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:45:10 trial068 ceph-mon[38805]: pgmap v1363: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:45:11.079 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:45:11.195 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:45:11.247 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1905 8 365281 365273 1.49789 0 - 0.0299513 2026-02-23T13:45:11.294 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:45:11.292+0000 7f26d2f66640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:45:11.294 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:45:11.292+0000 7f26d2f66640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:45:11.306 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:45:11.306 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:45:11.316 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:45:11.317 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:45:11.317 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:45:11.326 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:45:11.326 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:45:11.355 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:45:11 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3693760053' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:45:11.355 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:45:11.356 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:45:12.248 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1906 8 365281 365273 1.4971 0 - 0.0299513 2026-02-23T13:45:12.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:45:12 trial068 ceph-mon[38805]: pgmap v1364: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:45:13.247 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1907 8 365281 365273 1.49631 0 - 0.0299513 2026-02-23T13:45:13.357 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:45:13.475 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:45:13.574 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:45:13.572+0000 7fdcad0af640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:45:13.574 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:45:13.572+0000 7fdcad0af640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:45:13.586 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:45:13.586 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:45:13.596 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:45:13.596 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:45:13.597 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:45:13.607 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:45:13.607 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:45:13.637 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:45:13.638 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:45:14.248 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1908 8 365281 365273 1.49553 0 - 0.0299513 2026-02-23T13:45:14.303 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:45:13 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:45:13] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:45:14.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:45:13 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:45:13.974Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:45:14.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:45:14 trial068 ceph-mon[38805]: pgmap v1365: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:45:14.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:45:14 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1190430315' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:45:15.248 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1909 8 365281 365273 1.49475 0 - 0.0299513 2026-02-23T13:45:15.638 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:45:15.758 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:45:15.853 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:45:15.851+0000 7f8eb713b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:45:15.853 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:45:15.851+0000 7f8eb713b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:45:15.864 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:45:15.865 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:45:15.874 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:45:15.875 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:45:15.875 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:45:15.885 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:45:15.885 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:45:15.914 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:45:15.915 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:45:16.248 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1910 8 365281 365273 1.49396 0 - 0.0299513 2026-02-23T13:45:16.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:45:16 trial068 ceph-mon[38805]: pgmap v1366: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:45:16.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:45:16 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1845998967' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:45:17.248 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1911 8 365281 365273 1.49318 0 - 0.0299513 2026-02-23T13:45:17.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:45:17 trial068 ceph-mon[38805]: 5.7 scrub starts 2026-02-23T13:45:17.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:45:17 trial068 ceph-mon[38805]: 5.7 scrub ok 2026-02-23T13:45:17.916 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:45:18.034 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:45:18.138 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:45:18.136+0000 7f984b102640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:45:18.138 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:45:18.136+0000 7f984b102640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:45:18.150 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:45:18.151 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:45:18.160 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:45:18.161 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:45:18.161 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:45:18.172 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:45:18.172 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:45:18.204 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:45:18.205 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:45:18.248 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1912 8 365281 365273 1.4924 0 - 0.0299513 2026-02-23T13:45:18.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:45:18 trial068 ceph-mon[38805]: pgmap v1367: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:45:18.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:45:18 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3695490405' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:45:19.248 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1913 8 365281 365273 1.49162 0 - 0.0299513 2026-02-23T13:45:20.205 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:45:20.248 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1914 8 365281 365273 1.49084 0 - 0.0299513 2026-02-23T13:45:20.322 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:45:20.421 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:45:20.419+0000 7fc74fb4d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:45:20.421 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:45:20.419+0000 7fc74fb4d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:45:20.432 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:45:20.432 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:45:20.442 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:45:20.442 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:45:20.443 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:45:20.453 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:45:20.453 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:45:20.483 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:45:20 trial068 ceph-mon[38805]: pgmap v1368: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:45:20.484 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:45:20.485 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:45:21.248 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1915 8 365281 365273 1.49006 0 - 0.0299513 2026-02-23T13:45:21.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:45:21 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2924799900' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:45:21.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:45:21 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T13:45:21.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:45:21 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T13:45:22.248 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1916 8 365281 365273 1.48929 0 - 0.0299513 2026-02-23T13:45:22.486 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:45:22.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:45:22 trial068 ceph-mon[38805]: pgmap v1369: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:45:22.604 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:45:22.707 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:45:22.705+0000 7fa23ab29640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:45:22.707 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:45:22.705+0000 7fa23ab29640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:45:22.719 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:45:22.719 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:45:22.729 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:45:22.729 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:45:22.729 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:45:22.739 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:45:22.739 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:45:22.769 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:45:22.769 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:45:23.248 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1917 8 365281 365273 1.48851 0 - 0.0299513 2026-02-23T13:45:23.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:45:23 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/293987629' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:45:24.248 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1918 8 365281 365273 1.48773 0 - 0.0299513 2026-02-23T13:45:24.316 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:45:23 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:45:23] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:45:24.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:45:23 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:45:23.974Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:45:24.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:45:24 trial068 ceph-mon[38805]: pgmap v1370: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:45:24.770 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:45:24.889 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:45:24.990 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:45:24.988+0000 7fb61bffe640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:45:24.990 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:45:24.988+0000 7fb61bffe640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:45:25.002 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:45:25.003 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:45:25.012 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:45:25.013 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:45:25.013 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:45:25.023 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:45:25.024 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:45:25.072 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:45:25.073 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:45:25.248 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1919 8 365281 365273 1.48696 0 - 0.0299513 2026-02-23T13:45:25.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:45:25 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3288372493' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:45:26.248 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T13:45:26.248264+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T13:45:26.249 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T13:45:26.249 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1920 8 365281 365273 1.48618 0 - 0.0299513 2026-02-23T13:45:26.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:45:26 trial068 ceph-mon[38805]: pgmap v1371: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:45:27.074 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:45:27.191 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:45:27.248 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1921 8 365281 365273 1.48541 0 - 0.0299513 2026-02-23T13:45:27.286 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:45:27.284+0000 7f38cfcbb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:45:27.286 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:45:27.284+0000 7f38cfcbb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:45:27.298 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:45:27.298 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:45:27.308 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:45:27.309 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:45:27.309 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:45:27.319 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:45:27.319 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:45:27.347 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:45:27.347 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:45:27.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:45:27 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2078589903' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:45:28.249 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1922 8 365281 365273 1.48464 0 - 0.0299513 2026-02-23T13:45:28.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:45:28 trial068 ceph-mon[38805]: pgmap v1372: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:45:29.249 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1923 8 365281 365273 1.48386 0 - 0.0299513 2026-02-23T13:45:29.348 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:45:29.465 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:45:29.571 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:45:29.569+0000 7fe75fc89640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:45:29.571 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:45:29.569+0000 7fe75fc89640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:45:29.583 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:45:29.584 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:45:29.594 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:45:29.594 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:45:29.594 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:45:29.604 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:45:29.604 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:45:29.634 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:45:29.635 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:45:30.249 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1924 8 365281 365273 1.48309 0 - 0.0299513 2026-02-23T13:45:30.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:45:30 trial068 ceph-mon[38805]: pgmap v1373: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:45:30.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:45:30 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1375111779' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:45:31.249 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1925 8 365281 365273 1.48232 0 - 0.0299513 2026-02-23T13:45:31.636 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:45:31.754 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:45:31.856 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:45:31.854+0000 7fa4f6710640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:45:31.857 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:45:31.855+0000 7fa4f6710640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:45:31.868 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:45:31.868 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:45:31.878 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:45:31.879 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:45:31.879 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:45:31.888 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:45:31.889 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:45:31.920 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:45:31.921 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:45:32.249 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1926 8 365281 365273 1.48155 0 - 0.0299513 2026-02-23T13:45:32.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:45:32 trial068 ceph-mon[38805]: pgmap v1374: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:45:32.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:45:32 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3690169869' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:45:33.249 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1927 8 365281 365273 1.48078 0 - 0.0299513 2026-02-23T13:45:33.922 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:45:34.040 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:45:34.143 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:45:34.141+0000 7ff86cf04640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:45:34.143 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:45:34.142+0000 7ff86cf04640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:45:34.155 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:45:34.155 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:45:34.165 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:45:34.165 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:45:34.165 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:45:34.176 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:45:34.176 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:45:34.204 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:45:33 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:45:33] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:45:34.205 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:45:34.205 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:45:34.249 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1928 8 365281 365273 1.48002 0 - 0.0299513 2026-02-23T13:45:34.266 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:45:33 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:45:33.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:45:34.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:45:34 trial068 ceph-mon[38805]: pgmap v1375: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:45:34.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:45:34 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/177967121' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:45:35.249 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1929 8 365281 365273 1.47925 0 - 0.0299513 2026-02-23T13:45:36.206 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:45:36.249 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1930 8 365281 365273 1.47848 0 - 0.0299513 2026-02-23T13:45:36.322 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:45:36.423 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:45:36.421+0000 7f97bb5ea640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:45:36.423 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:45:36.421+0000 7f97bb5ea640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:45:36.435 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:45:36.435 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:45:36.445 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:45:36.445 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:45:36.445 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:45:36.455 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:45:36.455 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:45:36.487 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:45:36 trial068 ceph-mon[38805]: pgmap v1376: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:45:36.488 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:45:36.489 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:45:37.249 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1931 8 365281 365273 1.47772 0 - 0.0299513 2026-02-23T13:45:37.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:45:37 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2529995196' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:45:38.249 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1932 8 365281 365273 1.47695 0 - 0.0299513 2026-02-23T13:45:38.490 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:45:38.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:45:38 trial068 ceph-mon[38805]: pgmap v1377: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:45:38.606 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:45:38.701 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:45:38.699+0000 7fddb408a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:45:38.702 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:45:38.700+0000 7fddb408a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:45:38.713 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:45:38.713 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:45:38.723 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:45:38.724 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:45:38.724 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:45:38.734 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:45:38.734 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:45:38.764 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:45:38.765 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:45:39.249 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1933 8 365281 365273 1.47619 0 - 0.0299513 2026-02-23T13:45:39.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:45:39 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1109443528' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:45:40.249 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1934 8 365281 365273 1.47542 0 - 0.0299513 2026-02-23T13:45:40.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:45:40 trial068 ceph-mon[38805]: 5.8 scrub starts 2026-02-23T13:45:40.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:45:40 trial068 ceph-mon[38805]: 5.8 scrub ok 2026-02-23T13:45:40.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:45:40 trial068 ceph-mon[38805]: pgmap v1378: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:45:40.766 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:45:40.882 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:45:40.980 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:45:40.977+0000 7f0d28be0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:45:40.980 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:45:40.978+0000 7f0d28be0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:45:40.991 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:45:40.991 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:45:41.001 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:45:41.002 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:45:41.002 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:45:41.012 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:45:41.012 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:45:41.040 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:45:41.041 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:45:41.250 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1935 8 365281 365273 1.47466 0 - 0.0299513 2026-02-23T13:45:41.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:45:41 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T13:45:41.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:45:41 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T13:45:41.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:45:41 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2413445984' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:45:42.250 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1936 8 365281 365273 1.4739 0 - 0.0299513 2026-02-23T13:45:42.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:45:42 trial068 ceph-mon[38805]: pgmap v1379: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:45:43.042 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:45:43.158 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:45:43.250 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1937 8 365281 365273 1.47314 0 - 0.0299513 2026-02-23T13:45:43.256 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:45:43.254+0000 7fc1787f3640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:45:43.256 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:45:43.254+0000 7fc1787f3640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:45:43.268 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:45:43.268 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:45:43.279 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:45:43.279 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:45:43.280 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:45:43.291 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:45:43.291 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:45:43.320 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:45:43.320 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:45:43.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:45:43 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2026-02-23T13:45:43.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:45:43 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3389001707' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:45:44.250 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1938 8 365281 365273 1.47238 0 - 0.0299513 2026-02-23T13:45:44.342 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:45:43 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:45:43] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:45:44.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:45:43 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:45:43.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:45:44.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:45:44 trial068 ceph-mon[38805]: pgmap v1380: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:45:44.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:45:44 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-23T13:45:44.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:45:44 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2026-02-23T13:45:44.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:45:44 trial068 ceph-mon[38805]: pgmap v1381: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:45:44.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:45:44 trial068 ceph-mon[38805]: pgmap v1382: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:45:44.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:45:44 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T13:45:44.595 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:45:44 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2026-02-23T13:45:45.250 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1939 8 365281 365273 1.47162 0 - 0.0299513 2026-02-23T13:45:45.322 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:45:45.439 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:45:45.544 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:45:45.542+0000 7f0848e82640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:45:45.544 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:45:45.542+0000 7f0848e82640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:45:45.556 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:45:45.557 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:45:45.567 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:45:45.567 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:45:45.567 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:45:45.578 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:45:45.578 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:45:45.610 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:45:45.611 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:45:46.250 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T13:45:46.249668+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T13:45:46.250 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T13:45:46.250 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1940 8 365281 365273 1.47086 0 - 0.0299513 2026-02-23T13:45:46.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:45:46 trial068 ceph-mon[38805]: pgmap v1383: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:45:46.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:45:46 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/262644156' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:45:47.250 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1941 8 365281 365273 1.4701 0 - 0.0299513 2026-02-23T13:45:47.612 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:45:47.729 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:45:47.827 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:45:47.825+0000 7f0a3df18640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:45:47.827 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:45:47.825+0000 7f0a3df18640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:45:47.838 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:45:47.839 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:45:47.849 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:45:47.849 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:45:47.849 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:45:47.859 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:45:47.859 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:45:47.889 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:45:47.890 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:45:48.250 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1942 8 365281 365273 1.46935 0 - 0.0299513 2026-02-23T13:45:48.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:45:48 trial068 ceph-mon[38805]: pgmap v1384: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:45:48.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:45:48 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2362775626' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:45:49.250 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1943 8 365281 365273 1.46859 0 - 0.0299513 2026-02-23T13:45:49.891 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:45:50.009 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:45:50.104 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:45:50.102+0000 7f2848f6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:45:50.104 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:45:50.103+0000 7f2848f6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:45:50.116 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:45:50.116 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:45:50.126 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:45:50.127 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:45:50.127 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:45:50.137 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:45:50.137 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:45:50.167 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:45:50.168 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:45:50.250 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1944 8 365281 365273 1.46784 0 - 0.0299513 2026-02-23T13:45:50.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:45:50 trial068 ceph-mon[38805]: pgmap v1385: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:45:50.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:45:50 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/553372015' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:45:51.250 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1945 8 365281 365273 1.46708 0 - 0.0299513 2026-02-23T13:45:52.169 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:45:52.250 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1946 8 365281 365273 1.46633 0 - 0.0299513 2026-02-23T13:45:52.283 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:45:52.381 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:45:52.379+0000 7f779a058640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:45:52.381 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:45:52.380+0000 7f779a058640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:45:52.394 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:45:52.394 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:45:52.404 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:45:52.405 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:45:52.405 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:45:52.415 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:45:52.415 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:45:52.445 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:45:52.445 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:45:52.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:45:52 trial068 ceph-mon[38805]: pgmap v1386: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:45:52.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:45:52 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3343268049' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:45:53.250 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1947 8 365281 365273 1.46557 0 - 0.0299513 2026-02-23T13:45:53.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:45:53 trial068 ceph-mon[38805]: 5.a scrub starts 2026-02-23T13:45:53.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:45:53 trial068 ceph-mon[38805]: 5.a scrub ok 2026-02-23T13:45:54.250 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1948 8 365281 365273 1.46482 0 - 0.0299513 2026-02-23T13:45:54.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:45:53 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:45:53] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:45:54.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:45:53 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:45:53.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:45:54.446 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:45:54.566 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:45:54.609 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:45:54 trial068 ceph-mon[38805]: pgmap v1387: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:45:54.666 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:45:54.664+0000 7f8e0c85d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:45:54.666 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:45:54.664+0000 7f8e0c85d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:45:54.678 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:45:54.678 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:45:54.688 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:45:54.688 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:45:54.688 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:45:54.698 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:45:54.698 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:45:54.728 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:45:54.729 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:45:55.250 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1949 8 365281 365273 1.46407 0 - 0.0299513 2026-02-23T13:45:55.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:45:55 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2907232013' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:45:56.250 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1950 8 365281 365273 1.46332 0 - 0.0299513 2026-02-23T13:45:56.729 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:45:56.755 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:45:56 trial068 ceph-mon[38805]: pgmap v1388: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:45:56.847 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:45:56.943 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:45:56.941+0000 7f02f7414640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:45:56.943 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:45:56.941+0000 7f02f7414640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:45:56.954 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:45:56.955 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:45:56.965 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:45:56.965 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:45:56.965 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:45:56.976 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:45:56.976 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:45:57.006 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:45:57.007 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:45:57.250 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1951 8 365281 365273 1.46257 0 - 0.0299513 2026-02-23T13:45:57.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:45:57 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/721112569' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:45:58.251 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1952 8 365281 365273 1.46182 0 - 0.0299513 2026-02-23T13:45:58.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:45:58 trial068 ceph-mon[38805]: pgmap v1389: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:45:59.008 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:45:59.124 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:45:59.223 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:45:59.221+0000 7fc803405640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:45:59.223 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:45:59.221+0000 7fc803405640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:45:59.234 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:45:59.234 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:45:59.244 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:45:59.245 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:45:59.245 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:45:59.250 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1953 8 365281 365273 1.46107 0 - 0.0299513 2026-02-23T13:45:59.254 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:45:59.254 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:45:59.285 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:45:59.285 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:45:59.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:45:59 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2293670960' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:46:00.251 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1954 8 365281 365273 1.46032 0 - 0.0299513 2026-02-23T13:46:00.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:46:00 trial068 ceph-mon[38805]: pgmap v1390: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:46:01.251 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1955 8 365281 365273 1.45958 0 - 0.0299513 2026-02-23T13:46:01.286 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:46:01.402 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:46:01.498 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:46:01.496+0000 7f389542d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:46:01.498 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:46:01.497+0000 7f389542d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:46:01.511 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:46:01.511 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:46:01.521 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:46:01.522 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:46:01.522 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:46:01.532 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:46:01.532 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:46:01.562 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:46:01.562 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:46:02.251 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1956 8 365281 365273 1.45883 0 - 0.0299513 2026-02-23T13:46:02.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:46:02 trial068 ceph-mon[38805]: pgmap v1391: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:46:02.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:46:02 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1416060499' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:46:03.251 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1957 8 365281 365273 1.45808 0 - 0.0299513 2026-02-23T13:46:03.564 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:46:03.683 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:46:03.783 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:46:03.781+0000 7ff17f030640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:46:03.784 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:46:03.782+0000 7ff17f030640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:46:03.795 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:46:03.796 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:46:03.806 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:46:03.806 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:46:03.807 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:46:03.817 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:46:03.817 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:46:03.845 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:46:03.845 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:46:04.251 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1958 8 365281 365273 1.45734 0 - 0.0299513 2026-02-23T13:46:04.266 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:46:03 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:46:03.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:46:04.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:46:03 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:46:03] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:46:04.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:46:04 trial068 ceph-mon[38805]: pgmap v1392: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:46:04.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:46:04 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3770034465' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:46:05.251 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1959 8 365281 365273 1.4566 0 - 0.0299513 2026-02-23T13:46:05.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:46:05 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T13:46:05.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:46:05 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T13:46:05.846 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:46:05.965 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:46:06.067 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:46:06.065+0000 7f08bee57640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:46:06.067 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:46:06.066+0000 7f08bee57640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:46:06.080 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:46:06.080 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:46:06.090 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:46:06.090 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:46:06.091 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:46:06.100 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:46:06.101 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:46:06.131 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:46:06.131 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:46:06.251 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T13:46:06.251106+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T13:46:06.251 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T13:46:06.252 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1960 8 365281 365273 1.45585 0 - 0.0299513 2026-02-23T13:46:06.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:46:06 trial068 ceph-mon[38805]: pgmap v1393: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:46:06.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:46:06 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/627394558' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:46:07.251 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1961 8 365281 365273 1.45511 0 - 0.0299513 2026-02-23T13:46:08.133 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:46:08.250 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:46:08.251 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1962 8 365281 365273 1.45437 0 - 0.0299513 2026-02-23T13:46:08.348 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:46:08.345+0000 7fdcd2526640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:46:08.348 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:46:08.346+0000 7fdcd2526640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:46:08.361 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:46:08.361 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:46:08.371 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:46:08.371 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:46:08.371 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:46:08.381 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:46:08.382 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:46:08.411 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:46:08.411 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:46:08.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:46:08 trial068 ceph-mon[38805]: pgmap v1394: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:46:08.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:46:08 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1831702490' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:46:09.252 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1963 8 365281 365273 1.45363 0 - 0.0299513 2026-02-23T13:46:10.252 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1964 8 365281 365273 1.45289 0 - 0.0299513 2026-02-23T13:46:10.413 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:46:10.533 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:46:10.554 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:46:10 trial068 ceph-mon[38805]: pgmap v1395: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:46:10.631 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:46:10.629+0000 7f763b575640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:46:10.631 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:46:10.629+0000 7f763b575640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:46:10.642 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:46:10.643 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:46:10.652 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:46:10.653 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:46:10.653 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:46:10.663 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:46:10.663 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:46:10.692 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:46:10.692 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:46:11.252 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1965 8 365281 365273 1.45215 0 - 0.0299513 2026-02-23T13:46:11.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:46:11 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1790730607' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:46:12.252 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1966 8 365281 365273 1.45141 0 - 0.0299513 2026-02-23T13:46:12.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:46:12 trial068 ceph-mon[38805]: pgmap v1396: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:46:12.694 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:46:12.812 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:46:12.910 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:46:12.908+0000 7f1d42341640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:46:12.910 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:46:12.908+0000 7f1d42341640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:46:12.921 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:46:12.921 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:46:12.931 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:46:12.932 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:46:12.932 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:46:12.941 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:46:12.941 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:46:12.970 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:46:12.971 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:46:13.252 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1967 8 365281 365273 1.45067 0 - 0.0299513 2026-02-23T13:46:13.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:46:13 trial068 ceph-mon[38805]: osdmap e207: 4 total, 3 up, 4 in 2026-02-23T13:46:13.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:46:13 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2078871868' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:46:14.252 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1968 8 365281 365273 1.44994 0 - 0.0299513 2026-02-23T13:46:14.284 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:46:13 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:46:13] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:46:14.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:46:13 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:46:13.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:46:14.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:46:14 trial068 ceph-mon[38805]: pgmap v1398: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:46:14.972 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:46:15.091 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:46:15.191 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:46:15.189+0000 7f8ff99a8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:46:15.191 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:46:15.189+0000 7f8ff99a8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:46:15.203 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:46:15.204 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:46:15.213 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:46:15.214 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:46:15.214 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:46:15.224 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:46:15.224 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:46:15.252 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1969 8 365281 365273 1.4492 0 - 0.0299513 2026-02-23T13:46:15.258 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:46:15.259 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:46:15.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:46:15 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/570753373' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:46:16.252 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1970 8 365281 365273 1.44846 0 - 0.0299513 2026-02-23T13:46:16.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:46:16 trial068 ceph-mon[38805]: pgmap v1399: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:46:17.252 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1971 8 365281 365273 1.44773 0 - 0.0299513 2026-02-23T13:46:17.260 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:46:17.379 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:46:17.481 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:46:17.479+0000 7f1c5b1af640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:46:17.481 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:46:17.479+0000 7f1c5b1af640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:46:17.493 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:46:17.493 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:46:17.503 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:46:17.504 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:46:17.504 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:46:17.515 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:46:17.515 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:46:17.543 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:46:17 trial068 ceph-mon[38805]: 5.e scrub starts 2026-02-23T13:46:17.543 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:46:17 trial068 ceph-mon[38805]: 5.e scrub ok 2026-02-23T13:46:17.544 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:46:17.544 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:46:18.252 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1972 8 365281 365273 1.44699 0 - 0.0299513 2026-02-23T13:46:18.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:46:18 trial068 ceph-mon[38805]: pgmap v1400: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:46:18.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:46:18 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3227787907' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:46:19.252 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1973 8 365281 365273 1.44626 0 - 0.0299513 2026-02-23T13:46:19.545 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:46:19.569 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:46:19 trial068 ceph-mon[38805]: 5.5 scrub starts 2026-02-23T13:46:19.569 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:46:19 trial068 ceph-mon[38805]: 5.5 scrub ok 2026-02-23T13:46:19.661 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:46:19.763 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:46:19.761+0000 7fd12ac2e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:46:19.763 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:46:19.761+0000 7fd12ac2e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:46:19.776 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:46:19.776 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:46:19.786 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:46:19.787 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:46:19.787 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:46:19.797 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:46:19.797 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:46:19.824 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:46:19.825 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:46:20.252 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1974 8 365281 365273 1.44553 0 - 0.0299513 2026-02-23T13:46:20.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:46:20 trial068 ceph-mon[38805]: pgmap v1401: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:46:20.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:46:20 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2967765676' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:46:21.252 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1975 8 365281 365273 1.4448 0 - 0.0299513 2026-02-23T13:46:21.825 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:46:21.944 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:46:22.043 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:46:22.041+0000 7f04b0dc9640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:46:22.043 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:46:22.041+0000 7f04b0dc9640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:46:22.055 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:46:22.055 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:46:22.065 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:46:22.066 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:46:22.066 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:46:22.077 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:46:22.077 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:46:22.108 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:46:22.109 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:46:22.253 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1976 8 365281 365273 1.44406 0 - 0.0299513 2026-02-23T13:46:22.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:46:22 trial068 ceph-mon[38805]: pgmap v1402: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:46:22.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:46:22 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2602286222' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:46:23.253 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1977 8 365281 365273 1.44333 0 - 0.0299513 2026-02-23T13:46:24.109 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:46:24.228 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:46:24.250 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:46:23 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:46:23] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:46:24.252 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1978 8 365281 365273 1.4426 0 - 0.0299513 2026-02-23T13:46:24.326 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:46:24.324+0000 7fa209e72640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:46:24.327 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:46:24.325+0000 7fa209e72640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:46:24.339 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:46:24.339 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:46:24.350 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:46:24.350 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:46:24.350 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:46:24.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:46:23 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:46:23.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:46:24.360 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:46:24.361 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:46:24.391 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:46:24.392 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:46:24.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:46:24 trial068 ceph-mon[38805]: pgmap v1403: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:46:24.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:46:24 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/856651231' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:46:25.253 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1979 8 365281 365273 1.44188 0 - 0.0299513 2026-02-23T13:46:26.253 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T13:46:26.252692+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T13:46:26.253 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T13:46:26.253 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1980 8 365281 365273 1.44115 0 - 0.0299513 2026-02-23T13:46:26.393 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:46:26.511 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:46:26.607 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:46:26.605+0000 7f6f40b74640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:46:26.607 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:46:26.605+0000 7f6f40b74640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:46:26.619 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:46:26.619 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:46:26.629 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:46:26.629 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:46:26.629 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:46:26.639 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:46:26.639 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:46:26.667 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:46:26.668 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:46:26.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:46:26 trial068 ceph-mon[38805]: pgmap v1404: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:46:27.253 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1981 8 365281 365273 1.44042 0 - 0.0299513 2026-02-23T13:46:27.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:46:27 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/128941077' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:46:28.253 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1982 8 365281 365273 1.43969 0 - 0.0299513 2026-02-23T13:46:28.668 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:46:28.784 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:46:28.806 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:46:28 trial068 ceph-mon[38805]: pgmap v1405: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:46:28.877 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:46:28.875+0000 7f3318b28640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:46:28.878 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:46:28.876+0000 7f3318b28640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:46:28.889 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:46:28.890 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:46:28.900 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:46:28.900 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:46:28.900 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:46:28.910 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:46:28.911 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:46:28.938 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:46:28.939 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:46:29.253 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1983 8 365281 365273 1.43897 0 - 0.0299513 2026-02-23T13:46:29.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:46:29 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/190583771' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:46:30.253 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1984 8 365281 365273 1.43824 0 - 0.0299513 2026-02-23T13:46:30.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:46:30 trial068 ceph-mon[38805]: pgmap v1406: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:46:30.940 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:46:31.057 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:46:31.164 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:46:31.162+0000 7fd4ecf98640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:46:31.165 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:46:31.163+0000 7fd4ecf98640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:46:31.176 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:46:31.177 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:46:31.186 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:46:31.187 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:46:31.187 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:46:31.196 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:46:31.196 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:46:31.227 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:46:31.227 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:46:31.253 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1985 8 365281 365273 1.43752 0 - 0.0299513 2026-02-23T13:46:31.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:46:31 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/522350512' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:46:32.253 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1986 8 365281 365273 1.43679 0 - 0.0299513 2026-02-23T13:46:32.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:46:32 trial068 ceph-mon[38805]: pgmap v1407: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:46:33.228 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:46:33.253 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1987 8 365281 365273 1.43607 0 - 0.0299513 2026-02-23T13:46:33.346 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:46:33.450 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:46:33.448+0000 7f5efdc4f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:46:33.450 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:46:33.448+0000 7f5efdc4f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:46:33.462 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:46:33.462 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:46:33.472 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:46:33.472 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:46:33.472 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:46:33.483 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:46:33.483 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:46:33.514 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:46:33.514 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:46:33.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:46:33 trial068 ceph-mon[38805]: 5.7 scrub starts 2026-02-23T13:46:33.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:46:33 trial068 ceph-mon[38805]: 5.7 scrub ok 2026-02-23T13:46:33.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:46:33 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/87876107' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:46:34.253 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1988 8 365281 365273 1.43535 0 - 0.0299513 2026-02-23T13:46:34.266 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:46:33 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:46:33.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:46:34.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:46:33 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:46:33] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:46:34.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:46:34 trial068 ceph-mon[38805]: pgmap v1408: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:46:35.253 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1989 8 365281 365273 1.43463 0 - 0.0299513 2026-02-23T13:46:35.515 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:46:35.632 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:46:35.733 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:46:35.731+0000 7f393a16b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:46:35.733 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:46:35.731+0000 7f393a16b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:46:35.745 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:46:35.745 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:46:35.755 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:46:35.755 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:46:35.756 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:46:35.766 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:46:35.767 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:46:35.799 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:46:35.799 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:46:36.254 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1990 8 365281 365273 1.43391 0 - 0.0299513 2026-02-23T13:46:36.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:46:36 trial068 ceph-mon[38805]: pgmap v1409: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:46:36.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:46:36 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1885763090' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:46:37.254 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1991 8 365281 365273 1.43319 0 - 0.0299513 2026-02-23T13:46:37.800 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:46:37.916 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:46:38.020 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:46:38.018+0000 7f30224bf640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:46:38.020 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:46:38.018+0000 7f30224bf640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:46:38.031 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:46:38.032 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:46:38.041 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:46:38.042 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:46:38.042 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:46:38.052 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:46:38.052 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:46:38.083 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:46:38.083 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:46:38.254 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1992 8 365281 365273 1.43247 0 - 0.0299513 2026-02-23T13:46:38.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:46:38 trial068 ceph-mon[38805]: pgmap v1410: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:46:38.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:46:38 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2273135164' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:46:39.254 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1993 8 365281 365273 1.43175 0 - 0.0299513 2026-02-23T13:46:40.084 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:46:40.202 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:46:40.254 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1994 8 365281 365273 1.43103 0 - 0.0299513 2026-02-23T13:46:40.302 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:46:40.300+0000 7ff2c074c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:46:40.302 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:46:40.300+0000 7ff2c074c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:46:40.314 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:46:40.315 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:46:40.324 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:46:40.325 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:46:40.325 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:46:40.335 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:46:40.335 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:46:40.364 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:46:40 trial068 ceph-mon[38805]: pgmap v1411: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:46:40.365 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:46:40.365 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:46:41.254 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1995 8 365281 365273 1.43031 0 - 0.0299513 2026-02-23T13:46:41.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:46:41 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/4068785299' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:46:42.254 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1996 8 365281 365273 1.4296 0 - 0.0299513 2026-02-23T13:46:42.366 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:46:42.484 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:46:42.538 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:46:42 trial068 ceph-mon[38805]: pgmap v1412: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:46:42.583 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:46:42.582+0000 7fd2f1f87640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:46:42.584 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:46:42.582+0000 7fd2f1f87640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:46:42.596 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:46:42.596 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:46:42.606 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:46:42.607 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:46:42.607 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:46:42.617 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:46:42.617 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:46:42.648 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:46:42.648 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:46:43.255 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1997 8 365281 365273 1.42888 0 - 0.0299513 2026-02-23T13:46:43.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:46:43 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/217743959' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:46:44.255 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1998 8 365281 365273 1.42816 0 - 0.0299513 2026-02-23T13:46:44.289 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:46:43 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:46:43] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:46:44.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:46:43 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:46:43.974Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:46:44.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:46:44 trial068 ceph-mon[38805]: pgmap v1413: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:46:44.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:46:44 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2026-02-23T13:46:44.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:46:44 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T13:46:44.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:46:44 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T13:46:44.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:46:44 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T13:46:44.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:46:44 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T13:46:44.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:46:44 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-23T13:46:44.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:46:44 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2026-02-23T13:46:44.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:46:44 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T13:46:44.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:46:44 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2026-02-23T13:46:44.649 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:46:44.765 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:46:44.868 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:46:44.866+0000 7f4f440e2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:46:44.869 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:46:44.867+0000 7f4f440e2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:46:44.880 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:46:44.880 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:46:44.890 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:46:44.891 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:46:44.891 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:46:44.901 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:46:44.902 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:46:44.932 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:46:44.932 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:46:45.255 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 1999 8 365281 365273 1.42745 0 - 0.0299513 2026-02-23T13:46:45.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:46:45 trial068 ceph-mon[38805]: pgmap v1414: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:46:45.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:46:45 trial068 ceph-mon[38805]: pgmap v1415: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:46:45.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:46:45 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2703329983' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:46:46.255 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T13:46:46.254615+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T13:46:46.255 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T13:46:46.255 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2000 8 365281 365273 1.42674 0 - 0.0299513 2026-02-23T13:46:46.934 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:46:47.050 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:46:47.147 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:46:47.145+0000 7f77e3405640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:46:47.148 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:46:47.146+0000 7f77e3405640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:46:47.159 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:46:47.159 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:46:47.170 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:46:47.170 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:46:47.170 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:46:47.180 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:46:47.181 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:46:47.211 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:46:47.212 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:46:47.255 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2001 8 365281 365273 1.42602 0 - 0.0299513 2026-02-23T13:46:47.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:46:47 trial068 ceph-mon[38805]: pgmap v1416: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:46:47.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:46:47 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T13:46:47.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:46:47 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T13:46:47.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:46:47 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1692650333' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:46:48.255 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2002 8 365281 365273 1.42531 0 - 0.0299513 2026-02-23T13:46:49.213 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:46:49.255 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2003 8 365281 365273 1.4246 0 - 0.0299513 2026-02-23T13:46:49.331 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:46:49.435 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:46:49.433+0000 7fd22bd9a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:46:49.435 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:46:49.433+0000 7fd22bd9a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:46:49.447 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:46:49.447 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:46:49.458 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:46:49.459 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:46:49.459 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:46:49.469 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:46:49.469 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:46:49.495 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:46:49 trial068 ceph-mon[38805]: pgmap v1417: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:46:49.496 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:46:49.497 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:46:50.255 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2004 8 365281 365273 1.42389 0 - 0.0299513 2026-02-23T13:46:50.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:46:50 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/752142785' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:46:51.255 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2005 8 365281 365273 1.42318 0 - 0.0299513 2026-02-23T13:46:51.498 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:46:51.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:46:51 trial068 ceph-mon[38805]: pgmap v1418: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:46:51.617 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:46:51.724 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:46:51.722+0000 7f0748375640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:46:51.724 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:46:51.722+0000 7f0748375640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:46:51.736 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:46:51.737 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:46:51.747 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:46:51.747 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:46:51.747 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:46:51.757 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:46:51.757 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:46:51.785 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:46:51.786 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:46:52.255 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2006 8 365281 365273 1.42247 0 - 0.0299513 2026-02-23T13:46:52.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:46:52 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/4078804840' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:46:53.255 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2007 8 365281 365273 1.42176 0 - 0.0299513 2026-02-23T13:46:53.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:46:53 trial068 ceph-mon[38805]: pgmap v1419: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:46:53.787 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:46:53.905 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:46:54.003 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:46:54.001+0000 7f80e9658640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:46:54.003 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:46:54.001+0000 7f80e9658640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:46:54.015 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:46:54.015 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:46:54.025 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:46:54.026 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:46:54.026 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:46:54.035 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:46:54.036 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:46:54.064 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:46:53 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:46:53] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:46:54.065 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:46:54.065 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:46:54.255 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2008 8 365281 365273 1.42105 0 - 0.0299513 2026-02-23T13:46:54.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:46:54 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1923037066' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:46:54.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:46:53 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:46:53.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:46:55.255 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2009 8 365281 365273 1.42034 0 - 0.0299513 2026-02-23T13:46:55.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:46:55 trial068 ceph-mon[38805]: pgmap v1420: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:46:56.067 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:46:56.187 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:46:56.256 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2010 8 365281 365273 1.41964 0 - 0.0299513 2026-02-23T13:46:56.289 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:46:56.287+0000 7f10b512e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:46:56.289 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:46:56.287+0000 7f10b512e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:46:56.301 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:46:56.301 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:46:56.311 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:46:56.311 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:46:56.311 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:46:56.322 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:46:56.322 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:46:56.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:46:56 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2925585519' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:46:56.356 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:46:56.356 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:46:57.256 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2011 8 365281 365273 1.41893 0 - 0.0299513 2026-02-23T13:46:57.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:46:57 trial068 ceph-mon[38805]: pgmap v1421: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:46:58.256 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2012 8 365281 365273 1.41823 0 - 0.0299513 2026-02-23T13:46:58.357 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:46:58.476 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:46:58.575 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:46:58.573+0000 7f7daa958640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:46:58.576 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:46:58 trial068 ceph-mon[38805]: 5.8 scrub starts 2026-02-23T13:46:58.576 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:46:58 trial068 ceph-mon[38805]: 5.8 scrub ok 2026-02-23T13:46:58.576 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:46:58.574+0000 7f7daa958640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:46:58.588 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:46:58.588 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:46:58.598 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:46:58.598 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:46:58.598 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:46:58.608 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:46:58.608 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:46:58.639 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:46:58.640 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:46:59.256 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2013 8 365281 365273 1.41752 0 - 0.0299513 2026-02-23T13:46:59.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:46:59 trial068 ceph-mon[38805]: pgmap v1422: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:46:59.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:46:59 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1887111436' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:47:00.256 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2014 8 365281 365273 1.41682 0 - 0.0299513 2026-02-23T13:47:00.641 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:47:00.759 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:47:00.854 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:47:00.852+0000 7f0c59d00640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:47:00.854 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:47:00.852+0000 7f0c59d00640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:47:00.866 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:47:00.866 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:47:00.876 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:47:00.876 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:47:00.876 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:47:00.886 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:47:00.886 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:47:00.915 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:47:00.915 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:47:01.256 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2015 8 365281 365273 1.41612 0 - 0.0299513 2026-02-23T13:47:01.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:47:01 trial068 ceph-mon[38805]: pgmap v1423: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:47:01.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:47:01 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/491394422' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:47:02.256 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2016 8 365281 365273 1.41541 0 - 0.0299513 2026-02-23T13:47:02.916 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:47:03.033 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:47:03.129 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:47:03.127+0000 7f456c933640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:47:03.129 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:47:03.127+0000 7f456c933640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:47:03.140 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:47:03.141 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:47:03.151 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:47:03.151 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:47:03.151 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:47:03.161 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:47:03.161 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:47:03.192 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:47:03.192 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:47:03.256 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2017 8 365281 365273 1.41471 0 - 0.0299513 2026-02-23T13:47:03.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:47:03 trial068 ceph-mon[38805]: pgmap v1424: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:47:03.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:47:03 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/170657494' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:47:04.256 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2018 8 365281 365273 1.41401 0 - 0.0299513 2026-02-23T13:47:04.266 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:47:03 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:47:03.974Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:47:04.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:47:03 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:47:03] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:47:05.193 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:47:05.256 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2019 8 365281 365273 1.41331 0 - 0.0299513 2026-02-23T13:47:05.309 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:47:05.408 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:47:05.406+0000 7ff3aafcc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:47:05.408 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:47:05.406+0000 7ff3aafcc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:47:05.420 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:47:05.420 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:47:05.431 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:47:05.431 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:47:05.431 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:47:05.441 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:47:05.441 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:47:05.470 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:47:05 trial068 ceph-mon[38805]: pgmap v1425: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:47:05.471 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:47:05.471 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:47:06.256 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T13:47:06.256104+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T13:47:06.257 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T13:47:06.257 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2020 8 365281 365273 1.41261 0 - 0.0299513 2026-02-23T13:47:06.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:47:06 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/480701454' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:47:07.256 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2021 8 365281 365273 1.41191 0 - 0.0299513 2026-02-23T13:47:07.472 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:47:07.592 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:47:07.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:47:07 trial068 ceph-mon[38805]: pgmap v1426: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:47:07.688 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:47:07.687+0000 7f70c732a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:47:07.689 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:47:07.687+0000 7f70c732a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:47:07.700 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:47:07.701 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:47:07.711 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:47:07.711 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:47:07.711 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:47:07.722 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:47:07.722 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:47:07.752 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:47:07.753 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:47:08.257 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2022 8 365281 365273 1.41121 0 - 0.0299513 2026-02-23T13:47:08.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:47:08 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/244707120' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:47:09.256 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2023 8 365281 365273 1.41051 0 - 0.0299513 2026-02-23T13:47:09.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:47:09 trial068 ceph-mon[38805]: pgmap v1427: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:47:09.753 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:47:09.868 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:47:09.974 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:47:09.972+0000 7f97ca860640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:47:09.974 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:47:09.972+0000 7f97ca860640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:47:09.986 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:47:09.986 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:47:09.995 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:47:09.996 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:47:09.996 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:47:10.005 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:47:10.006 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:47:10.035 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:47:10.036 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:47:10.257 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2024 8 365281 365273 1.40982 0 - 0.0299513 2026-02-23T13:47:10.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:47:10 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2834906712' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:47:11.257 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2025 8 365281 365273 1.40912 0 - 0.0299513 2026-02-23T13:47:11.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:47:11 trial068 ceph-mon[38805]: pgmap v1428: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:47:11.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:47:11 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T13:47:11.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:47:11 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T13:47:12.037 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:47:12.152 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:47:12.256 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:47:12.254+0000 7f35e2d88640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:47:12.256 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:47:12.254+0000 7f35e2d88640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:47:12.257 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2026 8 365281 365273 1.40843 0 - 0.0299513 2026-02-23T13:47:12.268 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:47:12.268 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:47:12.278 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:47:12.279 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:47:12.279 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:47:12.288 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:47:12.289 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:47:12.319 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:47:12.320 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:47:12.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:47:12 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3921841074' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:47:13.257 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2027 8 365281 365273 1.40773 0 - 0.0299513 2026-02-23T13:47:13.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:47:13 trial068 ceph-mon[38805]: pgmap v1429: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:47:14.257 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2028 8 365281 365273 1.40704 0 - 0.0299513 2026-02-23T13:47:14.321 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:47:14.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:47:13 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:47:13] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:47:14.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:47:13 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:47:13.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:47:14.437 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:47:14.541 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:47:14.539+0000 7fec92969640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:47:14.541 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:47:14.539+0000 7fec92969640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:47:14.553 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:47:14.554 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:47:14.564 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:47:14.564 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:47:14.565 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:47:14.574 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:47:14.574 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:47:14.605 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:47:14.605 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:47:15.257 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2029 8 365281 365273 1.40634 0 - 0.0299513 2026-02-23T13:47:15.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:47:15 trial068 ceph-mon[38805]: pgmap v1430: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:47:15.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:47:15 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3321388355' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:47:16.257 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2030 8 365281 365273 1.40565 0 - 0.0299513 2026-02-23T13:47:16.606 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:47:16.723 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:47:16.823 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:47:16.821+0000 7f22918ee640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:47:16.823 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:47:16.821+0000 7f22918ee640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:47:16.835 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:47:16.835 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:47:16.845 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:47:16.845 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:47:16.845 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:47:16.855 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:47:16.855 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:47:16.885 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:47:16.886 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:47:17.257 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2031 8 365281 365273 1.40496 0 - 0.0299513 2026-02-23T13:47:17.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:47:17 trial068 ceph-mon[38805]: pgmap v1431: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:47:17.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:47:17 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2311053865' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:47:18.257 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2032 8 365281 365273 1.40427 0 - 0.0299513 2026-02-23T13:47:18.887 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:47:19.001 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:47:19.099 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:47:19.097+0000 7f13771aa640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:47:19.100 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:47:19.098+0000 7f13771aa640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:47:19.111 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:47:19.111 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:47:19.120 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:47:19.121 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:47:19.121 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:47:19.132 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:47:19.132 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:47:19.162 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:47:19.163 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:47:19.257 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2033 8 365281 365273 1.40358 0 - 0.0299513 2026-02-23T13:47:19.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:47:19 trial068 ceph-mon[38805]: pgmap v1432: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:47:19.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:47:19 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T13:47:19.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:47:19 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T13:47:19.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:47:19 trial068 ceph-mon[38805]: 5.e scrub starts 2026-02-23T13:47:19.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:47:19 trial068 ceph-mon[38805]: 5.e scrub ok 2026-02-23T13:47:19.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:47:19 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1821544695' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:47:20.257 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2034 8 365281 365273 1.40289 0 - 0.0299513 2026-02-23T13:47:21.164 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:47:21.257 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2035 8 365281 365273 1.4022 0 - 0.0299513 2026-02-23T13:47:21.283 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:47:21.385 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:47:21.383+0000 7fdd4d4ca640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:47:21.385 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:47:21.383+0000 7fdd4d4ca640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:47:21.397 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:47:21.397 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:47:21.408 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:47:21.408 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:47:21.408 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:47:21.419 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:47:21.419 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:47:21.449 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:47:21 trial068 ceph-mon[38805]: pgmap v1433: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:47:21.449 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:47:21.450 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:47:22.257 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2036 8 365281 365273 1.40151 0 - 0.0299513 2026-02-23T13:47:22.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:47:22 trial068 ceph-mon[38805]: 5.5 scrub starts 2026-02-23T13:47:22.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:47:22 trial068 ceph-mon[38805]: 5.5 scrub ok 2026-02-23T13:47:22.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:47:22 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1205208152' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:47:23.258 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2037 8 365281 365273 1.40082 0 - 0.0299513 2026-02-23T13:47:23.451 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:47:23.568 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:47:23.590 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:47:23 trial068 ceph-mon[38805]: pgmap v1434: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:47:23.669 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:47:23.667+0000 7f63dab79640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:47:23.669 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:47:23.667+0000 7f63dab79640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:47:23.681 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:47:23.681 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:47:23.691 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:47:23.691 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:47:23.692 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:47:23.701 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:47:23.702 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:47:23.732 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:47:23.732 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:47:24.258 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2038 8 365281 365273 1.40013 0 - 0.0299513 2026-02-23T13:47:24.330 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:47:23 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:47:23] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:47:24.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:47:23 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:47:23.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:47:24.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:47:24 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1254849579' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:47:24.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:47:24 trial068 ceph-mon[38805]: 5.a scrub starts 2026-02-23T13:47:25.258 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2039 8 365281 365273 1.39945 0 - 0.0299513 2026-02-23T13:47:25.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:47:25 trial068 ceph-mon[38805]: 5.a scrub ok 2026-02-23T13:47:25.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:47:25 trial068 ceph-mon[38805]: pgmap v1435: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:47:25.733 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:47:25.850 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:47:25.953 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:47:25.951+0000 7f4701f40640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:47:25.953 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:47:25.952+0000 7f4701f40640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:47:25.965 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:47:25.965 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:47:25.975 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:47:25.975 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:47:25.976 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:47:25.985 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:47:25.986 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:47:26.017 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:47:26.018 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:47:26.258 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T13:47:26.257654+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T13:47:26.258 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T13:47:26.258 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2040 8 365281 365273 1.39876 0 - 0.0299513 2026-02-23T13:47:26.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:47:26 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/4057257016' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:47:27.258 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2041 8 365281 365273 1.39808 0 - 0.0299513 2026-02-23T13:47:27.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:47:27 trial068 ceph-mon[38805]: pgmap v1436: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:47:28.019 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:47:28.136 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:47:28.231 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:47:28.229+0000 7f68a50e6640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:47:28.231 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:47:28.230+0000 7f68a50e6640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:47:28.243 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:47:28.243 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:47:28.253 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:47:28.253 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:47:28.253 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:47:28.258 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2042 8 365281 365273 1.39739 0 - 0.0299513 2026-02-23T13:47:28.262 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:47:28.263 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:47:28.293 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:47:28.294 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:47:28.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:47:28 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1526401772' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:47:29.258 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2043 8 365281 365273 1.39671 0 - 0.0299513 2026-02-23T13:47:29.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:47:29 trial068 ceph-mon[38805]: pgmap v1437: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:47:30.258 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2044 8 365281 365273 1.39602 0 - 0.0299513 2026-02-23T13:47:30.295 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:47:30.412 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:47:30.515 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:47:30.513+0000 7fee3518e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:47:30.516 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:47:30.514+0000 7fee3518e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:47:30.527 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:47:30.528 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:47:30.538 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:47:30.538 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:47:30.538 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:47:30.548 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:47:30.548 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:47:30.576 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:47:30.576 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:47:31.258 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2045 8 365281 365273 1.39534 0 - 0.0299513 2026-02-23T13:47:31.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:47:31 trial068 ceph-mon[38805]: pgmap v1438: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:47:31.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:47:31 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1113180006' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:47:32.258 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2046 8 365281 365273 1.39466 0 - 0.0299513 2026-02-23T13:47:32.577 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:47:32.692 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:47:32.788 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:47:32.786+0000 7f24e9534640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:47:32.788 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:47:32.786+0000 7f24e9534640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:47:32.800 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:47:32.800 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:47:32.810 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:47:32.810 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:47:32.811 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:47:32.820 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:47:32.820 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:47:32.849 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:47:32.850 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:47:33.258 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2047 8 365281 365273 1.39398 0 - 0.0299513 2026-02-23T13:47:33.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:47:33 trial068 ceph-mon[38805]: pgmap v1439: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:47:33.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:47:33 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/200517414' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:47:34.259 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2048 8 365281 365273 1.3933 0 - 0.0299513 2026-02-23T13:47:34.266 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:47:33 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:47:33.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:47:34.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:47:33 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:47:33] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:47:34.851 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:47:34.968 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:47:35.070 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:47:35.068+0000 7fe7de041640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:47:35.070 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:47:35.068+0000 7fe7de041640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:47:35.082 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:47:35.082 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:47:35.093 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:47:35.093 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:47:35.094 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:47:35.104 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:47:35.104 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:47:35.134 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:47:35.135 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:47:35.259 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2049 8 365281 365273 1.39262 0 - 0.0299513 2026-02-23T13:47:35.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:47:35 trial068 ceph-mon[38805]: pgmap v1440: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:47:35.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:47:35 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2832263274' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:47:36.259 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2050 8 365281 365273 1.39194 0 - 0.0299513 2026-02-23T13:47:37.136 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:47:37.255 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:47:37.259 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2051 8 365281 365273 1.39126 0 - 0.0299513 2026-02-23T13:47:37.356 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:47:37.354+0000 7fe7e5751640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:47:37.356 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:47:37.354+0000 7fe7e5751640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:47:37.368 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:47:37.368 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:47:37.378 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:47:37.379 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:47:37.379 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:47:37.389 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:47:37.389 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:47:37.419 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:47:37 trial068 ceph-mon[38805]: pgmap v1441: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:47:37.419 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:47:37.420 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:47:38.259 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2052 8 365281 365273 1.39058 0 - 0.0299513 2026-02-23T13:47:38.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:47:38 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2148411201' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:47:39.259 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2053 8 365281 365273 1.3899 0 - 0.0299513 2026-02-23T13:47:39.421 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:47:39.539 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:47:39.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:47:39 trial068 ceph-mon[38805]: pgmap v1442: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:47:39.634 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:47:39.632+0000 7f3ad74e3640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:47:39.634 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:47:39.632+0000 7f3ad74e3640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:47:39.647 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:47:39.647 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:47:39.657 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:47:39.657 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:47:39.657 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:47:39.667 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:47:39.668 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:47:39.698 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:47:39.698 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:47:40.259 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2054 8 365281 365273 1.38923 0 - 0.0299513 2026-02-23T13:47:40.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:47:40 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3226237843' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:47:41.259 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2055 8 365281 365273 1.38855 0 - 0.0299513 2026-02-23T13:47:41.700 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:47:41.724 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:47:41 trial068 ceph-mon[38805]: pgmap v1443: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:47:41.819 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:47:41.916 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:47:41.914+0000 7fa063daf640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:47:41.916 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:47:41.914+0000 7fa063daf640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:47:41.927 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:47:41.928 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:47:41.937 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:47:41.938 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:47:41.938 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:47:41.948 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:47:41.948 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:47:41.978 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:47:41.978 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:47:42.259 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2056 8 365281 365273 1.38788 0 - 0.0299513 2026-02-23T13:47:42.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:47:42 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2017239396' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:47:43.259 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2057 8 365281 365273 1.3872 0 - 0.0299513 2026-02-23T13:47:43.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:47:43 trial068 ceph-mon[38805]: pgmap v1444: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:47:43.980 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:47:44.095 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:47:44.196 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:47:44.194+0000 7f2ed3dec640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:47:44.196 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:47:44.194+0000 7f2ed3dec640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:47:44.209 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:47:44.210 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:47:44.221 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:47:44.221 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:47:44.221 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:47:44.232 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:47:44.233 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:47:44.233 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:47:43 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:47:43] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:47:44.259 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2058 8 365281 365273 1.38653 0 - 0.0299513 2026-02-23T13:47:44.263 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:47:44.264 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:47:44.307 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:47:43 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:47:43.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:47:44.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:47:44 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2026-02-23T13:47:44.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:47:44 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/622562975' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:47:45.259 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2059 8 365281 365273 1.38585 0 - 0.0299513 2026-02-23T13:47:45.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:47:45 trial068 ceph-mon[38805]: pgmap v1445: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:47:45.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:47:45 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T13:47:45.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:47:45 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T13:47:45.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:47:45 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T13:47:45.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:47:45 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T13:47:45.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:47:45 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-23T13:47:45.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:47:45 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2026-02-23T13:47:45.845 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:47:45 trial068 ceph-mon[38805]: pgmap v1446: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:47:45.845 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:47:45 trial068 ceph-mon[38805]: pgmap v1447: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:47:45.845 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:47:45 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T13:47:45.845 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:47:45 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2026-02-23T13:47:46.259 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T13:47:46.259220+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T13:47:46.260 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T13:47:46.260 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2060 8 365281 365273 1.38518 0 - 0.0299513 2026-02-23T13:47:46.265 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:47:46.384 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:47:46.482 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:47:46.480+0000 7f6a0c3f0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:47:46.482 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:47:46.480+0000 7f6a0c3f0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:47:46.494 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:47:46.494 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:47:46.505 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:47:46.505 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:47:46.506 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:47:46.516 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:47:46.517 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:47:46.545 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:47:46 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2956696025' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:47:46.546 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:47:46.547 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:47:47.260 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2061 8 365281 365273 1.38451 0 - 0.0299513 2026-02-23T13:47:47.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:47:47 trial068 ceph-mon[38805]: pgmap v1448: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:47:48.260 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2062 8 365281 365273 1.38384 0 - 0.0299513 2026-02-23T13:47:48.548 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:47:48.665 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:47:48.768 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:47:48.766+0000 7efffabdf640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:47:48.768 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:47:48.766+0000 7efffabdf640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:47:48.780 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:47:48.781 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:47:48.790 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:47:48.791 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:47:48.791 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:47:48.801 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:47:48.802 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:47:48.829 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:47:48 trial068 ceph-mon[38805]: pgmap v1449: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:47:48.830 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:47:48.830 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:47:49.260 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2063 8 365281 365273 1.38317 0 - 0.0299513 2026-02-23T13:47:50.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:47:49 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/636517934' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:47:50.260 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2064 8 365281 365273 1.3825 0 - 0.0299513 2026-02-23T13:47:50.831 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:47:50.951 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:47:50.973 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:47:50 trial068 ceph-mon[38805]: pgmap v1450: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:47:51.053 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:47:51.051+0000 7fd387297640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:47:51.054 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:47:51.052+0000 7fd387297640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:47:51.065 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:47:51.065 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:47:51.076 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:47:51.076 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:47:51.076 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:47:51.086 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:47:51.086 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:47:51.117 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:47:51.118 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:47:51.260 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2065 8 365281 365273 1.38183 0 - 0.0299513 2026-02-23T13:47:52.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:47:51 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1910021524' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:47:52.260 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2066 8 365281 365273 1.38116 0 - 0.0299513 2026-02-23T13:47:53.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:47:52 trial068 ceph-mon[38805]: pgmap v1451: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:47:53.118 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:47:53.235 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:47:53.260 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2067 8 365281 365273 1.38049 0 - 0.0299513 2026-02-23T13:47:53.334 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:47:53.333+0000 7fb25ff70640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:47:53.335 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:47:53.333+0000 7fb25ff70640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:47:53.347 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:47:53.347 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:47:53.356 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:47:53.357 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:47:53.357 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:47:53.366 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:47:53.366 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:47:53.397 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:47:53.397 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:47:53.974 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:47:53 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/322503649' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:47:54.260 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2068 8 365281 365273 1.37982 0 - 0.0299513 2026-02-23T13:47:54.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:47:53 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:47:53] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:47:54.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:47:53 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:47:53.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:47:55.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:47:54 trial068 ceph-mon[38805]: pgmap v1452: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:47:55.260 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2069 8 365281 365273 1.37916 0 - 0.0299513 2026-02-23T13:47:55.398 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:47:55.516 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:47:55.618 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:47:55.616+0000 7fcbca48d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:47:55.618 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:47:55.616+0000 7fcbca48d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:47:55.630 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:47:55.630 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:47:55.642 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:47:55.642 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:47:55.642 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:47:55.652 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:47:55.652 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:47:55.682 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:47:55.683 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:47:56.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:47:55 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1400861531' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:47:56.260 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2070 8 365281 365273 1.37849 0 - 0.0299513 2026-02-23T13:47:57.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:47:56 trial068 ceph-mon[38805]: pgmap v1453: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:47:57.260 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2071 8 365281 365273 1.37782 0 - 0.0299513 2026-02-23T13:47:57.684 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:47:57.800 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:47:57.901 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:47:57.899+0000 7fe1effc2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:47:57.901 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:47:57.899+0000 7fe1effc2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:47:57.913 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:47:57.913 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:47:57.923 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:47:57.923 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:47:57.923 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:47:57.933 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:47:57.934 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:47:57.961 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:47:57 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3930766224' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:47:57.962 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:47:57.963 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:47:58.260 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2072 8 365281 365273 1.37716 0 - 0.0299513 2026-02-23T13:47:59.260 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2073 8 365281 365273 1.37649 0 - 0.0299513 2026-02-23T13:47:59.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:47:58 trial068 ceph-mon[38805]: pgmap v1454: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:47:59.964 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:48:00.080 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:48:00.178 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:48:00.177+0000 7f1ea0b45640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:48:00.178 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:48:00.177+0000 7f1ea0b45640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:48:00.191 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:48:00.192 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:48:00.201 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:48:00.201 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:48:00.201 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:48:00.211 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:48:00.211 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:48:00.242 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:48:00.242 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:48:00.260 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2074 8 365281 365273 1.37583 0 - 0.0299513 2026-02-23T13:48:00.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:48:00 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3904530080' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:48:01.260 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2075 8 365281 365273 1.37517 0 - 0.0299513 2026-02-23T13:48:01.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:48:01 trial068 ceph-mon[38805]: pgmap v1455: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:48:02.243 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:48:02.260 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2076 8 365281 365273 1.3745 0 - 0.0299513 2026-02-23T13:48:02.361 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:48:02.466 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:48:02.464+0000 7f44f66a0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:48:02.466 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:48:02.464+0000 7f44f66a0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:48:02.478 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:48:02.478 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:48:02.488 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:48:02.489 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:48:02.489 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:48:02.498 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:48:02.499 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:48:02.530 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:48:02.530 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:48:02.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:48:02 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/933126855' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:48:03.260 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2077 8 365281 365273 1.37384 0 - 0.0299513 2026-02-23T13:48:03.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:48:03 trial068 ceph-mon[38805]: pgmap v1456: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:48:04.261 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2078 8 365281 365273 1.37318 0 - 0.0299513 2026-02-23T13:48:04.266 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:48:03 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:48:03.972Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:48:04.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:48:03 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:48:03] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:48:04.532 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:48:04.649 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:48:04.750 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:48:04.748+0000 7f38b218f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:48:04.751 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:48:04.749+0000 7f38b218f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:48:04.763 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:48:04.763 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:48:04.773 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:48:04.773 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:48:04.774 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:48:04 trial068 ceph-mon[38805]: 5.7 scrub starts 2026-02-23T13:48:04.774 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:48:04 trial068 ceph-mon[38805]: 5.7 scrub ok 2026-02-23T13:48:04.774 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:48:04 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T13:48:04.774 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:48:04 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T13:48:04.774 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:48:04.785 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:48:04.785 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:48:04.813 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:48:04.813 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:48:05.261 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2079 8 365281 365273 1.37252 0 - 0.0299513 2026-02-23T13:48:05.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:48:05 trial068 ceph-mon[38805]: pgmap v1457: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:48:05.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:48:05 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2539097971' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:48:06.261 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T13:48:06.260615+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T13:48:06.261 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T13:48:06.261 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2080 8 365281 365273 1.37186 0 - 0.0299513 2026-02-23T13:48:06.815 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:48:06.839 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:48:06 trial068 ceph-mon[38805]: 5.8 scrub starts 2026-02-23T13:48:06.839 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:48:06 trial068 ceph-mon[38805]: 5.8 scrub ok 2026-02-23T13:48:06.933 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:48:07.037 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:48:07.035+0000 7f37862e4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:48:07.038 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:48:07.036+0000 7f37862e4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:48:07.049 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:48:07.050 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:48:07.059 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:48:07.060 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:48:07.060 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:48:07.071 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:48:07.071 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:48:07.101 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:48:07.102 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:48:07.261 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2081 8 365281 365273 1.3712 0 - 0.0299513 2026-02-23T13:48:07.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:48:07 trial068 ceph-mon[38805]: pgmap v1458: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:48:07.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:48:07 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1904655076' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:48:08.261 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2082 8 365281 365273 1.37054 0 - 0.0299513 2026-02-23T13:48:09.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:48:08 trial068 ceph-mon[38805]: pgmap v1459: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:48:09.103 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:48:09.219 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:48:09.261 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2083 8 365281 365273 1.36989 0 - 0.0299513 2026-02-23T13:48:09.319 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:48:09.317+0000 7f8afae5c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:48:09.320 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:48:09.318+0000 7f8afae5c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:48:09.331 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:48:09.331 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:48:09.341 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:48:09.341 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:48:09.341 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:48:09.352 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:48:09.352 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:48:09.383 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:48:09.383 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:48:10.261 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2084 8 365281 365273 1.36923 0 - 0.0299513 2026-02-23T13:48:10.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:48:10 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2497726286' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:48:11.261 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2085 8 365281 365273 1.36857 0 - 0.0299513 2026-02-23T13:48:11.384 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:48:11.502 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:48:11.548 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:48:11 trial068 ceph-mon[38805]: pgmap v1460: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:48:11.603 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:48:11.601+0000 7fceb5ebb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:48:11.604 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:48:11.602+0000 7fceb5ebb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:48:11.616 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:48:11.617 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:48:11.627 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:48:11.627 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:48:11.627 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:48:11.637 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:48:11.638 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:48:11.668 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:48:11.668 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:48:12.261 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2086 8 365281 365273 1.36792 0 - 0.0299513 2026-02-23T13:48:12.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:48:12 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3833987369' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:48:13.261 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2087 8 365281 365273 1.36726 0 - 0.0299513 2026-02-23T13:48:13.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:48:13 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T13:48:13.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:48:13 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T13:48:13.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:48:13 trial068 ceph-mon[38805]: pgmap v1461: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:48:13.670 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:48:13.787 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:48:13.885 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:48:13.883+0000 7f810cc9b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:48:13.885 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:48:13.884+0000 7f810cc9b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:48:13.898 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:48:13.898 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:48:13.908 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:48:13.908 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:48:13.908 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:48:13.918 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:48:13.918 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:48:13.949 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:48:13.950 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:48:14.261 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2088 8 365281 365273 1.36661 0 - 0.0299513 2026-02-23T13:48:14.299 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:48:13 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:48:13] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:48:14.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:48:13 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:48:13.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:48:14.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:48:14 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/4243879679' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:48:15.261 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2089 8 365281 365273 1.36595 0 - 0.0299513 2026-02-23T13:48:15.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:48:15 trial068 ceph-mon[38805]: pgmap v1462: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:48:15.951 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:48:16.070 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:48:16.173 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:48:16.171+0000 7f8e89c9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:48:16.173 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:48:16.172+0000 7f8e89c9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:48:16.185 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:48:16.185 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:48:16.195 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:48:16.195 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:48:16.195 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:48:16.205 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:48:16.205 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:48:16.236 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:48:16.236 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:48:16.261 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2090 8 365281 365273 1.3653 0 - 0.0299513 2026-02-23T13:48:16.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:48:16 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/442722333' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:48:17.262 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2091 8 365281 365273 1.36464 0 - 0.0299513 2026-02-23T13:48:17.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:48:17 trial068 ceph-mon[38805]: pgmap v1463: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:48:18.237 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:48:18.262 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2092 8 365281 365273 1.36399 0 - 0.0299513 2026-02-23T13:48:18.354 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:48:18.458 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:48:18.456+0000 7f6b8b4fc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:48:18.458 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:48:18.456+0000 7f6b8b4fc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:48:18.469 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:48:18.470 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:48:18.479 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:48:18.480 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:48:18.480 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:48:18.489 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:48:18.490 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:48:18.522 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:48:18 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1277262569' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:48:18.523 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:48:18.523 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:48:19.262 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2093 8 365281 365273 1.36334 0 - 0.0299513 2026-02-23T13:48:19.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:48:19 trial068 ceph-mon[38805]: pgmap v1464: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:48:20.262 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2094 8 365281 365273 1.36269 0 - 0.0299513 2026-02-23T13:48:20.524 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:48:20.642 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:48:20.752 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:48:20.750+0000 7fbfe07a8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:48:20.752 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:48:20.750+0000 7fbfe07a8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:48:20.764 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:48:20.764 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:48:20.774 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:48:20.775 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:48:20.775 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:48:20.785 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:48:20.785 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:48:20.801 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:48:20 trial068 ceph-mon[38805]: pgmap v1465: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:48:20.815 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:48:20.816 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:48:21.262 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2095 8 365281 365273 1.36204 0 - 0.0299513 2026-02-23T13:48:22.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:48:21 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2214850088' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:48:22.262 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2096 8 365281 365273 1.36139 0 - 0.0299513 2026-02-23T13:48:22.818 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:48:22.937 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:48:22.958 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:48:22 trial068 ceph-mon[38805]: pgmap v1466: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:48:23.036 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:48:23.034+0000 7f03e3318640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:48:23.036 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:48:23.034+0000 7f03e3318640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:48:23.048 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:48:23.048 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:48:23.058 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:48:23.059 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:48:23.059 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:48:23.068 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:48:23.069 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:48:23.099 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:48:23.100 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:48:23.262 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2097 8 365281 365273 1.36074 0 - 0.0299513 2026-02-23T13:48:23.974 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:48:23 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3149532614' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:48:24.262 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2098 8 365281 365273 1.36009 0 - 0.0299513 2026-02-23T13:48:24.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:48:23 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:48:23] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:48:24.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:48:23 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:48:23.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:48:25.101 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:48:25.223 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:48:25.262 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2099 8 365281 365273 1.35944 0 - 0.0299513 2026-02-23T13:48:25.322 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:48:25.320+0000 7f3e15255640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:48:25.322 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:48:25.320+0000 7f3e15255640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:48:25.334 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:48:25.335 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:48:25.344 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:48:25.345 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:48:25.345 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:48:25.354 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:48:25.355 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:48:25.384 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:48:25 trial068 ceph-mon[38805]: pgmap v1467: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:48:25.385 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:48:25.386 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:48:26.262 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T13:48:26.262144+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T13:48:26.263 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T13:48:26.263 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2100 8 365281 365273 1.3588 0 - 0.0299513 2026-02-23T13:48:26.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:48:26 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2388968526' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:48:27.262 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2101 8 365281 365273 1.35815 0 - 0.0299513 2026-02-23T13:48:27.387 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:48:27.505 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:48:27.552 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:48:27 trial068 ceph-mon[38805]: pgmap v1468: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:48:27.597 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:48:27.595+0000 7f4a18352640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:48:27.597 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:48:27.595+0000 7f4a18352640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:48:27.609 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:48:27.609 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:48:27.619 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:48:27.620 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:48:27.620 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:48:27.629 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:48:27.630 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:48:27.675 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:48:27.676 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:48:28.263 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2102 8 365281 365273 1.3575 0 - 0.0299513 2026-02-23T13:48:28.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:48:28 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/909418026' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:48:29.263 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2103 8 365281 365273 1.35686 0 - 0.0299513 2026-02-23T13:48:29.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:48:29 trial068 ceph-mon[38805]: pgmap v1469: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:48:29.677 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:48:29.793 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:48:29.894 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:48:29.892+0000 7fa8b4945640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:48:29.894 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:48:29.892+0000 7fa8b4945640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:48:29.905 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:48:29.906 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:48:29.915 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:48:29.916 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:48:29.916 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:48:29.925 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:48:29.926 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:48:29.956 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:48:29.957 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:48:30.263 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2104 8 365281 365273 1.35621 0 - 0.0299513 2026-02-23T13:48:30.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:48:30 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2766027166' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:48:31.263 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2105 8 365281 365273 1.35557 0 - 0.0299513 2026-02-23T13:48:31.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:48:31 trial068 ceph-mon[38805]: pgmap v1470: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:48:31.958 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:48:32.076 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:48:32.180 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:48:32.178+0000 7f8b91bb7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:48:32.180 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:48:32.178+0000 7f8b91bb7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:48:32.192 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:48:32.192 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:48:32.202 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:48:32.202 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:48:32.202 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:48:32.212 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:48:32.212 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:48:32.245 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:48:32.246 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:48:32.263 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2106 8 365281 365273 1.35493 0 - 0.0299513 2026-02-23T13:48:32.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:48:32 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/4015726617' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:48:33.263 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2107 8 365281 365273 1.35428 0 - 0.0299513 2026-02-23T13:48:33.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:48:33 trial068 ceph-mon[38805]: pgmap v1471: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:48:34.246 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:48:34.263 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2108 8 365281 365273 1.35364 0 - 0.0299513 2026-02-23T13:48:34.270 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:48:33 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:48:33] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:48:34.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:48:33 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:48:33.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:48:34.363 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:48:34.465 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:48:34.463+0000 7fe0bf40c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:48:34.466 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:48:34.464+0000 7fe0bf40c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:48:34.477 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:48:34.477 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:48:34.487 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:48:34.488 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:48:34.488 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:48:34.497 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:48:34.497 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:48:34.526 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:48:34.527 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:48:35.263 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2109 8 365281 365273 1.353 0 - 0.0299513 2026-02-23T13:48:35.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:48:35 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T13:48:35.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:48:35 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T13:48:35.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:48:35 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/231754284' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:48:35.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:48:35 trial068 ceph-mon[38805]: pgmap v1472: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:48:35.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:48:35 trial068 ceph-mon[38805]: 5.e scrub starts 2026-02-23T13:48:35.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:48:35 trial068 ceph-mon[38805]: 5.e scrub ok 2026-02-23T13:48:36.263 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2110 8 365281 365273 1.35236 0 - 0.0299513 2026-02-23T13:48:36.528 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:48:36.647 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:48:36.746 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:48:36.744+0000 7fb162bad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:48:36.746 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:48:36.744+0000 7fb162bad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:48:36.759 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:48:36.759 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:48:36.769 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:48:36.769 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:48:36.769 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:48:36.780 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:48:36.781 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:48:36.811 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:48:36 trial068 ceph-mon[38805]: pgmap v1473: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:48:36.812 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:48:36.812 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:48:37.263 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2111 8 365281 365273 1.35172 0 - 0.0299513 2026-02-23T13:48:38.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:48:37 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1301224442' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:48:38.263 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2112 8 365281 365273 1.35108 0 - 0.0299513 2026-02-23T13:48:38.813 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:48:38.933 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:48:38.974 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:48:38 trial068 ceph-mon[38805]: pgmap v1474: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:48:39.030 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:48:39.028+0000 7f59ecbb8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:48:39.030 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:48:39.028+0000 7f59ecbb8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:48:39.042 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:48:39.042 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:48:39.051 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:48:39.052 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:48:39.052 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:48:39.063 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:48:39.063 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:48:39.092 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:48:39.093 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:48:39.263 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2113 8 365281 365273 1.35044 0 - 0.0299513 2026-02-23T13:48:40.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:48:39 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/474561826' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:48:40.263 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2114 8 365281 365273 1.3498 0 - 0.0299513 2026-02-23T13:48:41.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:48:40 trial068 ceph-mon[38805]: pgmap v1475: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:48:41.094 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:48:41.213 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:48:41.264 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2115 8 365281 365273 1.34916 0 - 0.0299513 2026-02-23T13:48:41.309 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:48:41.308+0000 7f7e31196640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:48:41.310 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:48:41.308+0000 7f7e31196640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:48:41.321 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:48:41.322 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:48:41.331 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:48:41.331 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:48:41.332 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:48:41.342 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:48:41.342 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:48:41.372 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:48:41.373 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:48:42.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:48:41 trial068 ceph-mon[38805]: 5.a scrub starts 2026-02-23T13:48:42.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:48:41 trial068 ceph-mon[38805]: 5.a scrub ok 2026-02-23T13:48:42.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:48:41 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/282917533' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:48:42.264 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2116 8 365281 365273 1.34852 0 - 0.0299513 2026-02-23T13:48:43.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:48:42 trial068 ceph-mon[38805]: pgmap v1476: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:48:43.264 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2117 8 365281 365273 1.34788 0 - 0.0299513 2026-02-23T13:48:43.374 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:48:43.492 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:48:43.590 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:48:43.588+0000 7fe821b03640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:48:43.591 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:48:43.588+0000 7fe821b03640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:48:43.602 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:48:43.602 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:48:43.612 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:48:43.612 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:48:43.613 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:48:43.623 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:48:43.623 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:48:43.654 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:48:43.654 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:48:43.974 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:48:43 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1578479820' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:48:44.264 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2118 8 365281 365273 1.34725 0 - 0.0299513 2026-02-23T13:48:44.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:48:43 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:48:43] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:48:44.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:48:43 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:48:43.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:48:45.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:48:44 trial068 ceph-mon[38805]: pgmap v1477: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:48:45.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:48:44 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2026-02-23T13:48:45.264 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2119 8 365281 365273 1.34661 0 - 0.0299513 2026-02-23T13:48:45.655 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:48:45.775 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:48:45.876 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:48:45.874+0000 7f155e69f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:48:45.876 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:48:45.874+0000 7f155e69f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:48:45.888 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:48:45.889 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:48:45.899 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:48:45.900 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:48:45.900 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:48:45.911 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:48:45.911 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:48:45.941 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:48:45 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-23T13:48:45.941 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:48:45 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2026-02-23T13:48:45.941 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:48:45 trial068 ceph-mon[38805]: pgmap v1478: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:48:45.941 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:48:45 trial068 ceph-mon[38805]: pgmap v1479: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:48:45.941 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:48:45 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T13:48:45.941 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:48:45 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2026-02-23T13:48:45.941 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:48:45 trial068 ceph-mon[38805]: 5.5 scrub starts 2026-02-23T13:48:45.942 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:48:45.943 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:48:46.264 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T13:48:46.263872+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T13:48:46.264 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T13:48:46.264 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2120 8 365281 365273 1.34598 0 - 0.0299513 2026-02-23T13:48:47.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:48:46 trial068 ceph-mon[38805]: 5.5 scrub ok 2026-02-23T13:48:47.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:48:46 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1935478287' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:48:47.264 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2121 8 365281 365273 1.34534 0 - 0.0299513 2026-02-23T13:48:47.944 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:48:47.978 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:48:47 trial068 ceph-mon[38805]: pgmap v1480: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:48:48.071 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:48:48.170 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:48:48.168+0000 7f30586f7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:48:48.170 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:48:48.168+0000 7f30586f7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:48:48.182 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:48:48.182 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:48:48.192 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:48:48.192 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:48:48.192 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:48:48.202 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:48:48.202 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:48:48.233 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:48:48.234 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:48:48.264 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2122 8 365281 365273 1.34471 0 - 0.0299513 2026-02-23T13:48:49.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:48:48 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2175944026' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:48:49.264 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2123 8 365281 365273 1.34408 0 - 0.0299513 2026-02-23T13:48:50.235 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:48:50.264 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2124 8 365281 365273 1.34344 0 - 0.0299513 2026-02-23T13:48:50.351 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:48:50.453 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:48:50.451+0000 7efcec911640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:48:50.453 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:48:50.451+0000 7efcec911640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:48:50.466 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:48:50.466 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:48:50.476 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:48:50.476 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:48:50.476 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:48:50.486 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:48:50.487 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:48:50.516 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:48:50 trial068 ceph-mon[38805]: pgmap v1481: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:48:50.516 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:48:50 trial068 ceph-mon[38805]: osdmap e208: 4 total, 3 up, 4 in 2026-02-23T13:48:50.517 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:48:50.518 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:48:51.264 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2125 8 365281 365273 1.34281 0 - 0.0299513 2026-02-23T13:48:51.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:48:51 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1384733761' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:48:52.264 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2126 8 365281 365273 1.34218 0 - 0.0299513 2026-02-23T13:48:52.519 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:48:52.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:48:52 trial068 ceph-mon[38805]: pgmap v1483: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:48:52.637 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:48:52.731 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:48:52.729+0000 7efc13b1a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:48:52.731 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:48:52.729+0000 7efc13b1a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:48:52.742 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:48:52.743 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:48:52.752 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:48:52.753 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:48:52.753 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:48:52.763 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:48:52.764 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:48:52.794 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:48:52.795 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:48:53.265 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2127 8 365281 365273 1.34155 0 - 0.0299513 2026-02-23T13:48:53.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:48:53 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2942308488' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:48:54.265 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2128 8 365281 365273 1.34092 0 - 0.0299513 2026-02-23T13:48:54.308 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:48:53 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:48:53] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:48:54.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:48:53 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:48:53.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:48:54.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:48:54 trial068 ceph-mon[38805]: pgmap v1484: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:48:54.796 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:48:54.912 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:48:55.007 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:48:55.004+0000 7f0065d86640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:48:55.007 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:48:55.005+0000 7f0065d86640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:48:55.018 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:48:55.018 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:48:55.029 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:48:55.029 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:48:55.029 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:48:55.039 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:48:55.039 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:48:55.067 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:48:55.068 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:48:55.265 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2129 8 365281 365273 1.34029 0 - 0.0299513 2026-02-23T13:48:55.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:48:55 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/942076161' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:48:56.265 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2130 8 365281 365273 1.33966 0 - 0.0299513 2026-02-23T13:48:56.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:48:56 trial068 ceph-mon[38805]: pgmap v1485: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:48:57.069 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:48:57.185 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:48:57.265 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2131 8 365281 365273 1.33903 0 - 0.0299513 2026-02-23T13:48:57.289 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:48:57.287+0000 7f1de6313640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:48:57.289 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:48:57.287+0000 7f1de6313640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:48:57.301 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:48:57.301 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:48:57.310 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:48:57.311 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:48:57.311 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:48:57.321 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:48:57.321 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:48:57.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:48:57 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3139121970' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:48:57.353 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:48:57.354 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:48:58.265 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2132 8 365281 365273 1.3384 0 - 0.0299513 2026-02-23T13:48:58.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:48:58 trial068 ceph-mon[38805]: pgmap v1486: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:48:59.265 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2133 8 365281 365273 1.33777 0 - 0.0299513 2026-02-23T13:48:59.355 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:48:59.474 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:48:59.569 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:48:59.568+0000 7f80c49c4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:48:59.570 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:48:59.568+0000 7f80c49c4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:48:59.582 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:48:59.582 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:48:59.593 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:48:59.593 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:48:59.593 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:48:59.603 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:48:59.603 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:48:59.635 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:48:59.635 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:49:00.265 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2134 8 365281 365273 1.33715 0 - 0.0299513 2026-02-23T13:49:00.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:49:00 trial068 ceph-mon[38805]: pgmap v1487: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:49:00.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:49:00 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/954877587' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:49:01.265 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2135 8 365281 365273 1.33652 0 - 0.0299513 2026-02-23T13:49:01.636 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:49:01.753 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:49:01.850 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:49:01.848+0000 7f45bbd96640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:49:01.850 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:49:01.848+0000 7f45bbd96640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:49:01.863 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:49:01.863 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:49:01.873 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:49:01.873 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:49:01.873 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:49:01.883 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:49:01.883 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:49:01.913 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:49:01.914 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:49:02.265 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2136 8 365281 365273 1.3359 0 - 0.0299513 2026-02-23T13:49:02.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:49:02 trial068 ceph-mon[38805]: pgmap v1488: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:49:02.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:49:02 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3041879972' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:49:03.265 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2137 8 365281 365273 1.33527 0 - 0.0299513 2026-02-23T13:49:03.915 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:49:04.034 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:49:04.133 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:49:04.132+0000 7f9da4340640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:49:04.134 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:49:04.132+0000 7f9da4340640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:49:04.145 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:49:04.145 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:49:04.155 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:49:04.156 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:49:04.156 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:49:04.165 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:49:04.166 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:49:04.195 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:49:03 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:49:03] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:49:04.196 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:49:04.196 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:49:04.265 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:49:03 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:49:03.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:49:04.265 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2138 8 365281 365273 1.33465 0 - 0.0299513 2026-02-23T13:49:04.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:49:04 trial068 ceph-mon[38805]: pgmap v1489: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:49:04.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:49:04 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1655816862' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:49:05.265 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2139 8 365281 365273 1.33402 0 - 0.0299513 2026-02-23T13:49:06.197 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:49:06.265 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T13:49:06.265359+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T13:49:06.266 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T13:49:06.266 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2140 8 365281 365273 1.3334 0 - 0.0299513 2026-02-23T13:49:06.313 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:49:06.410 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:49:06.409+0000 7f336aeed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:49:06.410 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:49:06.409+0000 7f336aeed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:49:06.422 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:49:06.422 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:49:06.432 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:49:06.432 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:49:06.433 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:49:06.443 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:49:06.443 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:49:06.472 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:49:06 trial068 ceph-mon[38805]: pgmap v1490: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:49:06.473 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:49:06.473 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:49:07.265 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2141 8 365281 365273 1.33278 0 - 0.0299513 2026-02-23T13:49:07.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:49:07 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2701635148' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:49:08.265 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2142 8 365281 365273 1.33215 0 - 0.0299513 2026-02-23T13:49:08.474 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:49:08.593 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:49:08.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:49:08 trial068 ceph-mon[38805]: pgmap v1491: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:49:08.690 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:49:08.688+0000 7f8451a92640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:49:08.690 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:49:08.689+0000 7f8451a92640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:49:08.703 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:49:08.704 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:49:08.713 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:49:08.714 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:49:08.714 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:49:08.724 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:49:08.725 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:49:08.756 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:49:08.756 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:49:09.266 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2143 8 365281 365273 1.33153 0 - 0.0299513 2026-02-23T13:49:09.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:49:09 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2386542863' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:49:10.266 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2144 8 365281 365273 1.33091 0 - 0.0299513 2026-02-23T13:49:10.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:49:10 trial068 ceph-mon[38805]: pgmap v1492: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:49:10.758 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:49:10.875 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:49:10.979 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:49:10.977+0000 7f1e65765640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:49:10.979 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:49:10.977+0000 7f1e65765640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:49:10.991 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:49:10.991 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:49:11.001 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:49:11.001 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:49:11.001 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:49:11.011 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:49:11.011 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:49:11.040 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:49:11.040 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:49:11.266 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2145 8 365281 365273 1.33029 0 - 0.0299513 2026-02-23T13:49:11.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:49:11 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1000092057' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:49:12.266 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2146 8 365281 365273 1.32967 0 - 0.0299513 2026-02-23T13:49:12.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:49:12 trial068 ceph-mon[38805]: pgmap v1493: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:49:13.042 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:49:13.159 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:49:13.260 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:49:13.258+0000 7f2e1733c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:49:13.261 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:49:13.259+0000 7f2e1733c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:49:13.266 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2147 8 365281 365273 1.32905 0 - 0.0299513 2026-02-23T13:49:13.272 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:49:13.273 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:49:13.282 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:49:13.283 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:49:13.283 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:49:13.293 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:49:13.293 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:49:13.323 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:49:13.324 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:49:13.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:49:13 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2098379347' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:49:14.267 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2148 8 365281 365273 1.32843 0 - 0.0299513 2026-02-23T13:49:14.330 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:49:13 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:49:13] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:49:14.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:49:13 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:49:13.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:49:14.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:49:14 trial068 ceph-mon[38805]: pgmap v1494: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:49:15.267 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2149 8 365281 365273 1.32781 0 - 0.0299513 2026-02-23T13:49:15.325 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:49:15.444 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:49:15.546 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:49:15.544+0000 7f7a639ce640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:49:15.546 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:49:15.544+0000 7f7a639ce640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:49:15.558 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:49:15.558 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:49:15.568 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:49:15.568 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:49:15.568 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:49:15.578 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:49:15.579 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:49:15.607 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:49:15.608 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:49:16.267 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2150 8 365281 365273 1.3272 0 - 0.0299513 2026-02-23T13:49:16.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:49:16 trial068 ceph-mon[38805]: pgmap v1495: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:49:16.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:49:16 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1803692387' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:49:16.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:49:16 trial068 ceph-mon[38805]: 5.8 scrub starts 2026-02-23T13:49:17.267 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2151 8 365281 365273 1.32658 0 - 0.0299513 2026-02-23T13:49:17.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:49:17 trial068 ceph-mon[38805]: 5.8 scrub ok 2026-02-23T13:49:17.609 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:49:17.724 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:49:17.822 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:49:17.821+0000 7eff3e59b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:49:17.823 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:49:17.821+0000 7eff3e59b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:49:17.834 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:49:17.835 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:49:17.845 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:49:17.846 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:49:17.846 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:49:17.856 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:49:17.856 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:49:17.886 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:49:17.887 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:49:18.267 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2152 8 365281 365273 1.32596 0 - 0.0299513 2026-02-23T13:49:18.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:49:18 trial068 ceph-mon[38805]: pgmap v1496: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:49:18.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:49:18 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3168978675' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:49:19.267 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2153 8 365281 365273 1.32535 0 - 0.0299513 2026-02-23T13:49:19.888 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:49:20.004 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:49:20.105 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:49:20.103+0000 7ff438596640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:49:20.105 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:49:20.104+0000 7ff438596640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:49:20.117 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:49:20.118 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:49:20.127 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:49:20.128 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:49:20.128 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:49:20.138 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:49:20.138 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:49:20.171 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:49:20.172 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:49:20.267 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2154 8 365281 365273 1.32473 0 - 0.0299513 2026-02-23T13:49:20.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:49:20 trial068 ceph-mon[38805]: pgmap v1497: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:49:20.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:49:20 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3377596205' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:49:21.267 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2155 8 365281 365273 1.32412 0 - 0.0299513 2026-02-23T13:49:21.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:49:21 trial068 ceph-mon[38805]: 5.7 scrub starts 2026-02-23T13:49:21.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:49:21 trial068 ceph-mon[38805]: 5.7 scrub ok 2026-02-23T13:49:21.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:49:21 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T13:49:21.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:49:21 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T13:49:22.173 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:49:22.267 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2156 8 365281 365273 1.3235 0 - 0.0299513 2026-02-23T13:49:22.288 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:49:22.388 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:49:22.386+0000 7f751202c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:49:22.388 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:49:22.387+0000 7f751202c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:49:22.400 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:49:22.401 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:49:22.410 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:49:22.411 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:49:22.411 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:49:22.422 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:49:22.422 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:49:22.449 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:49:22 trial068 ceph-mon[38805]: pgmap v1498: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:49:22.450 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:49:22.450 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:49:23.267 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2157 8 365281 365273 1.32289 0 - 0.0299513 2026-02-23T13:49:23.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:49:23 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/4023600135' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:49:24.267 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2158 8 365281 365273 1.32228 0 - 0.0299513 2026-02-23T13:49:24.341 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:49:23 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:49:23] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:49:24.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:49:23 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:49:23.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:49:24.452 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:49:24.570 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:49:24.592 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:49:24 trial068 ceph-mon[38805]: pgmap v1499: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:49:24.669 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:49:24.667+0000 7fedc6065640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:49:24.669 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:49:24.667+0000 7fedc6065640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:49:24.680 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:49:24.681 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:49:24.691 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:49:24.691 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:49:24.691 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:49:24.701 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:49:24.702 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:49:24.731 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:49:24.732 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:49:25.267 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2159 8 365281 365273 1.32166 0 - 0.0299513 2026-02-23T13:49:25.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:49:25 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T13:49:25.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:49:25 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T13:49:25.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:49:25 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2080196730' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:49:26.267 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T13:49:26.267328+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T13:49:26.268 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T13:49:26.268 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2160 8 365281 365273 1.32105 0 - 0.0299513 2026-02-23T13:49:26.733 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:49:26.759 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:49:26 trial068 ceph-mon[38805]: pgmap v1500: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:49:26.852 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:49:26.947 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:49:26.945+0000 7fdc237ee640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:49:26.947 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:49:26.945+0000 7fdc237ee640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:49:26.959 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:49:26.959 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:49:26.969 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:49:26.970 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:49:26.970 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:49:26.980 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:49:26.981 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:49:27.011 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:49:27.012 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:49:27.268 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2161 8 365281 365273 1.32044 0 - 0.0299513 2026-02-23T13:49:27.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:49:27 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2185581956' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:49:28.268 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2162 8 365281 365273 1.31983 0 - 0.0299513 2026-02-23T13:49:28.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:49:28 trial068 ceph-mon[38805]: pgmap v1501: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:49:29.012 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:49:29.130 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:49:29.234 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:49:29.232+0000 7f4460c8f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:49:29.234 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:49:29.232+0000 7f4460c8f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:49:29.246 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:49:29.246 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:49:29.256 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:49:29.257 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:49:29.257 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:49:29.266 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:49:29.267 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:49:29.267 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2163 8 365281 365273 1.31922 0 - 0.0299513 2026-02-23T13:49:29.296 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:49:29.297 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:49:29.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:49:29 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1656941760' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:49:30.268 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2164 8 365281 365273 1.31861 0 - 0.0299513 2026-02-23T13:49:30.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:49:30 trial068 ceph-mon[38805]: pgmap v1502: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:49:31.268 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2165 8 365281 365273 1.318 0 - 0.0299513 2026-02-23T13:49:31.298 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:49:31.417 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:49:31.517 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:49:31.515+0000 7fe7ed53c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:49:31.517 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:49:31.515+0000 7fe7ed53c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:49:31.529 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:49:31.529 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:49:31.539 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:49:31.540 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:49:31.540 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:49:31.550 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:49:31.550 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:49:31.581 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:49:31.581 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:49:32.268 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2166 8 365281 365273 1.31739 0 - 0.0299513 2026-02-23T13:49:32.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:49:32 trial068 ceph-mon[38805]: pgmap v1503: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:49:32.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:49:32 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1289498349' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:49:33.268 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2167 8 365281 365273 1.31678 0 - 0.0299513 2026-02-23T13:49:33.582 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:49:33.701 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:49:33.805 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:49:33.803+0000 7f7b5212a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:49:33.805 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:49:33.804+0000 7f7b5212a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:49:33.818 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:49:33.818 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:49:33.828 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:49:33.828 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:49:33.829 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:49:33.838 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:49:33.839 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:49:33.870 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:49:33.870 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:49:34.265 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:49:33 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:49:33.974Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:49:34.268 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2168 8 365281 365273 1.31618 0 - 0.0299513 2026-02-23T13:49:34.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:49:33 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:49:33] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:49:34.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:49:34 trial068 ceph-mon[38805]: pgmap v1504: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:49:34.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:49:34 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3189286534' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:49:35.268 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2169 8 365281 365273 1.31557 0 - 0.0299513 2026-02-23T13:49:35.871 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:49:35.991 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:49:36.091 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:49:36.089+0000 7f472e9a0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:49:36.091 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:49:36.089+0000 7f472e9a0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:49:36.103 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:49:36.103 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:49:36.114 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:49:36.114 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:49:36.114 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:49:36.124 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:49:36.124 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:49:36.153 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:49:36.153 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:49:36.268 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2170 8 365281 365273 1.31496 0 - 0.0299513 2026-02-23T13:49:36.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:49:36 trial068 ceph-mon[38805]: pgmap v1505: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:49:36.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:49:36 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1772995290' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:49:37.268 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2171 8 365281 365273 1.31436 0 - 0.0299513 2026-02-23T13:49:38.154 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:49:38.268 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2172 8 365281 365273 1.31375 0 - 0.0299513 2026-02-23T13:49:38.270 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:49:38.364 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:49:38.362+0000 7fcf14ca4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:49:38.364 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:49:38.362+0000 7fcf14ca4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:49:38.375 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:49:38.376 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:49:38.386 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:49:38.386 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:49:38.386 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:49:38.396 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:49:38.396 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:49:38.426 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:49:38.426 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:49:38.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:49:38 trial068 ceph-mon[38805]: pgmap v1506: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:49:39.268 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2173 8 365281 365273 1.31315 0 - 0.0299513 2026-02-23T13:49:39.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:49:39 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1780717137' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:49:40.268 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2174 8 365281 365273 1.31254 0 - 0.0299513 2026-02-23T13:49:40.427 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:49:40.547 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:49:40.645 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:49:40.643+0000 7f4b2bed3640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:49:40.646 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:49:40 trial068 ceph-mon[38805]: pgmap v1507: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:49:40.646 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:49:40.644+0000 7f4b2bed3640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:49:40.657 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:49:40.657 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:49:40.667 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:49:40.668 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:49:40.668 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:49:40.677 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:49:40.678 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:49:40.707 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:49:40.708 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:49:41.269 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2175 8 365281 365273 1.31194 0 - 0.0299513 2026-02-23T13:49:41.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:49:41 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3318760928' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:49:42.269 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2176 8 365281 365273 1.31134 0 - 0.0299513 2026-02-23T13:49:42.709 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:49:42.733 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:49:42 trial068 ceph-mon[38805]: pgmap v1508: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:49:42.825 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:49:42.923 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:49:42.922+0000 7fb1ee21a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:49:42.923 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:49:42.922+0000 7fb1ee21a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:49:42.935 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:49:42.936 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:49:42.946 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:49:42.946 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:49:42.946 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:49:42.956 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:49:42.956 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:49:42.986 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:49:42.987 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:49:43.269 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2177 8 365281 365273 1.31074 0 - 0.0299513 2026-02-23T13:49:43.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:49:43 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3388822725' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:49:44.269 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2178 8 365281 365273 1.31013 0 - 0.0299513 2026-02-23T13:49:44.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:49:43 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:49:43] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:49:44.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:49:43 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:49:43.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:49:44.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:49:44 trial068 ceph-mon[38805]: pgmap v1509: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:49:44.987 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:49:45.103 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:49:45.212 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:49:45.210+0000 7fad81e52640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:49:45.212 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:49:45.210+0000 7fad81e52640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:49:45.224 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:49:45.224 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:49:45.234 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:49:45.234 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:49:45.235 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:49:45.245 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:49:45.245 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:49:45.269 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2179 8 365281 365273 1.30953 0 - 0.0299513 2026-02-23T13:49:45.276 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:49:45.292 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:49:45.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:49:45 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2026-02-23T13:49:45.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:49:45 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-23T13:49:45.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:49:45 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2026-02-23T13:49:45.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:49:45 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T13:49:45.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:49:45 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2026-02-23T13:49:45.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:49:45 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/668840796' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:49:46.269 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T13:49:46.268868+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T13:49:46.269 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T13:49:46.269 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2180 8 365281 365273 1.30893 0 - 0.0299513 2026-02-23T13:49:46.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:49:46 trial068 ceph-mon[38805]: pgmap v1510: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:49:46.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:49:46 trial068 ceph-mon[38805]: pgmap v1511: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:49:46.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:49:46 trial068 ceph-mon[38805]: pgmap v1512: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:49:47.269 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2181 8 365281 365273 1.30833 0 - 0.0299513 2026-02-23T13:49:47.293 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:49:47.411 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:49:47.509 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:49:47.507+0000 7f20fa531640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:49:47.509 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:49:47.507+0000 7f20fa531640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:49:47.521 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:49:47.521 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:49:47.531 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:49:47.531 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:49:47.531 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:49:47.542 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:49:47.542 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:49:47.570 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:49:47.570 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:49:48.269 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2182 8 365281 365273 1.30773 0 - 0.0299513 2026-02-23T13:49:48.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:49:48 trial068 ceph-mon[38805]: pgmap v1513: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:49:48.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:49:48 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3729035966' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:49:49.269 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2183 8 365281 365273 1.30713 0 - 0.0299513 2026-02-23T13:49:49.571 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:49:49.689 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:49:49.710 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:49:49 trial068 ceph-mon[38805]: 5.5 scrub starts 2026-02-23T13:49:49.785 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:49:49.783+0000 7f6f5131e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:49:49.785 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:49:49.783+0000 7f6f5131e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:49:49.796 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:49:49.797 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:49:49.806 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:49:49.807 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:49:49.807 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:49:49.817 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:49:49.817 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:49:49.847 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:49:49.848 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:49:50.269 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2184 8 365281 365273 1.30653 0 - 0.0299513 2026-02-23T13:49:50.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:49:50 trial068 ceph-mon[38805]: 5.5 scrub ok 2026-02-23T13:49:50.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:49:50 trial068 ceph-mon[38805]: pgmap v1514: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:49:50.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:49:50 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3533222837' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:49:51.269 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2185 8 365281 365273 1.30594 0 - 0.0299513 2026-02-23T13:49:51.849 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:49:51.966 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:49:52.065 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:49:52.062+0000 7fdf11bc7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:49:52.065 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:49:52.063+0000 7fdf11bc7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:49:52.076 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:49:52.077 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:49:52.087 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:49:52.087 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:49:52.087 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:49:52.097 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:49:52.098 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:49:52.126 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:49:52.127 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:49:52.269 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2186 8 365281 365273 1.30534 0 - 0.0299513 2026-02-23T13:49:52.375 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:49:52 trial068 ceph-mon[38805]: pgmap v1515: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:49:52.376 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:49:52 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1646385711' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:49:53.270 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2187 8 365281 365273 1.30474 0 - 0.0299513 2026-02-23T13:49:54.128 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:49:54.247 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:49:54.270 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2188 8 365281 365273 1.30415 0 - 0.0299513 2026-02-23T13:49:54.270 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:49:53 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:49:53] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:49:54.348 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:49:54.347+0000 7f6ac58f9640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:49:54.349 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:49:54.347+0000 7f6ac58f9640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:49:54.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:49:53 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:49:53.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:49:54.361 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:49:54.361 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:49:54.371 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:49:54.371 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:49:54.372 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:49:54.382 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:49:54.382 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:49:54.412 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:49:54.412 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:49:54.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:49:54 trial068 ceph-mon[38805]: pgmap v1516: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:49:54.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:49:54 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2795121183' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:49:55.270 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2189 8 365281 365273 1.30355 0 - 0.0299513 2026-02-23T13:49:56.270 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2190 8 365281 365273 1.30296 0 - 0.0299513 2026-02-23T13:49:56.414 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:49:56.531 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:49:56.629 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:49:56.627+0000 7f160711b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:49:56.629 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:49:56.627+0000 7f160711b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:49:56.630 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:49:56 trial068 ceph-mon[38805]: pgmap v1517: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:49:56.641 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:49:56.641 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:49:56.651 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:49:56.651 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:49:56.651 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:49:56.661 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:49:56.661 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:49:56.692 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:49:56.693 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:49:57.270 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2191 8 365281 365273 1.30236 0 - 0.0299513 2026-02-23T13:49:57.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:49:57 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1038790132' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:49:58.270 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2192 8 365281 365273 1.30177 0 - 0.0299513 2026-02-23T13:49:58.693 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:49:58.719 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:49:58 trial068 ceph-mon[38805]: pgmap v1518: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:49:58.813 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:49:58.909 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:49:58.907+0000 7f70cf5ff640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:49:58.909 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:49:58.907+0000 7f70cf5ff640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:49:58.920 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:49:58.921 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:49:58.930 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:49:58.931 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:49:58.931 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:49:58.940 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:49:58.941 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:49:58.969 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:49:58.970 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:49:59.270 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2193 8 365281 365273 1.30117 0 - 0.0299513 2026-02-23T13:49:59.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:49:59 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2970698906' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:50:00.270 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2194 8 365281 365273 1.30058 0 - 0.0299513 2026-02-23T13:50:00.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:50:00 trial068 ceph-mon[38805]: pgmap v1519: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:50:00.971 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:50:01.090 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:50:01.195 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:50:01.193+0000 7f06ae1ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:50:01.195 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:50:01.193+0000 7f06ae1ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:50:01.208 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:50:01.208 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:50:01.218 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:50:01.219 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:50:01.219 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:50:01.229 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:50:01.229 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:50:01.259 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:50:01.259 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:50:01.270 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2195 8 365281 365273 1.29999 0 - 0.0299513 2026-02-23T13:50:01.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:50:01 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T13:50:01.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:50:01 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T13:50:01.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:50:01 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3048370193' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:50:02.270 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2196 8 365281 365273 1.2994 0 - 0.0299513 2026-02-23T13:50:02.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:50:02 trial068 ceph-mon[38805]: 5.e scrub starts 2026-02-23T13:50:02.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:50:02 trial068 ceph-mon[38805]: pgmap v1520: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:50:02.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:50:02 trial068 ceph-mon[38805]: 5.e scrub ok 2026-02-23T13:50:03.260 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:50:03.270 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2197 8 365281 365273 1.2988 0 - 0.0299513 2026-02-23T13:50:03.378 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:50:03.479 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:50:03.477+0000 7f832073a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:50:03.479 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:50:03.477+0000 7f832073a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:50:03.491 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:50:03.491 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:50:03.502 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:50:03.502 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:50:03.502 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:50:03.513 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:50:03.513 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:50:03.543 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:50:03.543 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:50:04.266 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:50:03 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:50:03.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:50:04.270 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2198 8 365281 365273 1.29821 0 - 0.0299513 2026-02-23T13:50:04.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:50:03 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:50:03] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:50:04.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:50:04 trial068 ceph-mon[38805]: pgmap v1521: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:50:04.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:50:04 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2699769990' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:50:05.270 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2199 8 365281 365273 1.29762 0 - 0.0299513 2026-02-23T13:50:05.544 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:50:05.662 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:50:05.759 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:50:05.757+0000 7f525cd9b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:50:05.759 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:50:05.757+0000 7f525cd9b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:50:05.770 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:50:05.771 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:50:05.780 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:50:05.781 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:50:05.781 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:50:05.791 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:50:05.791 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:50:05.821 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:50:05.821 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:50:06.271 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T13:50:06.270319+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T13:50:06.271 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T13:50:06.271 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2200 8 365281 365273 1.29703 0 - 0.0299513 2026-02-23T13:50:06.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:50:06 trial068 ceph-mon[38805]: pgmap v1522: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:50:06.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:50:06 trial068 ceph-mon[38805]: osdmap e209: 4 total, 3 up, 4 in 2026-02-23T13:50:06.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:50:06 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1666373464' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:50:07.270 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2201 8 365281 365273 1.29644 0 - 0.0299513 2026-02-23T13:50:07.822 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:50:07.939 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:50:08.048 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:50:08.046+0000 7efde2253640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:50:08.048 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:50:08.046+0000 7efde2253640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:50:08.059 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:50:08.060 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:50:08.070 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:50:08.070 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:50:08.070 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:50:08.080 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:50:08.080 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:50:08.111 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:50:08.111 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:50:08.271 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2202 8 365281 365273 1.29585 0 - 0.0299513 2026-02-23T13:50:08.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:50:08 trial068 ceph-mon[38805]: pgmap v1524: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:50:08.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:50:08 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/203754317' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:50:09.271 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2203 8 365281 365273 1.29527 0 - 0.0299513 2026-02-23T13:50:10.113 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:50:10.230 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:50:10.271 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2204 8 365281 365273 1.29468 0 - 0.0299513 2026-02-23T13:50:10.335 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:50:10.333+0000 7fd6105bb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:50:10.335 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:50:10.334+0000 7fd6105bb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:50:10.347 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:50:10.348 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:50:10.358 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:50:10.358 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:50:10.358 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:50:10.368 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:50:10.368 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:50:10.399 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:50:10.400 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:50:10.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:50:10 trial068 ceph-mon[38805]: pgmap v1525: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:50:10.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:50:10 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/399379562' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:50:11.271 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2205 8 365281 365273 1.29409 0 - 0.0299513 2026-02-23T13:50:11.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:50:11 trial068 ceph-mon[38805]: 5.a scrub starts 2026-02-23T13:50:12.271 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2206 8 365281 365273 1.2935 0 - 0.0299513 2026-02-23T13:50:12.401 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:50:12.518 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:50:12.618 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:50:12.616+0000 7f175e08c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:50:12.618 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:50:12.616+0000 7f175e08c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:50:12.630 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:50:12.630 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:50:12.641 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:50:12.641 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:50:12.641 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:50:12.652 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:50:12.652 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:50:12.652 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:50:12 trial068 ceph-mon[38805]: 5.a scrub ok 2026-02-23T13:50:12.653 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:50:12 trial068 ceph-mon[38805]: pgmap v1526: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:50:12.681 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:50:12.682 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:50:13.271 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2207 8 365281 365273 1.29292 0 - 0.0299513 2026-02-23T13:50:13.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:50:13 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1764954770' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:50:14.271 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2208 8 365281 365273 1.29233 0 - 0.0299513 2026-02-23T13:50:14.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:50:13 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:50:13] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:50:14.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:50:13 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:50:13.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:50:14.682 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:50:14.705 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:50:14 trial068 ceph-mon[38805]: pgmap v1527: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:50:14.799 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:50:14.899 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:50:14.897+0000 7f8e4d730640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:50:14.900 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:50:14.898+0000 7f8e4d730640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:50:14.911 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:50:14.911 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:50:14.921 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:50:14.922 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:50:14.922 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:50:14.932 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:50:14.932 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:50:14.962 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:50:14.963 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:50:15.271 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2209 8 365281 365273 1.29175 0 - 0.0299513 2026-02-23T13:50:15.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:50:15 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/748951984' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:50:16.271 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2210 8 365281 365273 1.29116 0 - 0.0299513 2026-02-23T13:50:16.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:50:16 trial068 ceph-mon[38805]: pgmap v1528: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:50:16.963 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:50:17.082 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:50:17.181 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:50:17.180+0000 7f3dbd103640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:50:17.182 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:50:17.180+0000 7f3dbd103640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:50:17.194 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:50:17.194 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:50:17.205 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:50:17.206 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:50:17.206 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:50:17.217 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:50:17.217 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:50:17.248 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:50:17.248 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:50:17.271 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2211 8 365281 365273 1.29058 0 - 0.0299513 2026-02-23T13:50:17.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:50:17 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/614566800' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:50:18.271 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2212 8 365281 365273 1.29 0 - 0.0299513 2026-02-23T13:50:18.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:50:18 trial068 ceph-mon[38805]: pgmap v1529: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:50:19.250 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:50:19.271 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2213 8 365281 365273 1.28941 0 - 0.0299513 2026-02-23T13:50:19.370 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:50:19.474 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:50:19.472+0000 7f328add2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:50:19.474 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:50:19.472+0000 7f328add2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:50:19.486 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:50:19.486 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:50:19.496 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:50:19.496 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:50:19.496 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:50:19.505 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:50:19.506 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:50:19.535 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:50:19.536 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:50:20.272 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2214 8 365281 365273 1.28883 0 - 0.0299513 2026-02-23T13:50:20.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:50:20 trial068 ceph-mon[38805]: pgmap v1530: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:50:20.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:50:20 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2530555488' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:50:21.272 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2215 8 365281 365273 1.28825 0 - 0.0299513 2026-02-23T13:50:21.536 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:50:21.655 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:50:21.755 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:50:21.753+0000 7f4a320c0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:50:21.755 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:50:21.753+0000 7f4a320c0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:50:21.767 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:50:21.767 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:50:21.777 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:50:21.778 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:50:21.778 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:50:21.787 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:50:21.788 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:50:21.817 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:50:21.818 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:50:22.272 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2216 8 365281 365273 1.28767 0 - 0.0299513 2026-02-23T13:50:22.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:50:22 trial068 ceph-mon[38805]: pgmap v1531: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:50:22.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:50:22 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1489661597' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:50:23.272 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2217 8 365281 365273 1.28709 0 - 0.0299513 2026-02-23T13:50:23.819 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:50:23.938 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:50:24.037 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:50:24.035+0000 7f16d7283640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:50:24.037 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:50:24.036+0000 7f16d7283640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:50:24.050 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:50:24.050 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:50:24.060 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:50:24.060 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:50:24.060 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:50:24.070 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:50:24.071 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:50:24.094 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:50:23 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:50:23] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:50:24.100 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:50:24.101 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:50:24.272 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2218 8 365281 365273 1.28651 0 - 0.0299513 2026-02-23T13:50:24.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:50:23 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:50:23.972Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:50:24.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:50:24 trial068 ceph-mon[38805]: pgmap v1532: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:50:24.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:50:24 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3507548637' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:50:25.272 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2219 8 365281 365273 1.28593 0 - 0.0299513 2026-02-23T13:50:26.102 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:50:26.218 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:50:26.272 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T13:50:26.271763+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T13:50:26.272 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T13:50:26.272 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2220 8 365281 365273 1.28535 0 - 0.0299513 2026-02-23T13:50:26.314 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:50:26.312+0000 7fdd69b70640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:50:26.314 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:50:26.312+0000 7fdd69b70640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:50:26.326 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:50:26.326 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:50:26.336 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:50:26.336 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:50:26.337 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:50:26.346 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:50:26.346 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:50:26.378 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:50:26.378 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:50:26.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:50:26 trial068 ceph-mon[38805]: pgmap v1533: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:50:26.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:50:26 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3766433468' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:50:27.272 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2221 8 365281 365273 1.28477 0 - 0.0299513 2026-02-23T13:50:28.272 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2222 8 365281 365273 1.28419 0 - 0.0299513 2026-02-23T13:50:28.379 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:50:28.497 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:50:28.599 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:50:28.597+0000 7f7595f80640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:50:28.599 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:50:28.597+0000 7f7595f80640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:50:28.611 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:50:28.612 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:50:28.622 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:50:28.622 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:50:28.622 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:50:28.632 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:50:28.632 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:50:28.653 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:50:28 trial068 ceph-mon[38805]: pgmap v1534: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:50:28.661 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:50:28.662 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:50:29.272 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2223 8 365281 365273 1.28361 0 - 0.0299513 2026-02-23T13:50:29.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:50:29 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2216756669' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:50:29.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:50:29 trial068 ceph-mon[38805]: 5.7 scrub starts 2026-02-23T13:50:30.272 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2224 8 365281 365273 1.28304 0 - 0.0299513 2026-02-23T13:50:30.663 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:50:30.685 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:50:30 trial068 ceph-mon[38805]: 5.7 scrub ok 2026-02-23T13:50:30.685 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:50:30 trial068 ceph-mon[38805]: pgmap v1535: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:50:30.777 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:50:30.876 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:50:30.874+0000 7f0447aa5640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:50:30.876 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:50:30.874+0000 7f0447aa5640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:50:30.888 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:50:30.888 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:50:30.898 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:50:30.899 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:50:30.899 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:50:30.909 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:50:30.909 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:50:30.940 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:50:30.940 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:50:31.272 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2225 8 365281 365273 1.28246 0 - 0.0299513 2026-02-23T13:50:31.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:50:31 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/776788519' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:50:32.273 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2226 8 365281 365273 1.28188 0 - 0.0299513 2026-02-23T13:50:32.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:50:32 trial068 ceph-mon[38805]: pgmap v1536: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:50:32.941 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:50:33.061 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:50:33.156 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:50:33.154+0000 7f9a26a37640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:50:33.156 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:50:33.154+0000 7f9a26a37640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:50:33.168 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:50:33.168 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:50:33.179 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:50:33.179 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:50:33.179 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:50:33.190 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:50:33.190 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:50:33.219 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:50:33.219 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:50:33.273 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2227 8 365281 365273 1.28131 0 - 0.0299513 2026-02-23T13:50:33.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:50:33 trial068 ceph-mon[38805]: 5.8 scrub starts 2026-02-23T13:50:33.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:50:33 trial068 ceph-mon[38805]: 5.8 scrub ok 2026-02-23T13:50:33.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:50:33 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2928998926' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:50:34.273 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2228 8 365281 365273 1.28073 0 - 0.0299513 2026-02-23T13:50:34.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:50:33 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:50:33] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:50:34.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:50:33 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:50:33.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:50:34.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:50:34 trial068 ceph-mon[38805]: pgmap v1537: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:50:35.221 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:50:35.273 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2229 8 365281 365273 1.28016 0 - 0.0299513 2026-02-23T13:50:35.339 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:50:35.435 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:50:35.433+0000 7fb1b941b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:50:35.435 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:50:35.433+0000 7fb1b941b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:50:35.447 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:50:35.448 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:50:35.458 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:50:35.459 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:50:35.459 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:50:35.469 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:50:35.470 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:50:35.500 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:50:35.500 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:50:36.273 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2230 8 365281 365273 1.27958 0 - 0.0299513 2026-02-23T13:50:36.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:50:36 trial068 ceph-mon[38805]: pgmap v1538: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:50:36.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:50:36 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3685754971' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:50:37.273 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2231 8 365281 365273 1.27901 0 - 0.0299513 2026-02-23T13:50:37.502 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:50:37.620 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:50:37.723 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:50:37.721+0000 7f460af2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:50:37.723 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:50:37.721+0000 7f460af2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:50:37.735 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:50:37.736 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:50:37.746 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:50:37.747 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:50:37.747 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:50:37.757 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:50:37.757 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:50:37.789 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:50:37.789 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:50:38.273 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2232 8 365281 365273 1.27844 0 - 0.0299513 2026-02-23T13:50:38.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:50:38 trial068 ceph-mon[38805]: pgmap v1539: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:50:38.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:50:38 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T13:50:38.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:50:38 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T13:50:38.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:50:38 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3878108445' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:50:39.273 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2233 8 365281 365273 1.27786 0 - 0.0299513 2026-02-23T13:50:39.790 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:50:39.910 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:50:40.013 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:50:40.011+0000 7f6950cf4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:50:40.013 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:50:40.011+0000 7f6950cf4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:50:40.025 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:50:40.025 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:50:40.035 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:50:40.036 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:50:40.036 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:50:40.047 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:50:40.047 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:50:40.076 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:50:40.077 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:50:40.273 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2234 8 365281 365273 1.27729 0 - 0.0299513 2026-02-23T13:50:40.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:50:40 trial068 ceph-mon[38805]: pgmap v1540: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:50:40.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:50:40 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1976801207' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:50:41.273 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2235 8 365281 365273 1.27672 0 - 0.0299513 2026-02-23T13:50:42.078 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:50:42.196 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:50:42.274 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2236 8 365281 365273 1.27615 0 - 0.0299513 2026-02-23T13:50:42.301 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:50:42.300+0000 7fad870a8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:50:42.302 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:50:42.300+0000 7fad870a8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:50:42.314 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:50:42.314 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:50:42.324 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:50:42.324 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:50:42.324 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:50:42.334 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:50:42.335 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:50:42.365 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:50:42.365 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:50:42.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:50:42 trial068 ceph-mon[38805]: pgmap v1541: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:50:42.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:50:42 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3314039271' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:50:43.274 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2237 8 365281 365273 1.27558 0 - 0.0299513 2026-02-23T13:50:44.274 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2238 8 365281 365273 1.27501 0 - 0.0299513 2026-02-23T13:50:44.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:50:43 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:50:43] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:50:44.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:50:43 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:50:43.974Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:50:44.366 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:50:44.486 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:50:44.583 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:50:44.581+0000 7f0cb64f6640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:50:44.583 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:50:44.581+0000 7f0cb64f6640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:50:44.594 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:50:44.595 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:50:44.604 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:50:44.605 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:50:44.605 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:50:44.615 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:50:44.615 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:50:44.644 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:50:44 trial068 ceph-mon[38805]: pgmap v1542: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:50:44.645 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:50:44.646 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:50:45.274 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2239 8 365281 365273 1.27444 0 - 0.0299513 2026-02-23T13:50:45.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:50:45 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3515304525' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:50:45.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:50:45 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T13:50:45.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:50:45 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T13:50:45.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:50:45 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2026-02-23T13:50:46.274 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T13:50:46.273626+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T13:50:46.274 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T13:50:46.274 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2240 8 365281 365273 1.27387 0 - 0.0299513 2026-02-23T13:50:46.647 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:50:46.767 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:50:46.790 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:50:46 trial068 ceph-mon[38805]: pgmap v1543: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:50:46.790 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:50:46 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-23T13:50:46.790 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:50:46 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2026-02-23T13:50:46.790 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:50:46 trial068 ceph-mon[38805]: pgmap v1544: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:50:46.790 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:50:46 trial068 ceph-mon[38805]: pgmap v1545: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:50:46.790 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:50:46 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T13:50:46.790 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:50:46 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2026-02-23T13:50:46.864 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:50:46.862+0000 7f0840ba2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:50:46.864 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:50:46.862+0000 7f0840ba2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:50:46.876 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:50:46.876 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:50:46.887 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:50:46.887 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:50:46.887 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:50:46.898 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:50:46.898 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:50:46.929 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:50:46.929 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:50:47.274 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2241 8 365281 365273 1.2733 0 - 0.0299513 2026-02-23T13:50:47.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:50:47 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/832176986' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:50:48.274 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2242 8 365281 365273 1.27273 0 - 0.0299513 2026-02-23T13:50:48.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:50:48 trial068 ceph-mon[38805]: pgmap v1546: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:50:48.930 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:50:49.048 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:50:49.145 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:50:49.143+0000 7f03f0cab640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:50:49.145 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:50:49.143+0000 7f03f0cab640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:50:49.157 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:50:49.158 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:50:49.167 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:50:49.168 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:50:49.168 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:50:49.178 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:50:49.178 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:50:49.208 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:50:49.209 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:50:49.274 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2243 8 365281 365273 1.27217 0 - 0.0299513 2026-02-23T13:50:49.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:50:49 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1523679736' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:50:50.274 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2244 8 365281 365273 1.2716 0 - 0.0299513 2026-02-23T13:50:50.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:50:50 trial068 ceph-mon[38805]: pgmap v1547: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:50:51.210 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:50:51.274 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2245 8 365281 365273 1.27103 0 - 0.0299513 2026-02-23T13:50:51.328 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:50:51.430 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:50:51.428+0000 7fd861a7e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:50:51.430 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:50:51.428+0000 7fd861a7e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:50:51.442 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:50:51.443 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:50:51.453 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:50:51.453 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:50:51.453 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:50:51.464 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:50:51.464 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:50:51.493 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:50:51 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3115485812' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:50:51.495 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:50:51.495 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:50:52.274 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2246 8 365281 365273 1.27047 0 - 0.0299513 2026-02-23T13:50:52.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:50:52 trial068 ceph-mon[38805]: pgmap v1548: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:50:53.274 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2247 8 365281 365273 1.2699 0 - 0.0299513 2026-02-23T13:50:53.496 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:50:53.614 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:50:53.717 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:50:53.715+0000 7ff623b80640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:50:53.717 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:50:53.715+0000 7ff623b80640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:50:53.729 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:50:53.730 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:50:53.739 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:50:53.740 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:50:53.740 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:50:53.749 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:50:53.750 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:50:53.779 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:50:53.779 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:50:54.093 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:50:53 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:50:53] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:50:54.274 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2248 8 365281 365273 1.26934 0 - 0.0299513 2026-02-23T13:50:54.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:50:53 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:50:53.974Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:50:54.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:50:54 trial068 ceph-mon[38805]: pgmap v1549: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:50:54.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:50:54 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2368804902' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:50:55.274 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2249 8 365281 365273 1.26877 0 - 0.0299513 2026-02-23T13:50:55.780 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:50:55.897 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:50:55.994 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:50:55.993+0000 7f81a29c9640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:50:55.995 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:50:55.993+0000 7f81a29c9640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:50:56.006 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:50:56.006 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:50:56.016 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:50:56.017 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:50:56.017 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:50:56.026 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:50:56.027 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:50:56.058 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:50:56.059 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:50:56.274 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2250 8 365281 365273 1.26821 0 - 0.0299513 2026-02-23T13:50:56.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:50:56 trial068 ceph-mon[38805]: pgmap v1550: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:50:56.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:50:56 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2319554943' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:50:57.274 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2251 8 365281 365273 1.26765 0 - 0.0299513 2026-02-23T13:50:58.060 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:50:58.176 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:50:58.274 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2252 8 365281 365273 1.26708 0 - 0.0299513 2026-02-23T13:50:58.281 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:50:58.278+0000 7f3e94ec7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:50:58.281 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:50:58.278+0000 7f3e94ec7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:50:58.293 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:50:58.293 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:50:58.303 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:50:58.304 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:50:58.304 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:50:58.314 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:50:58.314 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:50:58.343 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:50:58.343 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:50:58.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:50:58 trial068 ceph-mon[38805]: pgmap v1551: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:50:58.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:50:58 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1646130763' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:50:59.275 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2253 8 365281 365273 1.26652 0 - 0.0299513 2026-02-23T13:51:00.275 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2254 8 365281 365273 1.26596 0 - 0.0299513 2026-02-23T13:51:00.344 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:51:00.463 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:51:00.560 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:51:00.558+0000 7f8a83bbf640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:51:00.560 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:51:00.559+0000 7f8a83bbf640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:51:00.572 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:51:00.573 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:51:00.582 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:51:00.583 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:51:00.583 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:51:00.592 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:51:00.593 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:51:00.622 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:51:00 trial068 ceph-mon[38805]: pgmap v1552: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:51:00.623 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:51:00.623 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:51:01.275 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2255 8 365281 365273 1.2654 0 - 0.0299513 2026-02-23T13:51:01.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:51:01 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/950135850' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:51:02.275 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2256 8 365281 365273 1.26484 0 - 0.0299513 2026-02-23T13:51:02.624 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:51:02.741 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:51:02.763 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:51:02 trial068 ceph-mon[38805]: pgmap v1553: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:51:02.839 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:51:02.837+0000 7f81add38640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:51:02.840 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:51:02.838+0000 7f81add38640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:51:02.851 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:51:02.851 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:51:02.861 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:51:02.862 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:51:02.862 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:51:02.872 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:51:02.872 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:51:02.899 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:51:02.900 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:51:03.275 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2257 8 365281 365273 1.26428 0 - 0.0299513 2026-02-23T13:51:03.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:51:03 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3738350486' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:51:04.266 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:51:03 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:51:03.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:51:04.275 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2258 8 365281 365273 1.26372 0 - 0.0299513 2026-02-23T13:51:04.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:51:03 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:51:03] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:51:04.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:51:04 trial068 ceph-mon[38805]: pgmap v1554: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:51:04.900 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:51:05.018 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:51:05.125 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:51:05.123+0000 7fa298b41640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:51:05.125 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:51:05.123+0000 7fa298b41640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:51:05.138 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:51:05.138 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:51:05.148 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:51:05.148 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:51:05.148 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:51:05.159 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:51:05.160 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:51:05.190 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:51:05.191 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:51:05.275 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2259 8 365281 365273 1.26316 0 - 0.0299513 2026-02-23T13:51:05.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:51:05 trial068 ceph-mon[38805]: 5.5 scrub starts 2026-02-23T13:51:05.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:51:05 trial068 ceph-mon[38805]: 5.5 scrub ok 2026-02-23T13:51:05.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:51:05 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2511134095' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:51:06.275 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T13:51:06.275385+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T13:51:06.276 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T13:51:06.276 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2260 8 365281 365273 1.2626 0 - 0.0299513 2026-02-23T13:51:06.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:51:06 trial068 ceph-mon[38805]: pgmap v1555: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:51:07.192 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:51:07.276 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2261 8 365281 365273 1.26204 0 - 0.0299513 2026-02-23T13:51:07.308 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:51:07.407 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:51:07.405+0000 7f9ddd6d8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:51:07.407 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:51:07.405+0000 7f9ddd6d8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:51:07.419 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:51:07.419 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:51:07.429 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:51:07.429 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:51:07.429 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:51:07.439 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:51:07.440 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:51:07.473 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:51:07.473 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:51:07.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:51:07 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2739330565' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:51:08.276 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2262 8 365281 365273 1.26148 0 - 0.0299513 2026-02-23T13:51:08.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:51:08 trial068 ceph-mon[38805]: pgmap v1556: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:51:09.276 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2263 8 365281 365273 1.26092 0 - 0.0299513 2026-02-23T13:51:09.474 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:51:09.592 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:51:09.698 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:51:09.696+0000 7ff0ed3d3640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:51:09.698 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:51:09.696+0000 7ff0ed3d3640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:51:09.710 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:51:09.710 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:51:09.720 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:51:09.720 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:51:09.720 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:51:09.730 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:51:09.730 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:51:09.761 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:51:09.761 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:51:10.276 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2264 8 365281 365273 1.26037 0 - 0.0299513 2026-02-23T13:51:10.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:51:10 trial068 ceph-mon[38805]: pgmap v1557: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:51:10.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:51:10 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1449756583' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:51:11.276 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2265 8 365281 365273 1.25981 0 - 0.0299513 2026-02-23T13:51:11.762 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:51:11.881 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:51:11.982 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:51:11.980+0000 7f626c71c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:51:11.982 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:51:11.980+0000 7f626c71c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:51:11.994 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:51:11.994 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:51:12.004 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:51:12.004 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:51:12.004 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:51:12.014 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:51:12.014 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:51:12.044 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:51:12.044 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:51:12.276 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2266 8 365281 365273 1.25925 0 - 0.0299513 2026-02-23T13:51:12.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:51:12 trial068 ceph-mon[38805]: pgmap v1558: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:51:12.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:51:12 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3050995988' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:51:13.276 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2267 8 365281 365273 1.2587 0 - 0.0299513 2026-02-23T13:51:14.045 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:51:14.161 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:51:14.225 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:51:13 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:51:13] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:51:14.266 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:51:14.264+0000 7efdc862e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:51:14.266 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:51:14.265+0000 7efdc862e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:51:14.276 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2268 8 365281 365273 1.25814 0 - 0.0299513 2026-02-23T13:51:14.278 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:51:14.278 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:51:14.288 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:51:14.289 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:51:14.289 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:51:14.299 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:51:14.299 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:51:14.331 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:51:14.332 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:51:14.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:51:13 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:51:13.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:51:14.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:51:14 trial068 ceph-mon[38805]: pgmap v1559: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:51:14.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:51:14 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3403352621' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:51:15.276 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2269 8 365281 365273 1.25759 0 - 0.0299513 2026-02-23T13:51:15.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:51:15 trial068 ceph-mon[38805]: osdmap e210: 4 total, 3 up, 4 in 2026-02-23T13:51:16.276 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2270 8 365281 365273 1.25704 0 - 0.0299513 2026-02-23T13:51:16.333 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:51:16.452 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:51:16.555 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:51:16.553+0000 7f3cb3351640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:51:16.556 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:51:16.554+0000 7f3cb3351640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:51:16.568 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:51:16.568 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:51:16.577 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:51:16.578 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:51:16.578 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:51:16.588 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:51:16.589 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:51:16.618 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:51:16 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T13:51:16.619 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:51:16 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T13:51:16.619 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:51:16 trial068 ceph-mon[38805]: pgmap v1561: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:51:16.619 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:51:16 trial068 ceph-mon[38805]: 5.a scrub starts 2026-02-23T13:51:16.620 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:51:16.620 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:51:17.276 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2271 8 365281 365273 1.25648 0 - 0.0299513 2026-02-23T13:51:17.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:51:17 trial068 ceph-mon[38805]: 5.a scrub ok 2026-02-23T13:51:17.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:51:17 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3386277005' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:51:18.276 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2272 8 365281 365273 1.25593 0 - 0.0299513 2026-02-23T13:51:18.621 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:51:18.738 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:51:18.780 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:51:18 trial068 ceph-mon[38805]: pgmap v1562: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:51:18.836 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:51:18.834+0000 7f9eeddce640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:51:18.836 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:51:18.834+0000 7f9eeddce640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:51:18.849 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:51:18.849 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:51:18.859 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:51:18.860 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:51:18.860 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:51:18.870 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:51:18.870 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:51:18.901 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:51:18.901 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:51:19.276 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2273 8 365281 365273 1.25538 0 - 0.0299513 2026-02-23T13:51:19.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:51:19 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3372904314' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:51:20.276 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2274 8 365281 365273 1.25482 0 - 0.0299513 2026-02-23T13:51:20.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:51:20 trial068 ceph-mon[38805]: pgmap v1563: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:51:20.902 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:51:21.019 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:51:21.124 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:51:21.122+0000 7f23f4b46640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:51:21.124 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:51:21.123+0000 7f23f4b46640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:51:21.136 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:51:21.136 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:51:21.146 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:51:21.147 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:51:21.147 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:51:21.157 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:51:21.157 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:51:21.188 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:51:21.188 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:51:21.277 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2275 8 365281 365273 1.25427 0 - 0.0299513 2026-02-23T13:51:21.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:51:21 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2083487369' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:51:22.277 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2276 8 365281 365273 1.25372 0 - 0.0299513 2026-02-23T13:51:22.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:51:22 trial068 ceph-mon[38805]: pgmap v1564: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:51:23.189 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:51:23.277 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2277 8 365281 365273 1.25317 0 - 0.0299513 2026-02-23T13:51:23.308 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:51:23.406 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:51:23.404+0000 7fd337c5a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:51:23.406 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:51:23.404+0000 7fd337c5a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:51:23.417 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:51:23.418 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:51:23.427 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:51:23.428 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:51:23.428 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:51:23.438 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:51:23.438 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:51:23.466 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:51:23.467 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:51:23.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:51:23 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1042776098' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:51:24.277 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2278 8 365281 365273 1.25262 0 - 0.0299513 2026-02-23T13:51:24.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:51:23 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:51:23] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:51:24.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:51:23 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:51:23.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:51:24.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:51:24 trial068 ceph-mon[38805]: pgmap v1565: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:51:25.277 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2279 8 365281 365273 1.25207 0 - 0.0299513 2026-02-23T13:51:25.467 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:51:25.585 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:51:25.689 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:51:25.687+0000 7f2e882a9640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:51:25.689 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:51:25.687+0000 7f2e882a9640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:51:25.701 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:51:25.702 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:51:25.711 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:51:25.712 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:51:25.712 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:51:25.722 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:51:25.722 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:51:25.752 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:51:25.753 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:51:26.277 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T13:51:26.277054+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T13:51:26.278 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T13:51:26.278 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2280 8 365281 365273 1.25152 0 - 0.0299513 2026-02-23T13:51:26.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:51:26 trial068 ceph-mon[38805]: pgmap v1566: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:51:26.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:51:26 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2754789093' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:51:27.277 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2281 8 365281 365273 1.25097 0 - 0.0299513 2026-02-23T13:51:27.754 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:51:27.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:51:27 trial068 ceph-mon[38805]: 5.e scrub starts 2026-02-23T13:51:27.872 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:51:27.967 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:51:27.965+0000 7fb77f949640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:51:27.967 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:51:27.965+0000 7fb77f949640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:51:27.979 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:51:27.979 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:51:27.989 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:51:27.989 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:51:27.990 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:51:28.001 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:51:28.001 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:51:28.031 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:51:28.032 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:51:28.277 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2282 8 365281 365273 1.25043 0 - 0.0299513 2026-02-23T13:51:28.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:51:28 trial068 ceph-mon[38805]: 5.e scrub ok 2026-02-23T13:51:28.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:51:28 trial068 ceph-mon[38805]: pgmap v1567: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:51:28.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:51:28 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2283659038' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:51:29.277 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2283 8 365281 365273 1.24988 0 - 0.0299513 2026-02-23T13:51:30.033 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:51:30.149 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:51:30.249 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:51:30.247+0000 7f8eda1f0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:51:30.249 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:51:30.247+0000 7f8eda1f0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:51:30.261 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:51:30.261 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:51:30.271 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:51:30.272 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:51:30.272 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:51:30.277 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2284 8 365281 365273 1.24933 0 - 0.0299513 2026-02-23T13:51:30.282 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:51:30.282 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:51:30.312 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:51:30.313 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:51:30.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:51:30 trial068 ceph-mon[38805]: pgmap v1568: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:51:30.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:51:30 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3742596754' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:51:31.278 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2285 8 365281 365273 1.24878 0 - 0.0299513 2026-02-23T13:51:32.278 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2286 8 365281 365273 1.24824 0 - 0.0299513 2026-02-23T13:51:32.314 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:51:32.432 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:51:32.535 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:51:32.533+0000 7fc14563e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:51:32.536 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:51:32.534+0000 7fc14563e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:51:32.548 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:51:32.548 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:51:32.559 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:51:32.559 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:51:32.559 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:51:32.570 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:51:32.570 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:51:32.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:51:32 trial068 ceph-mon[38805]: pgmap v1569: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:51:32.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:51:32 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3551016988' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:51:32.600 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:51:32.601 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:51:33.278 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2287 8 365281 365273 1.24769 0 - 0.0299513 2026-02-23T13:51:34.266 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:51:33 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:51:33.974Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:51:34.278 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2288 8 365281 365273 1.24715 0 - 0.0299513 2026-02-23T13:51:34.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:51:33 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:51:33] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:51:34.602 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:51:34.718 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:51:34.819 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:51:34.817+0000 7fc5f4042640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:51:34.820 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:51:34.818+0000 7fc5f4042640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:51:34.820 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:51:34 trial068 ceph-mon[38805]: pgmap v1570: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:51:34.831 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:51:34.832 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:51:34.842 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:51:34.842 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:51:34.842 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:51:34.852 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:51:34.852 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:51:34.882 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:51:34.882 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:51:35.278 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2289 8 365281 365273 1.2466 0 - 0.0299513 2026-02-23T13:51:35.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:51:35 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3263478414' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:51:36.278 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2290 8 365281 365273 1.24606 0 - 0.0299513 2026-02-23T13:51:36.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:51:36 trial068 ceph-mon[38805]: pgmap v1571: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:51:36.883 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:51:37.002 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:51:37.101 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:51:37.099+0000 7f5eaa8d6640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:51:37.101 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:51:37.099+0000 7f5eaa8d6640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:51:37.113 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:51:37.113 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:51:37.123 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:51:37.124 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:51:37.124 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:51:37.134 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:51:37.134 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:51:37.165 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:51:37.166 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:51:37.278 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2291 8 365281 365273 1.24551 0 - 0.0299513 2026-02-23T13:51:37.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:51:37 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2502797397' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:51:38.278 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2292 8 365281 365273 1.24497 0 - 0.0299513 2026-02-23T13:51:38.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:51:38 trial068 ceph-mon[38805]: pgmap v1572: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:51:39.167 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:51:39.278 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2293 8 365281 365273 1.24443 0 - 0.0299513 2026-02-23T13:51:39.284 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:51:39.382 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:51:39.380+0000 7f83c4676640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:51:39.382 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:51:39.380+0000 7f83c4676640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:51:39.394 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:51:39.394 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:51:39.404 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:51:39.404 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:51:39.405 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:51:39.413 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:51:39.414 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:51:39.444 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:51:39.444 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:51:39.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:51:39 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1014459810' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:51:40.278 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2294 8 365281 365273 1.24388 0 - 0.0299513 2026-02-23T13:51:40.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:51:40 trial068 ceph-mon[38805]: pgmap v1573: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:51:41.278 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2295 8 365281 365273 1.24334 0 - 0.0299513 2026-02-23T13:51:41.446 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:51:41.563 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:51:41.659 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:51:41.657+0000 7f9561726640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:51:41.659 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:51:41.657+0000 7f9561726640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:51:41.672 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:51:41.672 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:51:41.682 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:51:41.682 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:51:41.683 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:51:41.692 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:51:41.693 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:51:41.723 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:51:41 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T13:51:41.723 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:51:41 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T13:51:41.724 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:51:41.724 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:51:42.279 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2296 8 365281 365273 1.2428 0 - 0.0299513 2026-02-23T13:51:42.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:51:42 trial068 ceph-mon[38805]: pgmap v1574: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:51:42.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:51:42 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2845485442' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:51:43.278 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2297 8 365281 365273 1.24226 0 - 0.0299513 2026-02-23T13:51:43.725 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:51:43.844 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:51:43.949 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:51:43.947+0000 7f8a0d860640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:51:43.949 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:51:43.947+0000 7f8a0d860640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:51:43.960 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:51:43.961 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:51:43.970 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:51:43.971 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:51:43.971 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:51:43.981 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:51:43.981 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:51:44.009 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:51:43 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:51:43] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:51:44.010 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:51:44.010 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:51:44.279 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2298 8 365281 365273 1.24172 0 - 0.0299513 2026-02-23T13:51:44.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:51:43 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:51:43.972Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:51:44.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:51:44 trial068 ceph-mon[38805]: pgmap v1575: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:51:44.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:51:44 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/630764083' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:51:45.279 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2299 8 365281 365273 1.24118 0 - 0.0299513 2026-02-23T13:51:45.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:51:45 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2026-02-23T13:51:46.011 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:51:46.128 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:51:46.227 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:51:46.225+0000 7f71e07dd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:51:46.228 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:51:46.226+0000 7f71e07dd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:51:46.240 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:51:46.240 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:51:46.250 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:51:46.251 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:51:46.251 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:51:46.261 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:51:46.262 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:51:46.279 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T13:51:46.278546+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T13:51:46.279 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T13:51:46.279 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2300 8 365281 365273 1.24064 0 - 0.0299513 2026-02-23T13:51:46.292 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:51:46.292 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:51:46.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:51:46 trial068 ceph-mon[38805]: pgmap v1576: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:51:46.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:51:46 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T13:51:46.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:51:46 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T13:51:46.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:51:46 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-23T13:51:46.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:51:46 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2026-02-23T13:51:46.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:51:46 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T13:51:46.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:51:46 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2026-02-23T13:51:46.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:51:46 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/41437138' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:51:47.279 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2301 8 365281 365273 1.2401 0 - 0.0299513 2026-02-23T13:51:47.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:51:47 trial068 ceph-mon[38805]: pgmap v1577: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:51:47.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:51:47 trial068 ceph-mon[38805]: pgmap v1578: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:51:48.279 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2302 8 365281 365273 1.23956 0 - 0.0299513 2026-02-23T13:51:48.293 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:51:48.414 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:51:48.509 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:51:48.507+0000 7f30deccc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:51:48.509 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:51:48.507+0000 7f30deccc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:51:48.521 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:51:48.522 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:51:48.531 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:51:48.532 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:51:48.532 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:51:48.542 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:51:48.542 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:51:48.570 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:51:48 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3439053175' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:51:48.570 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:51:48.571 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:51:49.279 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2303 8 365281 365273 1.23902 0 - 0.0299513 2026-02-23T13:51:49.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:51:49 trial068 ceph-mon[38805]: pgmap v1579: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:51:49.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:51:49 trial068 ceph-mon[38805]: 5.8 scrub starts 2026-02-23T13:51:49.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:51:49 trial068 ceph-mon[38805]: 5.8 scrub ok 2026-02-23T13:51:50.279 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2304 8 365281 365273 1.23849 0 - 0.0299513 2026-02-23T13:51:50.572 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:51:50.690 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:51:50.805 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:51:50.803+0000 7fa72cc98640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:51:50.805 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:51:50.803+0000 7fa72cc98640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:51:50.817 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:51:50.817 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:51:50.827 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:51:50.827 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:51:50.828 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:51:50.837 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:51:50.837 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:51:50.867 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:51:50.868 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:51:51.279 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2305 8 365281 365273 1.23795 0 - 0.0299513 2026-02-23T13:51:51.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:51:51 trial068 ceph-mon[38805]: pgmap v1580: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:51:51.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:51:51 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/164210621' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:51:52.279 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2306 8 365281 365273 1.23741 0 - 0.0299513 2026-02-23T13:51:52.869 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:51:52.986 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:51:53.083 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:51:53.081+0000 7f7ef0f02640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:51:53.083 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:51:53.081+0000 7f7ef0f02640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:51:53.095 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:51:53.095 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:51:53.105 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:51:53.106 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:51:53.106 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:51:53.117 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:51:53.117 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:51:53.147 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:51:53.147 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:51:53.279 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2307 8 365281 365273 1.23688 0 - 0.0299513 2026-02-23T13:51:53.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:51:53 trial068 ceph-mon[38805]: pgmap v1581: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:51:53.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:51:53 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3664051119' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:51:54.279 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2308 8 365281 365273 1.23634 0 - 0.0299513 2026-02-23T13:51:54.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:51:53 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:51:53] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:51:54.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:51:53 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:51:53.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:51:55.148 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:51:55.265 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:51:55.279 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2309 8 365281 365273 1.2358 0 - 0.0299513 2026-02-23T13:51:55.364 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:51:55.362+0000 7f39a7fb0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:51:55.364 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:51:55.362+0000 7f39a7fb0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:51:55.376 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:51:55.376 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:51:55.385 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:51:55.386 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:51:55.386 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:51:55.396 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:51:55.396 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:51:55.426 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:51:55.427 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:51:55.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:51:55 trial068 ceph-mon[38805]: pgmap v1582: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:51:55.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:51:55 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3926667500' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:51:56.279 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2310 8 365281 365273 1.23527 0 - 0.0299513 2026-02-23T13:51:57.280 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2311 8 365281 365273 1.23473 0 - 0.0299513 2026-02-23T13:51:57.428 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:51:57.546 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:51:57.647 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:51:57.645+0000 7f6576826640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:51:57.647 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:51:57.645+0000 7f6576826640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:51:57.659 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:51:57.659 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:51:57.669 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:51:57.670 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:51:57.670 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:51:57.679 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:51:57.680 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:51:57.710 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:51:57 trial068 ceph-mon[38805]: pgmap v1583: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:51:57.711 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:51:57.711 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:51:58.280 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2312 8 365281 365273 1.2342 0 - 0.0299513 2026-02-23T13:51:58.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:51:58 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3826470333' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:51:59.280 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2313 8 365281 365273 1.23367 0 - 0.0299513 2026-02-23T13:51:59.712 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:51:59.830 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:51:59.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:51:59 trial068 ceph-mon[38805]: pgmap v1584: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:51:59.928 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:51:59.926+0000 7f0472118640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:51:59.928 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:51:59.926+0000 7f0472118640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:51:59.940 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:51:59.940 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:51:59.950 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:51:59.950 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:51:59.951 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:51:59.960 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:51:59.961 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:51:59.990 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:51:59.991 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:52:00.280 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2314 8 365281 365273 1.23313 0 - 0.0299513 2026-02-23T13:52:00.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:52:00 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1650431549' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:52:01.280 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2315 8 365281 365273 1.2326 0 - 0.0299513 2026-02-23T13:52:01.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:52:01 trial068 ceph-mon[38805]: pgmap v1585: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:52:01.991 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:52:02.109 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:52:02.211 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:52:02.209+0000 7fc6f3a9f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:52:02.211 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:52:02.209+0000 7fc6f3a9f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:52:02.223 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:52:02.223 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:52:02.233 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:52:02.233 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:52:02.234 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:52:02.244 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:52:02.244 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:52:02.272 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:52:02.273 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:52:02.280 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2316 8 365281 365273 1.23207 0 - 0.0299513 2026-02-23T13:52:02.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:52:02 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1582352115' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:52:03.280 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2317 8 365281 365273 1.23154 0 - 0.0299513 2026-02-23T13:52:03.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:52:03 trial068 ceph-mon[38805]: pgmap v1586: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:52:03.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:52:03 trial068 ceph-mon[38805]: 5.7 scrub starts 2026-02-23T13:52:03.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:52:03 trial068 ceph-mon[38805]: 5.7 scrub ok 2026-02-23T13:52:04.266 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:52:03 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:52:03.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:52:04.274 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:52:04.280 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2318 8 365281 365273 1.23101 0 - 0.0299513 2026-02-23T13:52:04.298 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:52:03 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:52:03] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:52:04.392 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:52:04.495 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:52:04.493+0000 7f1423dc2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:52:04.496 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:52:04.494+0000 7f1423dc2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:52:04.507 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:52:04.508 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:52:04.517 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:52:04.518 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:52:04.518 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:52:04.529 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:52:04.529 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:52:04.561 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:52:04.561 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:52:05.280 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2319 8 365281 365273 1.23047 0 - 0.0299513 2026-02-23T13:52:05.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:52:05 trial068 ceph-mon[38805]: pgmap v1587: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:52:05.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:52:05 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/306762118' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:52:06.280 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T13:52:06.280033+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T13:52:06.280 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T13:52:06.281 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2320 8 365281 365273 1.22994 0 - 0.0299513 2026-02-23T13:52:06.562 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:52:06.680 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:52:06.778 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:52:06.776+0000 7f04c4443640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:52:06.778 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:52:06.776+0000 7f04c4443640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:52:06.790 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:52:06.790 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:52:06.800 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:52:06.800 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:52:06.800 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:52:06.810 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:52:06.811 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:52:06.841 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:52:06.841 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:52:07.280 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2321 8 365281 365273 1.22941 0 - 0.0299513 2026-02-23T13:52:07.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:52:07 trial068 ceph-mon[38805]: pgmap v1588: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:52:07.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:52:07 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2357746719' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:52:08.280 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2322 8 365281 365273 1.22889 0 - 0.0299513 2026-02-23T13:52:08.842 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:52:08.960 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:52:09.066 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:52:09.063+0000 7f1af8348640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:52:09.066 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:52:09.064+0000 7f1af8348640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:52:09.078 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:52:09.078 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:52:09.088 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:52:09.088 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:52:09.088 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:52:09.098 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:52:09.098 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:52:09.127 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:52:09.128 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:52:09.280 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2323 8 365281 365273 1.22836 0 - 0.0299513 2026-02-23T13:52:09.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:52:09 trial068 ceph-mon[38805]: pgmap v1589: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:52:09.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:52:09 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1435191380' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:52:10.280 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2324 8 365281 365273 1.22783 0 - 0.0299513 2026-02-23T13:52:11.128 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:52:11.246 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:52:11.280 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2325 8 365281 365273 1.2273 0 - 0.0299513 2026-02-23T13:52:11.341 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:52:11.339+0000 7fd823ec7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:52:11.341 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:52:11.339+0000 7fd823ec7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:52:11.353 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:52:11.353 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:52:11.363 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:52:11.363 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:52:11.363 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:52:11.373 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:52:11.373 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:52:11.401 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:52:11.402 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:52:11.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:52:11 trial068 ceph-mon[38805]: pgmap v1590: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:52:12.281 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2326 8 365281 365273 1.22677 0 - 0.0299513 2026-02-23T13:52:12.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:52:12 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1913106973' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:52:13.281 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2327 8 365281 365273 1.22624 0 - 0.0299513 2026-02-23T13:52:13.403 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:52:13.518 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:52:13.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:52:13 trial068 ceph-mon[38805]: pgmap v1591: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:52:13.616 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:52:13.613+0000 7efd493b9640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:52:13.616 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:52:13.614+0000 7efd493b9640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:52:13.627 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:52:13.628 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:52:13.637 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:52:13.638 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:52:13.638 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:52:13.648 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:52:13.648 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:52:13.678 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:52:13.679 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:52:14.281 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2328 8 365281 365273 1.22572 0 - 0.0299513 2026-02-23T13:52:14.334 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:52:13 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:52:13] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:52:14.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:52:13 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:52:13.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:52:14.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:52:14 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2008780963' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:52:15.281 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2329 8 365281 365273 1.22519 0 - 0.0299513 2026-02-23T13:52:15.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:52:15 trial068 ceph-mon[38805]: pgmap v1592: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:52:15.679 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:52:15.797 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:52:15.892 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:52:15.890+0000 7fc3c210b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:52:15.892 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:52:15.890+0000 7fc3c210b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:52:15.904 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:52:15.904 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:52:15.914 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:52:15.915 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:52:15.915 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:52:15.925 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:52:15.925 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:52:15.953 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:52:15.954 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:52:16.281 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2330 8 365281 365273 1.22467 0 - 0.0299513 2026-02-23T13:52:16.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:52:16 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/930245580' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:52:17.281 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2331 8 365281 365273 1.22414 0 - 0.0299513 2026-02-23T13:52:17.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:52:17 trial068 ceph-mon[38805]: pgmap v1593: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:52:17.954 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:52:18.072 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:52:18.172 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:52:18.170+0000 7f96c3ca1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:52:18.172 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:52:18.170+0000 7f96c3ca1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:52:18.184 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:52:18.184 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:52:18.194 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:52:18.194 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:52:18.194 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:52:18.204 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:52:18.204 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:52:18.235 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:52:18.235 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:52:18.281 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2332 8 365281 365273 1.22362 0 - 0.0299513 2026-02-23T13:52:18.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:52:18 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/683219299' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:52:19.281 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2333 8 365281 365273 1.22309 0 - 0.0299513 2026-02-23T13:52:19.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:52:19 trial068 ceph-mon[38805]: pgmap v1594: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:52:20.236 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:52:20.281 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2334 8 365281 365273 1.22257 0 - 0.0299513 2026-02-23T13:52:20.355 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:52:20.455 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:52:20.453+0000 7fec8618d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:52:20.455 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:52:20.453+0000 7fec8618d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:52:20.467 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:52:20.467 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:52:20.477 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:52:20.477 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:52:20.478 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:52:20.488 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:52:20.488 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:52:20.520 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:52:20.520 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:52:21.281 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2335 8 365281 365273 1.22204 0 - 0.0299513 2026-02-23T13:52:21.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:52:21 trial068 ceph-mon[38805]: pgmap v1595: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:52:21.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:52:21 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/7867315' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:52:22.281 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2336 8 365281 365273 1.22152 0 - 0.0299513 2026-02-23T13:52:22.521 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:52:22.638 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:52:22.738 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:52:22.736+0000 7f799944e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:52:22.738 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:52:22.736+0000 7f799944e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:52:22.750 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:52:22.750 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:52:22.760 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:52:22.760 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:52:22.760 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:52:22.770 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:52:22.770 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:52:22.798 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:52:22.799 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:52:23.281 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2337 8 365281 365273 1.221 0 - 0.0299513 2026-02-23T13:52:23.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:52:23 trial068 ceph-mon[38805]: pgmap v1596: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:52:23.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:52:23 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/609991833' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:52:24.281 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2338 8 365281 365273 1.22048 0 - 0.0299513 2026-02-23T13:52:24.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:52:23 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:52:23] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:52:24.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:52:23 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:52:23.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:52:24.800 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:52:24.918 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:52:25.019 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:52:25.017+0000 7f62229c8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:52:25.019 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:52:25.017+0000 7f62229c8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:52:25.031 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:52:25.032 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:52:25.042 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:52:25.043 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:52:25.043 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:52:25.053 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:52:25.053 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:52:25.084 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:52:25.084 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:52:25.282 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2339 8 365281 365273 1.21995 0 - 0.0299513 2026-02-23T13:52:25.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:52:25 trial068 ceph-mon[38805]: pgmap v1597: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:52:25.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:52:25 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/489569057' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:52:26.282 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T13:52:26.281437+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T13:52:26.282 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T13:52:26.282 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2340 8 365281 365273 1.21943 0 - 0.0299513 2026-02-23T13:52:27.085 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:52:27.210 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:52:27.282 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2341 8 365281 365273 1.21891 0 - 0.0299513 2026-02-23T13:52:27.304 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:52:27.302+0000 7f5510b5f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:52:27.304 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:52:27.302+0000 7f5510b5f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:52:27.315 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:52:27.316 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:52:27.326 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:52:27.326 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:52:27.326 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:52:27.336 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:52:27.336 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:52:27.366 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:52:27.366 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:52:27.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:52:27 trial068 ceph-mon[38805]: pgmap v1598: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:52:27.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:52:27 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T13:52:27.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:52:27 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T13:52:27.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:52:27 trial068 ceph-mon[38805]: 5.5 scrub starts 2026-02-23T13:52:27.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:52:27 trial068 ceph-mon[38805]: 5.5 scrub ok 2026-02-23T13:52:27.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:52:27 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2346788626' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:52:28.282 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2342 8 365281 365273 1.21839 0 - 0.0299513 2026-02-23T13:52:29.282 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2343 8 365281 365273 1.21787 0 - 0.0299513 2026-02-23T13:52:29.367 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:52:29.486 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:52:29.589 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:52:29.587+0000 7f45f32f6640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:52:29.589 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:52:29.588+0000 7f45f32f6640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:52:29.602 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:52:29 trial068 ceph-mon[38805]: pgmap v1599: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:52:29.602 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:52:29.602 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:52:29.612 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:52:29.613 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:52:29.613 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:52:29.623 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:52:29.623 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:52:29.652 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:52:29.652 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:52:30.282 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2344 8 365281 365273 1.21735 0 - 0.0299513 2026-02-23T13:52:30.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:52:30 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2144918124' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:52:31.282 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2345 8 365281 365273 1.21683 0 - 0.0299513 2026-02-23T13:52:31.654 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:52:31.678 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:52:31 trial068 ceph-mon[38805]: pgmap v1600: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:52:31.771 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:52:31.870 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:52:31.868+0000 7fadc66de640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:52:31.870 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:52:31.868+0000 7fadc66de640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:52:31.881 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:52:31.882 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:52:31.892 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:52:31.892 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:52:31.892 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:52:31.903 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:52:31.903 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:52:31.934 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:52:31.934 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:52:32.282 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2346 8 365281 365273 1.21631 0 - 0.0299513 2026-02-23T13:52:32.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:52:32 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/179081992' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:52:33.282 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2347 8 365281 365273 1.2158 0 - 0.0299513 2026-02-23T13:52:33.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:52:33 trial068 ceph-mon[38805]: pgmap v1601: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:52:33.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:52:33 trial068 ceph-mon[38805]: 5.a scrub starts 2026-02-23T13:52:33.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:52:33 trial068 ceph-mon[38805]: 5.a scrub ok 2026-02-23T13:52:33.936 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:52:34.053 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:52:34.152 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:52:34.150+0000 7fea5273d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:52:34.152 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:52:34.150+0000 7fea5273d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:52:34.165 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:52:34.165 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:52:34.175 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:52:34.175 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:52:34.175 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:52:34.185 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:52:34.185 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:52:34.213 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:52:33 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:52:33] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:52:34.214 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:52:34.215 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:52:34.266 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:52:33 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:52:33.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:52:34.282 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2348 8 365281 365273 1.21528 0 - 0.0299513 2026-02-23T13:52:34.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:52:34 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2951266240' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:52:35.282 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2349 8 365281 365273 1.21476 0 - 0.0299513 2026-02-23T13:52:35.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:52:35 trial068 ceph-mon[38805]: pgmap v1602: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:52:36.216 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:52:36.282 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2350 8 365281 365273 1.21424 0 - 0.0299513 2026-02-23T13:52:36.335 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:52:36.435 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:52:36.433+0000 7fd196b34640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:52:36.435 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:52:36.433+0000 7fd196b34640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:52:36.446 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:52:36.447 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:52:36.457 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:52:36.457 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:52:36.457 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:52:36.467 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:52:36.467 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:52:36.495 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:52:36.496 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:52:37.282 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2351 8 365281 365273 1.21373 0 - 0.0299513 2026-02-23T13:52:37.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:52:37 trial068 ceph-mon[38805]: pgmap v1603: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:52:37.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:52:37 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3170899957' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:52:38.282 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2352 8 365281 365273 1.21321 0 - 0.0299513 2026-02-23T13:52:38.497 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:52:38.615 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:52:38.713 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:52:38.711+0000 7f1dd3fff640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:52:38.713 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:52:38.712+0000 7f1dd3fff640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:52:38.725 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:52:38.725 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:52:38.735 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:52:38.736 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:52:38.736 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:52:38.746 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:52:38.746 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:52:38.777 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:52:38.778 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:52:39.283 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2353 8 365281 365273 1.2127 0 - 0.0299513 2026-02-23T13:52:39.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:52:39 trial068 ceph-mon[38805]: pgmap v1604: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:52:39.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:52:39 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/4230567512' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:52:40.283 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2354 8 365281 365273 1.21218 0 - 0.0299513 2026-02-23T13:52:40.780 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:52:40.897 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:52:40.998 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:52:40.996+0000 7f71d5492640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:52:40.999 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:52:40.996+0000 7f71d5492640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:52:41.011 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:52:41.011 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:52:41.022 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:52:41.022 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:52:41.022 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:52:41.032 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:52:41.033 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:52:41.064 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:52:41.065 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:52:41.283 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2355 8 365281 365273 1.21167 0 - 0.0299513 2026-02-23T13:52:41.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:52:41 trial068 ceph-mon[38805]: pgmap v1605: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:52:41.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:52:41 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2783708257' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:52:42.283 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2356 8 365281 365273 1.21115 0 - 0.0299513 2026-02-23T13:52:43.066 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:52:43.184 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:52:43.281 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:52:43.279+0000 7f4dd71f0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:52:43.281 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:52:43.279+0000 7f4dd71f0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:52:43.283 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2357 8 365281 365273 1.21064 0 - 0.0299513 2026-02-23T13:52:43.293 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:52:43.293 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:52:43.303 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:52:43.303 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:52:43.304 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:52:43.314 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:52:43.314 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:52:43.344 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:52:43.344 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:52:43.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:52:43 trial068 ceph-mon[38805]: pgmap v1606: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:52:43.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:52:43 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1666871428' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:52:44.283 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2358 8 365281 365273 1.21012 0 - 0.0299513 2026-02-23T13:52:44.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:52:43 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:52:43] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:52:44.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:52:43 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:52:43.974Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:52:45.283 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2359 8 365281 365273 1.20961 0 - 0.0299513 2026-02-23T13:52:45.346 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:52:45.466 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:52:45.562 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:52:45.560+0000 7f5d365df640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:52:45.563 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:52:45.560+0000 7f5d365df640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:52:45.574 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:52:45.574 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:52:45.584 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:52:45.584 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:52:45.585 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:52:45.594 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:52:45.595 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:52:45.610 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:52:45 trial068 ceph-mon[38805]: pgmap v1607: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:52:45.623 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:52:45.624 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:52:46.283 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T13:52:46.282931+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T13:52:46.283 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T13:52:46.284 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2360 8 365281 365273 1.2091 0 - 0.0299513 2026-02-23T13:52:46.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:52:46 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1421596996' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:52:46.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:52:46 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2026-02-23T13:52:46.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:52:46 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-23T13:52:46.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:52:46 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2026-02-23T13:52:46.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:52:46 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T13:52:46.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:52:46 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2026-02-23T13:52:47.283 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2361 8 365281 365273 1.20859 0 - 0.0299513 2026-02-23T13:52:47.625 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:52:47.655 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:52:47 trial068 ceph-mon[38805]: pgmap v1608: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:52:47.655 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:52:47 trial068 ceph-mon[38805]: pgmap v1609: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:52:47.655 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:52:47 trial068 ceph-mon[38805]: pgmap v1610: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:52:47.746 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:52:47.844 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:52:47.842+0000 7faad3add640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:52:47.844 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:52:47.843+0000 7faad3add640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:52:47.857 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:52:47.858 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:52:47.868 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:52:47.869 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:52:47.869 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:52:47.879 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:52:47.880 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:52:47.910 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:52:47.910 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:52:48.283 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2362 8 365281 365273 1.20807 0 - 0.0299513 2026-02-23T13:52:48.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:52:48 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/542910551' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:52:48.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:52:48 trial068 ceph-mon[38805]: 5.e scrub starts 2026-02-23T13:52:49.283 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2363 8 365281 365273 1.20756 0 - 0.0299513 2026-02-23T13:52:49.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:52:49 trial068 ceph-mon[38805]: pgmap v1611: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:52:49.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:52:49 trial068 ceph-mon[38805]: 5.e scrub ok 2026-02-23T13:52:49.911 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:52:50.029 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:52:50.129 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:52:50.127+0000 7f2fabdcf640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:52:50.129 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:52:50.127+0000 7f2fabdcf640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:52:50.141 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:52:50.141 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:52:50.151 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:52:50.152 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:52:50.152 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:52:50.163 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:52:50.163 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:52:50.193 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:52:50.193 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:52:50.283 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2364 8 365281 365273 1.20705 0 - 0.0299513 2026-02-23T13:52:50.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:52:50 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3605196127' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:52:51.284 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2365 8 365281 365273 1.20654 0 - 0.0299513 2026-02-23T13:52:51.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:52:51 trial068 ceph-mon[38805]: pgmap v1612: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:52:52.194 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:52:52.284 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2366 8 365281 365273 1.20603 0 - 0.0299513 2026-02-23T13:52:52.312 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:52:52.408 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:52:52.406+0000 7f14ffb5b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:52:52.409 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:52:52.406+0000 7f14ffb5b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:52:52.420 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:52:52.420 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:52:52.431 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:52:52.431 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:52:52.431 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:52:52.443 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:52:52.443 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:52:52.472 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:52:52.473 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:52:53.284 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2367 8 365281 365273 1.20552 0 - 0.0299513 2026-02-23T13:52:53.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:52:53 trial068 ceph-mon[38805]: pgmap v1613: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:52:53.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:52:53 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/514094710' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:52:54.284 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2368 8 365281 365273 1.20501 0 - 0.0299513 2026-02-23T13:52:54.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:52:53 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:52:53] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:52:54.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:52:53 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:52:53.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:52:54.474 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:52:54.595 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:52:54.690 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:52:54.688+0000 7f3005c66640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:52:54.691 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:52:54.688+0000 7f3005c66640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:52:54.703 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:52:54.703 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:52:54.713 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:52:54.713 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:52:54.713 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:52:54.723 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:52:54.723 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:52:54.755 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:52:54.755 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:52:55.284 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2369 8 365281 365273 1.2045 0 - 0.0299513 2026-02-23T13:52:55.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:52:55 trial068 ceph-mon[38805]: pgmap v1614: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:52:55.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:52:55 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3359039983' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:52:56.284 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2370 8 365281 365273 1.204 0 - 0.0299513 2026-02-23T13:52:56.756 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:52:56.782 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:52:56 trial068 ceph-mon[38805]: 5.8 scrub starts 2026-02-23T13:52:56.782 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:52:56 trial068 ceph-mon[38805]: 5.8 scrub ok 2026-02-23T13:52:56.876 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:52:56.971 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:52:56.969+0000 7f35863e7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:52:56.972 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:52:56.970+0000 7f35863e7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:52:56.983 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:52:56.984 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:52:56.994 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:52:56.994 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:52:56.994 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:52:57.004 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:52:57.004 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:52:57.035 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:52:57.036 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:52:57.284 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2371 8 365281 365273 1.20349 0 - 0.0299513 2026-02-23T13:52:57.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:52:57 trial068 ceph-mon[38805]: pgmap v1615: 17 pgs: 1 active+clean+scrubbing, 1 active+recovery_unfound+undersized+degraded, 4 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:52:57.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:52:57 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2648239643' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:52:58.284 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2372 8 365281 365273 1.20298 0 - 0.0299513 2026-02-23T13:52:59.036 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:52:59.155 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:52:59.258 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:52:59.256+0000 7f9a8c192640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:52:59.258 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:52:59.256+0000 7f9a8c192640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:52:59.270 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:52:59.270 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:52:59.280 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:52:59.280 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:52:59.280 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:52:59.284 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2373 8 365281 365273 1.20247 0 - 0.0299513 2026-02-23T13:52:59.290 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:52:59.291 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:52:59.319 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:52:59.319 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:52:59.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:52:59 trial068 ceph-mon[38805]: pgmap v1616: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 1 active+clean+scrubbing, 4 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:52:59.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:52:59 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T13:52:59.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:52:59 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T13:52:59.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:52:59 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2035699867' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:53:00.284 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2374 8 365281 365273 1.20197 0 - 0.0299513 2026-02-23T13:53:01.284 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2375 8 365281 365273 1.20146 0 - 0.0299513 2026-02-23T13:53:01.320 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:53:01.438 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:53:01.537 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:53:01.535+0000 7fe3136e2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:53:01.537 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:53:01.535+0000 7fe3136e2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:53:01.549 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:53:01.550 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:53:01.559 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:53:01.560 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:53:01.560 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:53:01.569 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:53:01.570 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:53:01.597 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:53:01 trial068 ceph-mon[38805]: pgmap v1617: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 1 active+clean+scrubbing, 4 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:53:01.598 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:53:01.599 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:53:02.284 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2376 8 365281 365273 1.20096 0 - 0.0299513 2026-02-23T13:53:02.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:53:02 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2059603711' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:53:03.284 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2377 8 365281 365273 1.20045 0 - 0.0299513 2026-02-23T13:53:03.600 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:53:03.719 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:53:03.740 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:53:03 trial068 ceph-mon[38805]: pgmap v1618: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:53:03.813 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:53:03.811+0000 7f55b5813640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:53:03.813 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:53:03.811+0000 7f55b5813640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:53:03.825 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:53:03.826 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:53:03.837 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:53:03.837 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:53:03.837 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:53:03.847 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:53:03.847 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:53:03.878 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:53:03.878 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:53:04.093 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:53:03 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:53:03] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:53:04.266 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:53:03 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:53:03.974Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:53:04.284 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2378 8 365281 365273 1.19995 0 - 0.0299513 2026-02-23T13:53:04.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:53:04 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/984734249' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:53:05.284 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2379 8 365281 365273 1.19944 0 - 0.0299513 2026-02-23T13:53:05.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:53:05 trial068 ceph-mon[38805]: pgmap v1619: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:53:05.879 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:53:05.998 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:53:06.095 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:53:06.093+0000 7f4e0831e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:53:06.095 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:53:06.093+0000 7f4e0831e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:53:06.107 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:53:06.107 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:53:06.117 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:53:06.117 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:53:06.118 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:53:06.127 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:53:06.128 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:53:06.158 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:53:06.158 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:53:06.285 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T13:53:06.284454+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T13:53:06.285 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T13:53:06.285 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2380 8 365281 365273 1.19894 0 - 0.0299513 2026-02-23T13:53:06.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:53:06 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3177501189' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:53:07.285 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2381 8 365281 365273 1.19843 0 - 0.0299513 2026-02-23T13:53:07.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:53:07 trial068 ceph-mon[38805]: pgmap v1620: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:53:07.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:53:07 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T13:53:07.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:53:07 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T13:53:08.159 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:53:08.277 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:53:08.285 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2382 8 365281 365273 1.19793 0 - 0.0299513 2026-02-23T13:53:08.372 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:53:08.370+0000 7f579f39d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:53:08.373 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:53:08.371+0000 7f579f39d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:53:08.385 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:53:08.385 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:53:08.395 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:53:08.395 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:53:08.395 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:53:08.405 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:53:08.406 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:53:08.436 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:53:08.436 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:53:08.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:53:08 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/494134184' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:53:09.285 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2383 8 365281 365273 1.19743 0 - 0.0299513 2026-02-23T13:53:09.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:53:09 trial068 ceph-mon[38805]: pgmap v1621: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:53:10.285 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2384 8 365281 365273 1.19693 0 - 0.0299513 2026-02-23T13:53:10.438 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:53:10.555 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:53:10.651 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:53:10.649+0000 7f664c4f7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:53:10.651 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:53:10.649+0000 7f664c4f7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:53:10.663 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:53:10.663 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:53:10.673 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:53:10.673 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:53:10.674 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:53:10.683 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:53:10.684 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:53:10.714 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:53:10.714 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:53:11.285 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2385 8 365281 365273 1.19642 0 - 0.0299513 2026-02-23T13:53:11.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:53:11 trial068 ceph-mon[38805]: pgmap v1622: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:53:11.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:53:11 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1933859849' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:53:11.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:53:11 trial068 ceph-mon[38805]: 5.7 scrub starts 2026-02-23T13:53:11.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:53:11 trial068 ceph-mon[38805]: 5.7 scrub ok 2026-02-23T13:53:12.285 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2386 8 365281 365273 1.19592 0 - 0.0299513 2026-02-23T13:53:12.715 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:53:12.832 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:53:12.931 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:53:12.929+0000 7fd6cb8c1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:53:12.931 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:53:12.929+0000 7fd6cb8c1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:53:12.943 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:53:12.943 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:53:12.953 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:53:12.954 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:53:12.954 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:53:12.964 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:53:12.965 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:53:12.996 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:53:12.996 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:53:13.285 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2387 8 365281 365273 1.19542 0 - 0.0299513 2026-02-23T13:53:13.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:53:13 trial068 ceph-mon[38805]: pgmap v1623: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:53:13.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:53:13 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/388209106' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:53:14.285 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2388 8 365281 365273 1.19492 0 - 0.0299513 2026-02-23T13:53:14.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:53:13 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:53:13] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:53:14.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:53:13 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:53:13.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:53:14.997 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:53:15.115 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:53:15.221 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:53:15.219+0000 7f988690b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:53:15.221 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:53:15.219+0000 7f988690b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:53:15.233 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:53:15.233 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:53:15.243 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:53:15.244 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:53:15.244 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:53:15.254 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:53:15.254 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:53:15.284 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:53:15.285 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:53:15.285 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2389 8 365281 365273 1.19442 0 - 0.0299513 2026-02-23T13:53:15.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:53:15 trial068 ceph-mon[38805]: pgmap v1624: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:53:15.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:53:15 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1356299320' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:53:16.285 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2390 8 365281 365273 1.19392 0 - 0.0299513 2026-02-23T13:53:17.285 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2391 8 365281 365273 1.19342 0 - 0.0299513 2026-02-23T13:53:17.286 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:53:17.405 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:53:17.504 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:53:17.501+0000 7feaaa4f6640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:53:17.504 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:53:17.502+0000 7feaaa4f6640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:53:17.515 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:53:17.515 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:53:17.525 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:53:17.526 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:53:17.526 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:53:17.536 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:53:17.536 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:53:17.565 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:53:17 trial068 ceph-mon[38805]: pgmap v1625: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:53:17.566 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:53:17.567 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:53:18.285 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2392 8 365281 365273 1.19292 0 - 0.0299513 2026-02-23T13:53:18.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:53:18 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2966789165' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:53:19.285 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2393 8 365281 365273 1.19242 0 - 0.0299513 2026-02-23T13:53:19.568 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:53:19.685 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:53:19.707 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:53:19 trial068 ceph-mon[38805]: pgmap v1626: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:53:19.779 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:53:19.777+0000 7fd89225b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:53:19.779 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:53:19.778+0000 7fd89225b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:53:19.791 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:53:19.792 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:53:19.802 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:53:19.802 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:53:19.802 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:53:19.813 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:53:19.813 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:53:19.842 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:53:19.842 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:53:20.286 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2394 8 365281 365273 1.19193 0 - 0.0299513 2026-02-23T13:53:20.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:53:20 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2272550884' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:53:21.285 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2395 8 365281 365273 1.19143 0 - 0.0299513 2026-02-23T13:53:21.843 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:53:21.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:53:21 trial068 ceph-mon[38805]: pgmap v1627: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:53:21.958 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:53:22.053 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:53:22.052+0000 7f29df9f6640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:53:22.053 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:53:22.052+0000 7f29df9f6640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:53:22.065 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:53:22.066 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:53:22.076 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:53:22.076 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:53:22.076 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:53:22.087 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:53:22.087 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:53:22.119 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:53:22.119 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:53:22.285 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2396 8 365281 365273 1.19093 0 - 0.0299513 2026-02-23T13:53:22.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:53:22 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/523878297' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:53:23.286 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2397 8 365281 365273 1.19043 0 - 0.0299513 2026-02-23T13:53:23.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:53:23 trial068 ceph-mon[38805]: pgmap v1628: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:53:24.120 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:53:24.239 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:53:24.261 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:53:23 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:53:23] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:53:24.286 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2398 8 365281 365273 1.18994 0 - 0.0299513 2026-02-23T13:53:24.336 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:53:24.334+0000 7ff1903db640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:53:24.336 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:53:24.334+0000 7ff1903db640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:53:24.348 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:53:24.348 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:53:24.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:53:23 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:53:23.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:53:24.358 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:53:24.359 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:53:24.359 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:53:24.368 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:53:24.369 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:53:24.397 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:53:24.398 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:53:24.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:53:24 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/4248147143' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:53:25.286 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2399 8 365281 365273 1.18944 0 - 0.0299513 2026-02-23T13:53:25.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:53:25 trial068 ceph-mon[38805]: pgmap v1629: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:53:26.286 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T13:53:26.285866+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T13:53:26.286 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T13:53:26.286 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2400 8 365281 365273 1.18895 0 - 0.0299513 2026-02-23T13:53:26.399 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:53:26.517 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:53:26.613 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:53:26.611+0000 7f6974270640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:53:26.613 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:53:26.611+0000 7f6974270640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:53:26.624 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:53:26.625 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:53:26.634 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:53:26.635 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:53:26.635 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:53:26.645 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:53:26.645 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:53:26.675 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:53:26.676 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:53:27.286 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2401 8 365281 365273 1.18845 0 - 0.0299513 2026-02-23T13:53:27.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:53:27 trial068 ceph-mon[38805]: pgmap v1630: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:53:27.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:53:27 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/780426599' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:53:28.286 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2402 8 365281 365273 1.18796 0 - 0.0299513 2026-02-23T13:53:28.677 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:53:28.795 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:53:28.900 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:53:28.898+0000 7f90e1646640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:53:28.900 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:53:28.898+0000 7f90e1646640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:53:28.912 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:53:28.913 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:53:28.922 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:53:28.923 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:53:28.923 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:53:28.932 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:53:28.933 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:53:28.962 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:53:28.963 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:53:29.286 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2403 8 365281 365273 1.18746 0 - 0.0299513 2026-02-23T13:53:29.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:53:29 trial068 ceph-mon[38805]: pgmap v1631: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:53:29.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:53:29 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/4103442639' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:53:30.286 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2404 8 365281 365273 1.18697 0 - 0.0299513 2026-02-23T13:53:30.964 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:53:31.081 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:53:31.187 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:53:31.185+0000 7f9d4409f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:53:31.187 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:53:31.185+0000 7f9d4409f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:53:31.199 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:53:31.200 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:53:31.209 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:53:31.210 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:53:31.210 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:53:31.220 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:53:31.220 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:53:31.264 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:53:31.265 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:53:31.286 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2405 8 365281 365273 1.18647 0 - 0.0299513 2026-02-23T13:53:31.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:53:31 trial068 ceph-mon[38805]: pgmap v1632: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:53:31.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:53:31 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/168121634' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:53:32.286 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2406 8 365281 365273 1.18598 0 - 0.0299513 2026-02-23T13:53:33.266 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:53:33.286 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2407 8 365281 365273 1.18549 0 - 0.0299513 2026-02-23T13:53:33.383 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:53:33.484 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:53:33.482+0000 7f0f7b831640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:53:33.484 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:53:33.482+0000 7f0f7b831640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:53:33.495 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:53:33.496 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:53:33.505 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:53:33.506 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:53:33.506 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:53:33.516 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:53:33.516 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:53:33.545 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:53:33 trial068 ceph-mon[38805]: pgmap v1633: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:53:33.546 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:53:33.546 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:53:34.266 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:53:33 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:53:33.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:53:34.287 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2408 8 365281 365273 1.185 0 - 0.0299513 2026-02-23T13:53:34.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:53:33 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:53:33] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:53:34.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:53:34 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/464443299' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:53:35.287 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2409 8 365281 365273 1.1845 0 - 0.0299513 2026-02-23T13:53:35.547 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:53:35.668 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:53:35.691 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:53:35 trial068 ceph-mon[38805]: pgmap v1634: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:53:35.768 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:53:35.765+0000 7fea772a8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:53:35.768 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:53:35.766+0000 7fea772a8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:53:35.779 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:53:35.780 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:53:35.789 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:53:35.790 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:53:35.790 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:53:35.799 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:53:35.800 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:53:35.830 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:53:35.830 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:53:36.287 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2410 8 365281 365273 1.18401 0 - 0.0299513 2026-02-23T13:53:36.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:53:36 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1821533695' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:53:37.287 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2411 8 365281 365273 1.18352 0 - 0.0299513 2026-02-23T13:53:37.832 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:53:37.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:53:37 trial068 ceph-mon[38805]: pgmap v1635: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:53:37.951 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:53:38.048 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:53:38.046+0000 7f096de11640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:53:38.048 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:53:38.046+0000 7f096de11640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:53:38.060 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:53:38.061 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:53:38.070 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:53:38.071 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:53:38.071 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:53:38.081 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:53:38.081 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:53:38.110 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:53:38.111 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:53:38.287 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2412 8 365281 365273 1.18303 0 - 0.0299513 2026-02-23T13:53:38.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:53:38 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3004146221' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:53:39.287 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2413 8 365281 365273 1.18254 0 - 0.0299513 2026-02-23T13:53:39.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:53:39 trial068 ceph-mon[38805]: pgmap v1636: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:53:39.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:53:39 trial068 ceph-mon[38805]: 5.a scrub starts 2026-02-23T13:53:39.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:53:39 trial068 ceph-mon[38805]: 5.a scrub ok 2026-02-23T13:53:40.112 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:53:40.229 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:53:40.287 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2414 8 365281 365273 1.18205 0 - 0.0299513 2026-02-23T13:53:40.323 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:53:40.321+0000 7f3ead572640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:53:40.323 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:53:40.321+0000 7f3ead572640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:53:40.335 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:53:40.335 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:53:40.345 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:53:40.345 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:53:40.346 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:53:40.356 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:53:40.357 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:53:40.386 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:53:40.387 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:53:40.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:53:40 trial068 ceph-mon[38805]: 5.5 scrub starts 2026-02-23T13:53:40.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:53:40 trial068 ceph-mon[38805]: 5.5 scrub ok 2026-02-23T13:53:40.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:53:40 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1652552225' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:53:41.287 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2415 8 365281 365273 1.18156 0 - 0.0299513 2026-02-23T13:53:41.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:53:41 trial068 ceph-mon[38805]: pgmap v1637: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:53:42.287 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2416 8 365281 365273 1.18107 0 - 0.0299513 2026-02-23T13:53:42.388 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:53:42.506 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:53:42.601 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:53:42.599+0000 7fb90f4f7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:53:42.602 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:53:42.600+0000 7fb90f4f7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:53:42.613 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:53:42.613 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:53:42.623 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:53:42.624 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:53:42.624 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:53:42.634 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:53:42.634 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:53:42.663 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:53:42.664 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:53:43.287 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2417 8 365281 365273 1.18058 0 - 0.0299513 2026-02-23T13:53:43.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:53:43 trial068 ceph-mon[38805]: pgmap v1638: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:53:43.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:53:43 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1739560673' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:53:44.287 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2418 8 365281 365273 1.1801 0 - 0.0299513 2026-02-23T13:53:44.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:53:43 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:53:43] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:53:44.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:53:43 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:53:43.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:53:44.665 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:53:44.784 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:53:44.882 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:53:44.880+0000 7f8885324640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:53:44.883 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:53:44.881+0000 7f8885324640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:53:44.894 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:53:44.895 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:53:44.905 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:53:44.905 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:53:44.905 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:53:44.915 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:53:44.916 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:53:44.945 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:53:44.946 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:53:45.287 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2419 8 365281 365273 1.17961 0 - 0.0299513 2026-02-23T13:53:45.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:53:45 trial068 ceph-mon[38805]: pgmap v1639: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:53:45.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:53:45 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3500466136' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:53:46.287 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T13:53:46.287350+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T13:53:46.288 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T13:53:46.288 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2420 8 365281 365273 1.17912 0 - 0.0299513 2026-02-23T13:53:46.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:53:46 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2026-02-23T13:53:46.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:53:46 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-23T13:53:46.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:53:46 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2026-02-23T13:53:46.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:53:46 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T13:53:46.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:53:46 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2026-02-23T13:53:46.947 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:53:47.066 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:53:47.165 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:53:47.163+0000 7f66b8eef640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:53:47.165 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:53:47.163+0000 7f66b8eef640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:53:47.177 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:53:47.177 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:53:47.187 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:53:47.188 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:53:47.188 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:53:47.198 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:53:47.199 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:53:47.231 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:53:47.231 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:53:47.287 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2421 8 365281 365273 1.17863 0 - 0.0299513 2026-02-23T13:53:47.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:53:47 trial068 ceph-mon[38805]: pgmap v1640: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:53:47.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:53:47 trial068 ceph-mon[38805]: pgmap v1641: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:53:47.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:53:47 trial068 ceph-mon[38805]: pgmap v1642: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:53:47.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:53:47 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3449093827' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:53:48.288 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2422 8 365281 365273 1.17815 0 - 0.0299513 2026-02-23T13:53:49.232 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:53:49.288 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2423 8 365281 365273 1.17766 0 - 0.0299513 2026-02-23T13:53:49.349 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:53:49.448 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:53:49.446+0000 7f09c5bbb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:53:49.449 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:53:49.447+0000 7f09c5bbb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:53:49.460 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:53:49.461 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:53:49.471 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:53:49.491 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:53:49.491 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:53:49.492 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:53:49.492 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:53:49.508 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:53:49 trial068 ceph-mon[38805]: pgmap v1643: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:53:49.509 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:53:49.510 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:53:50.288 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2424 8 365281 365273 1.17717 0 - 0.0299513 2026-02-23T13:53:50.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:53:50 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/869308664' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:53:51.288 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2425 8 365281 365273 1.17669 0 - 0.0299513 2026-02-23T13:53:51.510 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:53:51.627 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:53:51.679 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:53:51 trial068 ceph-mon[38805]: pgmap v1644: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:53:51.679 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:53:51 trial068 ceph-mon[38805]: osdmap e211: 4 total, 3 up, 4 in 2026-02-23T13:53:51.724 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:53:51.722+0000 7fc2d771b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:53:51.724 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:53:51.722+0000 7fc2d771b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:53:51.736 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:53:51.736 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:53:51.746 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:53:51.746 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:53:51.746 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:53:51.756 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:53:51.757 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:53:51.787 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:53:51.787 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:53:52.288 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2426 8 365281 365273 1.1762 0 - 0.0299513 2026-02-23T13:53:52.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:53:52 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3996389133' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:53:53.288 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2427 8 365281 365273 1.17572 0 - 0.0299513 2026-02-23T13:53:53.789 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:53:53.814 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:53:53 trial068 ceph-mon[38805]: pgmap v1646: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:53:53.906 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:53:54.003 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:53:54.002+0000 7fb3e53a9640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:53:54.003 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:53:54.002+0000 7fb3e53a9640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:53:54.015 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:53:54.016 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:53:54.026 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:53:54.027 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:53:54.027 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:53:54.037 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:53:54.037 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:53:54.067 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:53:53 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:53:53] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:53:54.068 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:53:54.068 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:53:54.288 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2428 8 365281 365273 1.17524 0 - 0.0299513 2026-02-23T13:53:54.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:53:53 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:53:53.972Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:53:54.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:53:54 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2224180625' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:53:55.288 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2429 8 365281 365273 1.17475 0 - 0.0299513 2026-02-23T13:53:55.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:53:55 trial068 ceph-mon[38805]: pgmap v1647: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:53:56.069 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:53:56.189 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:53:56.288 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:53:56.286+0000 7f8ef8147640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:53:56.288 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:53:56.286+0000 7f8ef8147640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:53:56.288 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2430 8 365281 365273 1.17427 0 - 0.0299513 2026-02-23T13:53:56.299 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:53:56.300 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:53:56.310 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:53:56.310 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:53:56.310 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:53:56.320 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:53:56.320 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:53:56.351 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:53:56.352 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:53:56.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:53:56 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/377614996' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:53:57.288 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2431 8 365281 365273 1.17379 0 - 0.0299513 2026-02-23T13:53:57.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:53:57 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T13:53:57.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:53:57 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T13:53:57.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:53:57 trial068 ceph-mon[38805]: pgmap v1648: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:53:58.288 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2432 8 365281 365273 1.1733 0 - 0.0299513 2026-02-23T13:53:58.353 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:53:58.473 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:53:58.578 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:53:58.576+0000 7f9367c22640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:53:58.578 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:53:58.576+0000 7f9367c22640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:53:58.590 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:53:58.591 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:53:58.601 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:53:58.601 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:53:58.601 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:53:58.611 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:53:58.611 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:53:58.642 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:53:58.642 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:53:59.289 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2433 8 365281 365273 1.17282 0 - 0.0299513 2026-02-23T13:53:59.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:53:59 trial068 ceph-mon[38805]: pgmap v1649: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:53:59.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:53:59 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1965203086' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:54:00.289 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2434 8 365281 365273 1.17234 0 - 0.0299513 2026-02-23T13:54:00.643 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:54:00.761 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:54:00.869 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:54:00.867+0000 7eff18f7a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:54:00.869 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:54:00.868+0000 7eff18f7a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:54:00.881 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:54:00.881 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:54:00.891 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:54:00.892 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:54:00.892 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:54:00.902 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:54:00.902 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:54:00.934 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:54:00.934 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:54:01.289 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2435 8 365281 365273 1.17186 0 - 0.0299513 2026-02-23T13:54:01.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:54:01 trial068 ceph-mon[38805]: pgmap v1650: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:54:01.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:54:01 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/590097884' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:54:02.289 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2436 8 365281 365273 1.17138 0 - 0.0299513 2026-02-23T13:54:02.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:54:02 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T13:54:02.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:54:02 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T13:54:02.935 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:54:03.052 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:54:03.151 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:54:03.149+0000 7f0c78d9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:54:03.151 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:54:03.149+0000 7f0c78d9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:54:03.163 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:54:03.163 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:54:03.173 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:54:03.174 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:54:03.174 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:54:03.184 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:54:03.184 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:54:03.212 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:54:03.212 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:54:03.289 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2437 8 365281 365273 1.1709 0 - 0.0299513 2026-02-23T13:54:03.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:54:03 trial068 ceph-mon[38805]: pgmap v1651: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:54:03.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:54:03 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/4064421914' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:54:04.266 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:54:03 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:54:03.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:54:04.289 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2438 8 365281 365273 1.17042 0 - 0.0299513 2026-02-23T13:54:04.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:54:03 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:54:03] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:54:05.213 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:54:05.289 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2439 8 365281 365273 1.16994 0 - 0.0299513 2026-02-23T13:54:05.333 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:54:05.438 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:54:05.435+0000 7f2a12fe8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:54:05.438 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:54:05.436+0000 7f2a12fe8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:54:05.450 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:54:05.450 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:54:05.460 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:54:05.461 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:54:05.461 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:54:05.471 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:54:05.471 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:54:05.499 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:54:05 trial068 ceph-mon[38805]: pgmap v1652: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:54:05.500 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:54:05 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1326640229' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:54:05.500 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:54:05.501 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:54:06.289 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T13:54:06.288827+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T13:54:06.289 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T13:54:06.289 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2440 8 365281 365273 1.16946 0 - 0.0299513 2026-02-23T13:54:06.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:54:06 trial068 ceph-mon[38805]: 5.8 scrub starts 2026-02-23T13:54:06.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:54:06 trial068 ceph-mon[38805]: 5.8 scrub ok 2026-02-23T13:54:07.289 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2441 8 365281 365273 1.16898 0 - 0.0299513 2026-02-23T13:54:07.501 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:54:07.616 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:54:07.710 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:54:07.708+0000 7f5b8fd20640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:54:07.710 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:54:07 trial068 ceph-mon[38805]: pgmap v1653: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:54:07.711 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:54:07.708+0000 7f5b8fd20640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:54:07.722 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:54:07.722 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:54:07.732 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:54:07.732 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:54:07.732 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:54:07.743 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:54:07.743 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:54:07.773 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:54:07.774 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:54:08.289 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2442 8 365281 365273 1.1685 0 - 0.0299513 2026-02-23T13:54:08.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:54:08 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2224736240' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:54:08.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:54:08 trial068 ceph-mon[38805]: 5.e scrub starts 2026-02-23T13:54:08.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:54:08 trial068 ceph-mon[38805]: 5.e scrub ok 2026-02-23T13:54:09.289 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2443 8 365281 365273 1.16802 0 - 0.0299513 2026-02-23T13:54:09.775 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:54:09.800 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:54:09 trial068 ceph-mon[38805]: pgmap v1654: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:54:09.893 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:54:09.996 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:54:09.994+0000 7fa03c786640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:54:09.996 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:54:09.994+0000 7fa03c786640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:54:10.009 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:54:10.009 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:54:10.019 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:54:10.019 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:54:10.019 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:54:10.030 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:54:10.031 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:54:10.058 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:54:10.059 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:54:10.289 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2444 8 365281 365273 1.16754 0 - 0.0299513 2026-02-23T13:54:10.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:54:10 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2026553575' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:54:11.289 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2445 8 365281 365273 1.16706 0 - 0.0299513 2026-02-23T13:54:11.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:54:11 trial068 ceph-mon[38805]: pgmap v1655: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:54:12.060 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:54:12.178 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:54:12.278 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:54:12.276+0000 7fa20ad94640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:54:12.278 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:54:12.276+0000 7fa20ad94640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:54:12.289 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2446 8 365281 365273 1.16659 0 - 0.0299513 2026-02-23T13:54:12.290 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:54:12.291 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:54:12.301 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:54:12.301 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:54:12.301 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:54:12.312 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:54:12.312 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:54:12.342 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:54:12.343 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:54:12.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:54:12 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1002472418' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:54:13.290 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2447 8 365281 365273 1.16611 0 - 0.0299513 2026-02-23T13:54:13.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:54:13 trial068 ceph-mon[38805]: pgmap v1656: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:54:14.290 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2448 8 365281 365273 1.16563 0 - 0.0299513 2026-02-23T13:54:14.343 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:54:14.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:54:13 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:54:13] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:54:14.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:54:13 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:54:13.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:54:14.464 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:54:14.561 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:54:14.559+0000 7fa2c4dff640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:54:14.561 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:54:14.559+0000 7fa2c4dff640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:54:14.574 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:54:14.574 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:54:14.584 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:54:14.585 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:54:14.585 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:54:14.595 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:54:14.595 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:54:14.625 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:54:14.626 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:54:15.290 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2449 8 365281 365273 1.16516 0 - 0.0299513 2026-02-23T13:54:15.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:54:15 trial068 ceph-mon[38805]: pgmap v1657: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:54:15.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:54:15 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/446693364' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:54:16.290 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2450 8 365281 365273 1.16468 0 - 0.0299513 2026-02-23T13:54:16.627 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:54:16.745 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:54:16.844 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:54:16.841+0000 7f12dbbbc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:54:16.844 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:54:16.842+0000 7f12dbbbc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:54:16.856 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:54:16.857 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:54:16.867 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:54:16.867 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:54:16.868 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:54:16.877 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:54:16.878 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:54:16.909 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:54:16.909 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:54:17.290 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2451 8 365281 365273 1.16421 0 - 0.0299513 2026-02-23T13:54:17.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:54:17 trial068 ceph-mon[38805]: pgmap v1658: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:54:17.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:54:17 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/4273347542' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:54:18.290 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2452 8 365281 365273 1.16373 0 - 0.0299513 2026-02-23T13:54:18.911 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:54:19.029 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:54:19.128 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:54:19.126+0000 7f695b4c4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:54:19.129 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:54:19.127+0000 7f695b4c4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:54:19.141 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:54:19.141 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:54:19.151 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:54:19.152 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:54:19.152 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:54:19.161 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:54:19.161 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:54:19.192 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:54:19.193 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:54:19.290 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2453 8 365281 365273 1.16326 0 - 0.0299513 2026-02-23T13:54:19.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:54:19 trial068 ceph-mon[38805]: pgmap v1659: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:54:19.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:54:19 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/197508258' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:54:20.290 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2454 8 365281 365273 1.16278 0 - 0.0299513 2026-02-23T13:54:21.193 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:54:21.290 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2455 8 365281 365273 1.16231 0 - 0.0299513 2026-02-23T13:54:21.311 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:54:21.412 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:54:21.411+0000 7f1d420bc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:54:21.413 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:54:21.411+0000 7f1d420bc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:54:21.425 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:54:21.425 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:54:21.435 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:54:21.435 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:54:21.436 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:54:21.445 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:54:21.446 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:54:21.476 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:54:21 trial068 ceph-mon[38805]: pgmap v1660: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:54:21.476 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:54:21 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/252724154' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:54:21.477 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:54:21.478 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:54:22.291 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2456 8 365281 365273 1.16184 0 - 0.0299513 2026-02-23T13:54:23.291 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2457 8 365281 365273 1.16136 0 - 0.0299513 2026-02-23T13:54:23.479 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:54:23.597 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:54:23.693 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:54:23.691+0000 7f8dac1f0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:54:23.694 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:54:23.692+0000 7f8dac1f0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:54:23.706 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:54:23.706 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:54:23.716 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:54:23.717 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:54:23.717 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:54:23 trial068 ceph-mon[38805]: pgmap v1661: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:54:23.717 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:54:23.727 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:54:23.727 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:54:23.773 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:54:23.773 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:54:24.291 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2458 8 365281 365273 1.16089 0 - 0.0299513 2026-02-23T13:54:24.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:54:23 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:54:23] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:54:24.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:54:23 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:54:23.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:54:24.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:54:24 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3474121932' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:54:25.291 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2459 8 365281 365273 1.16042 0 - 0.0299513 2026-02-23T13:54:25.774 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:54:25.799 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:54:25 trial068 ceph-mon[38805]: pgmap v1662: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:54:25.799 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:54:25 trial068 ceph-mon[38805]: 5.7 scrub starts 2026-02-23T13:54:25.800 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:54:25 trial068 ceph-mon[38805]: 5.7 scrub ok 2026-02-23T13:54:25.893 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:54:25.987 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:54:25.986+0000 7f0add0f9640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:54:25.988 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:54:25.986+0000 7f0add0f9640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:54:26.000 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:54:26.000 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:54:26.010 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:54:26.010 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:54:26.010 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:54:26.021 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:54:26.022 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:54:26.053 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:54:26.053 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:54:26.291 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T13:54:26.290714+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T13:54:26.291 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T13:54:26.291 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2460 8 365281 365273 1.15995 0 - 0.0299513 2026-02-23T13:54:26.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:54:26 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/579757390' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:54:27.291 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2461 8 365281 365273 1.15948 0 - 0.0299513 2026-02-23T13:54:27.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:54:27 trial068 ceph-mon[38805]: pgmap v1663: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:54:28.054 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:54:28.172 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:54:28.273 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:54:28.271+0000 7f45fea92640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:54:28.273 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:54:28.271+0000 7f45fea92640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:54:28.285 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:54:28.285 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:54:28.291 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2462 8 365281 365273 1.15901 0 - 0.0299513 2026-02-23T13:54:28.295 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:54:28.295 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:54:28.295 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:54:28.305 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:54:28.305 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:54:28.335 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:54:28.335 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:54:28.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:54:28 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/4215797648' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:54:29.291 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2463 8 365281 365273 1.15853 0 - 0.0299513 2026-02-23T13:54:29.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:54:29 trial068 ceph-mon[38805]: pgmap v1664: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:54:30.291 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2464 8 365281 365273 1.15806 0 - 0.0299513 2026-02-23T13:54:30.336 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:54:30.456 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:54:30.554 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:54:30.552+0000 7faedde85640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:54:30.555 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:54:30.553+0000 7faedde85640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:54:30.566 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:54:30.567 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:54:30.577 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:54:30.577 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:54:30.577 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:54:30.587 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:54:30.587 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:54:30.617 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:54:30.618 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:54:31.291 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2465 8 365281 365273 1.15759 0 - 0.0299513 2026-02-23T13:54:31.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:54:31 trial068 ceph-mon[38805]: pgmap v1665: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:54:31.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:54:31 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/147240661' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:54:32.291 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2466 8 365281 365273 1.15713 0 - 0.0299513 2026-02-23T13:54:32.619 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:54:32.737 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:54:32.834 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:54:32.832+0000 7fd33237c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:54:32.834 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:54:32.832+0000 7fd33237c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:54:32.845 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:54:32.846 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:54:32.856 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:54:32.856 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:54:32.856 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:54:32.866 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:54:32.866 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:54:32.898 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:54:32.899 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:54:33.291 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2467 8 365281 365273 1.15666 0 - 0.0299513 2026-02-23T13:54:33.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:54:33 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T13:54:33.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:54:33 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T13:54:33.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:54:33 trial068 ceph-mon[38805]: pgmap v1666: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:54:33.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:54:33 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1362754589' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:54:34.265 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:54:33 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:54:33.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:54:34.291 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2468 8 365281 365273 1.15619 0 - 0.0299513 2026-02-23T13:54:34.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:54:33 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:54:33] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:54:34.899 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:54:35.016 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:54:35.115 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:54:35.113+0000 7f248309b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:54:35.115 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:54:35.113+0000 7f248309b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:54:35.127 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:54:35.127 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:54:35.137 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:54:35.138 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:54:35.138 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:54:35.148 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:54:35.148 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:54:35.177 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:54:35.177 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:54:35.292 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2469 8 365281 365273 1.15572 0 - 0.0299513 2026-02-23T13:54:35.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:54:35 trial068 ceph-mon[38805]: pgmap v1667: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:54:35.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:54:35 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3538063839' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:54:36.292 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2470 8 365281 365273 1.15525 0 - 0.0299513 2026-02-23T13:54:37.179 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:54:37.292 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2471 8 365281 365273 1.15478 0 - 0.0299513 2026-02-23T13:54:37.294 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:54:37.390 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:54:37.388+0000 7f291972a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:54:37.390 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:54:37.388+0000 7f291972a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:54:37.402 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:54:37.402 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:54:37.412 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:54:37.412 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:54:37.413 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:54:37.422 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:54:37.422 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:54:37.452 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:54:37.452 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:54:37.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:54:37 trial068 ceph-mon[38805]: pgmap v1668: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:54:37.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:54:37 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2732248998' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:54:38.292 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2472 8 365281 365273 1.15432 0 - 0.0299513 2026-02-23T13:54:39.292 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2473 8 365281 365273 1.15385 0 - 0.0299513 2026-02-23T13:54:39.454 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:54:39.570 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:54:39.672 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:54:39.670+0000 7f7691d8a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:54:39.673 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:54:39.671+0000 7f7691d8a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:54:39.684 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:54:39.685 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:54:39.695 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:54:39.695 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:54:39.695 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:54:39.705 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:54:39.705 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:54:39.733 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:54:39 trial068 ceph-mon[38805]: pgmap v1669: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:54:39.734 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:54:39.735 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:54:40.292 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2474 8 365281 365273 1.15338 0 - 0.0299513 2026-02-23T13:54:40.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:54:40 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1624331986' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:54:41.292 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2475 8 365281 365273 1.15292 0 - 0.0299513 2026-02-23T13:54:41.735 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:54:41.760 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:54:41 trial068 ceph-mon[38805]: pgmap v1670: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:54:41.852 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:54:41.946 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:54:41.944+0000 7f2be0c75640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:54:41.946 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:54:41.944+0000 7f2be0c75640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:54:41.958 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:54:41.958 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:54:41.968 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:54:41.969 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:54:41.969 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:54:41.978 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:54:41.978 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:54:42.009 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:54:42.010 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:54:42.292 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2476 8 365281 365273 1.15245 0 - 0.0299513 2026-02-23T13:54:42.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:54:42 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3956077224' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:54:43.292 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2477 8 365281 365273 1.15199 0 - 0.0299513 2026-02-23T13:54:43.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:54:43 trial068 ceph-mon[38805]: pgmap v1671: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:54:44.010 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:54:44.128 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:54:44.228 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:54:44.226+0000 7fb80d0dd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:54:44.229 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:54:43 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:54:43] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:54:44.229 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:54:44.227+0000 7fb80d0dd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:54:44.240 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:54:44.241 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:54:44.250 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:54:44.251 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:54:44.251 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:54:44.260 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:54:44.260 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:54:44.290 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:54:44.291 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:54:44.292 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2478 8 365281 365273 1.15152 0 - 0.0299513 2026-02-23T13:54:44.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:54:43 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:54:43.974Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:54:44.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:54:44 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/107210612' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:54:45.292 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2479 8 365281 365273 1.15106 0 - 0.0299513 2026-02-23T13:54:45.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:54:45 trial068 ceph-mon[38805]: pgmap v1672: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:54:46.292 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:54:46.293 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T13:54:46.292145+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T13:54:46.293 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T13:54:46.293 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2480 8 365281 365273 1.15059 0 - 0.0299513 2026-02-23T13:54:46.406 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:54:46.503 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:54:46.501+0000 7fe26241d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:54:46.504 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:54:46.502+0000 7fe26241d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:54:46.516 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:54:46.517 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:54:46.527 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:54:46.527 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:54:46.527 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:54:46.537 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:54:46.538 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:54:46.566 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:54:46 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2026-02-23T13:54:46.567 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:54:46.568 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:54:47.292 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2481 8 365281 365273 1.15013 0 - 0.0299513 2026-02-23T13:54:47.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:54:47 trial068 ceph-mon[38805]: pgmap v1673: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:54:47.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:54:47 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-23T13:54:47.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:54:47 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2026-02-23T13:54:47.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:54:47 trial068 ceph-mon[38805]: pgmap v1674: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:54:47.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:54:47 trial068 ceph-mon[38805]: pgmap v1675: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:54:47.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:54:47 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T13:54:47.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:54:47 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2026-02-23T13:54:47.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:54:47 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/796872382' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:54:48.292 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2482 8 365281 365273 1.14967 0 - 0.0299513 2026-02-23T13:54:48.569 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:54:48.687 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:54:48.793 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:54:48.791+0000 7f8dd8ce7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:54:48.793 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:54:48.791+0000 7f8dd8ce7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:54:48.805 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:54:48.805 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:54:48.816 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:54:48.817 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:54:48.817 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:54:48.827 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:54:48.827 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:54:48.857 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:54:48.858 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:54:49.293 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2483 8 365281 365273 1.1492 0 - 0.0299513 2026-02-23T13:54:49.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:54:49 trial068 ceph-mon[38805]: pgmap v1676: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:54:49.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:54:49 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/663768799' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:54:50.293 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2484 8 365281 365273 1.14874 0 - 0.0299513 2026-02-23T13:54:50.859 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:54:50.978 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:54:51.079 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:54:51.077+0000 7f04bf1cc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:54:51.079 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:54:51.078+0000 7f04bf1cc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:54:51.091 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:54:51.092 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:54:51.101 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:54:51.101 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:54:51.102 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:54:51.111 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:54:51.112 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:54:51.143 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:54:51.143 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:54:51.293 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2485 8 365281 365273 1.14828 0 - 0.0299513 2026-02-23T13:54:51.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:54:51 trial068 ceph-mon[38805]: pgmap v1677: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:54:51.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:54:51 trial068 ceph-mon[38805]: 5.a scrub starts 2026-02-23T13:54:51.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:54:51 trial068 ceph-mon[38805]: 5.a scrub ok 2026-02-23T13:54:51.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:54:51 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/748871737' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:54:52.293 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2486 8 365281 365273 1.14782 0 - 0.0299513 2026-02-23T13:54:53.145 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:54:53.263 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:54:53.293 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2487 8 365281 365273 1.14735 0 - 0.0299513 2026-02-23T13:54:53.363 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:54:53.361+0000 7f4cf9cc7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:54:53.363 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:54:53.361+0000 7f4cf9cc7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:54:53.375 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:54:53.375 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:54:53.385 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:54:53.385 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:54:53.385 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:54:53.395 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:54:53.395 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:54:53.426 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:54:53.427 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:54:53.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:54:53 trial068 ceph-mon[38805]: pgmap v1678: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:54:53.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:54:53 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1305875909' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:54:54.293 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2488 8 365281 365273 1.14689 0 - 0.0299513 2026-02-23T13:54:54.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:54:53 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:54:53] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:54:54.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:54:53 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:54:53.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:54:55.293 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2489 8 365281 365273 1.14643 0 - 0.0299513 2026-02-23T13:54:55.428 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:54:55.547 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:54:55.602 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:54:55 trial068 ceph-mon[38805]: pgmap v1679: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:54:55.648 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:54:55.646+0000 7fba37d79640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:54:55.648 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:54:55.646+0000 7fba37d79640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:54:55.660 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:54:55.660 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:54:55.670 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:54:55.670 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:54:55.671 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:54:55.681 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:54:55.681 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:54:55.711 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:54:55.711 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:54:56.293 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2490 8 365281 365273 1.14597 0 - 0.0299513 2026-02-23T13:54:56.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:54:56 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1483098159' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:54:57.293 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2491 8 365281 365273 1.14551 0 - 0.0299513 2026-02-23T13:54:57.712 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:54:57.736 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:54:57 trial068 ceph-mon[38805]: pgmap v1680: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:54:57.830 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:54:57.934 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:54:57.932+0000 7fad4e1e9640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:54:57.935 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:54:57.933+0000 7fad4e1e9640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:54:57.946 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:54:57.946 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:54:57.956 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:54:57.957 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:54:57.957 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:54:57.967 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:54:57.968 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:54:57.997 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:54:57.998 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:54:58.294 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2492 8 365281 365273 1.14505 0 - 0.0299513 2026-02-23T13:54:58.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:54:58 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1345957134' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:54:59.294 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2493 8 365281 365273 1.14459 0 - 0.0299513 2026-02-23T13:54:59.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:54:59 trial068 ceph-mon[38805]: pgmap v1681: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:54:59.998 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:55:00.114 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:55:00.211 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:55:00.209+0000 7f108851e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:55:00.211 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:55:00.209+0000 7f108851e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:55:00.223 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:55:00.223 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:55:00.233 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:55:00.234 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:55:00.234 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:55:00.244 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:55:00.244 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:55:00.274 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:55:00.274 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:55:00.294 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2494 8 365281 365273 1.14413 0 - 0.0299513 2026-02-23T13:55:00.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:55:00 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2874516309' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:55:01.294 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2495 8 365281 365273 1.14368 0 - 0.0299513 2026-02-23T13:55:01.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:55:01 trial068 ceph-mon[38805]: pgmap v1682: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:55:02.275 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:55:02.294 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2496 8 365281 365273 1.14322 0 - 0.0299513 2026-02-23T13:55:02.392 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:55:02.499 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:55:02.497+0000 7f28351d5640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:55:02.499 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:55:02.497+0000 7f28351d5640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:55:02.511 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:55:02.511 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:55:02.521 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:55:02.521 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:55:02.522 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:55:02.531 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:55:02.531 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:55:02.561 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:55:02 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T13:55:02.561 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:55:02 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T13:55:02.562 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:55:02.562 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:55:03.294 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2497 8 365281 365273 1.14276 0 - 0.0299513 2026-02-23T13:55:03.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:55:03 trial068 ceph-mon[38805]: pgmap v1683: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:55:03.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:55:03 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/320983284' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:55:04.266 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:55:03 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:55:03.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:55:04.294 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2498 8 365281 365273 1.1423 0 - 0.0299513 2026-02-23T13:55:04.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:55:03 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:55:03] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:55:04.563 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:55:04.680 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:55:04.781 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:55:04.779+0000 7ff6d8fd7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:55:04.781 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:55:04.779+0000 7ff6d8fd7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:55:04.792 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:55:04.792 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:55:04.802 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:55:04.802 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:55:04.803 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:55:04.813 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:55:04.813 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:55:04.843 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:55:04.844 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:55:05.294 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2499 8 365281 365273 1.14185 0 - 0.0299513 2026-02-23T13:55:05.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:55:05 trial068 ceph-mon[38805]: pgmap v1684: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:55:05.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:55:05 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1251207182' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:55:06.294 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T13:55:06.293975+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T13:55:06.294 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T13:55:06.295 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2500 8 365281 365273 1.14139 0 - 0.0299513 2026-02-23T13:55:06.845 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:55:06.964 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:55:07.065 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:55:07.063+0000 7fa8cd41d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:55:07.065 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:55:07.063+0000 7fa8cd41d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:55:07.077 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:55:07.077 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:55:07.087 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:55:07.088 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:55:07.088 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:55:07.098 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:55:07.098 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:55:07.127 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:55:07.127 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:55:07.294 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2501 8 365281 365273 1.14093 0 - 0.0299513 2026-02-23T13:55:07.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:55:07 trial068 ceph-mon[38805]: pgmap v1685: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:55:07.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:55:07 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/914626737' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:55:08.294 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2502 8 365281 365273 1.14048 0 - 0.0299513 2026-02-23T13:55:08.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:55:08 trial068 ceph-mon[38805]: osdmap e212: 4 total, 3 up, 4 in 2026-02-23T13:55:09.128 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:55:09.246 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:55:09.294 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2503 8 365281 365273 1.14002 0 - 0.0299513 2026-02-23T13:55:09.349 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:55:09.347+0000 7fa8c7bf7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:55:09.349 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:55:09.348+0000 7fa8c7bf7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:55:09.361 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:55:09.362 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:55:09.371 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:55:09.372 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:55:09.372 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:55:09.382 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:55:09.382 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:55:09.409 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:55:09.410 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:55:09.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:55:09 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T13:55:09.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:55:09 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T13:55:09.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:55:09 trial068 ceph-mon[38805]: pgmap v1687: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:55:09.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:55:09 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2474828061' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:55:10.294 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2504 8 365281 365273 1.13957 0 - 0.0299513 2026-02-23T13:55:10.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:55:10 trial068 ceph-mon[38805]: 5.5 scrub starts 2026-02-23T13:55:10.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:55:10 trial068 ceph-mon[38805]: 5.5 scrub ok 2026-02-23T13:55:11.295 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2505 8 365281 365273 1.13911 0 - 0.0299513 2026-02-23T13:55:11.411 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:55:11.527 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:55:11.623 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:55:11.621+0000 7f964d1e0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:55:11.623 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:55:11.621+0000 7f964d1e0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:55:11.635 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:55:11.636 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:55:11.646 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:55:11.647 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:55:11.647 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:55:11.656 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:55:11.657 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:55:11.686 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:55:11.687 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:55:11.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:55:11 trial068 ceph-mon[38805]: pgmap v1688: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:55:12.295 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2506 8 365281 365273 1.13866 0 - 0.0299513 2026-02-23T13:55:12.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:55:12 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2920256096' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:55:13.295 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2507 8 365281 365273 1.1382 0 - 0.0299513 2026-02-23T13:55:13.688 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:55:13.806 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:55:13.828 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:55:13 trial068 ceph-mon[38805]: pgmap v1689: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:55:13.912 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:55:13.910+0000 7f6c83348640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:55:13.912 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:55:13.910+0000 7f6c83348640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:55:13.924 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:55:13.924 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:55:13.934 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:55:13.934 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:55:13.935 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:55:13.945 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:55:13.945 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:55:13.975 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:55:13.975 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:55:14.093 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:55:13 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:55:13] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:55:14.295 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2508 8 365281 365273 1.13775 0 - 0.0299513 2026-02-23T13:55:14.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:55:13 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:55:13.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:55:14.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:55:14 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/57303699' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:55:15.295 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2509 8 365281 365273 1.13729 0 - 0.0299513 2026-02-23T13:55:15.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:55:15 trial068 ceph-mon[38805]: pgmap v1690: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:55:15.977 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:55:16.093 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:55:16.192 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:55:16.190+0000 7ffb2c807640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:55:16.192 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:55:16.190+0000 7ffb2c807640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:55:16.204 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:55:16.205 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:55:16.215 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:55:16.215 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:55:16.215 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:55:16.225 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:55:16.225 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:55:16.254 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:55:16.255 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:55:16.295 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2510 8 365281 365273 1.13684 0 - 0.0299513 2026-02-23T13:55:16.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:55:16 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3620458496' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:55:17.295 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2511 8 365281 365273 1.13639 0 - 0.0299513 2026-02-23T13:55:17.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:55:17 trial068 ceph-mon[38805]: pgmap v1691: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:55:18.256 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:55:18.295 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2512 8 365281 365273 1.13594 0 - 0.0299513 2026-02-23T13:55:18.374 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:55:18.478 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:55:18.476+0000 7f50f46e7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:55:18.479 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:55:18.477+0000 7f50f46e7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:55:18.490 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:55:18.491 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:55:18.501 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:55:18.501 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:55:18.501 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:55:18.510 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:55:18.510 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:55:18.542 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:55:18.543 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:55:18.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:55:18 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1064583822' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:55:19.295 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2513 8 365281 365273 1.13548 0 - 0.0299513 2026-02-23T13:55:19.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:55:19 trial068 ceph-mon[38805]: pgmap v1692: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:55:20.295 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2514 8 365281 365273 1.13503 0 - 0.0299513 2026-02-23T13:55:20.544 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:55:20.665 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:55:20.770 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:55:20.768+0000 7f27725aa640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:55:20.771 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:55:20.768+0000 7f27725aa640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:55:20.782 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:55:20.782 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:55:20.792 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:55:20.793 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:55:20.793 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:55:20.803 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:55:20.804 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:55:20.833 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:55:20.834 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:55:21.295 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2515 8 365281 365273 1.13458 0 - 0.0299513 2026-02-23T13:55:21.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:55:21 trial068 ceph-mon[38805]: pgmap v1693: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:55:21.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:55:21 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2698797982' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:55:22.295 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2516 8 365281 365273 1.13413 0 - 0.0299513 2026-02-23T13:55:22.835 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:55:22.951 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:55:23.056 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:55:23.054+0000 7f644daef640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:55:23.056 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:55:23.054+0000 7f644daef640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:55:23.069 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:55:23.069 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:55:23.079 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:55:23.079 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:55:23.080 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:55:23.089 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:55:23.090 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:55:23.119 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:55:23.120 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:55:23.295 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2517 8 365281 365273 1.13368 0 - 0.0299513 2026-02-23T13:55:23.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:55:23 trial068 ceph-mon[38805]: pgmap v1694: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:55:23.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:55:23 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1406974586' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:55:24.296 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2518 8 365281 365273 1.13323 0 - 0.0299513 2026-02-23T13:55:24.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:55:23 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:55:23] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:55:24.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:55:23 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:55:23.974Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:55:25.121 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:55:25.242 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:55:25.296 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2519 8 365281 365273 1.13278 0 - 0.0299513 2026-02-23T13:55:25.344 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:55:25.342+0000 7fbf773c6640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:55:25.344 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:55:25.342+0000 7fbf773c6640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:55:25.355 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:55:25.355 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:55:25.365 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:55:25.366 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:55:25.366 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:55:25.377 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:55:25.377 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:55:25.408 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:55:25.409 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:55:25.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:55:25 trial068 ceph-mon[38805]: pgmap v1695: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:55:25.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:55:25 trial068 ceph-mon[38805]: 5.e scrub starts 2026-02-23T13:55:25.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:55:25 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3341015401' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:55:26.296 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T13:55:26.295555+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T13:55:26.296 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T13:55:26.296 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2520 8 365281 365273 1.13233 0 - 0.0299513 2026-02-23T13:55:26.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:55:26 trial068 ceph-mon[38805]: 5.e scrub ok 2026-02-23T13:55:27.296 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2521 8 365281 365273 1.13188 0 - 0.0299513 2026-02-23T13:55:27.410 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:55:27.531 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:55:27.632 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:55:27.630+0000 7f73eb4f4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:55:27.632 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:55:27.630+0000 7f73eb4f4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:55:27.644 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:55:27.644 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:55:27.654 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:55:27.655 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:55:27.655 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:55:27.665 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:55:27.666 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:55:27.697 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:55:27 trial068 ceph-mon[38805]: pgmap v1696: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:55:27.698 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:55:27.699 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:55:28.296 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2522 8 365281 365273 1.13143 0 - 0.0299513 2026-02-23T13:55:28.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:55:28 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1609850264' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:55:29.296 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2523 8 365281 365273 1.13098 0 - 0.0299513 2026-02-23T13:55:29.700 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:55:29.817 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:55:29.839 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:55:29 trial068 ceph-mon[38805]: pgmap v1697: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:55:29.929 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:55:29.927+0000 7fc04b5c1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:55:29.929 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:55:29.927+0000 7fc04b5c1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:55:29.941 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:55:29.941 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:55:29.951 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:55:29.952 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:55:29.952 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:55:29.961 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:55:29.962 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:55:29.993 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:55:29.993 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:55:30.296 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2524 8 365281 365273 1.13054 0 - 0.0299513 2026-02-23T13:55:30.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:55:30 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2413865230' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:55:31.296 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2525 8 365281 365273 1.13009 0 - 0.0299513 2026-02-23T13:55:31.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:55:31 trial068 ceph-mon[38805]: pgmap v1698: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:55:31.995 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:55:32.112 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:55:32.218 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:55:32.216+0000 7fb1c835d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:55:32.218 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:55:32.216+0000 7fb1c835d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:55:32.230 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:55:32.230 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:55:32.240 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:55:32.240 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:55:32.240 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:55:32.250 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:55:32.250 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:55:32.281 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:55:32.282 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:55:32.296 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2526 8 365281 365273 1.12964 0 - 0.0299513 2026-02-23T13:55:32.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:55:32 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1885119363' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:55:33.296 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2527 8 365281 365273 1.12919 0 - 0.0299513 2026-02-23T13:55:33.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:55:33 trial068 ceph-mon[38805]: pgmap v1699: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:55:34.282 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:55:34.296 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2528 8 365281 365273 1.12875 0 - 0.0299513 2026-02-23T13:55:34.305 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:55:33 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:55:33] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:55:34.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:55:33 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:55:33.974Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:55:34.400 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:55:34.502 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:55:34.500+0000 7fbb1eb21640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:55:34.503 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:55:34.501+0000 7fbb1eb21640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:55:34.514 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:55:34.515 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:55:34.524 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:55:34.525 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:55:34.525 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:55:34.535 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:55:34.535 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:55:34.566 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:55:34.566 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:55:35.296 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2529 8 365281 365273 1.1283 0 - 0.0299513 2026-02-23T13:55:35.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:55:35 trial068 ceph-mon[38805]: pgmap v1700: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:55:35.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:55:35 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1154095698' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:55:36.296 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2530 8 365281 365273 1.12785 0 - 0.0299513 2026-02-23T13:55:36.567 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:55:36.685 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:55:36.790 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:55:36.788+0000 7f1c8a436640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:55:36.790 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:55:36.788+0000 7f1c8a436640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:55:36.802 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:55:36.802 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:55:36.812 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:55:36.812 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:55:36.812 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:55:36.822 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:55:36.823 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:55:36.850 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:55:36.851 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:55:37.297 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2531 8 365281 365273 1.12741 0 - 0.0299513 2026-02-23T13:55:37.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:55:37 trial068 ceph-mon[38805]: pgmap v1701: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:55:37.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:55:37 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2710968834' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:55:38.297 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2532 8 365281 365273 1.12696 0 - 0.0299513 2026-02-23T13:55:38.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:55:38 trial068 ceph-mon[38805]: 5.8 scrub starts 2026-02-23T13:55:38.852 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:55:38.968 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:55:39.065 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:55:39.063+0000 7fc6546f9640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:55:39.065 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:55:39.063+0000 7fc6546f9640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:55:39.076 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:55:39.077 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:55:39.086 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:55:39.087 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:55:39.087 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:55:39.096 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:55:39.097 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:55:39.125 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:55:39.126 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:55:39.297 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2533 8 365281 365273 1.12652 0 - 0.0299513 2026-02-23T13:55:39.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:55:39 trial068 ceph-mon[38805]: 5.8 scrub ok 2026-02-23T13:55:39.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:55:39 trial068 ceph-mon[38805]: pgmap v1702: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:55:39.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:55:39 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3459497343' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:55:40.297 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2534 8 365281 365273 1.12607 0 - 0.0299513 2026-02-23T13:55:40.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:55:40 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T13:55:40.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:55:40 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T13:55:41.126 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:55:41.243 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:55:41.297 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2535 8 365281 365273 1.12563 0 - 0.0299513 2026-02-23T13:55:41.341 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:55:41.339+0000 7ff24997a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:55:41.341 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:55:41.339+0000 7ff24997a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:55:41.353 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:55:41.353 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:55:41.363 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:55:41.364 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:55:41.364 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:55:41.373 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:55:41.374 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:55:41.404 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:55:41.405 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:55:41.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:55:41 trial068 ceph-mon[38805]: pgmap v1703: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:55:41.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:55:41 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3003828394' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:55:42.297 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2536 8 365281 365273 1.12519 0 - 0.0299513 2026-02-23T13:55:43.297 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2537 8 365281 365273 1.12474 0 - 0.0299513 2026-02-23T13:55:43.406 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:55:43.526 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:55:43.630 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:55:43.628+0000 7f22385bf640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:55:43.630 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:55:43.628+0000 7f22385bf640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:55:43.642 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:55:43.642 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:55:43.652 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:55:43.653 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:55:43.654 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:55:43.662 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:55:43.662 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:55:43.692 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:55:43 trial068 ceph-mon[38805]: pgmap v1704: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:55:43.693 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:55:43.693 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:55:44.297 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2538 8 365281 365273 1.1243 0 - 0.0299513 2026-02-23T13:55:44.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:55:43 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:55:43] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:55:44.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:55:43 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:55:43.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:55:44.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:55:44 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/47234615' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:55:45.297 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2539 8 365281 365273 1.12386 0 - 0.0299513 2026-02-23T13:55:45.694 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:55:45.810 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:55:45.832 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:55:45 trial068 ceph-mon[38805]: pgmap v1705: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:55:45.912 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:55:45.910+0000 7f19a4ab2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:55:45.912 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:55:45.910+0000 7f19a4ab2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:55:45.924 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:55:45.925 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:55:45.934 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:55:45.935 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:55:45.935 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:55:45.945 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:55:45.946 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:55:45.976 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:55:45.976 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:55:46.297 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T13:55:46.297121+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T13:55:46.298 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T13:55:46.298 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2540 8 365281 365273 1.12341 0 - 0.0299513 2026-02-23T13:55:46.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:55:46 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/701268968' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:55:46.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:55:46 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2026-02-23T13:55:47.297 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2541 8 365281 365273 1.12297 0 - 0.0299513 2026-02-23T13:55:47.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:55:47 trial068 ceph-mon[38805]: pgmap v1706: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:55:47.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:55:47 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-23T13:55:47.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:55:47 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2026-02-23T13:55:47.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:55:47 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T13:55:47.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:55:47 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2026-02-23T13:55:47.977 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:55:48.093 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:55:48.190 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:55:48.188+0000 7f934f8b5640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:55:48.190 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:55:48.188+0000 7f934f8b5640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:55:48.202 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:55:48.202 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:55:48.212 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:55:48.213 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:55:48.213 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:55:48.224 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:55:48.225 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:55:48.256 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:55:48.257 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:55:48.297 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2542 8 365281 365273 1.12253 0 - 0.0299513 2026-02-23T13:55:48.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:55:48 trial068 ceph-mon[38805]: pgmap v1707: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:55:48.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:55:48 trial068 ceph-mon[38805]: pgmap v1708: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:55:48.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:55:48 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/4243267323' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:55:49.298 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2543 8 365281 365273 1.12209 0 - 0.0299513 2026-02-23T13:55:50.258 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:55:50.298 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2544 8 365281 365273 1.12165 0 - 0.0299513 2026-02-23T13:55:50.378 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:55:50.479 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:55:50.477+0000 7ff8b1fe5640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:55:50.479 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:55:50.477+0000 7ff8b1fe5640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:55:50.490 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:55:50.491 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:55:50.500 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:55:50.501 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:55:50.501 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:55:50.511 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:55:50.511 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:55:50.542 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:55:50.543 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:55:50.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:55:50 trial068 ceph-mon[38805]: pgmap v1709: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:55:50.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:55:50 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1927689585' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:55:51.298 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2545 8 365281 365273 1.12121 0 - 0.0299513 2026-02-23T13:55:52.298 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2546 8 365281 365273 1.12077 0 - 0.0299513 2026-02-23T13:55:52.544 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:55:52.663 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:55:52.768 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:55:52.765+0000 7f3b5a99c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:55:52.768 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:55:52.766+0000 7f3b5a99c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:55:52.780 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:55:52.780 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:55:52.790 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:55:52.791 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:55:52.791 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:55:52.801 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:55:52.801 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:55:52.830 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:55:52 trial068 ceph-mon[38805]: pgmap v1710: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:55:52.831 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:55:52.832 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:55:53.298 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2547 8 365281 365273 1.12033 0 - 0.0299513 2026-02-23T13:55:53.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:55:53 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1708065344' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:55:54.298 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2548 8 365281 365273 1.11989 0 - 0.0299513 2026-02-23T13:55:54.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:55:53 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:55:53] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:55:54.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:55:53 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:55:53.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:55:54.833 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:55:54.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:55:54 trial068 ceph-mon[38805]: pgmap v1711: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:55:54.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:55:54 trial068 ceph-mon[38805]: 5.7 scrub starts 2026-02-23T13:55:54.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:55:54 trial068 ceph-mon[38805]: 5.7 scrub ok 2026-02-23T13:55:54.952 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:55:55.049 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:55:55.047+0000 7f438c74a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:55:55.050 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:55:55.048+0000 7f438c74a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:55:55.061 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:55:55.061 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:55:55.071 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:55:55.071 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:55:55.072 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:55:55.082 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:55:55.082 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:55:55.113 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:55:55.114 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:55:55.298 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2549 8 365281 365273 1.11945 0 - 0.0299513 2026-02-23T13:55:55.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:55:55 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/480704179' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:55:56.298 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2550 8 365281 365273 1.11901 0 - 0.0299513 2026-02-23T13:55:56.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:55:56 trial068 ceph-mon[38805]: pgmap v1712: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:55:57.114 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:55:57.231 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:55:57.298 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2551 8 365281 365273 1.11857 0 - 0.0299513 2026-02-23T13:55:57.329 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:55:57.327+0000 7fd773297640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:55:57.329 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:55:57.328+0000 7fd773297640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:55:57.341 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:55:57.341 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:55:57.351 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:55:57.351 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:55:57.351 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:55:57.361 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:55:57.361 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:55:57.390 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:55:57.391 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:55:57.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:55:57 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1724848754' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:55:58.298 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2552 8 365281 365273 1.11813 0 - 0.0299513 2026-02-23T13:55:58.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:55:58 trial068 ceph-mon[38805]: pgmap v1713: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:55:59.298 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2553 8 365281 365273 1.11769 0 - 0.0299513 2026-02-23T13:55:59.392 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:55:59.514 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:55:59.611 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:55:59.609+0000 7f8c0bb37640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:55:59.611 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:55:59.609+0000 7f8c0bb37640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:55:59.623 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:55:59.624 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:55:59.633 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:55:59.634 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:55:59.634 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:55:59.644 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:55:59.645 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:55:59.675 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:55:59.676 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:56:00.298 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2554 8 365281 365273 1.11726 0 - 0.0299513 2026-02-23T13:56:00.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:56:00 trial068 ceph-mon[38805]: pgmap v1714: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:56:00.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:56:00 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1788791212' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:56:01.298 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2555 8 365281 365273 1.11682 0 - 0.0299513 2026-02-23T13:56:01.676 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:56:01.798 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:56:01.897 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:56:01.895+0000 7f11e59a5640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:56:01.897 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:56:01.895+0000 7f11e59a5640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:56:01.909 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:56:01.909 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:56:01.919 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:56:01.920 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:56:01.920 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:56:01.930 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:56:01.930 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:56:01.957 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:56:01.958 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:56:02.299 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2556 8 365281 365273 1.11638 0 - 0.0299513 2026-02-23T13:56:02.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:56:02 trial068 ceph-mon[38805]: pgmap v1715: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:56:02.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:56:02 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/4008502479' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:56:03.299 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2557 8 365281 365273 1.11594 0 - 0.0299513 2026-02-23T13:56:03.959 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:56:04.076 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:56:04.172 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:56:04.170+0000 7f48413b2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:56:04.172 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:56:04.170+0000 7f48413b2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:56:04.183 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:56:04.184 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:56:04.194 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:56:04.194 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:56:04.194 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:56:04.204 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:56:04.204 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:56:04.233 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:56:03 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:56:03] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:56:04.234 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:56:04.235 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:56:04.266 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:56:03 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:56:03.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:56:04.299 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2558 8 365281 365273 1.11551 0 - 0.0299513 2026-02-23T13:56:04.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:56:04 trial068 ceph-mon[38805]: pgmap v1716: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:56:04.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:56:04 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/713883499' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:56:05.299 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2559 8 365281 365273 1.11507 0 - 0.0299513 2026-02-23T13:56:06.236 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:56:06.299 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T13:56:06.299031+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T13:56:06.299 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T13:56:06.300 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2560 8 365281 365273 1.11464 0 - 0.0299513 2026-02-23T13:56:06.355 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:56:06.457 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:56:06.455+0000 7f400ab76640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:56:06.457 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:56:06.455+0000 7f400ab76640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:56:06.469 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:56:06.470 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:56:06.480 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:56:06.481 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:56:06.481 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:56:06.490 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:56:06.491 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:56:06.520 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:56:06.521 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:56:06.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:56:06 trial068 ceph-mon[38805]: pgmap v1717: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:56:06.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:56:06 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2489893092' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:56:07.299 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2561 8 365281 365273 1.1142 0 - 0.0299513 2026-02-23T13:56:08.299 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2562 8 365281 365273 1.11377 0 - 0.0299513 2026-02-23T13:56:08.522 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:56:08.639 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:56:08.739 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:56:08.737+0000 7f739d53e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:56:08.739 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:56:08.737+0000 7f739d53e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:56:08.752 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:56:08.752 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:56:08.763 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:56:08.763 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:56:08.763 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:56:08.773 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:56:08.773 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:56:08.800 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:56:08 trial068 ceph-mon[38805]: pgmap v1718: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:56:08.801 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:56:08.802 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:56:09.300 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2563 8 365281 365273 1.11333 0 - 0.0299513 2026-02-23T13:56:09.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:56:09 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1779971652' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:56:10.300 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2564 8 365281 365273 1.1129 0 - 0.0299513 2026-02-23T13:56:10.803 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:56:10.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:56:10 trial068 ceph-mon[38805]: pgmap v1719: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:56:10.922 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:56:11.029 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:56:11.027+0000 7f03e817e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:56:11.029 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:56:11.027+0000 7f03e817e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:56:11.041 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:56:11.041 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:56:11.051 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:56:11.052 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:56:11.052 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:56:11.061 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:56:11.061 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:56:11.091 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:56:11.091 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:56:11.300 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2565 8 365281 365273 1.11246 0 - 0.0299513 2026-02-23T13:56:11.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:56:11 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2009793079' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:56:12.300 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2566 8 365281 365273 1.11203 0 - 0.0299513 2026-02-23T13:56:12.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:56:12 trial068 ceph-mon[38805]: pgmap v1720: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:56:12.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:56:12 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T13:56:12.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:56:12 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T13:56:13.093 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:56:13.211 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:56:13.300 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2567 8 365281 365273 1.1116 0 - 0.0299513 2026-02-23T13:56:13.310 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:56:13.308+0000 7ff09d1b9640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:56:13.310 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:56:13.308+0000 7ff09d1b9640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:56:13.322 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:56:13.322 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:56:13.332 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:56:13.332 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:56:13.332 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:56:13.342 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:56:13.342 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:56:13.372 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:56:13.373 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:56:13.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:56:13 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2239241699' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:56:14.300 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2568 8 365281 365273 1.11116 0 - 0.0299513 2026-02-23T13:56:14.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:56:13 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:56:13] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:56:14.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:56:13 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:56:13.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:56:15.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:56:14 trial068 ceph-mon[38805]: pgmap v1721: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:56:15.300 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2569 8 365281 365273 1.11073 0 - 0.0299513 2026-02-23T13:56:15.373 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:56:15.490 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:56:15.598 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:56:15.595+0000 7f612be18640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:56:15.598 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:56:15.596+0000 7f612be18640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:56:15.610 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:56:15.610 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:56:15.620 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:56:15.621 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:56:15.621 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:56:15.631 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:56:15.631 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:56:15.660 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:56:15.661 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:56:16.300 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2570 8 365281 365273 1.1103 0 - 0.0299513 2026-02-23T13:56:17.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:56:16 trial068 ceph-mon[38805]: pgmap v1722: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:56:17.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:56:16 trial068 ceph-mon[38805]: osdmap e213: 4 total, 3 up, 4 in 2026-02-23T13:56:17.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:56:16 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/583467144' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:56:17.300 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2571 8 365281 365273 1.10987 0 - 0.0299513 2026-02-23T13:56:17.662 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:56:17.782 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:56:17.881 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:56:17.879+0000 7f4c2e6a3640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:56:17.881 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:56:17.879+0000 7f4c2e6a3640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:56:17.893 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:56:17.894 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:56:17.904 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:56:17.904 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:56:17.905 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:56:17.914 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:56:17.915 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:56:17.944 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:56:17.945 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:56:18.301 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2572 8 365281 365273 1.10944 0 - 0.0299513 2026-02-23T13:56:19.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:56:18 trial068 ceph-mon[38805]: pgmap v1724: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:56:19.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:56:18 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2495801148' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:56:19.301 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2573 8 365281 365273 1.10901 0 - 0.0299513 2026-02-23T13:56:19.946 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:56:20.065 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:56:20.170 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:56:20.166+0000 7f32d6120640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:56:20.170 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:56:20.166+0000 7f32d6120640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:56:20.180 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:56:20.181 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:56:20.191 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:56:20.191 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:56:20.191 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:56:20.201 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:56:20.201 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:56:20.230 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:56:20.231 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:56:20.301 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2574 8 365281 365273 1.10857 0 - 0.0299513 2026-02-23T13:56:21.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:56:20 trial068 ceph-mon[38805]: pgmap v1725: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:56:21.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:56:20 trial068 ceph-mon[38805]: 5.a scrub starts 2026-02-23T13:56:21.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:56:20 trial068 ceph-mon[38805]: 5.a scrub ok 2026-02-23T13:56:21.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:56:20 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2206222261' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:56:21.301 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2575 8 365281 365273 1.10814 0 - 0.0299513 2026-02-23T13:56:22.232 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:56:22.301 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2576 8 365281 365273 1.10771 0 - 0.0299513 2026-02-23T13:56:22.350 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:56:22.448 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:56:22.446+0000 7f9c9365a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:56:22.448 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:56:22.446+0000 7f9c9365a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:56:22.459 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:56:22.460 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:56:22.470 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:56:22.470 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:56:22.470 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:56:22.480 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:56:22.480 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:56:22.510 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:56:22.511 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:56:22.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:56:22 trial068 ceph-mon[38805]: pgmap v1726: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:56:22.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:56:22 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1168016685' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:56:23.301 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2577 8 365281 365273 1.10728 0 - 0.0299513 2026-02-23T13:56:24.301 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2578 8 365281 365273 1.10685 0 - 0.0299513 2026-02-23T13:56:24.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:56:23 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:56:23] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:56:24.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:56:23 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:56:23.974Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:56:24.512 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:56:24.630 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:56:24.731 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:56:24.729+0000 7f996f594640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:56:24.731 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:56:24.729+0000 7f996f594640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:56:24.743 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:56:24.743 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:56:24.754 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:56:24.754 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:56:24.754 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:56:24.765 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:56:24.765 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:56:24.796 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:56:24.797 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:56:25.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:56:24 trial068 ceph-mon[38805]: pgmap v1727: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:56:25.301 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2579 8 365281 365273 1.10643 0 - 0.0299513 2026-02-23T13:56:26.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:56:25 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3545986772' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:56:26.301 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T13:56:26.301040+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T13:56:26.302 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T13:56:26.302 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2580 8 365281 365273 1.106 0 - 0.0299513 2026-02-23T13:56:26.798 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:56:26.916 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:56:26.937 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:56:26 trial068 ceph-mon[38805]: pgmap v1728: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:56:26.937 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:56:26 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T13:56:26.938 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:56:26 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T13:56:27.024 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:56:27.022+0000 7f3ba99b7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:56:27.025 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:56:27.023+0000 7f3ba99b7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:56:27.037 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:56:27.037 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:56:27.046 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:56:27.047 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:56:27.047 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:56:27.057 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:56:27.057 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:56:27.085 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:56:27.086 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:56:27.301 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2581 8 365281 365273 1.10557 0 - 0.0299513 2026-02-23T13:56:28.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:56:27 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3525772819' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:56:28.301 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2582 8 365281 365273 1.10514 0 - 0.0299513 2026-02-23T13:56:29.087 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:56:29.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:56:28 trial068 ceph-mon[38805]: pgmap v1729: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:56:29.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:56:28 trial068 ceph-mon[38805]: 5.5 scrub starts 2026-02-23T13:56:29.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:56:28 trial068 ceph-mon[38805]: 5.5 scrub ok 2026-02-23T13:56:29.205 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:56:29.299 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:56:29.297+0000 7f8193f99640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:56:29.299 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:56:29.297+0000 7f8193f99640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:56:29.301 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2583 8 365281 365273 1.10471 0 - 0.0299513 2026-02-23T13:56:29.311 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:56:29.311 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:56:29.321 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:56:29.322 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:56:29.322 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:56:29.332 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:56:29.333 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:56:29.362 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:56:29.363 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:56:30.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:56:29 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2452569296' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:56:30.302 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2584 8 365281 365273 1.10428 0 - 0.0299513 2026-02-23T13:56:31.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:56:30 trial068 ceph-mon[38805]: pgmap v1730: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:56:31.302 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2585 8 365281 365273 1.10386 0 - 0.0299513 2026-02-23T13:56:31.364 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:56:31.484 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:56:31.585 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:56:31.583+0000 7f2b8c83b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:56:31.585 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:56:31.583+0000 7f2b8c83b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:56:31.597 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:56:31.597 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:56:31.607 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:56:31.608 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:56:31.608 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:56:31.617 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:56:31.618 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:56:31.646 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:56:31 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3142133940' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:56:31.648 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:56:31.649 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:56:32.302 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2586 8 365281 365273 1.10343 0 - 0.0299513 2026-02-23T13:56:33.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:56:32 trial068 ceph-mon[38805]: pgmap v1731: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:56:33.302 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2587 8 365281 365273 1.103 0 - 0.0299513 2026-02-23T13:56:33.649 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:56:33.768 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:56:33.871 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:56:33.869+0000 7f5c27299640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:56:33.871 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:56:33.869+0000 7f5c27299640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:56:33.883 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:56:33.883 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:56:33.894 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:56:33.895 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:56:33.895 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:56:33.905 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:56:33.905 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:56:33.937 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:56:33.938 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:56:34.266 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:56:33 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:56:33.972Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:56:34.302 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2588 8 365281 365273 1.10258 0 - 0.0299513 2026-02-23T13:56:34.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:56:33 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:56:33] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:56:35.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:56:34 trial068 ceph-mon[38805]: pgmap v1732: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:56:35.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:56:34 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3053718337' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:56:35.302 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2589 8 365281 365273 1.10215 0 - 0.0299513 2026-02-23T13:56:35.939 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:56:36.057 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:56:36.158 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:56:36.156+0000 7ff8f314c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:56:36.158 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:56:36.156+0000 7ff8f314c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:56:36.170 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:56:36.171 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:56:36.181 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:56:36.181 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:56:36.181 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:56:36.191 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:56:36.191 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:56:36.219 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:56:36.220 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:56:36.302 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2590 8 365281 365273 1.10173 0 - 0.0299513 2026-02-23T13:56:37.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:56:36 trial068 ceph-mon[38805]: pgmap v1733: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:56:37.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:56:36 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/993410646' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:56:37.302 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2591 8 365281 365273 1.1013 0 - 0.0299513 2026-02-23T13:56:38.220 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:56:38.302 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2592 8 365281 365273 1.10088 0 - 0.0299513 2026-02-23T13:56:38.339 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:56:38.436 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:56:38.434+0000 7f115059a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:56:38.437 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:56:38.434+0000 7f115059a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:56:38.448 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:56:38.448 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:56:38.458 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:56:38.459 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:56:38.459 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:56:38.468 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:56:38.468 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:56:38.498 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:56:38.498 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:56:38.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:56:38 trial068 ceph-mon[38805]: pgmap v1734: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:56:38.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:56:38 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3067171318' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:56:39.302 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2593 8 365281 365273 1.10045 0 - 0.0299513 2026-02-23T13:56:40.302 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2594 8 365281 365273 1.10003 0 - 0.0299513 2026-02-23T13:56:40.499 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:56:40.617 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:56:40.718 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:56:40.716+0000 7fadde771640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:56:40.718 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:56:40.716+0000 7fadde771640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:56:40.730 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:56:40.731 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:56:40.741 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:56:40.742 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:56:40.742 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:56:40.751 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:56:40.752 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:56:40.781 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:56:40 trial068 ceph-mon[38805]: pgmap v1735: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:56:40.782 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:56:40.782 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:56:41.302 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2595 8 365281 365273 1.0996 0 - 0.0299513 2026-02-23T13:56:42.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:56:41 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3238198786' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:56:42.302 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2596 8 365281 365273 1.09918 0 - 0.0299513 2026-02-23T13:56:42.783 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:56:42.901 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:56:42.923 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:56:42 trial068 ceph-mon[38805]: pgmap v1736: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:56:43.009 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:56:43.006+0000 7f5f1f1bf640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:56:43.009 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:56:43.007+0000 7f5f1f1bf640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:56:43.021 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:56:43.021 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:56:43.032 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:56:43.032 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:56:43.032 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:56:43.044 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:56:43.044 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:56:43.076 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:56:43.077 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:56:43.303 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2597 8 365281 365273 1.09876 0 - 0.0299513 2026-02-23T13:56:43.975 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:56:43 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3713368926' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:56:44.303 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2598 8 365281 365273 1.09833 0 - 0.0299513 2026-02-23T13:56:44.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:56:43 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:56:43] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:56:44.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:56:43 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:56:43.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:56:45.078 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:56:45.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:56:44 trial068 ceph-mon[38805]: pgmap v1737: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:56:45.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:56:44 trial068 ceph-mon[38805]: 5.8 scrub starts 2026-02-23T13:56:45.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:56:44 trial068 ceph-mon[38805]: 5.8 scrub ok 2026-02-23T13:56:45.196 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:56:45.299 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:56:45.297+0000 7fb061eff640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:56:45.299 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:56:45.297+0000 7fb061eff640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:56:45.303 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2599 8 365281 365273 1.09791 0 - 0.0299513 2026-02-23T13:56:45.311 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:56:45.311 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:56:45.321 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:56:45.322 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:56:45.322 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:56:45.332 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:56:45.332 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:56:45.362 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:56:45.362 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:56:46.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:56:45 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1258470567' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:56:46.303 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T13:56:46.302635+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T13:56:46.303 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T13:56:46.303 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2600 8 365281 365273 1.09749 0 - 0.0299513 2026-02-23T13:56:47.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:56:46 trial068 ceph-mon[38805]: pgmap v1738: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:56:47.303 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2601 8 365281 365273 1.09707 0 - 0.0299513 2026-02-23T13:56:47.363 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:56:47.482 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:56:47.577 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:56:47.575+0000 7f526290d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:56:47.577 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:56:47.576+0000 7f526290d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:56:47.589 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:56:47.590 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:56:47.599 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:56:47.600 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:56:47.600 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:56:47.611 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:56:47.612 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:56:47.643 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:56:47 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2026-02-23T13:56:47.643 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:56:47 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T13:56:47.643 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:56:47 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T13:56:47.643 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:56:47 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T13:56:47.643 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:56:47 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T13:56:47.643 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:56:47 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-23T13:56:47.643 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:56:47 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2026-02-23T13:56:47.643 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:56:47 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T13:56:47.643 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:56:47 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2026-02-23T13:56:47.643 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:56:47 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1260271371' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:56:47.644 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:56:47.645 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:56:48.303 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2602 8 365281 365273 1.09665 0 - 0.0299513 2026-02-23T13:56:49.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:56:48 trial068 ceph-mon[38805]: pgmap v1739: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:56:49.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:56:48 trial068 ceph-mon[38805]: pgmap v1740: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:56:49.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:56:48 trial068 ceph-mon[38805]: pgmap v1741: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:56:49.303 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2603 8 365281 365273 1.09622 0 - 0.0299513 2026-02-23T13:56:49.645 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:56:49.764 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:56:49.858 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:56:49.856+0000 7fd93d0e9640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:56:49.859 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:56:49.857+0000 7fd93d0e9640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:56:49.870 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:56:49.870 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:56:49.880 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:56:49.881 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:56:49.881 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:56:49.891 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:56:49.891 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:56:49.921 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:56:49.921 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:56:50.303 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2604 8 365281 365273 1.0958 0 - 0.0299513 2026-02-23T13:56:51.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:56:50 trial068 ceph-mon[38805]: pgmap v1742: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:56:51.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:56:50 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3355165297' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:56:51.303 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2605 8 365281 365273 1.09538 0 - 0.0299513 2026-02-23T13:56:51.922 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:56:52.041 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:56:52.149 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:56:52.147+0000 7f6cb6f7a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:56:52.149 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:56:52.147+0000 7f6cb6f7a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:56:52.161 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:56:52.161 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:56:52.171 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:56:52.172 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:56:52.172 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:56:52.183 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:56:52.183 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:56:52.210 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:56:52.211 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:56:52.303 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2606 8 365281 365273 1.09496 0 - 0.0299513 2026-02-23T13:56:53.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:56:52 trial068 ceph-mon[38805]: pgmap v1743: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:56:53.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:56:52 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1376719497' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:56:53.303 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2607 8 365281 365273 1.09454 0 - 0.0299513 2026-02-23T13:56:54.212 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:56:54.238 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:56:53 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:56:53] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:56:54.303 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2608 8 365281 365273 1.09412 0 - 0.0299513 2026-02-23T13:56:54.331 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:56:54.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:56:53 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:56:53.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:56:54.433 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:56:54.431+0000 7f5ee1486640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:56:54.433 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:56:54.431+0000 7f5ee1486640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:56:54.446 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:56:54.446 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:56:54.456 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:56:54.457 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:56:54.457 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:56:54.467 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:56:54.467 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:56:54.498 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:56:54.499 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:56:54.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:56:54 trial068 ceph-mon[38805]: pgmap v1744: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:56:54.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:56:54 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1667154388' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:56:55.304 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2609 8 365281 365273 1.0937 0 - 0.0299513 2026-02-23T13:56:56.304 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2610 8 365281 365273 1.09328 0 - 0.0299513 2026-02-23T13:56:56.500 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:56:56.620 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:56:56.718 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:56:56.716+0000 7fcafed15640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:56:56.718 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:56:56.716+0000 7fcafed15640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:56:56.730 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:56:56.731 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:56:56.741 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:56:56.741 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:56:56.742 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:56:56.752 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:56:56.752 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:56:56.784 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:56:56 trial068 ceph-mon[38805]: pgmap v1745: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:56:56.784 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:56:56.785 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:56:57.304 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2611 8 365281 365273 1.09287 0 - 0.0299513 2026-02-23T13:56:58.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:56:57 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/4263704702' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:56:58.304 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2612 8 365281 365273 1.09245 0 - 0.0299513 2026-02-23T13:56:58.786 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:56:58.903 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:56:58.925 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:56:58 trial068 ceph-mon[38805]: pgmap v1746: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:56:58.999 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:56:58.997+0000 7ff50f27b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:56:59.000 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:56:58.998+0000 7ff50f27b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:56:59.012 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:56:59.012 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:56:59.022 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:56:59.022 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:56:59.023 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:56:59.032 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:56:59.032 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:56:59.061 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:56:59.062 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:56:59.304 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2613 8 365281 365273 1.09203 0 - 0.0299513 2026-02-23T13:57:00.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:56:59 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1258811705' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:57:00.304 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2614 8 365281 365273 1.09161 0 - 0.0299513 2026-02-23T13:57:01.063 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:57:01.088 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:57:00 trial068 ceph-mon[38805]: pgmap v1747: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:57:01.181 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:57:01.280 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:57:01.278+0000 7f494fe21640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:57:01.280 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:57:01.278+0000 7f494fe21640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:57:01.292 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:57:01.293 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:57:01.302 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:57:01.303 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:57:01.303 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:57:01.304 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2615 8 365281 365273 1.09119 0 - 0.0299513 2026-02-23T13:57:01.313 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:57:01.313 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:57:01.343 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:57:01.344 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:57:02.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:57:01 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1071966881' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:57:02.304 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2616 8 365281 365273 1.09078 0 - 0.0299513 2026-02-23T13:57:03.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:57:02 trial068 ceph-mon[38805]: pgmap v1748: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:57:03.304 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2617 8 365281 365273 1.09036 0 - 0.0299513 2026-02-23T13:57:03.345 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:57:03.460 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:57:03.566 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:57:03.564+0000 7fc1afc93640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:57:03.566 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:57:03.565+0000 7fc1afc93640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:57:03.578 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:57:03.579 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:57:03.589 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:57:03.589 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:57:03.589 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:57:03.600 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:57:03.600 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:57:03.644 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:57:03.645 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:57:03.974 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:57:03 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2507875316' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:57:04.266 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:57:03 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:57:03.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:57:04.304 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2618 8 365281 365273 1.08994 0 - 0.0299513 2026-02-23T13:57:04.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:57:03 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:57:03] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:57:05.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:57:04 trial068 ceph-mon[38805]: 5.e scrub starts 2026-02-23T13:57:05.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:57:04 trial068 ceph-mon[38805]: pgmap v1749: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:57:05.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:57:04 trial068 ceph-mon[38805]: 5.e scrub ok 2026-02-23T13:57:05.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:57:04 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T13:57:05.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:57:04 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T13:57:05.304 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2619 8 365281 365273 1.08953 0 - 0.0299513 2026-02-23T13:57:05.646 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:57:05.766 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:57:05.865 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:57:05.863+0000 7f410bbcc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:57:05.866 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:57:05.864+0000 7f410bbcc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:57:05.878 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:57:05.878 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:57:05.888 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:57:05.889 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:57:05.889 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:57:05.899 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:57:05.900 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:57:05.909 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:57:05 trial068 ceph-mon[38805]: 5.7 scrub starts 2026-02-23T13:57:05.909 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:57:05 trial068 ceph-mon[38805]: 5.7 scrub ok 2026-02-23T13:57:05.930 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:57:05.931 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:57:06.304 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T13:57:06.304247+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T13:57:06.305 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T13:57:06.305 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2620 8 365281 365273 1.08911 0 - 0.0299513 2026-02-23T13:57:07.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:57:06 trial068 ceph-mon[38805]: pgmap v1750: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:57:07.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:57:06 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/4281003339' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:57:07.305 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2621 8 365281 365273 1.0887 0 - 0.0299513 2026-02-23T13:57:07.931 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:57:08.049 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:57:08.149 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:57:08.147+0000 7f8a374c7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:57:08.150 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:57:08.148+0000 7f8a374c7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:57:08.161 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:57:08.162 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:57:08.172 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:57:08.172 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:57:08.172 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:57:08.182 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:57:08.183 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:57:08.213 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:57:08.214 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:57:08.305 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2622 8 365281 365273 1.08828 0 - 0.0299513 2026-02-23T13:57:09.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:57:08 trial068 ceph-mon[38805]: pgmap v1751: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:57:09.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:57:08 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/274154988' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:57:09.305 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2623 8 365281 365273 1.08787 0 - 0.0299513 2026-02-23T13:57:10.215 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:57:10.305 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2624 8 365281 365273 1.08745 0 - 0.0299513 2026-02-23T13:57:10.334 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:57:10.433 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:57:10.431+0000 7fc5d7734640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:57:10.433 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:57:10.431+0000 7fc5d7734640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:57:10.445 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:57:10.445 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:57:10.455 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:57:10.456 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:57:10.456 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:57:10.466 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:57:10.466 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:57:10.495 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:57:10.496 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:57:10.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:57:10 trial068 ceph-mon[38805]: pgmap v1752: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:57:10.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:57:10 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2039860390' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:57:11.305 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2625 8 365281 365273 1.08704 0 - 0.0299513 2026-02-23T13:57:12.305 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2626 8 365281 365273 1.08662 0 - 0.0299513 2026-02-23T13:57:12.497 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:57:12.616 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:57:12.715 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:57:12.713+0000 7efec3200640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:57:12.715 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:57:12.713+0000 7efec3200640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:57:12.727 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:57:12.727 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:57:12.737 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:57:12.737 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:57:12.738 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:57:12.747 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:57:12.748 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:57:12.775 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:57:12 trial068 ceph-mon[38805]: pgmap v1753: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:57:12.776 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:57:12.776 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:57:13.305 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2627 8 365281 365273 1.08621 0 - 0.0299513 2026-02-23T13:57:13.975 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:57:13 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2439447192' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:57:14.305 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2628 8 365281 365273 1.0858 0 - 0.0299513 2026-02-23T13:57:14.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:57:13 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:57:13] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:57:14.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:57:13 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:57:13.974Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:57:14.777 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:57:14.896 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:57:14.919 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:57:14 trial068 ceph-mon[38805]: pgmap v1754: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:57:14.997 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:57:14.995+0000 7f6d1875a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:57:14.997 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:57:14.995+0000 7f6d1875a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:57:15.010 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:57:15.010 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:57:15.020 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:57:15.021 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:57:15.021 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:57:15.031 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:57:15.031 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:57:15.060 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:57:15.061 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:57:15.305 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2629 8 365281 365273 1.08538 0 - 0.0299513 2026-02-23T13:57:16.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:57:15 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/4035855163' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:57:16.305 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2630 8 365281 365273 1.08497 0 - 0.0299513 2026-02-23T13:57:17.062 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:57:17.086 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:57:16 trial068 ceph-mon[38805]: pgmap v1755: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:57:17.179 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:57:17.280 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:57:17.278+0000 7eff053d6640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:57:17.280 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:57:17.278+0000 7eff053d6640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:57:17.292 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:57:17.292 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:57:17.302 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:57:17.302 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:57:17.303 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:57:17.305 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2631 8 365281 365273 1.08456 0 - 0.0299513 2026-02-23T13:57:17.312 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:57:17.312 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:57:17.341 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:57:17.341 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:57:18.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:57:17 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3905889704' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:57:18.305 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2632 8 365281 365273 1.08415 0 - 0.0299513 2026-02-23T13:57:19.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:57:18 trial068 ceph-mon[38805]: pgmap v1756: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:57:19.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:57:18 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T13:57:19.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:57:18 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T13:57:19.305 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2633 8 365281 365273 1.08373 0 - 0.0299513 2026-02-23T13:57:19.343 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:57:19.459 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:57:19.558 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:57:19.556+0000 7ff5673f3640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:57:19.558 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:57:19.556+0000 7ff5673f3640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:57:19.570 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:57:19.570 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:57:19.580 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:57:19.581 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:57:19.581 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:57:19.590 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:57:19.590 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:57:19.619 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:57:19.620 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:57:20.305 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2634 8 365281 365273 1.08332 0 - 0.0299513 2026-02-23T13:57:20.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:57:20 trial068 ceph-mon[38805]: pgmap v1757: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:57:20.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:57:20 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3849757164' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:57:21.305 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2635 8 365281 365273 1.08291 0 - 0.0299513 2026-02-23T13:57:21.621 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:57:21.738 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:57:21.840 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:57:21.838+0000 7fb73f54a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:57:21.840 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:57:21.838+0000 7fb73f54a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:57:21.852 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:57:21.852 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:57:21.862 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:57:21.862 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:57:21.862 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:57:21.872 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:57:21.872 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:57:21.903 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:57:21.904 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:57:22.306 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2636 8 365281 365273 1.0825 0 - 0.0299513 2026-02-23T13:57:22.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:57:22 trial068 ceph-mon[38805]: pgmap v1758: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:57:22.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:57:22 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2396752067' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:57:23.306 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2637 8 365281 365273 1.08209 0 - 0.0299513 2026-02-23T13:57:23.905 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:57:24.024 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:57:24.124 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:57:24.122+0000 7f212e95c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:57:24.124 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:57:24.123+0000 7f212e95c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:57:24.137 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:57:24.137 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:57:24.147 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:57:24.147 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:57:24.148 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:57:24.157 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:57:24.158 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:57:24.186 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:57:23 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:57:23] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:57:24.187 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:57:24.187 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:57:24.306 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2638 8 365281 365273 1.08168 0 - 0.0299513 2026-02-23T13:57:24.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:57:23 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:57:23.972Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:57:24.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:57:24 trial068 ceph-mon[38805]: pgmap v1759: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:57:24.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:57:24 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3255678498' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:57:25.306 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2639 8 365281 365273 1.08127 0 - 0.0299513 2026-02-23T13:57:26.188 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:57:26.305 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:57:26.306 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T13:57:26.305627+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T13:57:26.306 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T13:57:26.306 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2640 8 365281 365273 1.08086 0 - 0.0299513 2026-02-23T13:57:26.401 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:57:26.398+0000 7f9beb3f2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:57:26.401 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:57:26.399+0000 7f9beb3f2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:57:26.412 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:57:26.413 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:57:26.422 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:57:26.423 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:57:26.423 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:57:26.432 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:57:26.432 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:57:26.463 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:57:26.464 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:57:26.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:57:26 trial068 ceph-mon[38805]: pgmap v1760: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:57:27.306 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2641 8 365281 365273 1.08045 0 - 0.0299513 2026-02-23T13:57:27.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:57:27 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/4161549013' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:57:28.306 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2642 8 365281 365273 1.08004 0 - 0.0299513 2026-02-23T13:57:28.465 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:57:28.581 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:57:28.626 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:57:28 trial068 ceph-mon[38805]: pgmap v1761: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:57:28.682 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:57:28.680+0000 7efc3accf640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:57:28.682 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:57:28.680+0000 7efc3accf640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:57:28.695 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:57:28.695 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:57:28.705 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:57:28.706 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:57:28.706 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:57:28.716 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:57:28.716 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:57:28.744 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:57:28.745 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:57:29.306 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2643 8 365281 365273 1.07963 0 - 0.0299513 2026-02-23T13:57:29.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:57:29 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2482603480' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:57:30.306 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2644 8 365281 365273 1.07922 0 - 0.0299513 2026-02-23T13:57:30.746 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:57:30.771 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:57:30 trial068 ceph-mon[38805]: 5.a scrub starts 2026-02-23T13:57:30.771 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:57:30 trial068 ceph-mon[38805]: 5.a scrub ok 2026-02-23T13:57:30.771 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:57:30 trial068 ceph-mon[38805]: pgmap v1762: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:57:30.864 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:57:30.962 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:57:30.960+0000 7f111f424640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:57:30.963 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:57:30.961+0000 7f111f424640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:57:30.974 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:57:30.975 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:57:30.984 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:57:30.985 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:57:30.985 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:57:30.995 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:57:30.995 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:57:31.024 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:57:31.025 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:57:31.306 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2645 8 365281 365273 1.07882 0 - 0.0299513 2026-02-23T13:57:31.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:57:31 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/330947120' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:57:32.307 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2646 8 365281 365273 1.07841 0 - 0.0299513 2026-02-23T13:57:32.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:57:32 trial068 ceph-mon[38805]: pgmap v1763: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:57:33.026 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:57:33.144 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:57:33.243 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:57:33.241+0000 7f35c887a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:57:33.243 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:57:33.241+0000 7f35c887a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:57:33.255 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:57:33.255 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:57:33.265 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:57:33.265 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:57:33.265 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:57:33.276 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:57:33.276 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:57:33.307 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2647 8 365281 365273 1.078 0 - 0.0299513 2026-02-23T13:57:33.309 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:57:33.309 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:57:33.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:57:33 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3473477206' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:57:34.266 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:57:33 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:57:33.974Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:57:34.307 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2648 8 365281 365273 1.07759 0 - 0.0299513 2026-02-23T13:57:34.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:57:33 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:57:33] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:57:34.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:57:34 trial068 ceph-mon[38805]: pgmap v1764: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:57:35.307 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2649 8 365281 365273 1.07719 0 - 0.0299513 2026-02-23T13:57:35.310 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:57:35.429 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:57:35.528 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:57:35.526+0000 7fb65f834640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:57:35.528 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:57:35.526+0000 7fb65f834640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:57:35.540 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:57:35.540 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:57:35.551 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:57:35.551 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:57:35.551 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:57:35.561 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:57:35.561 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:57:35.591 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:57:35.592 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:57:36.307 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2650 8 365281 365273 1.07678 0 - 0.0299513 2026-02-23T13:57:36.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:57:36 trial068 ceph-mon[38805]: pgmap v1765: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:57:36.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:57:36 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2383078833' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:57:37.307 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2651 8 365281 365273 1.07638 0 - 0.0299513 2026-02-23T13:57:37.592 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:57:37.709 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:57:37.807 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:57:37.805+0000 7f743e023640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:57:37.807 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:57:37.806+0000 7f743e023640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:57:37.820 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:57:37.820 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:57:37.830 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:57:37.830 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:57:37.830 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:57:37.840 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:57:37.841 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:57:37.870 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:57:37.870 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:57:38.307 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2652 8 365281 365273 1.07597 0 - 0.0299513 2026-02-23T13:57:38.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:57:38 trial068 ceph-mon[38805]: pgmap v1766: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:57:38.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:57:38 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/4182323131' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:57:39.307 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2653 8 365281 365273 1.07556 0 - 0.0299513 2026-02-23T13:57:39.871 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:57:39.988 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:57:40.085 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:57:40.083+0000 7f8f1baf0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:57:40.086 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:57:40.084+0000 7f8f1baf0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:57:40.097 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:57:40.097 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:57:40.107 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:57:40.108 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:57:40.108 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:57:40.118 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:57:40.118 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:57:40.150 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:57:40.151 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:57:40.307 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2654 8 365281 365273 1.07516 0 - 0.0299513 2026-02-23T13:57:40.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:57:40 trial068 ceph-mon[38805]: pgmap v1767: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:57:40.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:57:40 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3485261984' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:57:41.307 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2655 8 365281 365273 1.07475 0 - 0.0299513 2026-02-23T13:57:42.151 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:57:42.268 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:57:42.307 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2656 8 365281 365273 1.07435 0 - 0.0299513 2026-02-23T13:57:42.373 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:57:42.371+0000 7f15b9cfa640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:57:42.374 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:57:42.371+0000 7f15b9cfa640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:57:42.385 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:57:42.385 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:57:42.395 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:57:42.395 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:57:42.396 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:57:42.406 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:57:42.406 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:57:42.433 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:57:42 trial068 ceph-mon[38805]: pgmap v1768: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:57:42.433 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:57:42 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3522961170' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:57:42.434 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:57:42.434 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:57:43.307 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2657 8 365281 365273 1.07394 0 - 0.0299513 2026-02-23T13:57:44.308 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2658 8 365281 365273 1.07354 0 - 0.0299513 2026-02-23T13:57:44.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:57:43 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:57:43] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:57:44.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:57:43 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:57:43.974Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:57:44.435 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:57:44.554 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:57:44.650 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:57:44.648+0000 7f265c1d4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:57:44.650 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:57:44 trial068 ceph-mon[38805]: pgmap v1769: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:57:44.650 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:57:44.648+0000 7f265c1d4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:57:44.661 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:57:44.662 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:57:44.672 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:57:44.672 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:57:44.672 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:57:44.682 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:57:44.683 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:57:44.713 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:57:44.713 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:57:45.308 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2659 8 365281 365273 1.07314 0 - 0.0299513 2026-02-23T13:57:45.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:57:45 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3512484330' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:57:46.308 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T13:57:46.307443+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T13:57:46.308 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T13:57:46.308 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2660 8 365281 365273 1.07273 0 - 0.0299513 2026-02-23T13:57:46.715 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:57:46.739 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:57:46 trial068 ceph-mon[38805]: pgmap v1770: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:57:46.739 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:57:46 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T13:57:46.740 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:57:46 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T13:57:46.830 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:57:46.930 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:57:46.928+0000 7f0202797640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:57:46.930 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:57:46.928+0000 7f0202797640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:57:46.943 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:57:46.943 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:57:46.953 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:57:46.953 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:57:46.954 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:57:46.963 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:57:46.963 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:57:46.994 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:57:46.994 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:57:47.308 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2661 8 365281 365273 1.07233 0 - 0.0299513 2026-02-23T13:57:47.681 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:57:47 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2132662839' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:57:47.682 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:57:47 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2026-02-23T13:57:48.308 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2662 8 365281 365273 1.07193 0 - 0.0299513 2026-02-23T13:57:48.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:57:48 trial068 ceph-mon[38805]: pgmap v1771: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:57:48.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:57:48 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T13:57:48.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:57:48 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T13:57:48.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:57:48 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T13:57:48.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:57:48 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T13:57:48.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:57:48 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-23T13:57:48.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:57:48 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2026-02-23T13:57:48.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:57:48 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T13:57:48.845 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:57:48 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2026-02-23T13:57:48.996 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:57:49.113 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:57:49.213 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:57:49.211+0000 7f0a3c150640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:57:49.213 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:57:49.211+0000 7f0a3c150640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:57:49.225 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:57:49.225 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:57:49.235 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:57:49.235 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:57:49.235 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:57:49.247 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:57:49.247 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:57:49.278 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:57:49.279 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:57:49.308 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2663 8 365281 365273 1.07152 0 - 0.0299513 2026-02-23T13:57:49.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:57:49 trial068 ceph-mon[38805]: pgmap v1772: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:57:49.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:57:49 trial068 ceph-mon[38805]: pgmap v1773: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:57:49.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:57:49 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2522062486' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:57:50.308 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2664 8 365281 365273 1.07112 0 - 0.0299513 2026-02-23T13:57:51.280 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:57:51.308 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2665 8 365281 365273 1.07072 0 - 0.0299513 2026-02-23T13:57:51.399 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:57:51.504 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:57:51.502+0000 7fcdefa3d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:57:51.504 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:57:51.502+0000 7fcdefa3d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:57:51.516 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:57:51.516 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:57:51.526 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:57:51.526 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:57:51.526 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:57:51.537 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:57:51.538 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:57:51.566 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:57:51 trial068 ceph-mon[38805]: pgmap v1774: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:57:51.568 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:57:51.569 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:57:52.308 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2666 8 365281 365273 1.07032 0 - 0.0299513 2026-02-23T13:57:52.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:57:52 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2487087344' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:57:53.308 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2667 8 365281 365273 1.06992 0 - 0.0299513 2026-02-23T13:57:53.570 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:57:53.689 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:57:53.709 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:57:53 trial068 ceph-mon[38805]: pgmap v1775: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:57:53.789 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:57:53.787+0000 7fe23d4a4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:57:53.789 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:57:53.787+0000 7fe23d4a4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:57:53.801 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:57:53.801 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:57:53.811 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:57:53.812 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:57:53.812 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:57:53.822 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:57:53.822 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:57:53.853 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:57:53.854 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:57:54.308 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2668 8 365281 365273 1.06952 0 - 0.0299513 2026-02-23T13:57:54.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:57:53 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:57:53] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:57:54.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:57:53 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:57:53.974Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:57:54.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:57:54 trial068 ceph-mon[38805]: 5.5 scrub starts 2026-02-23T13:57:54.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:57:54 trial068 ceph-mon[38805]: 5.5 scrub ok 2026-02-23T13:57:54.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:57:54 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1983237038' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:57:55.308 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2669 8 365281 365273 1.06912 0 - 0.0299513 2026-02-23T13:57:55.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:57:55 trial068 ceph-mon[38805]: pgmap v1776: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:57:55.854 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:57:55.971 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:57:56.066 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:57:56.064+0000 7f2ec2ffa640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:57:56.066 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:57:56.064+0000 7f2ec2ffa640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:57:56.078 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:57:56.079 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:57:56.089 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:57:56.090 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:57:56.090 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:57:56.100 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:57:56.100 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:57:56.130 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:57:56.131 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:57:56.309 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2670 8 365281 365273 1.06872 0 - 0.0299513 2026-02-23T13:57:56.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:57:56 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2556658496' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:57:57.309 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2671 8 365281 365273 1.06832 0 - 0.0299513 2026-02-23T13:57:57.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:57:57 trial068 ceph-mon[38805]: pgmap v1777: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:57:58.132 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:57:58.252 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:57:58.309 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2672 8 365281 365273 1.06792 0 - 0.0299513 2026-02-23T13:57:58.351 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:57:58.349+0000 7f9cbb38a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:57:58.351 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:57:58.350+0000 7f9cbb38a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:57:58.363 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:57:58.364 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:57:58.374 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:57:58.374 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:57:58.374 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:57:58.385 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:57:58.385 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:57:58.415 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:57:58.415 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:57:58.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:57:58 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/4109735562' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:57:59.309 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2673 8 365281 365273 1.06752 0 - 0.0299513 2026-02-23T13:57:59.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:57:59 trial068 ceph-mon[38805]: pgmap v1778: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:58:00.309 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2674 8 365281 365273 1.06712 0 - 0.0299513 2026-02-23T13:58:00.417 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:58:00.533 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:58:00.636 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:58:00.634+0000 7fc62c90f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:58:00.636 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:58:00.634+0000 7fc62c90f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:58:00.648 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:58:00.648 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:58:00.658 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:58:00.659 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:58:00.659 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:58:00.668 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:58:00.669 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:58:00.698 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:58:00.698 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:58:01.309 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2675 8 365281 365273 1.06672 0 - 0.0299513 2026-02-23T13:58:01.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:58:01 trial068 ceph-mon[38805]: pgmap v1779: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:58:01.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:58:01 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/112031447' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:58:02.309 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2676 8 365281 365273 1.06632 0 - 0.0299513 2026-02-23T13:58:02.700 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:58:02.817 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:58:02.913 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:58:02.911+0000 7f541c251640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:58:02.913 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:58:02.911+0000 7f541c251640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:58:02.925 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:58:02.925 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:58:02.935 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:58:02.936 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:58:02.936 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:58:02.946 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:58:02.946 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:58:02.976 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:58:02.977 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:58:03.309 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2677 8 365281 365273 1.06592 0 - 0.0299513 2026-02-23T13:58:03.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:58:03 trial068 ceph-mon[38805]: pgmap v1780: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:58:03.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:58:03 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/4280001319' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:58:04.265 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:58:03 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:58:03.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:58:04.309 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2678 8 365281 365273 1.06552 0 - 0.0299513 2026-02-23T13:58:04.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:58:03 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:58:03] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:58:04.978 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:58:05.094 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:58:05.203 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:58:05.201+0000 7f976c801640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:58:05.203 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:58:05.201+0000 7f976c801640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:58:05.215 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:58:05.215 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:58:05.225 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:58:05.225 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:58:05.225 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:58:05.235 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:58:05.235 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:58:05.265 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:58:05.265 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:58:05.309 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2679 8 365281 365273 1.06513 0 - 0.0299513 2026-02-23T13:58:05.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:58:05 trial068 ceph-mon[38805]: pgmap v1781: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:58:05.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:58:05 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/891085989' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:58:06.309 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T13:58:06.308946+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T13:58:06.309 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T13:58:06.310 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2680 8 365281 365273 1.06473 0 - 0.0299513 2026-02-23T13:58:06.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:58:06 trial068 ceph-mon[38805]: 5.8 scrub starts 2026-02-23T13:58:06.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:58:06 trial068 ceph-mon[38805]: 5.8 scrub ok 2026-02-23T13:58:06.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:58:06 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T13:58:06.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:58:06 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T13:58:07.266 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:58:07.309 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2681 8 365281 365273 1.06433 0 - 0.0299513 2026-02-23T13:58:07.384 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:58:07.487 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:58:07.485+0000 7fc71bb60640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:58:07.487 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:58:07.485+0000 7fc71bb60640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:58:07.499 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:58:07.499 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:58:07.509 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:58:07.510 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:58:07.510 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:58:07.519 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:58:07.520 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:58:07.548 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:58:07 trial068 ceph-mon[38805]: pgmap v1782: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:58:07.549 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:58:07.550 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:58:08.309 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2682 8 365281 365273 1.06393 0 - 0.0299513 2026-02-23T13:58:08.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:58:08 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3949960953' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:58:09.309 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2683 8 365281 365273 1.06354 0 - 0.0299513 2026-02-23T13:58:09.551 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:58:09.667 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:58:09.688 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:58:09 trial068 ceph-mon[38805]: pgmap v1783: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:58:09.768 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:58:09.766+0000 7f023e629640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:58:09.768 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:58:09.766+0000 7f023e629640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:58:09.780 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:58:09.780 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:58:09.790 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:58:09.790 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:58:09.790 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:58:09.800 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:58:09.800 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:58:09.829 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:58:09.830 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:58:10.309 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2684 8 365281 365273 1.06314 0 - 0.0299513 2026-02-23T13:58:10.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:58:10 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/143522419' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:58:11.309 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2685 8 365281 365273 1.06275 0 - 0.0299513 2026-02-23T13:58:11.831 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:58:11.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:58:11 trial068 ceph-mon[38805]: pgmap v1784: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:58:11.950 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:58:12.063 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:58:12.061+0000 7fccc8981640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:58:12.063 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:58:12.061+0000 7fccc8981640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:58:12.075 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:58:12.075 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:58:12.085 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:58:12.085 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:58:12.086 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:58:12.095 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:58:12.096 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:58:12.125 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:58:12.126 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:58:12.310 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2686 8 365281 365273 1.06235 0 - 0.0299513 2026-02-23T13:58:12.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:58:12 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3468056159' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:58:13.310 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2687 8 365281 365273 1.06195 0 - 0.0299513 2026-02-23T13:58:13.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:58:13 trial068 ceph-mon[38805]: pgmap v1785: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:58:14.127 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:58:14.248 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:58:14.270 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:58:13 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:58:13] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:58:14.310 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2688 8 365281 365273 1.06156 0 - 0.0299513 2026-02-23T13:58:14.349 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:58:14.347+0000 7f9dffe03640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:58:14.350 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:58:14.348+0000 7f9dffe03640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:58:14.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:58:13 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:58:13.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:58:14.362 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:58:14.362 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:58:14.372 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:58:14.372 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:58:14.373 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:58:14.383 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:58:14.383 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:58:14.411 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:58:14.412 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:58:14.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:58:14 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3299398794' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:58:15.310 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2689 8 365281 365273 1.06116 0 - 0.0299513 2026-02-23T13:58:15.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:58:15 trial068 ceph-mon[38805]: pgmap v1786: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:58:16.310 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2690 8 365281 365273 1.06077 0 - 0.0299513 2026-02-23T13:58:16.413 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:58:16.532 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:58:16.635 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:58:16.632+0000 7f5bafa47640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:58:16.635 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:58:16.633+0000 7f5bafa47640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:58:16.646 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:58:16.647 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:58:16.656 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:58:16.657 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:58:16.657 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:58:16.668 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:58:16.668 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:58:16.682 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:58:16 trial068 ceph-mon[38805]: 5.7 scrub starts 2026-02-23T13:58:16.682 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:58:16 trial068 ceph-mon[38805]: 5.7 scrub ok 2026-02-23T13:58:16.697 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:58:16.697 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:58:17.310 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2691 8 365281 365273 1.06038 0 - 0.0299513 2026-02-23T13:58:17.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:58:17 trial068 ceph-mon[38805]: pgmap v1787: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:58:17.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:58:17 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/862350691' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:58:18.310 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2692 8 365281 365273 1.05998 0 - 0.0299513 2026-02-23T13:58:18.698 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:58:18.814 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:58:18.910 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:58:18.907+0000 7fc40b56d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:58:18.910 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:58:18.908+0000 7fc40b56d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:58:18.921 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:58:18.922 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:58:18.932 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:58:18.932 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:58:18.932 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:58:18.943 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:58:18.943 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:58:18.973 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:58:18.974 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:58:19.310 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2693 8 365281 365273 1.05959 0 - 0.0299513 2026-02-23T13:58:19.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:58:19 trial068 ceph-mon[38805]: pgmap v1788: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:58:19.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:58:19 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1661384712' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:58:20.310 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2694 8 365281 365273 1.05919 0 - 0.0299513 2026-02-23T13:58:20.975 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:58:21.094 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:58:21.196 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:58:21.195+0000 7fdf333fa640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:58:21.197 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:58:21.195+0000 7fdf333fa640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:58:21.209 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:58:21.209 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:58:21.219 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:58:21.219 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:58:21.219 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:58:21.229 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:58:21.229 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:58:21.257 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:58:21.258 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:58:21.310 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2695 8 365281 365273 1.0588 0 - 0.0299513 2026-02-23T13:58:21.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:58:21 trial068 ceph-mon[38805]: pgmap v1789: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:58:21.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:58:21 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3018454551' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:58:22.311 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2696 8 365281 365273 1.05841 0 - 0.0299513 2026-02-23T13:58:23.259 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:58:23.311 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2697 8 365281 365273 1.05802 0 - 0.0299513 2026-02-23T13:58:23.376 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:58:23.478 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:58:23.476+0000 7feeb2c2c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:58:23.479 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:58:23.477+0000 7feeb2c2c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:58:23.490 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:58:23.491 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:58:23.500 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:58:23.501 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:58:23.501 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:58:23.511 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:58:23.511 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:58:23.540 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:58:23 trial068 ceph-mon[38805]: pgmap v1790: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:58:23.541 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:58:23.542 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:58:24.311 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2698 8 365281 365273 1.05762 0 - 0.0299513 2026-02-23T13:58:24.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:58:23 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:58:23] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:58:24.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:58:23 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:58:23.974Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:58:24.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:58:24 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3632691822' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:58:25.311 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2699 8 365281 365273 1.05723 0 - 0.0299513 2026-02-23T13:58:25.543 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:58:25.661 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:58:25.706 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:58:25 trial068 ceph-mon[38805]: pgmap v1791: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:58:25.762 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:58:25.760+0000 7fb74d864640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:58:25.762 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:58:25.760+0000 7fb74d864640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:58:25.775 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:58:25.775 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:58:25.785 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:58:25.785 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:58:25.786 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:58:25.795 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:58:25.795 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:58:25.823 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:58:25.823 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:58:26.311 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T13:58:26.310771+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T13:58:26.311 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T13:58:26.312 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2700 8 365281 365273 1.05684 0 - 0.0299513 2026-02-23T13:58:26.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:58:26 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1971791068' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:58:27.311 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2701 8 365281 365273 1.05645 0 - 0.0299513 2026-02-23T13:58:27.824 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:58:27.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:58:27 trial068 ceph-mon[38805]: pgmap v1792: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:58:27.943 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:58:28.052 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:58:28.049+0000 7f2123c9c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:58:28.052 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:58:28.050+0000 7f2123c9c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:58:28.064 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:58:28.064 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:58:28.073 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:58:28.074 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:58:28.074 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:58:28.083 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:58:28.084 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:58:28.112 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:58:28.112 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:58:28.311 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2702 8 365281 365273 1.05606 0 - 0.0299513 2026-02-23T13:58:28.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:58:28 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2188199546' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:58:29.311 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2703 8 365281 365273 1.05567 0 - 0.0299513 2026-02-23T13:58:29.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:58:29 trial068 ceph-mon[38805]: pgmap v1793: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:58:30.113 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:58:30.230 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:58:30.311 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2704 8 365281 365273 1.05528 0 - 0.0299513 2026-02-23T13:58:30.331 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:58:30.329+0000 7f9c6b3a8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:58:30.331 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:58:30.329+0000 7f9c6b3a8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:58:30.343 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:58:30.343 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:58:30.353 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:58:30.353 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:58:30.354 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:58:30.363 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:58:30.363 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:58:30.393 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:58:30.393 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:58:30.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:58:30 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/4067755838' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:58:31.311 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2705 8 365281 365273 1.05489 0 - 0.0299513 2026-02-23T13:58:31.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:58:31 trial068 ceph-mon[38805]: pgmap v1794: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:58:32.311 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2706 8 365281 365273 1.0545 0 - 0.0299513 2026-02-23T13:58:32.394 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:58:32.511 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:58:32.613 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:58:32.611+0000 7fb34bad1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:58:32.613 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:58:32.611+0000 7fb34bad1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:58:32.625 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:58:32.625 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:58:32.636 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:58:32.636 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:58:32.636 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:58:32.646 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:58:32.647 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:58:32.690 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:58:32.690 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:58:33.312 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2707 8 365281 365273 1.05411 0 - 0.0299513 2026-02-23T13:58:33.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:58:33 trial068 ceph-mon[38805]: pgmap v1795: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:58:33.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:58:33 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1943177836' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:58:34.266 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:58:33 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:58:33.974Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:58:34.312 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2708 8 365281 365273 1.05372 0 - 0.0299513 2026-02-23T13:58:34.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:58:33 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:58:33] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:58:34.691 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:58:34.810 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:58:34.911 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:58:34.909+0000 7f2ce6bbd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:58:34.911 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:58:34.909+0000 7f2ce6bbd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:58:34.923 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:58:34.923 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:58:34.933 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:58:34.934 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:58:34.934 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:58:34.945 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:58:34.945 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:58:34.975 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:58:34.976 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:58:35.312 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2709 8 365281 365273 1.05333 0 - 0.0299513 2026-02-23T13:58:35.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:58:35 trial068 ceph-mon[38805]: pgmap v1796: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:58:35.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:58:35 trial068 ceph-mon[38805]: 5.e scrub starts 2026-02-23T13:58:35.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:58:35 trial068 ceph-mon[38805]: 5.e scrub ok 2026-02-23T13:58:35.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:58:35 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3456896680' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:58:36.312 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2710 8 365281 365273 1.05294 0 - 0.0299513 2026-02-23T13:58:36.977 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:58:37.095 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:58:37.196 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:58:37.194+0000 7f061b905640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:58:37.197 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:58:37.195+0000 7f061b905640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:58:37.208 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:58:37.209 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:58:37.218 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:58:37.219 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:58:37.219 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:58:37.229 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:58:37.229 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:58:37.260 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:58:37.261 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:58:37.312 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2711 8 365281 365273 1.05255 0 - 0.0299513 2026-02-23T13:58:37.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:58:37 trial068 ceph-mon[38805]: pgmap v1797: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:58:37.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:58:37 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/5342201' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:58:38.312 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2712 8 365281 365273 1.05216 0 - 0.0299513 2026-02-23T13:58:39.262 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:58:39.312 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2713 8 365281 365273 1.05178 0 - 0.0299513 2026-02-23T13:58:39.380 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:58:39.482 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:58:39.480+0000 7f075f352640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:58:39.482 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:58:39.480+0000 7f075f352640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:58:39.495 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:58:39.495 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:58:39.505 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:58:39.505 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:58:39.505 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:58:39.516 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:58:39.516 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:58:39.547 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:58:39 trial068 ceph-mon[38805]: pgmap v1798: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:58:39.547 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:58:39.548 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:58:40.312 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2714 8 365281 365273 1.05139 0 - 0.0299513 2026-02-23T13:58:40.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:58:40 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1277574810' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:58:41.312 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2715 8 365281 365273 1.051 0 - 0.0299513 2026-02-23T13:58:41.549 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:58:41.668 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:58:41.706 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:58:41 trial068 ceph-mon[38805]: pgmap v1799: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:58:41.763 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:58:41.761+0000 7f829746e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:58:41.763 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:58:41.761+0000 7f829746e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:58:41.775 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:58:41.776 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:58:41.786 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:58:41.786 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:58:41.786 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:58:41.796 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:58:41.796 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:58:41.827 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:58:41.828 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:58:42.312 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2716 8 365281 365273 1.05062 0 - 0.0299513 2026-02-23T13:58:42.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:58:42 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T13:58:42.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:58:42 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T13:58:42.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:58:42 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1899328639' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:58:43.312 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2717 8 365281 365273 1.05023 0 - 0.0299513 2026-02-23T13:58:43.829 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:58:43.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:58:43 trial068 ceph-mon[38805]: pgmap v1800: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:58:43.948 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:58:44.049 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:58:44.047+0000 7ff76536d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:58:44.049 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:58:44.047+0000 7ff76536d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:58:44.061 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:58:44.061 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:58:44.072 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:58:44.072 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:58:44.072 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:58:44.082 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:58:44.082 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:58:44.112 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:58:43 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:58:43] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:58:44.112 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:58:44.113 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:58:44.312 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2718 8 365281 365273 1.04984 0 - 0.0299513 2026-02-23T13:58:44.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:58:43 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:58:43.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:58:44.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:58:44 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3587897975' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:58:45.312 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2719 8 365281 365273 1.04946 0 - 0.0299513 2026-02-23T13:58:45.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:58:45 trial068 ceph-mon[38805]: pgmap v1801: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:58:46.114 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:58:46.232 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:58:46.313 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T13:58:46.312277+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T13:58:46.313 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T13:58:46.313 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2720 8 365281 365273 1.04907 0 - 0.0299513 2026-02-23T13:58:46.335 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:58:46.333+0000 7fbbd76b8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:58:46.335 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:58:46.333+0000 7fbbd76b8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:58:46.347 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:58:46.347 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:58:46.357 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:58:46.358 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:58:46.358 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:58:46.367 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:58:46.367 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:58:46.396 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:58:46.397 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:58:46.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:58:46 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/742691386' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:58:47.313 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2721 8 365281 365273 1.04868 0 - 0.0299513 2026-02-23T13:58:47.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:58:47 trial068 ceph-mon[38805]: pgmap v1802: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:58:48.313 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2722 8 365281 365273 1.0483 0 - 0.0299513 2026-02-23T13:58:48.398 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:58:48.515 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:58:48.629 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:58:48.627+0000 7f5cbd2b9640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:58:48.629 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:58:48.627+0000 7f5cbd2b9640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:58:48.641 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:58:48.641 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:58:48.652 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:58:48.652 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:58:48.652 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:58:48.663 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:58:48.664 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:58:48.673 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:58:48 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2026-02-23T13:58:48.673 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:58:48 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-23T13:58:48.673 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:58:48 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2026-02-23T13:58:48.673 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:58:48 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T13:58:48.674 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:58:48 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2026-02-23T13:58:48.695 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:58:48.696 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:58:49.313 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2723 8 365281 365273 1.04791 0 - 0.0299513 2026-02-23T13:58:49.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:58:49 trial068 ceph-mon[38805]: pgmap v1803: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:58:49.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:58:49 trial068 ceph-mon[38805]: pgmap v1804: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:58:49.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:58:49 trial068 ceph-mon[38805]: pgmap v1805: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:58:49.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:58:49 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1589341526' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:58:50.313 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2724 8 365281 365273 1.04753 0 - 0.0299513 2026-02-23T13:58:50.697 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:58:50.817 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:58:50.916 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:58:50.914+0000 7f5c31c01640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:58:50.917 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:58:50.915+0000 7f5c31c01640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:58:50.928 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:58:50.929 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:58:50.938 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:58:50.939 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:58:50.939 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:58:50.949 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:58:50.949 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:58:50.979 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:58:50.980 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:58:51.313 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2725 8 365281 365273 1.04715 0 - 0.0299513 2026-02-23T13:58:51.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:58:51 trial068 ceph-mon[38805]: pgmap v1806: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:58:51.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:58:51 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3169159154' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:58:52.313 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2726 8 365281 365273 1.04676 0 - 0.0299513 2026-02-23T13:58:52.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:58:52 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T13:58:52.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:58:52 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T13:58:52.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:58:52 trial068 ceph-mon[38805]: osdmap e214: 4 total, 3 up, 4 in 2026-02-23T13:58:52.981 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:58:53.098 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:58:53.200 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:58:53.198+0000 7fd44a9d7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:58:53.200 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:58:53.198+0000 7fd44a9d7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:58:53.212 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:58:53.212 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:58:53.222 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:58:53.223 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:58:53.223 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:58:53.233 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:58:53.233 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:58:53.264 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:58:53.264 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:58:53.313 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2727 8 365281 365273 1.04638 0 - 0.0299513 2026-02-23T13:58:53.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:58:53 trial068 ceph-mon[38805]: pgmap v1808: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:58:53.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:58:53 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1914091183' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:58:54.313 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2728 8 365281 365273 1.04599 0 - 0.0299513 2026-02-23T13:58:54.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:58:53 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:58:53] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:58:54.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:58:53 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:58:53.974Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:58:54.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:58:54 trial068 ceph-mon[38805]: 5.a scrub starts 2026-02-23T13:58:55.265 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:58:55.313 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2729 8 365281 365273 1.04561 0 - 0.0299513 2026-02-23T13:58:55.383 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:58:55.481 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:58:55.479+0000 7f12c53c7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:58:55.481 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:58:55.479+0000 7f12c53c7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:58:55.495 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:58:55.496 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:58:55.506 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:58:55.506 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:58:55.506 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:58:55.517 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:58:55.517 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:58:55.545 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:58:55 trial068 ceph-mon[38805]: 5.a scrub ok 2026-02-23T13:58:55.545 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:58:55 trial068 ceph-mon[38805]: pgmap v1809: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:58:55.546 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:58:55.546 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:58:56.314 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2730 8 365281 365273 1.04523 0 - 0.0299513 2026-02-23T13:58:56.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:58:56 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2249906509' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:58:57.314 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2731 8 365281 365273 1.04484 0 - 0.0299513 2026-02-23T13:58:57.547 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:58:57.665 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:58:57.733 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:58:57 trial068 ceph-mon[38805]: pgmap v1810: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:58:57.780 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:58:57.778+0000 7f00c754e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:58:57.780 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:58:57.778+0000 7f00c754e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:58:57.792 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:58:57.792 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:58:57.801 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:58:57.802 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:58:57.802 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:58:57.812 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:58:57.812 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:58:57.842 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:58:57.843 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:58:58.314 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2732 8 365281 365273 1.04446 0 - 0.0299513 2026-02-23T13:58:58.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:58:58 trial068 ceph-mon[38805]: 5.5 scrub starts 2026-02-23T13:58:58.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:58:58 trial068 ceph-mon[38805]: 5.5 scrub ok 2026-02-23T13:58:58.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:58:58 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1541862119' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:58:59.314 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2733 8 365281 365273 1.04408 0 - 0.0299513 2026-02-23T13:58:59.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:58:59 trial068 ceph-mon[38805]: pgmap v1811: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:58:59.844 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:58:59.960 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:59:00.060 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:59:00.058+0000 7fa659683640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:59:00.060 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:59:00.058+0000 7fa659683640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:59:00.072 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:59:00.072 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:59:00.082 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:59:00.082 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:59:00.083 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:59:00.093 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:59:00.093 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:59:00.124 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:59:00.124 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:59:00.314 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2734 8 365281 365273 1.0437 0 - 0.0299513 2026-02-23T13:59:00.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:59:00 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2492657562' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:59:01.314 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2735 8 365281 365273 1.04332 0 - 0.0299513 2026-02-23T13:59:01.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:59:01 trial068 ceph-mon[38805]: pgmap v1812: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:59:02.126 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:59:02.241 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:59:02.314 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2736 8 365281 365273 1.04294 0 - 0.0299513 2026-02-23T13:59:02.340 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:59:02.338+0000 7f80b202b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:59:02.341 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:59:02.339+0000 7f80b202b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:59:02.352 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:59:02.353 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:59:02.363 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:59:02.363 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:59:02.363 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:59:02.373 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:59:02.374 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:59:02.401 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:59:02.402 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:59:02.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:59:02 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1012192671' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:59:03.314 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2737 8 365281 365273 1.04255 0 - 0.0299513 2026-02-23T13:59:03.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:59:03 trial068 ceph-mon[38805]: pgmap v1813: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:59:04.266 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:59:03 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:59:03.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:59:04.314 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2738 8 365281 365273 1.04217 0 - 0.0299513 2026-02-23T13:59:04.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:59:03 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:59:03] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:59:04.403 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:59:04.521 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:59:04.626 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:59:04.624+0000 7f737558a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:59:04.627 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:59:04.624+0000 7f737558a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:59:04.639 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:59:04.639 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:59:04.648 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:59:04.649 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:59:04.649 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:59:04.660 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:59:04.660 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:59:04.691 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:59:04.691 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:59:05.314 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2739 8 365281 365273 1.04179 0 - 0.0299513 2026-02-23T13:59:05.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:59:05 trial068 ceph-mon[38805]: pgmap v1814: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:59:05.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:59:05 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3689334129' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:59:06.314 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T13:59:06.313994+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T13:59:06.315 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T13:59:06.315 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2740 8 365281 365273 1.04141 0 - 0.0299513 2026-02-23T13:59:06.693 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:59:06.809 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:59:06.916 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:59:06.914+0000 7fd324607640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:59:06.916 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:59:06.914+0000 7fd324607640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:59:06.927 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:59:06.928 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:59:06.937 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:59:06.938 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:59:06.938 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:59:06.948 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:59:06.948 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:59:06.979 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:59:06.980 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:59:07.314 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2741 8 365281 365273 1.04103 0 - 0.0299513 2026-02-23T13:59:07.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:59:07 trial068 ceph-mon[38805]: pgmap v1815: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:59:07.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:59:07 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1556119037' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:59:08.314 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2742 8 365281 365273 1.04065 0 - 0.0299513 2026-02-23T13:59:08.981 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:59:09.097 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:59:09.206 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:59:09.204+0000 7f7812e69640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:59:09.206 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:59:09.204+0000 7f7812e69640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:59:09.218 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:59:09.219 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:59:09.229 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:59:09.229 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:59:09.229 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:59:09.239 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:59:09.240 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:59:09.269 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:59:09.270 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:59:09.314 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2743 8 365281 365273 1.04027 0 - 0.0299513 2026-02-23T13:59:09.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:59:09 trial068 ceph-mon[38805]: pgmap v1816: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:59:09.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:59:09 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2971346658' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:59:10.314 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2744 8 365281 365273 1.03989 0 - 0.0299513 2026-02-23T13:59:11.271 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:59:11.314 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2745 8 365281 365273 1.03952 0 - 0.0299513 2026-02-23T13:59:11.388 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:59:11.482 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:59:11.480+0000 7f414e49e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:59:11.482 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:59:11.480+0000 7f414e49e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:59:11.495 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:59:11.496 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:59:11.506 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:59:11.506 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:59:11.506 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:59:11.517 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:59:11.517 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:59:11.545 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:59:11 trial068 ceph-mon[38805]: pgmap v1817: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:59:11.545 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:59:11 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2312188470' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:59:11.546 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:59:11.546 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:59:12.315 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2746 8 365281 365273 1.03914 0 - 0.0299513 2026-02-23T13:59:13.315 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2747 8 365281 365273 1.03876 0 - 0.0299513 2026-02-23T13:59:13.547 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:59:13.665 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:59:13.762 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:59:13.760+0000 7f5255ff8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:59:13.762 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:59:13.760+0000 7f5255ff8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:59:13.763 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:59:13 trial068 ceph-mon[38805]: pgmap v1818: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:59:13.774 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:59:13.775 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:59:13.785 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:59:13.785 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:59:13.785 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:59:13.795 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:59:13.796 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:59:13.823 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:59:13.824 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:59:14.094 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:59:13 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:59:13] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:59:14.315 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2748 8 365281 365273 1.03838 0 - 0.0299513 2026-02-23T13:59:14.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:59:13 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:59:13.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:59:14.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:59:14 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3295993746' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:59:15.315 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2749 8 365281 365273 1.038 0 - 0.0299513 2026-02-23T13:59:15.824 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:59:15.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:59:15 trial068 ceph-mon[38805]: pgmap v1819: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:59:15.941 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:59:16.038 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:59:16.036+0000 7fad5ddad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:59:16.039 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:59:16.037+0000 7fad5ddad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:59:16.050 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:59:16.051 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:59:16.061 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:59:16.062 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:59:16.062 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:59:16.072 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:59:16.072 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:59:16.103 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:59:16.104 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:59:16.315 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2750 8 365281 365273 1.03763 0 - 0.0299513 2026-02-23T13:59:16.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:59:16 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3866739419' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:59:17.315 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2751 8 365281 365273 1.03725 0 - 0.0299513 2026-02-23T13:59:17.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:59:17 trial068 ceph-mon[38805]: pgmap v1820: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:59:18.105 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:59:18.223 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:59:18.315 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2752 8 365281 365273 1.03687 0 - 0.0299513 2026-02-23T13:59:18.326 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:59:18.324+0000 7f1e0d9a0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:59:18.326 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:59:18.325+0000 7f1e0d9a0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:59:18.339 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:59:18.340 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:59:18.351 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:59:18.351 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:59:18.351 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:59:18.361 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:59:18.361 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:59:18.390 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:59:18.390 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:59:18.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:59:18 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2778746438' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:59:19.315 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2753 8 365281 365273 1.03649 0 - 0.0299513 2026-02-23T13:59:19.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:59:19 trial068 ceph-mon[38805]: pgmap v1821: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:59:20.315 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2754 8 365281 365273 1.03612 0 - 0.0299513 2026-02-23T13:59:20.391 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:59:20.509 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:59:20.614 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:59:20.612+0000 7f88b7ecd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:59:20.614 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:59:20.612+0000 7f88b7ecd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:59:20.626 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:59:20.626 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:59:20.636 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:59:20.636 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:59:20.637 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:59:20.646 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:59:20.646 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:59:20.676 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:59:20.677 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:59:21.316 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2755 8 365281 365273 1.03574 0 - 0.0299513 2026-02-23T13:59:21.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:59:21 trial068 ceph-mon[38805]: pgmap v1822: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:59:21.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:59:21 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2262965632' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:59:22.316 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2756 8 365281 365273 1.03537 0 - 0.0299513 2026-02-23T13:59:22.678 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:59:22.798 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:59:22.901 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:59:22.899+0000 7fcb4fbc2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:59:22.901 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:59:22.899+0000 7fcb4fbc2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:59:22.912 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:59:22.913 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:59:22.923 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:59:22.923 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:59:22.923 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:59:22.934 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:59:22.934 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:59:22.964 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:59:22.965 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:59:23.316 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2757 8 365281 365273 1.03499 0 - 0.0299513 2026-02-23T13:59:23.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:59:23 trial068 ceph-mon[38805]: pgmap v1823: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:59:23.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:59:23 trial068 ceph-mon[38805]: 5.8 scrub starts 2026-02-23T13:59:23.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:59:23 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2518994643' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:59:24.316 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2758 8 365281 365273 1.03462 0 - 0.0299513 2026-02-23T13:59:24.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:59:23 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:59:23] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:59:24.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:59:23 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:59:23.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:59:24.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:59:24 trial068 ceph-mon[38805]: 5.8 scrub ok 2026-02-23T13:59:24.966 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:59:25.085 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:59:25.192 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:59:25.190+0000 7f9cd9d58640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:59:25.192 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:59:25.190+0000 7f9cd9d58640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:59:25.204 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:59:25.204 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:59:25.214 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:59:25.215 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:59:25.215 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:59:25.225 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:59:25.225 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:59:25.256 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:59:25.256 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:59:25.316 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2759 8 365281 365273 1.03424 0 - 0.0299513 2026-02-23T13:59:25.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:59:25 trial068 ceph-mon[38805]: pgmap v1824: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:59:25.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:59:25 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/669570441' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:59:26.316 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T13:59:26.315690+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T13:59:26.316 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T13:59:26.316 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2760 8 365281 365273 1.03387 0 - 0.0299513 2026-02-23T13:59:27.257 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:59:27.316 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2761 8 365281 365273 1.03349 0 - 0.0299513 2026-02-23T13:59:27.374 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:59:27.472 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:59:27.470+0000 7f43d8e9a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:59:27.472 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:59:27.470+0000 7f43d8e9a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:59:27.483 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:59:27.483 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:59:27.493 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:59:27.494 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:59:27.494 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:59:27.503 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:59:27.504 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:59:27.533 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:59:27 trial068 ceph-mon[38805]: pgmap v1825: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:59:27.533 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:59:27 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/556281869' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:59:27.534 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:59:27.534 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:59:28.316 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2762 8 365281 365273 1.03312 0 - 0.0299513 2026-02-23T13:59:29.316 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2763 8 365281 365273 1.03274 0 - 0.0299513 2026-02-23T13:59:29.535 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:59:29.653 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:59:29.749 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:59:29.747+0000 7f223969e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:59:29.749 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:59:29.747+0000 7f223969e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:59:29.761 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:59:29.761 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:59:29.762 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:59:29 trial068 ceph-mon[38805]: pgmap v1826: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:59:29.771 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:59:29.771 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:59:29.772 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:59:29.781 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:59:29.782 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:59:29.812 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:59:29.812 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:59:30.316 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2764 8 365281 365273 1.03237 0 - 0.0299513 2026-02-23T13:59:30.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:59:30 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2318734797' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:59:31.316 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2765 8 365281 365273 1.032 0 - 0.0299513 2026-02-23T13:59:31.813 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:59:31.837 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:59:31 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T13:59:31.837 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:59:31 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T13:59:31.837 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:59:31 trial068 ceph-mon[38805]: pgmap v1827: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:59:31.929 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:59:32.023 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:59:32.021+0000 7f423ebb0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:59:32.023 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:59:32.021+0000 7f423ebb0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:59:32.035 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:59:32.035 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:59:32.045 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:59:32.045 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:59:32.045 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:59:32.055 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:59:32.055 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:59:32.084 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:59:32.084 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:59:32.316 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2766 8 365281 365273 1.03162 0 - 0.0299513 2026-02-23T13:59:32.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:59:32 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/589911100' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:59:33.316 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2767 8 365281 365273 1.03125 0 - 0.0299513 2026-02-23T13:59:33.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:59:33 trial068 ceph-mon[38805]: pgmap v1828: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:59:34.085 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:59:34.204 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:59:34.226 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:59:33 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:59:33] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:59:34.266 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:59:33 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:59:33.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:59:34.303 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:59:34.300+0000 7f0643ca0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:59:34.303 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:59:34.301+0000 7f0643ca0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:59:34.315 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:59:34.316 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:59:34.316 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2768 8 365281 365273 1.03088 0 - 0.0299513 2026-02-23T13:59:34.326 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:59:34.326 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:59:34.326 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:59:34.336 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:59:34.336 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:59:34.368 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:59:34.368 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:59:34.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:59:34 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3691333840' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:59:35.317 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2769 8 365281 365273 1.03051 0 - 0.0299513 2026-02-23T13:59:35.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:59:35 trial068 ceph-mon[38805]: pgmap v1829: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:59:36.317 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2770 8 365281 365273 1.03013 0 - 0.0299513 2026-02-23T13:59:36.369 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:59:36.487 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:59:36.595 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:59:36.593+0000 7fdc93a1f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:59:36.595 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:59:36.593+0000 7fdc93a1f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:59:36.607 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:59:36.607 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:59:36.618 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:59:36.619 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:59:36.619 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:59:36.629 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:59:36.629 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:59:36.659 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:59:36.660 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:59:37.317 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2771 8 365281 365273 1.02976 0 - 0.0299513 2026-02-23T13:59:37.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:59:37 trial068 ceph-mon[38805]: pgmap v1830: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:59:37.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:59:37 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2920520671' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:59:38.317 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2772 8 365281 365273 1.02939 0 - 0.0299513 2026-02-23T13:59:38.660 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:59:38.779 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:59:38.874 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:59:38.872+0000 7fe3606d4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:59:38.875 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:59:38.873+0000 7fe3606d4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:59:38.886 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:59:38.886 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:59:38.896 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:59:38.897 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:59:38.897 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:59:38.907 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:59:38.907 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:59:38.936 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:59:38.937 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:59:39.317 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2773 8 365281 365273 1.02902 0 - 0.0299513 2026-02-23T13:59:39.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:59:39 trial068 ceph-mon[38805]: pgmap v1831: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:59:39.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:59:39 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1632626037' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:59:40.317 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2774 8 365281 365273 1.02865 0 - 0.0299513 2026-02-23T13:59:40.938 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:59:41.056 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:59:41.156 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:59:41.154+0000 7f533691b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:59:41.156 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:59:41.155+0000 7f533691b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:59:41.168 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:59:41.169 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:59:41.178 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:59:41.179 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:59:41.179 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:59:41.188 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:59:41.189 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:59:41.219 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:59:41.220 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:59:41.317 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2775 8 365281 365273 1.02828 0 - 0.0299513 2026-02-23T13:59:41.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:59:41 trial068 ceph-mon[38805]: pgmap v1832: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:59:41.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:59:41 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/4013826022' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:59:42.317 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2776 8 365281 365273 1.02791 0 - 0.0299513 2026-02-23T13:59:43.221 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:59:43.317 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2777 8 365281 365273 1.02754 0 - 0.0299513 2026-02-23T13:59:43.337 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:59:43.438 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:59:43.436+0000 7f6992fba640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:59:43.438 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:59:43.437+0000 7f6992fba640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:59:43.451 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:59:43.451 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:59:43.461 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:59:43.461 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:59:43.461 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:59:43.472 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:59:43.472 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:59:43.501 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:59:43.501 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:59:43.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:59:43 trial068 ceph-mon[38805]: pgmap v1833: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:59:43.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:59:43 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/4031995921' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:59:44.317 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2778 8 365281 365273 1.02717 0 - 0.0299513 2026-02-23T13:59:44.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:59:43 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:59:43] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:59:44.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:59:43 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:59:43.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:59:45.317 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2779 8 365281 365273 1.0268 0 - 0.0299513 2026-02-23T13:59:45.502 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:59:45.620 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:59:45.726 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:59:45.724+0000 7f6b96864640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:59:45.726 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:59:45.725+0000 7f6b96864640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:59:45.738 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:59:45.738 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:59:45.748 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:59:45.748 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:59:45.748 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:59:45.759 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:59:45.759 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:59:45.771 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:59:45 trial068 ceph-mon[38805]: pgmap v1834: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:59:45.789 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:59:45.789 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:59:46.317 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T13:59:46.317175+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T13:59:46.317 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T13:59:46.317 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2780 8 365281 365273 1.02643 0 - 0.0299513 2026-02-23T13:59:46.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:59:46 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3588228047' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:59:47.318 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2781 8 365281 365273 1.02606 0 - 0.0299513 2026-02-23T13:59:47.790 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:59:47.815 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:59:47 trial068 ceph-mon[38805]: pgmap v1835: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:59:47.815 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:59:47 trial068 ceph-mon[38805]: 5.7 scrub starts 2026-02-23T13:59:47.815 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:59:47 trial068 ceph-mon[38805]: 5.7 scrub ok 2026-02-23T13:59:47.909 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:59:48.005 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:59:48.003+0000 7f106f5dd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:59:48.005 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:59:48.003+0000 7f106f5dd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:59:48.017 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:59:48.017 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:59:48.027 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:59:48.027 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:59:48.027 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:59:48.038 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:59:48.038 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:59:48.068 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:59:48.069 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:59:48.318 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2782 8 365281 365273 1.02569 0 - 0.0299513 2026-02-23T13:59:48.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:59:48 trial068 ceph-mon[38805]: 5.e scrub starts 2026-02-23T13:59:48.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:59:48 trial068 ceph-mon[38805]: 5.e scrub ok 2026-02-23T13:59:48.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:59:48 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/4184648025' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:59:48.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:59:48 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2026-02-23T13:59:48.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:59:48 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-23T13:59:48.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:59:48 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2026-02-23T13:59:48.845 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:59:48 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T13:59:48.845 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:59:48 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2026-02-23T13:59:49.318 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2783 8 365281 365273 1.02532 0 - 0.0299513 2026-02-23T13:59:49.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:59:49 trial068 ceph-mon[38805]: pgmap v1836: 17 pgs: 1 active+clean+scrubbing, 1 active+recovery_unfound+undersized+degraded, 4 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:59:49.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:59:49 trial068 ceph-mon[38805]: pgmap v1837: 17 pgs: 1 active+clean+scrubbing, 1 active+recovery_unfound+undersized+degraded, 4 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:59:49.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:59:49 trial068 ceph-mon[38805]: pgmap v1838: 17 pgs: 1 active+clean+scrubbing, 1 active+recovery_unfound+undersized+degraded, 4 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:59:50.069 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:59:50.188 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:59:50.293 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:59:50.291+0000 7fb33763a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:59:50.293 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:59:50.291+0000 7fb33763a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:59:50.305 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:59:50.305 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:59:50.315 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:59:50.316 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:59:50.316 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:59:50.318 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2784 8 365281 365273 1.02495 0 - 0.0299513 2026-02-23T13:59:50.326 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:59:50.327 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:59:50.358 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:59:50.358 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:59:50.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:59:50 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1776264537' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:59:51.318 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2785 8 365281 365273 1.02459 0 - 0.0299513 2026-02-23T13:59:51.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:59:51 trial068 ceph-mon[38805]: pgmap v1839: 17 pgs: 1 active+clean+scrubbing, 1 active+recovery_unfound+undersized+degraded, 4 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:59:52.318 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2786 8 365281 365273 1.02422 0 - 0.0299513 2026-02-23T13:59:52.359 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:59:52.477 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:59:52.575 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:59:52.573+0000 7f28c3c02640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:59:52.576 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:59:52.574+0000 7f28c3c02640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:59:52.588 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:59:52.588 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:59:52.598 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:59:52.599 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:59:52.599 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:59:52.610 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:59:52.610 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:59:52.658 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:59:52.659 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:59:53.318 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2787 8 365281 365273 1.02385 0 - 0.0299513 2026-02-23T13:59:53.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:59:53 trial068 ceph-mon[38805]: pgmap v1840: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:59:53.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:59:53 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1954951362' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:59:54.318 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2788 8 365281 365273 1.02348 0 - 0.0299513 2026-02-23T13:59:54.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 13:59:53 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:13:59:53] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T13:59:54.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 13:59:53 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T13:59:53.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T13:59:54.660 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:59:54.776 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:59:54.879 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:59:54.877+0000 7fd9756fe640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:59:54.879 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:59:54.877+0000 7fd9756fe640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:59:54.891 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:59:54.891 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:59:54.901 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:59:54.902 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:59:54.902 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:59:54.912 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:59:54.912 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:59:54.941 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:59:54.942 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:59:55.318 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2789 8 365281 365273 1.02312 0 - 0.0299513 2026-02-23T13:59:55.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:59:55 trial068 ceph-mon[38805]: pgmap v1841: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:59:55.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:59:55 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3031002523' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:59:56.318 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2790 8 365281 365273 1.02275 0 - 0.0299513 2026-02-23T13:59:56.943 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T13:59:57.060 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:59:57.159 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:59:57.157+0000 7f766c3b5640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:59:57.160 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:59:57.158+0000 7f766c3b5640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:59:57.172 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:59:57.172 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:59:57.182 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:59:57.183 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:59:57.183 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:59:57.192 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:59:57.193 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:59:57.223 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:59:57.224 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:59:57.318 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2791 8 365281 365273 1.02238 0 - 0.0299513 2026-02-23T13:59:57.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:59:57 trial068 ceph-mon[38805]: pgmap v1842: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:59:57.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:59:57 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3519330431' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T13:59:58.318 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2792 8 365281 365273 1.02202 0 - 0.0299513 2026-02-23T13:59:59.225 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T13:59:59.318 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2793 8 365281 365273 1.02165 0 - 0.0299513 2026-02-23T13:59:59.341 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T13:59:59.442 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:59:59.440+0000 7f6089781640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:59:59.442 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T13:59:59.440+0000 7f6089781640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T13:59:59.454 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T13:59:59.454 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T13:59:59.465 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T13:59:59.465 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T13:59:59.465 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T13:59:59.475 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T13:59:59.475 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T13:59:59.505 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T13:59:59.506 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T13:59:59.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:59:59 trial068 ceph-mon[38805]: pgmap v1843: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T13:59:59.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 13:59:59 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1445016221' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T14:00:00.318 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2794 8 365281 365273 1.02128 0 - 0.0299513 2026-02-23T14:00:01.319 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2795 8 365281 365273 1.02092 0 - 0.0299513 2026-02-23T14:00:01.507 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T14:00:01.624 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:00:01.732 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:00:01.730+0000 7efd9b1fd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:00:01.732 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:00:01.730+0000 7efd9b1fd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:00:01.744 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T14:00:01.744 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T14:00:01.754 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T14:00:01.755 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T14:00:01.755 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T14:00:01.764 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T14:00:01.765 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T14:00:01.791 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:00:01 trial068 ceph-mon[38805]: pgmap v1844: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:00:01.792 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T14:00:01.793 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T14:00:02.319 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2796 8 365281 365273 1.02055 0 - 0.0299513 2026-02-23T14:00:02.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:00:02 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3844958775' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T14:00:03.319 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2797 8 365281 365273 1.02019 0 - 0.0299513 2026-02-23T14:00:03.794 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T14:00:03.820 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:00:03 trial068 ceph-mon[38805]: pgmap v1845: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:00:03.915 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:00:04.017 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:00:04.015+0000 7f0904548640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:00:04.017 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:00:04.015+0000 7f0904548640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:00:04.029 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T14:00:04.030 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T14:00:04.040 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T14:00:04.041 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T14:00:04.041 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T14:00:04.050 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T14:00:04.051 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T14:00:04.082 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:00:03 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:00:03] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:00:04.083 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T14:00:04.084 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T14:00:04.266 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:00:03 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:00:03.972Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:00:04.319 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2798 8 365281 365273 1.01982 0 - 0.0299513 2026-02-23T14:00:04.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:00:04 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T14:00:04.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:00:04 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T14:00:04.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:00:04 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2065862317' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T14:00:05.319 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2799 8 365281 365273 1.01946 0 - 0.0299513 2026-02-23T14:00:05.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:00:05 trial068 ceph-mon[38805]: pgmap v1846: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:00:06.084 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T14:00:06.203 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:00:06.305 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:00:06.303+0000 7f3ac3e37640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:00:06.306 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:00:06.304+0000 7f3ac3e37640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:00:06.318 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T14:00:06.318 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T14:00:06.319 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T14:00:06.318680+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T14:00:06.319 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T14:00:06.319 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2800 8 365281 365273 1.0191 0 - 0.0299513 2026-02-23T14:00:06.328 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T14:00:06.329 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T14:00:06.329 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T14:00:06.338 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T14:00:06.338 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T14:00:06.366 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T14:00:06.367 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T14:00:06.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:00:06 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2547005460' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T14:00:07.319 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2801 8 365281 365273 1.01873 0 - 0.0299513 2026-02-23T14:00:07.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:00:07 trial068 ceph-mon[38805]: pgmap v1847: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:00:08.319 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2802 8 365281 365273 1.01837 0 - 0.0299513 2026-02-23T14:00:08.368 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T14:00:08.484 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:00:08.584 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:00:08.582+0000 7f6c205c3640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:00:08.584 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:00:08.582+0000 7f6c205c3640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:00:08.596 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T14:00:08.596 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T14:00:08.606 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T14:00:08.607 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T14:00:08.607 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T14:00:08.616 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T14:00:08.616 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T14:00:08.645 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T14:00:08.645 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T14:00:09.319 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2803 8 365281 365273 1.01801 0 - 0.0299513 2026-02-23T14:00:09.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:00:09 trial068 ceph-mon[38805]: pgmap v1848: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:00:09.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:00:09 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2911411425' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T14:00:10.319 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2804 8 365281 365273 1.01764 0 - 0.0299513 2026-02-23T14:00:10.646 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T14:00:10.763 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:00:10.804 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:00:10 trial068 ceph-mon[38805]: 5.5 scrub starts 2026-02-23T14:00:10.804 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:00:10 trial068 ceph-mon[38805]: 5.5 scrub ok 2026-02-23T14:00:10.804 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:00:10 trial068 ceph-mon[38805]: osdmap e215: 4 total, 3 up, 4 in 2026-02-23T14:00:10.859 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:00:10.858+0000 7f219144b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:00:10.859 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:00:10.858+0000 7f219144b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:00:10.871 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T14:00:10.871 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T14:00:10.881 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T14:00:10.882 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T14:00:10.882 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T14:00:10.892 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T14:00:10.892 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T14:00:10.922 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T14:00:10.922 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T14:00:11.319 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2805 8 365281 365273 1.01728 0 - 0.0299513 2026-02-23T14:00:11.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:00:11 trial068 ceph-mon[38805]: pgmap v1850: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:00:11.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:00:11 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/4230269109' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T14:00:12.319 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2806 8 365281 365273 1.01692 0 - 0.0299513 2026-02-23T14:00:12.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:00:12 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T14:00:12.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:00:12 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T14:00:12.923 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T14:00:13.043 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:00:13.138 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:00:13.136+0000 7fd893034640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:00:13.138 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:00:13.136+0000 7fd893034640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:00:13.149 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T14:00:13.150 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T14:00:13.159 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T14:00:13.160 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T14:00:13.160 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T14:00:13.169 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T14:00:13.170 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T14:00:13.199 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T14:00:13.200 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T14:00:13.320 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2807 8 365281 365273 1.01656 0 - 0.0299513 2026-02-23T14:00:13.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:00:13 trial068 ceph-mon[38805]: pgmap v1851: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:00:13.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:00:13 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/946970685' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T14:00:14.320 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2808 8 365281 365273 1.01619 0 - 0.0299513 2026-02-23T14:00:14.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:00:13 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:00:13] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:00:14.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:00:13 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:00:13.974Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:00:15.201 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T14:00:15.319 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:00:15.319 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2809 8 365281 365273 1.01583 0 - 0.0299513 2026-02-23T14:00:15.422 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:00:15.420+0000 7f2ab05eb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:00:15.423 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:00:15.421+0000 7f2ab05eb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:00:15.434 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T14:00:15.435 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T14:00:15.445 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T14:00:15.445 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T14:00:15.445 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T14:00:15.455 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T14:00:15.455 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T14:00:15.484 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:00:15 trial068 ceph-mon[38805]: pgmap v1852: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:00:15.486 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T14:00:15.486 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T14:00:16.320 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2810 8 365281 365273 1.01547 0 - 0.0299513 2026-02-23T14:00:16.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:00:16 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3803204529' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T14:00:17.320 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2811 8 365281 365273 1.01511 0 - 0.0299513 2026-02-23T14:00:17.487 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T14:00:17.606 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:00:17.654 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:00:17 trial068 ceph-mon[38805]: pgmap v1853: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:00:17.709 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:00:17.706+0000 7f2f1a03a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:00:17.709 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:00:17.707+0000 7f2f1a03a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:00:17.720 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T14:00:17.721 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T14:00:17.731 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T14:00:17.731 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T14:00:17.731 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T14:00:17.741 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T14:00:17.741 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T14:00:17.769 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T14:00:17.769 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T14:00:18.320 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2812 8 365281 365273 1.01475 0 - 0.0299513 2026-02-23T14:00:18.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:00:18 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/4029963610' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T14:00:19.320 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2813 8 365281 365273 1.01439 0 - 0.0299513 2026-02-23T14:00:19.770 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T14:00:19.795 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:00:19 trial068 ceph-mon[38805]: 5.a scrub starts 2026-02-23T14:00:19.795 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:00:19 trial068 ceph-mon[38805]: 5.a scrub ok 2026-02-23T14:00:19.795 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:00:19 trial068 ceph-mon[38805]: pgmap v1854: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:00:19.888 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:00:19.997 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:00:19.995+0000 7f63e8fe3640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:00:19.997 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:00:19.995+0000 7f63e8fe3640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:00:20.009 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T14:00:20.009 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T14:00:20.018 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T14:00:20.019 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T14:00:20.019 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T14:00:20.029 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T14:00:20.029 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T14:00:20.057 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T14:00:20.058 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T14:00:20.320 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2814 8 365281 365273 1.01403 0 - 0.0299513 2026-02-23T14:00:20.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:00:20 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1001067697' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T14:00:21.320 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2815 8 365281 365273 1.01367 0 - 0.0299513 2026-02-23T14:00:21.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:00:21 trial068 ceph-mon[38805]: pgmap v1855: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:00:22.059 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T14:00:22.176 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:00:22.274 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:00:22.272+0000 7f1dc117a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:00:22.274 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:00:22.272+0000 7f1dc117a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:00:22.285 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T14:00:22.286 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T14:00:22.295 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T14:00:22.296 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T14:00:22.296 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T14:00:22.306 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T14:00:22.306 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T14:00:22.320 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2816 8 365281 365273 1.01331 0 - 0.0299513 2026-02-23T14:00:22.339 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T14:00:22.340 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T14:00:22.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:00:22 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2900089156' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T14:00:23.320 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2817 8 365281 365273 1.01295 0 - 0.0299513 2026-02-23T14:00:23.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:00:23 trial068 ceph-mon[38805]: pgmap v1856: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:00:24.320 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2818 8 365281 365273 1.01259 0 - 0.0299513 2026-02-23T14:00:24.341 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T14:00:24.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:00:23 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:00:23] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:00:24.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:00:23 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:00:23.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:00:24.460 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:00:24.562 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:00:24.559+0000 7fdfcfcb6640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:00:24.562 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:00:24.560+0000 7fdfcfcb6640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:00:24.574 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T14:00:24.574 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T14:00:24.583 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T14:00:24.584 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T14:00:24.584 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T14:00:24.594 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T14:00:24.595 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T14:00:24.623 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T14:00:24.624 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T14:00:25.320 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2819 8 365281 365273 1.01223 0 - 0.0299513 2026-02-23T14:00:25.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:00:25 trial068 ceph-mon[38805]: pgmap v1857: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:00:25.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:00:25 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1100766006' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T14:00:26.320 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T14:00:26.320196+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T14:00:26.321 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T14:00:26.321 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2820 8 365281 365273 1.01187 0 - 0.0299513 2026-02-23T14:00:26.625 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T14:00:26.744 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:00:26.841 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:00:26.839+0000 7fac0e510640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:00:26.841 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:00:26.839+0000 7fac0e510640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:00:26.853 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T14:00:26.854 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T14:00:26.863 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T14:00:26.864 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T14:00:26.864 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T14:00:26.874 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T14:00:26.874 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T14:00:26.904 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T14:00:26.905 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T14:00:27.321 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2821 8 365281 365273 1.01151 0 - 0.0299513 2026-02-23T14:00:27.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:00:27 trial068 ceph-mon[38805]: pgmap v1858: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:00:27.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:00:27 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/4100657351' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T14:00:28.321 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2822 8 365281 365273 1.01115 0 - 0.0299513 2026-02-23T14:00:28.906 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T14:00:29.023 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:00:29.124 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:00:29.122+0000 7f9e68714640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:00:29.124 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:00:29.122+0000 7f9e68714640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:00:29.135 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T14:00:29.135 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T14:00:29.145 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T14:00:29.145 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T14:00:29.146 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T14:00:29.155 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T14:00:29.155 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T14:00:29.187 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T14:00:29.187 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T14:00:29.321 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2823 8 365281 365273 1.01079 0 - 0.0299513 2026-02-23T14:00:29.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:00:29 trial068 ceph-mon[38805]: pgmap v1859: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:00:29.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:00:29 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3521506371' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T14:00:30.321 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2824 8 365281 365273 1.01044 0 - 0.0299513 2026-02-23T14:00:31.189 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T14:00:31.305 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:00:31.321 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2825 8 365281 365273 1.01008 0 - 0.0299513 2026-02-23T14:00:31.403 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:00:31.401+0000 7fab10b8f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:00:31.403 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:00:31.401+0000 7fab10b8f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:00:31.415 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T14:00:31.415 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T14:00:31.425 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T14:00:31.426 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T14:00:31.426 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T14:00:31.435 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T14:00:31.435 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T14:00:31.465 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T14:00:31.465 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T14:00:31.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:00:31 trial068 ceph-mon[38805]: pgmap v1860: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:00:31.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:00:31 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3074313181' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T14:00:32.321 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2826 8 365281 365273 1.00972 0 - 0.0299513 2026-02-23T14:00:33.321 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2827 8 365281 365273 1.00936 0 - 0.0299513 2026-02-23T14:00:33.466 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T14:00:33.584 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:00:33.688 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:00:33.685+0000 7f25fe293640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:00:33.688 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:00:33.686+0000 7f25fe293640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:00:33.700 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T14:00:33.700 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T14:00:33.710 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T14:00:33.711 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T14:00:33.711 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T14:00:33.721 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T14:00:33.721 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T14:00:33.748 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:00:33 trial068 ceph-mon[38805]: pgmap v1861: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:00:33.749 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T14:00:33.750 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T14:00:34.094 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:00:33 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:00:33] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:00:34.266 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:00:33 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:00:33.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:00:34.321 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2828 8 365281 365273 1.00901 0 - 0.0299513 2026-02-23T14:00:34.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:00:34 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/226742601' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T14:00:35.321 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2829 8 365281 365273 1.00865 0 - 0.0299513 2026-02-23T14:00:35.751 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T14:00:35.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:00:35 trial068 ceph-mon[38805]: pgmap v1862: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:00:35.869 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:00:35.966 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:00:35.964+0000 7f1c14261640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:00:35.967 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:00:35.964+0000 7f1c14261640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:00:35.979 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T14:00:35.979 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T14:00:35.989 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T14:00:35.989 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T14:00:35.989 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T14:00:35.999 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T14:00:36.000 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T14:00:36.028 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T14:00:36.028 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T14:00:36.321 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2830 8 365281 365273 1.00829 0 - 0.0299513 2026-02-23T14:00:36.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:00:36 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3825371182' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T14:00:37.321 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2831 8 365281 365273 1.00794 0 - 0.0299513 2026-02-23T14:00:37.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:00:37 trial068 ceph-mon[38805]: pgmap v1863: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:00:38.030 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T14:00:38.147 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:00:38.244 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:00:38.242+0000 7fb05a147640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:00:38.245 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:00:38.243+0000 7fb05a147640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:00:38.256 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T14:00:38.257 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T14:00:38.266 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T14:00:38.267 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T14:00:38.267 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T14:00:38.276 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T14:00:38.277 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T14:00:38.305 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T14:00:38.306 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T14:00:38.321 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2832 8 365281 365273 1.00758 0 - 0.0299513 2026-02-23T14:00:38.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:00:38 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2518600806' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T14:00:39.321 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2833 8 365281 365273 1.00723 0 - 0.0299513 2026-02-23T14:00:39.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:00:39 trial068 ceph-mon[38805]: pgmap v1864: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:00:40.307 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T14:00:40.321 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2834 8 365281 365273 1.00687 0 - 0.0299513 2026-02-23T14:00:40.424 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:00:40.523 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:00:40.521+0000 7ff0b7371640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:00:40.523 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:00:40.521+0000 7ff0b7371640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:00:40.535 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T14:00:40.535 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T14:00:40.545 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T14:00:40.546 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T14:00:40.546 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T14:00:40.556 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T14:00:40.556 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T14:00:40.584 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:00:40 trial068 ceph-mon[38805]: 5.8 scrub starts 2026-02-23T14:00:40.584 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:00:40 trial068 ceph-mon[38805]: 5.8 scrub ok 2026-02-23T14:00:40.584 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:00:40 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/514656399' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T14:00:40.585 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T14:00:40.586 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T14:00:41.322 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2835 8 365281 365273 1.00652 0 - 0.0299513 2026-02-23T14:00:41.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:00:41 trial068 ceph-mon[38805]: pgmap v1865: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:00:42.322 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2836 8 365281 365273 1.00616 0 - 0.0299513 2026-02-23T14:00:42.587 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T14:00:42.706 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:00:42.802 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:00:42.800+0000 7fe951ff2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:00:42.802 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:00:42.800+0000 7fe951ff2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:00:42.813 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T14:00:42.814 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T14:00:42.823 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T14:00:42.824 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T14:00:42.824 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T14:00:42.834 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T14:00:42.834 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T14:00:42.866 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T14:00:42.866 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T14:00:43.322 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2837 8 365281 365273 1.00581 0 - 0.0299513 2026-02-23T14:00:43.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:00:43 trial068 ceph-mon[38805]: pgmap v1866: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:00:43.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:00:43 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2991660654' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T14:00:44.322 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2838 8 365281 365273 1.00545 0 - 0.0299513 2026-02-23T14:00:44.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:00:43 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:00:43] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:00:44.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:00:43 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:00:43.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:00:44.867 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T14:00:44.986 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:00:45.087 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:00:45.085+0000 7ff410758640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:00:45.088 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:00:45.086+0000 7ff410758640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:00:45.099 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T14:00:45.100 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T14:00:45.110 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T14:00:45.111 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T14:00:45.111 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T14:00:45.121 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T14:00:45.121 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T14:00:45.151 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T14:00:45.152 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T14:00:45.322 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2839 8 365281 365273 1.0051 0 - 0.0299513 2026-02-23T14:00:45.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:00:45 trial068 ceph-mon[38805]: pgmap v1867: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:00:45.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:00:45 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1288815589' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T14:00:46.322 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T14:00:46.322070+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T14:00:46.323 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T14:00:46.323 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2840 8 365281 365273 1.00474 0 - 0.0299513 2026-02-23T14:00:47.153 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T14:00:47.271 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:00:47.322 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2841 8 365281 365273 1.00439 0 - 0.0299513 2026-02-23T14:00:47.369 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:00:47.367+0000 7ff4f79ab640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:00:47.369 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:00:47.367+0000 7ff4f79ab640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:00:47.381 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T14:00:47.381 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T14:00:47.391 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T14:00:47.391 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T14:00:47.391 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T14:00:47.401 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T14:00:47.401 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T14:00:47.432 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T14:00:47.433 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T14:00:47.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:00:47 trial068 ceph-mon[38805]: pgmap v1868: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:00:47.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:00:47 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/209462534' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T14:00:48.323 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2842 8 365281 365273 1.00404 0 - 0.0299513 2026-02-23T14:00:48.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:00:48 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2026-02-23T14:00:49.323 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2843 8 365281 365273 1.00368 0 - 0.0299513 2026-02-23T14:00:49.434 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T14:00:49.550 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:00:49.649 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:00:49.647+0000 7fe6ab5e8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:00:49.649 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:00:49.647+0000 7fe6ab5e8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:00:49.661 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T14:00:49.661 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T14:00:49.671 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T14:00:49.672 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T14:00:49.672 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T14:00:49.682 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T14:00:49.682 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T14:00:49.715 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:00:49 trial068 ceph-mon[38805]: pgmap v1869: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:00:49.715 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:00:49 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-23T14:00:49.715 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:00:49 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2026-02-23T14:00:49.716 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:00:49 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T14:00:49.716 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:00:49 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2026-02-23T14:00:49.716 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T14:00:49.717 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T14:00:50.323 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2844 8 365281 365273 1.00333 0 - 0.0299513 2026-02-23T14:00:50.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:00:50 trial068 ceph-mon[38805]: pgmap v1870: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:00:50.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:00:50 trial068 ceph-mon[38805]: pgmap v1871: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:00:50.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:00:50 trial068 ceph-mon[38805]: 5.7 scrub starts 2026-02-23T14:00:50.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:00:50 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/119099836' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T14:00:50.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:00:50 trial068 ceph-mon[38805]: 5.7 scrub ok 2026-02-23T14:00:51.323 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2845 8 365281 365273 1.00298 0 - 0.0299513 2026-02-23T14:00:51.718 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T14:00:51.836 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:00:51.932 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:00:51.930+0000 7f6ae8200640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:00:51.933 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:00:51.930+0000 7f6ae8200640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:00:51.944 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T14:00:51.944 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T14:00:51.954 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T14:00:51.954 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T14:00:51.954 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T14:00:51.965 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T14:00:51.965 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T14:00:51.996 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T14:00:51.996 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T14:00:52.323 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2846 8 365281 365273 1.00262 0 - 0.0299513 2026-02-23T14:00:52.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:00:52 trial068 ceph-mon[38805]: pgmap v1872: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:00:52.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:00:52 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2488405161' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T14:00:53.323 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2847 8 365281 365273 1.00227 0 - 0.0299513 2026-02-23T14:00:53.998 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T14:00:54.114 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:00:54.213 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:00:54.211+0000 7fdf63ad6640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:00:54.213 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:00:54.211+0000 7fdf63ad6640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:00:54.225 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T14:00:54.226 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T14:00:54.226 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:00:53 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:00:53] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:00:54.237 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T14:00:54.237 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T14:00:54.237 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T14:00:54.248 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T14:00:54.248 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T14:00:54.280 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T14:00:54.281 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T14:00:54.323 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2848 8 365281 365273 1.00192 0 - 0.0299513 2026-02-23T14:00:54.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:00:53 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:00:53.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:00:54.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:00:54 trial068 ceph-mon[38805]: pgmap v1873: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:00:54.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:00:54 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T14:00:54.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:00:54 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T14:00:54.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:00:54 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1991856992' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T14:00:55.323 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2849 8 365281 365273 1.00157 0 - 0.0299513 2026-02-23T14:00:56.281 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T14:00:56.323 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2850 8 365281 365273 1.00122 0 - 0.0299513 2026-02-23T14:00:56.400 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:00:56.500 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:00:56.498+0000 7f162b63f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:00:56.501 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:00:56.498+0000 7f162b63f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:00:56.512 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T14:00:56.512 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T14:00:56.522 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T14:00:56.522 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T14:00:56.522 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T14:00:56.532 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T14:00:56.533 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T14:00:56.563 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T14:00:56.564 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T14:00:56.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:00:56 trial068 ceph-mon[38805]: pgmap v1874: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:00:56.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:00:56 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3030927758' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T14:00:57.323 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2851 8 365281 365273 1.00087 0 - 0.0299513 2026-02-23T14:00:58.323 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2852 8 365281 365273 1.00052 0 - 0.0299513 2026-02-23T14:00:58.565 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T14:00:58.681 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:00:58.780 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:00:58.778+0000 7fd5b1e2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:00:58.780 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:00:58.778+0000 7fd5b1e2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:00:58.792 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T14:00:58.792 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T14:00:58.802 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T14:00:58.802 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T14:00:58.802 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T14:00:58.812 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T14:00:58.813 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T14:00:58.822 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:00:58 trial068 ceph-mon[38805]: pgmap v1875: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:00:58.843 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T14:00:58.843 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T14:00:59.323 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2853 8 365281 365273 1.00016 0 - 0.0299513 2026-02-23T14:00:59.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:00:59 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1544056905' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T14:01:00.323 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2854 8 365281 365273 0.999814 0 - 0.0299513 2026-02-23T14:01:00.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:01:00 trial068 ceph-mon[38805]: pgmap v1876: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:01:00.844 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T14:01:00.960 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:01:01.063 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:01:01.061+0000 7ff6f80e9640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:01:01.063 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:01:01.061+0000 7ff6f80e9640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:01:01.075 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T14:01:01.075 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T14:01:01.085 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T14:01:01.086 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T14:01:01.086 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T14:01:01.097 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T14:01:01.097 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T14:01:01.128 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T14:01:01.128 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T14:01:01.323 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2855 8 365281 365273 0.999464 0 - 0.0299513 2026-02-23T14:01:01.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:01:01 trial068 ceph-mon[38805]: 5.e scrub starts 2026-02-23T14:01:01.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:01:01 trial068 ceph-mon[38805]: 5.e scrub ok 2026-02-23T14:01:01.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:01:01 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2416721880' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T14:01:02.323 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2856 8 365281 365273 0.999114 0 - 0.0299513 2026-02-23T14:01:02.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:01:02 trial068 ceph-mon[38805]: pgmap v1877: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:01:03.129 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T14:01:03.250 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:01:03.324 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2857 8 365281 365273 0.998765 0 - 0.0299513 2026-02-23T14:01:03.351 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:01:03.349+0000 7fc701259640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:01:03.351 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:01:03.349+0000 7fc701259640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:01:03.364 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T14:01:03.364 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T14:01:03.374 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T14:01:03.375 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T14:01:03.375 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T14:01:03.386 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T14:01:03.386 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T14:01:03.416 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T14:01:03.417 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T14:01:03.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:01:03 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/535978870' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T14:01:04.324 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2858 8 365281 365273 0.998415 0 - 0.0299513 2026-02-23T14:01:04.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:01:03 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:01:03] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:01:04.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:01:03 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:01:03.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:01:04.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:01:04 trial068 ceph-mon[38805]: pgmap v1878: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:01:05.324 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2859 8 365281 365273 0.998066 0 - 0.0299513 2026-02-23T14:01:05.417 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T14:01:05.535 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:01:05.636 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:01:05.634+0000 7f66d05a4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:01:05.636 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:01:05.634+0000 7f66d05a4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:01:05.649 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T14:01:05.649 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T14:01:05.658 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T14:01:05.659 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T14:01:05.659 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T14:01:05.669 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T14:01:05.669 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T14:01:05.697 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T14:01:05.697 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T14:01:06.324 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T14:01:06.323550+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T14:01:06.324 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T14:01:06.324 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2860 8 365281 365273 0.997717 0 - 0.0299513 2026-02-23T14:01:06.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:01:06 trial068 ceph-mon[38805]: pgmap v1879: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:01:06.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:01:06 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1542174759' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T14:01:07.324 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2861 8 365281 365273 0.997368 0 - 0.0299513 2026-02-23T14:01:07.699 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T14:01:07.816 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:01:07.914 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:01:07.912+0000 7f6171b26640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:01:07.914 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:01:07.912+0000 7f6171b26640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:01:07.926 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T14:01:07.926 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T14:01:07.936 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T14:01:07.936 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T14:01:07.937 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T14:01:07.946 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T14:01:07.946 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T14:01:07.977 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T14:01:07.978 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T14:01:08.324 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2862 8 365281 365273 0.99702 0 - 0.0299513 2026-02-23T14:01:08.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:01:08 trial068 ceph-mon[38805]: pgmap v1880: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:01:08.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:01:08 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1997731346' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T14:01:09.324 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2863 8 365281 365273 0.996671 0 - 0.0299513 2026-02-23T14:01:09.979 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T14:01:10.099 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:01:10.202 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:01:10.200+0000 7eff55907640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:01:10.202 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:01:10.200+0000 7eff55907640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:01:10.214 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T14:01:10.215 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T14:01:10.225 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T14:01:10.225 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T14:01:10.225 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T14:01:10.235 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T14:01:10.235 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T14:01:10.265 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T14:01:10.266 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T14:01:10.324 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2864 8 365281 365273 0.996323 0 - 0.0299513 2026-02-23T14:01:10.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:01:10 trial068 ceph-mon[38805]: pgmap v1881: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:01:10.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:01:10 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1977164727' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T14:01:11.324 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2865 8 365281 365273 0.995976 0 - 0.0299513 2026-02-23T14:01:12.267 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T14:01:12.324 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2866 8 365281 365273 0.995628 0 - 0.0299513 2026-02-23T14:01:12.385 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:01:12.487 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:01:12.485+0000 7f4159d27640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:01:12.487 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:01:12.485+0000 7f4159d27640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:01:12.499 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T14:01:12.499 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T14:01:12.509 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T14:01:12.509 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T14:01:12.510 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T14:01:12.519 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T14:01:12.519 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T14:01:12.549 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T14:01:12.550 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T14:01:12.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:01:12 trial068 ceph-mon[38805]: pgmap v1882: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:01:12.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:01:12 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/265393988' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T14:01:13.324 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2867 8 365281 365273 0.995281 0 - 0.0299513 2026-02-23T14:01:14.324 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2868 8 365281 365273 0.994934 0 - 0.0299513 2026-02-23T14:01:14.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:01:13 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:01:13] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:01:14.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:01:13 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:01:13.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:01:14.551 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T14:01:14.669 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:01:14.766 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:01:14.764+0000 7eff76d0f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:01:14.767 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:01:14.764+0000 7eff76d0f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:01:14.778 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T14:01:14.779 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T14:01:14.789 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T14:01:14.789 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T14:01:14.789 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T14:01:14.799 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T14:01:14.799 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T14:01:14.831 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:01:14 trial068 ceph-mon[38805]: pgmap v1883: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:01:14.831 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:01:14 trial068 ceph-mon[38805]: 5.5 scrub starts 2026-02-23T14:01:14.831 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:01:14 trial068 ceph-mon[38805]: 5.5 scrub ok 2026-02-23T14:01:14.832 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T14:01:14.832 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T14:01:15.325 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2869 8 365281 365273 0.994587 0 - 0.0299513 2026-02-23T14:01:15.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:01:15 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T14:01:15.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:01:15 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T14:01:15.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:01:15 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1033045039' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T14:01:16.325 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2870 8 365281 365273 0.99424 0 - 0.0299513 2026-02-23T14:01:16.833 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T14:01:16.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:01:16 trial068 ceph-mon[38805]: pgmap v1884: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:01:16.952 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:01:17.050 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:01:17.048+0000 7f655715d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:01:17.050 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:01:17.048+0000 7f655715d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:01:17.062 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T14:01:17.062 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T14:01:17.072 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T14:01:17.073 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T14:01:17.073 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T14:01:17.083 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T14:01:17.084 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T14:01:17.115 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T14:01:17.115 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T14:01:17.325 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2871 8 365281 365273 0.993894 0 - 0.0299513 2026-02-23T14:01:17.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:01:17 trial068 ceph-mon[38805]: osdmap e216: 4 total, 3 up, 4 in 2026-02-23T14:01:17.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:01:17 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3198129347' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T14:01:18.325 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2872 8 365281 365273 0.993548 0 - 0.0299513 2026-02-23T14:01:19.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:01:18 trial068 ceph-mon[38805]: pgmap v1886: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:01:19.116 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T14:01:19.233 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:01:19.325 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2873 8 365281 365273 0.993202 0 - 0.0299513 2026-02-23T14:01:19.332 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:01:19.330+0000 7fee32f76640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:01:19.332 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:01:19.330+0000 7fee32f76640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:01:19.344 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T14:01:19.344 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T14:01:19.354 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T14:01:19.355 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T14:01:19.355 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T14:01:19.365 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T14:01:19.365 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T14:01:19.394 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T14:01:19.394 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T14:01:19.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:01:19 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/744863686' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T14:01:20.325 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2874 8 365281 365273 0.992857 0 - 0.0299513 2026-02-23T14:01:21.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:01:20 trial068 ceph-mon[38805]: pgmap v1887: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:01:21.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:01:20 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T14:01:21.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:01:20 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T14:01:21.325 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2875 8 365281 365273 0.992511 0 - 0.0299513 2026-02-23T14:01:21.395 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T14:01:21.516 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:01:21.618 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:01:21.616+0000 7f7fce20a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:01:21.618 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:01:21.616+0000 7f7fce20a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:01:21.630 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T14:01:21.631 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T14:01:21.640 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T14:01:21.641 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T14:01:21.641 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T14:01:21.651 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T14:01:21.651 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T14:01:21.679 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T14:01:21.680 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T14:01:22.325 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2876 8 365281 365273 0.992166 0 - 0.0299513 2026-02-23T14:01:23.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:01:22 trial068 ceph-mon[38805]: pgmap v1888: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:01:23.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:01:22 trial068 ceph-mon[38805]: 5.a scrub starts 2026-02-23T14:01:23.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:01:22 trial068 ceph-mon[38805]: 5.a scrub ok 2026-02-23T14:01:23.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:01:22 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/946474075' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T14:01:23.325 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2877 8 365281 365273 0.991821 0 - 0.0299513 2026-02-23T14:01:23.681 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T14:01:23.798 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:01:23.895 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:01:23.893+0000 7f7ed6340640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:01:23.895 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:01:23.893+0000 7f7ed6340640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:01:23.908 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T14:01:23.908 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T14:01:23.918 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T14:01:23.919 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T14:01:23.919 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T14:01:23.930 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T14:01:23.930 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T14:01:23.965 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T14:01:23.966 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T14:01:24.325 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2878 8 365281 365273 0.991477 0 - 0.0299513 2026-02-23T14:01:24.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:01:23 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:01:23] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:01:24.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:01:23 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:01:23.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:01:25.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:01:24 trial068 ceph-mon[38805]: pgmap v1889: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:01:25.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:01:24 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1564714305' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T14:01:25.325 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2879 8 365281 365273 0.991132 0 - 0.0299513 2026-02-23T14:01:25.967 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T14:01:26.084 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:01:26.182 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:01:26.180+0000 7faf56d87640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:01:26.183 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:01:26.181+0000 7faf56d87640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:01:26.194 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T14:01:26.194 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T14:01:26.204 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T14:01:26.205 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T14:01:26.205 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T14:01:26.214 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T14:01:26.214 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T14:01:26.245 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T14:01:26.245 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T14:01:26.325 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T14:01:26.325081+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T14:01:26.326 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T14:01:26.326 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2880 8 365281 365273 0.990788 0 - 0.0299513 2026-02-23T14:01:27.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:01:26 trial068 ceph-mon[38805]: pgmap v1890: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:01:27.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:01:26 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2250282112' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T14:01:27.325 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2881 8 365281 365273 0.990444 0 - 0.0299513 2026-02-23T14:01:28.246 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T14:01:28.325 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2882 8 365281 365273 0.990101 0 - 0.0299513 2026-02-23T14:01:28.365 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:01:28.464 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:01:28.462+0000 7f1eb126e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:01:28.464 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:01:28.462+0000 7f1eb126e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:01:28.476 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T14:01:28.476 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T14:01:28.486 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T14:01:28.486 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T14:01:28.487 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T14:01:28.496 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T14:01:28.497 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T14:01:28.527 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T14:01:28.527 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T14:01:28.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:01:28 trial068 ceph-mon[38805]: pgmap v1891: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:01:28.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:01:28 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/4173518926' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T14:01:29.326 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2883 8 365281 365273 0.989757 0 - 0.0299513 2026-02-23T14:01:30.326 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2884 8 365281 365273 0.989414 0 - 0.0299513 2026-02-23T14:01:30.528 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T14:01:30.648 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:01:30.751 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:01:30.749+0000 7f7242192640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:01:30.751 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:01:30.749+0000 7f7242192640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:01:30.763 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T14:01:30.763 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T14:01:30.773 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T14:01:30.773 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T14:01:30.774 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T14:01:30.783 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T14:01:30.783 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T14:01:30.816 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:01:30 trial068 ceph-mon[38805]: pgmap v1892: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:01:30.817 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T14:01:30.818 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T14:01:31.326 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2885 8 365281 365273 0.989071 0 - 0.0299513 2026-02-23T14:01:32.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:01:31 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/4067234536' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T14:01:32.326 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2886 8 365281 365273 0.988728 0 - 0.0299513 2026-02-23T14:01:32.819 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T14:01:32.936 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:01:32.959 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:01:32 trial068 ceph-mon[38805]: pgmap v1893: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:01:33.030 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:01:33.028+0000 7f5fbbfff640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:01:33.030 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:01:33.028+0000 7f5fbbfff640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:01:33.042 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T14:01:33.042 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T14:01:33.052 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T14:01:33.052 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T14:01:33.053 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T14:01:33.062 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T14:01:33.063 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T14:01:33.093 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T14:01:33.094 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T14:01:33.326 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2887 8 365281 365273 0.988386 0 - 0.0299513 2026-02-23T14:01:33.974 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:01:33 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3448673483' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T14:01:34.266 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:01:33 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:01:33.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:01:34.326 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2888 8 365281 365273 0.988044 0 - 0.0299513 2026-02-23T14:01:34.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:01:33 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:01:33] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:01:35.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:01:34 trial068 ceph-mon[38805]: pgmap v1894: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:01:35.095 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T14:01:35.211 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:01:35.310 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:01:35.308+0000 7f0d3614d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:01:35.310 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:01:35.308+0000 7f0d3614d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:01:35.321 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T14:01:35.322 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T14:01:35.326 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2889 8 365281 365273 0.987702 0 - 0.0299513 2026-02-23T14:01:35.331 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T14:01:35.332 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T14:01:35.332 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T14:01:35.341 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T14:01:35.342 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T14:01:35.371 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T14:01:35.372 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T14:01:35.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:01:35 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2532648623' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T14:01:36.326 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2890 8 365281 365273 0.98736 0 - 0.0299513 2026-02-23T14:01:37.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:01:36 trial068 ceph-mon[38805]: pgmap v1895: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:01:37.326 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2891 8 365281 365273 0.987018 0 - 0.0299513 2026-02-23T14:01:37.373 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T14:01:37.490 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:01:37.589 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:01:37.587+0000 7f9e1dbf7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:01:37.589 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:01:37.587+0000 7f9e1dbf7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:01:37.601 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T14:01:37.601 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T14:01:37.611 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T14:01:37.612 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T14:01:37.612 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T14:01:37.622 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T14:01:37.622 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T14:01:37.651 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:01:37 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1927221049' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T14:01:37.652 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T14:01:37.653 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T14:01:38.326 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2892 8 365281 365273 0.986677 0 - 0.0299513 2026-02-23T14:01:39.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:01:38 trial068 ceph-mon[38805]: pgmap v1896: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:01:39.326 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2893 8 365281 365273 0.986336 0 - 0.0299513 2026-02-23T14:01:39.654 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T14:01:39.772 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:01:39.865 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:01:39.863+0000 7fb16aec8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:01:39.866 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:01:39.864+0000 7fb16aec8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:01:39.877 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T14:01:39.878 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T14:01:39.888 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T14:01:39.889 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T14:01:39.889 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T14:01:39.898 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T14:01:39.898 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T14:01:39.929 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T14:01:39.930 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T14:01:40.326 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2894 8 365281 365273 0.985995 0 - 0.0299513 2026-02-23T14:01:41.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:01:40 trial068 ceph-mon[38805]: pgmap v1897: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:01:41.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:01:40 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2102030138' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T14:01:41.326 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2895 8 365281 365273 0.985655 0 - 0.0299513 2026-02-23T14:01:41.931 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T14:01:42.047 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:01:42.154 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:01:42.152+0000 7f4a940a6640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:01:42.155 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:01:42.153+0000 7f4a940a6640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:01:42.166 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T14:01:42.167 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T14:01:42.176 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T14:01:42.177 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T14:01:42.177 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T14:01:42.187 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T14:01:42.187 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T14:01:42.217 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T14:01:42.217 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T14:01:42.327 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2896 8 365281 365273 0.985314 0 - 0.0299513 2026-02-23T14:01:43.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:01:42 trial068 ceph-mon[38805]: pgmap v1898: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:01:43.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:01:42 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1401360007' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T14:01:43.327 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2897 8 365281 365273 0.984974 0 - 0.0299513 2026-02-23T14:01:44.218 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T14:01:44.242 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:01:43 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:01:43] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:01:44.327 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2898 8 365281 365273 0.984634 0 - 0.0299513 2026-02-23T14:01:44.335 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:01:44.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:01:43 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:01:43.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:01:44.431 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:01:44.429+0000 7f86fc4dd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:01:44.431 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:01:44.429+0000 7f86fc4dd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:01:44.443 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T14:01:44.443 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T14:01:44.453 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T14:01:44.454 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T14:01:44.454 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T14:01:44.464 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T14:01:44.465 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T14:01:44.494 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T14:01:44.494 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T14:01:44.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:01:44 trial068 ceph-mon[38805]: pgmap v1899: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:01:44.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:01:44 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/4198420797' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T14:01:45.327 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2899 8 365281 365273 0.984295 0 - 0.0299513 2026-02-23T14:01:46.327 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T14:01:46.326550+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T14:01:46.327 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T14:01:46.327 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2900 8 365281 365273 0.983955 0 - 0.0299513 2026-02-23T14:01:46.495 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T14:01:46.615 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:01:46.711 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:01:46.709+0000 7f741e097640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:01:46.711 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:01:46.709+0000 7f741e097640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:01:46.724 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T14:01:46.724 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T14:01:46.734 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T14:01:46.735 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T14:01:46.735 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T14:01:46.745 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T14:01:46.746 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T14:01:46.775 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:01:46 trial068 ceph-mon[38805]: pgmap v1900: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:01:46.776 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T14:01:46.777 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T14:01:47.327 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2901 8 365281 365273 0.983616 0 - 0.0299513 2026-02-23T14:01:48.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:01:47 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3590760864' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T14:01:48.327 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2902 8 365281 365273 0.983277 0 - 0.0299513 2026-02-23T14:01:48.777 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T14:01:48.897 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:01:48.921 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:01:48 trial068 ceph-mon[38805]: pgmap v1901: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:01:48.999 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:01:48.997+0000 7f5b81d68640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:01:48.999 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:01:48.997+0000 7f5b81d68640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:01:49.011 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T14:01:49.011 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T14:01:49.021 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T14:01:49.022 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T14:01:49.023 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T14:01:49.033 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T14:01:49.033 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T14:01:49.064 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T14:01:49.065 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T14:01:49.327 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2903 8 365281 365273 0.982938 0 - 0.0299513 2026-02-23T14:01:50.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:01:49 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2026-02-23T14:01:50.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:01:49 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3117520367' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T14:01:50.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:01:49 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-23T14:01:50.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:01:49 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2026-02-23T14:01:50.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:01:49 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T14:01:50.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:01:49 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2026-02-23T14:01:50.327 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2904 8 365281 365273 0.9826 0 - 0.0299513 2026-02-23T14:01:51.065 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T14:01:51.090 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:01:50 trial068 ceph-mon[38805]: pgmap v1902: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:01:51.090 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:01:50 trial068 ceph-mon[38805]: pgmap v1903: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:01:51.090 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:01:50 trial068 ceph-mon[38805]: pgmap v1904: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:01:51.191 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:01:51.295 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:01:51.292+0000 7f351988f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:01:51.295 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:01:51.293+0000 7f351988f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:01:51.306 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T14:01:51.307 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T14:01:51.316 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T14:01:51.317 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T14:01:51.317 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T14:01:51.327 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2905 8 365281 365273 0.982262 0 - 0.0299513 2026-02-23T14:01:51.328 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T14:01:51.328 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T14:01:51.357 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T14:01:51.358 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T14:01:52.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:01:51 trial068 ceph-mon[38805]: 5.7 scrub starts 2026-02-23T14:01:52.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:01:51 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1414563851' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T14:01:52.327 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2906 8 365281 365273 0.981924 0 - 0.0299513 2026-02-23T14:01:53.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:01:52 trial068 ceph-mon[38805]: pgmap v1905: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:01:53.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:01:52 trial068 ceph-mon[38805]: 5.7 scrub ok 2026-02-23T14:01:53.327 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2907 8 365281 365273 0.981586 0 - 0.0299513 2026-02-23T14:01:53.359 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T14:01:53.475 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:01:53.573 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:01:53.571+0000 7f3820316640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:01:53.573 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:01:53.571+0000 7f3820316640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:01:53.585 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T14:01:53.585 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T14:01:53.595 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T14:01:53.595 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T14:01:53.595 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T14:01:53.605 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T14:01:53.606 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T14:01:53.637 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T14:01:53.637 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T14:01:53.975 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:01:53 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/592546838' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T14:01:54.327 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2908 8 365281 365273 0.981248 0 - 0.0299513 2026-02-23T14:01:54.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:01:53 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:01:53] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:01:54.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:01:53 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:01:53.974Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:01:55.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:01:54 trial068 ceph-mon[38805]: pgmap v1906: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:01:55.328 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2909 8 365281 365273 0.980911 0 - 0.0299513 2026-02-23T14:01:55.638 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T14:01:55.757 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:01:55.854 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:01:55.852+0000 7f1c5e31b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:01:55.854 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:01:55.852+0000 7f1c5e31b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:01:55.866 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T14:01:55.867 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T14:01:55.877 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T14:01:55.878 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T14:01:55.878 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T14:01:55.887 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T14:01:55.888 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T14:01:55.917 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T14:01:55.917 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T14:01:56.328 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2910 8 365281 365273 0.980574 0 - 0.0299513 2026-02-23T14:01:57.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:01:56 trial068 ceph-mon[38805]: pgmap v1907: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:01:57.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:01:56 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1612825489' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T14:01:57.328 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2911 8 365281 365273 0.980237 0 - 0.0299513 2026-02-23T14:01:57.918 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T14:01:58.036 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:01:58.131 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:01:58.129+0000 7fad3b524640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:01:58.131 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:01:58.129+0000 7fad3b524640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:01:58.143 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T14:01:58.144 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T14:01:58.154 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T14:01:58.155 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T14:01:58.155 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T14:01:58.164 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T14:01:58.165 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T14:01:58.195 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T14:01:58.196 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T14:01:58.328 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2912 8 365281 365273 0.979901 0 - 0.0299513 2026-02-23T14:01:59.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:01:58 trial068 ceph-mon[38805]: pgmap v1908: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:01:59.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:01:58 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3598128564' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T14:01:59.328 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2913 8 365281 365273 0.979564 0 - 0.0299513 2026-02-23T14:02:00.197 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T14:02:00.315 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:02:00.328 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2914 8 365281 365273 0.979228 0 - 0.0299513 2026-02-23T14:02:00.412 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:02:00.410+0000 7fe08d7db640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:02:00.412 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:02:00.410+0000 7fe08d7db640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:02:00.424 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T14:02:00.424 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T14:02:00.434 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T14:02:00.434 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T14:02:00.434 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T14:02:00.445 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T14:02:00.445 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T14:02:00.477 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T14:02:00.478 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T14:02:00.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:02:00 trial068 ceph-mon[38805]: pgmap v1909: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:02:00.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:02:00 trial068 ceph-mon[38805]: 5.8 scrub starts 2026-02-23T14:02:00.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:02:00 trial068 ceph-mon[38805]: 5.8 scrub ok 2026-02-23T14:02:00.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:02:00 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1789531813' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T14:02:01.328 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2915 8 365281 365273 0.978892 0 - 0.0299513 2026-02-23T14:02:02.328 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2916 8 365281 365273 0.978556 0 - 0.0299513 2026-02-23T14:02:02.479 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T14:02:02.598 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:02:02.701 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:02:02.699+0000 7fde66097640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:02:02.701 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:02:02.699+0000 7fde66097640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:02:02.713 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T14:02:02.714 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T14:02:02.723 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T14:02:02.724 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T14:02:02.724 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T14:02:02.733 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T14:02:02.734 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T14:02:02.761 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:02:02 trial068 ceph-mon[38805]: pgmap v1910: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:02:02.761 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:02:02 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T14:02:02.761 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:02:02 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T14:02:02.762 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T14:02:02.762 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T14:02:03.328 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2917 8 365281 365273 0.978221 0 - 0.0299513 2026-02-23T14:02:03.974 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:02:03 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2532675827' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T14:02:04.328 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2918 8 365281 365273 0.977886 0 - 0.0299513 2026-02-23T14:02:04.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:02:03 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:02:03] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:02:04.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:02:03 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:02:03.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:02:04.763 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T14:02:04.881 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:02:04.902 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:02:04 trial068 ceph-mon[38805]: pgmap v1911: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:02:04.981 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:02:04.979+0000 7fc45cc3a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:02:04.982 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:02:04.980+0000 7fc45cc3a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:02:04.994 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T14:02:04.994 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T14:02:05.004 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T14:02:05.005 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T14:02:05.005 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T14:02:05.015 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T14:02:05.015 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T14:02:05.047 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T14:02:05.048 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T14:02:05.328 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2919 8 365281 365273 0.977551 0 - 0.0299513 2026-02-23T14:02:06.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:02:05 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1480225939' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T14:02:06.328 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T14:02:06.328141+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T14:02:06.329 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T14:02:06.329 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2920 8 365281 365273 0.977216 0 - 0.0299513 2026-02-23T14:02:07.049 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T14:02:07.073 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:02:06 trial068 ceph-mon[38805]: pgmap v1912: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:02:07.168 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:02:07.274 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:02:07.272+0000 7f4bedf63640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:02:07.274 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:02:07.272+0000 7f4bedf63640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:02:07.286 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T14:02:07.286 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T14:02:07.296 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T14:02:07.296 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T14:02:07.297 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T14:02:07.306 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T14:02:07.306 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T14:02:07.329 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2921 8 365281 365273 0.976881 0 - 0.0299513 2026-02-23T14:02:07.335 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T14:02:07.336 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T14:02:08.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:02:07 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3398763798' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T14:02:08.329 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2922 8 365281 365273 0.976547 0 - 0.0299513 2026-02-23T14:02:09.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:02:08 trial068 ceph-mon[38805]: pgmap v1913: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:02:09.329 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2923 8 365281 365273 0.976213 0 - 0.0299513 2026-02-23T14:02:09.337 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T14:02:09.456 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:02:09.558 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:02:09.556+0000 7f56b55cf640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:02:09.558 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:02:09.556+0000 7f56b55cf640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:02:09.570 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T14:02:09.570 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T14:02:09.580 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T14:02:09.580 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T14:02:09.580 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T14:02:09.590 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T14:02:09.590 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T14:02:09.620 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T14:02:09.621 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T14:02:10.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:02:09 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1120694208' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T14:02:10.329 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2924 8 365281 365273 0.975879 0 - 0.0299513 2026-02-23T14:02:11.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:02:10 trial068 ceph-mon[38805]: pgmap v1914: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:02:11.329 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2925 8 365281 365273 0.975545 0 - 0.0299513 2026-02-23T14:02:11.622 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T14:02:11.739 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:02:11.838 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:02:11.835+0000 7fe75d9d7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:02:11.838 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:02:11.836+0000 7fe75d9d7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:02:11.849 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T14:02:11.850 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T14:02:11.859 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T14:02:11.860 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T14:02:11.860 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T14:02:11.870 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T14:02:11.871 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T14:02:11.901 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T14:02:11.901 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T14:02:12.329 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2926 8 365281 365273 0.975212 0 - 0.0299513 2026-02-23T14:02:13.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:02:12 trial068 ceph-mon[38805]: pgmap v1915: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:02:13.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:02:12 trial068 ceph-mon[38805]: 5.e scrub starts 2026-02-23T14:02:13.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:02:12 trial068 ceph-mon[38805]: 5.e scrub ok 2026-02-23T14:02:13.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:02:12 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/261403474' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T14:02:13.329 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2927 8 365281 365273 0.974879 0 - 0.0299513 2026-02-23T14:02:13.902 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T14:02:14.019 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:02:14.120 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:02:14.118+0000 7f83d6b5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:02:14.121 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:02:14.118+0000 7f83d6b5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:02:14.133 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T14:02:14.134 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T14:02:14.143 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T14:02:14.144 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T14:02:14.144 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T14:02:14.154 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T14:02:14.154 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T14:02:14.183 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:02:13 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:02:13] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:02:14.185 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T14:02:14.186 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T14:02:14.329 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2928 8 365281 365273 0.974546 0 - 0.0299513 2026-02-23T14:02:14.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:02:13 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:02:13.972Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:02:15.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:02:14 trial068 ceph-mon[38805]: pgmap v1916: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:02:15.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:02:14 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3912413179' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T14:02:15.329 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2929 8 365281 365273 0.974213 0 - 0.0299513 2026-02-23T14:02:16.186 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T14:02:16.305 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:02:16.329 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2930 8 365281 365273 0.973881 0 - 0.0299513 2026-02-23T14:02:16.410 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:02:16.408+0000 7ff421149640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:02:16.410 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:02:16.408+0000 7ff421149640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:02:16.422 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T14:02:16.422 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T14:02:16.432 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T14:02:16.432 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T14:02:16.433 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T14:02:16.443 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T14:02:16.443 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T14:02:16.473 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T14:02:16.473 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T14:02:16.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:02:16 trial068 ceph-mon[38805]: pgmap v1917: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:02:16.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:02:16 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1072929745' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T14:02:17.329 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2931 8 365281 365273 0.973548 0 - 0.0299513 2026-02-23T14:02:18.329 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2932 8 365281 365273 0.973216 0 - 0.0299513 2026-02-23T14:02:18.475 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T14:02:18.591 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:02:18.692 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:02:18.689+0000 7f0b07ed0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:02:18.692 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:02:18.690+0000 7f0b07ed0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:02:18.703 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T14:02:18.703 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T14:02:18.713 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T14:02:18.714 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T14:02:18.714 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T14:02:18.724 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T14:02:18.724 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T14:02:18.752 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:02:18 trial068 ceph-mon[38805]: pgmap v1918: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:02:18.752 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T14:02:18.753 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T14:02:19.329 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2933 8 365281 365273 0.972885 0 - 0.0299513 2026-02-23T14:02:20.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:02:19 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2190117656' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T14:02:20.330 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2934 8 365281 365273 0.972553 0 - 0.0299513 2026-02-23T14:02:20.754 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T14:02:20.871 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:02:20.914 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:02:20 trial068 ceph-mon[38805]: pgmap v1919: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:02:20.970 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:02:20.968+0000 7f6d4d78e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:02:20.970 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:02:20.968+0000 7f6d4d78e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:02:20.982 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T14:02:20.982 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T14:02:20.992 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T14:02:20.993 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T14:02:20.993 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T14:02:21.002 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T14:02:21.003 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T14:02:21.033 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T14:02:21.034 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T14:02:21.330 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2935 8 365281 365273 0.972222 0 - 0.0299513 2026-02-23T14:02:22.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:02:21 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1203389707' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T14:02:22.330 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2936 8 365281 365273 0.97189 0 - 0.0299513 2026-02-23T14:02:23.035 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T14:02:23.061 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:02:22 trial068 ceph-mon[38805]: pgmap v1920: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:02:23.153 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:02:23.261 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:02:23.259+0000 7f7340ad5640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:02:23.261 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:02:23.259+0000 7f7340ad5640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:02:23.273 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T14:02:23.273 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T14:02:23.283 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T14:02:23.284 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T14:02:23.284 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T14:02:23.294 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T14:02:23.294 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T14:02:23.324 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T14:02:23.325 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T14:02:23.330 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2937 8 365281 365273 0.97156 0 - 0.0299513 2026-02-23T14:02:23.975 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:02:23 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3652705300' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T14:02:24.330 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2938 8 365281 365273 0.971229 0 - 0.0299513 2026-02-23T14:02:24.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:02:23 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:02:23] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:02:24.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:02:23 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:02:23.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:02:25.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:02:24 trial068 ceph-mon[38805]: pgmap v1921: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:02:25.326 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T14:02:25.330 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2939 8 365281 365273 0.970898 0 - 0.0299513 2026-02-23T14:02:25.448 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:02:25.552 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:02:25.550+0000 7f9f97577640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:02:25.552 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:02:25.550+0000 7f9f97577640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:02:25.564 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T14:02:25.564 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T14:02:25.573 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T14:02:25.574 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T14:02:25.574 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T14:02:25.584 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T14:02:25.585 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T14:02:25.616 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T14:02:25.616 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T14:02:26.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:02:25 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2651773362' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T14:02:26.330 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T14:02:26.329660+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T14:02:26.330 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T14:02:26.331 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2940 8 365281 365273 0.970568 0 - 0.0299513 2026-02-23T14:02:27.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:02:26 trial068 ceph-mon[38805]: pgmap v1922: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:02:27.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:02:26 trial068 ceph-mon[38805]: 5.a scrub starts 2026-02-23T14:02:27.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:02:26 trial068 ceph-mon[38805]: 5.a scrub ok 2026-02-23T14:02:27.330 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2941 8 365281 365273 0.970238 0 - 0.0299513 2026-02-23T14:02:27.618 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T14:02:27.737 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:02:27.836 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:02:27.833+0000 7f5820532640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:02:27.836 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:02:27.834+0000 7f5820532640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:02:27.847 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T14:02:27.848 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T14:02:27.858 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T14:02:27.858 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T14:02:27.858 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T14:02:27.868 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T14:02:27.868 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T14:02:27.898 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T14:02:27.898 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T14:02:28.330 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2942 8 365281 365273 0.969908 0 - 0.0299513 2026-02-23T14:02:29.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:02:28 trial068 ceph-mon[38805]: pgmap v1923: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:02:29.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:02:28 trial068 ceph-mon[38805]: 5.5 scrub starts 2026-02-23T14:02:29.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:02:28 trial068 ceph-mon[38805]: 5.5 scrub ok 2026-02-23T14:02:29.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:02:28 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/8984150' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T14:02:29.330 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2943 8 365281 365273 0.969579 0 - 0.0299513 2026-02-23T14:02:29.899 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T14:02:30.016 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:02:30.113 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:02:30.111+0000 7ff90c8ba640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:02:30.113 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:02:30.111+0000 7ff90c8ba640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:02:30.125 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T14:02:30.125 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T14:02:30.135 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T14:02:30.136 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T14:02:30.136 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T14:02:30.145 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T14:02:30.145 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T14:02:30.175 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T14:02:30.175 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T14:02:30.330 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2944 8 365281 365273 0.969249 0 - 0.0299513 2026-02-23T14:02:31.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:02:30 trial068 ceph-mon[38805]: pgmap v1924: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:02:31.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:02:30 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1425213792' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T14:02:31.330 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2945 8 365281 365273 0.96892 0 - 0.0299513 2026-02-23T14:02:32.176 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T14:02:32.292 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:02:32.330 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2946 8 365281 365273 0.968591 0 - 0.0299513 2026-02-23T14:02:32.386 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:02:32.384+0000 7ff682b51640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:02:32.386 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:02:32.384+0000 7ff682b51640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:02:32.398 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T14:02:32.398 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T14:02:32.407 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T14:02:32.408 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T14:02:32.408 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T14:02:32.418 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T14:02:32.418 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T14:02:32.448 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T14:02:32.448 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T14:02:33.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:02:32 trial068 ceph-mon[38805]: pgmap v1925: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:02:33.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:02:32 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/486776850' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T14:02:33.331 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2947 8 365281 365273 0.968263 0 - 0.0299513 2026-02-23T14:02:34.266 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:02:33 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:02:33.974Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:02:34.331 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2948 8 365281 365273 0.967934 0 - 0.0299513 2026-02-23T14:02:34.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:02:33 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:02:33] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:02:34.450 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T14:02:34.566 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:02:34.662 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:02:34.660+0000 7f78fbe50640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:02:34.663 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:02:34.660+0000 7f78fbe50640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:02:34.675 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T14:02:34.675 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T14:02:34.685 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T14:02:34.685 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T14:02:34.686 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T14:02:34.695 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T14:02:34.695 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T14:02:34.725 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:02:34 trial068 ceph-mon[38805]: pgmap v1926: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:02:34.725 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:02:34 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3793515398' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T14:02:34.726 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T14:02:34.726 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T14:02:35.331 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2949 8 365281 365273 0.967606 0 - 0.0299513 2026-02-23T14:02:36.331 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2950 8 365281 365273 0.967278 0 - 0.0299513 2026-02-23T14:02:36.728 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T14:02:36.844 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:02:36.942 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:02:36.940+0000 7f0d77f3c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:02:36.942 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:02:36.940+0000 7f0d77f3c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:02:36.942 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:02:36 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T14:02:36.943 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:02:36 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T14:02:36.943 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:02:36 trial068 ceph-mon[38805]: pgmap v1927: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:02:36.954 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T14:02:36.954 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T14:02:36.964 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T14:02:36.964 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T14:02:36.964 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T14:02:36.974 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T14:02:36.974 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T14:02:37.005 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T14:02:37.006 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T14:02:37.331 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2951 8 365281 365273 0.96695 0 - 0.0299513 2026-02-23T14:02:38.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:02:37 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/195528220' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T14:02:38.331 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2952 8 365281 365273 0.966623 0 - 0.0299513 2026-02-23T14:02:39.007 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T14:02:39.033 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:02:38 trial068 ceph-mon[38805]: pgmap v1928: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:02:39.126 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:02:39.224 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:02:39.222+0000 7f87695e5640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:02:39.224 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:02:39.222+0000 7f87695e5640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:02:39.236 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T14:02:39.237 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T14:02:39.247 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T14:02:39.248 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T14:02:39.248 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T14:02:39.256 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T14:02:39.257 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T14:02:39.287 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T14:02:39.287 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T14:02:39.331 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2953 8 365281 365273 0.966295 0 - 0.0299513 2026-02-23T14:02:40.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:02:39 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2876358556' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T14:02:40.331 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2954 8 365281 365273 0.965968 0 - 0.0299513 2026-02-23T14:02:41.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:02:40 trial068 ceph-mon[38805]: pgmap v1929: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:02:41.289 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T14:02:41.331 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2955 8 365281 365273 0.965641 0 - 0.0299513 2026-02-23T14:02:41.407 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:02:41.505 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:02:41.503+0000 7fbc0598a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:02:41.506 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:02:41.503+0000 7fbc0598a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:02:41.517 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T14:02:41.517 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T14:02:41.527 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T14:02:41.527 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T14:02:41.527 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T14:02:41.537 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T14:02:41.537 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T14:02:41.567 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T14:02:41.568 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T14:02:41.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:02:41 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T14:02:41.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:02:41 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T14:02:41.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:02:41 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/535305778' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T14:02:42.331 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2956 8 365281 365273 0.965315 0 - 0.0299513 2026-02-23T14:02:43.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:02:42 trial068 ceph-mon[38805]: pgmap v1930: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:02:43.331 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2957 8 365281 365273 0.964988 0 - 0.0299513 2026-02-23T14:02:43.569 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T14:02:43.687 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:02:43.786 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:02:43.784+0000 7f810fec2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:02:43.786 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:02:43.784+0000 7f810fec2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:02:43.798 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T14:02:43.798 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T14:02:43.808 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T14:02:43.808 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T14:02:43.808 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T14:02:43.818 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T14:02:43.818 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T14:02:43.849 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T14:02:43.849 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T14:02:44.331 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2958 8 365281 365273 0.964662 0 - 0.0299513 2026-02-23T14:02:44.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:02:43 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:02:43] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:02:44.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:02:43 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:02:43.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:02:45.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:02:44 trial068 ceph-mon[38805]: pgmap v1931: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:02:45.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:02:44 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/345949278' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T14:02:45.331 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2959 8 365281 365273 0.964336 0 - 0.0299513 2026-02-23T14:02:45.850 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T14:02:45.969 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:02:46.066 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:02:46.063+0000 7f93d19af640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:02:46.066 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:02:46.064+0000 7f93d19af640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:02:46.077 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T14:02:46.077 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T14:02:46.087 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T14:02:46.088 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T14:02:46.088 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T14:02:46.097 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T14:02:46.098 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T14:02:46.128 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T14:02:46.128 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T14:02:46.331 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T14:02:46.331147+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T14:02:46.332 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T14:02:46.332 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2960 8 365281 365273 0.96401 0 - 0.0299513 2026-02-23T14:02:47.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:02:46 trial068 ceph-mon[38805]: pgmap v1932: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:02:47.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:02:46 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1383104826' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T14:02:47.332 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2961 8 365281 365273 0.963685 0 - 0.0299513 2026-02-23T14:02:48.129 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T14:02:48.247 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:02:48.332 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2962 8 365281 365273 0.963359 0 - 0.0299513 2026-02-23T14:02:48.343 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:02:48.341+0000 7f1f3be5c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:02:48.343 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:02:48.341+0000 7f1f3be5c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:02:48.354 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T14:02:48.355 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T14:02:48.364 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T14:02:48.364 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T14:02:48.364 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T14:02:48.374 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T14:02:48.374 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T14:02:48.401 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T14:02:48.402 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T14:02:49.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:02:48 trial068 ceph-mon[38805]: pgmap v1933: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:02:49.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:02:48 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/983108996' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T14:02:49.332 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2963 8 365281 365273 0.963034 0 - 0.0299513 2026-02-23T14:02:50.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:02:49 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2026-02-23T14:02:50.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:02:49 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-23T14:02:50.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:02:49 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2026-02-23T14:02:50.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:02:49 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T14:02:50.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:02:49 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2026-02-23T14:02:50.332 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2964 8 365281 365273 0.962709 0 - 0.0299513 2026-02-23T14:02:50.403 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T14:02:50.522 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:02:50.623 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:02:50.621+0000 7fbd50eab640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:02:50.624 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:02:50.622+0000 7fbd50eab640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:02:50.636 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T14:02:50.636 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T14:02:50.646 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T14:02:50.646 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T14:02:50.646 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T14:02:50.657 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T14:02:50.657 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T14:02:50.688 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T14:02:50.689 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T14:02:51.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:02:50 trial068 ceph-mon[38805]: pgmap v1934: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:02:51.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:02:50 trial068 ceph-mon[38805]: pgmap v1935: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:02:51.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:02:50 trial068 ceph-mon[38805]: pgmap v1936: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:02:51.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:02:50 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1657249894' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T14:02:51.332 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2965 8 365281 365273 0.962385 0 - 0.0299513 2026-02-23T14:02:52.332 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2966 8 365281 365273 0.96206 0 - 0.0299513 2026-02-23T14:02:52.689 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T14:02:52.808 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:02:52.906 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:02:52.904+0000 7f8134fea640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:02:52.906 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:02:52.904+0000 7f8134fea640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:02:52.917 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T14:02:52.917 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T14:02:52.927 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T14:02:52.928 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T14:02:52.928 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T14:02:52.939 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T14:02:52.939 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T14:02:52.940 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:02:52 trial068 ceph-mon[38805]: pgmap v1937: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:02:52.971 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T14:02:52.971 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T14:02:53.332 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2967 8 365281 365273 0.961736 0 - 0.0299513 2026-02-23T14:02:53.974 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:02:53 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3865298689' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T14:02:54.332 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2968 8 365281 365273 0.961412 0 - 0.0299513 2026-02-23T14:02:54.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:02:53 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:02:53] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:02:54.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:02:53 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:02:53.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:02:54.972 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T14:02:54.997 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:02:54 trial068 ceph-mon[38805]: pgmap v1938: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:02:55.090 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:02:55.195 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:02:55.193+0000 7f0896811640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:02:55.196 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:02:55.194+0000 7f0896811640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:02:55.207 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T14:02:55.208 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T14:02:55.218 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T14:02:55.218 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T14:02:55.218 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T14:02:55.228 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T14:02:55.228 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T14:02:55.260 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T14:02:55.260 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T14:02:55.332 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2969 8 365281 365273 0.961088 0 - 0.0299513 2026-02-23T14:02:56.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:02:55 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2375375031' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T14:02:56.332 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2970 8 365281 365273 0.960764 0 - 0.0299513 2026-02-23T14:02:57.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:02:56 trial068 ceph-mon[38805]: pgmap v1939: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:02:57.261 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T14:02:57.332 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2971 8 365281 365273 0.960441 0 - 0.0299513 2026-02-23T14:02:57.380 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:02:57.476 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:02:57.473+0000 7f9cd5429640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:02:57.476 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:02:57.474+0000 7f9cd5429640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:02:57.487 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T14:02:57.488 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T14:02:57.498 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T14:02:57.498 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T14:02:57.498 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T14:02:57.509 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T14:02:57.509 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T14:02:57.554 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T14:02:57.554 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T14:02:57.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:02:57 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/54273405' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T14:02:58.332 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2972 8 365281 365273 0.960118 0 - 0.0299513 2026-02-23T14:02:59.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:02:58 trial068 ceph-mon[38805]: pgmap v1940: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:02:59.332 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2973 8 365281 365273 0.959795 0 - 0.0299513 2026-02-23T14:02:59.556 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T14:02:59.674 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:02:59.774 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:02:59.772+0000 7ff1907c0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:02:59.774 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:02:59.772+0000 7ff1907c0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:02:59.787 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T14:02:59.787 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T14:02:59.797 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T14:02:59.797 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T14:02:59.797 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T14:02:59.806 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T14:02:59.807 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T14:02:59.837 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T14:02:59.837 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T14:03:00.333 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2974 8 365281 365273 0.959472 0 - 0.0299513 2026-02-23T14:03:01.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:03:00 trial068 ceph-mon[38805]: pgmap v1941: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:03:01.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:03:00 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2234070268' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T14:03:01.333 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2975 8 365281 365273 0.95915 0 - 0.0299513 2026-02-23T14:03:01.838 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T14:03:01.955 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:03:02.055 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:03:02.053+0000 7f1cc7de8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:03:02.055 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:03:02.053+0000 7f1cc7de8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:03:02.067 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T14:03:02.067 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T14:03:02.079 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T14:03:02.079 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T14:03:02.079 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T14:03:02.089 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T14:03:02.089 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T14:03:02.117 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T14:03:02.118 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T14:03:02.333 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2976 8 365281 365273 0.958827 0 - 0.0299513 2026-02-23T14:03:03.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:03:02 trial068 ceph-mon[38805]: pgmap v1942: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:03:03.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:03:02 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2760182937' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T14:03:03.333 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2977 8 365281 365273 0.958505 0 - 0.0299513 2026-02-23T14:03:04.118 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T14:03:04.238 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:03:04.259 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:03:03 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:03:03] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:03:04.265 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:03:03 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:03:03.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:03:04.333 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2978 8 365281 365273 0.958183 0 - 0.0299513 2026-02-23T14:03:04.337 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:03:04.335+0000 7f15e868f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:03:04.337 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:03:04.335+0000 7f15e868f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:03:04.349 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T14:03:04.349 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T14:03:04.359 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T14:03:04.359 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T14:03:04.360 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T14:03:04.369 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T14:03:04.369 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T14:03:04.399 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T14:03:04.399 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T14:03:05.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:03:04 trial068 ceph-mon[38805]: pgmap v1943: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:03:05.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:03:04 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/4041855545' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T14:03:05.333 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2979 8 365281 365273 0.957862 0 - 0.0299513 2026-02-23T14:03:06.333 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T14:03:06.332968+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T14:03:06.334 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T14:03:06.334 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2980 8 365281 365273 0.95754 0 - 0.0299513 2026-02-23T14:03:06.400 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T14:03:06.519 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:03:06.614 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:03:06.611+0000 7f77bbb48640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:03:06.614 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:03:06.612+0000 7f77bbb48640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:03:06.625 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T14:03:06.626 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T14:03:06.635 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T14:03:06.636 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T14:03:06.636 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T14:03:06.647 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T14:03:06.647 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T14:03:06.678 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T14:03:06.678 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T14:03:07.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:03:06 trial068 ceph-mon[38805]: pgmap v1944: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:03:07.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:03:06 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/320544196' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T14:03:07.333 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2981 8 365281 365273 0.957219 0 - 0.0299513 2026-02-23T14:03:08.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:03:07 trial068 ceph-mon[38805]: 5.7 scrub starts 2026-02-23T14:03:08.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:03:07 trial068 ceph-mon[38805]: 5.7 scrub ok 2026-02-23T14:03:08.333 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2982 8 365281 365273 0.956898 0 - 0.0299513 2026-02-23T14:03:08.679 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T14:03:08.796 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:03:08.892 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:03:08.890+0000 7f4e775cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:03:08.893 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:03:08.891+0000 7f4e775cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:03:08.904 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T14:03:08.904 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T14:03:08.914 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T14:03:08.914 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T14:03:08.914 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T14:03:08.924 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T14:03:08.925 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T14:03:08.954 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:03:08 trial068 ceph-mon[38805]: pgmap v1945: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:03:08.955 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T14:03:08.955 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T14:03:09.334 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2983 8 365281 365273 0.956577 0 - 0.0299513 2026-02-23T14:03:10.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:03:09 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3247943004' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T14:03:10.334 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2984 8 365281 365273 0.956257 0 - 0.0299513 2026-02-23T14:03:10.957 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T14:03:10.981 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:03:10 trial068 ceph-mon[38805]: pgmap v1946: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:03:11.074 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:03:11.169 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:03:11.167+0000 7f38219a9640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:03:11.169 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:03:11.167+0000 7f38219a9640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:03:11.181 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T14:03:11.182 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T14:03:11.191 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T14:03:11.192 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T14:03:11.192 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T14:03:11.201 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T14:03:11.202 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T14:03:11.230 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T14:03:11.230 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T14:03:11.334 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2985 8 365281 365273 0.955936 0 - 0.0299513 2026-02-23T14:03:12.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:03:11 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3194810931' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T14:03:12.334 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2986 8 365281 365273 0.955616 0 - 0.0299513 2026-02-23T14:03:13.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:03:12 trial068 ceph-mon[38805]: pgmap v1947: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:03:13.231 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T14:03:13.334 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2987 8 365281 365273 0.955296 0 - 0.0299513 2026-02-23T14:03:13.347 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:03:13.445 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:03:13.442+0000 7fe3f3a64640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:03:13.445 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:03:13.443+0000 7fe3f3a64640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:03:13.457 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T14:03:13.457 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T14:03:13.467 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T14:03:13.467 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T14:03:13.468 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T14:03:13.477 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T14:03:13.477 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T14:03:13.509 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T14:03:13.509 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T14:03:13.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:03:13 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/514996' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T14:03:14.334 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2988 8 365281 365273 0.954977 0 - 0.0299513 2026-02-23T14:03:14.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:03:13 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:03:13] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:03:14.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:03:13 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:03:13.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:03:15.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:03:14 trial068 ceph-mon[38805]: pgmap v1948: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:03:15.334 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2989 8 365281 365273 0.954657 0 - 0.0299513 2026-02-23T14:03:15.510 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T14:03:15.630 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:03:15.727 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:03:15.725+0000 7f0d9ca40640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:03:15.727 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:03:15.725+0000 7f0d9ca40640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:03:15.739 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T14:03:15.739 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T14:03:15.750 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T14:03:15.750 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T14:03:15.750 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T14:03:15.760 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T14:03:15.760 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T14:03:15.789 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T14:03:15.790 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T14:03:16.334 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2990 8 365281 365273 0.954338 0 - 0.0299513 2026-02-23T14:03:17.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:03:16 trial068 ceph-mon[38805]: pgmap v1949: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:03:17.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:03:16 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/404290712' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T14:03:17.334 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2991 8 365281 365273 0.954019 0 - 0.0299513 2026-02-23T14:03:17.791 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T14:03:17.911 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:03:18.007 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:03:18.005+0000 7f0ba02b8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:03:18.007 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:03:18.005+0000 7f0ba02b8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:03:18.019 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T14:03:18.020 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T14:03:18.029 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T14:03:18.030 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T14:03:18.030 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T14:03:18.039 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T14:03:18.040 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T14:03:18.068 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T14:03:18.068 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T14:03:18.334 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2992 8 365281 365273 0.9537 0 - 0.0299513 2026-02-23T14:03:19.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:03:18 trial068 ceph-mon[38805]: pgmap v1950: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:03:19.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:03:18 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2922782419' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T14:03:19.334 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2993 8 365281 365273 0.953381 0 - 0.0299513 2026-02-23T14:03:20.070 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T14:03:20.189 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:03:20.292 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:03:20.290+0000 7f4186423640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:03:20.292 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:03:20.290+0000 7f4186423640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:03:20.304 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T14:03:20.304 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T14:03:20.314 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T14:03:20.314 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T14:03:20.314 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T14:03:20.324 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T14:03:20.324 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T14:03:20.334 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2994 8 365281 365273 0.953063 0 - 0.0299513 2026-02-23T14:03:20.354 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T14:03:20.355 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T14:03:21.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:03:20 trial068 ceph-mon[38805]: pgmap v1951: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:03:21.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:03:20 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1662541218' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T14:03:21.335 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2995 8 365281 365273 0.952745 0 - 0.0299513 2026-02-23T14:03:22.335 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2996 8 365281 365273 0.952427 0 - 0.0299513 2026-02-23T14:03:22.355 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T14:03:22.475 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:03:22.573 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:03:22.571+0000 7f021752a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:03:22.573 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:03:22.571+0000 7f021752a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:03:22.585 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T14:03:22.586 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T14:03:22.595 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T14:03:22.596 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T14:03:22.596 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T14:03:22.605 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T14:03:22.606 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T14:03:22.636 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T14:03:22.636 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T14:03:23.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:03:22 trial068 ceph-mon[38805]: pgmap v1952: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:03:23.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:03:22 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1904307129' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T14:03:23.335 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2997 8 365281 365273 0.952109 0 - 0.0299513 2026-02-23T14:03:24.335 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2998 8 365281 365273 0.951791 0 - 0.0299513 2026-02-23T14:03:24.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:03:23 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:03:23] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:03:24.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:03:23 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:03:23.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:03:24.637 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T14:03:24.756 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:03:24.855 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:03:24.853+0000 7f4f71326640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:03:24.855 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:03:24.853+0000 7f4f71326640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:03:24.867 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T14:03:24.867 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T14:03:24.877 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T14:03:24.877 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T14:03:24.878 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T14:03:24.887 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T14:03:24.887 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T14:03:24.917 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:03:24 trial068 ceph-mon[38805]: pgmap v1953: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:03:24.918 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T14:03:24.919 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T14:03:25.335 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 2999 8 365281 365273 0.951474 0 - 0.0299513 2026-02-23T14:03:26.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:03:25 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1670762280' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T14:03:26.335 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T14:03:26.334582+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T14:03:26.335 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T14:03:26.335 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3000 8 365281 365273 0.951157 0 - 0.0299513 2026-02-23T14:03:26.920 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T14:03:27.036 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:03:27.059 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:03:26 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T14:03:27.059 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:03:26 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T14:03:27.060 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:03:26 trial068 ceph-mon[38805]: pgmap v1954: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:03:27.136 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:03:27.134+0000 7f05cb967640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:03:27.136 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:03:27.134+0000 7f05cb967640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:03:27.148 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T14:03:27.148 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T14:03:27.158 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T14:03:27.158 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T14:03:27.159 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T14:03:27.170 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T14:03:27.170 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T14:03:27.200 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T14:03:27.201 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T14:03:27.335 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3001 8 365281 365273 0.95084 0 - 0.0299513 2026-02-23T14:03:28.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:03:27 trial068 ceph-mon[38805]: 5.e scrub starts 2026-02-23T14:03:28.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:03:27 trial068 ceph-mon[38805]: 5.e scrub ok 2026-02-23T14:03:28.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:03:27 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/83842943' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T14:03:28.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:03:27 trial068 ceph-mon[38805]: pgmap v1955: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:03:28.335 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3002 8 365281 365273 0.950523 0 - 0.0299513 2026-02-23T14:03:29.202 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T14:03:29.320 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:03:29.335 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3003 8 365281 365273 0.950207 0 - 0.0299513 2026-02-23T14:03:29.417 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:03:29.414+0000 7fc3e20e4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:03:29.417 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:03:29.415+0000 7fc3e20e4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:03:29.428 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T14:03:29.428 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T14:03:29.438 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T14:03:29.438 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T14:03:29.439 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T14:03:29.449 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T14:03:29.449 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T14:03:29.480 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T14:03:29.480 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T14:03:30.335 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3004 8 365281 365273 0.94989 0 - 0.0299513 2026-02-23T14:03:30.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:03:30 trial068 ceph-mon[38805]: pgmap v1956: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:03:30.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:03:30 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3561103825' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T14:03:31.335 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3005 8 365281 365273 0.949574 0 - 0.0299513 2026-02-23T14:03:31.481 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T14:03:31.601 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:03:31.700 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:03:31.698+0000 7fb42343c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:03:31.700 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:03:31.699+0000 7fb42343c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:03:31.712 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T14:03:31.713 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T14:03:31.723 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T14:03:31.723 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T14:03:31.723 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T14:03:31.734 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T14:03:31.734 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T14:03:31.765 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T14:03:31.766 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T14:03:32.336 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3006 8 365281 365273 0.949258 0 - 0.0299513 2026-02-23T14:03:32.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:03:32 trial068 ceph-mon[38805]: pgmap v1957: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:03:32.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:03:32 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/338769334' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T14:03:33.335 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3007 8 365281 365273 0.948943 0 - 0.0299513 2026-02-23T14:03:33.767 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T14:03:33.791 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:03:33 trial068 ceph-mon[38805]: 5.8 scrub starts 2026-02-23T14:03:33.792 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:03:33 trial068 ceph-mon[38805]: 5.8 scrub ok 2026-02-23T14:03:33.884 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:03:33.984 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:03:33.982+0000 7f5c258a8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:03:33.984 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:03:33.982+0000 7f5c258a8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:03:33.997 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T14:03:33.998 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T14:03:34.008 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T14:03:34.008 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T14:03:34.008 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T14:03:34.019 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T14:03:34.019 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T14:03:34.051 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:03:33 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:03:33] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:03:34.052 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T14:03:34.053 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T14:03:34.266 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:03:33 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:03:33.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:03:34.336 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3008 8 365281 365273 0.948627 0 - 0.0299513 2026-02-23T14:03:34.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:03:34 trial068 ceph-mon[38805]: pgmap v1958: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:03:34.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:03:34 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/511774657' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T14:03:35.336 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3009 8 365281 365273 0.948312 0 - 0.0299513 2026-02-23T14:03:36.054 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T14:03:36.174 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:03:36.270 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:03:36.267+0000 7feb7b02d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:03:36.270 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:03:36.268+0000 7feb7b02d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:03:36.282 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T14:03:36.282 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T14:03:36.292 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T14:03:36.292 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T14:03:36.293 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T14:03:36.303 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T14:03:36.303 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T14:03:36.331 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T14:03:36.331 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T14:03:36.336 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3010 8 365281 365273 0.947997 0 - 0.0299513 2026-02-23T14:03:36.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:03:36 trial068 ceph-mon[38805]: pgmap v1959: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:03:36.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:03:36 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2218135668' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T14:03:37.336 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3011 8 365281 365273 0.947682 0 - 0.0299513 2026-02-23T14:03:38.332 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T14:03:38.336 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3012 8 365281 365273 0.947367 0 - 0.0299513 2026-02-23T14:03:38.451 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:03:38.560 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:03:38.558+0000 7f52e72d9640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:03:38.561 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:03:38.558+0000 7f52e72d9640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:03:38.572 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T14:03:38.572 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T14:03:38.583 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T14:03:38.584 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T14:03:38.584 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T14:03:38.594 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T14:03:38.594 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T14:03:38.611 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:03:38 trial068 ceph-mon[38805]: pgmap v1960: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:03:38.625 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T14:03:38.626 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T14:03:39.336 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3013 8 365281 365273 0.947053 0 - 0.0299513 2026-02-23T14:03:39.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:03:39 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/919549374' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T14:03:40.336 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3014 8 365281 365273 0.946739 0 - 0.0299513 2026-02-23T14:03:40.626 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T14:03:40.744 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:03:40.766 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:03:40 trial068 ceph-mon[38805]: pgmap v1961: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:03:40.842 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:03:40.840+0000 7effd51b5640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:03:40.843 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:03:40.841+0000 7effd51b5640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:03:40.855 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T14:03:40.855 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T14:03:40.865 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T14:03:40.865 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T14:03:40.865 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T14:03:40.875 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T14:03:40.875 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T14:03:40.905 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T14:03:40.906 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T14:03:41.336 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3015 8 365281 365273 0.946425 0 - 0.0299513 2026-02-23T14:03:41.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:03:41 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3693506903' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T14:03:42.336 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3016 8 365281 365273 0.946111 0 - 0.0299513 2026-02-23T14:03:42.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:03:42 trial068 ceph-mon[38805]: pgmap v1962: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:03:42.907 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T14:03:43.025 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:03:43.124 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:03:43.122+0000 7facb7e1d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:03:43.124 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:03:43.122+0000 7facb7e1d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:03:43.136 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T14:03:43.137 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T14:03:43.147 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T14:03:43.148 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T14:03:43.148 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T14:03:43.158 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T14:03:43.159 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T14:03:43.189 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T14:03:43.190 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T14:03:43.336 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3017 8 365281 365273 0.945797 0 - 0.0299513 2026-02-23T14:03:43.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:03:43 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2929457011' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T14:03:44.337 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3018 8 365281 365273 0.945484 0 - 0.0299513 2026-02-23T14:03:44.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:03:43 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:03:43] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:03:44.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:03:43 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:03:43.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:03:44.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:03:44 trial068 ceph-mon[38805]: pgmap v1963: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:03:45.191 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T14:03:45.310 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:03:45.337 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3019 8 365281 365273 0.945171 0 - 0.0299513 2026-02-23T14:03:45.412 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:03:45.410+0000 7f0d28c69640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:03:45.412 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:03:45.410+0000 7f0d28c69640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:03:45.423 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T14:03:45.424 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T14:03:45.434 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T14:03:45.434 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T14:03:45.434 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T14:03:45.444 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T14:03:45.444 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T14:03:45.473 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:03:45 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3733389787' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T14:03:45.474 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T14:03:45.474 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T14:03:46.337 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T14:03:46.336444+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T14:03:46.337 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T14:03:46.337 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3020 8 365281 365273 0.944858 0 - 0.0299513 2026-02-23T14:03:46.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:03:46 trial068 ceph-mon[38805]: pgmap v1964: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:03:47.337 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3021 8 365281 365273 0.944545 0 - 0.0299513 2026-02-23T14:03:47.475 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T14:03:47.592 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:03:47.686 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:03:47.684+0000 7f25697dd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:03:47.686 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:03:47.685+0000 7f25697dd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:03:47.698 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T14:03:47.698 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T14:03:47.708 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T14:03:47.708 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T14:03:47.709 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T14:03:47.719 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T14:03:47.719 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T14:03:47.750 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T14:03:47.750 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T14:03:48.337 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3022 8 365281 365273 0.944232 0 - 0.0299513 2026-02-23T14:03:48.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:03:48 trial068 ceph-mon[38805]: pgmap v1965: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:03:48.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:03:48 trial068 ceph-mon[38805]: 5.a scrub starts 2026-02-23T14:03:48.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:03:48 trial068 ceph-mon[38805]: 5.a scrub ok 2026-02-23T14:03:48.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:03:48 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2911815587' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T14:03:49.337 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3023 8 365281 365273 0.94392 0 - 0.0299513 2026-02-23T14:03:49.751 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T14:03:49.774 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:03:49 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T14:03:49.774 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:03:49 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T14:03:49.775 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:03:49 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2026-02-23T14:03:49.866 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:03:49.972 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:03:49.969+0000 7f4fdc125640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:03:49.972 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:03:49.970+0000 7f4fdc125640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:03:49.984 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T14:03:49.984 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T14:03:49.995 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T14:03:49.995 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T14:03:49.995 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T14:03:50.006 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T14:03:50.007 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T14:03:50.037 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T14:03:50.038 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T14:03:50.337 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3024 8 365281 365273 0.943608 0 - 0.0299513 2026-02-23T14:03:50.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:03:50 trial068 ceph-mon[38805]: pgmap v1966: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:03:50.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:03:50 trial068 ceph-mon[38805]: 5.5 scrub starts 2026-02-23T14:03:50.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:03:50 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-23T14:03:50.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:03:50 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2026-02-23T14:03:50.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:03:50 trial068 ceph-mon[38805]: pgmap v1967: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:03:50.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:03:50 trial068 ceph-mon[38805]: pgmap v1968: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:03:50.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:03:50 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T14:03:50.845 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:03:50 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2026-02-23T14:03:50.845 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:03:50 trial068 ceph-mon[38805]: 5.5 scrub ok 2026-02-23T14:03:50.845 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:03:50 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/416747086' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T14:03:51.337 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3025 8 365281 365273 0.943296 0 - 0.0299513 2026-02-23T14:03:52.039 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T14:03:52.156 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:03:52.254 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:03:52.251+0000 7f4502372640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:03:52.254 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:03:52.252+0000 7f4502372640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:03:52.265 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T14:03:52.266 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T14:03:52.275 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T14:03:52.276 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T14:03:52.276 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T14:03:52.286 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T14:03:52.286 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T14:03:52.315 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T14:03:52.316 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T14:03:52.337 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3026 8 365281 365273 0.942984 0 - 0.0299513 2026-02-23T14:03:53.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:03:52 trial068 ceph-mon[38805]: pgmap v1969: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:03:53.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:03:52 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/5395684' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T14:03:53.337 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3027 8 365281 365273 0.942673 0 - 0.0299513 2026-02-23T14:03:54.317 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T14:03:54.337 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3028 8 365281 365273 0.942361 0 - 0.0299513 2026-02-23T14:03:54.341 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:03:53 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:03:53] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:03:54.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:03:53 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:03:53.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:03:54.436 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:03:54.534 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:03:54.532+0000 7f51ec68f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:03:54.534 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:03:54.532+0000 7f51ec68f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:03:54.545 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T14:03:54.546 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T14:03:54.555 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T14:03:54.556 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T14:03:54.556 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T14:03:54.566 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T14:03:54.567 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T14:03:54.615 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T14:03:54.615 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T14:03:54.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:03:54 trial068 ceph-mon[38805]: pgmap v1970: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:03:54.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:03:54 trial068 ceph-mon[38805]: osdmap e217: 4 total, 3 up, 4 in 2026-02-23T14:03:54.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:03:54 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3580560862' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T14:03:55.337 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3029 8 365281 365273 0.94205 0 - 0.0299513 2026-02-23T14:03:56.337 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3030 8 365281 365273 0.941739 0 - 0.0299513 2026-02-23T14:03:56.617 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T14:03:56.733 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:03:56.835 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:03:56.833+0000 7f896fc9f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:03:56.835 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:03:56.833+0000 7f896fc9f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:03:56.847 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T14:03:56.847 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T14:03:56.857 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T14:03:56.857 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T14:03:56.857 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T14:03:56.867 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T14:03:56.868 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T14:03:56.868 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:03:56 trial068 ceph-mon[38805]: pgmap v1972: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:03:56.897 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T14:03:56.897 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T14:03:57.338 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3031 8 365281 365273 0.941429 0 - 0.0299513 2026-02-23T14:03:58.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:03:57 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/398980987' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T14:03:58.338 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3032 8 365281 365273 0.941118 0 - 0.0299513 2026-02-23T14:03:58.898 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T14:03:58.922 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:03:58 trial068 ceph-mon[38805]: pgmap v1973: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:03:59.015 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:03:59.112 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:03:59.110+0000 7ff209e94640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:03:59.112 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:03:59.110+0000 7ff209e94640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:03:59.124 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T14:03:59.124 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T14:03:59.134 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T14:03:59.135 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T14:03:59.135 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T14:03:59.145 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T14:03:59.145 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T14:03:59.175 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T14:03:59.175 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T14:03:59.338 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3033 8 365281 365273 0.940808 0 - 0.0299513 2026-02-23T14:04:00.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:03:59 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/550673399' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T14:04:00.338 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3034 8 365281 365273 0.940498 0 - 0.0299513 2026-02-23T14:04:01.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:04:00 trial068 ceph-mon[38805]: pgmap v1974: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:04:01.176 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T14:04:01.293 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:04:01.338 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3035 8 365281 365273 0.940188 0 - 0.0299513 2026-02-23T14:04:01.385 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:04:01.383+0000 7fb2c1c73640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:04:01.385 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:04:01.383+0000 7fb2c1c73640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:04:01.397 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T14:04:01.397 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T14:04:01.407 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T14:04:01.408 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T14:04:01.408 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T14:04:01.417 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T14:04:01.418 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T14:04:01.445 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T14:04:01.446 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T14:04:02.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:04:01 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1594293911' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T14:04:02.338 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3036 8 365281 365273 0.939878 0 - 0.0299513 2026-02-23T14:04:03.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:04:02 trial068 ceph-mon[38805]: pgmap v1975: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:04:03.338 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3037 8 365281 365273 0.939569 0 - 0.0299513 2026-02-23T14:04:03.447 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T14:04:03.567 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:04:03.670 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:04:03.668+0000 7fca8234c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:04:03.670 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:04:03.668+0000 7fca8234c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:04:03.682 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T14:04:03.682 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T14:04:03.692 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T14:04:03.692 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T14:04:03.692 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T14:04:03.702 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T14:04:03.702 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T14:04:03.730 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:04:03 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T14:04:03.730 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:04:03 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T14:04:03.730 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T14:04:03.731 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T14:04:04.094 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:04:03 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:04:03] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:04:04.266 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:04:03 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:04:03.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:04:04.338 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3038 8 365281 365273 0.939259 0 - 0.0299513 2026-02-23T14:04:05.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:04:04 trial068 ceph-mon[38805]: pgmap v1976: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:04:05.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:04:04 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/929350371' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T14:04:05.338 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3039 8 365281 365273 0.93895 0 - 0.0299513 2026-02-23T14:04:05.732 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T14:04:05.852 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:04:05.949 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:04:05.947+0000 7f166c0c9640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:04:05.949 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:04:05.947+0000 7f166c0c9640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:04:05.961 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T14:04:05.961 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T14:04:05.971 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T14:04:05.971 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T14:04:05.971 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T14:04:05.981 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T14:04:05.981 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T14:04:06.011 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T14:04:06.011 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T14:04:06.338 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T14:04:06.337962+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T14:04:06.339 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T14:04:06.339 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3040 8 365281 365273 0.938641 0 - 0.0299513 2026-02-23T14:04:07.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:04:06 trial068 ceph-mon[38805]: pgmap v1977: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:04:07.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:04:06 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1795748679' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T14:04:07.338 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3041 8 365281 365273 0.938333 0 - 0.0299513 2026-02-23T14:04:08.013 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-23T14:04:08.130 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:04:08.235 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:04:08.233+0000 7feff49dd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:04:08.235 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:04:08.233+0000 7feff49dd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:04:08.247 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T14:04:08.248 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T14:04:08.257 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T14:04:08.258 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T14:04:08.258 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T14:04:08.269 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T14:04:08.269 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T14:04:08.303 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T14:04:08.304 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T14:04:08.338 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3042 8 365281 365273 0.938024 0 - 0.0299513 2026-02-23T14:04:09.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:04:08 trial068 ceph-mon[38805]: 5.7 scrub starts 2026-02-23T14:04:09.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:04:08 trial068 ceph-mon[38805]: 5.7 scrub ok 2026-02-23T14:04:09.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:04:08 trial068 ceph-mon[38805]: pgmap v1978: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:04:09.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:04:08 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/4221392176' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T14:04:09.338 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3043 8 365281 365273 0.937716 0 - 0.0299513 2026-02-23T14:04:10.304 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-23T14:04:10.338 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3044 8 365281 365273 0.937408 0 - 0.0299513 2026-02-23T14:04:10.420 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:04:10.529 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:04:10.527+0000 7fd7d6c67640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:04:10.529 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:04:10.527+0000 7fd7d6c67640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:04:10.541 INFO:teuthology.orchestra.run.trial068.stdout:osd.0: {} 2026-02-23T14:04:10.541 INFO:teuthology.orchestra.run.trial068.stderr:osd.0: osd_enable_op_tracker = '' 2026-02-23T14:04:10.551 INFO:teuthology.orchestra.run.trial068.stdout:osd.1: {} 2026-02-23T14:04:10.552 INFO:teuthology.orchestra.run.trial068.stderr:osd.1: osd_enable_op_tracker = '' 2026-02-23T14:04:10.552 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-23T14:04:10.562 INFO:teuthology.orchestra.run.trial068.stdout:osd.3: {} 2026-02-23T14:04:10.562 INFO:teuthology.orchestra.run.trial068.stderr:osd.3: osd_enable_op_tracker = '' 2026-02-23T14:04:10.593 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-23T14:04:10.594 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-23T14:04:10.642 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) ^^^^^^^^^^^^^^ File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/teuthology/task/full_sequential.py", line 37, in task mgr.__exit__(*exc_info) File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_shraddhaag_ceph_a78040ddef3ba5a489697ca2fe95df59a7d8bd08/qa/tasks/radosbench.py", line 142, in task run.wait(radosbench.values(), timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/teuthology/orchestra/run.py", line 479, in wait check_time() File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (500) after waiting for 3000 seconds 2026-02-23T14:04:10.645 DEBUG:teuthology.run_tasks:Unwinding manager thrashosds 2026-02-23T14:04:10.659 INFO:tasks.thrashosds:joining thrashosds 2026-02-23T14:04:10.660 INFO:tasks.thrashosds.thrasher:joining the do_optrack_toggle greenlet 2026-02-23T14:04:10.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:04:10 trial068 ceph-mon[38805]: pgmap v1979: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:04:10.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:04:10 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/391855560' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2026-02-23T14:04:11.339 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3045 8 365281 365273 0.9371 0 - 0.0299513 2026-02-23T14:04:12.339 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3046 8 365281 365273 0.936793 0 - 0.0299513 2026-02-23T14:04:12.594 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2026-02-23T14:04:12.594 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:04:12.711 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:04:12.814 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:04:12.812+0000 7f98da2d6640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:04:12.815 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:04:12.812+0000 7f98da2d6640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:04:12.923 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:04:12.923 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":217,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:03:53.610644+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:01:14.553379+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:03:48.616954+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:59:30.175917+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:04:12.923 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:04:12 trial068 ceph-mon[38805]: pgmap v1980: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:04:13.339 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3047 8 365281 365273 0.936485 0 - 0.0299513 2026-02-23T14:04:13.975 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:04:13 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2872656797' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:04:14.339 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3048 8 365281 365273 0.936178 0 - 0.0299513 2026-02-23T14:04:14.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:04:13 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:04:13] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:04:14.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:04:13 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:04:13.974Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:04:15.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:04:14 trial068 ceph-mon[38805]: pgmap v1981: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:04:15.339 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3049 8 365281 365273 0.935871 0 - 0.0299513 2026-02-23T14:04:15.959 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:04:16.077 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:04:16.174 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:04:16.171+0000 7f7ced009640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:04:16.174 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:04:16.172+0000 7f7ced009640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:04:16.282 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:04:16.282 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":217,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:03:53.610644+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:01:14.553379+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:03:48.616954+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:59:30.175917+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:04:16.339 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3050 8 365281 365273 0.935564 0 - 0.0299513 2026-02-23T14:04:17.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:04:16 trial068 ceph-mon[38805]: pgmap v1982: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:04:17.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:04:16 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/624692791' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:04:17.339 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3051 8 365281 365273 0.935257 0 - 0.0299513 2026-02-23T14:04:18.339 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3052 8 365281 365273 0.934951 0 - 0.0299513 2026-02-23T14:04:19.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:04:18 trial068 ceph-mon[38805]: pgmap v1983: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:04:19.318 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:04:19.339 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3053 8 365281 365273 0.934645 0 - 0.0299513 2026-02-23T14:04:19.436 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:04:19.539 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:04:19.537+0000 7f05e0c40640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:04:19.540 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:04:19.538+0000 7f05e0c40640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:04:19.648 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:04:19.648 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":217,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:03:53.610644+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:01:14.553379+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:03:48.616954+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:59:30.175917+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:04:20.339 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3054 8 365281 365273 0.934339 0 - 0.0299513 2026-02-23T14:04:21.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:04:20 trial068 ceph-mon[38805]: pgmap v1984: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:04:21.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:04:20 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2996376091' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:04:21.339 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3055 8 365281 365273 0.934033 0 - 0.0299513 2026-02-23T14:04:22.340 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3056 8 365281 365273 0.933727 0 - 0.0299513 2026-02-23T14:04:22.686 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:04:22.802 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:04:22.901 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:04:22 trial068 ceph-mon[38805]: pgmap v1985: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:04:22.902 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:04:22.899+0000 7f156a767640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:04:22.902 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:04:22.899+0000 7f156a767640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:04:23.009 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:04:23.009 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":217,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:03:53.610644+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:01:14.553379+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:03:48.616954+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:59:30.175917+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:04:23.340 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3057 8 365281 365273 0.933422 0 - 0.0299513 2026-02-23T14:04:23.974 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:04:23 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1758661405' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:04:24.340 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3058 8 365281 365273 0.933116 0 - 0.0299513 2026-02-23T14:04:24.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:04:23 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:04:23] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:04:24.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:04:23 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:04:23.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:04:25.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:04:24 trial068 ceph-mon[38805]: pgmap v1986: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:04:25.340 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3059 8 365281 365273 0.932811 0 - 0.0299513 2026-02-23T14:04:26.050 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:04:26.168 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:04:26.270 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:04:26.268+0000 7feeaca82640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:04:26.271 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:04:26.268+0000 7feeaca82640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:04:26.340 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T14:04:26.339603+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T14:04:26.340 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T14:04:26.340 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3060 8 365281 365273 0.932507 0 - 0.0299513 2026-02-23T14:04:26.379 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:04:26.379 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":217,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:03:53.610644+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:01:14.553379+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:03:48.616954+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:59:30.175917+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:04:27.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:04:26 trial068 ceph-mon[38805]: pgmap v1987: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:04:27.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:04:26 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1425270873' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:04:27.340 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3061 8 365281 365273 0.932202 0 - 0.0299513 2026-02-23T14:04:28.340 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3062 8 365281 365273 0.931897 0 - 0.0299513 2026-02-23T14:04:29.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:04:28 trial068 ceph-mon[38805]: pgmap v1988: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:04:29.340 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3063 8 365281 365273 0.931593 0 - 0.0299513 2026-02-23T14:04:29.416 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:04:29.534 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:04:29.632 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:04:29.630+0000 7f144ad1f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:04:29.632 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:04:29.630+0000 7f144ad1f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:04:29.741 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:04:29.742 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":217,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:03:53.610644+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:01:14.553379+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:03:48.616954+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:59:30.175917+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:04:30.340 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3064 8 365281 365273 0.931289 0 - 0.0299513 2026-02-23T14:04:31.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:04:30 trial068 ceph-mon[38805]: pgmap v1989: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:04:31.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:04:30 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1771306907' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:04:31.340 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3065 8 365281 365273 0.930985 0 - 0.0299513 2026-02-23T14:04:32.340 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3066 8 365281 365273 0.930682 0 - 0.0299513 2026-02-23T14:04:32.780 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:04:32.897 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:04:32.919 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:04:32 trial068 ceph-mon[38805]: 5.e scrub starts 2026-02-23T14:04:32.919 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:04:32 trial068 ceph-mon[38805]: 5.e scrub ok 2026-02-23T14:04:32.919 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:04:32 trial068 ceph-mon[38805]: pgmap v1990: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:04:32.997 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:04:32.994+0000 7f41b8a77640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:04:32.997 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:04:32.995+0000 7f41b8a77640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:04:33.107 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:04:33.107 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":217,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:03:53.610644+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:01:14.553379+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:03:48.616954+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:59:30.175917+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:04:33.340 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3067 8 365281 365273 0.930378 0 - 0.0299513 2026-02-23T14:04:33.974 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:04:33 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3890177369' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:04:34.266 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:04:33 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:04:33.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:04:34.340 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3068 8 365281 365273 0.930075 0 - 0.0299513 2026-02-23T14:04:34.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:04:33 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:04:33] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:04:35.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:04:34 trial068 ceph-mon[38805]: pgmap v1991: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:04:35.341 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3069 8 365281 365273 0.929772 0 - 0.0299513 2026-02-23T14:04:36.146 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:04:36.264 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:04:36.341 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3070 8 365281 365273 0.929469 0 - 0.0299513 2026-02-23T14:04:36.365 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:04:36.363+0000 7f4132a0d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:04:36.365 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:04:36.363+0000 7f4132a0d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:04:36.474 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:04:36.474 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":217,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:03:53.610644+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:01:14.553379+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:03:48.616954+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:59:30.175917+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:04:36.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:04:36 trial068 ceph-mon[38805]: 5.8 scrub starts 2026-02-23T14:04:36.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:04:36 trial068 ceph-mon[38805]: 5.8 scrub ok 2026-02-23T14:04:36.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:04:36 trial068 ceph-mon[38805]: pgmap v1992: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:04:36.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:04:36 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T14:04:36.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:04:36 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T14:04:36.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:04:36 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1155397201' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:04:37.341 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3071 8 365281 365273 0.929166 0 - 0.0299513 2026-02-23T14:04:38.341 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3072 8 365281 365273 0.928864 0 - 0.0299513 2026-02-23T14:04:39.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:04:38 trial068 ceph-mon[38805]: pgmap v1993: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:04:39.341 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3073 8 365281 365273 0.928562 0 - 0.0299513 2026-02-23T14:04:39.512 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:04:39.630 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:04:39.726 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:04:39.724+0000 7fb50dd4f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:04:39.727 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:04:39.724+0000 7fb50dd4f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:04:39.836 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:04:39.836 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":217,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:03:53.610644+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:01:14.553379+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:03:48.616954+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:59:30.175917+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:04:40.341 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3074 8 365281 365273 0.92826 0 - 0.0299513 2026-02-23T14:04:41.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:04:40 trial068 ceph-mon[38805]: pgmap v1994: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:04:41.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:04:40 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3681218128' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:04:41.341 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3075 8 365281 365273 0.927958 0 - 0.0299513 2026-02-23T14:04:42.341 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3076 8 365281 365273 0.927656 0 - 0.0299513 2026-02-23T14:04:42.874 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:04:42.991 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:04:43.013 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:04:42 trial068 ceph-mon[38805]: pgmap v1995: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:04:43.090 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:04:43.088+0000 7f3bd7917640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:04:43.091 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:04:43.089+0000 7f3bd7917640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:04:43.201 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:04:43.201 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":217,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:03:53.610644+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:01:14.553379+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:03:48.616954+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:59:30.175917+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:04:43.341 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3077 8 365281 365273 0.927355 0 - 0.0299513 2026-02-23T14:04:43.974 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:04:43 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3060558733' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:04:44.341 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3078 8 365281 365273 0.927053 0 - 0.0299513 2026-02-23T14:04:44.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:04:43 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:04:43] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:04:44.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:04:43 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:04:43.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:04:45.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:04:44 trial068 ceph-mon[38805]: pgmap v1996: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:04:45.341 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3079 8 365281 365273 0.926752 0 - 0.0299513 2026-02-23T14:04:46.238 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:04:46.341 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T14:04:46.340926+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T14:04:46.341 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T14:04:46.342 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3080 8 365281 365273 0.926451 0 - 0.0299513 2026-02-23T14:04:46.355 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:04:46.452 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:04:46.450+0000 7f79a4ffb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:04:46.452 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:04:46.450+0000 7f79a4ffb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:04:46.562 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:04:46.562 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":217,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:03:53.610644+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:01:14.553379+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:03:48.616954+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:59:30.175917+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:04:46.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:04:46 trial068 ceph-mon[38805]: pgmap v1997: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:04:46.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:04:46 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2094805431' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:04:47.341 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3081 8 365281 365273 0.926151 0 - 0.0299513 2026-02-23T14:04:48.341 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3082 8 365281 365273 0.92585 0 - 0.0299513 2026-02-23T14:04:49.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:04:48 trial068 ceph-mon[38805]: pgmap v1998: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:04:49.341 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3083 8 365281 365273 0.92555 0 - 0.0299513 2026-02-23T14:04:49.599 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:04:49.718 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:04:49.828 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:04:49.825+0000 7f56b3dfe640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:04:49.828 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:04:49.826+0000 7f56b3dfe640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:04:49.937 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:04:49.942 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":217,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:03:53.610644+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:01:14.553379+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:03:48.616954+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:59:30.175917+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:04:49.942 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:04:49 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2026-02-23T14:04:50.342 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3084 8 365281 365273 0.92525 0 - 0.0299513 2026-02-23T14:04:51.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:04:50 trial068 ceph-mon[38805]: pgmap v1999: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:04:51.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:04:50 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-23T14:04:51.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:04:50 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2026-02-23T14:04:51.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:04:50 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T14:04:51.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:04:50 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2026-02-23T14:04:51.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:04:50 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/4188532722' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:04:51.342 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3085 8 365281 365273 0.92495 0 - 0.0299513 2026-02-23T14:04:52.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:04:51 trial068 ceph-mon[38805]: pgmap v2000: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:04:52.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:04:51 trial068 ceph-mon[38805]: pgmap v2001: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:04:52.342 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3086 8 365281 365273 0.92465 0 - 0.0299513 2026-02-23T14:04:52.977 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:04:53.093 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:04:53.195 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:04:53.193+0000 7ff082e5b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:04:53.195 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:04:53.193+0000 7ff082e5b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:04:53.305 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:04:53.305 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":217,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:03:53.610644+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:01:14.553379+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:03:48.616954+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:59:30.175917+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:04:53.342 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3087 8 365281 365273 0.924351 0 - 0.0299513 2026-02-23T14:04:53.975 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:04:53 trial068 ceph-mon[38805]: pgmap v2002: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:04:53.975 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:04:53 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/117923193' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:04:54.342 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3088 8 365281 365273 0.924051 0 - 0.0299513 2026-02-23T14:04:54.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:04:53 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:04:53] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:04:54.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:04:53 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:04:53.974Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:04:55.342 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3089 8 365281 365273 0.923752 0 - 0.0299513 2026-02-23T14:04:56.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:04:55 trial068 ceph-mon[38805]: pgmap v2003: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:04:56.342 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3090 8 365281 365273 0.923453 0 - 0.0299513 2026-02-23T14:04:56.342 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:04:56.460 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:04:56.557 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:04:56.555+0000 7f1f2b35c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:04:56.558 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:04:56.556+0000 7f1f2b35c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:04:56.666 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:04:56.666 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":217,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:03:53.610644+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:01:14.553379+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:03:48.616954+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:59:30.175917+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:04:57.342 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3091 8 365281 365273 0.923154 0 - 0.0299513 2026-02-23T14:04:58.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:04:57 trial068 ceph-mon[38805]: pgmap v2004: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:04:58.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:04:57 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2817492519' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:04:58.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:04:57 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T14:04:58.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:04:57 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T14:04:58.342 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3092 8 365281 365273 0.922856 0 - 0.0299513 2026-02-23T14:04:59.342 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3093 8 365281 365273 0.922557 0 - 0.0299513 2026-02-23T14:04:59.710 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:04:59.828 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:04:59.930 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:04:59.927+0000 7fb7c5134640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:04:59.930 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:04:59 trial068 ceph-mon[38805]: pgmap v2005: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:04:59.930 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:04:59 trial068 ceph-mon[38805]: 5.a scrub starts 2026-02-23T14:04:59.930 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:04:59 trial068 ceph-mon[38805]: 5.a scrub ok 2026-02-23T14:04:59.931 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:04:59.928+0000 7fb7c5134640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:05:00.038 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:05:00.039 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":217,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:03:53.610644+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:01:14.553379+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:03:48.616954+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:59:30.175917+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:05:00.342 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3094 8 365281 365273 0.922259 0 - 0.0299513 2026-02-23T14:05:01.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:05:00 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/4075208096' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:05:01.342 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3095 8 365281 365273 0.921961 0 - 0.0299513 2026-02-23T14:05:02.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:05:01 trial068 ceph-mon[38805]: pgmap v2006: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:05:02.342 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3096 8 365281 365273 0.921664 0 - 0.0299513 2026-02-23T14:05:03.075 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:05:03.195 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:05:03.298 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:05:03.295+0000 7fc6a82d2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:05:03.298 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:05:03.295+0000 7fc6a82d2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:05:03.342 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3097 8 365281 365273 0.921366 0 - 0.0299513 2026-02-23T14:05:03.406 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:05:03.406 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":217,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:03:53.610644+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:01:14.553379+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:03:48.616954+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:59:30.175917+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:05:03.974 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:05:03 trial068 ceph-mon[38805]: pgmap v2007: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:05:03.974 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:05:03 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1071154098' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:05:04.266 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:05:03 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:05:03.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:05:04.343 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3098 8 365281 365273 0.921069 0 - 0.0299513 2026-02-23T14:05:04.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:05:03 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:05:03] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:05:05.343 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3099 8 365281 365273 0.920771 0 - 0.0299513 2026-02-23T14:05:06.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:05:05 trial068 ceph-mon[38805]: pgmap v2008: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:05:06.343 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T14:05:06.342383+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T14:05:06.343 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T14:05:06.343 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3100 8 365281 365273 0.920474 0 - 0.0299513 2026-02-23T14:05:06.444 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:05:06.565 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:05:06.661 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:05:06.659+0000 7f84c9886640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:05:06.662 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:05:06.659+0000 7f84c9886640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:05:06.771 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:05:06.771 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":217,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:03:53.610644+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:01:14.553379+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:03:48.616954+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:59:30.175917+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:05:07.343 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3101 8 365281 365273 0.920177 0 - 0.0299513 2026-02-23T14:05:08.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:05:07 trial068 ceph-mon[38805]: pgmap v2009: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:05:08.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:05:07 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1752477033' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:05:08.343 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3102 8 365281 365273 0.919881 0 - 0.0299513 2026-02-23T14:05:09.343 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3103 8 365281 365273 0.919584 0 - 0.0299513 2026-02-23T14:05:09.809 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:05:09.927 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:05:09.949 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:05:09 trial068 ceph-mon[38805]: pgmap v2010: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:05:10.024 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:05:10.021+0000 7fa82d3d6640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:05:10.024 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:05:10.022+0000 7fa82d3d6640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:05:10.133 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:05:10.133 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":217,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:03:53.610644+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:01:14.553379+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:03:48.616954+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:59:30.175917+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:05:10.343 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3104 8 365281 365273 0.919288 0 - 0.0299513 2026-02-23T14:05:11.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:05:10 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2972103460' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:05:11.343 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3105 8 365281 365273 0.918992 0 - 0.0299513 2026-02-23T14:05:12.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:05:11 trial068 ceph-mon[38805]: pgmap v2011: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:05:12.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:05:11 trial068 ceph-mon[38805]: osdmap e218: 4 total, 3 up, 4 in 2026-02-23T14:05:12.343 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3106 8 365281 365273 0.918696 0 - 0.0299513 2026-02-23T14:05:13.171 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:05:13.289 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:05:13.343 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3107 8 365281 365273 0.9184 0 - 0.0299513 2026-02-23T14:05:13.386 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:05:13.384+0000 7f4a65499640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:05:13.386 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:05:13.384+0000 7f4a65499640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:05:13.496 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:05:13.496 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":218,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:05:10.676625+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:01:14.553379+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:03:48.616954+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:04:35.751423+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:05:13.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:05:13 trial068 ceph-mon[38805]: pgmap v2013: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:05:13.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:05:13 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/58326542' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:05:14.343 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3108 8 365281 365273 0.918105 0 - 0.0299513 2026-02-23T14:05:14.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:05:13 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:05:13] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:05:14.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:05:13 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:05:13.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:05:15.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:05:14 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T14:05:15.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:05:14 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T14:05:15.343 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3109 8 365281 365273 0.91781 0 - 0.0299513 2026-02-23T14:05:16.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:05:15 trial068 ceph-mon[38805]: pgmap v2014: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:05:16.343 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3110 8 365281 365273 0.917515 0 - 0.0299513 2026-02-23T14:05:16.548 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:05:16.666 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:05:16.766 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:05:16.763+0000 7f5524668640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:05:16.766 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:05:16.764+0000 7f5524668640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:05:16.874 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:05:16.874 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":218,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:05:10.676625+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:01:14.553379+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:03:48.616954+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:04:35.751423+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:05:17.344 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3111 8 365281 365273 0.91722 0 - 0.0299513 2026-02-23T14:05:18.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:05:17 trial068 ceph-mon[38805]: pgmap v2015: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:05:18.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:05:17 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3470956132' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:05:18.344 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3112 8 365281 365273 0.916925 0 - 0.0299513 2026-02-23T14:05:19.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:05:18 trial068 ceph-mon[38805]: 5.5 scrub starts 2026-02-23T14:05:19.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:05:18 trial068 ceph-mon[38805]: 5.5 scrub ok 2026-02-23T14:05:19.344 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3113 8 365281 365273 0.91663 0 - 0.0299513 2026-02-23T14:05:19.910 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:05:19.936 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:05:19 trial068 ceph-mon[38805]: pgmap v2016: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:05:20.028 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:05:20.126 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:05:20.124+0000 7f344754a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:05:20.126 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:05:20.124+0000 7f344754a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:05:20.236 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:05:20.236 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":218,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:05:10.676625+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:01:14.553379+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:03:48.616954+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:04:35.751423+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:05:20.344 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3114 8 365281 365273 0.916336 0 - 0.0299513 2026-02-23T14:05:21.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:05:20 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/897969433' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:05:21.344 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3115 8 365281 365273 0.916042 0 - 0.0299513 2026-02-23T14:05:22.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:05:21 trial068 ceph-mon[38805]: pgmap v2017: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:05:22.344 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3116 8 365281 365273 0.915748 0 - 0.0299513 2026-02-23T14:05:23.275 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:05:23.344 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3117 8 365281 365273 0.915454 0 - 0.0299513 2026-02-23T14:05:23.396 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:05:23.502 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:05:23.500+0000 7f62b0cdc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:05:23.503 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:05:23.500+0000 7f62b0cdc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:05:23.613 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:05:23.613 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":218,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:05:10.676625+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:01:14.553379+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:03:48.616954+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:04:35.751423+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:05:23.974 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:05:23 trial068 ceph-mon[38805]: pgmap v2018: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:05:23.974 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:05:23 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3912803274' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:05:24.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:05:23 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:05:23] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:05:24.344 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3118 8 365281 365273 0.91516 0 - 0.0299513 2026-02-23T14:05:24.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:05:23 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:05:23.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:05:25.344 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3119 8 365281 365273 0.914867 0 - 0.0299513 2026-02-23T14:05:26.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:05:25 trial068 ceph-mon[38805]: pgmap v2019: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:05:26.344 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T14:05:26.343857+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T14:05:26.344 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T14:05:26.345 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3120 8 365281 365273 0.914574 0 - 0.0299513 2026-02-23T14:05:26.651 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:05:26.767 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:05:26.863 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:05:26.861+0000 7fb0287cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:05:26.864 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:05:26.861+0000 7fb0287cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:05:26.976 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:05:26.976 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":218,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:05:10.676625+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:01:14.553379+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:03:48.616954+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:04:35.751423+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:05:27.344 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3121 8 365281 365273 0.914281 0 - 0.0299513 2026-02-23T14:05:28.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:05:27 trial068 ceph-mon[38805]: pgmap v2020: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:05:28.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:05:27 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1873152894' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:05:28.344 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3122 8 365281 365273 0.913988 0 - 0.0299513 2026-02-23T14:05:29.344 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3123 8 365281 365273 0.913695 0 - 0.0299513 2026-02-23T14:05:30.014 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:05:30.040 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:05:29 trial068 ceph-mon[38805]: pgmap v2021: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:05:30.131 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:05:30.237 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:05:30.235+0000 7fbad43fe640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:05:30.237 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:05:30.235+0000 7fbad43fe640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:05:30.344 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3124 8 365281 365273 0.913403 0 - 0.0299513 2026-02-23T14:05:30.347 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:05:30.348 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":218,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:05:10.676625+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:01:14.553379+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:03:48.616954+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:04:35.751423+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:05:31.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:05:30 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/993338213' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:05:31.345 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3125 8 365281 365273 0.913111 0 - 0.0299513 2026-02-23T14:05:32.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:05:31 trial068 ceph-mon[38805]: pgmap v2022: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:05:32.345 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3126 8 365281 365273 0.912818 0 - 0.0299513 2026-02-23T14:05:33.345 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3127 8 365281 365273 0.912527 0 - 0.0299513 2026-02-23T14:05:33.388 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:05:33.507 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:05:33.605 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:05:33.602+0000 7fdfeef68640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:05:33.605 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:05:33.603+0000 7fdfeef68640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:05:33.714 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:05:33.714 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":218,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:05:10.676625+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:01:14.553379+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:03:48.616954+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:04:35.751423+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:05:33.975 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:05:33 trial068 ceph-mon[38805]: pgmap v2023: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:05:34.266 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:05:33 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:05:33.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:05:34.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:05:33 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:05:33] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:05:34.345 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3128 8 365281 365273 0.912235 0 - 0.0299513 2026-02-23T14:05:35.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:05:34 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2046408918' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:05:35.345 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3129 8 365281 365273 0.911943 0 - 0.0299513 2026-02-23T14:05:36.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:05:35 trial068 ceph-mon[38805]: pgmap v2024: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:05:36.345 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3130 8 365281 365273 0.911652 0 - 0.0299513 2026-02-23T14:05:36.751 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:05:36.867 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:05:36.966 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:05:36.964+0000 7f1a63894640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:05:36.966 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:05:36.964+0000 7f1a63894640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:05:37.076 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:05:37.076 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":218,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:05:10.676625+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:01:14.553379+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:03:48.616954+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:04:35.751423+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:05:37.345 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3131 8 365281 365273 0.911361 0 - 0.0299513 2026-02-23T14:05:38.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:05:37 trial068 ceph-mon[38805]: pgmap v2025: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:05:38.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:05:37 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3739529819' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:05:38.345 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3132 8 365281 365273 0.91107 0 - 0.0299513 2026-02-23T14:05:39.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:05:38 trial068 ceph-mon[38805]: 5.7 scrub starts 2026-02-23T14:05:39.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:05:38 trial068 ceph-mon[38805]: 5.7 scrub ok 2026-02-23T14:05:39.345 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3133 8 365281 365273 0.910779 0 - 0.0299513 2026-02-23T14:05:40.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:05:39 trial068 ceph-mon[38805]: pgmap v2026: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:05:40.113 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:05:40.232 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:05:40.330 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:05:40.328+0000 7f060a09e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:05:40.330 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:05:40.328+0000 7f060a09e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:05:40.345 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3134 8 365281 365273 0.910488 0 - 0.0299513 2026-02-23T14:05:40.445 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:05:40.445 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":218,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:05:10.676625+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:01:14.553379+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:03:48.616954+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:04:35.751423+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:05:41.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:05:40 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/656764660' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:05:41.345 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3135 8 365281 365273 0.910198 0 - 0.0299513 2026-02-23T14:05:42.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:05:41 trial068 ceph-mon[38805]: pgmap v2027: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:05:42.345 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3136 8 365281 365273 0.909908 0 - 0.0299513 2026-02-23T14:05:43.345 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3137 8 365281 365273 0.909618 0 - 0.0299513 2026-02-23T14:05:43.481 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:05:43.600 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:05:43.699 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:05:43.697+0000 7fbb43fc7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:05:43.699 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:05:43.697+0000 7fbb43fc7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:05:43.808 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:05:43.808 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":218,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:05:10.676625+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:01:14.553379+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:03:48.616954+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:04:35.751423+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:05:43.809 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:05:43 trial068 ceph-mon[38805]: pgmap v2028: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:05:44.094 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:05:43 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:05:43] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:05:44.345 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3138 8 365281 365273 0.909328 0 - 0.0299513 2026-02-23T14:05:44.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:05:43 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:05:43.974Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:05:45.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:05:44 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1229128898' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:05:45.345 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3139 8 365281 365273 0.909038 0 - 0.0299513 2026-02-23T14:05:46.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:05:45 trial068 ceph-mon[38805]: pgmap v2029: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:05:46.345 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T14:05:46.345175+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T14:05:46.346 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T14:05:46.346 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3140 8 365281 365273 0.908749 0 - 0.0299513 2026-02-23T14:05:46.845 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:05:46.963 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:05:47.061 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:05:47.059+0000 7ffbf7869640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:05:47.062 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:05:47.059+0000 7ffbf7869640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:05:47.170 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:05:47.170 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":218,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:05:10.676625+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:01:14.553379+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:03:48.616954+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:04:35.751423+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:05:47.346 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3141 8 365281 365273 0.908459 0 - 0.0299513 2026-02-23T14:05:48.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:05:47 trial068 ceph-mon[38805]: pgmap v2030: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:05:48.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:05:47 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2016628054' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:05:48.346 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3142 8 365281 365273 0.90817 0 - 0.0299513 2026-02-23T14:05:49.346 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3143 8 365281 365273 0.907881 0 - 0.0299513 2026-02-23T14:05:50.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:05:49 trial068 ceph-mon[38805]: pgmap v2031: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:05:50.208 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:05:50.325 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:05:50.346 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3144 8 365281 365273 0.907592 0 - 0.0299513 2026-02-23T14:05:50.424 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:05:50.422+0000 7f049380c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:05:50.424 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:05:50.422+0000 7f049380c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:05:50.533 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:05:50.534 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":218,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:05:10.676625+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:01:14.553379+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:03:48.616954+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:04:35.751423+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:05:50.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:05:50 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2026-02-23T14:05:50.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:05:50 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T14:05:50.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:05:50 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T14:05:50.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:05:50 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-23T14:05:50.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:05:50 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2026-02-23T14:05:50.845 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:05:50 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T14:05:50.845 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:05:50 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2026-02-23T14:05:50.845 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:05:50 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1167918628' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:05:51.346 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3145 8 365281 365273 0.907304 0 - 0.0299513 2026-02-23T14:05:52.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:05:51 trial068 ceph-mon[38805]: pgmap v2032: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:05:52.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:05:51 trial068 ceph-mon[38805]: pgmap v2033: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:05:52.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:05:51 trial068 ceph-mon[38805]: pgmap v2034: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:05:52.346 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3146 8 365281 365273 0.907015 0 - 0.0299513 2026-02-23T14:05:53.346 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3147 8 365281 365273 0.906727 0 - 0.0299513 2026-02-23T14:05:53.571 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:05:53.689 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:05:53.791 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:05:53.789+0000 7f17db29d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:05:53.791 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:05:53.789+0000 7f17db29d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:05:53.900 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:05:53.901 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":218,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:05:10.676625+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:01:14.553379+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:03:48.616954+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:04:35.751423+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:05:53.901 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:05:53 trial068 ceph-mon[38805]: pgmap v2035: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:05:54.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:05:53 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:05:53] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:05:54.346 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3148 8 365281 365273 0.906439 0 - 0.0299513 2026-02-23T14:05:54.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:05:53 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:05:53.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:05:55.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:05:54 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2985009247' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:05:55.346 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3149 8 365281 365273 0.906151 0 - 0.0299513 2026-02-23T14:05:56.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:05:55 trial068 ceph-mon[38805]: pgmap v2036: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:05:56.346 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3150 8 365281 365273 0.905864 0 - 0.0299513 2026-02-23T14:05:56.938 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:05:56.995 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:05:56 trial068 ceph-mon[38805]: 5.8 scrub starts 2026-02-23T14:05:56.995 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:05:56 trial068 ceph-mon[38805]: 5.8 scrub ok 2026-02-23T14:05:57.058 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:05:57.160 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:05:57.157+0000 7f854035e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:05:57.160 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:05:57.158+0000 7f854035e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:05:57.269 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:05:57.269 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":218,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:05:10.676625+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:01:14.553379+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:03:48.616954+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:04:35.751423+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:05:57.346 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3151 8 365281 365273 0.905576 0 - 0.0299513 2026-02-23T14:05:58.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:05:57 trial068 ceph-mon[38805]: pgmap v2037: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:05:58.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:05:57 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1222744921' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:05:58.346 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3152 8 365281 365273 0.905289 0 - 0.0299513 2026-02-23T14:05:59.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:05:58 trial068 ceph-mon[38805]: 5.e scrub starts 2026-02-23T14:05:59.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:05:58 trial068 ceph-mon[38805]: 5.e scrub ok 2026-02-23T14:05:59.346 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3153 8 365281 365273 0.905002 0 - 0.0299513 2026-02-23T14:06:00.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:05:59 trial068 ceph-mon[38805]: pgmap v2038: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:06:00.307 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:06:00.346 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3154 8 365281 365273 0.904715 0 - 0.0299513 2026-02-23T14:06:00.423 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:06:00.524 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:06:00.522+0000 7f9f73afa640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:06:00.524 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:06:00.522+0000 7f9f73afa640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:06:00.632 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:06:00.632 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":218,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:05:10.676625+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:01:14.553379+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:03:48.616954+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:04:35.751423+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:06:01.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:06:00 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1703619619' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:06:01.347 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3155 8 365281 365273 0.904428 0 - 0.0299513 2026-02-23T14:06:02.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:06:01 trial068 ceph-mon[38805]: pgmap v2039: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:06:02.347 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3156 8 365281 365273 0.904142 0 - 0.0299513 2026-02-23T14:06:03.347 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3157 8 365281 365273 0.903855 0 - 0.0299513 2026-02-23T14:06:03.669 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:06:03.790 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:06:03.890 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:06:03.888+0000 7f8aaabf0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:06:03.891 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:06:03.888+0000 7f8aaabf0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:06:03.974 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:06:03 trial068 ceph-mon[38805]: pgmap v2040: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:06:03.999 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:06:04.000 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":218,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:05:10.676625+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:01:14.553379+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:03:48.616954+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:04:35.751423+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:06:04.266 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:06:03 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:06:03.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:06:04.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:06:03 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:06:03] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:06:04.347 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3158 8 365281 365273 0.903569 0 - 0.0299513 2026-02-23T14:06:05.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:06:04 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/917314475' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:06:05.347 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3159 8 365281 365273 0.903283 0 - 0.0299513 2026-02-23T14:06:06.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:06:05 trial068 ceph-mon[38805]: pgmap v2041: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:06:06.347 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T14:06:06.346523+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T14:06:06.347 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T14:06:06.347 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3160 8 365281 365273 0.902997 0 - 0.0299513 2026-02-23T14:06:07.037 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:06:07.154 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:06:07.260 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:06:07.258+0000 7f3ccf608640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:06:07.261 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:06:07.258+0000 7f3ccf608640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:06:07.347 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3161 8 365281 365273 0.902711 0 - 0.0299513 2026-02-23T14:06:07.369 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:06:07.369 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":218,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:05:10.676625+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:01:14.553379+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:03:48.616954+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:04:35.751423+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:06:08.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:06:07 trial068 ceph-mon[38805]: pgmap v2042: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:06:08.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:06:07 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/172017898' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:06:08.347 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3162 8 365281 365273 0.902426 0 - 0.0299513 2026-02-23T14:06:09.347 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3163 8 365281 365273 0.902141 0 - 0.0299513 2026-02-23T14:06:10.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:06:09 trial068 ceph-mon[38805]: pgmap v2043: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:06:10.347 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3164 8 365281 365273 0.901856 0 - 0.0299513 2026-02-23T14:06:10.423 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:06:10.540 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:06:10.647 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:06:10.645+0000 7f5d69bba640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:06:10.647 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:06:10.645+0000 7f5d69bba640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:06:10.756 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:06:10.756 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":218,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:05:10.676625+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:01:14.553379+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:03:48.616954+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:04:35.751423+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:06:11.347 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3165 8 365281 365273 0.901571 0 - 0.0299513 2026-02-23T14:06:12.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:06:11 trial068 ceph-mon[38805]: pgmap v2044: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:06:12.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:06:11 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2083411207' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:06:12.347 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3166 8 365281 365273 0.901286 0 - 0.0299513 2026-02-23T14:06:13.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:06:12 trial068 ceph-mon[38805]: 5.a scrub starts 2026-02-23T14:06:13.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:06:12 trial068 ceph-mon[38805]: 5.a scrub ok 2026-02-23T14:06:13.347 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3167 8 365281 365273 0.901001 0 - 0.0299513 2026-02-23T14:06:13.794 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:06:13.912 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:06:14.009 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:06:13 trial068 ceph-mon[38805]: pgmap v2045: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:06:14.010 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:06:13 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:06:13] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:06:14.010 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:06:14.007+0000 7fa4565d7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:06:14.010 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:06:14.007+0000 7fa4565d7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:06:14.118 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:06:14.118 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":218,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:05:10.676625+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:01:14.553379+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:03:48.616954+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:04:35.751423+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:06:14.347 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3168 8 365281 365273 0.900717 0 - 0.0299513 2026-02-23T14:06:14.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:06:13 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:06:13.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:06:15.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:06:14 trial068 ceph-mon[38805]: pgmap v2046: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:06:15.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:06:14 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3673610496' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:06:15.348 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3169 8 365281 365273 0.900433 0 - 0.0299513 2026-02-23T14:06:16.348 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3170 8 365281 365273 0.900149 0 - 0.0299513 2026-02-23T14:06:17.156 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:06:17.274 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:06:17.348 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3171 8 365281 365273 0.899865 0 - 0.0299513 2026-02-23T14:06:17.383 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:06:17.382+0000 7f9f877ec640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:06:17.384 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:06:17.382+0000 7f9f877ec640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:06:17.494 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:06:17.494 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":218,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:05:10.676625+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:01:14.553379+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:03:48.616954+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:04:35.751423+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:06:17.494 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:06:17 trial068 ceph-mon[38805]: pgmap v2047: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:06:18.348 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3172 8 365281 365273 0.899581 0 - 0.0299513 2026-02-23T14:06:18.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:06:18 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2717882767' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:06:19.348 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3173 8 365281 365273 0.899297 0 - 0.0299513 2026-02-23T14:06:19.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:06:19 trial068 ceph-mon[38805]: pgmap v2048: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:06:20.348 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3174 8 365281 365273 0.899014 0 - 0.0299513 2026-02-23T14:06:20.532 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:06:20.557 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:06:20 trial068 ceph-mon[38805]: osdmap e219: 4 total, 3 up, 4 in 2026-02-23T14:06:20.650 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:06:20.747 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:06:20.745+0000 7f197abbc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:06:20.747 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:06:20.745+0000 7f197abbc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:06:20.855 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:06:20.856 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":219,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:06:19.146565+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:05:13.564547+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:03:48.616954+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:04:35.751423+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:06:21.348 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3175 8 365281 365273 0.898731 0 - 0.0299513 2026-02-23T14:06:21.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:06:21 trial068 ceph-mon[38805]: pgmap v2050: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:06:21.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:06:21 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1415529107' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:06:22.348 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3176 8 365281 365273 0.898448 0 - 0.0299513 2026-02-23T14:06:23.348 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3177 8 365281 365273 0.898165 0 - 0.0299513 2026-02-23T14:06:23.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:06:23 trial068 ceph-mon[38805]: pgmap v2051: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:06:23.893 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:06:24.011 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:06:24.118 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:06:24.115+0000 7f801ab33640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:06:24.118 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:06:24.116+0000 7f801ab33640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:06:24.226 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:06:24.227 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":219,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:06:19.146565+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:05:13.564547+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:03:48.616954+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:04:35.751423+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:06:24.227 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:06:23 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:06:23] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:06:24.348 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3178 8 365281 365273 0.897883 0 - 0.0299513 2026-02-23T14:06:24.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:06:23 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:06:23.972Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:06:24.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:06:24 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/434123723' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:06:25.348 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3179 8 365281 365273 0.8976 0 - 0.0299513 2026-02-23T14:06:25.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:06:25 trial068 ceph-mon[38805]: pgmap v2052: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:06:26.348 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T14:06:26.347976+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T14:06:26.349 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T14:06:26.349 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3180 8 365281 365273 0.897318 0 - 0.0299513 2026-02-23T14:06:26.745 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:06:26 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T14:06:26.745 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:06:26 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T14:06:26.746 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:06:26 trial068 ceph-mon[38805]: 5.5 scrub starts 2026-02-23T14:06:26.746 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:06:26 trial068 ceph-mon[38805]: 5.5 scrub ok 2026-02-23T14:06:27.264 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:06:27.349 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3181 8 365281 365273 0.897036 0 - 0.0299513 2026-02-23T14:06:27.382 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:06:27.483 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:06:27.481+0000 7f2e425b3640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:06:27.483 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:06:27.481+0000 7f2e425b3640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:06:27.593 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:06:27.593 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":219,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:06:19.146565+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:05:13.564547+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:03:48.616954+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:04:35.751423+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:06:27.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:06:27 trial068 ceph-mon[38805]: pgmap v2053: 17 pgs: 1 active+clean+scrubbing, 1 active+recovery_unfound+undersized+degraded, 4 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:06:27.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:06:27 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T14:06:27.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:06:27 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T14:06:28.348 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3182 8 365281 365273 0.896754 0 - 0.0299513 2026-02-23T14:06:28.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:06:28 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3829360258' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:06:29.348 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3183 8 365281 365273 0.896472 0 - 0.0299513 2026-02-23T14:06:29.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:06:29 trial068 ceph-mon[38805]: pgmap v2054: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 1 active+clean+scrubbing, 4 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:06:30.349 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3184 8 365281 365273 0.896191 0 - 0.0299513 2026-02-23T14:06:30.630 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:06:30.748 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:06:30.846 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:06:30.843+0000 7ffa15863640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:06:30.846 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:06:30.844+0000 7ffa15863640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:06:30.954 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:06:30.955 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":219,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:06:19.146565+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:05:13.564547+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:03:48.616954+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:04:35.751423+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:06:31.349 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3185 8 365281 365273 0.895909 0 - 0.0299513 2026-02-23T14:06:31.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:06:31 trial068 ceph-mon[38805]: pgmap v2055: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 1 active+clean+scrubbing, 4 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:06:31.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:06:31 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2267334665' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:06:32.349 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3186 8 365281 365273 0.895628 0 - 0.0299513 2026-02-23T14:06:33.349 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3187 8 365281 365273 0.895347 0 - 0.0299513 2026-02-23T14:06:33.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:06:33 trial068 ceph-mon[38805]: pgmap v2056: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:06:33.992 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:06:34.108 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:06:34.205 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:06:34.203+0000 7f46810f6640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:06:34.205 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:06:34.203+0000 7f46810f6640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:06:34.266 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:06:33 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:06:33.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:06:34.316 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:06:34.316 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":219,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:06:19.146565+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:05:13.564547+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:03:48.616954+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:04:35.751423+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:06:34.317 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:06:33 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:06:33] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:06:34.349 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3188 8 365281 365273 0.895066 0 - 0.0299513 2026-02-23T14:06:34.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:06:34 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3146612056' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:06:35.349 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3189 8 365281 365273 0.894786 0 - 0.0299513 2026-02-23T14:06:35.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:06:35 trial068 ceph-mon[38805]: pgmap v2057: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:06:36.349 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3190 8 365281 365273 0.894505 0 - 0.0299513 2026-02-23T14:06:37.349 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3191 8 365281 365273 0.894225 0 - 0.0299513 2026-02-23T14:06:37.354 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:06:37.470 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:06:37.569 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:06:37.567+0000 7ff986443640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:06:37.569 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:06:37.567+0000 7ff986443640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:06:37.677 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:06:37.677 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":219,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:06:19.146565+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:05:13.564547+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:03:48.616954+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:04:35.751423+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:06:37.677 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:06:37 trial068 ceph-mon[38805]: pgmap v2058: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:06:38.349 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3192 8 365281 365273 0.893945 0 - 0.0299513 2026-02-23T14:06:38.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:06:38 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1122756689' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:06:39.349 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3193 8 365281 365273 0.893665 0 - 0.0299513 2026-02-23T14:06:39.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:06:39 trial068 ceph-mon[38805]: pgmap v2059: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:06:40.349 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3194 8 365281 365273 0.893385 0 - 0.0299513 2026-02-23T14:06:40.714 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:06:40.831 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:06:40.928 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:06:40.926+0000 7f56aedf3640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:06:40.928 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:06:40.926+0000 7f56aedf3640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:06:41.037 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:06:41.037 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":219,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:06:19.146565+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:05:13.564547+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:03:48.616954+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:04:35.751423+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:06:41.349 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3195 8 365281 365273 0.893105 0 - 0.0299513 2026-02-23T14:06:41.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:06:41 trial068 ceph-mon[38805]: pgmap v2060: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:06:41.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:06:41 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3437491386' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:06:42.349 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3196 8 365281 365273 0.892826 0 - 0.0299513 2026-02-23T14:06:43.349 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3197 8 365281 365273 0.892546 0 - 0.0299513 2026-02-23T14:06:43.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:06:43 trial068 ceph-mon[38805]: pgmap v2061: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:06:44.076 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:06:44.193 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:06:44.238 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:06:43 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:06:43] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:06:44.295 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:06:44.293+0000 7f8a52cd5640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:06:44.296 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:06:44.293+0000 7f8a52cd5640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:06:44.349 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3198 8 365281 365273 0.892267 0 - 0.0299513 2026-02-23T14:06:44.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:06:43 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:06:43.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:06:44.403 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:06:44.403 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":219,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:06:19.146565+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:05:13.564547+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:03:48.616954+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:04:35.751423+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:06:44.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:06:44 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1653877975' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:06:45.350 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3199 8 365281 365273 0.891988 0 - 0.0299513 2026-02-23T14:06:45.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:06:45 trial068 ceph-mon[38805]: pgmap v2062: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:06:46.350 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T14:06:46.349367+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T14:06:46.350 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T14:06:46.350 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3200 8 365281 365273 0.89171 0 - 0.0299513 2026-02-23T14:06:47.350 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3201 8 365281 365273 0.891431 0 - 0.0299513 2026-02-23T14:06:47.440 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:06:47.560 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:06:47.664 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:06:47.662+0000 7f528a30d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:06:47.664 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:06:47.662+0000 7f528a30d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:06:47.774 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:06:47.774 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":219,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:06:19.146565+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:05:13.564547+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:03:48.616954+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:04:35.751423+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:06:47.775 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:06:47 trial068 ceph-mon[38805]: pgmap v2063: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:06:48.350 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3202 8 365281 365273 0.891153 0 - 0.0299513 2026-02-23T14:06:48.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:06:48 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2796542239' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:06:49.350 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3203 8 365281 365273 0.890875 0 - 0.0299513 2026-02-23T14:06:49.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:06:49 trial068 ceph-mon[38805]: pgmap v2064: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:06:50.350 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3204 8 365281 365273 0.890596 0 - 0.0299513 2026-02-23T14:06:50.812 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:06:50.835 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:06:50 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2026-02-23T14:06:50.835 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:06:50 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T14:06:50.835 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:06:50 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T14:06:50.836 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:06:50 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T14:06:50.836 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:06:50 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T14:06:50.928 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:06:51.028 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:06:51.026+0000 7f0490eb9640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:06:51.028 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:06:51.026+0000 7f0490eb9640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:06:51.136 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:06:51.136 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":219,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:06:19.146565+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:05:13.564547+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:03:48.616954+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:04:35.751423+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:06:51.350 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3205 8 365281 365273 0.890319 0 - 0.0299513 2026-02-23T14:06:51.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:06:51 trial068 ceph-mon[38805]: pgmap v2065: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:06:51.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:06:51 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-23T14:06:51.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:06:51 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2026-02-23T14:06:51.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:06:51 trial068 ceph-mon[38805]: pgmap v2066: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:06:51.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:06:51 trial068 ceph-mon[38805]: pgmap v2067: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:06:51.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:06:51 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T14:06:51.845 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:06:51 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2026-02-23T14:06:51.845 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:06:51 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3078645170' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:06:52.350 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3206 8 365281 365273 0.890041 0 - 0.0299513 2026-02-23T14:06:53.350 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3207 8 365281 365273 0.889763 0 - 0.0299513 2026-02-23T14:06:53.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:06:53 trial068 ceph-mon[38805]: pgmap v2068: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:06:54.175 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:06:54.293 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:06:54.316 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:06:53 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:06:53] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:06:54.350 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3208 8 365281 365273 0.889486 0 - 0.0299513 2026-02-23T14:06:54.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:06:53 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:06:53.974Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:06:54.394 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:06:54.391+0000 7f50542d0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:06:54.394 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:06:54.392+0000 7f50542d0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:06:54.503 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:06:54.503 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":219,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:06:19.146565+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:05:13.564547+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:03:48.616954+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:04:35.751423+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:06:55.350 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3209 8 365281 365273 0.889209 0 - 0.0299513 2026-02-23T14:06:55.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:06:55 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/553561588' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:06:55.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:06:55 trial068 ceph-mon[38805]: pgmap v2069: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:06:56.350 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3210 8 365281 365273 0.888932 0 - 0.0299513 2026-02-23T14:06:57.351 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3211 8 365281 365273 0.888655 0 - 0.0299513 2026-02-23T14:06:57.542 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:06:57.659 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:06:57.756 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:06:57.754+0000 7f2d7bbaf640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:06:57.756 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:06:57.754+0000 7f2d7bbaf640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:06:57.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:06:57 trial068 ceph-mon[38805]: pgmap v2070: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:06:57.867 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:06:57.867 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":219,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:06:19.146565+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:05:13.564547+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:03:48.616954+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:04:35.751423+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:06:58.351 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3212 8 365281 365273 0.888378 0 - 0.0299513 2026-02-23T14:06:58.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:06:58 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/4011905056' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:06:59.351 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3213 8 365281 365273 0.888102 0 - 0.0299513 2026-02-23T14:06:59.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:06:59 trial068 ceph-mon[38805]: pgmap v2071: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:07:00.351 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3214 8 365281 365273 0.887825 0 - 0.0299513 2026-02-23T14:07:00.918 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:07:01.035 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:07:01.136 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:07:01.134+0000 7fc13e262640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:07:01.136 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:07:01.134+0000 7fc13e262640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:07:01.244 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:07:01.244 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":219,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:06:19.146565+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:05:13.564547+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:03:48.616954+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:04:35.751423+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:07:01.351 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3215 8 365281 365273 0.887549 0 - 0.0299513 2026-02-23T14:07:01.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:07:01 trial068 ceph-mon[38805]: pgmap v2072: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:07:01.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:07:01 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1780973738' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:07:02.351 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3216 8 365281 365273 0.887273 0 - 0.0299513 2026-02-23T14:07:02.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:07:02 trial068 ceph-mon[38805]: 5.7 scrub starts 2026-02-23T14:07:03.351 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3217 8 365281 365273 0.886998 0 - 0.0299513 2026-02-23T14:07:03.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:07:03 trial068 ceph-mon[38805]: 5.7 scrub ok 2026-02-23T14:07:03.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:07:03 trial068 ceph-mon[38805]: pgmap v2073: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:07:04.266 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:07:03 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:07:03.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:07:04.281 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:07:04.304 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:07:03 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:07:03] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:07:04.351 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3218 8 365281 365273 0.886722 0 - 0.0299513 2026-02-23T14:07:04.397 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:07:04.497 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:07:04.495+0000 7f94ddd8e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:07:04.498 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:07:04.496+0000 7f94ddd8e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:07:04.606 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:07:04.606 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":219,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:06:19.146565+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:05:13.564547+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:03:48.616954+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:04:35.751423+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:07:05.351 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3219 8 365281 365273 0.886446 0 - 0.0299513 2026-02-23T14:07:05.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:07:05 trial068 ceph-mon[38805]: pgmap v2074: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:07:05.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:07:05 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/4234391152' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:07:06.351 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T14:07:06.350835+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T14:07:06.351 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T14:07:06.352 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3220 8 365281 365273 0.886171 0 - 0.0299513 2026-02-23T14:07:07.351 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3221 8 365281 365273 0.885896 0 - 0.0299513 2026-02-23T14:07:07.643 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:07:07.760 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:07:07.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:07:07 trial068 ceph-mon[38805]: pgmap v2075: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:07:07.863 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:07:07.861+0000 7fa69065e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:07:07.863 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:07:07.861+0000 7fa69065e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:07:07.977 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:07:07.977 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":219,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:06:19.146565+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:05:13.564547+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:03:48.616954+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:04:35.751423+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:07:08.351 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3222 8 365281 365273 0.885621 0 - 0.0299513 2026-02-23T14:07:08.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:07:08 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1801476848' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:07:09.351 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3223 8 365281 365273 0.885346 0 - 0.0299513 2026-02-23T14:07:09.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:07:09 trial068 ceph-mon[38805]: pgmap v2076: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:07:10.351 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3224 8 365281 365273 0.885072 0 - 0.0299513 2026-02-23T14:07:11.016 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:07:11.133 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:07:11.232 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:07:11.229+0000 7fbc87a84640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:07:11.232 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:07:11.230+0000 7fbc87a84640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:07:11.344 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:07:11.345 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":219,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:06:19.146565+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:05:13.564547+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:03:48.616954+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:04:35.751423+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:07:11.351 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3225 8 365281 365273 0.884797 0 - 0.0299513 2026-02-23T14:07:11.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:07:11 trial068 ceph-mon[38805]: pgmap v2077: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:07:11.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:07:11 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/67409406' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:07:12.352 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3226 8 365281 365273 0.884523 0 - 0.0299513 2026-02-23T14:07:13.352 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3227 8 365281 365273 0.884249 0 - 0.0299513 2026-02-23T14:07:13.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:07:13 trial068 ceph-mon[38805]: pgmap v2078: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:07:14.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:07:13 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:07:13] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:07:14.352 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3228 8 365281 365273 0.883975 0 - 0.0299513 2026-02-23T14:07:14.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:07:13 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:07:13.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:07:14.382 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:07:14.501 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:07:14.606 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:07:14.604+0000 7f3993cd9640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:07:14.607 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:07:14.605+0000 7f3993cd9640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:07:14.716 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:07:14.716 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":219,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:06:19.146565+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:05:13.564547+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:03:48.616954+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:04:35.751423+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:07:15.352 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3229 8 365281 365273 0.883701 0 - 0.0299513 2026-02-23T14:07:15.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:07:15 trial068 ceph-mon[38805]: pgmap v2079: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:07:15.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:07:15 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1139501871' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:07:16.352 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3230 8 365281 365273 0.883428 0 - 0.0299513 2026-02-23T14:07:17.352 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3231 8 365281 365273 0.883154 0 - 0.0299513 2026-02-23T14:07:17.769 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:07:17.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:07:17 trial068 ceph-mon[38805]: pgmap v2080: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:07:17.887 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:07:17.983 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:07:17.981+0000 7fb5323c7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:07:17.984 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:07:17.981+0000 7fb5323c7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:07:18.094 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:07:18.094 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":219,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:06:19.146565+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:05:13.564547+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:03:48.616954+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:04:35.751423+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:07:18.352 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3232 8 365281 365273 0.882881 0 - 0.0299513 2026-02-23T14:07:18.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:07:18 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3933281758' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:07:19.352 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3233 8 365281 365273 0.882608 0 - 0.0299513 2026-02-23T14:07:19.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:07:19 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T14:07:19.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:07:19 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T14:07:19.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:07:19 trial068 ceph-mon[38805]: pgmap v2081: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:07:20.352 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3234 8 365281 365273 0.882335 0 - 0.0299513 2026-02-23T14:07:21.132 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:07:21.248 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:07:21.347 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:07:21.345+0000 7ff022a45640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:07:21.348 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:07:21.345+0000 7ff022a45640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:07:21.352 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3235 8 365281 365273 0.882062 0 - 0.0299513 2026-02-23T14:07:21.456 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:07:21.457 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":219,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:06:19.146565+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:05:13.564547+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:03:48.616954+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:04:35.751423+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:07:21.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:07:21 trial068 ceph-mon[38805]: pgmap v2082: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:07:21.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:07:21 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3419546630' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:07:22.352 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3236 8 365281 365273 0.88179 0 - 0.0299513 2026-02-23T14:07:23.352 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3237 8 365281 365273 0.881517 0 - 0.0299513 2026-02-23T14:07:23.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:07:23 trial068 ceph-mon[38805]: pgmap v2083: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:07:24.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:07:23 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:07:23] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:07:24.352 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3238 8 365281 365273 0.881245 0 - 0.0299513 2026-02-23T14:07:24.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:07:23 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:07:23.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:07:24.493 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:07:24.610 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:07:24.709 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:07:24.706+0000 7ff253a1e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:07:24.709 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:07:24.707+0000 7ff253a1e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:07:24.817 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:07:24.818 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":219,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:06:19.146565+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:05:13.564547+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:03:48.616954+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:04:35.751423+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:07:24.818 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:07:24 trial068 ceph-mon[38805]: 5.a scrub starts 2026-02-23T14:07:24.818 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:07:24 trial068 ceph-mon[38805]: 5.a scrub ok 2026-02-23T14:07:25.353 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3239 8 365281 365273 0.880973 0 - 0.0299513 2026-02-23T14:07:25.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:07:25 trial068 ceph-mon[38805]: pgmap v2084: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:07:25.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:07:25 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3287976058' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:07:26.353 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T14:07:26.352316+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T14:07:26.353 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T14:07:26.353 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3240 8 365281 365273 0.880701 0 - 0.0299513 2026-02-23T14:07:27.353 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3241 8 365281 365273 0.880429 0 - 0.0299513 2026-02-23T14:07:27.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:07:27 trial068 ceph-mon[38805]: pgmap v2085: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:07:27.855 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:07:27.976 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:07:28.084 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:07:28.082+0000 7fef689a0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:07:28.084 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:07:28.082+0000 7fef689a0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:07:28.194 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:07:28.195 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":219,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:06:19.146565+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:05:13.564547+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:03:48.616954+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:04:35.751423+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:07:28.353 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3242 8 365281 365273 0.880158 0 - 0.0299513 2026-02-23T14:07:28.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:07:28 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1115411223' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:07:29.353 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3243 8 365281 365273 0.879886 0 - 0.0299513 2026-02-23T14:07:29.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:07:29 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T14:07:29.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:07:29 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T14:07:29.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:07:29 trial068 ceph-mon[38805]: 5.8 scrub starts 2026-02-23T14:07:29.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:07:29 trial068 ceph-mon[38805]: pgmap v2086: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:07:29.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:07:29 trial068 ceph-mon[38805]: 5.8 scrub ok 2026-02-23T14:07:30.353 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3244 8 365281 365273 0.879615 0 - 0.0299513 2026-02-23T14:07:31.233 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:07:31.353 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:07:31.353 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3245 8 365281 365273 0.879344 0 - 0.0299513 2026-02-23T14:07:31.454 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:07:31.451+0000 7f357bb64640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:07:31.454 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:07:31.452+0000 7f357bb64640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:07:31.562 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:07:31.562 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":219,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:06:19.146565+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:05:13.564547+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:03:48.616954+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:04:35.751423+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:07:31.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:07:31 trial068 ceph-mon[38805]: pgmap v2087: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:07:31.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:07:31 trial068 ceph-mon[38805]: 5.e scrub starts 2026-02-23T14:07:31.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:07:31 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3907917888' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:07:32.353 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3246 8 365281 365273 0.879073 0 - 0.0299513 2026-02-23T14:07:32.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:07:32 trial068 ceph-mon[38805]: 5.e scrub ok 2026-02-23T14:07:33.353 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3247 8 365281 365273 0.878802 0 - 0.0299513 2026-02-23T14:07:33.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:07:33 trial068 ceph-mon[38805]: pgmap v2088: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:07:34.266 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:07:33 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:07:33.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:07:34.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:07:33 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:07:33] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:07:34.353 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3248 8 365281 365273 0.878532 0 - 0.0299513 2026-02-23T14:07:34.599 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:07:34.717 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:07:34.817 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:07:34.815+0000 7f909dc5b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:07:34.817 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:07:34.815+0000 7f909dc5b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:07:34.926 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:07:34.926 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":219,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:06:19.146565+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:05:13.564547+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:03:48.616954+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:04:35.751423+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:07:35.353 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3249 8 365281 365273 0.878261 0 - 0.0299513 2026-02-23T14:07:35.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:07:35 trial068 ceph-mon[38805]: pgmap v2089: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:07:35.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:07:35 trial068 ceph-mon[38805]: 5.5 scrub starts 2026-02-23T14:07:35.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:07:35 trial068 ceph-mon[38805]: 5.5 scrub ok 2026-02-23T14:07:35.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:07:35 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2563053962' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:07:36.353 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3250 8 365281 365273 0.877991 0 - 0.0299513 2026-02-23T14:07:37.354 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3251 8 365281 365273 0.877721 0 - 0.0299513 2026-02-23T14:07:37.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:07:37 trial068 ceph-mon[38805]: pgmap v2090: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:07:37.977 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:07:38.096 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:07:38.201 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:07:38.199+0000 7f72bf8cc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:07:38.202 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:07:38.200+0000 7f72bf8cc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:07:38.313 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:07:38.313 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":219,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:06:19.146565+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:05:13.564547+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:03:48.616954+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:04:35.751423+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:07:38.353 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3252 8 365281 365273 0.877451 0 - 0.0299513 2026-02-23T14:07:38.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:07:38 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1752742854' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:07:39.354 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3253 8 365281 365273 0.877181 0 - 0.0299513 2026-02-23T14:07:39.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:07:39 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T14:07:39.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:07:39 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T14:07:39.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:07:39 trial068 ceph-mon[38805]: pgmap v2091: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:07:40.354 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3254 8 365281 365273 0.876912 0 - 0.0299513 2026-02-23T14:07:41.351 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:07:41.355 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3255 8 365281 365273 0.876642 0 - 0.0299513 2026-02-23T14:07:41.470 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:07:41.574 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:07:41.572+0000 7fef1aeab640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:07:41.574 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:07:41.572+0000 7fef1aeab640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:07:41.685 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:07:41.685 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":219,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:06:19.146565+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:05:13.564547+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:03:48.616954+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:04:35.751423+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:07:41.686 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:07:41 trial068 ceph-mon[38805]: pgmap v2092: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:07:42.354 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3256 8 365281 365273 0.876373 0 - 0.0299513 2026-02-23T14:07:42.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:07:42 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2692563749' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:07:43.354 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3257 8 365281 365273 0.876104 0 - 0.0299513 2026-02-23T14:07:43.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:07:43 trial068 ceph-mon[38805]: pgmap v2093: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:07:44.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:07:43 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:07:43] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:07:44.354 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3258 8 365281 365273 0.875835 0 - 0.0299513 2026-02-23T14:07:44.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:07:43 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:07:43.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:07:44.724 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:07:44.843 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:07:44.944 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:07:44.941+0000 7fb5d2c64640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:07:44.944 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:07:44.942+0000 7fb5d2c64640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:07:45.053 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:07:45.053 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":219,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:06:19.146565+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:05:13.564547+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:03:48.616954+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:04:35.751423+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:07:45.354 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3259 8 365281 365273 0.875567 0 - 0.0299513 2026-02-23T14:07:45.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:07:45 trial068 ceph-mon[38805]: pgmap v2094: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:07:45.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:07:45 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2235603957' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:07:46.354 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T14:07:46.353843+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T14:07:46.354 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T14:07:46.355 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3260 8 365281 365273 0.875298 0 - 0.0299513 2026-02-23T14:07:47.354 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3261 8 365281 365273 0.87503 0 - 0.0299513 2026-02-23T14:07:47.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:07:47 trial068 ceph-mon[38805]: pgmap v2095: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:07:48.091 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:07:48.208 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:07:48.308 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:07:48.306+0000 7fe7f0b93640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:07:48.308 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:07:48.306+0000 7fe7f0b93640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:07:48.354 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3262 8 365281 365273 0.874761 0 - 0.0299513 2026-02-23T14:07:48.416 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:07:48.416 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":219,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:06:19.146565+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:05:13.564547+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:03:48.616954+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:04:35.751423+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:07:48.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:07:48 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1796902283' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:07:49.354 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3263 8 365281 365273 0.874493 0 - 0.0299513 2026-02-23T14:07:50.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:07:49 trial068 ceph-mon[38805]: pgmap v2096: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:07:50.355 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3264 8 365281 365273 0.874225 0 - 0.0299513 2026-02-23T14:07:51.005 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:07:50 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2026-02-23T14:07:51.355 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3265 8 365281 365273 0.873958 0 - 0.0299513 2026-02-23T14:07:51.455 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:07:51.573 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:07:51.683 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:07:51.681+0000 7fd8bc71c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:07:51.684 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:07:51.681+0000 7fd8bc71c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:07:51.742 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:07:51 trial068 ceph-mon[38805]: pgmap v2097: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:07:51.742 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:07:51 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T14:07:51.742 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:07:51 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T14:07:51.742 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:07:51 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T14:07:51.742 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:07:51 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T14:07:51.742 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:07:51 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T14:07:51.742 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:07:51 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T14:07:51.743 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:07:51 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T14:07:51.743 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:07:51 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T14:07:51.795 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:07:51.796 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":219,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:06:19.146565+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:05:13.564547+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:03:48.616954+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:04:35.751423+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:07:52.355 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3266 8 365281 365273 0.87369 0 - 0.0299513 2026-02-23T14:07:52.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:07:52 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2279997919' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:07:52.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:07:52 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T14:07:52.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:07:52 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T14:07:52.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:07:52 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T14:07:52.845 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:07:52 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T14:07:52.845 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:07:52 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-23T14:07:52.845 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:07:52 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2026-02-23T14:07:52.845 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:07:52 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T14:07:52.845 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:07:52 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2026-02-23T14:07:53.355 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3267 8 365281 365273 0.873423 0 - 0.0299513 2026-02-23T14:07:53.974 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:07:53 trial068 ceph-mon[38805]: pgmap v2098: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:07:53.974 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:07:53 trial068 ceph-mon[38805]: pgmap v2099: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:07:54.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:07:53 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:07:53] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:07:54.355 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3268 8 365281 365273 0.873155 0 - 0.0299513 2026-02-23T14:07:54.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:07:53 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:07:53.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:07:54.835 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:07:54.953 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:07:55.055 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:07:55.053+0000 7f79bf76f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:07:55.055 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:07:55.053+0000 7f79bf76f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:07:55.165 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:07:55.166 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":219,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:06:19.146565+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:05:13.564547+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:03:48.616954+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:04:35.751423+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:07:55.355 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3269 8 365281 365273 0.872888 0 - 0.0299513 2026-02-23T14:07:56.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:07:55 trial068 ceph-mon[38805]: pgmap v2100: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:07:56.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:07:55 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1618213386' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:07:56.355 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3270 8 365281 365273 0.872621 0 - 0.0299513 2026-02-23T14:07:57.355 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3271 8 365281 365273 0.872354 0 - 0.0299513 2026-02-23T14:07:57.896 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:07:57 trial068 ceph-mon[38805]: pgmap v2101: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:07:58.219 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:07:58.338 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:07:58.355 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3272 8 365281 365273 0.872088 0 - 0.0299513 2026-02-23T14:07:58.441 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:07:58.439+0000 7fc19c52b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:07:58.441 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:07:58.439+0000 7fc19c52b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:07:58.552 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:07:58.552 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":219,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:06:19.146565+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:05:13.564547+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:03:48.616954+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:04:35.751423+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:07:58.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:07:58 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1729501807' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:07:59.355 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3273 8 365281 365273 0.871821 0 - 0.0299513 2026-02-23T14:08:00.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:07:59 trial068 ceph-mon[38805]: pgmap v2102: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:08:00.355 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3274 8 365281 365273 0.871555 0 - 0.0299513 2026-02-23T14:08:01.355 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3275 8 365281 365273 0.871289 0 - 0.0299513 2026-02-23T14:08:01.590 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:08:01.708 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:08:01.809 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:08:01.806+0000 7fd3c3ce5640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:08:01.809 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:08:01.807+0000 7fd3c3ce5640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:08:01.920 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:08:01.920 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":219,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:06:19.146565+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:05:13.564547+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:03:48.616954+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:04:35.751423+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:08:01.920 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:08:01 trial068 ceph-mon[38805]: pgmap v2103: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:08:02.355 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3276 8 365281 365273 0.871023 0 - 0.0299513 2026-02-23T14:08:03.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:08:02 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3594845413' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:08:03.355 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3277 8 365281 365273 0.870757 0 - 0.0299513 2026-02-23T14:08:03.974 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:08:03 trial068 ceph-mon[38805]: pgmap v2104: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:08:04.266 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:08:03 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:08:03.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:08:04.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:08:03 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:08:03] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:08:04.355 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3278 8 365281 365273 0.870492 0 - 0.0299513 2026-02-23T14:08:04.957 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:08:05.077 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:08:05.176 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:08:05.173+0000 7f5dece70640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:08:05.176 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:08:05.174+0000 7f5dece70640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:08:05.285 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:08:05.285 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":219,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:06:19.146565+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:05:13.564547+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:03:48.616954+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:04:35.751423+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:08:05.356 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3279 8 365281 365273 0.870226 0 - 0.0299513 2026-02-23T14:08:06.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:08:05 trial068 ceph-mon[38805]: pgmap v2105: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:08:06.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:08:05 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3041792681' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:08:06.356 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T14:08:06.355252+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T14:08:06.356 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T14:08:06.356 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3280 8 365281 365273 0.869961 0 - 0.0299513 2026-02-23T14:08:07.356 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3281 8 365281 365273 0.869696 0 - 0.0299513 2026-02-23T14:08:08.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:08:07 trial068 ceph-mon[38805]: pgmap v2106: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:08:08.323 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:08:08.356 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3282 8 365281 365273 0.869431 0 - 0.0299513 2026-02-23T14:08:08.442 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:08:08.541 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:08:08.539+0000 7ff27d42f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:08:08.541 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:08:08.539+0000 7ff27d42f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:08:08.650 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:08:08.650 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":219,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:06:19.146565+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:05:13.564547+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:03:48.616954+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:04:35.751423+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:08:09.356 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3283 8 365281 365273 0.869166 0 - 0.0299513 2026-02-23T14:08:10.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:08:09 trial068 ceph-mon[38805]: pgmap v2107: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:08:10.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:08:09 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/876647933' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:08:10.356 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3284 8 365281 365273 0.868901 0 - 0.0299513 2026-02-23T14:08:11.356 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3285 8 365281 365273 0.868637 0 - 0.0299513 2026-02-23T14:08:11.688 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:08:11.805 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:08:11.863 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:08:11 trial068 ceph-mon[38805]: pgmap v2108: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:08:11.909 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:08:11.907+0000 7ff336cd8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:08:11.909 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:08:11.907+0000 7ff336cd8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:08:12.018 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:08:12.018 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":219,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:06:19.146565+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:05:13.564547+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:03:48.616954+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:04:35.751423+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:08:12.356 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3286 8 365281 365273 0.868372 0 - 0.0299513 2026-02-23T14:08:13.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:08:12 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/687791874' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:08:13.356 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3287 8 365281 365273 0.868108 0 - 0.0299513 2026-02-23T14:08:13.974 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:08:13 trial068 ceph-mon[38805]: pgmap v2109: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:08:14.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:08:13 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:08:13] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:08:14.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:08:13 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:08:13.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:08:14.356 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3288 8 365281 365273 0.867844 0 - 0.0299513 2026-02-23T14:08:15.055 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:08:15.173 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:08:15.278 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:08:15.275+0000 7f7795e0b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:08:15.278 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:08:15.276+0000 7f7795e0b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:08:15.356 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3289 8 365281 365273 0.86758 0 - 0.0299513 2026-02-23T14:08:15.389 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:08:15.390 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":219,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:06:19.146565+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:05:13.564547+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:03:48.616954+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:04:35.751423+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:08:15.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:08:15 trial068 ceph-mon[38805]: pgmap v2110: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:08:15.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:08:15 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2784403250' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:08:16.357 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3290 8 365281 365273 0.867317 0 - 0.0299513 2026-02-23T14:08:17.357 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3291 8 365281 365273 0.867053 0 - 0.0299513 2026-02-23T14:08:18.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:08:17 trial068 ceph-mon[38805]: pgmap v2111: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:08:18.357 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3292 8 365281 365273 0.86679 0 - 0.0299513 2026-02-23T14:08:18.428 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:08:18.546 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:08:18.645 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:08:18.643+0000 7f1508f2f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:08:18.645 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:08:18.643+0000 7f1508f2f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:08:18.754 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:08:18.754 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":219,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:06:19.146565+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:05:13.564547+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:03:48.616954+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:04:35.751423+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:08:19.357 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3293 8 365281 365273 0.866526 0 - 0.0299513 2026-02-23T14:08:20.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:08:19 trial068 ceph-mon[38805]: pgmap v2112: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:08:20.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:08:19 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/652907932' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:08:20.357 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3294 8 365281 365273 0.866263 0 - 0.0299513 2026-02-23T14:08:21.357 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3295 8 365281 365273 0.866 0 - 0.0299513 2026-02-23T14:08:21.792 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:08:21.909 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:08:21.932 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:08:21 trial068 ceph-mon[38805]: pgmap v2113: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:08:22.007 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:08:22.005+0000 7fc73706a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:08:22.007 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:08:22.005+0000 7fc73706a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:08:22.115 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:08:22.115 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":219,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:06:19.146565+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:05:13.564547+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:03:48.616954+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:04:35.751423+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:08:22.357 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3296 8 365281 365273 0.865738 0 - 0.0299513 2026-02-23T14:08:23.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:08:22 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3893812042' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:08:23.357 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3297 8 365281 365273 0.865475 0 - 0.0299513 2026-02-23T14:08:23.974 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:08:23 trial068 ceph-mon[38805]: pgmap v2114: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:08:24.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:08:23 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:08:23] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:08:24.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:08:23 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:08:23.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:08:24.357 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3298 8 365281 365273 0.865213 0 - 0.0299513 2026-02-23T14:08:25.173 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:08:25.292 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:08:25.357 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3299 8 365281 365273 0.86495 0 - 0.0299513 2026-02-23T14:08:25.390 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:08:25.388+0000 7f01941bc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:08:25.390 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:08:25.388+0000 7f01941bc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:08:25.501 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:08:25.501 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":219,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:06:19.146565+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:05:13.564547+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:03:48.616954+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:04:35.751423+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:08:25.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:08:25 trial068 ceph-mon[38805]: pgmap v2115: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:08:25.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:08:25 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3185267988' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:08:26.357 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T14:08:26.356826+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T14:08:26.357 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T14:08:26.357 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3300 8 365281 365273 0.864688 0 - 0.0299513 2026-02-23T14:08:27.357 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3301 8 365281 365273 0.864426 0 - 0.0299513 2026-02-23T14:08:28.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:08:27 trial068 ceph-mon[38805]: pgmap v2116: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:08:28.357 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3302 8 365281 365273 0.864165 0 - 0.0299513 2026-02-23T14:08:28.539 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:08:28.657 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:08:28.758 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:08:28.756+0000 7fc285009640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:08:28.759 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:08:28.757+0000 7fc285009640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:08:28.867 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:08:28.867 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":219,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:06:19.146565+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:05:13.564547+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:03:48.616954+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:04:35.751423+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:08:29.357 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3303 8 365281 365273 0.863903 0 - 0.0299513 2026-02-23T14:08:30.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:08:29 trial068 ceph-mon[38805]: pgmap v2117: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:08:30.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:08:29 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2553165036' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:08:30.357 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3304 8 365281 365273 0.863641 0 - 0.0299513 2026-02-23T14:08:31.358 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3305 8 365281 365273 0.86338 0 - 0.0299513 2026-02-23T14:08:31.905 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:08:31.930 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:08:31 trial068 ceph-mon[38805]: pgmap v2118: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:08:32.022 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:08:32.119 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:08:32.117+0000 7f15dca64640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:08:32.119 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:08:32.117+0000 7f15dca64640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:08:32.228 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:08:32.229 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":219,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:06:19.146565+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:05:13.564547+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:03:48.616954+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:04:35.751423+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:08:32.358 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3306 8 365281 365273 0.863119 0 - 0.0299513 2026-02-23T14:08:33.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:08:32 trial068 ceph-mon[38805]: 5.a scrub starts 2026-02-23T14:08:33.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:08:32 trial068 ceph-mon[38805]: 5.a scrub ok 2026-02-23T14:08:33.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:08:32 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T14:08:33.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:08:32 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T14:08:33.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:08:32 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/4131542758' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:08:33.358 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3307 8 365281 365273 0.862858 0 - 0.0299513 2026-02-23T14:08:33.975 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:08:33 trial068 ceph-mon[38805]: pgmap v2119: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:08:34.266 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:08:33 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:08:33.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:08:34.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:08:33 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:08:33] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:08:34.358 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3308 8 365281 365273 0.862597 0 - 0.0299513 2026-02-23T14:08:35.266 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:08:35.358 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3309 8 365281 365273 0.862336 0 - 0.0299513 2026-02-23T14:08:35.384 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:08:35.481 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:08:35.479+0000 7ff96b958640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:08:35.482 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:08:35.479+0000 7ff96b958640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:08:35.590 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:08:35.591 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":219,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:06:19.146565+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:05:13.564547+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:03:48.616954+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:04:35.751423+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:08:35.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:08:35 trial068 ceph-mon[38805]: pgmap v2120: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:08:35.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:08:35 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/247957899' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:08:36.358 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3310 8 365281 365273 0.862076 0 - 0.0299513 2026-02-23T14:08:37.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:08:36 trial068 ceph-mon[38805]: 5.7 scrub starts 2026-02-23T14:08:37.358 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3311 8 365281 365273 0.861816 0 - 0.0299513 2026-02-23T14:08:38.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:08:37 trial068 ceph-mon[38805]: 5.7 scrub ok 2026-02-23T14:08:38.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:08:37 trial068 ceph-mon[38805]: pgmap v2121: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:08:38.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:08:37 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T14:08:38.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:08:37 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T14:08:38.358 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3312 8 365281 365273 0.861555 0 - 0.0299513 2026-02-23T14:08:38.628 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:08:38.747 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:08:38.847 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:08:38.845+0000 7ff5781d4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:08:38.847 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:08:38.845+0000 7ff5781d4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:08:38.955 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:08:38.955 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":219,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:06:19.146565+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:05:13.564547+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:03:48.616954+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:04:35.751423+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:08:39.358 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3313 8 365281 365273 0.861295 0 - 0.0299513 2026-02-23T14:08:40.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:08:39 trial068 ceph-mon[38805]: pgmap v2122: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:08:40.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:08:39 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/4220770639' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:08:40.358 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3314 8 365281 365273 0.861035 0 - 0.0299513 2026-02-23T14:08:41.358 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3315 8 365281 365273 0.860776 0 - 0.0299513 2026-02-23T14:08:41.992 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:08:42.017 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:08:41 trial068 ceph-mon[38805]: pgmap v2123: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:08:42.114 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:08:42.215 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:08:42.213+0000 7f6e8d690640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:08:42.215 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:08:42.213+0000 7f6e8d690640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:08:42.325 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:08:42.326 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":219,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:06:19.146565+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:05:13.564547+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:03:48.616954+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:04:35.751423+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:08:42.358 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3316 8 365281 365273 0.860516 0 - 0.0299513 2026-02-23T14:08:43.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:08:42 trial068 ceph-mon[38805]: 5.e scrub starts 2026-02-23T14:08:43.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:08:42 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2657785650' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:08:43.359 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3317 8 365281 365273 0.860257 0 - 0.0299513 2026-02-23T14:08:43.974 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:08:43 trial068 ceph-mon[38805]: 5.e scrub ok 2026-02-23T14:08:43.974 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:08:43 trial068 ceph-mon[38805]: pgmap v2124: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:08:44.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:08:43 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:08:43] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:08:44.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:08:43 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:08:43.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:08:44.359 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3318 8 365281 365273 0.859997 0 - 0.0299513 2026-02-23T14:08:45.359 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3319 8 365281 365273 0.859738 0 - 0.0299513 2026-02-23T14:08:45.365 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:08:45.484 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:08:45.581 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:08:45.579+0000 7f462c1bc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:08:45.581 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:08:45.579+0000 7f462c1bc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:08:45.690 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:08:45.690 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":219,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:06:19.146565+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:05:13.564547+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:03:48.616954+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:04:35.751423+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:08:45.691 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:08:45 trial068 ceph-mon[38805]: pgmap v2125: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:08:46.359 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T14:08:46.358554+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T14:08:46.359 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T14:08:46.359 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3320 8 365281 365273 0.859479 0 - 0.0299513 2026-02-23T14:08:47.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:08:46 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/595929144' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:08:47.359 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3321 8 365281 365273 0.85922 0 - 0.0299513 2026-02-23T14:08:48.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:08:47 trial068 ceph-mon[38805]: pgmap v2126: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:08:48.359 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3322 8 365281 365273 0.858962 0 - 0.0299513 2026-02-23T14:08:48.728 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:08:48.845 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:08:48.942 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:08:48.940+0000 7f6f0380e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:08:48.942 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:08:48.940+0000 7f6f0380e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:08:49.055 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:08:49.056 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":219,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:06:19.146565+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:05:13.564547+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:03:48.616954+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:04:35.751423+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:08:49.359 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3323 8 365281 365273 0.858703 0 - 0.0299513 2026-02-23T14:08:50.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:08:49 trial068 ceph-mon[38805]: pgmap v2127: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:08:50.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:08:49 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/823684748' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:08:50.359 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3324 8 365281 365273 0.858445 0 - 0.0299513 2026-02-23T14:08:51.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:08:50 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T14:08:51.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:08:50 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T14:08:51.359 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3325 8 365281 365273 0.858187 0 - 0.0299513 2026-02-23T14:08:52.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:08:51 trial068 ceph-mon[38805]: pgmap v2128: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:08:52.094 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:08:52.212 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:08:52.314 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:08:52.312+0000 7f55f5076640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:08:52.314 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:08:52.312+0000 7f55f5076640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:08:52.359 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3326 8 365281 365273 0.857929 0 - 0.0299513 2026-02-23T14:08:52.425 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:08:52.425 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":219,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:06:19.146565+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:05:13.564547+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:03:48.616954+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:04:35.751423+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:08:52.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:08:52 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2026-02-23T14:08:52.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:08:52 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2659404451' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:08:52.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:08:52 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-23T14:08:52.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:08:52 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2026-02-23T14:08:53.359 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3327 8 365281 365273 0.857671 0 - 0.0299513 2026-02-23T14:08:53.975 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:08:53 trial068 ceph-mon[38805]: pgmap v2129: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:08:53.975 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:08:53 trial068 ceph-mon[38805]: pgmap v2130: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:08:53.975 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:08:53 trial068 ceph-mon[38805]: pgmap v2131: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:08:53.975 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:08:53 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T14:08:53.975 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:08:53 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2026-02-23T14:08:54.250 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:08:53 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:08:53] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:08:54.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:08:53 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:08:53.974Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:08:54.359 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3328 8 365281 365273 0.857413 0 - 0.0299513 2026-02-23T14:08:55.360 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3329 8 365281 365273 0.857156 0 - 0.0299513 2026-02-23T14:08:55.465 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:08:55.581 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:08:55.688 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:08:55.686+0000 7fb4b193b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:08:55.689 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:08:55.687+0000 7fb4b193b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:08:55.797 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:08:55.798 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":220,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:08:55.456944+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:05:13.564547+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:08:49.362470+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:04:35.751423+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:08:55.798 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:08:55 trial068 ceph-mon[38805]: pgmap v2132: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:08:56.360 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3330 8 365281 365273 0.856898 0 - 0.0299513 2026-02-23T14:08:56.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:08:56 trial068 ceph-mon[38805]: osdmap e220: 4 total, 3 up, 4 in 2026-02-23T14:08:56.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:08:56 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/162752034' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:08:57.360 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3331 8 365281 365273 0.856641 0 - 0.0299513 2026-02-23T14:08:57.745 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:08:57 trial068 ceph-mon[38805]: pgmap v2134: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:08:58.360 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3332 8 365281 365273 0.856384 0 - 0.0299513 2026-02-23T14:08:58.835 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:08:58.953 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:08:59.056 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:08:59.054+0000 7f487effa640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:08:59.056 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:08:59.054+0000 7f487effa640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:08:59.170 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:08:59.170 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":220,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:08:55.456944+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:05:13.564547+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:08:49.362470+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:04:35.751423+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:08:59.360 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3333 8 365281 365273 0.856127 0 - 0.0299513 2026-02-23T14:09:00.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:08:59 trial068 ceph-mon[38805]: pgmap v2135: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:09:00.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:08:59 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2269966066' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:09:00.360 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3334 8 365281 365273 0.85587 0 - 0.0299513 2026-02-23T14:09:01.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:09:00 trial068 ceph-mon[38805]: 5.8 scrub starts 2026-02-23T14:09:01.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:09:00 trial068 ceph-mon[38805]: 5.8 scrub ok 2026-02-23T14:09:01.360 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3335 8 365281 365273 0.855614 0 - 0.0299513 2026-02-23T14:09:02.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:09:01 trial068 ceph-mon[38805]: pgmap v2136: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:09:02.207 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:09:02.322 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:09:02.360 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3336 8 365281 365273 0.855357 0 - 0.0299513 2026-02-23T14:09:02.426 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:09:02.424+0000 7f756a6ce640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:09:02.426 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:09:02.424+0000 7f756a6ce640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:09:02.535 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:09:02.535 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":220,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:08:55.456944+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:05:13.564547+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:08:49.362470+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:04:35.751423+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:09:02.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:09:02 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3190810624' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:09:03.360 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3337 8 365281 365273 0.855101 0 - 0.0299513 2026-02-23T14:09:03.975 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:09:03 trial068 ceph-mon[38805]: 5.5 scrub starts 2026-02-23T14:09:03.975 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:09:03 trial068 ceph-mon[38805]: 5.5 scrub ok 2026-02-23T14:09:03.975 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:09:03 trial068 ceph-mon[38805]: pgmap v2137: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:09:04.266 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:09:03 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:09:03.974Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:09:04.287 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:09:03 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:09:03] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:09:04.360 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3338 8 365281 365273 0.854845 0 - 0.0299513 2026-02-23T14:09:05.361 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3339 8 365281 365273 0.854589 0 - 0.0299513 2026-02-23T14:09:05.572 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:09:05.692 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:09:05.715 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:09:05 trial068 ceph-mon[38805]: pgmap v2138: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:09:05.792 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:09:05.790+0000 7f4e1936a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:09:05.793 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:09:05.791+0000 7f4e1936a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:09:05.901 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:09:05.901 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":220,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:08:55.456944+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:05:13.564547+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:08:49.362470+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:04:35.751423+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:09:06.360 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T14:09:06.360029+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T14:09:06.361 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T14:09:06.361 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3340 8 365281 365273 0.854333 0 - 0.0299513 2026-02-23T14:09:06.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:09:06 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/4102980470' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:09:07.360 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3341 8 365281 365273 0.854077 0 - 0.0299513 2026-02-23T14:09:07.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:09:07 trial068 ceph-mon[38805]: pgmap v2139: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:09:08.361 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3342 8 365281 365273 0.853821 0 - 0.0299513 2026-02-23T14:09:08.940 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:09:09.057 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:09:09.163 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:09:09.161+0000 7f120c88e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:09:09.163 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:09:09.161+0000 7f120c88e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:09:09.274 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:09:09.274 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":220,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:08:55.456944+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:05:13.564547+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:08:49.362470+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:04:35.751423+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:09:09.361 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3343 8 365281 365273 0.853566 0 - 0.0299513 2026-02-23T14:09:10.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:09:09 trial068 ceph-mon[38805]: pgmap v2140: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:09:10.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:09:09 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2245488078' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:09:10.361 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3344 8 365281 365273 0.853311 0 - 0.0299513 2026-02-23T14:09:11.361 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3345 8 365281 365273 0.853056 0 - 0.0299513 2026-02-23T14:09:12.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:09:11 trial068 ceph-mon[38805]: pgmap v2141: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:09:12.312 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:09:12.361 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3346 8 365281 365273 0.852801 0 - 0.0299513 2026-02-23T14:09:12.427 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:09:12.532 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:09:12.529+0000 7fd1fb4eb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:09:12.532 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:09:12.529+0000 7fd1fb4eb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:09:12.641 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:09:12.641 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":220,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:08:55.456944+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:05:13.564547+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:08:49.362470+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:04:35.751423+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:09:13.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:09:12 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3543090840' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:09:13.361 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3347 8 365281 365273 0.852546 0 - 0.0299513 2026-02-23T14:09:13.975 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:09:13 trial068 ceph-mon[38805]: pgmap v2142: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:09:14.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:09:13 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:09:13] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:09:14.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:09:13 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:09:13.974Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:09:14.361 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3348 8 365281 365273 0.852291 0 - 0.0299513 2026-02-23T14:09:15.361 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3349 8 365281 365273 0.852037 0 - 0.0299513 2026-02-23T14:09:15.679 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:09:15.800 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:09:15.822 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:09:15 trial068 ceph-mon[38805]: pgmap v2143: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:09:15.900 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:09:15.898+0000 7faacba8f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:09:15.900 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:09:15.898+0000 7faacba8f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:09:16.009 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:09:16.009 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":220,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:08:55.456944+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:05:13.564547+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:08:49.362470+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:04:35.751423+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:09:16.361 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3350 8 365281 365273 0.851783 0 - 0.0299513 2026-02-23T14:09:16.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:09:16 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/4242705825' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:09:17.361 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3351 8 365281 365273 0.851528 0 - 0.0299513 2026-02-23T14:09:17.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:09:17 trial068 ceph-mon[38805]: pgmap v2144: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:09:18.361 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3352 8 365281 365273 0.851274 0 - 0.0299513 2026-02-23T14:09:19.046 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:09:19.163 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:09:19.262 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:09:19.260+0000 7f13d613f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:09:19.263 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:09:19.261+0000 7f13d613f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:09:19.361 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3353 8 365281 365273 0.85102 0 - 0.0299513 2026-02-23T14:09:19.372 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:09:19.372 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":220,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:08:55.456944+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:05:13.564547+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:08:49.362470+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:04:35.751423+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:09:20.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:09:19 trial068 ceph-mon[38805]: pgmap v2145: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:09:20.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:09:19 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/450347773' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:09:20.361 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3354 8 365281 365273 0.850767 0 - 0.0299513 2026-02-23T14:09:21.362 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3355 8 365281 365273 0.850513 0 - 0.0299513 2026-02-23T14:09:22.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:09:21 trial068 ceph-mon[38805]: pgmap v2146: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:09:22.362 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3356 8 365281 365273 0.85026 0 - 0.0299513 2026-02-23T14:09:22.410 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:09:22.529 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:09:22.633 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:09:22.630+0000 7f3c0d325640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:09:22.633 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:09:22.631+0000 7f3c0d325640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:09:22.743 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:09:22.743 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":220,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:08:55.456944+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:05:13.564547+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:08:49.362470+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:04:35.751423+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:09:23.362 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3357 8 365281 365273 0.850006 0 - 0.0299513 2026-02-23T14:09:23.973 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:09:23 trial068 ceph-mon[38805]: pgmap v2147: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:09:23.973 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:09:23 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1048525191' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:09:24.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:09:23 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:09:23] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:09:24.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:09:23 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:09:23.972Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:09:24.362 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3358 8 365281 365273 0.849753 0 - 0.0299513 2026-02-23T14:09:25.362 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3359 8 365281 365273 0.8495 0 - 0.0299513 2026-02-23T14:09:25.782 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:09:25.899 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:09:25.951 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:09:25 trial068 ceph-mon[38805]: pgmap v2148: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:09:26.004 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:09:26.002+0000 7efc04486640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:09:26.004 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:09:26.002+0000 7efc04486640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:09:26.113 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:09:26.113 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":220,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:08:55.456944+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:05:13.564547+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:08:49.362470+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:04:35.751423+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:09:26.362 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T14:09:26.361539+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T14:09:26.362 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T14:09:26.362 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3360 8 365281 365273 0.849247 0 - 0.0299513 2026-02-23T14:09:27.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:09:26 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/984792174' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:09:27.362 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3361 8 365281 365273 0.848995 0 - 0.0299513 2026-02-23T14:09:28.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:09:27 trial068 ceph-mon[38805]: pgmap v2149: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:09:28.362 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3362 8 365281 365273 0.848742 0 - 0.0299513 2026-02-23T14:09:29.151 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:09:29.268 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:09:29.362 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3363 8 365281 365273 0.84849 0 - 0.0299513 2026-02-23T14:09:29.366 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:09:29.364+0000 7f86c0a1d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:09:29.366 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:09:29.365+0000 7f86c0a1d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:09:29.476 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:09:29.476 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":220,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:08:55.456944+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:05:13.564547+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:08:49.362470+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:04:35.751423+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:09:29.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:09:29 trial068 ceph-mon[38805]: pgmap v2150: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:09:29.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:09:29 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2527653865' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:09:30.362 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3364 8 365281 365273 0.848238 0 - 0.0299513 2026-02-23T14:09:31.362 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3365 8 365281 365273 0.847986 0 - 0.0299513 2026-02-23T14:09:32.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:09:31 trial068 ceph-mon[38805]: pgmap v2151: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:09:32.362 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3366 8 365281 365273 0.847734 0 - 0.0299513 2026-02-23T14:09:32.514 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:09:32.635 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:09:32.736 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:09:32.734+0000 7f1981fc7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:09:32.736 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:09:32.734+0000 7f1981fc7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:09:32.846 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:09:32.846 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":220,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:08:55.456944+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:05:13.564547+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:08:49.362470+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:04:35.751423+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:09:33.362 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3367 8 365281 365273 0.847482 0 - 0.0299513 2026-02-23T14:09:33.974 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:09:33 trial068 ceph-mon[38805]: pgmap v2152: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:09:33.975 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:09:33 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/4198167181' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:09:34.266 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:09:33 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:09:33.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:09:34.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:09:33 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:09:33] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:09:34.362 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3368 8 365281 365273 0.84723 0 - 0.0299513 2026-02-23T14:09:35.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:09:34 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T14:09:35.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:09:34 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T14:09:35.362 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3369 8 365281 365273 0.846979 0 - 0.0299513 2026-02-23T14:09:35.883 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:09:36.003 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:09:36.025 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:09:35 trial068 ceph-mon[38805]: pgmap v2153: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:09:36.104 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:09:36.102+0000 7fd83e0d9640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:09:36.104 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:09:36.102+0000 7fd83e0d9640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:09:36.212 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:09:36.212 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":220,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:08:55.456944+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:05:13.564547+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:08:49.362470+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:04:35.751423+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:09:36.363 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3370 8 365281 365273 0.846727 0 - 0.0299513 2026-02-23T14:09:37.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:09:36 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2305679314' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:09:37.363 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3371 8 365281 365273 0.846476 0 - 0.0299513 2026-02-23T14:09:38.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:09:37 trial068 ceph-mon[38805]: pgmap v2154: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:09:38.363 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3372 8 365281 365273 0.846225 0 - 0.0299513 2026-02-23T14:09:39.249 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:09:39.363 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3373 8 365281 365273 0.845974 0 - 0.0299513 2026-02-23T14:09:39.367 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:09:39.469 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:09:39.467+0000 7f6e40c32640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:09:39.469 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:09:39.467+0000 7f6e40c32640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:09:39.579 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:09:39.579 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":220,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:08:55.456944+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:05:13.564547+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:08:49.362470+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:04:35.751423+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:09:39.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:09:39 trial068 ceph-mon[38805]: pgmap v2155: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:09:39.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:09:39 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2979937988' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:09:40.363 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3374 8 365281 365273 0.845724 0 - 0.0299513 2026-02-23T14:09:41.363 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3375 8 365281 365273 0.845473 0 - 0.0299513 2026-02-23T14:09:42.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:09:41 trial068 ceph-mon[38805]: pgmap v2156: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:09:42.363 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3376 8 365281 365273 0.845223 0 - 0.0299513 2026-02-23T14:09:42.617 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:09:42.737 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:09:42.843 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:09:42.841+0000 7f8ff747d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:09:42.843 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:09:42.841+0000 7f8ff747d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:09:42.953 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:09:42.954 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":220,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:08:55.456944+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:05:13.564547+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:08:49.362470+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:04:35.751423+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:09:42.954 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:09:42 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T14:09:42.954 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:09:42 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T14:09:43.363 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3377 8 365281 365273 0.844972 0 - 0.0299513 2026-02-23T14:09:43.974 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:09:43 trial068 ceph-mon[38805]: pgmap v2157: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:09:43.974 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:09:43 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1283647627' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:09:44.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:09:43 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:09:43] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:09:44.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:09:43 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:09:43.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:09:44.363 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3378 8 365281 365273 0.844722 0 - 0.0299513 2026-02-23T14:09:45.363 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3379 8 365281 365273 0.844472 0 - 0.0299513 2026-02-23T14:09:45.991 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:09:46.015 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:09:45 trial068 ceph-mon[38805]: 5.a scrub starts 2026-02-23T14:09:46.015 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:09:45 trial068 ceph-mon[38805]: 5.a scrub ok 2026-02-23T14:09:46.015 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:09:45 trial068 ceph-mon[38805]: pgmap v2158: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:09:46.106 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:09:46.208 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:09:46.206+0000 7f9b617fe640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:09:46.208 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:09:46.206+0000 7f9b617fe640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:09:46.317 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:09:46.317 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":220,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:08:55.456944+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:05:13.564547+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:08:49.362470+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:04:35.751423+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:09:46.363 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T14:09:46.362924+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T14:09:46.363 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T14:09:46.364 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3380 8 365281 365273 0.844222 0 - 0.0299513 2026-02-23T14:09:47.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:09:46 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3531531029' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:09:47.363 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3381 8 365281 365273 0.843973 0 - 0.0299513 2026-02-23T14:09:48.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:09:47 trial068 ceph-mon[38805]: pgmap v2159: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:09:48.363 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3382 8 365281 365273 0.843723 0 - 0.0299513 2026-02-23T14:09:49.363 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3383 8 365281 365273 0.843474 0 - 0.0299513 2026-02-23T14:09:49.372 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:09:49.491 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:09:49.593 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:09:49.591+0000 7fe78b968640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:09:49.593 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:09:49.591+0000 7fe78b968640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:09:49.702 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:09:49.702 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":220,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:08:55.456944+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:05:13.564547+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:08:49.362470+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:04:35.751423+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:09:50.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:09:49 trial068 ceph-mon[38805]: pgmap v2160: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:09:50.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:09:49 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2697420318' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:09:50.364 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3384 8 365281 365273 0.843224 0 - 0.0299513 2026-02-23T14:09:51.364 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3385 8 365281 365273 0.842975 0 - 0.0299513 2026-02-23T14:09:52.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:09:51 trial068 ceph-mon[38805]: pgmap v2161: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:09:52.364 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3386 8 365281 365273 0.842726 0 - 0.0299513 2026-02-23T14:09:52.740 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:09:52.855 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:09:52.969 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:09:52.967+0000 7f323373c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:09:52.969 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:09:52.967+0000 7f323373c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:09:53.077 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:09:53.078 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":220,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:08:55.456944+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:05:13.564547+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:08:49.362470+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:04:35.751423+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:09:53.078 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:09:52 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2026-02-23T14:09:53.364 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3387 8 365281 365273 0.842478 0 - 0.0299513 2026-02-23T14:09:54.094 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:09:53 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:09:53] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:09:54.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:09:53 trial068 ceph-mon[38805]: pgmap v2162: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:09:54.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:09:53 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-23T14:09:54.095 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:09:53 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2026-02-23T14:09:54.095 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:09:53 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T14:09:54.095 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:09:53 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2026-02-23T14:09:54.095 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:09:53 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2540243535' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:09:54.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:09:53 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:09:53.974Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:09:54.364 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3388 8 365281 365273 0.842229 0 - 0.0299513 2026-02-23T14:09:55.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:09:54 trial068 ceph-mon[38805]: pgmap v2163: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:09:55.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:09:54 trial068 ceph-mon[38805]: pgmap v2164: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:09:55.364 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3389 8 365281 365273 0.84198 0 - 0.0299513 2026-02-23T14:09:56.114 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:09:56.233 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:09:56.333 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:09:56.331+0000 7f1384f90640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:09:56.333 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:09:56.331+0000 7f1384f90640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:09:56.364 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3390 8 365281 365273 0.841732 0 - 0.0299513 2026-02-23T14:09:56.441 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:09:56.441 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":220,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:08:55.456944+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:05:13.564547+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:08:49.362470+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:04:35.751423+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:09:57.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:09:56 trial068 ceph-mon[38805]: pgmap v2165: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:09:57.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:09:56 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/659639016' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:09:57.364 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3391 8 365281 365273 0.841484 0 - 0.0299513 2026-02-23T14:09:58.364 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3392 8 365281 365273 0.841236 0 - 0.0299513 2026-02-23T14:09:59.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:09:58 trial068 ceph-mon[38805]: pgmap v2166: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:09:59.364 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3393 8 365281 365273 0.840988 0 - 0.0299513 2026-02-23T14:09:59.482 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:09:59.596 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:09:59.699 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:09:59.697+0000 7f32de5b7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:09:59.699 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:09:59.697+0000 7f32de5b7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:09:59.810 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:09:59.810 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":220,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:08:55.456944+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:05:13.564547+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:08:49.362470+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:04:35.751423+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:10:00.365 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3394 8 365281 365273 0.84074 0 - 0.0299513 2026-02-23T14:10:01.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:10:00 trial068 ceph-mon[38805]: pgmap v2167: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:10:01.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:10:00 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T14:10:01.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:10:00 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T14:10:01.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:10:00 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2131576066' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:10:01.365 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3395 8 365281 365273 0.840492 0 - 0.0299513 2026-02-23T14:10:02.365 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3396 8 365281 365273 0.840245 0 - 0.0299513 2026-02-23T14:10:02.847 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:10:02.967 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:10:02.989 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:10:02 trial068 ceph-mon[38805]: pgmap v2168: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:10:03.073 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:10:03.070+0000 7f9e09b50640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:10:03.073 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:10:03.071+0000 7f9e09b50640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:10:03.181 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:10:03.181 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":220,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:08:55.456944+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:05:13.564547+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:08:49.362470+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:04:35.751423+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:10:03.365 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3397 8 365281 365273 0.839997 0 - 0.0299513 2026-02-23T14:10:04.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:10:03 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/740547922' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:10:04.094 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:10:03 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:10:03] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:10:04.266 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:10:03 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:10:03.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:10:04.365 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3398 8 365281 365273 0.83975 0 - 0.0299513 2026-02-23T14:10:05.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:10:04 trial068 ceph-mon[38805]: pgmap v2169: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:10:05.365 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3399 8 365281 365273 0.839503 0 - 0.0299513 2026-02-23T14:10:06.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:10:05 trial068 ceph-mon[38805]: 5.7 scrub starts 2026-02-23T14:10:06.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:10:05 trial068 ceph-mon[38805]: 5.7 scrub ok 2026-02-23T14:10:06.218 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:10:06.338 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:10:06.365 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T14:10:06.364596+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T14:10:06.365 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T14:10:06.365 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3400 8 365281 365273 0.839256 0 - 0.0299513 2026-02-23T14:10:06.435 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:10:06.432+0000 7ff20f2cc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:10:06.435 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:10:06.433+0000 7ff20f2cc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:10:06.544 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:10:06.545 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":220,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:08:55.456944+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:05:13.564547+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:08:49.362470+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:04:35.751423+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:10:06.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:10:06 trial068 ceph-mon[38805]: pgmap v2170: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:10:06.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:10:06 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2713081985' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:10:07.365 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3401 8 365281 365273 0.83901 0 - 0.0299513 2026-02-23T14:10:08.365 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3402 8 365281 365273 0.838763 0 - 0.0299513 2026-02-23T14:10:09.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:10:08 trial068 ceph-mon[38805]: pgmap v2171: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:10:09.365 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3403 8 365281 365273 0.838516 0 - 0.0299513 2026-02-23T14:10:09.581 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:10:09.700 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:10:09.795 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:10:09.793+0000 7fd614167640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:10:09.795 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:10:09.793+0000 7fd614167640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:10:09.904 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:10:09.904 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":220,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:08:55.456944+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:05:13.564547+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:08:49.362470+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:04:35.751423+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:10:10.365 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3404 8 365281 365273 0.83827 0 - 0.0299513 2026-02-23T14:10:11.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:10:10 trial068 ceph-mon[38805]: pgmap v2172: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:10:11.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:10:10 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/256958936' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:10:11.365 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3405 8 365281 365273 0.838024 0 - 0.0299513 2026-02-23T14:10:12.365 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3406 8 365281 365273 0.837778 0 - 0.0299513 2026-02-23T14:10:12.940 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:10:13.059 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:10:13.082 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:10:12 trial068 ceph-mon[38805]: pgmap v2173: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:10:13.082 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:10:12 trial068 ceph-mon[38805]: 5.e scrub starts 2026-02-23T14:10:13.082 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:10:12 trial068 ceph-mon[38805]: 5.e scrub ok 2026-02-23T14:10:13.082 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:10:12 trial068 ceph-mon[38805]: osdmap e221: 4 total, 3 up, 4 in 2026-02-23T14:10:13.159 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:10:13.157+0000 7f48421a0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:10:13.159 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:10:13.157+0000 7f48421a0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:10:13.269 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:10:13.269 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":221,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:10:11.738124+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:05:13.564547+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:08:49.362470+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:09:41.751921+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:10:13.365 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3407 8 365281 365273 0.837532 0 - 0.0299513 2026-02-23T14:10:14.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:10:13 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1604099078' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:10:14.094 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:10:13 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:10:13] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:10:14.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:10:13 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:10:13.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:10:14.365 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3408 8 365281 365273 0.837286 0 - 0.0299513 2026-02-23T14:10:15.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:10:14 trial068 ceph-mon[38805]: pgmap v2175: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:10:15.366 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3409 8 365281 365273 0.837041 0 - 0.0299513 2026-02-23T14:10:16.306 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:10:16.366 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3410 8 365281 365273 0.836795 0 - 0.0299513 2026-02-23T14:10:16.424 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:10:16.523 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:10:16.521+0000 7fb88ff79640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:10:16.524 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:10:16.522+0000 7fb88ff79640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:10:16.634 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:10:16.635 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":221,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:10:11.738124+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:05:13.564547+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:08:49.362470+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:09:41.751921+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:10:17.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:10:16 trial068 ceph-mon[38805]: pgmap v2176: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:10:17.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:10:16 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/4153746540' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:10:17.366 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3411 8 365281 365273 0.83655 0 - 0.0299513 2026-02-23T14:10:18.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:10:17 trial068 ceph-mon[38805]: 5.5 scrub starts 2026-02-23T14:10:18.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:10:17 trial068 ceph-mon[38805]: 5.5 scrub ok 2026-02-23T14:10:18.366 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3412 8 365281 365273 0.836305 0 - 0.0299513 2026-02-23T14:10:19.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:10:18 trial068 ceph-mon[38805]: pgmap v2177: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:10:19.366 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3413 8 365281 365273 0.83606 0 - 0.0299513 2026-02-23T14:10:19.674 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:10:19.794 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:10:19.891 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:10:19.889+0000 7faad66bf640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:10:19.891 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:10:19.889+0000 7faad66bf640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:10:20.001 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:10:20.002 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":221,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:10:11.738124+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:05:13.564547+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:08:49.362470+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:09:41.751921+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:10:20.366 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3414 8 365281 365273 0.835815 0 - 0.0299513 2026-02-23T14:10:21.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:10:20 trial068 ceph-mon[38805]: pgmap v2178: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:10:21.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:10:20 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1970178129' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:10:21.366 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3415 8 365281 365273 0.83557 0 - 0.0299513 2026-02-23T14:10:22.366 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3416 8 365281 365273 0.835325 0 - 0.0299513 2026-02-23T14:10:23.038 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:10:23.064 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:10:22 trial068 ceph-mon[38805]: pgmap v2179: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:10:23.158 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:10:23.254 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:10:23.252+0000 7f06414f8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:10:23.254 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:10:23.252+0000 7f06414f8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:10:23.362 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:10:23.362 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":221,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:10:11.738124+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:05:13.564547+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:08:49.362470+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:09:41.751921+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:10:23.366 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3417 8 365281 365273 0.835081 0 - 0.0299513 2026-02-23T14:10:24.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:10:23 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1146003454' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:10:24.094 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:10:23 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:10:23] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:10:24.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:10:23 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:10:23.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:10:24.366 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3418 8 365281 365273 0.834837 0 - 0.0299513 2026-02-23T14:10:25.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:10:24 trial068 ceph-mon[38805]: pgmap v2180: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:10:25.366 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3419 8 365281 365273 0.834592 0 - 0.0299513 2026-02-23T14:10:26.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:10:25 trial068 ceph-mon[38805]: pgmap v2181: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:10:26.366 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T14:10:26.366050+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T14:10:26.367 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T14:10:26.367 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3420 8 365281 365273 0.834348 0 - 0.0299513 2026-02-23T14:10:26.415 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:10:26.535 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:10:26.644 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:10:26.642+0000 7f9434ff8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:10:26.645 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:10:26.643+0000 7f9434ff8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:10:26.753 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:10:26.753 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":221,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:10:11.738124+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:05:13.564547+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:08:49.362470+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:09:41.751921+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:10:27.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:10:26 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2394962622' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:10:27.366 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3421 8 365281 365273 0.834105 0 - 0.0299513 2026-02-23T14:10:28.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:10:27 trial068 ceph-mon[38805]: pgmap v2182: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:10:28.367 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3422 8 365281 365273 0.833861 0 - 0.0299513 2026-02-23T14:10:29.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:10:28 trial068 ceph-mon[38805]: 5.8 scrub starts 2026-02-23T14:10:29.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:10:28 trial068 ceph-mon[38805]: 5.8 scrub ok 2026-02-23T14:10:29.367 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3423 8 365281 365273 0.833617 0 - 0.0299513 2026-02-23T14:10:29.790 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:10:29.908 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:10:30.014 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:10:30.012+0000 7fc7bb922640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:10:30.014 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:10:30.012+0000 7fc7bb922640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:10:30.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:10:29 trial068 ceph-mon[38805]: pgmap v2183: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:10:30.122 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:10:30.122 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":221,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:10:11.738124+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:05:13.564547+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:08:49.362470+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:09:41.751921+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:10:30.367 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3424 8 365281 365273 0.833374 0 - 0.0299513 2026-02-23T14:10:31.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:10:30 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3015017746' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:10:31.367 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3425 8 365281 365273 0.83313 0 - 0.0299513 2026-02-23T14:10:32.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:10:31 trial068 ceph-mon[38805]: pgmap v2184: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:10:32.367 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3426 8 365281 365273 0.832887 0 - 0.0299513 2026-02-23T14:10:33.159 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:10:33.278 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:10:33.367 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3427 8 365281 365273 0.832644 0 - 0.0299513 2026-02-23T14:10:33.381 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:10:33.379+0000 7fe100da1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:10:33.382 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:10:33.380+0000 7fe100da1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:10:33.490 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:10:33.490 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":221,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:10:11.738124+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:05:13.564547+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:08:49.362470+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:09:41.751921+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:10:34.266 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:10:33 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:10:33.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:10:34.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:10:33 trial068 ceph-mon[38805]: pgmap v2185: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:10:34.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:10:33 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1934744498' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:10:34.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:10:33 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:10:33] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:10:34.367 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3428 8 365281 365273 0.832401 0 - 0.0299513 2026-02-23T14:10:35.367 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3429 8 365281 365273 0.832159 0 - 0.0299513 2026-02-23T14:10:36.367 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3430 8 365281 365273 0.831916 0 - 0.0299513 2026-02-23T14:10:36.549 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:10:36.664 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:10:36.721 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:10:36 trial068 ceph-mon[38805]: pgmap v2186: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:10:36.773 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:10:36.771+0000 7f5902eba640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:10:36.774 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:10:36.772+0000 7f5902eba640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:10:36.881 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:10:36.881 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":221,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:10:11.738124+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:05:13.564547+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:08:49.362470+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:09:41.751921+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:10:37.367 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3431 8 365281 365273 0.831673 0 - 0.0299513 2026-02-23T14:10:37.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:10:37 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3527502520' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:10:38.367 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3432 8 365281 365273 0.831431 0 - 0.0299513 2026-02-23T14:10:38.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:10:38 trial068 ceph-mon[38805]: pgmap v2187: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:10:39.367 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3433 8 365281 365273 0.831189 0 - 0.0299513 2026-02-23T14:10:39.918 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:10:40.036 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:10:40.137 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:10:40.135+0000 7f03d9084640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:10:40.138 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:10:40.135+0000 7f03d9084640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:10:40.246 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:10:40.247 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":221,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:10:11.738124+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:05:13.564547+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:08:49.362470+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:09:41.751921+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:10:40.368 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3434 8 365281 365273 0.830947 0 - 0.0299513 2026-02-23T14:10:40.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:10:40 trial068 ceph-mon[38805]: pgmap v2188: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:10:40.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:10:40 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2317408962' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:10:41.368 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3435 8 365281 365273 0.830705 0 - 0.0299513 2026-02-23T14:10:42.368 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3436 8 365281 365273 0.830463 0 - 0.0299513 2026-02-23T14:10:42.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:10:42 trial068 ceph-mon[38805]: pgmap v2189: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:10:43.300 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:10:43.368 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3437 8 365281 365273 0.830222 0 - 0.0299513 2026-02-23T14:10:43.418 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:10:43.516 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:10:43.513+0000 7f2f125e7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:10:43.516 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:10:43.514+0000 7f2f125e7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:10:43.625 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:10:43.625 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":221,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:10:11.738124+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:05:13.564547+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:08:49.362470+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:09:41.751921+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:10:44.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:10:43 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:10:43] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:10:44.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:10:43 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:10:43.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:10:44.368 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3438 8 365281 365273 0.82998 0 - 0.0299513 2026-02-23T14:10:44.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:10:44 trial068 ceph-mon[38805]: pgmap v2190: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:10:44.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:10:44 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3821205606' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:10:45.368 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3439 8 365281 365273 0.829739 0 - 0.0299513 2026-02-23T14:10:46.368 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T14:10:46.367560+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T14:10:46.368 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T14:10:46.368 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3440 8 365281 365273 0.829498 0 - 0.0299513 2026-02-23T14:10:46.663 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:10:46.781 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:10:46.803 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:10:46 trial068 ceph-mon[38805]: pgmap v2191: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:10:46.882 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:10:46.879+0000 7fae7e1a1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:10:46.882 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:10:46.880+0000 7fae7e1a1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:10:46.989 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:10:46.990 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":221,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:10:11.738124+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:05:13.564547+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:08:49.362470+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:09:41.751921+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:10:47.368 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3441 8 365281 365273 0.829257 0 - 0.0299513 2026-02-23T14:10:47.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:10:47 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/906986418' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:10:48.368 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3442 8 365281 365273 0.829016 0 - 0.0299513 2026-02-23T14:10:48.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:10:48 trial068 ceph-mon[38805]: pgmap v2192: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:10:49.368 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3443 8 365281 365273 0.828775 0 - 0.0299513 2026-02-23T14:10:50.026 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:10:50.147 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:10:50.252 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:10:50.250+0000 7f60f7102640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:10:50.253 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:10:50.251+0000 7f60f7102640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:10:50.361 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:10:50.361 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":221,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:10:11.738124+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:05:13.564547+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:08:49.362470+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:09:41.751921+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:10:50.368 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3444 8 365281 365273 0.828534 0 - 0.0299513 2026-02-23T14:10:50.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:10:50 trial068 ceph-mon[38805]: pgmap v2193: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:10:50.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:10:50 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1572335579' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:10:51.368 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3445 8 365281 365273 0.828294 0 - 0.0299513 2026-02-23T14:10:52.368 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3446 8 365281 365273 0.828053 0 - 0.0299513 2026-02-23T14:10:52.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:10:52 trial068 ceph-mon[38805]: pgmap v2194: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:10:52.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:10:52 trial068 ceph-mon[38805]: 5.a scrub starts 2026-02-23T14:10:52.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:10:52 trial068 ceph-mon[38805]: 5.a scrub ok 2026-02-23T14:10:53.368 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3447 8 365281 365273 0.827813 0 - 0.0299513 2026-02-23T14:10:53.399 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:10:53.516 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:10:53.612 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:10:53.609+0000 7f24dffff640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:10:53.612 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:10:53.610+0000 7f24dffff640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:10:53.723 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:10:53.723 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":221,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:10:11.738124+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:05:13.564547+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:08:49.362470+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:09:41.751921+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:10:53.724 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:10:53 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2026-02-23T14:10:53.724 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:10:53 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-23T14:10:53.724 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:10:53 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2026-02-23T14:10:53.724 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:10:53 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T14:10:53.724 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:10:53 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2026-02-23T14:10:54.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:10:53 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:10:53] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:10:54.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:10:53 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:10:53.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:10:54.368 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3448 8 365281 365273 0.827573 0 - 0.0299513 2026-02-23T14:10:54.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:10:54 trial068 ceph-mon[38805]: pgmap v2195: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:10:54.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:10:54 trial068 ceph-mon[38805]: pgmap v2196: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:10:54.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:10:54 trial068 ceph-mon[38805]: pgmap v2197: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:10:54.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:10:54 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3576731113' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:10:55.369 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3449 8 365281 365273 0.827333 0 - 0.0299513 2026-02-23T14:10:56.369 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3450 8 365281 365273 0.827093 0 - 0.0299513 2026-02-23T14:10:56.761 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:10:56.787 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:10:56 trial068 ceph-mon[38805]: pgmap v2198: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:10:56.880 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:10:56.978 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:10:56.976+0000 7ff3992b5640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:10:56.978 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:10:56.976+0000 7ff3992b5640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:10:57.088 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:10:57.088 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":221,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:10:11.738124+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:05:13.564547+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:08:49.362470+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:09:41.751921+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:10:57.369 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3451 8 365281 365273 0.826854 0 - 0.0299513 2026-02-23T14:10:57.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:10:57 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/267660475' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:10:58.369 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3452 8 365281 365273 0.826614 0 - 0.0299513 2026-02-23T14:10:58.745 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:10:58 trial068 ceph-mon[38805]: pgmap v2199: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:10:59.369 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3453 8 365281 365273 0.826375 0 - 0.0299513 2026-02-23T14:11:00.127 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:11:00.244 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:11:00.344 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:11:00.342+0000 7f423e60d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:11:00.344 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:11:00.342+0000 7f423e60d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:11:00.369 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3454 8 365281 365273 0.826135 0 - 0.0299513 2026-02-23T14:11:00.453 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:11:00.454 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":221,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:10:11.738124+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:05:13.564547+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:08:49.362470+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:09:41.751921+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:11:00.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:11:00 trial068 ceph-mon[38805]: pgmap v2200: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:11:00.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:11:00 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3704265372' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:11:01.369 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3455 8 365281 365273 0.825896 0 - 0.0299513 2026-02-23T14:11:02.369 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3456 8 365281 365273 0.825657 0 - 0.0299513 2026-02-23T14:11:02.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:11:02 trial068 ceph-mon[38805]: pgmap v2201: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:11:03.369 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3457 8 365281 365273 0.825418 0 - 0.0299513 2026-02-23T14:11:03.491 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:11:03.611 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:11:03.714 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:11:03.712+0000 7fa6511c0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:11:03.714 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:11:03.712+0000 7fa6511c0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:11:03.826 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:11:03.826 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":221,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:10:11.738124+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:05:13.564547+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:08:49.362470+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:09:41.751921+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:11:03.827 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:11:03 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T14:11:03.827 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:11:03 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T14:11:04.094 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:11:03 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:11:03] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:11:04.266 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:11:03 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:11:03.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:11:04.369 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3458 8 365281 365273 0.82518 0 - 0.0299513 2026-02-23T14:11:04.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:11:04 trial068 ceph-mon[38805]: pgmap v2202: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:11:04.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:11:04 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3811967730' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:11:05.370 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3459 8 365281 365273 0.824941 0 - 0.0299513 2026-02-23T14:11:06.370 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T14:11:06.369298+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T14:11:06.370 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T14:11:06.370 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3460 8 365281 365273 0.824703 0 - 0.0299513 2026-02-23T14:11:06.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:11:06 trial068 ceph-mon[38805]: pgmap v2203: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:11:06.864 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:11:06.981 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:11:07.079 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:11:07.077+0000 7fbe4beed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:11:07.079 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:11:07.077+0000 7fbe4beed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:11:07.189 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:11:07.189 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":221,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:10:11.738124+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:05:13.564547+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:08:49.362470+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:09:41.751921+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:11:07.370 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3461 8 365281 365273 0.824464 0 - 0.0299513 2026-02-23T14:11:07.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:11:07 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3264786266' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:11:08.370 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3462 8 365281 365273 0.824226 0 - 0.0299513 2026-02-23T14:11:08.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:11:08 trial068 ceph-mon[38805]: pgmap v2204: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:11:09.370 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3463 8 365281 365273 0.823988 0 - 0.0299513 2026-02-23T14:11:10.227 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:11:10.347 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:11:10.370 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3464 8 365281 365273 0.82375 0 - 0.0299513 2026-02-23T14:11:10.449 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:11:10.446+0000 7f8e57bf1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:11:10.449 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:11:10.447+0000 7f8e57bf1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:11:10.557 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:11:10.557 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":221,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:10:11.738124+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:05:13.564547+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:08:49.362470+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:09:41.751921+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:11:10.557 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:11:10 trial068 ceph-mon[38805]: pgmap v2205: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:11:11.370 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3465 8 365281 365273 0.823513 0 - 0.0299513 2026-02-23T14:11:11.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:11:11 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1683488473' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:11:12.370 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3466 8 365281 365273 0.823275 0 - 0.0299513 2026-02-23T14:11:12.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:11:12 trial068 ceph-mon[38805]: pgmap v2206: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:11:13.370 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3467 8 365281 365273 0.823038 0 - 0.0299513 2026-02-23T14:11:13.595 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:11:13.713 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:11:13.756 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:11:13 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T14:11:13.756 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:11:13 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T14:11:13.813 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:11:13.811+0000 7f9d677b1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:11:13.814 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:11:13.812+0000 7f9d677b1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:11:13.923 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:11:13.923 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":221,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:10:11.738124+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:05:13.564547+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:08:49.362470+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:09:41.751921+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:11:14.094 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:11:13 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:11:13] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:11:14.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:11:13 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:11:13.972Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:11:14.370 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3468 8 365281 365273 0.8228 0 - 0.0299513 2026-02-23T14:11:14.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:11:14 trial068 ceph-mon[38805]: pgmap v2207: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:11:14.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:11:14 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1846152941' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:11:15.370 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3469 8 365281 365273 0.822563 0 - 0.0299513 2026-02-23T14:11:16.370 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3470 8 365281 365273 0.822326 0 - 0.0299513 2026-02-23T14:11:16.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:11:16 trial068 ceph-mon[38805]: pgmap v2208: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:11:16.960 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:11:17.076 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:11:17.173 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:11:17.171+0000 7f66d0c7d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:11:17.174 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:11:17.172+0000 7f66d0c7d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:11:17.283 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:11:17.283 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":221,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:10:11.738124+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:05:13.564547+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:08:49.362470+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:09:41.751921+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:11:17.370 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3471 8 365281 365273 0.822089 0 - 0.0299513 2026-02-23T14:11:17.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:11:17 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3291387682' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:11:18.370 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3472 8 365281 365273 0.821852 0 - 0.0299513 2026-02-23T14:11:18.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:11:18 trial068 ceph-mon[38805]: pgmap v2209: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:11:19.371 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3473 8 365281 365273 0.821616 0 - 0.0299513 2026-02-23T14:11:20.320 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:11:20.371 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3474 8 365281 365273 0.821379 0 - 0.0299513 2026-02-23T14:11:20.436 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:11:20.539 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:11:20.537+0000 7f7b8d7e8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:11:20.539 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:11:20.537+0000 7f7b8d7e8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:11:20.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:11:20 trial068 ceph-mon[38805]: pgmap v2210: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:11:20.647 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:11:20.647 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":222,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:11:20.508259+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:11:02.003888+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:08:49.362470+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:09:41.751921+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:11:21.371 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3475 8 365281 365273 0.821143 0 - 0.0299513 2026-02-23T14:11:21.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:11:21 trial068 ceph-mon[38805]: osdmap e222: 4 total, 3 up, 4 in 2026-02-23T14:11:21.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:11:21 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3206260223' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:11:22.371 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3476 8 365281 365273 0.820907 0 - 0.0299513 2026-02-23T14:11:22.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:11:22 trial068 ceph-mon[38805]: pgmap v2212: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:11:23.371 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3477 8 365281 365273 0.820671 0 - 0.0299513 2026-02-23T14:11:23.686 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:11:23.805 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:11:23.905 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:11:23.903+0000 7f91e43b1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:11:23.905 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:11:23.903+0000 7f91e43b1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:11:24.016 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:11:24.016 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":222,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:11:20.508259+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:11:02.003888+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:08:49.362470+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:09:41.751921+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:11:24.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:11:23 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:11:23] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:11:24.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:11:23 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:11:23.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:11:24.371 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3478 8 365281 365273 0.820435 0 - 0.0299513 2026-02-23T14:11:24.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:11:24 trial068 ceph-mon[38805]: pgmap v2213: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:11:24.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:11:24 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2288554535' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:11:25.371 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3479 8 365281 365273 0.820199 0 - 0.0299513 2026-02-23T14:11:26.371 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T14:11:26.370593+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T14:11:26.371 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T14:11:26.371 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3480 8 365281 365273 0.819963 0 - 0.0299513 2026-02-23T14:11:26.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:11:26 trial068 ceph-mon[38805]: pgmap v2214: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:11:27.054 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:11:27.174 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:11:27.270 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:11:27.268+0000 7f06016b3640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:11:27.271 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:11:27.268+0000 7f06016b3640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:11:27.371 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3481 8 365281 365273 0.819728 0 - 0.0299513 2026-02-23T14:11:27.379 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:11:27.379 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":222,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:11:20.508259+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:11:02.003888+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:08:49.362470+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:09:41.751921+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:11:27.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:11:27 trial068 ceph-mon[38805]: 5.7 scrub starts 2026-02-23T14:11:27.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:11:27 trial068 ceph-mon[38805]: 5.7 scrub ok 2026-02-23T14:11:27.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:11:27 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/349600474' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:11:28.371 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3482 8 365281 365273 0.819492 0 - 0.0299513 2026-02-23T14:11:28.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:11:28 trial068 ceph-mon[38805]: pgmap v2215: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:11:29.371 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3483 8 365281 365273 0.819257 0 - 0.0299513 2026-02-23T14:11:29.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:11:29 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T14:11:29.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:11:29 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T14:11:30.371 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3484 8 365281 365273 0.819022 0 - 0.0299513 2026-02-23T14:11:30.416 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:11:30.535 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:11:30.634 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:11:30.632+0000 7f4b2481c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:11:30.634 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:11:30.632+0000 7f4b2481c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:11:30.743 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:11:30.744 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":222,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:11:20.508259+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:11:02.003888+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:08:49.362470+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:09:41.751921+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:11:30.744 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:11:30 trial068 ceph-mon[38805]: pgmap v2216: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:11:31.371 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3485 8 365281 365273 0.818787 0 - 0.0299513 2026-02-23T14:11:31.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:11:31 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3569549094' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:11:32.371 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3486 8 365281 365273 0.818552 0 - 0.0299513 2026-02-23T14:11:32.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:11:32 trial068 ceph-mon[38805]: 5.5 scrub starts 2026-02-23T14:11:32.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:11:32 trial068 ceph-mon[38805]: pgmap v2217: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:11:32.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:11:32 trial068 ceph-mon[38805]: 5.5 scrub ok 2026-02-23T14:11:33.371 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3487 8 365281 365273 0.818317 0 - 0.0299513 2026-02-23T14:11:33.783 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:11:33.903 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:11:34.005 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:11:34.003+0000 7f870eb27640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:11:34.005 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:11:34.003+0000 7f870eb27640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:11:34.094 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:11:33 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:11:33] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:11:34.113 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:11:34.114 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":222,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:11:20.508259+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:11:02.003888+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:08:49.362470+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:09:41.751921+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:11:34.266 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:11:33 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:11:33.972Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:11:34.371 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3488 8 365281 365273 0.818083 0 - 0.0299513 2026-02-23T14:11:34.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:11:34 trial068 ceph-mon[38805]: pgmap v2218: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:11:34.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:11:34 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/259330564' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:11:35.372 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3489 8 365281 365273 0.817848 0 - 0.0299513 2026-02-23T14:11:36.372 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3490 8 365281 365273 0.817614 0 - 0.0299513 2026-02-23T14:11:36.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:11:36 trial068 ceph-mon[38805]: pgmap v2219: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:11:37.150 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:11:37.271 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:11:37.372 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:11:37.369+0000 7ff5fd611640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:11:37.372 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:11:37.370+0000 7ff5fd611640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:11:37.372 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3491 8 365281 365273 0.81738 0 - 0.0299513 2026-02-23T14:11:37.481 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:11:37.481 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":222,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:11:20.508259+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:11:02.003888+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:08:49.362470+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:09:41.751921+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:11:37.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:11:37 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/4121271234' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:11:38.372 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3492 8 365281 365273 0.817145 0 - 0.0299513 2026-02-23T14:11:38.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:11:38 trial068 ceph-mon[38805]: pgmap v2220: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:11:39.372 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3493 8 365281 365273 0.816911 0 - 0.0299513 2026-02-23T14:11:40.372 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3494 8 365281 365273 0.816678 0 - 0.0299513 2026-02-23T14:11:40.538 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:11:40.653 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:11:40.750 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:11:40.747+0000 7f070faff640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:11:40.750 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:11:40.748+0000 7f070faff640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:11:40.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:11:40 trial068 ceph-mon[38805]: pgmap v2221: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:11:40.861 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:11:40.861 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":222,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:11:20.508259+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:11:02.003888+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:08:49.362470+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:09:41.751921+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:11:41.372 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3495 8 365281 365273 0.816444 0 - 0.0299513 2026-02-23T14:11:41.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:11:41 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3295654683' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:11:42.372 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3496 8 365281 365273 0.81621 0 - 0.0299513 2026-02-23T14:11:42.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:11:42 trial068 ceph-mon[38805]: pgmap v2222: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:11:43.372 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3497 8 365281 365273 0.815977 0 - 0.0299513 2026-02-23T14:11:43.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:11:43 trial068 ceph-mon[38805]: 5.e scrub starts 2026-02-23T14:11:43.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:11:43 trial068 ceph-mon[38805]: 5.e scrub ok 2026-02-23T14:11:43.898 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:11:44.017 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:11:44.123 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:11:44.120+0000 7efff7dfc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:11:44.123 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:11:44.121+0000 7efff7dfc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:11:44.232 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:11:44.232 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":222,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:11:20.508259+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:11:02.003888+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:08:49.362470+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:09:41.751921+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:11:44.232 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:11:43 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:11:43] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:11:44.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:11:43 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:11:43.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:11:44.372 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3498 8 365281 365273 0.815744 0 - 0.0299513 2026-02-23T14:11:44.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:11:44 trial068 ceph-mon[38805]: pgmap v2223: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:11:44.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:11:44 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3552296895' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:11:45.372 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3499 8 365281 365273 0.815511 0 - 0.0299513 2026-02-23T14:11:45.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:11:45 trial068 ceph-mon[38805]: 5.8 scrub starts 2026-02-23T14:11:45.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:11:45 trial068 ceph-mon[38805]: 5.8 scrub ok 2026-02-23T14:11:46.372 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T14:11:46.371942+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T14:11:46.373 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T14:11:46.373 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3500 8 365281 365273 0.815278 0 - 0.0299513 2026-02-23T14:11:46.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:11:46 trial068 ceph-mon[38805]: pgmap v2224: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:11:47.269 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:11:47.372 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3501 8 365281 365273 0.815045 0 - 0.0299513 2026-02-23T14:11:47.391 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:11:47.489 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:11:47.487+0000 7f4863536640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:11:47.489 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:11:47.487+0000 7f4863536640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:11:47.598 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:11:47.598 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":222,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:11:20.508259+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:11:02.003888+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:08:49.362470+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:09:41.751921+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:11:48.373 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3502 8 365281 365273 0.814812 0 - 0.0299513 2026-02-23T14:11:48.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:11:48 trial068 ceph-mon[38805]: pgmap v2225: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:11:48.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:11:48 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2348648093' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:11:49.373 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3503 8 365281 365273 0.814579 0 - 0.0299513 2026-02-23T14:11:50.373 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3504 8 365281 365273 0.814347 0 - 0.0299513 2026-02-23T14:11:50.635 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:11:50.753 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:11:50.796 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:11:50 trial068 ceph-mon[38805]: pgmap v2226: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:11:50.855 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:11:50.852+0000 7f7651193640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:11:50.855 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:11:50.853+0000 7f7651193640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:11:50.965 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:11:50.966 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":222,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:11:20.508259+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:11:02.003888+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:08:49.362470+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:09:41.751921+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:11:51.373 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3505 8 365281 365273 0.814115 0 - 0.0299513 2026-02-23T14:11:51.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:11:51 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/4091077535' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:11:52.373 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3506 8 365281 365273 0.813882 0 - 0.0299513 2026-02-23T14:11:52.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:11:52 trial068 ceph-mon[38805]: pgmap v2227: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:11:53.373 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3507 8 365281 365273 0.81365 0 - 0.0299513 2026-02-23T14:11:53.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:11:53 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2026-02-23T14:11:53.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:11:53 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-23T14:11:53.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:11:53 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2026-02-23T14:11:53.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:11:53 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T14:11:53.845 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:11:53 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2026-02-23T14:11:54.002 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:11:54.119 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:11:54.220 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:11:54.218+0000 7f62bcdf3640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:11:54.220 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:11:54.218+0000 7f62bcdf3640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:11:54.233 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:11:53 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:11:53] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:11:54.331 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:11:54.331 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":222,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:11:20.508259+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:11:02.003888+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:08:49.362470+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:09:41.751921+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:11:54.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:11:53 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:11:53.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:11:54.373 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3508 8 365281 365273 0.813418 0 - 0.0299513 2026-02-23T14:11:54.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:11:54 trial068 ceph-mon[38805]: pgmap v2228: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:11:54.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:11:54 trial068 ceph-mon[38805]: pgmap v2229: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:11:54.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:11:54 trial068 ceph-mon[38805]: pgmap v2230: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:11:54.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:11:54 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1021286156' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:11:55.373 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3509 8 365281 365273 0.813187 0 - 0.0299513 2026-02-23T14:11:56.373 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3510 8 365281 365273 0.812955 0 - 0.0299513 2026-02-23T14:11:56.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:11:56 trial068 ceph-mon[38805]: pgmap v2231: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:11:57.369 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:11:57.373 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3511 8 365281 365273 0.812723 0 - 0.0299513 2026-02-23T14:11:57.485 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:11:57.586 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:11:57.583+0000 7fb49db0d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:11:57.586 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:11:57.583+0000 7fb49db0d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:11:57.695 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:11:57.695 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":222,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:11:20.508259+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:11:02.003888+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:08:49.362470+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:09:41.751921+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:11:58.373 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3512 8 365281 365273 0.812492 0 - 0.0299513 2026-02-23T14:11:58.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:11:58 trial068 ceph-mon[38805]: pgmap v2232: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:11:58.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:11:58 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2688699426' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:11:59.373 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3513 8 365281 365273 0.812261 0 - 0.0299513 2026-02-23T14:12:00.373 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3514 8 365281 365273 0.81203 0 - 0.0299513 2026-02-23T14:12:00.733 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:12:00.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:12:00 trial068 ceph-mon[38805]: pgmap v2233: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:12:00.852 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:12:00.954 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:12:00.952+0000 7fb610fa9640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:12:00.954 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:12:00.952+0000 7fb610fa9640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:12:01.062 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:12:01.063 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":222,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:11:20.508259+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:11:02.003888+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:08:49.362470+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:09:41.751921+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:12:01.373 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3515 8 365281 365273 0.811799 0 - 0.0299513 2026-02-23T14:12:01.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:12:01 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2068816302' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:12:02.374 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3516 8 365281 365273 0.811568 0 - 0.0299513 2026-02-23T14:12:02.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:12:02 trial068 ceph-mon[38805]: pgmap v2234: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:12:03.373 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3517 8 365281 365273 0.811337 0 - 0.0299513 2026-02-23T14:12:04.099 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:12:04.216 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:12:04.239 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:12:03 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:12:03] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:12:04.266 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:12:03 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:12:03.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:12:04.318 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:12:04.316+0000 7f0d43bd2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:12:04.318 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:12:04.316+0000 7f0d43bd2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:12:04.373 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3518 8 365281 365273 0.811106 0 - 0.0299513 2026-02-23T14:12:04.427 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:12:04.427 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":222,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:11:20.508259+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:11:02.003888+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:08:49.362470+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:09:41.751921+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:12:04.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:12:04 trial068 ceph-mon[38805]: pgmap v2235: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:12:04.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:12:04 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1391244650' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:12:05.374 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3519 8 365281 365273 0.810876 0 - 0.0299513 2026-02-23T14:12:06.374 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T14:12:06.373272+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T14:12:06.374 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T14:12:06.374 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3520 8 365281 365273 0.810645 0 - 0.0299513 2026-02-23T14:12:06.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:12:06 trial068 ceph-mon[38805]: pgmap v2236: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:12:07.374 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3521 8 365281 365273 0.810415 0 - 0.0299513 2026-02-23T14:12:07.464 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:12:07.583 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:12:07.686 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:12:07.683+0000 7f2e11cf0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:12:07.686 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:12:07.684+0000 7f2e11cf0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:12:07.797 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:12:07.797 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":222,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:11:20.508259+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:11:02.003888+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:08:49.362470+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:09:41.751921+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:12:08.374 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3522 8 365281 365273 0.810185 0 - 0.0299513 2026-02-23T14:12:08.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:12:08 trial068 ceph-mon[38805]: pgmap v2237: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:12:08.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:12:08 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3886205597' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:12:09.374 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3523 8 365281 365273 0.809955 0 - 0.0299513 2026-02-23T14:12:10.374 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3524 8 365281 365273 0.809725 0 - 0.0299513 2026-02-23T14:12:10.835 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:12:10.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:12:10 trial068 ceph-mon[38805]: pgmap v2238: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:12:10.953 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:12:11.052 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:12:11.050+0000 7f6bedfd9640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:12:11.053 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:12:11.050+0000 7f6bedfd9640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:12:11.163 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:12:11.163 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":222,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:11:20.508259+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:11:02.003888+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:08:49.362470+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:09:41.751921+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:12:11.374 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3525 8 365281 365273 0.809496 0 - 0.0299513 2026-02-23T14:12:11.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:12:11 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/427427636' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:12:12.374 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3526 8 365281 365273 0.809266 0 - 0.0299513 2026-02-23T14:12:12.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:12:12 trial068 ceph-mon[38805]: pgmap v2239: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:12:13.374 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3527 8 365281 365273 0.809037 0 - 0.0299513 2026-02-23T14:12:14.202 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:12:14.228 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:12:13 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:12:13] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:12:14.322 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:12:14.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:12:13 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:12:13.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:12:14.374 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3528 8 365281 365273 0.808807 0 - 0.0299513 2026-02-23T14:12:14.424 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:12:14.422+0000 7f008558a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:12:14.424 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:12:14.422+0000 7f008558a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:12:14.536 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:12:14.537 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":222,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:11:20.508259+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:11:02.003888+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:08:49.362470+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:09:41.751921+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:12:14.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:12:14 trial068 ceph-mon[38805]: pgmap v2240: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:12:14.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:12:14 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1941160167' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:12:15.374 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3529 8 365281 365273 0.808578 0 - 0.0299513 2026-02-23T14:12:16.374 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3530 8 365281 365273 0.808349 0 - 0.0299513 2026-02-23T14:12:16.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:12:16 trial068 ceph-mon[38805]: pgmap v2241: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:12:17.374 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3531 8 365281 365273 0.80812 0 - 0.0299513 2026-02-23T14:12:17.578 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:12:17.694 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:12:17.789 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:12:17.787+0000 7f8af7d89640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:12:17.789 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:12:17.787+0000 7f8af7d89640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:12:17.897 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:12:17.898 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":222,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:11:20.508259+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:11:02.003888+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:08:49.362470+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:09:41.751921+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:12:18.375 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3532 8 365281 365273 0.807891 0 - 0.0299513 2026-02-23T14:12:18.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:12:18 trial068 ceph-mon[38805]: pgmap v2242: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:12:18.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:12:18 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2478790058' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:12:19.374 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3533 8 365281 365273 0.807663 0 - 0.0299513 2026-02-23T14:12:20.375 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3534 8 365281 365273 0.807434 0 - 0.0299513 2026-02-23T14:12:20.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:12:20 trial068 ceph-mon[38805]: pgmap v2243: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:12:20.935 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:12:21.053 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:12:21.155 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:12:21.152+0000 7ffb7220f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:12:21.155 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:12:21.153+0000 7ffb7220f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:12:21.267 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:12:21.268 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":222,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:11:20.508259+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:11:02.003888+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:08:49.362470+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:09:41.751921+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:12:21.375 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3535 8 365281 365273 0.807206 0 - 0.0299513 2026-02-23T14:12:21.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:12:21 trial068 ceph-mon[38805]: 5.a scrub starts 2026-02-23T14:12:21.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:12:21 trial068 ceph-mon[38805]: 5.a scrub ok 2026-02-23T14:12:21.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:12:21 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1593539222' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:12:22.375 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3536 8 365281 365273 0.806977 0 - 0.0299513 2026-02-23T14:12:22.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:12:22 trial068 ceph-mon[38805]: pgmap v2244: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:12:23.375 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3537 8 365281 365273 0.806749 0 - 0.0299513 2026-02-23T14:12:24.305 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:12:24.330 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:12:23 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:12:23] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:12:24.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:12:23 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:12:23.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:12:24.375 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3538 8 365281 365273 0.806521 0 - 0.0299513 2026-02-23T14:12:24.423 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:12:24.527 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:12:24.525+0000 7f7a77212640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:12:24.527 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:12:24.525+0000 7f7a77212640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:12:24.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:12:24 trial068 ceph-mon[38805]: pgmap v2245: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:12:24.636 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:12:24.636 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":222,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:11:20.508259+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:11:02.003888+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:08:49.362470+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:09:41.751921+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:12:25.375 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3539 8 365281 365273 0.806293 0 - 0.0299513 2026-02-23T14:12:25.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:12:25 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/4142006697' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:12:26.375 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T14:12:26.374740+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T14:12:26.375 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T14:12:26.376 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3540 8 365281 365273 0.806066 0 - 0.0299513 2026-02-23T14:12:26.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:12:26 trial068 ceph-mon[38805]: pgmap v2246: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:12:27.375 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3541 8 365281 365273 0.805838 0 - 0.0299513 2026-02-23T14:12:27.674 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:12:27.795 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:12:27.897 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:12:27.894+0000 7feb95903640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:12:27.897 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:12:27.895+0000 7feb95903640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:12:28.005 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:12:28.006 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":222,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:11:20.508259+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:11:02.003888+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:08:49.362470+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:09:41.751921+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:12:28.375 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3542 8 365281 365273 0.80561 0 - 0.0299513 2026-02-23T14:12:28.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:12:28 trial068 ceph-mon[38805]: pgmap v2247: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:12:28.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:12:28 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3980910525' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:12:29.375 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3543 8 365281 365273 0.805383 0 - 0.0299513 2026-02-23T14:12:29.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:12:29 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T14:12:29.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:12:29 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T14:12:30.375 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3544 8 365281 365273 0.805156 0 - 0.0299513 2026-02-23T14:12:30.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:12:30 trial068 ceph-mon[38805]: pgmap v2248: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:12:31.044 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:12:31.163 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:12:31.262 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:12:31.260+0000 7fe32a44d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:12:31.262 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:12:31.260+0000 7fe32a44d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:12:31.370 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:12:31.370 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":222,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:11:20.508259+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:11:02.003888+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:08:49.362470+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:09:41.751921+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:12:31.375 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3545 8 365281 365273 0.804929 0 - 0.0299513 2026-02-23T14:12:31.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:12:31 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3461864148' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:12:32.375 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3546 8 365281 365273 0.804702 0 - 0.0299513 2026-02-23T14:12:32.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:12:32 trial068 ceph-mon[38805]: pgmap v2249: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:12:33.376 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3547 8 365281 365273 0.804475 0 - 0.0299513 2026-02-23T14:12:34.266 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:12:33 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:12:33.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:12:34.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:12:33 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:12:33] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:12:34.376 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3548 8 365281 365273 0.804248 0 - 0.0299513 2026-02-23T14:12:34.407 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:12:34.527 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:12:34.630 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:12:34.627+0000 7f586cb90640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:12:34.630 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:12:34.628+0000 7f586cb90640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:12:34.739 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:12:34.739 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":222,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:11:20.508259+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:11:02.003888+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:08:49.362470+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:09:41.751921+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:12:34.739 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:12:34 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T14:12:34.740 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:12:34 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T14:12:34.740 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:12:34 trial068 ceph-mon[38805]: pgmap v2250: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:12:35.376 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3549 8 365281 365273 0.804021 0 - 0.0299513 2026-02-23T14:12:35.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:12:35 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T14:12:35.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:12:35 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T14:12:35.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:12:35 trial068 ceph-mon[38805]: 5.5 scrub starts 2026-02-23T14:12:35.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:12:35 trial068 ceph-mon[38805]: 5.5 scrub ok 2026-02-23T14:12:35.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:12:35 trial068 ceph-mon[38805]: 5.7 scrub starts 2026-02-23T14:12:35.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:12:35 trial068 ceph-mon[38805]: 5.7 scrub ok 2026-02-23T14:12:35.845 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:12:35 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3407715624' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:12:36.376 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3550 8 365281 365273 0.803795 0 - 0.0299513 2026-02-23T14:12:36.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:12:36 trial068 ceph-mon[38805]: pgmap v2251: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:12:37.376 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3551 8 365281 365273 0.803569 0 - 0.0299513 2026-02-23T14:12:37.777 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:12:37.894 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:12:37.991 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:12:37.989+0000 7f4456d77640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:12:37.991 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:12:37.989+0000 7f4456d77640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:12:38.100 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:12:38.101 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":222,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:11:20.508259+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:11:02.003888+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:08:49.362470+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:09:41.751921+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:12:38.376 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3552 8 365281 365273 0.803342 0 - 0.0299513 2026-02-23T14:12:38.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:12:38 trial068 ceph-mon[38805]: pgmap v2252: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:12:38.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:12:38 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1485833878' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:12:39.376 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3553 8 365281 365273 0.803116 0 - 0.0299513 2026-02-23T14:12:40.376 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3554 8 365281 365273 0.80289 0 - 0.0299513 2026-02-23T14:12:40.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:12:40 trial068 ceph-mon[38805]: pgmap v2253: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:12:41.137 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:12:41.258 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:12:41.359 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:12:41.357+0000 7f63ce561640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:12:41.359 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:12:41.357+0000 7f63ce561640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:12:41.376 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3555 8 365281 365273 0.802664 0 - 0.0299513 2026-02-23T14:12:41.467 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:12:41.468 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":222,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:11:20.508259+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:11:02.003888+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:08:49.362470+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:09:41.751921+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:12:41.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:12:41 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2700759569' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:12:42.376 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3556 8 365281 365273 0.802439 0 - 0.0299513 2026-02-23T14:12:42.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:12:42 trial068 ceph-mon[38805]: pgmap v2254: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:12:43.376 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3557 8 365281 365273 0.802213 0 - 0.0299513 2026-02-23T14:12:44.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:12:43 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:12:43] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:12:44.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:12:43 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:12:43.974Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:12:44.376 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3558 8 365281 365273 0.801988 0 - 0.0299513 2026-02-23T14:12:44.505 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:12:44.625 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:12:44.728 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:12:44.725+0000 7f369bce5640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:12:44.728 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:12:44.726+0000 7f369bce5640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:12:44.836 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:12:44.836 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":222,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:11:20.508259+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:11:02.003888+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:08:49.362470+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:09:41.751921+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:12:44.837 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:12:44 trial068 ceph-mon[38805]: pgmap v2255: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:12:45.376 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3559 8 365281 365273 0.801762 0 - 0.0299513 2026-02-23T14:12:45.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:12:45 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1297764436' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:12:46.376 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T14:12:46.376140+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T14:12:46.377 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T14:12:46.377 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3560 8 365281 365273 0.801537 0 - 0.0299513 2026-02-23T14:12:47.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:12:46 trial068 ceph-mon[38805]: pgmap v2256: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:12:47.377 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3561 8 365281 365273 0.801312 0 - 0.0299513 2026-02-23T14:12:47.874 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:12:47.993 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:12:48.094 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:12:48.092+0000 7fe9b9a42640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:12:48.094 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:12:48.092+0000 7fe9b9a42640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:12:48.206 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:12:48.206 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":222,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:11:20.508259+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:11:02.003888+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:08:49.362470+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:09:41.751921+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:12:48.377 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3562 8 365281 365273 0.801087 0 - 0.0299513 2026-02-23T14:12:49.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:12:48 trial068 ceph-mon[38805]: pgmap v2257: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:12:49.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:12:48 trial068 ceph-mon[38805]: 5.e scrub starts 2026-02-23T14:12:49.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:12:48 trial068 ceph-mon[38805]: 5.e scrub ok 2026-02-23T14:12:49.095 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:12:48 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3090850424' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:12:49.377 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3563 8 365281 365273 0.800862 0 - 0.0299513 2026-02-23T14:12:50.377 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3564 8 365281 365273 0.800638 0 - 0.0299513 2026-02-23T14:12:51.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:12:50 trial068 ceph-mon[38805]: pgmap v2258: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:12:51.245 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:12:51.363 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:12:51.377 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3565 8 365281 365273 0.800413 0 - 0.0299513 2026-02-23T14:12:51.461 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:12:51.459+0000 7f00a1c99640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:12:51.462 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:12:51.459+0000 7f00a1c99640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:12:51.571 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:12:51.571 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":222,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:11:20.508259+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:11:02.003888+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:08:49.362470+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:09:41.751921+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:12:51.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:12:51 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1796182037' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:12:52.377 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3566 8 365281 365273 0.800189 0 - 0.0299513 2026-02-23T14:12:53.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:12:52 trial068 ceph-mon[38805]: pgmap v2259: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:12:53.377 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3567 8 365281 365273 0.799964 0 - 0.0299513 2026-02-23T14:12:53.975 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:12:53 trial068 ceph-mon[38805]: 5.8 scrub starts 2026-02-23T14:12:53.975 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:12:53 trial068 ceph-mon[38805]: 5.8 scrub ok 2026-02-23T14:12:53.975 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:12:53 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2026-02-23T14:12:54.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:12:53 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:12:53] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:12:54.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:12:53 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:12:53.974Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:12:54.377 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3568 8 365281 365273 0.79974 0 - 0.0299513 2026-02-23T14:12:54.609 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:12:54.724 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:12:54.828 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:12:54.825+0000 7fec92534640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:12:54.828 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:12:54.826+0000 7fec92534640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:12:54.937 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:12:54.937 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":222,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:11:20.508259+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:11:02.003888+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:08:49.362470+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:09:41.751921+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:12:54.937 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:12:54 trial068 ceph-mon[38805]: pgmap v2260: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:12:54.937 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:12:54 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-23T14:12:54.938 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:12:54 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2026-02-23T14:12:54.938 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:12:54 trial068 ceph-mon[38805]: pgmap v2261: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:12:54.938 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:12:54 trial068 ceph-mon[38805]: pgmap v2262: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:12:54.938 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:12:54 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T14:12:54.938 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:12:54 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2026-02-23T14:12:55.377 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3569 8 365281 365273 0.799516 0 - 0.0299513 2026-02-23T14:12:56.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:12:55 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3400066720' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:12:56.377 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3570 8 365281 365273 0.799292 0 - 0.0299513 2026-02-23T14:12:57.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:12:56 trial068 ceph-mon[38805]: pgmap v2263: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:12:57.377 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3571 8 365281 365273 0.799068 0 - 0.0299513 2026-02-23T14:12:57.974 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:12:58.092 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:12:58.187 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:12:58.185+0000 7f75005bd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:12:58.187 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:12:58.185+0000 7f75005bd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:12:58.296 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:12:58.296 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":222,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:11:20.508259+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:11:02.003888+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:08:49.362470+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:09:41.751921+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:12:58.377 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3572 8 365281 365273 0.798844 0 - 0.0299513 2026-02-23T14:12:59.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:12:58 trial068 ceph-mon[38805]: pgmap v2264: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:12:59.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:12:58 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2628844016' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:12:59.377 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3573 8 365281 365273 0.798621 0 - 0.0299513 2026-02-23T14:13:00.377 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3574 8 365281 365273 0.798397 0 - 0.0299513 2026-02-23T14:13:01.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:13:00 trial068 ceph-mon[38805]: pgmap v2265: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:13:01.333 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:13:01.377 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3575 8 365281 365273 0.798174 0 - 0.0299513 2026-02-23T14:13:01.453 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:13:01.557 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:13:01.555+0000 7fc62f450640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:13:01.557 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:13:01.555+0000 7fc62f450640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:13:01.667 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:13:01.667 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":222,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:11:20.508259+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:11:02.003888+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:08:49.362470+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:09:41.751921+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:13:02.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:13:01 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/747000614' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:13:02.377 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3576 8 365281 365273 0.797951 0 - 0.0299513 2026-02-23T14:13:03.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:13:02 trial068 ceph-mon[38805]: pgmap v2266: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:13:03.378 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3577 8 365281 365273 0.797728 0 - 0.0299513 2026-02-23T14:13:04.266 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:13:03 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:13:03.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:13:04.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:13:03 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:13:03] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:13:04.378 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3578 8 365281 365273 0.797505 0 - 0.0299513 2026-02-23T14:13:04.704 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:13:04.823 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:13:04.925 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:13:04.923+0000 7ff5b9538640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:13:04.925 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:13:04.923+0000 7ff5b9538640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:13:05.035 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:13:05.035 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":222,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:11:20.508259+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:11:02.003888+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:08:49.362470+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:09:41.751921+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:13:05.036 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:13:04 trial068 ceph-mon[38805]: pgmap v2267: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:13:05.378 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3579 8 365281 365273 0.797282 0 - 0.0299513 2026-02-23T14:13:06.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:13:05 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3158841319' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:13:06.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:13:05 trial068 ceph-mon[38805]: pgmap v2268: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:13:06.378 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T14:13:06.377405+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T14:13:06.378 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T14:13:06.378 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3580 8 365281 365273 0.797059 0 - 0.0299513 2026-02-23T14:13:07.378 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3581 8 365281 365273 0.796837 0 - 0.0299513 2026-02-23T14:13:08.073 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:13:08.192 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:13:08.293 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:13:08.290+0000 7f19e23cc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:13:08.293 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:13:08.291+0000 7f19e23cc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:13:08.378 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3582 8 365281 365273 0.796614 0 - 0.0299513 2026-02-23T14:13:08.404 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:13:08.404 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":222,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:11:20.508259+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:11:02.003888+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:08:49.362470+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:09:41.751921+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:13:09.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:13:08 trial068 ceph-mon[38805]: pgmap v2269: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:13:09.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:13:08 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3209340360' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:13:09.378 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3583 8 365281 365273 0.796392 0 - 0.0299513 2026-02-23T14:13:10.378 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3584 8 365281 365273 0.79617 0 - 0.0299513 2026-02-23T14:13:11.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:13:10 trial068 ceph-mon[38805]: pgmap v2270: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:13:11.378 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3585 8 365281 365273 0.795948 0 - 0.0299513 2026-02-23T14:13:11.443 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:13:11.560 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:13:11.663 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:13:11.661+0000 7f9e769a7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:13:11.663 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:13:11.661+0000 7f9e769a7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:13:11.772 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:13:11.773 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":222,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:11:20.508259+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:11:02.003888+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:08:49.362470+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:09:41.751921+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:13:12.379 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3586 8 365281 365273 0.795726 0 - 0.0299513 2026-02-23T14:13:13.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:13:12 trial068 ceph-mon[38805]: pgmap v2271: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:13:13.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:13:12 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1310218160' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:13:13.379 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3587 8 365281 365273 0.795504 0 - 0.0299513 2026-02-23T14:13:14.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:13:13 trial068 ceph-mon[38805]: pgmap v2272: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:13:14.094 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:13:13 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:13:13] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:13:14.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:13:13 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:13:13.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:13:14.378 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3588 8 365281 365273 0.795282 0 - 0.0299513 2026-02-23T14:13:14.809 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:13:14.928 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:13:15.033 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:13:15.030+0000 7f660076e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:13:15.033 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:13:15.031+0000 7f660076e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:13:15.143 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:13:15.144 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":222,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:11:20.508259+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:11:02.003888+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:08:49.362470+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:09:41.751921+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:13:15.379 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3589 8 365281 365273 0.795061 0 - 0.0299513 2026-02-23T14:13:15.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:13:15 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/4115747771' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:13:16.379 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3590 8 365281 365273 0.794839 0 - 0.0299513 2026-02-23T14:13:16.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:13:16 trial068 ceph-mon[38805]: pgmap v2273: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:13:17.379 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3591 8 365281 365273 0.794618 0 - 0.0299513 2026-02-23T14:13:18.181 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:13:18.298 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:13:18.379 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3592 8 365281 365273 0.794397 0 - 0.0299513 2026-02-23T14:13:18.398 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:13:18.396+0000 7fc907de4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:13:18.398 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:13:18.396+0000 7fc907de4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:13:18.510 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:13:18.510 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":222,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:11:20.508259+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:11:02.003888+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:08:49.362470+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:09:41.751921+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:13:19.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:13:18 trial068 ceph-mon[38805]: pgmap v2274: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:13:19.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:13:18 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/607066048' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:13:19.379 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3593 8 365281 365273 0.794175 0 - 0.0299513 2026-02-23T14:13:20.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:13:19 trial068 ceph-mon[38805]: pgmap v2275: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:13:20.379 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3594 8 365281 365273 0.793954 0 - 0.0299513 2026-02-23T14:13:21.379 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3595 8 365281 365273 0.793734 0 - 0.0299513 2026-02-23T14:13:21.547 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:13:21.664 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:13:21.768 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:13:21.766+0000 7f3281021640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:13:21.768 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:13:21.766+0000 7f3281021640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:13:21.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:13:21 trial068 ceph-mon[38805]: pgmap v2276: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:13:21.879 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:13:21.879 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":222,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:11:20.508259+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:11:02.003888+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:08:49.362470+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:09:41.751921+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:13:22.379 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3596 8 365281 365273 0.793513 0 - 0.0299513 2026-02-23T14:13:23.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:13:22 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/4272766053' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:13:23.379 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3597 8 365281 365273 0.793292 0 - 0.0299513 2026-02-23T14:13:24.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:13:23 trial068 ceph-mon[38805]: pgmap v2277: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:13:24.094 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:13:23 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:13:23] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:13:24.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:13:23 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:13:23.974Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:13:24.379 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3598 8 365281 365273 0.793072 0 - 0.0299513 2026-02-23T14:13:24.916 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:13:25.033 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:13:25.133 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:13:25.131+0000 7f7f9bd57640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:13:25.133 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:13:25.131+0000 7f7f9bd57640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:13:25.244 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:13:25.244 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":222,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:11:20.508259+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:11:02.003888+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:08:49.362470+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:09:41.751921+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:13:25.379 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3599 8 365281 365273 0.792851 0 - 0.0299513 2026-02-23T14:13:25.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:13:25 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/4113027852' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:13:26.380 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T14:13:26.379079+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T14:13:26.380 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T14:13:26.380 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3600 8 365281 365273 0.792631 0 - 0.0299513 2026-02-23T14:13:26.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:13:26 trial068 ceph-mon[38805]: pgmap v2278: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:13:27.380 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3601 8 365281 365273 0.792411 0 - 0.0299513 2026-02-23T14:13:28.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:13:27 trial068 ceph-mon[38805]: pgmap v2279: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:13:28.301 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:13:28.380 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3602 8 365281 365273 0.792191 0 - 0.0299513 2026-02-23T14:13:28.419 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:13:28.528 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:13:28.526+0000 7f761d9bc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:13:28.528 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:13:28.526+0000 7f761d9bc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:13:28.636 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:13:28.636 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":222,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:11:20.508259+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:11:02.003888+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:08:49.362470+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:09:41.751921+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:13:29.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:13:28 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1065272865' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:13:29.380 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3603 8 365281 365273 0.791971 0 - 0.0299513 2026-02-23T14:13:30.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:13:29 trial068 ceph-mon[38805]: pgmap v2280: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:13:30.380 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3604 8 365281 365273 0.791752 0 - 0.0299513 2026-02-23T14:13:31.380 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3605 8 365281 365273 0.791532 0 - 0.0299513 2026-02-23T14:13:31.673 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:13:31.792 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:13:31.892 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:13:31.890+0000 7f804b040640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:13:31.892 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:13:31.890+0000 7f804b040640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:13:32.002 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:13:32.003 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":222,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:11:20.508259+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:11:02.003888+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:08:49.362470+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:09:41.751921+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:13:32.003 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:13:31 trial068 ceph-mon[38805]: pgmap v2281: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:13:32.380 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3606 8 365281 365273 0.791312 0 - 0.0299513 2026-02-23T14:13:33.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:13:32 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1462940651' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:13:33.380 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3607 8 365281 365273 0.791093 0 - 0.0299513 2026-02-23T14:13:34.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:13:33 trial068 ceph-mon[38805]: pgmap v2282: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:13:34.094 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:13:33 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:13:33] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:13:34.266 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:13:33 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:13:33.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:13:34.380 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3608 8 365281 365273 0.790874 0 - 0.0299513 2026-02-23T14:13:35.041 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:13:35.160 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:13:35.267 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:13:35.264+0000 7fce82805640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:13:35.267 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:13:35.265+0000 7fce82805640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:13:35.375 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:13:35.375 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":222,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:11:20.508259+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:11:02.003888+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:08:49.362470+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:09:41.751921+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:13:35.380 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3609 8 365281 365273 0.790655 0 - 0.0299513 2026-02-23T14:13:35.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:13:35 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/490972110' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:13:36.380 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3610 8 365281 365273 0.790436 0 - 0.0299513 2026-02-23T14:13:36.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:13:36 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T14:13:36.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:13:36 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T14:13:36.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:13:36 trial068 ceph-mon[38805]: pgmap v2283: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:13:37.380 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3611 8 365281 365273 0.790217 0 - 0.0299513 2026-02-23T14:13:38.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:13:37 trial068 ceph-mon[38805]: pgmap v2284: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:13:38.380 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3612 8 365281 365273 0.789998 0 - 0.0299513 2026-02-23T14:13:38.412 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:13:38.532 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:13:38.635 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:13:38.632+0000 7f5c7bf31640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:13:38.635 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:13:38.632+0000 7f5c7bf31640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:13:38.745 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:13:38.745 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":222,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:11:20.508259+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:11:02.003888+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:08:49.362470+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:09:41.751921+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:13:39.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:13:38 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/352158082' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:13:39.380 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3613 8 365281 365273 0.789779 0 - 0.0299513 2026-02-23T14:13:40.380 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3614 8 365281 365273 0.789561 0 - 0.0299513 2026-02-23T14:13:40.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:13:40 trial068 ceph-mon[38805]: pgmap v2285: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:13:41.380 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3615 8 365281 365273 0.789342 0 - 0.0299513 2026-02-23T14:13:41.783 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:13:41.902 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:13:42.002 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:13:42.000+0000 7fd12ed39640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:13:42.002 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:13:42.000+0000 7fd12ed39640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:13:42.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:13:41 trial068 ceph-mon[38805]: pgmap v2286: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:13:42.114 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:13:42.115 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":222,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:11:20.508259+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:11:02.003888+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:08:49.362470+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:09:41.751921+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:13:42.381 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3616 8 365281 365273 0.789124 0 - 0.0299513 2026-02-23T14:13:43.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:13:42 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2115462810' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:13:43.381 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3617 8 365281 365273 0.788906 0 - 0.0299513 2026-02-23T14:13:44.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:13:43 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T14:13:44.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:13:43 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T14:13:44.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:13:43 trial068 ceph-mon[38805]: pgmap v2287: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:13:44.094 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:13:43 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:13:43] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:13:44.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:13:43 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:13:43.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:13:44.381 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3618 8 365281 365273 0.788688 0 - 0.0299513 2026-02-23T14:13:45.171 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:13:45.290 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:13:45.381 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3619 8 365281 365273 0.78847 0 - 0.0299513 2026-02-23T14:13:45.392 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:13:45.390+0000 7f1292dd6640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:13:45.392 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:13:45.390+0000 7f1292dd6640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:13:45.502 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:13:45.503 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":222,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:11:20.508259+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:11:02.003888+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:08:49.362470+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:09:41.751921+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:13:45.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:13:45 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2344242627' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:13:46.381 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T14:13:46.380592+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T14:13:46.381 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T14:13:46.381 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3620 8 365281 365273 0.788252 0 - 0.0299513 2026-02-23T14:13:46.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:13:46 trial068 ceph-mon[38805]: pgmap v2288: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:13:47.381 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3621 8 365281 365273 0.788034 0 - 0.0299513 2026-02-23T14:13:48.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:13:47 trial068 ceph-mon[38805]: 5.a scrub starts 2026-02-23T14:13:48.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:13:47 trial068 ceph-mon[38805]: 5.a scrub ok 2026-02-23T14:13:48.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:13:47 trial068 ceph-mon[38805]: pgmap v2289: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:13:48.381 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3622 8 365281 365273 0.787817 0 - 0.0299513 2026-02-23T14:13:48.540 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:13:48.659 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:13:48.759 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:13:48.757+0000 7f125f42d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:13:48.759 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:13:48.757+0000 7f125f42d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:13:48.868 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:13:48.868 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":222,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:11:20.508259+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:11:02.003888+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:08:49.362470+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:09:41.751921+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:13:49.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:13:48 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/279055606' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:13:49.381 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3623 8 365281 365273 0.787599 0 - 0.0299513 2026-02-23T14:13:50.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:13:49 trial068 ceph-mon[38805]: 5.7 scrub starts 2026-02-23T14:13:50.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:13:49 trial068 ceph-mon[38805]: 5.7 scrub ok 2026-02-23T14:13:50.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:13:49 trial068 ceph-mon[38805]: pgmap v2290: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:13:50.381 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3624 8 365281 365273 0.787382 0 - 0.0299513 2026-02-23T14:13:51.381 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3625 8 365281 365273 0.787165 0 - 0.0299513 2026-02-23T14:13:51.905 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:13:52.024 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:13:52.069 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:13:51 trial068 ceph-mon[38805]: pgmap v2291: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:13:52.121 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:13:52.118+0000 7f06b64fb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:13:52.121 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:13:52.119+0000 7f06b64fb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:13:52.230 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:13:52.231 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":222,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:11:20.508259+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:11:02.003888+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:08:49.362470+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:09:41.751921+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:13:52.381 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3626 8 365281 365273 0.786948 0 - 0.0299513 2026-02-23T14:13:53.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:13:52 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/614601430' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:13:53.381 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3627 8 365281 365273 0.786731 0 - 0.0299513 2026-02-23T14:13:54.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:13:53 trial068 ceph-mon[38805]: pgmap v2292: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:13:54.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:13:53 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2026-02-23T14:13:54.095 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:13:53 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:13:53] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:13:54.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:13:53 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:13:53.974Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:13:54.381 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3628 8 365281 365273 0.786514 0 - 0.0299513 2026-02-23T14:13:55.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:13:54 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-23T14:13:55.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:13:54 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2026-02-23T14:13:55.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:13:54 trial068 ceph-mon[38805]: pgmap v2293: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:13:55.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:13:54 trial068 ceph-mon[38805]: pgmap v2294: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:13:55.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:13:54 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T14:13:55.095 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:13:54 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2026-02-23T14:13:55.268 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:13:55.382 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3629 8 365281 365273 0.786297 0 - 0.0299513 2026-02-23T14:13:55.383 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:13:55.487 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:13:55.485+0000 7f0918b9b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:13:55.487 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:13:55.485+0000 7f0918b9b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:13:55.597 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:13:55.597 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":222,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:11:20.508259+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:11:02.003888+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:08:49.362470+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:09:41.751921+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:13:56.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:13:55 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T14:13:56.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:13:55 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T14:13:56.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:13:55 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1764860900' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:13:56.382 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3630 8 365281 365273 0.786081 0 - 0.0299513 2026-02-23T14:13:57.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:13:56 trial068 ceph-mon[38805]: osdmap e223: 4 total, 3 up, 4 in 2026-02-23T14:13:57.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:13:56 trial068 ceph-mon[38805]: 5.5 scrub starts 2026-02-23T14:13:57.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:13:56 trial068 ceph-mon[38805]: pgmap v2296: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:13:57.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:13:56 trial068 ceph-mon[38805]: 5.5 scrub ok 2026-02-23T14:13:57.382 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3631 8 365281 365273 0.785864 0 - 0.0299513 2026-02-23T14:13:58.382 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3632 8 365281 365273 0.785648 0 - 0.0299513 2026-02-23T14:13:58.636 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:13:58.754 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:13:58.857 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:13:58.854+0000 7f9c74d6b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:13:58.857 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:13:58.855+0000 7f9c74d6b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:13:58.966 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:13:58.967 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":223,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:13:55.824287+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:11:02.003888+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:13:34.834650+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:09:41.751921+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:13:59.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:13:59 trial068 ceph-mon[38805]: pgmap v2297: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:13:59.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:13:59 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1262150280' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:13:59.382 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3633 8 365281 365273 0.785431 0 - 0.0299513 2026-02-23T14:14:00.382 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3634 8 365281 365273 0.785215 0 - 0.0299513 2026-02-23T14:14:01.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:14:01 trial068 ceph-mon[38805]: pgmap v2298: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:14:01.382 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3635 8 365281 365273 0.784999 0 - 0.0299513 2026-02-23T14:14:02.003 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:14:02.120 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:14:02.224 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:14:02.222+0000 7f85b0520640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:14:02.225 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:14:02.222+0000 7f85b0520640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:14:02.334 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:14:02.334 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":223,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:13:55.824287+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:11:02.003888+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:13:34.834650+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:09:41.751921+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:14:02.382 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3636 8 365281 365273 0.784783 0 - 0.0299513 2026-02-23T14:14:03.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:14:03 trial068 ceph-mon[38805]: pgmap v2299: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:14:03.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:14:03 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3483914681' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:14:03.382 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3637 8 365281 365273 0.784568 0 - 0.0299513 2026-02-23T14:14:04.266 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:14:03 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:14:03.972Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:14:04.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:14:03 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:14:03] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:14:04.382 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3638 8 365281 365273 0.784352 0 - 0.0299513 2026-02-23T14:14:05.370 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:14:05.382 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3639 8 365281 365273 0.784136 0 - 0.0299513 2026-02-23T14:14:05.488 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:14:05.591 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:14:05.589+0000 7f606c8af640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:14:05.591 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:14:05.589+0000 7f606c8af640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:14:05.703 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:14:05.703 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":223,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:13:55.824287+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:11:02.003888+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:13:34.834650+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:09:41.751921+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:14:05.704 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:14:05 trial068 ceph-mon[38805]: pgmap v2300: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:14:06.382 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T14:14:06.382041+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T14:14:06.383 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T14:14:06.383 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3640 8 365281 365273 0.783921 0 - 0.0299513 2026-02-23T14:14:06.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:14:06 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/4046607284' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:14:07.383 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3641 8 365281 365273 0.783706 0 - 0.0299513 2026-02-23T14:14:07.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:14:07 trial068 ceph-mon[38805]: pgmap v2301: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:14:08.383 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3642 8 365281 365273 0.783491 0 - 0.0299513 2026-02-23T14:14:08.742 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:14:08.858 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:14:08.956 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:14:08.954+0000 7f47c2b99640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:14:08.956 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:14:08.954+0000 7f47c2b99640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:14:09.066 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:14:09.066 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":223,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:13:55.824287+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:11:02.003888+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:13:34.834650+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:09:41.751921+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:14:09.383 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3643 8 365281 365273 0.783275 0 - 0.0299513 2026-02-23T14:14:09.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:14:09 trial068 ceph-mon[38805]: pgmap v2302: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:14:09.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:14:09 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3010430975' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:14:10.383 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3644 8 365281 365273 0.78306 0 - 0.0299513 2026-02-23T14:14:11.383 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3645 8 365281 365273 0.782846 0 - 0.0299513 2026-02-23T14:14:11.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:14:11 trial068 ceph-mon[38805]: pgmap v2303: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:14:12.104 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:14:12.222 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:14:12.322 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:14:12.320+0000 7f9f3f2ef640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:14:12.323 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:14:12.320+0000 7f9f3f2ef640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:14:12.383 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3646 8 365281 365273 0.782631 0 - 0.0299513 2026-02-23T14:14:12.432 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:14:12.432 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":223,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:13:55.824287+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:11:02.003888+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:13:34.834650+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:09:41.751921+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:14:12.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:14:12 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2782694029' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:14:13.383 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3647 8 365281 365273 0.782416 0 - 0.0299513 2026-02-23T14:14:13.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:14:13 trial068 ceph-mon[38805]: pgmap v2304: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:14:14.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:14:13 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:14:13] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:14:14.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:14:13 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:14:13.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:14:14.383 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3648 8 365281 365273 0.782202 0 - 0.0299513 2026-02-23T14:14:14.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:14:14 trial068 ceph-mon[38805]: 5.8 scrub starts 2026-02-23T14:14:14.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:14:14 trial068 ceph-mon[38805]: 5.8 scrub ok 2026-02-23T14:14:15.383 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3649 8 365281 365273 0.781987 0 - 0.0299513 2026-02-23T14:14:15.469 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:14:15.587 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:14:15.688 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:14:15.685+0000 7efcadd5f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:14:15.688 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:14:15.685+0000 7efcadd5f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:14:15.796 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:14:15.796 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":223,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:13:55.824287+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:11:02.003888+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:13:34.834650+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:09:41.751921+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:14:15.797 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:14:15 trial068 ceph-mon[38805]: pgmap v2305: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:14:16.384 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3650 8 365281 365273 0.781773 0 - 0.0299513 2026-02-23T14:14:16.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:14:16 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3326824679' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:14:17.384 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3651 8 365281 365273 0.781559 0 - 0.0299513 2026-02-23T14:14:17.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:14:17 trial068 ceph-mon[38805]: pgmap v2306: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:14:18.384 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3652 8 365281 365273 0.781345 0 - 0.0299513 2026-02-23T14:14:18.833 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:14:18.954 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:14:19.061 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:14:19.059+0000 7f73b9eab640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:14:19.061 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:14:19.059+0000 7f73b9eab640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:14:19.172 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:14:19.173 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":223,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:13:55.824287+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:11:02.003888+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:13:34.834650+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:09:41.751921+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:14:19.384 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3653 8 365281 365273 0.781131 0 - 0.0299513 2026-02-23T14:14:19.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:14:19 trial068 ceph-mon[38805]: pgmap v2307: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:14:19.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:14:19 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/317664953' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:14:20.384 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3654 8 365281 365273 0.780917 0 - 0.0299513 2026-02-23T14:14:20.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:14:20 trial068 ceph-mon[38805]: 5.e scrub starts 2026-02-23T14:14:20.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:14:20 trial068 ceph-mon[38805]: 5.e scrub ok 2026-02-23T14:14:21.384 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3655 8 365281 365273 0.780704 0 - 0.0299513 2026-02-23T14:14:21.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:14:21 trial068 ceph-mon[38805]: pgmap v2308: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:14:22.212 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:14:22.329 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:14:22.384 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3656 8 365281 365273 0.78049 0 - 0.0299513 2026-02-23T14:14:22.427 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:14:22.425+0000 7fdc33eb1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:14:22.427 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:14:22.425+0000 7fdc33eb1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:14:22.536 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:14:22.536 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":223,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:13:55.824287+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:11:02.003888+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:13:34.834650+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:09:41.751921+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:14:23.384 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3657 8 365281 365273 0.780277 0 - 0.0299513 2026-02-23T14:14:23.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:14:23 trial068 ceph-mon[38805]: pgmap v2309: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:14:23.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:14:23 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2045698420' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:14:24.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:14:23 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:14:23] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:14:24.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:14:23 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:14:23.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:14:24.384 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3658 8 365281 365273 0.780064 0 - 0.0299513 2026-02-23T14:14:25.384 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3659 8 365281 365273 0.77985 0 - 0.0299513 2026-02-23T14:14:25.574 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:14:25.692 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:14:25.793 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:14:25.790+0000 7fef1e8fb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:14:25.793 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:14:25.791+0000 7fef1e8fb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:14:25.793 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:14:25 trial068 ceph-mon[38805]: pgmap v2310: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:14:25.903 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:14:25.903 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":223,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:13:55.824287+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:11:02.003888+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:13:34.834650+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:09:41.751921+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:14:26.384 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T14:14:26.383783+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T14:14:26.384 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T14:14:26.385 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3660 8 365281 365273 0.779637 0 - 0.0299513 2026-02-23T14:14:26.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:14:26 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/332333399' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:14:27.384 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3661 8 365281 365273 0.779424 0 - 0.0299513 2026-02-23T14:14:27.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:14:27 trial068 ceph-mon[38805]: pgmap v2311: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:14:28.384 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3662 8 365281 365273 0.779211 0 - 0.0299513 2026-02-23T14:14:28.939 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:14:29.054 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:14:29.151 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:14:29.149+0000 7f50c9975640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:14:29.152 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:14:29.149+0000 7f50c9975640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:14:29.261 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:14:29.261 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":223,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:13:55.824287+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:11:02.003888+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:13:34.834650+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:09:41.751921+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:14:29.384 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3663 8 365281 365273 0.778999 0 - 0.0299513 2026-02-23T14:14:29.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:14:29 trial068 ceph-mon[38805]: pgmap v2312: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:14:29.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:14:29 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3884919639' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:14:30.385 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3664 8 365281 365273 0.778786 0 - 0.0299513 2026-02-23T14:14:31.385 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3665 8 365281 365273 0.778574 0 - 0.0299513 2026-02-23T14:14:31.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:14:31 trial068 ceph-mon[38805]: pgmap v2313: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:14:32.299 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:14:32.385 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3666 8 365281 365273 0.778361 0 - 0.0299513 2026-02-23T14:14:32.416 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:14:32.527 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:14:32.524+0000 7f114e01c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:14:32.527 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:14:32.525+0000 7f114e01c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:14:32.637 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:14:32.637 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":223,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:13:55.824287+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:11:02.003888+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:13:34.834650+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:09:41.751921+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:14:33.385 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3667 8 365281 365273 0.778149 0 - 0.0299513 2026-02-23T14:14:33.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:14:33 trial068 ceph-mon[38805]: pgmap v2314: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:14:33.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:14:33 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2045950300' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:14:34.266 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:14:33 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:14:33.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:14:34.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:14:33 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:14:33] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:14:34.385 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3668 8 365281 365273 0.777937 0 - 0.0299513 2026-02-23T14:14:35.385 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3669 8 365281 365273 0.777725 0 - 0.0299513 2026-02-23T14:14:35.675 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:14:35.794 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:14:35.816 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:14:35 trial068 ceph-mon[38805]: pgmap v2315: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:14:35.897 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:14:35.894+0000 7f6ecf404640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:14:35.897 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:14:35.895+0000 7f6ecf404640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:14:36.004 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:14:36.004 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":223,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:13:55.824287+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:11:02.003888+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:13:34.834650+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:09:41.751921+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:14:36.385 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3670 8 365281 365273 0.777513 0 - 0.0299513 2026-02-23T14:14:36.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:14:36 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2317996816' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:14:37.385 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3671 8 365281 365273 0.777301 0 - 0.0299513 2026-02-23T14:14:37.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:14:37 trial068 ceph-mon[38805]: pgmap v2316: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:14:38.385 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3672 8 365281 365273 0.777089 0 - 0.0299513 2026-02-23T14:14:39.041 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:14:39.158 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:14:39.260 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:14:39.258+0000 7fae35f34640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:14:39.260 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:14:39.258+0000 7fae35f34640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:14:39.369 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:14:39.369 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":223,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:13:55.824287+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:11:02.003888+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:13:34.834650+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:09:41.751921+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:14:39.385 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3673 8 365281 365273 0.776878 0 - 0.0299513 2026-02-23T14:14:39.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:14:39 trial068 ceph-mon[38805]: pgmap v2317: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:14:39.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:14:39 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3692962540' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:14:40.385 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3674 8 365281 365273 0.776666 0 - 0.0299513 2026-02-23T14:14:41.385 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3675 8 365281 365273 0.776455 0 - 0.0299513 2026-02-23T14:14:41.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:14:41 trial068 ceph-mon[38805]: pgmap v2318: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:14:42.385 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3676 8 365281 365273 0.776244 0 - 0.0299513 2026-02-23T14:14:42.406 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:14:42.523 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:14:42.623 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:14:42.620+0000 7fc362385640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:14:42.623 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:14:42.621+0000 7fc362385640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:14:42.731 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:14:42.732 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":223,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:13:55.824287+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:11:02.003888+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:13:34.834650+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:09:41.751921+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:14:43.385 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3677 8 365281 365273 0.776033 0 - 0.0299513 2026-02-23T14:14:43.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:14:43 trial068 ceph-mon[38805]: pgmap v2319: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:14:43.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:14:43 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2119036848' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:14:44.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:14:43 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:14:43] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:14:44.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:14:43 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:14:43.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:14:44.385 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3678 8 365281 365273 0.775822 0 - 0.0299513 2026-02-23T14:14:45.386 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3679 8 365281 365273 0.775611 0 - 0.0299513 2026-02-23T14:14:45.769 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:14:45.794 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:14:45 trial068 ceph-mon[38805]: pgmap v2320: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:14:45.887 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:14:45.996 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:14:45.993+0000 7fa342864640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:14:45.996 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:14:45.993+0000 7fa342864640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:14:46.110 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:14:46.110 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":223,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:13:55.824287+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:11:02.003888+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:13:34.834650+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:09:41.751921+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:14:46.386 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T14:14:46.385216+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T14:14:46.386 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T14:14:46.386 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3680 8 365281 365273 0.7754 0 - 0.0299513 2026-02-23T14:14:46.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:14:46 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2613263823' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:14:47.386 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3681 8 365281 365273 0.775189 0 - 0.0299513 2026-02-23T14:14:47.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:14:47 trial068 ceph-mon[38805]: pgmap v2321: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:14:48.386 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3682 8 365281 365273 0.774979 0 - 0.0299513 2026-02-23T14:14:49.163 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:14:49.281 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:14:49.377 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:14:49.375+0000 7fe2414df640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:14:49.378 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:14:49.376+0000 7fe2414df640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:14:49.386 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3683 8 365281 365273 0.774769 0 - 0.0299513 2026-02-23T14:14:49.495 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:14:49.495 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":223,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:13:55.824287+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:11:02.003888+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:13:34.834650+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:09:41.751921+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:14:49.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:14:49 trial068 ceph-mon[38805]: pgmap v2322: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:14:49.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:14:49 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2246477004' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:14:50.386 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3684 8 365281 365273 0.774558 0 - 0.0299513 2026-02-23T14:14:51.386 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3685 8 365281 365273 0.774348 0 - 0.0299513 2026-02-23T14:14:51.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:14:51 trial068 ceph-mon[38805]: pgmap v2323: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:14:52.386 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3686 8 365281 365273 0.774138 0 - 0.0299513 2026-02-23T14:14:52.533 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:14:52.650 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:14:52.754 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:14:52.752+0000 7fbec31d3640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:14:52.754 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:14:52.752+0000 7fbec31d3640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:14:52.864 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:14:52.864 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":223,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:13:55.824287+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:11:02.003888+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:13:34.834650+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:09:41.751921+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:14:53.386 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3687 8 365281 365273 0.773928 0 - 0.0299513 2026-02-23T14:14:53.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:14:53 trial068 ceph-mon[38805]: pgmap v2324: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:14:53.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:14:53 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T14:14:53.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:14:53 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T14:14:53.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:14:53 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/555417333' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:14:54.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:14:53 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:14:53] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:14:54.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:14:53 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:14:53.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:14:54.386 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3688 8 365281 365273 0.773718 0 - 0.0299513 2026-02-23T14:14:54.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:14:54 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2026-02-23T14:14:54.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:14:54 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-23T14:14:54.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:14:54 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2026-02-23T14:14:54.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:14:54 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T14:14:54.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:14:54 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2026-02-23T14:14:55.386 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3689 8 365281 365273 0.773508 0 - 0.0299513 2026-02-23T14:14:55.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:14:55 trial068 ceph-mon[38805]: pgmap v2325: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:14:55.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:14:55 trial068 ceph-mon[38805]: pgmap v2326: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:14:55.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:14:55 trial068 ceph-mon[38805]: pgmap v2327: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:14:55.901 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:14:56.016 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:14:56.120 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:14:56.118+0000 7fc88df83640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:14:56.120 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:14:56.118+0000 7fc88df83640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:14:56.229 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:14:56.229 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":223,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:13:55.824287+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:11:02.003888+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:13:34.834650+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:09:41.751921+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:14:56.386 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3690 8 365281 365273 0.773299 0 - 0.0299513 2026-02-23T14:14:56.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:14:56 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2658122366' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:14:57.386 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3691 8 365281 365273 0.773089 0 - 0.0299513 2026-02-23T14:14:57.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:14:57 trial068 ceph-mon[38805]: pgmap v2328: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:14:58.386 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3692 8 365281 365273 0.77288 0 - 0.0299513 2026-02-23T14:14:59.266 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:14:59.385 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:14:59.386 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3693 8 365281 365273 0.772671 0 - 0.0299513 2026-02-23T14:14:59.486 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:14:59.483+0000 7f0a12f1e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:14:59.486 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:14:59.484+0000 7f0a12f1e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:14:59.596 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:14:59.597 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":223,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:13:55.824287+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:11:02.003888+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:13:34.834650+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:09:41.751921+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:14:59.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:14:59 trial068 ceph-mon[38805]: pgmap v2329: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:15:00.387 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3694 8 365281 365273 0.772461 0 - 0.0299513 2026-02-23T14:15:00.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:15:00 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1417164788' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:15:01.387 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3695 8 365281 365273 0.772252 0 - 0.0299513 2026-02-23T14:15:01.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:15:01 trial068 ceph-mon[38805]: pgmap v2330: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:15:02.387 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3696 8 365281 365273 0.772043 0 - 0.0299513 2026-02-23T14:15:02.638 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:15:02.757 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:15:02.805 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:15:02 trial068 ceph-mon[38805]: 5.7 scrub starts 2026-02-23T14:15:02.805 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:15:02 trial068 ceph-mon[38805]: 5.a scrub starts 2026-02-23T14:15:02.805 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:15:02 trial068 ceph-mon[38805]: 5.7 scrub ok 2026-02-23T14:15:02.857 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:15:02.855+0000 7fd1c23d3640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:15:02.857 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:15:02.855+0000 7fd1c23d3640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:15:02.965 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:15:02.965 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":223,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:13:55.824287+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:11:02.003888+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:13:34.834650+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:09:41.751921+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:15:03.387 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3697 8 365281 365273 0.771835 0 - 0.0299513 2026-02-23T14:15:03.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:15:03 trial068 ceph-mon[38805]: 5.a scrub ok 2026-02-23T14:15:03.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:15:03 trial068 ceph-mon[38805]: pgmap v2331: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:15:03.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:15:03 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T14:15:03.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:15:03 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T14:15:03.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:15:03 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3597581135' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:15:04.266 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:15:03 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:15:03.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:15:04.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:15:03 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:15:03] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:15:04.387 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3698 8 365281 365273 0.771626 0 - 0.0299513 2026-02-23T14:15:05.387 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3699 8 365281 365273 0.771417 0 - 0.0299513 2026-02-23T14:15:05.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:15:05 trial068 ceph-mon[38805]: pgmap v2332: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:15:06.003 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:15:06.122 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:15:06.231 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:15:06.228+0000 7f54ef648640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:15:06.231 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:15:06.229+0000 7f54ef648640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:15:06.339 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:15:06.339 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":223,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:13:55.824287+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:11:02.003888+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:13:34.834650+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:09:41.751921+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:15:06.387 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T14:15:06.386582+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T14:15:06.387 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T14:15:06.387 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3700 8 365281 365273 0.771209 0 - 0.0299513 2026-02-23T14:15:06.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:15:06 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/499444822' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:15:07.387 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3701 8 365281 365273 0.771 0 - 0.0299513 2026-02-23T14:15:07.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:15:07 trial068 ceph-mon[38805]: pgmap v2333: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:15:08.387 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3702 8 365281 365273 0.770792 0 - 0.0299513 2026-02-23T14:15:09.375 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:15:09.387 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3703 8 365281 365273 0.770584 0 - 0.0299513 2026-02-23T14:15:09.494 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:15:09.595 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:15:09.593+0000 7ff03c02f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:15:09.595 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:15:09.593+0000 7ff03c02f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:15:09.708 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:15:09.708 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":223,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:13:55.824287+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:11:02.003888+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:13:34.834650+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:09:41.751921+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:15:09.708 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:15:09 trial068 ceph-mon[38805]: pgmap v2334: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:15:10.387 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3704 8 365281 365273 0.770376 0 - 0.0299513 2026-02-23T14:15:10.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:15:10 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/544283225' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:15:11.387 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3705 8 365281 365273 0.770168 0 - 0.0299513 2026-02-23T14:15:11.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:15:11 trial068 ceph-mon[38805]: pgmap v2335: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:15:11.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:15:11 trial068 ceph-mon[38805]: 5.5 scrub starts 2026-02-23T14:15:11.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:15:11 trial068 ceph-mon[38805]: 5.5 scrub ok 2026-02-23T14:15:12.387 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3706 8 365281 365273 0.76996 0 - 0.0299513 2026-02-23T14:15:12.747 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:15:12.863 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:15:12.963 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:15:12.961+0000 7fc5304d1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:15:12.963 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:15:12.961+0000 7fc5304d1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:15:13.072 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:15:13.072 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":223,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:13:55.824287+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:11:02.003888+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:13:34.834650+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:09:41.751921+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:15:13.387 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3707 8 365281 365273 0.769753 0 - 0.0299513 2026-02-23T14:15:13.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:15:13 trial068 ceph-mon[38805]: pgmap v2336: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:15:13.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:15:13 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T14:15:13.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:15:13 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T14:15:13.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:15:13 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3345284727' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:15:14.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:15:13 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:15:13] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:15:14.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:15:13 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:15:13.974Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:15:14.387 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3708 8 365281 365273 0.769545 0 - 0.0299513 2026-02-23T14:15:14.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:15:14 trial068 ceph-mon[38805]: osdmap e224: 4 total, 3 up, 4 in 2026-02-23T14:15:15.388 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3709 8 365281 365273 0.769337 0 - 0.0299513 2026-02-23T14:15:15.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:15:15 trial068 ceph-mon[38805]: pgmap v2338: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:15:16.111 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:15:16.227 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:15:16.328 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:15:16.326+0000 7f89ee27e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:15:16.329 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:15:16.326+0000 7f89ee27e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:15:16.388 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3710 8 365281 365273 0.76913 0 - 0.0299513 2026-02-23T14:15:16.441 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:15:16.441 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":224,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:15:13.565036+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:11:02.003888+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:13:34.834650+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:15:02.352244+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:15:16.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:15:16 trial068 ceph-mon[38805]: 5.8 scrub starts 2026-02-23T14:15:16.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:15:16 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/264756310' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:15:17.388 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3711 8 365281 365273 0.768923 0 - 0.0299513 2026-02-23T14:15:17.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:15:17 trial068 ceph-mon[38805]: 5.8 scrub ok 2026-02-23T14:15:17.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:15:17 trial068 ceph-mon[38805]: pgmap v2339: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:15:18.388 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3712 8 365281 365273 0.768716 0 - 0.0299513 2026-02-23T14:15:19.388 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3713 8 365281 365273 0.768509 0 - 0.0299513 2026-02-23T14:15:19.481 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:15:19.597 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:15:19.703 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:15:19.701+0000 7f1def807640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:15:19.703 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:15:19.701+0000 7f1def807640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:15:19.813 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:15:19.813 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":224,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:15:13.565036+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:11:02.003888+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:13:34.834650+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:15:02.352244+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:15:19.813 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:15:19 trial068 ceph-mon[38805]: pgmap v2340: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:15:20.388 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3714 8 365281 365273 0.768302 0 - 0.0299513 2026-02-23T14:15:20.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:15:20 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/200190516' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:15:21.388 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3715 8 365281 365273 0.768095 0 - 0.0299513 2026-02-23T14:15:21.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:15:21 trial068 ceph-mon[38805]: pgmap v2341: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:15:22.388 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3716 8 365281 365273 0.767888 0 - 0.0299513 2026-02-23T14:15:22.849 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:15:22.967 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:15:23.066 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:15:23.064+0000 7f5e922cd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:15:23.067 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:15:23.064+0000 7f5e922cd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:15:23.176 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:15:23.176 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":224,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:15:13.565036+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:11:02.003888+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:13:34.834650+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:15:02.352244+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:15:23.388 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3717 8 365281 365273 0.767682 0 - 0.0299513 2026-02-23T14:15:23.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:15:23 trial068 ceph-mon[38805]: pgmap v2342: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:15:23.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:15:23 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3969079859' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:15:24.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:15:23 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:15:23] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:15:24.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:15:23 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:15:23.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:15:24.388 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3718 8 365281 365273 0.767475 0 - 0.0299513 2026-02-23T14:15:25.388 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3719 8 365281 365273 0.767269 0 - 0.0299513 2026-02-23T14:15:25.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:15:25 trial068 ceph-mon[38805]: pgmap v2343: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:15:26.213 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:15:26.330 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:15:26.388 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T14:15:26.387991+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T14:15:26.389 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T14:15:26.389 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3720 8 365281 365273 0.767063 0 - 0.0299513 2026-02-23T14:15:26.430 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:15:26.428+0000 7f4033e20640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:15:26.431 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:15:26.429+0000 7f4033e20640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:15:26.540 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:15:26.540 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":224,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:15:13.565036+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:11:02.003888+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:13:34.834650+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:15:02.352244+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:15:26.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:15:26 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3299598336' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:15:27.388 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3721 8 365281 365273 0.766856 0 - 0.0299513 2026-02-23T14:15:27.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:15:27 trial068 ceph-mon[38805]: pgmap v2344: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:15:28.388 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3722 8 365281 365273 0.76665 0 - 0.0299513 2026-02-23T14:15:29.389 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3723 8 365281 365273 0.766444 0 - 0.0299513 2026-02-23T14:15:29.578 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:15:29.694 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:15:29.795 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:15:29.793+0000 7f8356270640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:15:29.795 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:15:29.793+0000 7f8356270640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:15:29.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:15:29 trial068 ceph-mon[38805]: pgmap v2345: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:15:29.905 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:15:29.906 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":224,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:15:13.565036+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:11:02.003888+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:13:34.834650+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:15:02.352244+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:15:30.389 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3724 8 365281 365273 0.766239 0 - 0.0299513 2026-02-23T14:15:30.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:15:30 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3686689506' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:15:31.389 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3725 8 365281 365273 0.766033 0 - 0.0299513 2026-02-23T14:15:31.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:15:31 trial068 ceph-mon[38805]: pgmap v2346: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:15:32.389 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3726 8 365281 365273 0.765827 0 - 0.0299513 2026-02-23T14:15:32.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:15:32 trial068 ceph-mon[38805]: 5.e scrub starts 2026-02-23T14:15:32.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:15:32 trial068 ceph-mon[38805]: 5.e scrub ok 2026-02-23T14:15:32.942 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:15:33.060 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:15:33.157 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:15:33.155+0000 7f46b2724640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:15:33.158 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:15:33.155+0000 7f46b2724640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:15:33.267 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:15:33.267 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":224,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:15:13.565036+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:11:02.003888+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:13:34.834650+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:15:02.352244+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:15:33.389 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3727 8 365281 365273 0.765622 0 - 0.0299513 2026-02-23T14:15:33.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:15:33 trial068 ceph-mon[38805]: pgmap v2347: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:15:33.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:15:33 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/933977466' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:15:34.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:15:33 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:15:33] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:15:34.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:15:33 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:15:33.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:15:34.389 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3728 8 365281 365273 0.765417 0 - 0.0299513 2026-02-23T14:15:35.389 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3729 8 365281 365273 0.765211 0 - 0.0299513 2026-02-23T14:15:35.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:15:35 trial068 ceph-mon[38805]: pgmap v2348: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:15:36.305 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:15:36.389 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3730 8 365281 365273 0.765006 0 - 0.0299513 2026-02-23T14:15:36.422 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:15:36.521 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:15:36.519+0000 7f0704631640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:15:36.521 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:15:36.519+0000 7f0704631640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:15:36.631 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:15:36.631 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":224,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:15:13.565036+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:11:02.003888+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:13:34.834650+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:15:02.352244+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:15:37.389 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3731 8 365281 365273 0.764801 0 - 0.0299513 2026-02-23T14:15:37.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:15:37 trial068 ceph-mon[38805]: pgmap v2349: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:15:37.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:15:37 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1391312455' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:15:38.389 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3732 8 365281 365273 0.764596 0 - 0.0299513 2026-02-23T14:15:39.389 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3733 8 365281 365273 0.764391 0 - 0.0299513 2026-02-23T14:15:39.668 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:15:39.787 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:15:39.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:15:39 trial068 ceph-mon[38805]: pgmap v2350: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:15:39.893 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:15:39.891+0000 7f644bd71640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:15:39.894 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:15:39.891+0000 7f644bd71640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:15:40.007 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:15:40.008 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":224,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:15:13.565036+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:11:02.003888+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:13:34.834650+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:15:02.352244+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:15:40.389 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3734 8 365281 365273 0.764187 0 - 0.0299513 2026-02-23T14:15:40.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:15:40 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1781495483' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:15:41.389 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3735 8 365281 365273 0.763982 0 - 0.0299513 2026-02-23T14:15:42.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:15:41 trial068 ceph-mon[38805]: pgmap v2351: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:15:42.390 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3736 8 365281 365273 0.763778 0 - 0.0299513 2026-02-23T14:15:43.045 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:15:43.165 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:15:43.270 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:15:43.268+0000 7f98c10ec640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:15:43.271 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:15:43.268+0000 7f98c10ec640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:15:43.382 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:15:43.382 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":224,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:15:13.565036+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:11:02.003888+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:13:34.834650+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:15:02.352244+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:15:43.389 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3737 8 365281 365273 0.763573 0 - 0.0299513 2026-02-23T14:15:43.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:15:43 trial068 ceph-mon[38805]: pgmap v2352: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:15:43.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:15:43 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2969808838' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:15:44.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:15:43 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:15:43] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:15:44.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:15:43 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:15:43.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:15:44.390 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3738 8 365281 365273 0.763369 0 - 0.0299513 2026-02-23T14:15:45.390 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3739 8 365281 365273 0.763165 0 - 0.0299513 2026-02-23T14:15:46.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:15:45 trial068 ceph-mon[38805]: pgmap v2353: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:15:46.390 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T14:15:46.389344+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T14:15:46.390 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T14:15:46.390 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3740 8 365281 365273 0.762961 0 - 0.0299513 2026-02-23T14:15:46.420 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:15:46.540 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:15:46.642 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:15:46.640+0000 7fe2762cf640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:15:46.642 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:15:46.640+0000 7fe2762cf640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:15:46.752 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:15:46.752 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":224,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:15:13.565036+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:11:02.003888+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:13:34.834650+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:15:02.352244+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:15:47.390 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3741 8 365281 365273 0.762757 0 - 0.0299513 2026-02-23T14:15:48.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:15:47 trial068 ceph-mon[38805]: pgmap v2354: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:15:48.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:15:47 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/385893562' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:15:48.390 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3742 8 365281 365273 0.762553 0 - 0.0299513 2026-02-23T14:15:49.390 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3743 8 365281 365273 0.762349 0 - 0.0299513 2026-02-23T14:15:49.789 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:15:49.907 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:15:49.930 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:15:49 trial068 ceph-mon[38805]: pgmap v2355: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:15:50.012 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:15:50.009+0000 7fbacea88640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:15:50.012 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:15:50.010+0000 7fbacea88640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:15:50.122 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:15:50.122 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":224,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:15:13.565036+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:11:02.003888+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:13:34.834650+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:15:02.352244+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:15:50.390 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3744 8 365281 365273 0.762146 0 - 0.0299513 2026-02-23T14:15:51.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:15:50 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2439698612' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:15:51.390 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3745 8 365281 365273 0.761942 0 - 0.0299513 2026-02-23T14:15:52.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:15:51 trial068 ceph-mon[38805]: pgmap v2356: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:15:52.390 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3746 8 365281 365273 0.761739 0 - 0.0299513 2026-02-23T14:15:53.160 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:15:53.279 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:15:53.384 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:15:53.382+0000 7fbcd47e4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:15:53.384 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:15:53.382+0000 7fbcd47e4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:15:53.390 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3747 8 365281 365273 0.761535 0 - 0.0299513 2026-02-23T14:15:53.492 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:15:53.493 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":224,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:15:13.565036+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:11:02.003888+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:13:34.834650+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:15:02.352244+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:15:53.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:15:53 trial068 ceph-mon[38805]: pgmap v2357: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:15:53.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:15:53 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T14:15:53.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:15:53 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T14:15:53.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:15:53 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3236433738' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:15:54.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:15:53 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:15:53] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:15:54.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:15:53 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:15:53.974Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:15:54.390 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3748 8 365281 365273 0.761332 0 - 0.0299513 2026-02-23T14:15:55.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:15:54 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2026-02-23T14:15:55.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:15:54 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-23T14:15:55.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:15:54 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2026-02-23T14:15:55.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:15:54 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T14:15:55.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:15:54 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2026-02-23T14:15:55.390 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3749 8 365281 365273 0.761129 0 - 0.0299513 2026-02-23T14:15:56.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:15:55 trial068 ceph-mon[38805]: pgmap v2358: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:15:56.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:15:55 trial068 ceph-mon[38805]: pgmap v2359: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:15:56.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:15:55 trial068 ceph-mon[38805]: pgmap v2360: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:15:56.390 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3750 8 365281 365273 0.760926 0 - 0.0299513 2026-02-23T14:15:56.530 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:15:56.648 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:15:56.754 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:15:56.752+0000 7fba0c2f7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:15:56.755 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:15:56.753+0000 7fba0c2f7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:15:56.864 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:15:56.864 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":224,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:15:13.565036+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:11:02.003888+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:13:34.834650+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:15:02.352244+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:15:57.390 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3751 8 365281 365273 0.760723 0 - 0.0299513 2026-02-23T14:15:58.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:15:57 trial068 ceph-mon[38805]: pgmap v2361: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:15:58.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:15:57 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1495519177' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:15:58.391 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3752 8 365281 365273 0.760521 0 - 0.0299513 2026-02-23T14:15:59.391 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3753 8 365281 365273 0.760318 0 - 0.0299513 2026-02-23T14:15:59.896 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:15:59 trial068 ceph-mon[38805]: pgmap v2362: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:15:59.899 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:16:00.016 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:16:00.117 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:16:00.115+0000 7f5eee41d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:16:00.117 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:16:00.115+0000 7f5eee41d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:16:00.228 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:16:00.228 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":224,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:15:13.565036+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:11:02.003888+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:13:34.834650+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:15:02.352244+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:16:00.391 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3754 8 365281 365273 0.760115 0 - 0.0299513 2026-02-23T14:16:01.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:16:00 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1883252572' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:16:01.391 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3755 8 365281 365273 0.759913 0 - 0.0299513 2026-02-23T14:16:02.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:16:01 trial068 ceph-mon[38805]: pgmap v2363: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:16:02.391 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3756 8 365281 365273 0.759711 0 - 0.0299513 2026-02-23T14:16:03.267 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:16:03.384 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:16:03.391 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3757 8 365281 365273 0.759508 0 - 0.0299513 2026-02-23T14:16:03.481 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:16:03.478+0000 7f2a0a5dc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:16:03.481 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:16:03.479+0000 7f2a0a5dc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:16:03.590 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:16:03.590 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":224,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:15:13.565036+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:11:02.003888+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:13:34.834650+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:15:02.352244+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:16:03.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:16:03 trial068 ceph-mon[38805]: pgmap v2364: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:16:03.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:16:03 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3541624507' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:16:04.266 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:16:03 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:16:03.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:16:04.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:16:03 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:16:03] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:16:04.391 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3758 8 365281 365273 0.759306 0 - 0.0299513 2026-02-23T14:16:05.391 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3759 8 365281 365273 0.759104 0 - 0.0299513 2026-02-23T14:16:06.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:16:05 trial068 ceph-mon[38805]: pgmap v2365: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:16:06.391 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T14:16:06.390816+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T14:16:06.391 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T14:16:06.391 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3760 8 365281 365273 0.758902 0 - 0.0299513 2026-02-23T14:16:06.628 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:16:06.744 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:16:06.842 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:16:06.840+0000 7fa99a3db640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:16:06.842 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:16:06.840+0000 7fa99a3db640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:16:06.951 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:16:06.951 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":224,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:15:13.565036+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:11:02.003888+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:13:34.834650+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:15:02.352244+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:16:07.391 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3761 8 365281 365273 0.758701 0 - 0.0299513 2026-02-23T14:16:08.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:16:07 trial068 ceph-mon[38805]: pgmap v2366: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:16:08.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:16:07 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3984846394' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:16:08.391 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3762 8 365281 365273 0.758499 0 - 0.0299513 2026-02-23T14:16:09.391 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3763 8 365281 365273 0.758297 0 - 0.0299513 2026-02-23T14:16:10.003 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:16:10.028 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:16:09 trial068 ceph-mon[38805]: pgmap v2367: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:16:10.122 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:16:10.223 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:16:10.221+0000 7f9686bb1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:16:10.223 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:16:10.221+0000 7f9686bb1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:16:10.332 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:16:10.333 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":224,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:15:13.565036+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:11:02.003888+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:13:34.834650+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:15:02.352244+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:16:10.391 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3764 8 365281 365273 0.758096 0 - 0.0299513 2026-02-23T14:16:11.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:16:10 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2889336120' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:16:11.391 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3765 8 365281 365273 0.757895 0 - 0.0299513 2026-02-23T14:16:12.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:16:11 trial068 ceph-mon[38805]: pgmap v2368: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:16:12.391 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3766 8 365281 365273 0.757693 0 - 0.0299513 2026-02-23T14:16:13.370 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:16:13.392 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3767 8 365281 365273 0.757492 0 - 0.0299513 2026-02-23T14:16:13.487 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:16:13.591 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:16:13.589+0000 7efc3852a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:16:13.591 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:16:13.589+0000 7efc3852a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:16:13.702 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:16:13.702 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":224,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:15:13.565036+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:11:02.003888+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:13:34.834650+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:15:02.352244+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:16:13.702 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:16:13 trial068 ceph-mon[38805]: pgmap v2369: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:16:14.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:16:13 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:16:13] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:16:14.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:16:13 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:16:13.974Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:16:14.392 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3768 8 365281 365273 0.757291 0 - 0.0299513 2026-02-23T14:16:15.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:16:14 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1640421742' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:16:15.392 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3769 8 365281 365273 0.75709 0 - 0.0299513 2026-02-23T14:16:16.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:16:15 trial068 ceph-mon[38805]: pgmap v2370: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:16:16.392 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3770 8 365281 365273 0.756889 0 - 0.0299513 2026-02-23T14:16:16.740 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:16:16.860 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:16:16.882 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:16:16 trial068 ceph-mon[38805]: 5.a scrub starts 2026-02-23T14:16:16.883 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:16:16 trial068 ceph-mon[38805]: 5.a scrub ok 2026-02-23T14:16:16.956 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:16:16.954+0000 7f6946efd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:16:16.957 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:16:16.955+0000 7f6946efd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:16:17.068 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:16:17.068 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":224,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:15:13.565036+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:11:02.003888+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:13:34.834650+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:15:02.352244+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:16:17.392 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3771 8 365281 365273 0.756689 0 - 0.0299513 2026-02-23T14:16:18.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:16:17 trial068 ceph-mon[38805]: pgmap v2371: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:16:18.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:16:17 trial068 ceph-mon[38805]: 5.7 scrub starts 2026-02-23T14:16:18.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:16:17 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/84158842' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:16:18.392 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3772 8 365281 365273 0.756488 0 - 0.0299513 2026-02-23T14:16:19.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:16:18 trial068 ceph-mon[38805]: 5.7 scrub ok 2026-02-23T14:16:19.392 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3773 8 365281 365273 0.756288 0 - 0.0299513 2026-02-23T14:16:20.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:16:19 trial068 ceph-mon[38805]: pgmap v2372: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:16:20.107 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:16:20.225 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:16:20.331 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:16:20.329+0000 7f80a685d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:16:20.331 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:16:20.329+0000 7f80a685d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:16:20.392 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3774 8 365281 365273 0.756087 0 - 0.0299513 2026-02-23T14:16:20.439 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:16:20.439 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":224,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:15:13.565036+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:11:02.003888+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:13:34.834650+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:15:02.352244+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:16:20.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:16:20 trial068 ceph-mon[38805]: 5.5 scrub starts 2026-02-23T14:16:20.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:16:20 trial068 ceph-mon[38805]: 5.5 scrub ok 2026-02-23T14:16:20.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:16:20 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1509915302' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:16:21.392 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3775 8 365281 365273 0.755887 0 - 0.0299513 2026-02-23T14:16:22.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:16:21 trial068 ceph-mon[38805]: pgmap v2373: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:16:22.392 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3776 8 365281 365273 0.755687 0 - 0.0299513 2026-02-23T14:16:23.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:16:22 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T14:16:23.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:16:22 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T14:16:23.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:16:22 trial068 ceph-mon[38805]: osdmap e225: 4 total, 3 up, 4 in 2026-02-23T14:16:23.392 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3777 8 365281 365273 0.755487 0 - 0.0299513 2026-02-23T14:16:23.476 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:16:23.591 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:16:23.689 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:16:23.687+0000 7f4c718e6640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:16:23.689 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:16:23.687+0000 7f4c718e6640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:16:23.798 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:16:23 trial068 ceph-mon[38805]: pgmap v2375: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:16:23.799 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:16:23.799 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":225,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:16:21.632768+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:16:21.569603+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:13:34.834650+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:15:02.352244+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:16:24.094 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:16:23 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:16:23] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:16:24.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:16:23 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:16:23.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:16:24.393 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3778 8 365281 365273 0.755287 0 - 0.0299513 2026-02-23T14:16:25.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:16:24 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/4292453456' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:16:25.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:16:24 trial068 ceph-mon[38805]: osdmap e226: 4 total, 3 up, 4 in 2026-02-23T14:16:25.393 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3779 8 365281 365273 0.755087 0 - 0.0299513 2026-02-23T14:16:26.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:16:25 trial068 ceph-mon[38805]: pgmap v2376: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:16:26.393 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T14:16:26.392584+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T14:16:26.393 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T14:16:26.393 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3780 8 365281 365273 0.754887 0 - 0.0299513 2026-02-23T14:16:26.849 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:16:26.966 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:16:26.988 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:16:26 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T14:16:26.989 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:16:26 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T14:16:27.070 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:16:27.068+0000 7f90265f5640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:16:27.070 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:16:27.068+0000 7f90265f5640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:16:27.180 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:16:27.180 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":226,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:16:24.513874+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:16:21.569603+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:13:34.834650+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:15:02.352244+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:16:27.393 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3781 8 365281 365273 0.754687 0 - 0.0299513 2026-02-23T14:16:28.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:16:27 trial068 ceph-mon[38805]: pgmap v2378: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:16:28.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:16:27 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/841690118' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:16:28.393 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3782 8 365281 365273 0.754488 0 - 0.0299513 2026-02-23T14:16:29.393 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3783 8 365281 365273 0.754288 0 - 0.0299513 2026-02-23T14:16:29.896 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:16:29 trial068 ceph-mon[38805]: pgmap v2379: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:16:29.896 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:16:29 trial068 ceph-mon[38805]: osdmap e227: 4 total, 3 up, 4 in 2026-02-23T14:16:30.216 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:16:30.335 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:16:30.393 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3784 8 365281 365273 0.754089 0 - 0.0299513 2026-02-23T14:16:30.433 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:16:30.431+0000 7fbed3294640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:16:30.433 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:16:30.431+0000 7fbed3294640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:16:30.542 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:16:30.542 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":227,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:16:29.514924+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:16:21.569603+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:13:34.834650+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:15:02.352244+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:16:30.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:16:30 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3990178821' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:16:31.393 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3785 8 365281 365273 0.75389 0 - 0.0299513 2026-02-23T14:16:32.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:16:31 trial068 ceph-mon[38805]: pgmap v2381: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:16:32.393 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3786 8 365281 365273 0.753691 0 - 0.0299513 2026-02-23T14:16:33.393 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3787 8 365281 365273 0.753492 0 - 0.0299513 2026-02-23T14:16:33.579 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:16:33.697 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:16:33.795 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:16:33.793+0000 7f709fb42640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:16:33.795 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:16:33.793+0000 7f709fb42640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:16:33.903 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:16:33.904 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":227,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:16:29.514924+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:16:21.569603+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:13:34.834650+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:15:02.352244+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:16:33.904 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:16:33 trial068 ceph-mon[38805]: pgmap v2382: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:16:34.266 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:16:33 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:16:33.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:16:34.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:16:33 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:16:33] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:16:34.393 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3788 8 365281 365273 0.753293 0 - 0.0299513 2026-02-23T14:16:35.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:16:34 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3297236970' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:16:35.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:16:34 trial068 ceph-mon[38805]: osdmap e228: 4 total, 3 up, 4 in 2026-02-23T14:16:35.394 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3789 8 365281 365273 0.753094 0 - 0.0299513 2026-02-23T14:16:36.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:16:35 trial068 ceph-mon[38805]: pgmap v2383: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:16:36.394 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3790 8 365281 365273 0.752895 0 - 0.0299513 2026-02-23T14:16:36.941 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:16:36.967 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:16:36 trial068 ceph-mon[38805]: 5.8 scrub starts 2026-02-23T14:16:36.967 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:16:36 trial068 ceph-mon[38805]: 5.8 scrub ok 2026-02-23T14:16:37.060 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:16:37.182 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:16:37.180+0000 7fa3e0c5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:16:37.182 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:16:37.180+0000 7fa3e0c5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:16:37.291 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:16:37.292 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":228,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:16:34.515755+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:16:21.569603+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:13:34.834650+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:15:02.352244+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:16:37.394 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3791 8 365281 365273 0.752697 0 - 0.0299513 2026-02-23T14:16:38.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:16:37 trial068 ceph-mon[38805]: pgmap v2385: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:16:38.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:16:37 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1968072511' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:16:38.394 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3792 8 365281 365273 0.752498 0 - 0.0299513 2026-02-23T14:16:39.394 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3793 8 365281 365273 0.7523 0 - 0.0299513 2026-02-23T14:16:40.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:16:39 trial068 ceph-mon[38805]: pgmap v2386: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:16:40.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:16:39 trial068 ceph-mon[38805]: osdmap e229: 4 total, 3 up, 4 in 2026-02-23T14:16:40.328 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:16:40.394 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3794 8 365281 365273 0.752101 0 - 0.0299513 2026-02-23T14:16:40.446 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:16:40.546 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:16:40.544+0000 7fe2eb1db640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:16:40.546 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:16:40.544+0000 7fe2eb1db640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:16:40.656 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:16:40.656 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":229,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:16:39.517039+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:16:21.569603+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:13:34.834650+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:15:02.352244+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:16:41.394 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3795 8 365281 365273 0.751903 0 - 0.0299513 2026-02-23T14:16:42.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:16:41 trial068 ceph-mon[38805]: pgmap v2388: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:16:42.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:16:41 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/4292172867' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:16:42.394 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3796 8 365281 365273 0.751705 0 - 0.0299513 2026-02-23T14:16:43.394 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3797 8 365281 365273 0.751507 0 - 0.0299513 2026-02-23T14:16:43.694 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:16:43.811 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:16:43.914 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:16:43.912+0000 7f36f2035640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:16:43.915 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:16:43 trial068 ceph-mon[38805]: pgmap v2389: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:16:43.915 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:16:43.913+0000 7f36f2035640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:16:44.024 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:16:44.024 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":229,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:16:39.517039+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:16:21.569603+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:13:34.834650+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:15:02.352244+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:16:44.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:16:43 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:16:43] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:16:44.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:16:43 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:16:43.972Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:16:44.394 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3798 8 365281 365273 0.751309 0 - 0.0299513 2026-02-23T14:16:45.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:16:44 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/651255643' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:16:45.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:16:44 trial068 ceph-mon[38805]: osdmap e230: 4 total, 3 up, 4 in 2026-02-23T14:16:45.394 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3799 8 365281 365273 0.751112 0 - 0.0299513 2026-02-23T14:16:46.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:16:45 trial068 ceph-mon[38805]: pgmap v2390: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:16:46.394 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T14:16:46.393804+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T14:16:46.394 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T14:16:46.395 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3800 8 365281 365273 0.750914 0 - 0.0299513 2026-02-23T14:16:47.062 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:16:47.181 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:16:47.282 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:16:47.279+0000 7ff007cee640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:16:47.282 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:16:47.280+0000 7ff007cee640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:16:47.393 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:16:47.393 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":230,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:16:44.517977+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:16:21.569603+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:13:34.834650+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:15:02.352244+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:16:47.394 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3801 8 365281 365273 0.750716 0 - 0.0299513 2026-02-23T14:16:48.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:16:47 trial068 ceph-mon[38805]: pgmap v2392: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:16:48.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:16:47 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3822869739' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:16:48.394 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3802 8 365281 365273 0.750519 0 - 0.0299513 2026-02-23T14:16:49.394 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3803 8 365281 365273 0.750322 0 - 0.0299513 2026-02-23T14:16:50.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:16:49 trial068 ceph-mon[38805]: pgmap v2393: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:16:50.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:16:49 trial068 ceph-mon[38805]: osdmap e231: 4 total, 3 up, 4 in 2026-02-23T14:16:50.394 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3804 8 365281 365273 0.750124 0 - 0.0299513 2026-02-23T14:16:50.430 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:16:50.548 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:16:50.649 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:16:50.647+0000 7fdb444d3640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:16:50.649 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:16:50.647+0000 7fdb444d3640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:16:50.757 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:16:50.757 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":231,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:16:49.519290+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:16:21.569603+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:13:34.834650+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:15:02.352244+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:16:51.395 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3805 8 365281 365273 0.749927 0 - 0.0299513 2026-02-23T14:16:52.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:16:51 trial068 ceph-mon[38805]: pgmap v2395: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:16:52.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:16:51 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/204483115' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:16:52.395 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3806 8 365281 365273 0.74973 0 - 0.0299513 2026-02-23T14:16:53.395 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3807 8 365281 365273 0.749533 0 - 0.0299513 2026-02-23T14:16:53.796 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:16:53.914 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:16:53.936 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:16:53 trial068 ceph-mon[38805]: pgmap v2396: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:16:54.026 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:16:54.024+0000 7fcbe8dfe640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:16:54.027 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:16:54.025+0000 7fcbe8dfe640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:16:54.136 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:16:54.136 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":231,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:16:49.519290+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:16:21.569603+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:13:34.834650+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:15:02.352244+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:16:54.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:16:53 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:16:53] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:16:54.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:16:53 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:16:53.972Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:16:54.395 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3808 8 365281 365273 0.749336 0 - 0.0299513 2026-02-23T14:16:55.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:16:54 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/63637582' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:16:55.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:16:54 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2026-02-23T14:16:55.395 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3809 8 365281 365273 0.74914 0 - 0.0299513 2026-02-23T14:16:56.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:16:55 trial068 ceph-mon[38805]: pgmap v2397: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:16:56.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:16:55 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T14:16:56.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:16:55 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T14:16:56.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:16:55 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T14:16:56.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:16:55 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T14:16:56.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:16:55 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-23T14:16:56.095 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:16:55 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2026-02-23T14:16:56.095 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:16:55 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T14:16:56.095 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:16:55 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2026-02-23T14:16:56.395 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3810 8 365281 365273 0.748943 0 - 0.0299513 2026-02-23T14:16:57.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:16:56 trial068 ceph-mon[38805]: pgmap v2398: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:16:57.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:16:56 trial068 ceph-mon[38805]: pgmap v2399: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:16:57.175 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:16:57.290 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:16:57.394 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:16:57.392+0000 7f5d27628640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:16:57.394 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:16:57.392+0000 7f5d27628640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:16:57.395 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3811 8 365281 365273 0.748747 0 - 0.0299513 2026-02-23T14:16:57.504 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:16:57.504 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":231,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:16:49.519290+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:16:21.569603+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:13:34.834650+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:15:02.352244+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:16:57.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:16:57 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3639494116' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:16:58.395 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3812 8 365281 365273 0.74855 0 - 0.0299513 2026-02-23T14:16:59.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:16:58 trial068 ceph-mon[38805]: pgmap v2400: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:16:59.395 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3813 8 365281 365273 0.748354 0 - 0.0299513 2026-02-23T14:17:00.395 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3814 8 365281 365273 0.748158 0 - 0.0299513 2026-02-23T14:17:00.540 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:17:00.659 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:17:00.757 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:17:00.755+0000 7f90e8028640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:17:00.757 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:17:00.755+0000 7f90e8028640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:17:00.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:17:00 trial068 ceph-mon[38805]: pgmap v2401: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:17:00.866 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:17:00.866 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":231,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:16:49.519290+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:16:21.569603+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:13:34.834650+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:15:02.352244+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:17:01.395 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3815 8 365281 365273 0.747962 0 - 0.0299513 2026-02-23T14:17:02.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:17:01 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3967845691' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:17:02.395 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3816 8 365281 365273 0.747766 0 - 0.0299513 2026-02-23T14:17:03.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:17:02 trial068 ceph-mon[38805]: pgmap v2402: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:17:03.396 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3817 8 365281 365273 0.74757 0 - 0.0299513 2026-02-23T14:17:03.918 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:17:03.974 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:17:03 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T14:17:03.974 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:17:03 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T14:17:04.038 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:17:04.142 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:17:04.140+0000 7fbb927ce640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:17:04.143 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:17:04.140+0000 7fbb927ce640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:17:04.252 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:17:04.252 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":231,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:16:49.519290+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:16:21.569603+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:13:34.834650+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:15:02.352244+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:17:04.252 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:17:03 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:17:03] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:17:04.266 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:17:03 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:17:03.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:17:04.395 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3818 8 365281 365273 0.747374 0 - 0.0299513 2026-02-23T14:17:05.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:17:04 trial068 ceph-mon[38805]: pgmap v2403: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:17:05.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:17:04 trial068 ceph-mon[38805]: 5.e scrub starts 2026-02-23T14:17:05.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:17:04 trial068 ceph-mon[38805]: 5.e scrub ok 2026-02-23T14:17:05.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:17:04 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2319368160' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:17:05.396 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3819 8 365281 365273 0.747178 0 - 0.0299513 2026-02-23T14:17:06.396 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T14:17:06.395135+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T14:17:06.396 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T14:17:06.396 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3820 8 365281 365273 0.746983 0 - 0.0299513 2026-02-23T14:17:07.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:17:06 trial068 ceph-mon[38805]: pgmap v2404: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:17:07.313 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:17:07.396 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3821 8 365281 365273 0.746787 0 - 0.0299513 2026-02-23T14:17:07.432 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:17:07.529 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:17:07.527+0000 7f6adbfff640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:17:07.529 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:17:07.527+0000 7f6adbfff640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:17:07.639 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:17:07.639 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":231,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:16:49.519290+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:16:21.569603+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:13:34.834650+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:15:02.352244+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:17:08.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:17:07 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1303312986' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:17:08.396 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3822 8 365281 365273 0.746592 0 - 0.0299513 2026-02-23T14:17:09.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:17:08 trial068 ceph-mon[38805]: pgmap v2405: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:17:09.396 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3823 8 365281 365273 0.746396 0 - 0.0299513 2026-02-23T14:17:10.396 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3824 8 365281 365273 0.746201 0 - 0.0299513 2026-02-23T14:17:10.676 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:17:10.794 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:17:10.893 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:17:10.891+0000 7ff8dcd13640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:17:10.893 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:17:10.891+0000 7ff8dcd13640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:17:11.004 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:17:11.004 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":231,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:16:49.519290+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:16:21.569603+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:13:34.834650+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:15:02.352244+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:17:11.005 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:17:10 trial068 ceph-mon[38805]: pgmap v2406: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:17:11.396 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3825 8 365281 365273 0.746006 0 - 0.0299513 2026-02-23T14:17:12.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:17:11 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2895283229' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:17:12.396 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3826 8 365281 365273 0.745811 0 - 0.0299513 2026-02-23T14:17:13.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:17:12 trial068 ceph-mon[38805]: pgmap v2407: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:17:13.396 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3827 8 365281 365273 0.745616 0 - 0.0299513 2026-02-23T14:17:14.055 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:17:14.175 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:17:14.226 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:17:13 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:17:13] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:17:14.273 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:17:14.271+0000 7f2814a87640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:17:14.273 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:17:14.271+0000 7f2814a87640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:17:14.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:17:13 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:17:13.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:17:14.383 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:17:14.383 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":231,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:16:49.519290+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:16:21.569603+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:13:34.834650+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:15:02.352244+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:17:14.396 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3828 8 365281 365273 0.745421 0 - 0.0299513 2026-02-23T14:17:15.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:17:14 trial068 ceph-mon[38805]: pgmap v2408: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:17:15.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:17:14 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1251354492' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:17:15.397 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3829 8 365281 365273 0.745227 0 - 0.0299513 2026-02-23T14:17:16.397 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3830 8 365281 365273 0.745032 0 - 0.0299513 2026-02-23T14:17:17.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:17:16 trial068 ceph-mon[38805]: pgmap v2409: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:17:17.397 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3831 8 365281 365273 0.744838 0 - 0.0299513 2026-02-23T14:17:17.422 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:17:17.541 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:17:17.645 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:17:17.643+0000 7faa2c84b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:17:17.645 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:17:17.643+0000 7faa2c84b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:17:17.757 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:17:17.758 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":231,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:16:49.519290+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:16:21.569603+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:13:34.834650+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:15:02.352244+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:17:18.397 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3832 8 365281 365273 0.744643 0 - 0.0299513 2026-02-23T14:17:19.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:17:18 trial068 ceph-mon[38805]: pgmap v2410: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:17:19.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:17:18 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/420500569' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:17:19.397 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3833 8 365281 365273 0.744449 0 - 0.0299513 2026-02-23T14:17:20.397 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3834 8 365281 365273 0.744255 0 - 0.0299513 2026-02-23T14:17:20.795 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:17:20.913 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:17:21.010 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:17:21.007+0000 7f2452722640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:17:21.010 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:17:20 trial068 ceph-mon[38805]: pgmap v2411: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:17:21.011 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:17:21.008+0000 7f2452722640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:17:21.119 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:17:21.120 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":231,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:16:49.519290+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:16:21.569603+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:13:34.834650+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:15:02.352244+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:17:21.397 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3835 8 365281 365273 0.744061 0 - 0.0299513 2026-02-23T14:17:22.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:17:21 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/26446539' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:17:22.397 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3836 8 365281 365273 0.743867 0 - 0.0299513 2026-02-23T14:17:23.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:17:22 trial068 ceph-mon[38805]: pgmap v2412: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:17:23.397 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3837 8 365281 365273 0.743673 0 - 0.0299513 2026-02-23T14:17:24.157 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:17:24.274 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:17:24.297 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:17:23 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:17:23] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:17:24.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:17:23 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:17:23.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:17:24.376 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:17:24.374+0000 7fa34e19c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:17:24.377 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:17:24.375+0000 7fa34e19c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:17:24.397 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3838 8 365281 365273 0.743479 0 - 0.0299513 2026-02-23T14:17:24.487 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:17:24.487 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":231,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:16:49.519290+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:16:21.569603+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:13:34.834650+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:15:02.352244+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:17:25.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:17:24 trial068 ceph-mon[38805]: pgmap v2413: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:17:25.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:17:24 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/775352505' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:17:25.397 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3839 8 365281 365273 0.743286 0 - 0.0299513 2026-02-23T14:17:26.397 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T14:17:26.396734+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T14:17:26.397 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T14:17:26.398 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3840 8 365281 365273 0.743092 0 - 0.0299513 2026-02-23T14:17:27.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:17:26 trial068 ceph-mon[38805]: pgmap v2414: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:17:27.397 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3841 8 365281 365273 0.742899 0 - 0.0299513 2026-02-23T14:17:27.524 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:17:27.645 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:17:27.744 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:17:27.742+0000 7febcb482640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:17:27.745 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:17:27.743+0000 7febcb482640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:17:27.856 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:17:27.857 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":231,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:16:49.519290+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:16:21.569603+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:13:34.834650+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:15:02.352244+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:17:28.397 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3842 8 365281 365273 0.742705 0 - 0.0299513 2026-02-23T14:17:29.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:17:28 trial068 ceph-mon[38805]: pgmap v2415: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:17:29.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:17:28 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T14:17:29.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:17:28 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T14:17:29.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:17:28 trial068 ceph-mon[38805]: 5.7 scrub starts 2026-02-23T14:17:29.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:17:28 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1805668998' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:17:29.095 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:17:28 trial068 ceph-mon[38805]: 5.7 scrub ok 2026-02-23T14:17:29.397 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3843 8 365281 365273 0.742512 0 - 0.0299513 2026-02-23T14:17:30.397 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3844 8 365281 365273 0.742319 0 - 0.0299513 2026-02-23T14:17:30.895 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:17:31.013 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:17:31.035 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:17:30 trial068 ceph-mon[38805]: pgmap v2416: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:17:31.114 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:17:31.112+0000 7f976b605640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:17:31.115 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:17:31.113+0000 7f976b605640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:17:31.223 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:17:31.224 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":231,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:16:49.519290+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:16:21.569603+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:13:34.834650+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:15:02.352244+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:17:31.397 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3845 8 365281 365273 0.742126 0 - 0.0299513 2026-02-23T14:17:32.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:17:31 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2090793970' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:17:32.398 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3846 8 365281 365273 0.741933 0 - 0.0299513 2026-02-23T14:17:33.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:17:32 trial068 ceph-mon[38805]: pgmap v2417: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:17:33.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:17:32 trial068 ceph-mon[38805]: 5.a scrub starts 2026-02-23T14:17:33.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:17:32 trial068 ceph-mon[38805]: 5.a scrub ok 2026-02-23T14:17:33.398 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3847 8 365281 365273 0.74174 0 - 0.0299513 2026-02-23T14:17:34.261 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:17:34.266 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:17:33 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:17:33.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:17:34.285 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:17:33 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:17:33] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:17:34.377 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:17:34.398 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3848 8 365281 365273 0.741547 0 - 0.0299513 2026-02-23T14:17:34.473 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:17:34.471+0000 7f9b5e651640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:17:34.474 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:17:34.472+0000 7f9b5e651640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:17:34.584 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:17:34.584 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":231,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:16:49.519290+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:16:21.569603+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:13:34.834650+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:15:02.352244+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:17:34.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:17:34 trial068 ceph-mon[38805]: pgmap v2418: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:17:34.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:17:34 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2764318877' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:17:35.398 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3849 8 365281 365273 0.741354 0 - 0.0299513 2026-02-23T14:17:36.398 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3850 8 365281 365273 0.741162 0 - 0.0299513 2026-02-23T14:17:37.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:17:36 trial068 ceph-mon[38805]: pgmap v2419: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:17:37.398 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3851 8 365281 365273 0.740969 0 - 0.0299513 2026-02-23T14:17:37.623 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:17:37.740 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:17:37.844 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:17:37.842+0000 7f2662629640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:17:37.844 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:17:37.842+0000 7f2662629640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:17:37.954 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:17:37.954 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":231,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:16:49.519290+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:16:21.569603+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:13:34.834650+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:15:02.352244+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:17:37.955 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:17:37 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T14:17:37.955 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:17:37 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T14:17:37.955 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:17:37 trial068 ceph-mon[38805]: 5.5 scrub starts 2026-02-23T14:17:37.955 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:17:37 trial068 ceph-mon[38805]: 5.5 scrub ok 2026-02-23T14:17:38.398 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3852 8 365281 365273 0.740777 0 - 0.0299513 2026-02-23T14:17:39.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:17:38 trial068 ceph-mon[38805]: pgmap v2420: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:17:39.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:17:38 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/16217616' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:17:39.398 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3853 8 365281 365273 0.740585 0 - 0.0299513 2026-02-23T14:17:40.398 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3854 8 365281 365273 0.740393 0 - 0.0299513 2026-02-23T14:17:40.991 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:17:41.015 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:17:40 trial068 ceph-mon[38805]: pgmap v2421: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:17:41.109 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:17:41.207 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:17:41.205+0000 7f6160437640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:17:41.207 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:17:41.205+0000 7f6160437640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:17:41.317 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:17:41.318 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":231,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:16:49.519290+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:16:21.569603+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:13:34.834650+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:15:02.352244+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:17:41.398 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3855 8 365281 365273 0.740201 0 - 0.0299513 2026-02-23T14:17:42.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:17:41 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3900594467' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:17:42.398 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3856 8 365281 365273 0.740009 0 - 0.0299513 2026-02-23T14:17:43.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:17:42 trial068 ceph-mon[38805]: pgmap v2422: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:17:43.398 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3857 8 365281 365273 0.739817 0 - 0.0299513 2026-02-23T14:17:44.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:17:43 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:17:43] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:17:44.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:17:43 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:17:43.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:17:44.357 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:17:44.398 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3858 8 365281 365273 0.739625 0 - 0.0299513 2026-02-23T14:17:44.473 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:17:44.576 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:17:44.573+0000 7fbe17e5a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:17:44.576 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:17:44.574+0000 7fbe17e5a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:17:44.685 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:17:44.685 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":231,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:16:49.519290+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:16:21.569603+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:13:34.834650+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:15:02.352244+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:17:45.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:17:44 trial068 ceph-mon[38805]: pgmap v2423: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:17:45.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:17:44 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2214714966' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:17:45.398 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3859 8 365281 365273 0.739433 0 - 0.0299513 2026-02-23T14:17:46.399 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T14:17:46.398150+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T14:17:46.399 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T14:17:46.399 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3860 8 365281 365273 0.739242 0 - 0.0299513 2026-02-23T14:17:47.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:17:46 trial068 ceph-mon[38805]: pgmap v2424: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:17:47.399 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3861 8 365281 365273 0.73905 0 - 0.0299513 2026-02-23T14:17:47.724 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:17:47.841 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:17:47.942 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:17:47.940+0000 7fa3004a2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:17:47.942 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:17:47.940+0000 7fa3004a2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:17:48.053 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:17:48.053 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":231,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:16:49.519290+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:16:21.569603+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:13:34.834650+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:15:02.352244+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:17:48.399 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3862 8 365281 365273 0.738859 0 - 0.0299513 2026-02-23T14:17:49.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:17:48 trial068 ceph-mon[38805]: pgmap v2425: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:17:49.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:17:48 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3853643168' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:17:49.399 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3863 8 365281 365273 0.738668 0 - 0.0299513 2026-02-23T14:17:50.399 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3864 8 365281 365273 0.738477 0 - 0.0299513 2026-02-23T14:17:51.088 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:17:51.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:17:50 trial068 ceph-mon[38805]: pgmap v2426: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:17:51.208 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:17:51.306 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:17:51.304+0000 7f5e554cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:17:51.306 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:17:51.304+0000 7f5e554cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:17:51.399 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3865 8 365281 365273 0.738285 0 - 0.0299513 2026-02-23T14:17:51.417 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:17:51.417 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":231,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:16:49.519290+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:16:21.569603+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:13:34.834650+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:15:02.352244+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:17:52.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:17:51 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/54528797' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:17:52.399 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3866 8 365281 365273 0.738095 0 - 0.0299513 2026-02-23T14:17:53.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:17:52 trial068 ceph-mon[38805]: pgmap v2427: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:17:53.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:17:52 trial068 ceph-mon[38805]: 5.8 scrub starts 2026-02-23T14:17:53.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:17:52 trial068 ceph-mon[38805]: 5.8 scrub ok 2026-02-23T14:17:53.399 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3867 8 365281 365273 0.737904 0 - 0.0299513 2026-02-23T14:17:54.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:17:53 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:17:53] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:17:54.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:17:53 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:17:53.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:17:54.399 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3868 8 365281 365273 0.737713 0 - 0.0299513 2026-02-23T14:17:54.457 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:17:54.576 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:17:54.680 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:17:54.678+0000 7f2629648640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:17:54.681 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:17:54.678+0000 7f2629648640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:17:54.789 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:17:54.789 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":231,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:16:49.519290+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:16:21.569603+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:13:34.834650+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:15:02.352244+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:17:55.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:17:54 trial068 ceph-mon[38805]: pgmap v2428: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:17:55.399 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3869 8 365281 365273 0.737522 0 - 0.0299513 2026-02-23T14:17:56.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:17:55 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/564859130' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:17:56.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:17:55 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2026-02-23T14:17:56.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:17:55 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T14:17:56.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:17:55 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T14:17:56.095 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:17:55 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T14:17:56.095 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:17:55 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T14:17:56.095 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:17:55 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-23T14:17:56.095 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:17:55 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2026-02-23T14:17:56.095 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:17:55 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T14:17:56.095 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:17:55 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2026-02-23T14:17:56.399 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3870 8 365281 365273 0.737332 0 - 0.0299513 2026-02-23T14:17:57.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:17:56 trial068 ceph-mon[38805]: pgmap v2429: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:17:57.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:17:56 trial068 ceph-mon[38805]: pgmap v2430: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:17:57.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:17:56 trial068 ceph-mon[38805]: pgmap v2431: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:17:57.400 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3871 8 365281 365273 0.737141 0 - 0.0299513 2026-02-23T14:17:57.826 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:17:57.945 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:17:58.047 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:17:58.044+0000 7fe1f5427640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:17:58.047 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:17:58.045+0000 7fe1f5427640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:17:58.156 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:17:58.156 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":231,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:16:49.519290+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:16:21.569603+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:13:34.834650+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:15:02.352244+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:17:58.399 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3872 8 365281 365273 0.736951 0 - 0.0299513 2026-02-23T14:17:59.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:17:58 trial068 ceph-mon[38805]: pgmap v2432: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:17:59.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:17:58 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3562849645' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:17:59.399 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3873 8 365281 365273 0.736761 0 - 0.0299513 2026-02-23T14:18:00.399 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3874 8 365281 365273 0.73657 0 - 0.0299513 2026-02-23T14:18:01.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:18:00 trial068 ceph-mon[38805]: pgmap v2433: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:18:01.195 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:18:01.311 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:18:01.400 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3875 8 365281 365273 0.73638 0 - 0.0299513 2026-02-23T14:18:01.413 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:18:01.411+0000 7f581c5dc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:18:01.413 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:18:01.411+0000 7f581c5dc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:18:01.521 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:18:01.521 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":231,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:16:49.519290+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:16:21.569603+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:13:34.834650+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:15:02.352244+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:18:01.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:18:01 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3042405437' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:18:02.400 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3876 8 365281 365273 0.73619 0 - 0.0299513 2026-02-23T14:18:03.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:18:02 trial068 ceph-mon[38805]: pgmap v2434: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:18:03.400 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3877 8 365281 365273 0.736 0 - 0.0299513 2026-02-23T14:18:04.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:18:03 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:18:03] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:18:04.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:18:03 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:18:03.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:18:04.400 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3878 8 365281 365273 0.735811 0 - 0.0299513 2026-02-23T14:18:04.559 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:18:04.675 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:18:04.778 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:18:04.775+0000 7f9b6b072640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:18:04.778 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:18:04.776+0000 7f9b6b072640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:18:04.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:18:04 trial068 ceph-mon[38805]: pgmap v2435: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:18:04.887 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:18:04.887 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":231,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:16:49.519290+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:16:21.569603+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:13:34.834650+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:15:02.352244+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:18:05.400 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3879 8 365281 365273 0.735621 0 - 0.0299513 2026-02-23T14:18:06.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:18:05 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3942380890' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:18:06.400 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T14:18:06.399519+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T14:18:06.400 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T14:18:06.400 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3880 8 365281 365273 0.735431 0 - 0.0299513 2026-02-23T14:18:07.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:18:06 trial068 ceph-mon[38805]: pgmap v2436: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:18:07.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:18:06 trial068 ceph-mon[38805]: 5.e scrub starts 2026-02-23T14:18:07.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:18:06 trial068 ceph-mon[38805]: 5.e scrub ok 2026-02-23T14:18:07.400 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3881 8 365281 365273 0.735242 0 - 0.0299513 2026-02-23T14:18:07.926 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:18:08.045 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:18:08.146 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:18:08.144+0000 7f3dac877640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:18:08.146 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:18:08.144+0000 7f3dac877640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:18:08.254 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:18:08.254 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":231,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:16:49.519290+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:16:21.569603+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:13:34.834650+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:15:02.352244+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:18:08.400 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3882 8 365281 365273 0.735052 0 - 0.0299513 2026-02-23T14:18:09.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:18:08 trial068 ceph-mon[38805]: pgmap v2437: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:18:09.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:18:08 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1057551634' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:18:09.400 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3883 8 365281 365273 0.734863 0 - 0.0299513 2026-02-23T14:18:10.400 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3884 8 365281 365273 0.734674 0 - 0.0299513 2026-02-23T14:18:11.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:18:10 trial068 ceph-mon[38805]: pgmap v2438: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:18:11.291 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:18:11.400 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3885 8 365281 365273 0.734485 0 - 0.0299513 2026-02-23T14:18:11.409 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:18:11.508 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:18:11.506+0000 7f466e8d0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:18:11.509 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:18:11.507+0000 7f466e8d0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:18:11.617 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:18:11.618 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":231,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:16:49.519290+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:16:21.569603+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:13:34.834650+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:15:02.352244+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:18:12.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:18:11 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3000677597' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:18:12.400 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3886 8 365281 365273 0.734296 0 - 0.0299513 2026-02-23T14:18:13.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:18:12 trial068 ceph-mon[38805]: pgmap v2439: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:18:13.400 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3887 8 365281 365273 0.734107 0 - 0.0299513 2026-02-23T14:18:14.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:18:13 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:18:13] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:18:14.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:18:13 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:18:13.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:18:14.400 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3888 8 365281 365273 0.733918 0 - 0.0299513 2026-02-23T14:18:14.656 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:18:14.772 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:18:14.878 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:18:14.875+0000 7f9c93309640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:18:14.878 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:18:14.876+0000 7f9c93309640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:18:14.986 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:18:14.986 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":231,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:16:49.519290+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:16:21.569603+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:13:34.834650+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:15:02.352244+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:18:14.987 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:18:14 trial068 ceph-mon[38805]: pgmap v2440: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:18:15.401 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3889 8 365281 365273 0.733729 0 - 0.0299513 2026-02-23T14:18:16.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:18:15 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3634608642' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:18:16.401 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3890 8 365281 365273 0.733541 0 - 0.0299513 2026-02-23T14:18:17.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:18:16 trial068 ceph-mon[38805]: pgmap v2441: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:18:17.401 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3891 8 365281 365273 0.733352 0 - 0.0299513 2026-02-23T14:18:18.039 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:18:18.159 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:18:18.256 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:18:18.254+0000 7fc807437640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:18:18.256 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:18:18.254+0000 7fc807437640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:18:18.365 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:18:18.365 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":231,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:16:49.519290+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:16:21.569603+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:13:34.834650+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:15:02.352244+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:18:18.401 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3892 8 365281 365273 0.733164 0 - 0.0299513 2026-02-23T14:18:19.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:18:18 trial068 ceph-mon[38805]: pgmap v2442: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:18:19.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:18:18 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1645765931' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:18:19.401 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3893 8 365281 365273 0.732975 0 - 0.0299513 2026-02-23T14:18:20.401 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3894 8 365281 365273 0.732787 0 - 0.0299513 2026-02-23T14:18:20.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:18:20 trial068 ceph-mon[38805]: pgmap v2443: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:18:21.401 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3895 8 365281 365273 0.732599 0 - 0.0299513 2026-02-23T14:18:21.402 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:18:21.520 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:18:21.619 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:18:21.616+0000 7f571b292640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:18:21.619 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:18:21.617+0000 7f571b292640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:18:21.728 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:18:21.728 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":231,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:16:49.519290+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:16:21.569603+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:13:34.834650+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:15:02.352244+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:18:22.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:18:21 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/4025671584' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:18:22.401 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3896 8 365281 365273 0.732411 0 - 0.0299513 2026-02-23T14:18:23.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:18:22 trial068 ceph-mon[38805]: pgmap v2444: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:18:23.401 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3897 8 365281 365273 0.732223 0 - 0.0299513 2026-02-23T14:18:24.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:18:23 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:18:23] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:18:24.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:18:23 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:18:23.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:18:24.401 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3898 8 365281 365273 0.732035 0 - 0.0299513 2026-02-23T14:18:24.766 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:18:24.884 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:18:24.989 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:18:24.986+0000 7fdc437d3640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:18:24.989 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:18:24.986+0000 7fdc437d3640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:18:25.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:18:24 trial068 ceph-mon[38805]: pgmap v2445: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:18:25.098 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:18:25.098 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":231,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:16:49.519290+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:16:21.569603+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:13:34.834650+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:15:02.352244+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:18:25.401 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3899 8 365281 365273 0.731848 0 - 0.0299513 2026-02-23T14:18:26.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:18:25 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3160516520' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:18:26.401 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T14:18:26.400768+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T14:18:26.401 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T14:18:26.402 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3900 8 365281 365273 0.73166 0 - 0.0299513 2026-02-23T14:18:27.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:18:26 trial068 ceph-mon[38805]: pgmap v2446: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:18:27.401 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3901 8 365281 365273 0.731472 0 - 0.0299513 2026-02-23T14:18:28.135 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:18:28.254 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:18:28.359 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:18:28.356+0000 7f7df2700640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:18:28.359 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:18:28.357+0000 7f7df2700640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:18:28.401 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3902 8 365281 365273 0.731285 0 - 0.0299513 2026-02-23T14:18:28.468 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:18:28.468 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":231,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:16:49.519290+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:16:21.569603+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:13:34.834650+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:15:02.352244+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:18:29.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:18:28 trial068 ceph-mon[38805]: pgmap v2447: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:18:29.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:18:28 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3301516300' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:18:29.401 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3903 8 365281 365273 0.731098 0 - 0.0299513 2026-02-23T14:18:30.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:18:29 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T14:18:30.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:18:29 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T14:18:30.402 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3904 8 365281 365273 0.73091 0 - 0.0299513 2026-02-23T14:18:31.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:18:30 trial068 ceph-mon[38805]: pgmap v2448: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:18:31.402 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3905 8 365281 365273 0.730723 0 - 0.0299513 2026-02-23T14:18:31.505 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:18:31.626 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:18:31.735 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:18:31.733+0000 7f39b2e9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:18:31.735 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:18:31.733+0000 7f39b2e9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:18:31.845 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:18:31.846 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":231,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:16:49.519290+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:16:21.569603+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:13:34.834650+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:15:02.352244+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:18:32.402 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3906 8 365281 365273 0.730536 0 - 0.0299513 2026-02-23T14:18:33.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:18:32 trial068 ceph-mon[38805]: pgmap v2449: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:18:33.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:18:32 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3236589698' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:18:33.402 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3907 8 365281 365273 0.730349 0 - 0.0299513 2026-02-23T14:18:34.266 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:18:33 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:18:33.974Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:18:34.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:18:33 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:18:33] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:18:34.402 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3908 8 365281 365273 0.730162 0 - 0.0299513 2026-02-23T14:18:34.885 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:18:35.004 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:18:35.052 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:18:34 trial068 ceph-mon[38805]: pgmap v2450: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:18:35.111 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:18:35.109+0000 7f108ef99640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:18:35.111 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:18:35.109+0000 7f108ef99640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:18:35.221 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:18:35.221 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":231,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:16:49.519290+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:16:21.569603+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:13:34.834650+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:15:02.352244+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:18:35.402 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3909 8 365281 365273 0.729975 0 - 0.0299513 2026-02-23T14:18:36.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:18:35 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3000181664' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:18:36.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:18:35 trial068 ceph-mon[38805]: pgmap v2451: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:18:36.402 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3910 8 365281 365273 0.729789 0 - 0.0299513 2026-02-23T14:18:37.402 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3911 8 365281 365273 0.729602 0 - 0.0299513 2026-02-23T14:18:38.259 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:18:38.376 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:18:38.402 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3912 8 365281 365273 0.729416 0 - 0.0299513 2026-02-23T14:18:38.474 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:18:38.472+0000 7f14efe0d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:18:38.474 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:18:38.472+0000 7f14efe0d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:18:38.584 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:18:38.584 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":231,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:16:49.519290+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:16:21.569603+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:13:34.834650+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:15:02.352244+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:18:38.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:18:38 trial068 ceph-mon[38805]: pgmap v2452: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:18:38.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:18:38 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1204914571' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:18:39.402 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3913 8 365281 365273 0.729229 0 - 0.0299513 2026-02-23T14:18:40.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:18:39 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T14:18:40.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:18:39 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T14:18:40.402 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3914 8 365281 365273 0.729043 0 - 0.0299513 2026-02-23T14:18:41.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:18:40 trial068 ceph-mon[38805]: pgmap v2453: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:18:41.402 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3915 8 365281 365273 0.728857 0 - 0.0299513 2026-02-23T14:18:41.622 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:18:41.741 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:18:41.842 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:18:41.840+0000 7f83e2105640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:18:41.842 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:18:41.840+0000 7f83e2105640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:18:41.951 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:18:41.951 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":231,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:16:49.519290+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:16:21.569603+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:13:34.834650+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:15:02.352244+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:18:42.402 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3916 8 365281 365273 0.728671 0 - 0.0299513 2026-02-23T14:18:43.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:18:42 trial068 ceph-mon[38805]: pgmap v2454: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:18:43.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:18:42 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1839317436' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:18:43.402 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3917 8 365281 365273 0.728485 0 - 0.0299513 2026-02-23T14:18:44.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:18:43 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:18:43] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:18:44.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:18:43 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:18:43.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:18:44.402 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3918 8 365281 365273 0.728299 0 - 0.0299513 2026-02-23T14:18:44.989 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:18:45.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:18:44 trial068 ceph-mon[38805]: pgmap v2455: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:18:45.108 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:18:45.209 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:18:45.206+0000 7f6e01fbd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:18:45.209 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:18:45.207+0000 7f6e01fbd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:18:45.321 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:18:45.321 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":231,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:16:49.519290+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:16:21.569603+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:13:34.834650+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:15:02.352244+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:18:45.402 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3919 8 365281 365273 0.728113 0 - 0.0299513 2026-02-23T14:18:46.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:18:45 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2902076719' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:18:46.403 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T14:18:46.402150+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T14:18:46.403 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T14:18:46.403 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3920 8 365281 365273 0.727927 0 - 0.0299513 2026-02-23T14:18:47.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:18:46 trial068 ceph-mon[38805]: pgmap v2456: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:18:47.403 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3921 8 365281 365273 0.727741 0 - 0.0299513 2026-02-23T14:18:48.358 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:18:48.403 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3922 8 365281 365273 0.727556 0 - 0.0299513 2026-02-23T14:18:48.476 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:18:48.579 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:18:48.577+0000 7fe04c9ee640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:18:48.579 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:18:48.577+0000 7fe04c9ee640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:18:48.693 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:18:48.693 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":231,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:16:49.519290+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:16:21.569603+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:13:34.834650+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:15:02.352244+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:18:49.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:18:48 trial068 ceph-mon[38805]: pgmap v2457: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:18:49.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:18:48 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/751421647' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:18:49.403 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3923 8 365281 365273 0.72737 0 - 0.0299513 2026-02-23T14:18:50.403 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3924 8 365281 365273 0.727185 0 - 0.0299513 2026-02-23T14:18:50.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:18:50 trial068 ceph-mon[38805]: pgmap v2458: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:18:51.403 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3925 8 365281 365273 0.727 0 - 0.0299513 2026-02-23T14:18:51.733 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:18:51.852 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:18:51.955 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:18:51.953+0000 7fc6a18f4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:18:51.956 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:18:51.953+0000 7fc6a18f4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:18:52.067 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:18:52.067 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":231,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:16:49.519290+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:16:21.569603+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:13:34.834650+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:15:02.352244+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:18:52.403 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3926 8 365281 365273 0.726815 0 - 0.0299513 2026-02-23T14:18:53.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:18:52 trial068 ceph-mon[38805]: pgmap v2459: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:18:53.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:18:52 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1316578296' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:18:53.403 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3927 8 365281 365273 0.726629 0 - 0.0299513 2026-02-23T14:18:54.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:18:53 trial068 ceph-mon[38805]: 5.a scrub starts 2026-02-23T14:18:54.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:18:53 trial068 ceph-mon[38805]: 5.a scrub ok 2026-02-23T14:18:54.094 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:18:53 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:18:53] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:18:54.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:18:53 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:18:53.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:18:54.403 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3928 8 365281 365273 0.726444 0 - 0.0299513 2026-02-23T14:18:55.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:18:54 trial068 ceph-mon[38805]: pgmap v2460: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:18:55.104 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:18:55.222 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:18:55.320 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:18:55.318+0000 7f7fae54e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:18:55.320 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:18:55.318+0000 7f7fae54e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:18:55.403 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3929 8 365281 365273 0.72626 0 - 0.0299513 2026-02-23T14:18:55.429 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:18:55.429 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":231,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:16:49.519290+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:16:21.569603+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:13:34.834650+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:15:02.352244+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:18:56.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:18:55 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2484429732' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:18:56.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:18:55 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2026-02-23T14:18:56.403 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3930 8 365281 365273 0.726075 0 - 0.0299513 2026-02-23T14:18:57.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:18:56 trial068 ceph-mon[38805]: pgmap v2461: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:18:57.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:18:56 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-23T14:18:57.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:18:56 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2026-02-23T14:18:57.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:18:56 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T14:18:57.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:18:56 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2026-02-23T14:18:57.403 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3931 8 365281 365273 0.72589 0 - 0.0299513 2026-02-23T14:18:58.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:18:57 trial068 ceph-mon[38805]: pgmap v2462: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:18:58.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:18:57 trial068 ceph-mon[38805]: pgmap v2463: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:18:58.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:18:57 trial068 ceph-mon[38805]: 5.5 scrub starts 2026-02-23T14:18:58.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:18:57 trial068 ceph-mon[38805]: 5.5 scrub ok 2026-02-23T14:18:58.403 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3932 8 365281 365273 0.725705 0 - 0.0299513 2026-02-23T14:18:58.485 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:18:58.602 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:18:58.701 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:18:58.699+0000 7fc7b12d0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:18:58.701 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:18:58.699+0000 7fc7b12d0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:18:58.809 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:18:58.810 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":232,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:18:57.772265+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:16:21.569603+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:18:28.405361+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:15:02.352244+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:18:59.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:18:58 trial068 ceph-mon[38805]: osdmap e232: 4 total, 3 up, 4 in 2026-02-23T14:18:59.403 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3933 8 365281 365273 0.725521 0 - 0.0299513 2026-02-23T14:19:00.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:18:59 trial068 ceph-mon[38805]: pgmap v2465: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:19:00.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:18:59 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3925818263' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:19:00.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:18:59 trial068 ceph-mon[38805]: osdmap e233: 4 total, 3 up, 4 in 2026-02-23T14:19:00.404 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3934 8 365281 365273 0.725337 0 - 0.0299513 2026-02-23T14:19:01.404 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3935 8 365281 365273 0.725152 0 - 0.0299513 2026-02-23T14:19:01.849 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:19:01.968 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:19:02.069 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:19:02.067+0000 7f3f381c8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:19:02.069 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:19:02.067+0000 7f3f381c8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:19:02.070 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:19:01 trial068 ceph-mon[38805]: pgmap v2467: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:19:02.179 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:19:02.179 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":233,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:18:59.536451+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:16:21.569603+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:18:28.405361+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:15:02.352244+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:19:02.404 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3936 8 365281 365273 0.724968 0 - 0.0299513 2026-02-23T14:19:03.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:19:02 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3505416069' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:19:03.404 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3937 8 365281 365273 0.724784 0 - 0.0299513 2026-02-23T14:19:04.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:19:03 trial068 ceph-mon[38805]: pgmap v2468: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:19:04.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:19:03 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T14:19:04.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:19:03 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T14:19:04.094 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:19:03 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:19:03] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:19:04.266 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:19:03 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:19:03.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:19:04.404 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3938 8 365281 365273 0.7246 0 - 0.0299513 2026-02-23T14:19:05.218 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:19:05.337 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:19:05.404 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3939 8 365281 365273 0.724416 0 - 0.0299513 2026-02-23T14:19:05.438 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:19:05.435+0000 7fd609d42640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:19:05.438 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:19:05.436+0000 7fd609d42640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:19:05.546 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:19:05.546 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":233,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:18:59.536451+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:16:21.569603+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:18:28.405361+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:15:02.352244+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:19:05.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:19:05 trial068 ceph-mon[38805]: pgmap v2469: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:19:05.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:19:05 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1418158690' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:19:06.404 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T14:19:06.403455+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T14:19:06.404 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T14:19:06.404 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3940 8 365281 365273 0.724232 0 - 0.0299513 2026-02-23T14:19:07.404 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3941 8 365281 365273 0.724048 0 - 0.0299513 2026-02-23T14:19:08.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:19:07 trial068 ceph-mon[38805]: pgmap v2470: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:19:08.404 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3942 8 365281 365273 0.723865 0 - 0.0299513 2026-02-23T14:19:08.584 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:19:08.703 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:19:08.809 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:19:08.807+0000 7fe9a8f0a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:19:08.809 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:19:08.807+0000 7fe9a8f0a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:19:08.919 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:19:08.919 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":233,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:18:59.536451+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:16:21.569603+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:18:28.405361+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:15:02.352244+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:19:08.919 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:19:08 trial068 ceph-mon[38805]: 5.8 scrub starts 2026-02-23T14:19:08.920 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:19:08 trial068 ceph-mon[38805]: 5.8 scrub ok 2026-02-23T14:19:09.404 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3943 8 365281 365273 0.723681 0 - 0.0299513 2026-02-23T14:19:10.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:19:09 trial068 ceph-mon[38805]: pgmap v2471: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:19:10.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:19:09 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/676645320' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:19:10.404 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3944 8 365281 365273 0.723497 0 - 0.0299513 2026-02-23T14:19:11.404 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3945 8 365281 365273 0.723314 0 - 0.0299513 2026-02-23T14:19:11.956 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:19:12.075 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:19:12.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:19:11 trial068 ceph-mon[38805]: pgmap v2472: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:19:12.180 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:19:12.178+0000 7f978fca9640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:19:12.180 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:19:12.178+0000 7f978fca9640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:19:12.296 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:19:12.296 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":233,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:18:59.536451+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:16:21.569603+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:18:28.405361+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:15:02.352244+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:19:12.404 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3946 8 365281 365273 0.723131 0 - 0.0299513 2026-02-23T14:19:13.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:19:12 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1972801172' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:19:13.404 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3947 8 365281 365273 0.722948 0 - 0.0299513 2026-02-23T14:19:14.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:19:13 trial068 ceph-mon[38805]: pgmap v2473: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:19:14.094 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:19:13 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:19:13] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:19:14.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:19:13 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:19:13.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:19:14.404 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3948 8 365281 365273 0.722764 0 - 0.0299513 2026-02-23T14:19:15.334 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:19:15.404 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3949 8 365281 365273 0.722581 0 - 0.0299513 2026-02-23T14:19:15.451 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:19:15.549 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:19:15.546+0000 7f3a8f905640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:19:15.549 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:19:15.547+0000 7f3a8f905640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:19:15.658 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:19:15.659 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":233,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:18:59.536451+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:16:21.569603+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:18:28.405361+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:15:02.352244+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:19:15.659 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:19:15 trial068 ceph-mon[38805]: pgmap v2474: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:19:16.405 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3950 8 365281 365273 0.722398 0 - 0.0299513 2026-02-23T14:19:16.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:19:16 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3260227846' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:19:17.405 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3951 8 365281 365273 0.722216 0 - 0.0299513 2026-02-23T14:19:17.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:19:17 trial068 ceph-mon[38805]: pgmap v2475: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:19:17.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:19:17 trial068 ceph-mon[38805]: 5.7 scrub starts 2026-02-23T14:19:17.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:19:17 trial068 ceph-mon[38805]: 5.7 scrub ok 2026-02-23T14:19:18.405 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3952 8 365281 365273 0.722033 0 - 0.0299513 2026-02-23T14:19:18.698 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:19:18.819 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:19:18.925 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:19:18.923+0000 7f3788bc5640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:19:18.925 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:19:18.923+0000 7f3788bc5640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:19:19.035 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:19:19.035 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":233,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:18:59.536451+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:16:21.569603+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:18:28.405361+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:15:02.352244+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:19:19.405 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3953 8 365281 365273 0.72185 0 - 0.0299513 2026-02-23T14:19:19.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:19:19 trial068 ceph-mon[38805]: pgmap v2476: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:19:19.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:19:19 trial068 ceph-mon[38805]: 5.e scrub starts 2026-02-23T14:19:19.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:19:19 trial068 ceph-mon[38805]: 5.e scrub ok 2026-02-23T14:19:19.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:19:19 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/4017193667' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:19:20.405 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3954 8 365281 365273 0.721668 0 - 0.0299513 2026-02-23T14:19:21.405 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3955 8 365281 365273 0.721485 0 - 0.0299513 2026-02-23T14:19:21.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:19:21 trial068 ceph-mon[38805]: pgmap v2477: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:19:22.073 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:19:22.192 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:19:22.294 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:19:22.292+0000 7f5857554640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:19:22.294 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:19:22.292+0000 7f5857554640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:19:22.404 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:19:22.405 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":233,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:18:59.536451+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:16:21.569603+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:18:28.405361+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:15:02.352244+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:19:22.405 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3956 8 365281 365273 0.721303 0 - 0.0299513 2026-02-23T14:19:22.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:19:22 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1823439222' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:19:23.405 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3957 8 365281 365273 0.721121 0 - 0.0299513 2026-02-23T14:19:23.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:19:23 trial068 ceph-mon[38805]: pgmap v2478: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:19:24.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:19:23 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:19:23] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:19:24.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:19:23 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:19:23.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:19:24.405 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3958 8 365281 365273 0.720938 0 - 0.0299513 2026-02-23T14:19:25.405 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3959 8 365281 365273 0.720756 0 - 0.0299513 2026-02-23T14:19:25.443 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:19:25.562 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:19:25.662 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:19:25.660+0000 7f1875477640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:19:25.662 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:19:25.660+0000 7f1875477640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:19:25.773 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:19:25.773 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":233,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:18:59.536451+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:16:21.569603+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:18:28.405361+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:15:02.352244+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:19:25.774 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:19:25 trial068 ceph-mon[38805]: pgmap v2479: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:19:26.405 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T14:19:26.404898+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T14:19:26.406 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T14:19:26.406 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3960 8 365281 365273 0.720574 0 - 0.0299513 2026-02-23T14:19:26.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:19:26 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/633449328' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:19:27.405 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3961 8 365281 365273 0.720392 0 - 0.0299513 2026-02-23T14:19:27.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:19:27 trial068 ceph-mon[38805]: pgmap v2480: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:19:28.405 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3962 8 365281 365273 0.720211 0 - 0.0299513 2026-02-23T14:19:28.812 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:19:28.931 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:19:29.035 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:19:29.033+0000 7fd128448640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:19:29.035 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:19:29.033+0000 7fd128448640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:19:29.145 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:19:29.145 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":233,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:18:59.536451+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:16:21.569603+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:18:28.405361+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:15:02.352244+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:19:29.405 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3963 8 365281 365273 0.720029 0 - 0.0299513 2026-02-23T14:19:29.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:19:29 trial068 ceph-mon[38805]: pgmap v2481: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:19:29.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:19:29 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3296291643' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:19:30.406 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3964 8 365281 365273 0.719847 0 - 0.0299513 2026-02-23T14:19:31.406 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3965 8 365281 365273 0.719666 0 - 0.0299513 2026-02-23T14:19:31.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:19:31 trial068 ceph-mon[38805]: pgmap v2482: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:19:32.182 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:19:32.297 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:19:32.404 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:19:32.402+0000 7f5877805640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:19:32.404 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:19:32.402+0000 7f5877805640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:19:32.406 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3966 8 365281 365273 0.719484 0 - 0.0299513 2026-02-23T14:19:32.512 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:19:32.512 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":233,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:18:59.536451+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:16:21.569603+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:18:28.405361+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:15:02.352244+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:19:32.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:19:32 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3524800900' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:19:33.406 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3967 8 365281 365273 0.719303 0 - 0.0299513 2026-02-23T14:19:33.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:19:33 trial068 ceph-mon[38805]: pgmap v2483: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:19:34.266 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:19:33 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:19:33.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:19:34.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:19:33 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:19:33] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:19:34.406 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3968 8 365281 365273 0.719122 0 - 0.0299513 2026-02-23T14:19:35.406 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3969 8 365281 365273 0.71894 0 - 0.0299513 2026-02-23T14:19:35.549 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:19:35.667 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:19:35.767 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:19:35.764+0000 7fded75b0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:19:35.767 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:19:35.764+0000 7fded75b0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:19:35.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:19:35 trial068 ceph-mon[38805]: pgmap v2484: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:19:35.875 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:19:35.876 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":233,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:18:59.536451+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:16:21.569603+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:18:28.405361+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:15:02.352244+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:19:36.406 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3970 8 365281 365273 0.718759 0 - 0.0299513 2026-02-23T14:19:36.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:19:36 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2586014466' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:19:37.406 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3971 8 365281 365273 0.718578 0 - 0.0299513 2026-02-23T14:19:37.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:19:37 trial068 ceph-mon[38805]: pgmap v2485: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:19:38.406 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3972 8 365281 365273 0.718397 0 - 0.0299513 2026-02-23T14:19:38.913 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:19:39.031 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:19:39.130 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:19:39.127+0000 7f61f4b3c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:19:39.130 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:19:39.128+0000 7f61f4b3c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:19:39.238 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:19:39.238 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":233,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:18:59.536451+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:16:21.569603+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:18:28.405361+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:15:02.352244+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:19:39.406 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3973 8 365281 365273 0.718217 0 - 0.0299513 2026-02-23T14:19:39.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:19:39 trial068 ceph-mon[38805]: pgmap v2486: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:19:39.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:19:39 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2545283082' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:19:40.406 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3974 8 365281 365273 0.718036 0 - 0.0299513 2026-02-23T14:19:41.406 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3975 8 365281 365273 0.717855 0 - 0.0299513 2026-02-23T14:19:41.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:19:41 trial068 ceph-mon[38805]: pgmap v2487: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:19:42.275 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:19:42.393 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:19:42.406 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3976 8 365281 365273 0.717675 0 - 0.0299513 2026-02-23T14:19:42.492 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:19:42.490+0000 7f1427fcb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:19:42.493 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:19:42.490+0000 7f1427fcb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:19:42.601 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:19:42.601 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":233,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:18:59.536451+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:16:21.569603+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:18:28.405361+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:15:02.352244+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:19:43.406 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3977 8 365281 365273 0.717494 0 - 0.0299513 2026-02-23T14:19:43.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:19:43 trial068 ceph-mon[38805]: pgmap v2488: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:19:43.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:19:43 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/4106785025' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:19:44.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:19:43 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:19:43] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:19:44.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:19:43 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:19:43.974Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:19:44.406 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3978 8 365281 365273 0.717314 0 - 0.0299513 2026-02-23T14:19:45.406 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3979 8 365281 365273 0.717134 0 - 0.0299513 2026-02-23T14:19:45.638 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:19:45.758 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:19:45.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:19:45 trial068 ceph-mon[38805]: pgmap v2489: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:19:45.858 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:19:45.856+0000 7fe0ed2bb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:19:45.858 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:19:45.856+0000 7fe0ed2bb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:19:45.966 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:19:45.966 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":233,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:18:59.536451+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:16:21.569603+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:18:28.405361+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:15:02.352244+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:19:46.407 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T14:19:46.406153+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T14:19:46.407 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T14:19:46.407 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3980 8 365281 365273 0.716953 0 - 0.0299513 2026-02-23T14:19:46.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:19:46 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/857937038' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:19:47.407 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3981 8 365281 365273 0.716773 0 - 0.0299513 2026-02-23T14:19:47.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:19:47 trial068 ceph-mon[38805]: pgmap v2490: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:19:48.407 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3982 8 365281 365273 0.716593 0 - 0.0299513 2026-02-23T14:19:49.003 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:19:49.119 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:19:49.216 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:19:49.214+0000 7fc9dee99640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:19:49.217 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:19:49.215+0000 7fc9dee99640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:19:49.326 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:19:49.326 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":233,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:18:59.536451+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:16:21.569603+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:18:28.405361+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:15:02.352244+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:19:49.407 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3983 8 365281 365273 0.716413 0 - 0.0299513 2026-02-23T14:19:49.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:19:49 trial068 ceph-mon[38805]: pgmap v2491: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:19:49.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:19:49 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T14:19:49.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:19:49 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T14:19:49.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:19:49 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/845943513' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:19:50.407 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3984 8 365281 365273 0.716234 0 - 0.0299513 2026-02-23T14:19:51.407 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3985 8 365281 365273 0.716054 0 - 0.0299513 2026-02-23T14:19:51.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:19:51 trial068 ceph-mon[38805]: pgmap v2492: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:19:52.364 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:19:52.407 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3986 8 365281 365273 0.715874 0 - 0.0299513 2026-02-23T14:19:52.484 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:19:52.584 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:19:52.581+0000 7fca42c3a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:19:52.584 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:19:52.582+0000 7fca42c3a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:19:52.693 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:19:52.693 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":233,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:18:59.536451+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:16:21.569603+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:18:28.405361+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:15:02.352244+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:19:53.407 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3987 8 365281 365273 0.715695 0 - 0.0299513 2026-02-23T14:19:53.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:19:53 trial068 ceph-mon[38805]: pgmap v2493: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:19:53.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:19:53 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/419773697' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:19:54.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:19:53 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:19:53] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:19:54.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:19:53 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:19:53.974Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:19:54.407 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3988 8 365281 365273 0.715515 0 - 0.0299513 2026-02-23T14:19:55.407 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3989 8 365281 365273 0.715336 0 - 0.0299513 2026-02-23T14:19:55.731 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:19:55.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:19:55 trial068 ceph-mon[38805]: pgmap v2494: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:19:55.850 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:19:55.946 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:19:55.944+0000 7efca0a53640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:19:55.946 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:19:55.944+0000 7efca0a53640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:19:56.058 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:19:56.058 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":233,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:18:59.536451+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:16:21.569603+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:18:28.405361+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:15:02.352244+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:19:56.407 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3990 8 365281 365273 0.715156 0 - 0.0299513 2026-02-23T14:19:56.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:19:56 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2026-02-23T14:19:56.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:19:56 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3159399386' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:19:56.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:19:56 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-23T14:19:56.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:19:56 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2026-02-23T14:19:56.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:19:56 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T14:19:56.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:19:56 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2026-02-23T14:19:57.407 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3991 8 365281 365273 0.714977 0 - 0.0299513 2026-02-23T14:19:57.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:19:57 trial068 ceph-mon[38805]: pgmap v2495: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:19:57.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:19:57 trial068 ceph-mon[38805]: pgmap v2496: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:19:57.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:19:57 trial068 ceph-mon[38805]: pgmap v2497: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:19:58.407 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3992 8 365281 365273 0.714798 0 - 0.0299513 2026-02-23T14:19:59.099 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:19:59.218 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:19:59.324 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:19:59.322+0000 7f3e25fb3640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:19:59.324 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:19:59.322+0000 7f3e25fb3640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:19:59.407 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3993 8 365281 365273 0.714619 0 - 0.0299513 2026-02-23T14:19:59.435 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:19:59.435 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":233,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:18:59.536451+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:16:21.569603+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:18:28.405361+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:15:02.352244+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:19:59.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:19:59 trial068 ceph-mon[38805]: pgmap v2498: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:19:59.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:19:59 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3200363041' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:20:00.408 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3994 8 365281 365273 0.71444 0 - 0.0299513 2026-02-23T14:20:01.408 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3995 8 365281 365273 0.714261 0 - 0.0299513 2026-02-23T14:20:01.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:20:01 trial068 ceph-mon[38805]: pgmap v2499: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:20:02.408 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3996 8 365281 365273 0.714083 0 - 0.0299513 2026-02-23T14:20:02.472 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:20:02.590 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:20:02.696 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:20:02.694+0000 7f61331b7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:20:02.696 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:20:02.694+0000 7f61331b7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:20:02.804 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:20:02.804 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":233,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:18:59.536451+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:16:21.569603+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:18:28.405361+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:15:02.352244+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:20:03.408 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3997 8 365281 365273 0.713904 0 - 0.0299513 2026-02-23T14:20:03.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:20:03 trial068 ceph-mon[38805]: pgmap v2500: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:20:03.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:20:03 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T14:20:03.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:20:03 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T14:20:03.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:20:03 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1576553286' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:20:04.266 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:20:03 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:20:03.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:20:04.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:20:03 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:20:03] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:20:04.408 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3998 8 365281 365273 0.713725 0 - 0.0299513 2026-02-23T14:20:05.408 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 3999 8 365281 365273 0.713547 0 - 0.0299513 2026-02-23T14:20:05.841 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:20:05.845 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:20:05 trial068 ceph-mon[38805]: pgmap v2501: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:20:05.960 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:20:06.063 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:20:06.061+0000 7fa1ab310640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:20:06.063 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:20:06.061+0000 7fa1ab310640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:20:06.175 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:20:06.175 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":233,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:18:59.536451+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:16:21.569603+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:18:28.405361+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:15:02.352244+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:20:06.408 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T14:20:06.407518+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T14:20:06.408 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T14:20:06.408 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4000 8 365281 365273 0.713369 0 - 0.0299513 2026-02-23T14:20:06.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:20:06 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/783623307' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:20:07.408 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4001 8 365281 365273 0.71319 0 - 0.0299513 2026-02-23T14:20:07.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:20:07 trial068 ceph-mon[38805]: pgmap v2502: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:20:08.408 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4002 8 365281 365273 0.713012 0 - 0.0299513 2026-02-23T14:20:09.214 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:20:09.331 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:20:09.408 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4003 8 365281 365273 0.712834 0 - 0.0299513 2026-02-23T14:20:09.437 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:20:09.435+0000 7f1f9dd95640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:20:09.437 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:20:09.435+0000 7f1f9dd95640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:20:09.545 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:20:09.546 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":233,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:18:59.536451+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:16:21.569603+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:18:28.405361+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:15:02.352244+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:20:09.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:20:09 trial068 ceph-mon[38805]: pgmap v2503: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:20:09.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:20:09 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1040305334' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:20:10.408 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4004 8 365281 365273 0.712656 0 - 0.0299513 2026-02-23T14:20:10.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:20:10 trial068 ceph-mon[38805]: 5.a scrub starts 2026-02-23T14:20:10.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:20:10 trial068 ceph-mon[38805]: 5.a scrub ok 2026-02-23T14:20:11.408 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4005 8 365281 365273 0.712478 0 - 0.0299513 2026-02-23T14:20:11.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:20:11 trial068 ceph-mon[38805]: pgmap v2504: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:20:12.408 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4006 8 365281 365273 0.7123 0 - 0.0299513 2026-02-23T14:20:12.583 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:20:12.700 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:20:12.804 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:20:12.802+0000 7f151fb10640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:20:12.804 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:20:12.802+0000 7f151fb10640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:20:12.915 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:20:12.915 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":233,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:18:59.536451+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:16:21.569603+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:18:28.405361+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:15:02.352244+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:20:13.408 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4007 8 365281 365273 0.712122 0 - 0.0299513 2026-02-23T14:20:13.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:20:13 trial068 ceph-mon[38805]: pgmap v2505: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:20:13.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:20:13 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1385048391' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:20:14.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:20:13 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:20:13] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:20:14.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:20:13 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:20:13.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:20:14.408 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4008 8 365281 365273 0.711945 0 - 0.0299513 2026-02-23T14:20:15.409 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4009 8 365281 365273 0.711767 0 - 0.0299513 2026-02-23T14:20:15.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:20:15 trial068 ceph-mon[38805]: pgmap v2506: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:20:15.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:20:15 trial068 ceph-mon[38805]: osdmap e234: 4 total, 3 up, 4 in 2026-02-23T14:20:15.953 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:20:16.069 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:20:16.166 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:20:16.164+0000 7f8e3aa1e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:20:16.166 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:20:16.164+0000 7f8e3aa1e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:20:16.278 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:20:16.278 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":234,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:20:14.586609+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:16:21.569603+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:18:28.405361+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:20:02.384818+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:20:16.409 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4010 8 365281 365273 0.71159 0 - 0.0299513 2026-02-23T14:20:16.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:20:16 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3296926168' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:20:17.409 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4011 8 365281 365273 0.711412 0 - 0.0299513 2026-02-23T14:20:17.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:20:17 trial068 ceph-mon[38805]: pgmap v2508: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:20:18.409 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4012 8 365281 365273 0.711235 0 - 0.0299513 2026-02-23T14:20:19.316 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:20:19.409 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4013 8 365281 365273 0.711058 0 - 0.0299513 2026-02-23T14:20:19.434 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:20:19.532 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:20:19.530+0000 7f8a55b90640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:20:19.532 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:20:19.530+0000 7f8a55b90640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:20:19.642 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:20:19.642 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":235,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:20:19.547376+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:16:21.569603+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:18:28.405361+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:20:02.384818+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:20:20.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:20:19 trial068 ceph-mon[38805]: 5.5 scrub starts 2026-02-23T14:20:20.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:20:19 trial068 ceph-mon[38805]: 5.5 scrub ok 2026-02-23T14:20:20.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:20:19 trial068 ceph-mon[38805]: pgmap v2509: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:20:20.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:20:19 trial068 ceph-mon[38805]: osdmap e235: 4 total, 3 up, 4 in 2026-02-23T14:20:20.409 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4014 8 365281 365273 0.710881 0 - 0.0299513 2026-02-23T14:20:21.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:20:20 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3836069802' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:20:21.409 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4015 8 365281 365273 0.710704 0 - 0.0299513 2026-02-23T14:20:22.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:20:21 trial068 ceph-mon[38805]: pgmap v2511: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:20:22.409 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4016 8 365281 365273 0.710527 0 - 0.0299513 2026-02-23T14:20:22.684 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:20:22.799 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:20:22.902 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:20:22.899+0000 7f4c43e4f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:20:22.902 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:20:22.899+0000 7f4c43e4f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:20:23.011 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:20:23.011 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":235,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:20:19.547376+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:16:21.569603+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:18:28.405361+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:20:02.384818+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:20:23.409 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4017 8 365281 365273 0.71035 0 - 0.0299513 2026-02-23T14:20:23.975 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:20:23 trial068 ceph-mon[38805]: pgmap v2512: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:20:23.975 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:20:23 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1605670470' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:20:24.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:20:23 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:20:23] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:20:24.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:20:23 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:20:23.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:20:24.409 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4018 8 365281 365273 0.710173 0 - 0.0299513 2026-02-23T14:20:25.410 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4019 8 365281 365273 0.709996 0 - 0.0299513 2026-02-23T14:20:26.051 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:20:26.074 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:20:25 trial068 ceph-mon[38805]: pgmap v2513: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:20:26.167 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:20:26.270 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:20:26.268+0000 7f8f3bce2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:20:26.270 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:20:26.268+0000 7f8f3bce2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:20:26.379 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:20:26.379 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":235,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:20:19.547376+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:16:21.569603+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:18:28.405361+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:20:02.384818+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:20:26.410 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T14:20:26.409107+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T14:20:26.410 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T14:20:26.410 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4020 8 365281 365273 0.70982 0 - 0.0299513 2026-02-23T14:20:26.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:20:26 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3203812984' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:20:27.410 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4021 8 365281 365273 0.709643 0 - 0.0299513 2026-02-23T14:20:28.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:20:27 trial068 ceph-mon[38805]: pgmap v2514: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:20:28.410 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4022 8 365281 365273 0.709467 0 - 0.0299513 2026-02-23T14:20:29.410 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4023 8 365281 365273 0.70929 0 - 0.0299513 2026-02-23T14:20:29.416 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:20:29.532 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:20:29.631 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:20:29.629+0000 7f57c7e86640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:20:29.631 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:20:29.629+0000 7f57c7e86640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:20:29.741 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:20:29.741 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":235,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:20:19.547376+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:16:21.569603+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:18:28.405361+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:20:02.384818+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:20:29.742 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:20:29 trial068 ceph-mon[38805]: pgmap v2515: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:20:30.410 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4024 8 365281 365273 0.709114 0 - 0.0299513 2026-02-23T14:20:31.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:20:30 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3512566752' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:20:31.410 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4025 8 365281 365273 0.708938 0 - 0.0299513 2026-02-23T14:20:31.995 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:20:31 trial068 ceph-mon[38805]: pgmap v2516: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:20:31.995 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:20:31 trial068 ceph-mon[38805]: 5.e scrub starts 2026-02-23T14:20:32.410 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4026 8 365281 365273 0.708762 0 - 0.0299513 2026-02-23T14:20:32.780 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:20:32.897 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:20:32.919 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:20:32 trial068 ceph-mon[38805]: 5.e scrub ok 2026-02-23T14:20:32.999 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:20:32.996+0000 7f4bd4349640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:20:32.999 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:20:32.997+0000 7f4bd4349640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:20:33.109 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:20:33.109 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":235,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:20:19.547376+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:16:21.569603+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:18:28.405361+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:20:02.384818+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:20:33.410 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4027 8 365281 365273 0.708586 0 - 0.0299513 2026-02-23T14:20:33.974 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:20:33 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T14:20:33.974 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:20:33 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T14:20:33.974 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:20:33 trial068 ceph-mon[38805]: pgmap v2517: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:20:33.974 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:20:33 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1988954524' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:20:34.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:20:33 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:20:33] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:20:34.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:20:33 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:20:33.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:20:34.410 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4028 8 365281 365273 0.70841 0 - 0.0299513 2026-02-23T14:20:35.410 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4029 8 365281 365273 0.708234 0 - 0.0299513 2026-02-23T14:20:36.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:20:35 trial068 ceph-mon[38805]: 5.8 scrub starts 2026-02-23T14:20:36.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:20:35 trial068 ceph-mon[38805]: pgmap v2518: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:20:36.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:20:35 trial068 ceph-mon[38805]: 5.8 scrub ok 2026-02-23T14:20:36.146 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:20:36.261 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:20:36.356 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:20:36.353+0000 7fcd171a8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:20:36.356 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:20:36.354+0000 7fcd171a8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:20:36.410 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4030 8 365281 365273 0.708058 0 - 0.0299513 2026-02-23T14:20:36.464 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:20:36.465 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":235,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:20:19.547376+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:16:21.569603+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:18:28.405361+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:20:02.384818+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:20:36.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:20:36 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2744844533' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:20:37.410 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4031 8 365281 365273 0.707883 0 - 0.0299513 2026-02-23T14:20:38.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:20:37 trial068 ceph-mon[38805]: pgmap v2519: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:20:38.410 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4032 8 365281 365273 0.707707 0 - 0.0299513 2026-02-23T14:20:39.410 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4033 8 365281 365273 0.707531 0 - 0.0299513 2026-02-23T14:20:39.501 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:20:39.620 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:20:39.722 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:20:39.720+0000 7f06c43ac640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:20:39.722 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:20:39.720+0000 7f06c43ac640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:20:39.830 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:20:39.831 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":235,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:20:19.547376+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:16:21.569603+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:18:28.405361+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:20:02.384818+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:20:39.831 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:20:39 trial068 ceph-mon[38805]: pgmap v2520: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:20:40.410 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4034 8 365281 365273 0.707356 0 - 0.0299513 2026-02-23T14:20:41.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:20:40 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/654613898' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:20:41.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:20:40 trial068 ceph-mon[38805]: 5.7 scrub starts 2026-02-23T14:20:41.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:20:40 trial068 ceph-mon[38805]: 5.7 scrub ok 2026-02-23T14:20:41.410 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4035 8 365281 365273 0.707181 0 - 0.0299513 2026-02-23T14:20:42.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:20:41 trial068 ceph-mon[38805]: pgmap v2521: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:20:42.411 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4036 8 365281 365273 0.707006 0 - 0.0299513 2026-02-23T14:20:42.866 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:20:42.985 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:20:43.087 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:20:43.085+0000 7fefdc8a0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:20:43.088 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:20:43.085+0000 7fefdc8a0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:20:43.197 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:20:43.197 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":235,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:20:19.547376+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:16:21.569603+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:18:28.405361+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:20:02.384818+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:20:43.411 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4037 8 365281 365273 0.70683 0 - 0.0299513 2026-02-23T14:20:43.974 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:20:43 trial068 ceph-mon[38805]: pgmap v2522: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:20:43.974 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:20:43 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/4114880866' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:20:44.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:20:43 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:20:43] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:20:44.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:20:43 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:20:43.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:20:44.411 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4038 8 365281 365273 0.706655 0 - 0.0299513 2026-02-23T14:20:45.411 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4039 8 365281 365273 0.70648 0 - 0.0299513 2026-02-23T14:20:46.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:20:45 trial068 ceph-mon[38805]: pgmap v2523: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:20:46.235 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:20:46.353 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:20:46.411 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T14:20:46.410411+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T14:20:46.411 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T14:20:46.411 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4040 8 365281 365273 0.706306 0 - 0.0299513 2026-02-23T14:20:46.464 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:20:46.462+0000 7f1c99b75640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:20:46.465 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:20:46.462+0000 7f1c99b75640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:20:46.572 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:20:46.572 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":235,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:20:19.547376+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:16:21.569603+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:18:28.405361+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:20:02.384818+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:20:46.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:20:46 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/144599822' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:20:47.411 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4041 8 365281 365273 0.706131 0 - 0.0299513 2026-02-23T14:20:48.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:20:47 trial068 ceph-mon[38805]: pgmap v2524: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:20:48.411 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4042 8 365281 365273 0.705956 0 - 0.0299513 2026-02-23T14:20:49.411 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4043 8 365281 365273 0.705781 0 - 0.0299513 2026-02-23T14:20:49.609 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:20:49.728 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:20:49.823 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:20:49.821+0000 7fb7d05fd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:20:49.823 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:20:49.821+0000 7fb7d05fd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:20:49.932 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:20:49.932 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":235,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:20:19.547376+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:16:21.569603+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:18:28.405361+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:20:02.384818+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:20:49.933 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:20:49 trial068 ceph-mon[38805]: pgmap v2525: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:20:50.411 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4044 8 365281 365273 0.705607 0 - 0.0299513 2026-02-23T14:20:51.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:20:50 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1441627273' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:20:51.411 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4045 8 365281 365273 0.705433 0 - 0.0299513 2026-02-23T14:20:52.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:20:51 trial068 ceph-mon[38805]: pgmap v2526: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:20:52.411 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4046 8 365281 365273 0.705258 0 - 0.0299513 2026-02-23T14:20:52.969 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:20:53.088 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:20:53.185 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:20:53.183+0000 7f48d0734640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:20:53.186 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:20:53.183+0000 7f48d0734640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:20:53.294 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:20:53.295 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":235,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:20:19.547376+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:16:21.569603+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:18:28.405361+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:20:02.384818+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:20:53.411 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4047 8 365281 365273 0.705084 0 - 0.0299513 2026-02-23T14:20:53.975 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:20:53 trial068 ceph-mon[38805]: pgmap v2527: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:20:53.975 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:20:53 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T14:20:53.975 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:20:53 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T14:20:53.975 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:20:53 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/262180483' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:20:54.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:20:53 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:20:53] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:20:54.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:20:53 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:20:53.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:20:54.411 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4048 8 365281 365273 0.70491 0 - 0.0299513 2026-02-23T14:20:55.412 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4049 8 365281 365273 0.704736 0 - 0.0299513 2026-02-23T14:20:56.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:20:55 trial068 ceph-mon[38805]: pgmap v2528: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:20:56.332 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:20:56.412 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4050 8 365281 365273 0.704562 0 - 0.0299513 2026-02-23T14:20:56.448 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:20:56.561 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:20:56.559+0000 7f5c187a8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:20:56.562 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:20:56.560+0000 7f5c187a8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:20:56.670 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:20:56.670 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":235,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:20:19.547376+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:16:21.569603+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:18:28.405361+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:20:02.384818+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:20:57.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:20:56 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2026-02-23T14:20:57.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:20:56 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-23T14:20:57.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:20:56 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2026-02-23T14:20:57.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:20:56 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T14:20:57.095 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:20:56 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2026-02-23T14:20:57.412 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4051 8 365281 365273 0.704388 0 - 0.0299513 2026-02-23T14:20:58.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:20:57 trial068 ceph-mon[38805]: pgmap v2529: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:20:58.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:20:57 trial068 ceph-mon[38805]: pgmap v2530: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:20:58.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:20:57 trial068 ceph-mon[38805]: pgmap v2531: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:20:58.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:20:57 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3742591125' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:20:58.412 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4052 8 365281 365273 0.704214 0 - 0.0299513 2026-02-23T14:20:59.412 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4053 8 365281 365273 0.70404 0 - 0.0299513 2026-02-23T14:20:59.709 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:20:59.826 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:20:59.929 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:20:59 trial068 ceph-mon[38805]: pgmap v2532: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:20:59.930 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:20:59.927+0000 7fe855f48640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:20:59.930 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:20:59.927+0000 7fe855f48640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:21:00.038 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:21:00.038 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":235,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:20:19.547376+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:16:21.569603+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:18:28.405361+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:20:02.384818+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:21:00.412 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4054 8 365281 365273 0.703866 0 - 0.0299513 2026-02-23T14:21:01.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:21:00 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3046727193' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:21:01.412 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4055 8 365281 365273 0.703693 0 - 0.0299513 2026-02-23T14:21:02.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:21:01 trial068 ceph-mon[38805]: pgmap v2533: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:21:02.412 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4056 8 365281 365273 0.703519 0 - 0.0299513 2026-02-23T14:21:03.074 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:21:03.193 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:21:03.292 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:21:03.290+0000 7fc505662640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:21:03.292 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:21:03.290+0000 7fc505662640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:21:03.402 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:21:03.402 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":235,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:20:19.547376+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:16:21.569603+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:18:28.405361+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:20:02.384818+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:21:03.412 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4057 8 365281 365273 0.703346 0 - 0.0299513 2026-02-23T14:21:03.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:21:03 trial068 ceph-mon[38805]: pgmap v2534: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:21:03.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:21:03 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1613934520' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:21:04.266 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:21:03 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:21:03.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:21:04.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:21:03 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:21:03] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:21:04.412 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4058 8 365281 365273 0.703173 0 - 0.0299513 2026-02-23T14:21:05.412 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4059 8 365281 365273 0.702999 0 - 0.0299513 2026-02-23T14:21:06.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:21:05 trial068 ceph-mon[38805]: pgmap v2535: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:21:06.412 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T14:21:06.411720+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T14:21:06.412 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T14:21:06.412 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4060 8 365281 365273 0.702826 0 - 0.0299513 2026-02-23T14:21:06.439 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:21:06.556 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:21:06.656 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:21:06.654+0000 7f85dcdfc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:21:06.656 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:21:06.654+0000 7f85dcdfc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:21:06.765 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:21:06.765 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":235,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:20:19.547376+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:16:21.569603+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:18:28.405361+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:20:02.384818+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:21:07.412 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4061 8 365281 365273 0.702653 0 - 0.0299513 2026-02-23T14:21:08.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:21:07 trial068 ceph-mon[38805]: pgmap v2536: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:21:08.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:21:07 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/969553899' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:21:08.412 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4062 8 365281 365273 0.70248 0 - 0.0299513 2026-02-23T14:21:09.412 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4063 8 365281 365273 0.702307 0 - 0.0299513 2026-02-23T14:21:09.801 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:21:09.921 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:21:09.944 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:21:09 trial068 ceph-mon[38805]: pgmap v2537: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:21:10.023 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:21:10.020+0000 7f851b92d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:21:10.023 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:21:10.021+0000 7f851b92d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:21:10.132 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:21:10.132 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":235,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:20:19.547376+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:16:21.569603+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:18:28.405361+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:20:02.384818+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:21:10.412 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4064 8 365281 365273 0.702135 0 - 0.0299513 2026-02-23T14:21:11.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:21:10 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1317732617' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:21:11.412 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4065 8 365281 365273 0.701962 0 - 0.0299513 2026-02-23T14:21:12.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:21:11 trial068 ceph-mon[38805]: pgmap v2538: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:21:12.413 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4066 8 365281 365273 0.701789 0 - 0.0299513 2026-02-23T14:21:13.170 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:21:13.288 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:21:13.391 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:21:13.389+0000 7efc8cf6f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:21:13.391 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:21:13.389+0000 7efc8cf6f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:21:13.412 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4067 8 365281 365273 0.701617 0 - 0.0299513 2026-02-23T14:21:13.500 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:21:13.500 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":235,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:20:19.547376+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:16:21.569603+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:18:28.405361+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:20:02.384818+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:21:13.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:21:13 trial068 ceph-mon[38805]: pgmap v2539: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:21:13.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:21:13 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3608557700' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:21:14.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:21:13 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:21:13] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:21:14.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:21:13 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:21:13.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:21:14.413 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4068 8 365281 365273 0.701444 0 - 0.0299513 2026-02-23T14:21:15.413 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4069 8 365281 365273 0.701272 0 - 0.0299513 2026-02-23T14:21:16.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:21:15 trial068 ceph-mon[38805]: pgmap v2540: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:21:16.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:21:15 trial068 ceph-mon[38805]: 5.a scrub starts 2026-02-23T14:21:16.413 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4070 8 365281 365273 0.701099 0 - 0.0299513 2026-02-23T14:21:16.538 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:21:16.656 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:21:16.759 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:21:16.757+0000 7fa6c1f4f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:21:16.759 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:21:16.757+0000 7fa6c1f4f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:21:16.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:21:16 trial068 ceph-mon[38805]: 5.a scrub ok 2026-02-23T14:21:16.870 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:21:16.871 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":235,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:20:19.547376+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:16:21.569603+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:18:28.405361+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:20:02.384818+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:21:17.413 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4071 8 365281 365273 0.700927 0 - 0.0299513 2026-02-23T14:21:18.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:21:17 trial068 ceph-mon[38805]: pgmap v2541: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:21:18.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:21:17 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/393973173' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:21:18.413 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4072 8 365281 365273 0.700755 0 - 0.0299513 2026-02-23T14:21:19.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:21:18 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T14:21:19.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:21:18 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T14:21:19.413 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4073 8 365281 365273 0.700583 0 - 0.0299513 2026-02-23T14:21:19.908 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:21:19.933 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:21:19 trial068 ceph-mon[38805]: pgmap v2542: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:21:20.026 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:21:20.128 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:21:20.126+0000 7f335c8f6640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:21:20.128 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:21:20.126+0000 7f335c8f6640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:21:20.236 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:21:20.237 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":235,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:20:19.547376+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:16:21.569603+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:18:28.405361+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:20:02.384818+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:21:20.413 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4074 8 365281 365273 0.700411 0 - 0.0299513 2026-02-23T14:21:21.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:21:20 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1187551281' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:21:21.413 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4075 8 365281 365273 0.700239 0 - 0.0299513 2026-02-23T14:21:22.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:21:21 trial068 ceph-mon[38805]: pgmap v2543: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:21:22.413 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4076 8 365281 365273 0.700067 0 - 0.0299513 2026-02-23T14:21:23.276 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:21:23.393 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:21:23.413 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4077 8 365281 365273 0.699896 0 - 0.0299513 2026-02-23T14:21:23.491 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:21:23.489+0000 7f97a7ba3640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:21:23.491 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:21:23.489+0000 7f97a7ba3640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:21:23.600 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:21:23.601 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":235,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:20:19.547376+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:16:21.569603+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:18:28.405361+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:20:02.384818+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:21:23.974 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:21:23 trial068 ceph-mon[38805]: pgmap v2544: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:21:23.974 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:21:23 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2010650859' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:21:24.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:21:23 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:21:23] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:21:24.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:21:23 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:21:23.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:21:24.413 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4078 8 365281 365273 0.699724 0 - 0.0299513 2026-02-23T14:21:25.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:21:24 trial068 ceph-mon[38805]: osdmap e236: 4 total, 3 up, 4 in 2026-02-23T14:21:25.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:21:24 trial068 ceph-mon[38805]: osdmap e237: 4 total, 3 up, 4 in 2026-02-23T14:21:25.413 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4079 8 365281 365273 0.699553 0 - 0.0299513 2026-02-23T14:21:26.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:21:25 trial068 ceph-mon[38805]: pgmap v2546: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:21:26.414 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T14:21:26.413009+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T14:21:26.414 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T14:21:26.414 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4080 8 365281 365273 0.699381 0 - 0.0299513 2026-02-23T14:21:26.637 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:21:26.754 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:21:26.860 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:21:26.858+0000 7fc215a1e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:21:26.860 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:21:26.858+0000 7fc215a1e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:21:26.969 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:21:26.970 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":237,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:21:24.555495+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:20:32.175012+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:18:28.405361+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:20:02.384818+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:21:27.414 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4081 8 365281 365273 0.69921 0 - 0.0299513 2026-02-23T14:21:28.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:21:27 trial068 ceph-mon[38805]: pgmap v2548: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:21:28.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:21:27 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/616435720' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:21:28.414 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4082 8 365281 365273 0.699038 0 - 0.0299513 2026-02-23T14:21:29.414 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4083 8 365281 365273 0.698867 0 - 0.0299513 2026-02-23T14:21:30.009 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:21:30.033 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:21:29 trial068 ceph-mon[38805]: pgmap v2549: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:21:30.126 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:21:30.228 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:21:30.225+0000 7fd07a7b0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:21:30.228 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:21:30.226+0000 7fd07a7b0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:21:30.340 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:21:30.340 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":237,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:21:24.555495+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:20:32.175012+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:18:28.405361+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:20:02.384818+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:21:30.414 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4084 8 365281 365273 0.698696 0 - 0.0299513 2026-02-23T14:21:31.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:21:30 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1223441645' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:21:31.414 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4085 8 365281 365273 0.698525 0 - 0.0299513 2026-02-23T14:21:32.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:21:31 trial068 ceph-mon[38805]: pgmap v2550: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:21:32.414 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4086 8 365281 365273 0.698354 0 - 0.0299513 2026-02-23T14:21:33.379 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:21:33.414 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4087 8 365281 365273 0.698183 0 - 0.0299513 2026-02-23T14:21:33.497 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:21:33.597 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:21:33.595+0000 7f58a1b6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:21:33.597 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:21:33.595+0000 7f58a1b6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:21:33.709 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:21:33.709 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":237,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:21:24.555495+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:20:32.175012+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:18:28.405361+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:20:02.384818+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:21:33.975 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:21:33 trial068 ceph-mon[38805]: pgmap v2551: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:21:34.266 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:21:33 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:21:33.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:21:34.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:21:33 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:21:33] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:21:34.414 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4088 8 365281 365273 0.698012 0 - 0.0299513 2026-02-23T14:21:35.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:21:34 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/882376989' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:21:35.414 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4089 8 365281 365273 0.697842 0 - 0.0299513 2026-02-23T14:21:36.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:21:35 trial068 ceph-mon[38805]: pgmap v2552: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:21:36.414 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4090 8 365281 365273 0.697671 0 - 0.0299513 2026-02-23T14:21:36.747 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:21:36.865 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:21:36.967 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:21:36.964+0000 7fe796c77640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:21:36.967 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:21:36.965+0000 7fe796c77640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:21:37.074 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:21:37.075 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":237,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:21:24.555495+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:20:32.175012+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:18:28.405361+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:20:02.384818+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:21:37.414 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4091 8 365281 365273 0.697501 0 - 0.0299513 2026-02-23T14:21:38.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:21:37 trial068 ceph-mon[38805]: pgmap v2553: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:21:38.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:21:37 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1325538223' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:21:38.414 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4092 8 365281 365273 0.69733 0 - 0.0299513 2026-02-23T14:21:39.414 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4093 8 365281 365273 0.69716 0 - 0.0299513 2026-02-23T14:21:40.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:21:39 trial068 ceph-mon[38805]: pgmap v2554: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:21:40.111 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:21:40.231 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:21:40.327 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:21:40.325+0000 7f6c6c972640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:21:40.327 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:21:40.325+0000 7f6c6c972640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:21:40.414 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4094 8 365281 365273 0.69699 0 - 0.0299513 2026-02-23T14:21:40.437 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:21:40.437 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":237,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:21:24.555495+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:20:32.175012+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:18:28.405361+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:20:02.384818+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:21:40.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:21:40 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1557091531' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:21:41.414 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4095 8 365281 365273 0.696819 0 - 0.0299513 2026-02-23T14:21:42.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:21:41 trial068 ceph-mon[38805]: pgmap v2555: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:21:42.414 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4096 8 365281 365273 0.696649 0 - 0.0299513 2026-02-23T14:21:43.414 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4097 8 365281 365273 0.696479 0 - 0.0299513 2026-02-23T14:21:43.475 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:21:43.594 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:21:43.692 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:21:43.689+0000 7f924ab13640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:21:43.692 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:21:43.690+0000 7f924ab13640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:21:43.801 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:21:43.801 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":237,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:21:24.555495+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:20:32.175012+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:18:28.405361+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:20:02.384818+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:21:43.801 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:21:43 trial068 ceph-mon[38805]: pgmap v2556: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:21:44.094 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:21:43 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:21:43] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:21:44.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:21:43 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:21:43.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:21:44.414 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4098 8 365281 365273 0.696309 0 - 0.0299513 2026-02-23T14:21:45.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:21:44 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3995352879' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:21:45.415 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4099 8 365281 365273 0.696139 0 - 0.0299513 2026-02-23T14:21:46.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:21:45 trial068 ceph-mon[38805]: pgmap v2557: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:21:46.415 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T14:21:46.414242+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T14:21:46.415 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T14:21:46.415 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4100 8 365281 365273 0.69597 0 - 0.0299513 2026-02-23T14:21:46.838 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:21:46.959 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:21:46.981 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:21:46 trial068 ceph-mon[38805]: 5.5 scrub starts 2026-02-23T14:21:46.981 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:21:46 trial068 ceph-mon[38805]: 5.5 scrub ok 2026-02-23T14:21:47.068 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:21:47.066+0000 7f5faa25f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:21:47.068 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:21:47.066+0000 7f5faa25f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:21:47.177 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:21:47.177 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":237,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:21:24.555495+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:20:32.175012+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:18:28.405361+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:20:02.384818+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:21:47.415 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4101 8 365281 365273 0.6958 0 - 0.0299513 2026-02-23T14:21:48.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:21:47 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T14:21:48.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:21:47 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T14:21:48.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:21:47 trial068 ceph-mon[38805]: pgmap v2558: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:21:48.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:21:47 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/407823584' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:21:48.415 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4102 8 365281 365273 0.69563 0 - 0.0299513 2026-02-23T14:21:49.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:21:48 trial068 ceph-mon[38805]: 5.8 scrub starts 2026-02-23T14:21:49.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:21:48 trial068 ceph-mon[38805]: 5.8 scrub ok 2026-02-23T14:21:49.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:21:48 trial068 ceph-mon[38805]: 5.e scrub starts 2026-02-23T14:21:49.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:21:48 trial068 ceph-mon[38805]: 5.e scrub ok 2026-02-23T14:21:49.415 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4103 8 365281 365273 0.695461 0 - 0.0299513 2026-02-23T14:21:50.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:21:49 trial068 ceph-mon[38805]: 5.7 scrub starts 2026-02-23T14:21:50.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:21:49 trial068 ceph-mon[38805]: 5.7 scrub ok 2026-02-23T14:21:50.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:21:49 trial068 ceph-mon[38805]: pgmap v2559: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:21:50.215 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:21:50.334 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:21:50.415 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4104 8 365281 365273 0.695291 0 - 0.0299513 2026-02-23T14:21:50.436 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:21:50.434+0000 7f37fd21d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:21:50.436 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:21:50.434+0000 7f37fd21d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:21:50.546 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:21:50.546 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":237,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:21:24.555495+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:20:32.175012+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:18:28.405361+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:20:02.384818+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:21:50.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:21:50 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3980705440' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:21:51.415 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4105 8 365281 365273 0.695122 0 - 0.0299513 2026-02-23T14:21:52.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:21:51 trial068 ceph-mon[38805]: pgmap v2560: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:21:52.415 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4106 8 365281 365273 0.694953 0 - 0.0299513 2026-02-23T14:21:53.415 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4107 8 365281 365273 0.694783 0 - 0.0299513 2026-02-23T14:21:53.583 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:21:53.700 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:21:53.805 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:21:53.802+0000 7f5800bf1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:21:53.805 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:21:53.803+0000 7f5800bf1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:21:53.912 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:21:53.913 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":237,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:21:24.555495+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:20:32.175012+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:18:28.405361+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:20:02.384818+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:21:53.913 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:21:53 trial068 ceph-mon[38805]: pgmap v2561: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:21:54.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:21:53 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:21:53] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:21:54.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:21:53 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:21:53.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:21:54.415 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4108 8 365281 365273 0.694614 0 - 0.0299513 2026-02-23T14:21:55.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:21:54 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1326833251' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:21:55.415 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4109 8 365281 365273 0.694445 0 - 0.0299513 2026-02-23T14:21:56.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:21:55 trial068 ceph-mon[38805]: pgmap v2562: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:21:56.415 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4110 8 365281 365273 0.694276 0 - 0.0299513 2026-02-23T14:21:56.949 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:21:56.974 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:21:56 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2026-02-23T14:21:57.068 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:21:57.176 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:21:57.173+0000 7f8e07db4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:21:57.176 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:21:57.174+0000 7f8e07db4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:21:57.286 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:21:57.287 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":237,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:21:24.555495+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:20:32.175012+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:18:28.405361+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:20:02.384818+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:21:57.415 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4111 8 365281 365273 0.694107 0 - 0.0299513 2026-02-23T14:21:58.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:21:57 trial068 ceph-mon[38805]: pgmap v2563: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:21:58.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:21:57 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-23T14:21:58.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:21:57 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2026-02-23T14:21:58.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:21:57 trial068 ceph-mon[38805]: pgmap v2564: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:21:58.095 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:21:57 trial068 ceph-mon[38805]: pgmap v2565: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:21:58.095 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:21:57 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T14:21:58.095 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:21:57 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2026-02-23T14:21:58.095 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:21:57 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3821111800' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:21:58.415 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4112 8 365281 365273 0.693939 0 - 0.0299513 2026-02-23T14:21:59.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:21:58 trial068 ceph-mon[38805]: pgmap v2566: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:21:59.415 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4113 8 365281 365273 0.69377 0 - 0.0299513 2026-02-23T14:22:00.325 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:22:00.415 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4114 8 365281 365273 0.693601 0 - 0.0299513 2026-02-23T14:22:00.444 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:22:00.541 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:22:00.539+0000 7f5caf22b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:22:00.542 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:22:00.540+0000 7f5caf22b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:22:00.652 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:22:00.653 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":237,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:21:24.555495+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:20:32.175012+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:18:28.405361+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:20:02.384818+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:22:01.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:22:00 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1375110979' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:22:01.415 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4115 8 365281 365273 0.693433 0 - 0.0299513 2026-02-23T14:22:02.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:22:01 trial068 ceph-mon[38805]: pgmap v2567: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:22:02.416 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4116 8 365281 365273 0.693264 0 - 0.0299513 2026-02-23T14:22:03.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:22:02 trial068 ceph-mon[38805]: pgmap v2568: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:22:03.416 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4117 8 365281 365273 0.693096 0 - 0.0299513 2026-02-23T14:22:03.690 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:22:03.811 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:22:03.920 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:22:03.918+0000 7f9b2c8bd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:22:03.920 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:22:03.918+0000 7f9b2c8bd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:22:04.029 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:22:04.029 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":237,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:21:24.555495+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:20:32.175012+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:18:28.405361+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:20:02.384818+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:22:04.266 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:22:03 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:22:03.972Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:22:04.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:22:04 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/803816335' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:22:04.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:22:03 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:22:03] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:22:04.416 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4118 8 365281 365273 0.692927 0 - 0.0299513 2026-02-23T14:22:05.416 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4119 8 365281 365273 0.692759 0 - 0.0299513 2026-02-23T14:22:05.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:22:05 trial068 ceph-mon[38805]: pgmap v2569: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:22:06.416 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T14:22:06.415444+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T14:22:06.416 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T14:22:06.416 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4120 8 365281 365273 0.692591 0 - 0.0299513 2026-02-23T14:22:07.066 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:22:07.185 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:22:07.208 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:22:06 trial068 ceph-mon[38805]: pgmap v2570: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:22:07.284 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:22:07.281+0000 7faac214f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:22:07.284 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:22:07.282+0000 7faac214f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:22:07.394 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:22:07.394 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":237,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:21:24.555495+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:20:32.175012+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:18:28.405361+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:20:02.384818+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:22:07.416 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4121 8 365281 365273 0.692423 0 - 0.0299513 2026-02-23T14:22:08.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:22:07 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/98683427' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:22:08.416 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4122 8 365281 365273 0.692255 0 - 0.0299513 2026-02-23T14:22:09.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:22:08 trial068 ceph-mon[38805]: pgmap v2571: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:22:09.416 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4123 8 365281 365273 0.692087 0 - 0.0299513 2026-02-23T14:22:10.416 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4124 8 365281 365273 0.691919 0 - 0.0299513 2026-02-23T14:22:10.431 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:22:10.550 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:22:10.648 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:22:10.646+0000 7f26955fc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:22:10.648 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:22:10.646+0000 7f26955fc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:22:10.758 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:22:10.758 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":237,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:21:24.555495+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:20:32.175012+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:18:28.405361+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:20:02.384818+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:22:11.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:22:10 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1732894411' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:22:11.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:22:10 trial068 ceph-mon[38805]: pgmap v2572: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:22:11.416 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4125 8 365281 365273 0.691752 0 - 0.0299513 2026-02-23T14:22:12.416 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4126 8 365281 365273 0.691584 0 - 0.0299513 2026-02-23T14:22:13.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:22:12 trial068 ceph-mon[38805]: pgmap v2573: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:22:13.416 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4127 8 365281 365273 0.691416 0 - 0.0299513 2026-02-23T14:22:13.798 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:22:13.917 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:22:14.013 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:22:14.011+0000 7f02b2b49640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:22:14.013 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:22:14.011+0000 7f02b2b49640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:22:14.094 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:22:13 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:22:13] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:22:14.123 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:22:14.123 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":237,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:21:24.555495+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:20:32.175012+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:18:28.405361+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:20:02.384818+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:22:14.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:22:13 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:22:13.972Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:22:14.416 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4128 8 365281 365273 0.691249 0 - 0.0299513 2026-02-23T14:22:14.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:22:14 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/4032041130' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:22:15.417 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4129 8 365281 365273 0.691081 0 - 0.0299513 2026-02-23T14:22:15.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:22:15 trial068 ceph-mon[38805]: pgmap v2574: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:22:16.417 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4130 8 365281 365273 0.690914 0 - 0.0299513 2026-02-23T14:22:17.159 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:22:17.185 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:22:16 trial068 ceph-mon[38805]: pgmap v2575: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:22:17.278 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:22:17.377 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:22:17.375+0000 7f74ef51a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:22:17.377 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:22:17.375+0000 7f74ef51a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:22:17.417 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4131 8 365281 365273 0.690747 0 - 0.0299513 2026-02-23T14:22:17.485 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:22:17.486 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":237,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:21:24.555495+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:20:32.175012+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:18:28.405361+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:20:02.384818+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:22:18.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:22:17 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/388952732' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:22:18.417 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4132 8 365281 365273 0.69058 0 - 0.0299513 2026-02-23T14:22:19.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:22:18 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T14:22:19.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:22:18 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T14:22:19.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:22:18 trial068 ceph-mon[38805]: pgmap v2576: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:22:19.417 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4133 8 365281 365273 0.690413 0 - 0.0299513 2026-02-23T14:22:20.417 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4134 8 365281 365273 0.690246 0 - 0.0299513 2026-02-23T14:22:20.524 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:22:20.642 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:22:20.740 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:22:20.737+0000 7f3e82a8c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:22:20.740 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:22:20.738+0000 7f3e82a8c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:22:20.850 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:22:20.850 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":237,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:21:24.555495+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:20:32.175012+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:18:28.405361+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:20:02.384818+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:22:21.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:22:20 trial068 ceph-mon[38805]: pgmap v2577: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:22:21.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:22:20 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1617962396' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:22:21.417 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4135 8 365281 365273 0.690079 0 - 0.0299513 2026-02-23T14:22:22.417 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4136 8 365281 365273 0.689912 0 - 0.0299513 2026-02-23T14:22:23.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:22:22 trial068 ceph-mon[38805]: pgmap v2578: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:22:23.417 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4137 8 365281 365273 0.689745 0 - 0.0299513 2026-02-23T14:22:23.887 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:22:24.005 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:22:24.102 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:22:24.100+0000 7fa67b2bb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:22:24.102 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:22:24.100+0000 7fa67b2bb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:22:24.212 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:22:24.213 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":237,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:21:24.555495+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:20:32.175012+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:18:28.405361+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:20:02.384818+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:22:24.213 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:22:23 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:22:23] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:22:24.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:22:23 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:22:23.972Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:22:24.417 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4138 8 365281 365273 0.689578 0 - 0.0299513 2026-02-23T14:22:24.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:22:24 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3265370758' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:22:25.417 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4139 8 365281 365273 0.689412 0 - 0.0299513 2026-02-23T14:22:25.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:22:25 trial068 ceph-mon[38805]: 5.a scrub starts 2026-02-23T14:22:25.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:22:25 trial068 ceph-mon[38805]: pgmap v2579: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:22:25.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:22:25 trial068 ceph-mon[38805]: 5.a scrub ok 2026-02-23T14:22:26.417 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T14:22:26.416839+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T14:22:26.418 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T14:22:26.418 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4140 8 365281 365273 0.689245 0 - 0.0299513 2026-02-23T14:22:27.268 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:22:27.293 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:22:26 trial068 ceph-mon[38805]: pgmap v2580: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:22:27.387 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:22:27.417 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4141 8 365281 365273 0.689079 0 - 0.0299513 2026-02-23T14:22:27.486 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:22:27.484+0000 7fe0ad6ca640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:22:27.486 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:22:27.484+0000 7fe0ad6ca640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:22:27.594 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:22:27.594 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":237,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:21:24.555495+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:20:32.175012+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:18:28.405361+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:20:02.384818+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:22:28.251 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:22:27 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3758759643' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:22:28.418 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4142 8 365281 365273 0.688912 0 - 0.0299513 2026-02-23T14:22:29.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:22:28 trial068 ceph-mon[38805]: pgmap v2581: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:22:29.418 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4143 8 365281 365273 0.688746 0 - 0.0299513 2026-02-23T14:22:30.418 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4144 8 365281 365273 0.68858 0 - 0.0299513 2026-02-23T14:22:30.631 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:22:30.752 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:22:30.855 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:22:30.853+0000 7f3d878ae640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:22:30.855 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:22:30.853+0000 7f3d878ae640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:22:30.966 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:22:30.966 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":237,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:21:24.555495+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:20:32.175012+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:18:28.405361+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:20:02.384818+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:22:30.967 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:22:30 trial068 ceph-mon[38805]: pgmap v2582: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:22:31.418 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4145 8 365281 365273 0.688414 0 - 0.0299513 2026-02-23T14:22:32.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:22:31 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1783178699' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:22:32.418 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4146 8 365281 365273 0.688248 0 - 0.0299513 2026-02-23T14:22:33.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:22:32 trial068 ceph-mon[38805]: pgmap v2583: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:22:33.418 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4147 8 365281 365273 0.688082 0 - 0.0299513 2026-02-23T14:22:34.004 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:22:34.126 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:22:34.224 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:22:34.222+0000 7f5f63019640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:22:34.224 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:22:34.222+0000 7f5f63019640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:22:34.240 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:22:33 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:22:33] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:22:34.337 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:22:34.337 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":237,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:21:24.555495+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:20:32.175012+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:18:28.405361+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:20:02.384818+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:22:34.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:22:33 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:22:33.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:22:34.418 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4148 8 365281 365273 0.687916 0 - 0.0299513 2026-02-23T14:22:34.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:22:34 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/188414306' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:22:35.418 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4149 8 365281 365273 0.68775 0 - 0.0299513 2026-02-23T14:22:35.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:22:35 trial068 ceph-mon[38805]: pgmap v2584: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:22:36.418 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4150 8 365281 365273 0.687584 0 - 0.0299513 2026-02-23T14:22:37.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:22:36 trial068 ceph-mon[38805]: pgmap v2585: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:22:37.373 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:22:37.418 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4151 8 365281 365273 0.687419 0 - 0.0299513 2026-02-23T14:22:37.492 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:22:37.595 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:22:37.593+0000 7efe97e2a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:22:37.596 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:22:37.593+0000 7efe97e2a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:22:37.706 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:22:37.706 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":237,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:21:24.555495+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:20:32.175012+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:18:28.405361+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:20:02.384818+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:22:38.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:22:37 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T14:22:38.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:22:37 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T14:22:38.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:22:37 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2287335346' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:22:38.418 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4152 8 365281 365273 0.687253 0 - 0.0299513 2026-02-23T14:22:39.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:22:38 trial068 ceph-mon[38805]: pgmap v2586: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:22:39.418 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4153 8 365281 365273 0.687088 0 - 0.0299513 2026-02-23T14:22:40.418 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4154 8 365281 365273 0.686922 0 - 0.0299513 2026-02-23T14:22:40.745 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:22:40.862 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:22:40.960 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:22:40.958+0000 7fc64086c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:22:40.960 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:22:40.958+0000 7fc64086c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:22:41.070 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:22:41.070 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":237,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:21:24.555495+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:20:32.175012+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:18:28.405361+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:20:02.384818+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:22:41.071 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:22:40 trial068 ceph-mon[38805]: pgmap v2587: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:22:41.418 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4155 8 365281 365273 0.686757 0 - 0.0299513 2026-02-23T14:22:42.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:22:41 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2681519507' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:22:42.418 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4156 8 365281 365273 0.686592 0 - 0.0299513 2026-02-23T14:22:43.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:22:42 trial068 ceph-mon[38805]: pgmap v2588: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:22:43.418 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4157 8 365281 365273 0.686427 0 - 0.0299513 2026-02-23T14:22:44.107 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:22:44.225 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:22:44.247 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:22:43 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:22:43] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:22:44.330 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:22:44.328+0000 7fb8c1bac640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:22:44.330 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:22:44.328+0000 7fb8c1bac640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:22:44.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:22:43 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:22:43.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:22:44.419 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4158 8 365281 365273 0.686262 0 - 0.0299513 2026-02-23T14:22:44.440 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:22:44.440 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":237,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:21:24.555495+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:20:32.175012+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:18:28.405361+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:20:02.384818+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:22:44.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:22:44 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3639004481' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:22:45.419 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4159 8 365281 365273 0.686097 0 - 0.0299513 2026-02-23T14:22:45.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:22:45 trial068 ceph-mon[38805]: pgmap v2589: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:22:46.419 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T14:22:46.418408+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T14:22:46.419 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T14:22:46.419 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4160 8 365281 365273 0.685932 0 - 0.0299513 2026-02-23T14:22:47.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:22:46 trial068 ceph-mon[38805]: pgmap v2590: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:22:47.419 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4161 8 365281 365273 0.685767 0 - 0.0299513 2026-02-23T14:22:47.478 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:22:47.595 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:22:47.695 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:22:47.692+0000 7f048f237640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:22:47.695 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:22:47.693+0000 7f048f237640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:22:47.805 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:22:47.805 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":237,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:21:24.555495+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:20:32.175012+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:18:28.405361+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:20:02.384818+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:22:48.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:22:47 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2844047459' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:22:48.419 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4162 8 365281 365273 0.685602 0 - 0.0299513 2026-02-23T14:22:49.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:22:48 trial068 ceph-mon[38805]: pgmap v2591: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:22:49.419 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4163 8 365281 365273 0.685437 0 - 0.0299513 2026-02-23T14:22:50.419 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4164 8 365281 365273 0.685273 0 - 0.0299513 2026-02-23T14:22:50.844 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:22:50.960 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:22:51.068 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:22:51.065+0000 7ff5af332640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:22:51.068 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:22:51.066+0000 7ff5af332640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:22:51.176 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:22:51.177 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":237,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:21:24.555495+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:20:32.175012+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:18:28.405361+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:20:02.384818+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:22:51.177 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:22:50 trial068 ceph-mon[38805]: pgmap v2592: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:22:51.419 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4165 8 365281 365273 0.685108 0 - 0.0299513 2026-02-23T14:22:52.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:22:51 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2032483318' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:22:52.419 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4166 8 365281 365273 0.684944 0 - 0.0299513 2026-02-23T14:22:53.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:22:52 trial068 ceph-mon[38805]: pgmap v2593: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:22:53.419 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4167 8 365281 365273 0.684779 0 - 0.0299513 2026-02-23T14:22:54.214 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:22:54.240 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:22:53 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:22:53] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:22:54.335 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:22:54.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:22:53 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:22:53.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:22:54.419 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4168 8 365281 365273 0.684615 0 - 0.0299513 2026-02-23T14:22:54.429 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:22:54.427+0000 7fd7cd596640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:22:54.429 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:22:54.427+0000 7fd7cd596640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:22:54.539 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:22:54.539 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":237,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:21:24.555495+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:20:32.175012+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:18:28.405361+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:20:02.384818+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:22:54.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:22:54 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2342300166' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:22:55.419 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4169 8 365281 365273 0.684451 0 - 0.0299513 2026-02-23T14:22:55.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:22:55 trial068 ceph-mon[38805]: pgmap v2594: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:22:56.419 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4170 8 365281 365273 0.684287 0 - 0.0299513 2026-02-23T14:22:57.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:22:56 trial068 ceph-mon[38805]: pgmap v2595: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:22:57.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:22:56 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2026-02-23T14:22:57.419 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4171 8 365281 365273 0.684123 0 - 0.0299513 2026-02-23T14:22:57.578 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:22:57.695 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:22:57.795 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:22:57.792+0000 7fb507523640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:22:57.795 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:22:57.793+0000 7fb507523640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:22:57.907 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:22:57.908 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":237,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:21:24.555495+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:20:32.175012+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:18:28.405361+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:20:02.384818+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:22:57.908 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:22:57 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-23T14:22:57.908 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:22:57 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2026-02-23T14:22:57.908 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:22:57 trial068 ceph-mon[38805]: pgmap v2596: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:22:57.908 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:22:57 trial068 ceph-mon[38805]: pgmap v2597: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:22:57.908 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:22:57 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T14:22:57.908 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:22:57 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2026-02-23T14:22:58.420 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4172 8 365281 365273 0.683959 0 - 0.0299513 2026-02-23T14:22:59.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:22:58 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2961215902' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:22:59.420 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4173 8 365281 365273 0.683795 0 - 0.0299513 2026-02-23T14:23:00.420 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4174 8 365281 365273 0.683631 0 - 0.0299513 2026-02-23T14:23:00.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:23:00 trial068 ceph-mon[38805]: pgmap v2598: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:23:00.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:23:00 trial068 ceph-mon[38805]: 5.e scrub starts 2026-02-23T14:23:00.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:23:00 trial068 ceph-mon[38805]: 5.e scrub ok 2026-02-23T14:23:00.947 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:23:01.067 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:23:01.167 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:23:01.165+0000 7f6f8eb26640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:23:01.168 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:23:01.165+0000 7f6f8eb26640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:23:01.280 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:23:01.280 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":237,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:21:24.555495+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:20:32.175012+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:18:28.405361+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:20:02.384818+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:23:01.420 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4175 8 365281 365273 0.683467 0 - 0.0299513 2026-02-23T14:23:01.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:23:01 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3125773563' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:23:02.420 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4176 8 365281 365273 0.683304 0 - 0.0299513 2026-02-23T14:23:02.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:23:02 trial068 ceph-mon[38805]: pgmap v2599: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:23:02.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:23:02 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T14:23:02.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:23:02 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T14:23:03.420 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4177 8 365281 365273 0.68314 0 - 0.0299513 2026-02-23T14:23:04.265 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:23:03 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:23:03.974Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:23:04.320 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:23:04.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:23:03 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:23:03] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:23:04.420 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4178 8 365281 365273 0.682976 0 - 0.0299513 2026-02-23T14:23:04.437 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:23:04.538 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:23:04.536+0000 7f8a29203640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:23:04.539 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:23:04.536+0000 7f8a29203640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:23:04.646 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:23:04.646 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":237,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:21:24.555495+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:20:32.175012+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:18:28.405361+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:20:02.384818+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:23:04.647 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:23:04 trial068 ceph-mon[38805]: pgmap v2600: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:23:05.420 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4179 8 365281 365273 0.682813 0 - 0.0299513 2026-02-23T14:23:05.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:23:05 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/891904488' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:23:06.420 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T14:23:06.419807+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T14:23:06.420 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T14:23:06.421 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4180 8 365281 365273 0.68265 0 - 0.0299513 2026-02-23T14:23:06.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:23:06 trial068 ceph-mon[38805]: pgmap v2601: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:23:07.420 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4181 8 365281 365273 0.682486 0 - 0.0299513 2026-02-23T14:23:07.686 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:23:07.806 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:23:07.902 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:23:07.900+0000 7f3deb9cc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:23:07.902 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:23:07.900+0000 7f3deb9cc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:23:08.013 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:23:08.013 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":237,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:21:24.555495+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:20:32.175012+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:18:28.405361+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:20:02.384818+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:23:08.420 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4182 8 365281 365273 0.682323 0 - 0.0299513 2026-02-23T14:23:08.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:23:08 trial068 ceph-mon[38805]: pgmap v2602: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:23:08.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:23:08 trial068 ceph-mon[38805]: 5.5 scrub starts 2026-02-23T14:23:08.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:23:08 trial068 ceph-mon[38805]: 5.5 scrub ok 2026-02-23T14:23:08.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:23:08 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1447516967' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:23:09.420 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4183 8 365281 365273 0.68216 0 - 0.0299513 2026-02-23T14:23:10.420 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4184 8 365281 365273 0.681997 0 - 0.0299513 2026-02-23T14:23:10.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:23:10 trial068 ceph-mon[38805]: pgmap v2603: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:23:11.051 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:23:11.168 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:23:11.268 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:23:11.266+0000 7f3dda07f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:23:11.268 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:23:11.266+0000 7f3dda07f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:23:11.378 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:23:11.378 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":237,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:21:24.555495+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:20:32.175012+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:18:28.405361+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:20:02.384818+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:23:11.420 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4185 8 365281 365273 0.681834 0 - 0.0299513 2026-02-23T14:23:11.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:23:11 trial068 ceph-mon[38805]: 5.7 scrub starts 2026-02-23T14:23:11.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:23:11 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1467633769' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:23:12.421 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4186 8 365281 365273 0.681671 0 - 0.0299513 2026-02-23T14:23:12.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:23:12 trial068 ceph-mon[38805]: pgmap v2604: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:23:12.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:23:12 trial068 ceph-mon[38805]: 5.7 scrub ok 2026-02-23T14:23:13.421 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4187 8 365281 365273 0.681508 0 - 0.0299513 2026-02-23T14:23:14.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:23:13 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:23:13] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:23:14.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:23:13 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:23:13.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:23:14.415 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:23:14.421 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4188 8 365281 365273 0.681346 0 - 0.0299513 2026-02-23T14:23:14.534 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:23:14.638 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:23:14.635+0000 7feac5193640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:23:14.638 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:23:14.636+0000 7feac5193640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:23:14.747 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:23:14.747 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":237,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:21:24.555495+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:20:32.175012+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:18:28.405361+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:20:02.384818+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:23:14.748 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:23:14 trial068 ceph-mon[38805]: pgmap v2605: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:23:15.421 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4189 8 365281 365273 0.681183 0 - 0.0299513 2026-02-23T14:23:15.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:23:15 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1813691937' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:23:16.421 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4190 8 365281 365273 0.68102 0 - 0.0299513 2026-02-23T14:23:16.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:23:16 trial068 ceph-mon[38805]: pgmap v2606: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:23:17.421 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4191 8 365281 365273 0.680858 0 - 0.0299513 2026-02-23T14:23:17.801 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:23:17.921 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:23:18.031 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:23:18.028+0000 7f52541cd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:23:18.031 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:23:18.029+0000 7f52541cd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:23:18.144 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:23:18.144 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":237,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:21:24.555495+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:20:32.175012+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:18:28.405361+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:20:02.384818+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:23:18.421 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4192 8 365281 365273 0.680696 0 - 0.0299513 2026-02-23T14:23:18.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:23:18 trial068 ceph-mon[38805]: pgmap v2607: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:23:18.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:23:18 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3602453151' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:23:19.421 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4193 8 365281 365273 0.680533 0 - 0.0299513 2026-02-23T14:23:20.421 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4194 8 365281 365273 0.680371 0 - 0.0299513 2026-02-23T14:23:20.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:23:20 trial068 ceph-mon[38805]: pgmap v2608: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:23:21.181 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:23:21.300 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:23:21.400 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:23:21.398+0000 7f1714488640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:23:21.401 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:23:21.399+0000 7f1714488640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:23:21.421 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4195 8 365281 365273 0.680209 0 - 0.0299513 2026-02-23T14:23:21.513 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:23:21.513 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":237,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:21:24.555495+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:20:32.175012+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:18:28.405361+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:20:02.384818+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:23:21.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:23:21 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2395059071' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:23:22.421 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4196 8 365281 365273 0.680047 0 - 0.0299513 2026-02-23T14:23:23.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:23:22 trial068 ceph-mon[38805]: pgmap v2609: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:23:23.421 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4197 8 365281 365273 0.679885 0 - 0.0299513 2026-02-23T14:23:24.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:23:23 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:23:23] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:23:24.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:23:23 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:23:23.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:23:24.421 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4198 8 365281 365273 0.679723 0 - 0.0299513 2026-02-23T14:23:24.551 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:23:24.668 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:23:24.770 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:23:24.768+0000 7f08cbb71640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:23:24.770 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:23:24.768+0000 7f08cbb71640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:23:24.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:23:24 trial068 ceph-mon[38805]: pgmap v2610: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:23:24.882 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:23:24.882 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":237,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:21:24.555495+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:20:32.175012+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:18:28.405361+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:20:02.384818+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:23:25.422 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4199 8 365281 365273 0.679561 0 - 0.0299513 2026-02-23T14:23:26.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:23:25 trial068 ceph-mon[38805]: 5.8 scrub starts 2026-02-23T14:23:26.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:23:25 trial068 ceph-mon[38805]: 5.8 scrub ok 2026-02-23T14:23:26.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:23:25 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/4017628037' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:23:26.422 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T14:23:26.421243+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T14:23:26.422 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T14:23:26.422 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4200 8 365281 365273 0.679399 0 - 0.0299513 2026-02-23T14:23:27.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:23:26 trial068 ceph-mon[38805]: pgmap v2611: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:23:27.422 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4201 8 365281 365273 0.679237 0 - 0.0299513 2026-02-23T14:23:27.920 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:23:28.040 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:23:28.146 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:23:28.144+0000 7f380e83a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:23:28.146 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:23:28.144+0000 7f380e83a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:23:28.256 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:23:28.256 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":237,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:21:24.555495+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:20:32.175012+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:18:28.405361+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:20:02.384818+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:23:28.422 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4202 8 365281 365273 0.679076 0 - 0.0299513 2026-02-23T14:23:29.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:23:28 trial068 ceph-mon[38805]: pgmap v2612: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:23:29.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:23:28 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2959804007' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:23:29.422 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4203 8 365281 365273 0.678914 0 - 0.0299513 2026-02-23T14:23:30.422 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4204 8 365281 365273 0.678753 0 - 0.0299513 2026-02-23T14:23:31.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:23:30 trial068 ceph-mon[38805]: pgmap v2613: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:23:31.293 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:23:31.409 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:23:31.422 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4205 8 365281 365273 0.678591 0 - 0.0299513 2026-02-23T14:23:31.511 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:23:31.510+0000 7f3a78ac5640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:23:31.511 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:23:31.510+0000 7f3a78ac5640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:23:31.620 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:23:31.620 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":237,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:21:24.555495+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:20:32.175012+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:18:28.405361+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:20:02.384818+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:23:32.422 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4206 8 365281 365273 0.67843 0 - 0.0299513 2026-02-23T14:23:33.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:23:32 trial068 ceph-mon[38805]: pgmap v2614: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:23:33.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:23:32 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3094546764' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:23:33.422 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4207 8 365281 365273 0.678269 0 - 0.0299513 2026-02-23T14:23:34.266 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:23:33 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:23:33.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:23:34.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:23:33 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:23:33] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:23:34.422 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4208 8 365281 365273 0.678107 0 - 0.0299513 2026-02-23T14:23:34.661 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:23:34.777 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:23:34.881 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:23:34.879+0000 7f62e638f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:23:34.881 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:23:34 trial068 ceph-mon[38805]: pgmap v2615: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:23:34.882 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:23:34.879+0000 7f62e638f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:23:34.989 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:23:34.990 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":237,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:21:24.555495+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:20:32.175012+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:18:28.405361+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:20:02.384818+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:23:35.422 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4209 8 365281 365273 0.677946 0 - 0.0299513 2026-02-23T14:23:36.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:23:35 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3074179985' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:23:36.422 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4210 8 365281 365273 0.677785 0 - 0.0299513 2026-02-23T14:23:37.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:23:36 trial068 ceph-mon[38805]: pgmap v2616: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:23:37.422 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4211 8 365281 365273 0.677624 0 - 0.0299513 2026-02-23T14:23:38.026 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:23:38.051 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:23:37 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T14:23:38.051 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:23:37 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T14:23:38.147 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:23:38.247 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:23:38.245+0000 7f5430cca640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:23:38.247 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:23:38.245+0000 7f5430cca640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:23:38.357 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:23:38.357 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":237,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:21:24.555495+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:20:32.175012+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:18:28.405361+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:20:02.384818+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:23:38.422 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4212 8 365281 365273 0.677463 0 - 0.0299513 2026-02-23T14:23:39.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:23:38 trial068 ceph-mon[38805]: pgmap v2617: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:23:39.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:23:38 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/295147643' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:23:39.422 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4213 8 365281 365273 0.677303 0 - 0.0299513 2026-02-23T14:23:40.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:23:39 trial068 ceph-mon[38805]: 5.a scrub starts 2026-02-23T14:23:40.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:23:39 trial068 ceph-mon[38805]: 5.a scrub ok 2026-02-23T14:23:40.423 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4214 8 365281 365273 0.677142 0 - 0.0299513 2026-02-23T14:23:41.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:23:40 trial068 ceph-mon[38805]: pgmap v2618: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:23:41.395 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:23:41.423 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4215 8 365281 365273 0.676981 0 - 0.0299513 2026-02-23T14:23:41.511 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:23:41.617 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:23:41.615+0000 7f0da036e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:23:41.617 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:23:41.615+0000 7f0da036e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:23:41.730 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:23:41.731 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":237,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:21:24.555495+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:20:32.175012+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:18:28.405361+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:20:02.384818+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:23:42.423 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4216 8 365281 365273 0.676821 0 - 0.0299513 2026-02-23T14:23:43.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:23:42 trial068 ceph-mon[38805]: pgmap v2619: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:23:43.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:23:42 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2950581847' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:23:43.423 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4217 8 365281 365273 0.67666 0 - 0.0299513 2026-02-23T14:23:44.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:23:43 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:23:43] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:23:44.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:23:43 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:23:43.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:23:44.423 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4218 8 365281 365273 0.6765 0 - 0.0299513 2026-02-23T14:23:44.767 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:23:44.884 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:23:44.907 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:23:44 trial068 ceph-mon[38805]: pgmap v2620: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:23:44.985 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:23:44.983+0000 7f7d8bedd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:23:44.986 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:23:44.983+0000 7f7d8bedd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:23:45.094 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:23:45.094 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":237,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:21:24.555495+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:20:32.175012+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:18:28.405361+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:20:02.384818+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:23:45.423 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4219 8 365281 365273 0.676339 0 - 0.0299513 2026-02-23T14:23:46.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:23:45 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2504432443' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:23:46.423 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T14:23:46.422719+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T14:23:46.423 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T14:23:46.423 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4220 8 365281 365273 0.676179 0 - 0.0299513 2026-02-23T14:23:47.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:23:46 trial068 ceph-mon[38805]: pgmap v2621: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:23:47.423 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4221 8 365281 365273 0.676019 0 - 0.0299513 2026-02-23T14:23:48.131 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:23:48.249 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:23:48.348 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:23:48.346+0000 7fd2dff52640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:23:48.348 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:23:48.346+0000 7fd2dff52640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:23:48.423 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4222 8 365281 365273 0.675859 0 - 0.0299513 2026-02-23T14:23:48.457 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:23:48.457 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":237,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:21:24.555495+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:20:32.175012+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:18:28.405361+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:20:02.384818+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:23:48.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:23:48 trial068 ceph-mon[38805]: pgmap v2622: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:23:48.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:23:48 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/554611977' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:23:49.423 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4223 8 365281 365273 0.675699 0 - 0.0299513 2026-02-23T14:23:50.423 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4224 8 365281 365273 0.675539 0 - 0.0299513 2026-02-23T14:23:51.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:23:50 trial068 ceph-mon[38805]: pgmap v2623: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:23:51.423 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4225 8 365281 365273 0.675379 0 - 0.0299513 2026-02-23T14:23:51.494 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:23:51.612 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:23:51.708 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:23:51.706+0000 7f9ade721640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:23:51.709 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:23:51.706+0000 7f9ade721640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:23:51.819 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:23:51.819 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":237,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:21:24.555495+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:20:32.175012+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:18:28.405361+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:20:02.384818+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:23:52.424 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4226 8 365281 365273 0.675219 0 - 0.0299513 2026-02-23T14:23:53.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:23:52 trial068 ceph-mon[38805]: pgmap v2624: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:23:53.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:23:52 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1275187419' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:23:53.423 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4227 8 365281 365273 0.675059 0 - 0.0299513 2026-02-23T14:23:54.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:23:53 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:23:53] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:23:54.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:23:53 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:23:53.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:23:54.423 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4228 8 365281 365273 0.6749 0 - 0.0299513 2026-02-23T14:23:54.856 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:23:54.878 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:23:54 trial068 ceph-mon[38805]: pgmap v2625: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:23:54.970 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:23:55.067 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:23:55.065+0000 7f6db3c0c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:23:55.067 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:23:55.065+0000 7f6db3c0c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:23:55.178 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:23:55.178 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":237,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:21:24.555495+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:20:32.175012+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:18:28.405361+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:20:02.384818+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:23:55.424 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4229 8 365281 365273 0.67474 0 - 0.0299513 2026-02-23T14:23:56.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:23:55 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2904921163' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:23:56.424 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4230 8 365281 365273 0.674581 0 - 0.0299513 2026-02-23T14:23:57.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:23:56 trial068 ceph-mon[38805]: pgmap v2626: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:23:57.424 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4231 8 365281 365273 0.674421 0 - 0.0299513 2026-02-23T14:23:58.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:23:57 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2026-02-23T14:23:58.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:23:57 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-23T14:23:58.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:23:57 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2026-02-23T14:23:58.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:23:57 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T14:23:58.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:23:57 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2026-02-23T14:23:58.214 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:23:58.330 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:23:58.424 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4232 8 365281 365273 0.674262 0 - 0.0299513 2026-02-23T14:23:58.430 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:23:58.428+0000 7f81a7131640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:23:58.430 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:23:58.428+0000 7f81a7131640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:23:58.540 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:23:58.541 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":237,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:21:24.555495+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:20:32.175012+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:18:28.405361+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:20:02.384818+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:23:58.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:23:58 trial068 ceph-mon[38805]: pgmap v2627: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:23:58.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:23:58 trial068 ceph-mon[38805]: pgmap v2628: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:23:58.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:23:58 trial068 ceph-mon[38805]: pgmap v2629: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:23:58.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:23:58 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T14:23:58.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:23:58 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T14:23:58.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:23:58 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2510302462' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:23:59.424 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4233 8 365281 365273 0.674102 0 - 0.0299513 2026-02-23T14:24:00.424 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4234 8 365281 365273 0.673943 0 - 0.0299513 2026-02-23T14:24:01.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:24:00 trial068 ceph-mon[38805]: pgmap v2630: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:24:01.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:24:00 trial068 ceph-mon[38805]: osdmap e238: 4 total, 3 up, 4 in 2026-02-23T14:24:01.424 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4235 8 365281 365273 0.673784 0 - 0.0299513 2026-02-23T14:24:01.580 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:24:01.699 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:24:01.800 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:24:01.798+0000 7f166f7f8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:24:01.801 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:24:01.798+0000 7f166f7f8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:24:01.911 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:24:01.911 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":238,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:23:59.627735+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:20:32.175012+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:23:36.246673+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:20:02.384818+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:24:02.424 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4236 8 365281 365273 0.673625 0 - 0.0299513 2026-02-23T14:24:03.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:24:02 trial068 ceph-mon[38805]: pgmap v2632: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:24:03.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:24:02 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/970747954' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:24:03.424 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4237 8 365281 365273 0.673466 0 - 0.0299513 2026-02-23T14:24:04.266 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:24:03 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:24:03.974Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:24:04.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:24:03 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:24:03] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:24:04.424 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4238 8 365281 365273 0.673307 0 - 0.0299513 2026-02-23T14:24:04.948 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:24:04.974 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:24:04 trial068 ceph-mon[38805]: pgmap v2633: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:24:04.974 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:24:04 trial068 ceph-mon[38805]: osdmap e239: 4 total, 3 up, 4 in 2026-02-23T14:24:05.066 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:24:05.165 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:24:05.163+0000 7f1c37706640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:24:05.165 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:24:05.163+0000 7f1c37706640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:24:05.274 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:24:05.274 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":239,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:24:04.576030+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:20:32.175012+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:23:36.246673+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:20:02.384818+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:24:05.424 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4239 8 365281 365273 0.673148 0 - 0.0299513 2026-02-23T14:24:06.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:24:05 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/674800788' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:24:06.424 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T14:24:06.423948+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T14:24:06.425 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T14:24:06.425 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4240 8 365281 365273 0.67299 0 - 0.0299513 2026-02-23T14:24:07.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:24:06 trial068 ceph-mon[38805]: pgmap v2635: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:24:07.424 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4241 8 365281 365273 0.672831 0 - 0.0299513 2026-02-23T14:24:08.311 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:24:08.424 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4242 8 365281 365273 0.672672 0 - 0.0299513 2026-02-23T14:24:08.432 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:24:08.534 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:24:08.532+0000 7fd744a5b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:24:08.534 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:24:08.532+0000 7fd744a5b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:24:08.644 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:24:08.644 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":239,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:24:04.576030+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:20:32.175012+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:23:36.246673+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:20:02.384818+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:24:09.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:24:08 trial068 ceph-mon[38805]: pgmap v2636: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:24:09.425 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4243 8 365281 365273 0.672514 0 - 0.0299513 2026-02-23T14:24:10.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:24:09 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1389716801' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:24:10.425 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4244 8 365281 365273 0.672355 0 - 0.0299513 2026-02-23T14:24:11.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:24:10 trial068 ceph-mon[38805]: pgmap v2637: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:24:11.425 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4245 8 365281 365273 0.672197 0 - 0.0299513 2026-02-23T14:24:11.681 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:24:11.800 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:24:11.895 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:24:11.892+0000 7f59addb6640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:24:11.895 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:24:11.892+0000 7f59addb6640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:24:12.006 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:24:12.006 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":239,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:24:04.576030+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:20:32.175012+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:23:36.246673+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:20:02.384818+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:24:12.425 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4246 8 365281 365273 0.672039 0 - 0.0299513 2026-02-23T14:24:13.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:24:12 trial068 ceph-mon[38805]: pgmap v2638: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:24:13.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:24:12 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2960785610' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:24:13.425 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4247 8 365281 365273 0.67188 0 - 0.0299513 2026-02-23T14:24:14.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:24:13 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:24:13] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:24:14.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:24:13 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:24:13.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:24:14.425 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4248 8 365281 365273 0.671722 0 - 0.0299513 2026-02-23T14:24:15.043 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:24:15.067 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:24:14 trial068 ceph-mon[38805]: pgmap v2639: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:24:15.163 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:24:15.267 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:24:15.265+0000 7f9ae134f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:24:15.267 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:24:15.265+0000 7f9ae134f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:24:15.379 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:24:15.380 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":239,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:24:04.576030+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:20:32.175012+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:23:36.246673+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:20:02.384818+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:24:15.425 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4249 8 365281 365273 0.671564 0 - 0.0299513 2026-02-23T14:24:16.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:24:15 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/115306223' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:24:16.425 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4250 8 365281 365273 0.671406 0 - 0.0299513 2026-02-23T14:24:17.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:24:16 trial068 ceph-mon[38805]: pgmap v2640: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:24:17.425 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4251 8 365281 365273 0.671248 0 - 0.0299513 2026-02-23T14:24:18.418 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:24:18.425 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4252 8 365281 365273 0.67109 0 - 0.0299513 2026-02-23T14:24:18.537 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:24:18.637 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:24:18.635+0000 7f29252d2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:24:18.638 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:24:18.635+0000 7f29252d2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:24:18.746 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:24:18.747 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":239,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:24:04.576030+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:20:32.175012+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:23:36.246673+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:20:02.384818+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:24:18.747 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:24:18 trial068 ceph-mon[38805]: pgmap v2641: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:24:18.747 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:24:18 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T14:24:18.747 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:24:18 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T14:24:19.425 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4253 8 365281 365273 0.670933 0 - 0.0299513 2026-02-23T14:24:20.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:24:19 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/709662664' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:24:20.425 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4254 8 365281 365273 0.670775 0 - 0.0299513 2026-02-23T14:24:21.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:24:20 trial068 ceph-mon[38805]: 5.5 scrub starts 2026-02-23T14:24:21.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:24:20 trial068 ceph-mon[38805]: 5.5 scrub ok 2026-02-23T14:24:21.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:24:20 trial068 ceph-mon[38805]: pgmap v2642: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:24:21.425 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4255 8 365281 365273 0.670617 0 - 0.0299513 2026-02-23T14:24:21.785 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:24:21.902 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:24:22.003 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:24:22.001+0000 7f6701e35640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:24:22.003 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:24:22.001+0000 7f6701e35640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:24:22.114 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:24:22.114 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":239,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:24:04.576030+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:20:32.175012+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:23:36.246673+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:20:02.384818+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:24:22.426 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4256 8 365281 365273 0.67046 0 - 0.0299513 2026-02-23T14:24:23.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:24:22 trial068 ceph-mon[38805]: pgmap v2643: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:24:23.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:24:22 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1284909633' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:24:23.425 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4257 8 365281 365273 0.670302 0 - 0.0299513 2026-02-23T14:24:24.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:24:23 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:24:23] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:24:24.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:24:23 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:24:23.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:24:24.426 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4258 8 365281 365273 0.670145 0 - 0.0299513 2026-02-23T14:24:25.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:24:24 trial068 ceph-mon[38805]: pgmap v2644: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:24:25.153 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:24:25.271 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:24:25.375 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:24:25.372+0000 7f63d0c13640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:24:25.375 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:24:25.373+0000 7f63d0c13640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:24:25.426 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4259 8 365281 365273 0.669987 0 - 0.0299513 2026-02-23T14:24:25.486 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:24:25.486 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":239,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:24:04.576030+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:20:32.175012+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:23:36.246673+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:20:02.384818+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:24:25.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:24:25 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2985947739' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:24:26.426 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T14:24:26.425358+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T14:24:26.426 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T14:24:26.426 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4260 8 365281 365273 0.66983 0 - 0.0299513 2026-02-23T14:24:27.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:24:26 trial068 ceph-mon[38805]: pgmap v2645: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:24:27.426 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4261 8 365281 365273 0.669673 0 - 0.0299513 2026-02-23T14:24:28.426 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4262 8 365281 365273 0.669516 0 - 0.0299513 2026-02-23T14:24:28.526 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:24:28.645 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:24:28.745 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:24:28.743+0000 7f84abb5f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:24:28.745 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:24:28.743+0000 7f84abb5f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:24:28.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:24:28 trial068 ceph-mon[38805]: pgmap v2646: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:24:28.856 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:24:28.856 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":239,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:24:04.576030+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:20:32.175012+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:23:36.246673+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:20:02.384818+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:24:29.426 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4263 8 365281 365273 0.669359 0 - 0.0299513 2026-02-23T14:24:30.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:24:29 trial068 ceph-mon[38805]: 5.e scrub starts 2026-02-23T14:24:30.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:24:29 trial068 ceph-mon[38805]: 5.e scrub ok 2026-02-23T14:24:30.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:24:29 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2178622799' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:24:30.426 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4264 8 365281 365273 0.669202 0 - 0.0299513 2026-02-23T14:24:31.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:24:30 trial068 ceph-mon[38805]: 5.8 scrub starts 2026-02-23T14:24:31.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:24:30 trial068 ceph-mon[38805]: 5.8 scrub ok 2026-02-23T14:24:31.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:24:30 trial068 ceph-mon[38805]: pgmap v2647: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:24:31.426 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4265 8 365281 365273 0.669045 0 - 0.0299513 2026-02-23T14:24:31.898 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:24:32.018 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:24:32.120 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:24:32.117+0000 7f0eb7c63640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:24:32.120 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:24:32.118+0000 7f0eb7c63640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:24:32.230 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:24:32.230 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":239,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:24:04.576030+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:20:32.175012+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:23:36.246673+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:20:02.384818+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:24:32.426 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4266 8 365281 365273 0.668888 0 - 0.0299513 2026-02-23T14:24:33.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:24:32 trial068 ceph-mon[38805]: pgmap v2648: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:24:33.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:24:32 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1663934327' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:24:33.426 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4267 8 365281 365273 0.668731 0 - 0.0299513 2026-02-23T14:24:34.266 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:24:33 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:24:33.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:24:34.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:24:33 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:24:33] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:24:34.426 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4268 8 365281 365273 0.668575 0 - 0.0299513 2026-02-23T14:24:35.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:24:34 trial068 ceph-mon[38805]: pgmap v2649: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:24:35.270 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:24:35.389 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:24:35.426 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4269 8 365281 365273 0.668418 0 - 0.0299513 2026-02-23T14:24:35.492 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:24:35.489+0000 7f18360d6640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:24:35.492 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:24:35.490+0000 7f18360d6640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:24:35.601 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:24:35.601 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":239,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:24:04.576030+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:20:32.175012+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:23:36.246673+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:20:02.384818+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:24:36.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:24:35 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3984622522' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:24:36.426 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4270 8 365281 365273 0.668261 0 - 0.0299513 2026-02-23T14:24:37.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:24:36 trial068 ceph-mon[38805]: pgmap v2650: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:24:37.427 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4271 8 365281 365273 0.668105 0 - 0.0299513 2026-02-23T14:24:38.427 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4272 8 365281 365273 0.667949 0 - 0.0299513 2026-02-23T14:24:38.640 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:24:38.760 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:24:38.863 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:24:38.860+0000 7f17e1a50640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:24:38.863 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:24:38.861+0000 7f17e1a50640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:24:38.971 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:24:38.972 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":239,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:24:04.576030+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:20:32.175012+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:23:36.246673+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:20:02.384818+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:24:38.972 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:24:38 trial068 ceph-mon[38805]: pgmap v2651: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:24:39.427 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4273 8 365281 365273 0.667792 0 - 0.0299513 2026-02-23T14:24:40.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:24:39 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2470765301' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:24:40.427 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4274 8 365281 365273 0.667636 0 - 0.0299513 2026-02-23T14:24:41.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:24:40 trial068 ceph-mon[38805]: pgmap v2652: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:24:41.427 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4275 8 365281 365273 0.66748 0 - 0.0299513 2026-02-23T14:24:42.008 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:24:42.125 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:24:42.225 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:24:42.223+0000 7f21be48c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:24:42.226 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:24:42.223+0000 7f21be48c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:24:42.334 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:24:42.334 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":239,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:24:04.576030+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:20:32.175012+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:23:36.246673+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:20:02.384818+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:24:42.427 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4276 8 365281 365273 0.667324 0 - 0.0299513 2026-02-23T14:24:43.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:24:42 trial068 ceph-mon[38805]: 5.7 scrub starts 2026-02-23T14:24:43.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:24:42 trial068 ceph-mon[38805]: pgmap v2653: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:24:43.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:24:42 trial068 ceph-mon[38805]: 5.7 scrub ok 2026-02-23T14:24:43.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:24:42 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/755206126' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:24:43.427 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4277 8 365281 365273 0.667168 0 - 0.0299513 2026-02-23T14:24:44.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:24:43 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:24:43] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:24:44.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:24:43 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:24:43.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:24:44.427 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4278 8 365281 365273 0.667012 0 - 0.0299513 2026-02-23T14:24:45.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:24:44 trial068 ceph-mon[38805]: pgmap v2654: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:24:45.372 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:24:45.427 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4279 8 365281 365273 0.666856 0 - 0.0299513 2026-02-23T14:24:45.490 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:24:45.589 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:24:45.586+0000 7fa89f0d9640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:24:45.589 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:24:45.587+0000 7fa89f0d9640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:24:45.697 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:24:45.697 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":239,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:24:04.576030+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:20:32.175012+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:23:36.246673+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:20:02.384818+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:24:46.427 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T14:24:46.426759+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T14:24:46.427 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T14:24:46.428 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4280 8 365281 365273 0.6667 0 - 0.0299513 2026-02-23T14:24:47.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:24:46 trial068 ceph-mon[38805]: pgmap v2655: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:24:47.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:24:46 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1072759801' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:24:47.427 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4281 8 365281 365273 0.666544 0 - 0.0299513 2026-02-23T14:24:48.427 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4282 8 365281 365273 0.666389 0 - 0.0299513 2026-02-23T14:24:48.734 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:24:48.852 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:24:48.954 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:24:48.952+0000 7f8438a9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:24:48.954 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:24:48 trial068 ceph-mon[38805]: pgmap v2656: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:24:48.955 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:24:48.952+0000 7f8438a9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:24:49.063 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:24:49.064 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":239,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:24:04.576030+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:20:32.175012+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:23:36.246673+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:20:02.384818+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:24:49.427 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4283 8 365281 365273 0.666233 0 - 0.0299513 2026-02-23T14:24:50.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:24:49 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3457175347' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:24:50.428 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4284 8 365281 365273 0.666078 0 - 0.0299513 2026-02-23T14:24:51.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:24:50 trial068 ceph-mon[38805]: pgmap v2657: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:24:51.428 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4285 8 365281 365273 0.665922 0 - 0.0299513 2026-02-23T14:24:52.101 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:24:52.220 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:24:52.323 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:24:52.321+0000 7fbd7d311640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:24:52.323 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:24:52.321+0000 7fbd7d311640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:24:52.428 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4286 8 365281 365273 0.665767 0 - 0.0299513 2026-02-23T14:24:52.432 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:24:52.432 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":239,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:24:04.576030+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:20:32.175012+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:23:36.246673+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:20:02.384818+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:24:52.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:24:52 trial068 ceph-mon[38805]: pgmap v2658: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:24:52.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:24:52 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2054681538' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:24:53.428 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4287 8 365281 365273 0.665611 0 - 0.0299513 2026-02-23T14:24:54.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:24:53 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:24:53] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:24:54.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:24:53 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:24:53.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:24:54.428 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4288 8 365281 365273 0.665456 0 - 0.0299513 2026-02-23T14:24:55.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:24:54 trial068 ceph-mon[38805]: pgmap v2659: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:24:55.428 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4289 8 365281 365273 0.665301 0 - 0.0299513 2026-02-23T14:24:55.471 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:24:55.587 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:24:55.689 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:24:55.687+0000 7fde0b45b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:24:55.689 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:24:55.687+0000 7fde0b45b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:24:55.799 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:24:55.799 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":239,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:24:04.576030+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:20:32.175012+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:23:36.246673+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:20:02.384818+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:24:56.428 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4290 8 365281 365273 0.665146 0 - 0.0299513 2026-02-23T14:24:57.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:24:56 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T14:24:57.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:24:56 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T14:24:57.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:24:56 trial068 ceph-mon[38805]: 5.a scrub starts 2026-02-23T14:24:57.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:24:56 trial068 ceph-mon[38805]: pgmap v2660: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:24:57.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:24:56 trial068 ceph-mon[38805]: 5.a scrub ok 2026-02-23T14:24:57.095 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:24:56 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/980795312' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:24:57.428 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4291 8 365281 365273 0.664991 0 - 0.0299513 2026-02-23T14:24:58.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:24:57 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2026-02-23T14:24:58.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:24:57 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-23T14:24:58.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:24:57 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2026-02-23T14:24:58.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:24:57 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T14:24:58.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:24:57 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2026-02-23T14:24:58.428 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4292 8 365281 365273 0.664836 0 - 0.0299513 2026-02-23T14:24:58.836 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:24:58.954 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:24:58.977 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:24:58 trial068 ceph-mon[38805]: pgmap v2661: 17 pgs: 1 active+clean+scrubbing, 1 active+recovery_unfound+undersized+degraded, 4 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:24:58.977 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:24:58 trial068 ceph-mon[38805]: pgmap v2662: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 1 active+clean+scrubbing, 4 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:24:58.977 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:24:58 trial068 ceph-mon[38805]: pgmap v2663: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 1 active+clean+scrubbing, 4 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:24:59.057 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:24:59.055+0000 7fcf8bd0b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:24:59.057 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:24:59.055+0000 7fcf8bd0b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:24:59.168 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:24:59.169 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":239,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:24:04.576030+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:20:32.175012+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:23:36.246673+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:20:02.384818+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:24:59.428 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4293 8 365281 365273 0.664681 0 - 0.0299513 2026-02-23T14:25:00.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:24:59 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/4012793046' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:25:00.428 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4294 8 365281 365273 0.664526 0 - 0.0299513 2026-02-23T14:25:01.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:25:00 trial068 ceph-mon[38805]: pgmap v2664: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 1 active+clean+scrubbing, 4 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:25:01.428 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4295 8 365281 365273 0.664372 0 - 0.0299513 2026-02-23T14:25:02.205 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:25:02.323 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:25:02.422 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:25:02.420+0000 7f1abf66b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:25:02.422 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:25:02.420+0000 7f1abf66b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:25:02.428 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4296 8 365281 365273 0.664217 0 - 0.0299513 2026-02-23T14:25:02.532 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:25:02.532 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":239,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:24:04.576030+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:20:32.175012+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:23:36.246673+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:20:02.384818+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:25:02.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:25:02 trial068 ceph-mon[38805]: pgmap v2665: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:25:02.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:25:02 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2210280771' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:25:03.429 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4297 8 365281 365273 0.664062 0 - 0.0299513 2026-02-23T14:25:04.266 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:25:03 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:25:03.974Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:25:04.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:25:03 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:25:03] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:25:04.428 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4298 8 365281 365273 0.663908 0 - 0.0299513 2026-02-23T14:25:05.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:25:04 trial068 ceph-mon[38805]: pgmap v2666: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:25:05.429 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4299 8 365281 365273 0.663753 0 - 0.0299513 2026-02-23T14:25:05.569 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:25:05.686 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:25:05.784 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:25:05.782+0000 7fdf86d52640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:25:05.785 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:25:05.782+0000 7fdf86d52640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:25:05.896 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:25:05.896 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":239,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:24:04.576030+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:20:32.175012+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:23:36.246673+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:20:02.384818+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:25:06.429 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T14:25:06.428292+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T14:25:06.429 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T14:25:06.429 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4300 8 365281 365273 0.663599 0 - 0.0299513 2026-02-23T14:25:07.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:25:06 trial068 ceph-mon[38805]: pgmap v2667: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:25:07.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:25:06 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1130636845' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:25:07.429 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4301 8 365281 365273 0.663445 0 - 0.0299513 2026-02-23T14:25:08.429 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4302 8 365281 365273 0.663291 0 - 0.0299513 2026-02-23T14:25:08.933 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:25:08.959 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:25:08 trial068 ceph-mon[38805]: pgmap v2668: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:25:09.052 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:25:09.151 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:25:09.149+0000 7f1b9550f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:25:09.151 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:25:09.149+0000 7f1b9550f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:25:09.261 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:25:09.261 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":239,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:24:04.576030+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:20:32.175012+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:23:36.246673+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:20:02.384818+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:25:09.429 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4303 8 365281 365273 0.663136 0 - 0.0299513 2026-02-23T14:25:10.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:25:09 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2475235945' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:25:10.429 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4304 8 365281 365273 0.662982 0 - 0.0299513 2026-02-23T14:25:11.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:25:10 trial068 ceph-mon[38805]: pgmap v2669: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:25:11.429 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4305 8 365281 365273 0.662828 0 - 0.0299513 2026-02-23T14:25:12.299 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:25:12.417 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:25:12.429 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4306 8 365281 365273 0.662674 0 - 0.0299513 2026-02-23T14:25:12.527 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:25:12.525+0000 7f53d1f72640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:25:12.527 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:25:12.525+0000 7f53d1f72640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:25:12.638 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:25:12.638 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":239,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:24:04.576030+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:20:32.175012+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:23:36.246673+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:20:02.384818+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:25:13.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:25:12 trial068 ceph-mon[38805]: pgmap v2670: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:25:13.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:25:12 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3485068474' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:25:13.429 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4307 8 365281 365273 0.662521 0 - 0.0299513 2026-02-23T14:25:14.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:25:13 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:25:13] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:25:14.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:25:13 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:25:13.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:25:14.429 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4308 8 365281 365273 0.662367 0 - 0.0299513 2026-02-23T14:25:15.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:25:14 trial068 ceph-mon[38805]: pgmap v2671: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:25:15.429 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4309 8 365281 365273 0.662213 0 - 0.0299513 2026-02-23T14:25:15.676 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:25:15.795 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:25:15.895 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:25:15.893+0000 7fecf663d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:25:15.895 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:25:15.893+0000 7fecf663d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:25:16.008 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:25:16.008 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":240,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:25:15.692769+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:20:32.175012+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:23:36.246673+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:23:57.967252+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:25:16.429 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4310 8 365281 365273 0.662059 0 - 0.0299513 2026-02-23T14:25:17.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:25:16 trial068 ceph-mon[38805]: pgmap v2672: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:25:17.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:25:16 trial068 ceph-mon[38805]: osdmap e240: 4 total, 3 up, 4 in 2026-02-23T14:25:17.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:25:16 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2757311925' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:25:17.429 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4311 8 365281 365273 0.661906 0 - 0.0299513 2026-02-23T14:25:18.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:25:17 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T14:25:18.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:25:17 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T14:25:18.430 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4312 8 365281 365273 0.661752 0 - 0.0299513 2026-02-23T14:25:19.047 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:25:19.071 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:25:18 trial068 ceph-mon[38805]: pgmap v2674: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:25:19.164 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:25:19.265 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:25:19.262+0000 7f1a32383640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:25:19.265 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:25:19.263+0000 7f1a32383640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:25:19.375 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:25:19.375 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":240,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:25:15.692769+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:20:32.175012+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:23:36.246673+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:23:57.967252+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:25:19.430 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4313 8 365281 365273 0.661599 0 - 0.0299513 2026-02-23T14:25:20.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:25:19 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1728940920' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:25:20.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:25:19 trial068 ceph-mon[38805]: osdmap e241: 4 total, 3 up, 4 in 2026-02-23T14:25:20.430 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4314 8 365281 365273 0.661446 0 - 0.0299513 2026-02-23T14:25:21.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:25:20 trial068 ceph-mon[38805]: pgmap v2675: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:25:21.430 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4315 8 365281 365273 0.661292 0 - 0.0299513 2026-02-23T14:25:22.412 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:25:22.430 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4316 8 365281 365273 0.661139 0 - 0.0299513 2026-02-23T14:25:22.530 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:25:22.627 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:25:22.625+0000 7fdf08200640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:25:22.627 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:25:22.625+0000 7fdf08200640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:25:22.737 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:25:22.737 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":241,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:25:19.587927+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:20:32.175012+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:23:36.246673+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:23:57.967252+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:25:23.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:25:22 trial068 ceph-mon[38805]: pgmap v2677: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:25:23.430 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4317 8 365281 365273 0.660986 0 - 0.0299513 2026-02-23T14:25:23.975 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:25:23 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1439089571' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:25:24.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:25:23 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:25:23] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:25:24.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:25:23 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:25:23.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:25:24.430 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4318 8 365281 365273 0.660833 0 - 0.0299513 2026-02-23T14:25:25.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:25:24 trial068 ceph-mon[38805]: pgmap v2678: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:25:25.430 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4319 8 365281 365273 0.66068 0 - 0.0299513 2026-02-23T14:25:25.776 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:25:25.895 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:25:26.000 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:25:25.998+0000 7f26fb401640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:25:26.000 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:25:25.998+0000 7f26fb401640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:25:26.111 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:25:26.111 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":241,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:25:19.587927+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:20:32.175012+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:23:36.246673+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:23:57.967252+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:25:26.430 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T14:25:26.429813+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T14:25:26.430 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T14:25:26.431 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4320 8 365281 365273 0.660527 0 - 0.0299513 2026-02-23T14:25:27.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:25:26 trial068 ceph-mon[38805]: pgmap v2679: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:25:27.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:25:26 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2208886816' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:25:27.430 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4321 8 365281 365273 0.660374 0 - 0.0299513 2026-02-23T14:25:28.430 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4322 8 365281 365273 0.660221 0 - 0.0299513 2026-02-23T14:25:29.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:25:28 trial068 ceph-mon[38805]: pgmap v2680: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:25:29.149 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:25:29.266 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:25:29.368 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:25:29.366+0000 7f85597e7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:25:29.368 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:25:29.366+0000 7f85597e7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:25:29.430 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4323 8 365281 365273 0.660068 0 - 0.0299513 2026-02-23T14:25:29.477 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:25:29.477 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":241,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:25:19.587927+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:20:32.175012+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:23:36.246673+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:23:57.967252+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:25:29.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:25:29 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3418924564' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:25:30.430 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4324 8 365281 365273 0.659916 0 - 0.0299513 2026-02-23T14:25:31.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:25:30 trial068 ceph-mon[38805]: pgmap v2681: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:25:31.431 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4325 8 365281 365273 0.659763 0 - 0.0299513 2026-02-23T14:25:32.431 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4326 8 365281 365273 0.659611 0 - 0.0299513 2026-02-23T14:25:32.514 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:25:32.633 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:25:32.731 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:25:32.729+0000 7fd9dcada640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:25:32.732 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:25:32.729+0000 7fd9dcada640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:25:32.844 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:25:32.844 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":241,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:25:19.587927+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:20:32.175012+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:23:36.246673+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:23:57.967252+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:25:32.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:25:32 trial068 ceph-mon[38805]: pgmap v2682: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:25:32.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:25:32 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T14:25:32.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:25:32 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T14:25:33.431 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4327 8 365281 365273 0.659458 0 - 0.0299513 2026-02-23T14:25:33.974 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:25:33 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3639510824' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:25:34.265 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:25:33 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:25:33.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:25:34.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:25:33 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:25:33] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:25:34.431 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4328 8 365281 365273 0.659306 0 - 0.0299513 2026-02-23T14:25:35.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:25:34 trial068 ceph-mon[38805]: pgmap v2683: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:25:35.431 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4329 8 365281 365273 0.659154 0 - 0.0299513 2026-02-23T14:25:35.882 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:25:35.998 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:25:36.097 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:25:36.094+0000 7fd036855640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:25:36.097 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:25:36.095+0000 7fd036855640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:25:36.207 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:25:36.208 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":241,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:25:19.587927+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:20:32.175012+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:23:36.246673+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:23:57.967252+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:25:36.431 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4330 8 365281 365273 0.659001 0 - 0.0299513 2026-02-23T14:25:37.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:25:36 trial068 ceph-mon[38805]: pgmap v2684: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:25:37.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:25:36 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1502303085' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:25:37.431 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4331 8 365281 365273 0.658849 0 - 0.0299513 2026-02-23T14:25:38.431 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4332 8 365281 365273 0.658697 0 - 0.0299513 2026-02-23T14:25:39.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:25:38 trial068 ceph-mon[38805]: pgmap v2685: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:25:39.246 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:25:39.364 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:25:39.432 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4333 8 365281 365273 0.658545 0 - 0.0299513 2026-02-23T14:25:39.465 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:25:39.463+0000 7f986e5ab640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:25:39.465 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:25:39.463+0000 7f986e5ab640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:25:39.579 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:25:39.579 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":241,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:25:19.587927+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:20:32.175012+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:23:36.246673+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:23:57.967252+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:25:39.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:25:39 trial068 ceph-mon[38805]: 5.8 scrub starts 2026-02-23T14:25:39.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:25:39 trial068 ceph-mon[38805]: 5.8 scrub ok 2026-02-23T14:25:39.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:25:39 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3858445689' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:25:40.432 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4334 8 365281 365273 0.658393 0 - 0.0299513 2026-02-23T14:25:41.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:25:40 trial068 ceph-mon[38805]: pgmap v2686: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:25:41.432 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4335 8 365281 365273 0.658241 0 - 0.0299513 2026-02-23T14:25:42.432 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4336 8 365281 365273 0.658089 0 - 0.0299513 2026-02-23T14:25:42.617 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:25:42.735 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:25:42.840 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:25:42.838+0000 7f3bab6be640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:25:42.841 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:25:42.838+0000 7f3bab6be640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:25:42.950 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:25:42.950 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":241,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:25:19.587927+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:20:32.175012+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:23:36.246673+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:23:57.967252+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:25:42.950 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:25:42 trial068 ceph-mon[38805]: pgmap v2687: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:25:43.432 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4337 8 365281 365273 0.657938 0 - 0.0299513 2026-02-23T14:25:43.974 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:25:43 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/4054569954' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:25:44.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:25:43 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:25:43] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:25:44.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:25:43 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:25:43.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:25:44.432 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4338 8 365281 365273 0.657786 0 - 0.0299513 2026-02-23T14:25:45.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:25:44 trial068 ceph-mon[38805]: pgmap v2688: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:25:45.432 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4339 8 365281 365273 0.657634 0 - 0.0299513 2026-02-23T14:25:46.004 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:25:46.123 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:25:46.221 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:25:46.219+0000 7f9bd114d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:25:46.221 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:25:46.219+0000 7f9bd114d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:25:46.334 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:25:46.334 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":241,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:25:19.587927+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:20:32.175012+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:23:36.246673+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:23:57.967252+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:25:46.432 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T14:25:46.431767+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T14:25:46.432 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T14:25:46.433 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4340 8 365281 365273 0.657483 0 - 0.0299513 2026-02-23T14:25:47.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:25:46 trial068 ceph-mon[38805]: pgmap v2689: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:25:47.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:25:46 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2856401710' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:25:47.432 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4341 8 365281 365273 0.657331 0 - 0.0299513 2026-02-23T14:25:48.432 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4342 8 365281 365273 0.65718 0 - 0.0299513 2026-02-23T14:25:49.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:25:48 trial068 ceph-mon[38805]: 5.5 scrub starts 2026-02-23T14:25:49.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:25:48 trial068 ceph-mon[38805]: pgmap v2690: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:25:49.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:25:48 trial068 ceph-mon[38805]: 5.5 scrub ok 2026-02-23T14:25:49.374 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:25:49.432 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4343 8 365281 365273 0.657029 0 - 0.0299513 2026-02-23T14:25:49.488 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:25:49.587 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:25:49.584+0000 7f81e7e36640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:25:49.587 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:25:49.585+0000 7f81e7e36640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:25:49.697 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:25:49.697 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":241,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:25:19.587927+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:20:32.175012+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:23:36.246673+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:23:57.967252+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:25:50.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:25:49 trial068 ceph-mon[38805]: 5.e scrub starts 2026-02-23T14:25:50.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:25:49 trial068 ceph-mon[38805]: 5.e scrub ok 2026-02-23T14:25:50.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:25:49 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3588391537' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:25:50.432 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4344 8 365281 365273 0.656877 0 - 0.0299513 2026-02-23T14:25:51.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:25:50 trial068 ceph-mon[38805]: pgmap v2691: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:25:51.432 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4345 8 365281 365273 0.656726 0 - 0.0299513 2026-02-23T14:25:52.433 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4346 8 365281 365273 0.656575 0 - 0.0299513 2026-02-23T14:25:52.735 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:25:52.853 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:25:52.955 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:25:52.952+0000 7ff02ae2c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:25:52.955 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:25:52.953+0000 7ff02ae2c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:25:53.063 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:25:53.063 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":241,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:25:19.587927+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:20:32.175012+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:23:36.246673+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:23:57.967252+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:25:53.064 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:25:52 trial068 ceph-mon[38805]: pgmap v2692: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:25:53.433 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4347 8 365281 365273 0.656424 0 - 0.0299513 2026-02-23T14:25:54.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:25:53 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3784137431' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:25:54.094 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:25:53 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:25:53] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:25:54.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:25:53 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:25:53.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:25:54.433 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4348 8 365281 365273 0.656273 0 - 0.0299513 2026-02-23T14:25:55.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:25:54 trial068 ceph-mon[38805]: pgmap v2693: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:25:55.433 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4349 8 365281 365273 0.656122 0 - 0.0299513 2026-02-23T14:25:56.101 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:25:56.220 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:25:56.323 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:25:56.321+0000 7f9a91fc2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:25:56.323 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:25:56.321+0000 7f9a91fc2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:25:56.432 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:25:56.432 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":241,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:25:19.587927+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:20:32.175012+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:23:36.246673+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:23:57.967252+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:25:56.433 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4350 8 365281 365273 0.655971 0 - 0.0299513 2026-02-23T14:25:57.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:25:56 trial068 ceph-mon[38805]: pgmap v2694: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:25:57.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:25:56 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1788778477' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:25:57.433 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4351 8 365281 365273 0.655821 0 - 0.0299513 2026-02-23T14:25:58.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:25:57 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2026-02-23T14:25:58.433 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4352 8 365281 365273 0.65567 0 - 0.0299513 2026-02-23T14:25:59.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:25:58 trial068 ceph-mon[38805]: pgmap v2695: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:25:59.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:25:58 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-23T14:25:59.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:25:58 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2026-02-23T14:25:59.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:25:58 trial068 ceph-mon[38805]: pgmap v2696: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:25:59.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:25:58 trial068 ceph-mon[38805]: pgmap v2697: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:25:59.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:25:58 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T14:25:59.095 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:25:58 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2026-02-23T14:25:59.433 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4353 8 365281 365273 0.655519 0 - 0.0299513 2026-02-23T14:25:59.469 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:25:59.585 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:25:59.684 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:25:59.682+0000 7f3e45124640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:25:59.685 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:25:59.683+0000 7f3e45124640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:25:59.794 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:25:59.794 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":241,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:25:19.587927+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:20:32.175012+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:23:36.246673+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:23:57.967252+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:26:00.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:25:59 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/4134235446' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:26:00.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:25:59 trial068 ceph-mon[38805]: pgmap v2698: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:26:00.433 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4354 8 365281 365273 0.655369 0 - 0.0299513 2026-02-23T14:26:01.433 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4355 8 365281 365273 0.655218 0 - 0.0299513 2026-02-23T14:26:01.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:26:01 trial068 ceph-mon[38805]: 5.7 scrub starts 2026-02-23T14:26:01.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:26:01 trial068 ceph-mon[38805]: 5.7 scrub ok 2026-02-23T14:26:02.433 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4356 8 365281 365273 0.655068 0 - 0.0299513 2026-02-23T14:26:02.495 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:26:02 trial068 ceph-mon[38805]: pgmap v2699: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:26:02.831 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:26:02.950 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:26:03.049 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:26:03.047+0000 7fd521595640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:26:03.050 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:26:03.047+0000 7fd521595640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:26:03.161 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:26:03.161 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":241,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:25:19.587927+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:20:32.175012+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:23:36.246673+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:23:57.967252+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:26:03.434 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4357 8 365281 365273 0.654918 0 - 0.0299513 2026-02-23T14:26:03.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:26:03 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/45737758' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:26:04.266 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:26:03 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:26:03.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:26:04.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:26:04 trial068 ceph-mon[38805]: pgmap v2700: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:26:04.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:26:03 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:26:03] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:26:04.433 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4358 8 365281 365273 0.654767 0 - 0.0299513 2026-02-23T14:26:05.434 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4359 8 365281 365273 0.654617 0 - 0.0299513 2026-02-23T14:26:06.199 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:26:06.225 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:26:05 trial068 ceph-mon[38805]: pgmap v2701: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:26:06.318 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:26:06.419 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:26:06.417+0000 7f892bda1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:26:06.420 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:26:06.417+0000 7f892bda1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:26:06.433 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T14:26:06.433173+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T14:26:06.434 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T14:26:06.434 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4360 8 365281 365273 0.654467 0 - 0.0299513 2026-02-23T14:26:06.527 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:26:06.528 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":241,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:25:19.587927+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:20:32.175012+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:23:36.246673+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:23:57.967252+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:26:07.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:26:06 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/774190780' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:26:07.434 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4361 8 365281 365273 0.654317 0 - 0.0299513 2026-02-23T14:26:08.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:26:07 trial068 ceph-mon[38805]: pgmap v2702: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:26:08.434 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4362 8 365281 365273 0.654167 0 - 0.0299513 2026-02-23T14:26:09.434 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4363 8 365281 365273 0.654017 0 - 0.0299513 2026-02-23T14:26:09.569 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:26:09.686 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:26:09.788 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:26:09.785+0000 7f67a1640640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:26:09.788 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:26:09.786+0000 7f67a1640640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:26:09.896 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:26:09.896 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":241,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:25:19.587927+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:20:32.175012+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:23:36.246673+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:23:57.967252+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:26:10.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:26:09 trial068 ceph-mon[38805]: pgmap v2703: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:26:10.434 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4364 8 365281 365273 0.653867 0 - 0.0299513 2026-02-23T14:26:11.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:26:10 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1404493075' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:26:11.434 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4365 8 365281 365273 0.653717 0 - 0.0299513 2026-02-23T14:26:12.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:26:11 trial068 ceph-mon[38805]: pgmap v2704: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:26:12.434 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4366 8 365281 365273 0.653568 0 - 0.0299513 2026-02-23T14:26:12.934 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:26:13.054 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:26:13.153 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:26:13.150+0000 7fe6a6aba640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:26:13.153 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:26:13.151+0000 7fe6a6aba640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:26:13.262 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:26:13.262 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":241,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:25:19.587927+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:20:32.175012+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:23:36.246673+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:23:57.967252+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:26:13.434 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4367 8 365281 365273 0.653418 0 - 0.0299513 2026-02-23T14:26:13.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:26:13 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/474901481' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:26:14.313 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:26:13 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:26:13] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:26:14.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:26:13 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:26:13.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:26:14.434 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4368 8 365281 365273 0.653268 0 - 0.0299513 2026-02-23T14:26:14.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:26:14 trial068 ceph-mon[38805]: pgmap v2705: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:26:15.434 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4369 8 365281 365273 0.653119 0 - 0.0299513 2026-02-23T14:26:16.300 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:26:16.323 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:26:15 trial068 ceph-mon[38805]: pgmap v2706: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:26:16.417 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:26:16.434 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4370 8 365281 365273 0.652969 0 - 0.0299513 2026-02-23T14:26:16.513 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:26:16.511+0000 7f31da283640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:26:16.513 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:26:16.511+0000 7f31da283640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:26:16.622 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:26:16.622 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":241,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:25:19.587927+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:20:32.175012+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:23:36.246673+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:23:57.967252+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:26:17.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:26:16 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3776726086' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:26:17.434 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4371 8 365281 365273 0.65282 0 - 0.0299513 2026-02-23T14:26:18.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:26:17 trial068 ceph-mon[38805]: pgmap v2707: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:26:18.434 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4372 8 365281 365273 0.652671 0 - 0.0299513 2026-02-23T14:26:19.434 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4373 8 365281 365273 0.652521 0 - 0.0299513 2026-02-23T14:26:19.660 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:26:19.776 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:26:19.875 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:26:19.872+0000 7fee979c3640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:26:19.875 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:26:19.872+0000 7fee979c3640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:26:19.982 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:26:19.982 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":241,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:25:19.587927+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:20:32.175012+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:23:36.246673+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:23:57.967252+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:26:20.434 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4374 8 365281 365273 0.652372 0 - 0.0299513 2026-02-23T14:26:21.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:26:20 trial068 ceph-mon[38805]: pgmap v2708: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:26:21.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:26:20 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/87159537' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:26:21.434 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4375 8 365281 365273 0.652223 0 - 0.0299513 2026-02-23T14:26:22.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:26:21 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T14:26:22.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:26:21 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T14:26:22.435 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4376 8 365281 365273 0.652074 0 - 0.0299513 2026-02-23T14:26:23.019 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:26:23.045 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:26:22 trial068 ceph-mon[38805]: pgmap v2709: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:26:23.139 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:26:23.238 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:26:23.236+0000 7fd0de8be640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:26:23.238 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:26:23.236+0000 7fd0de8be640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:26:23.345 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:26:23.345 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":241,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:25:19.587927+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:20:32.175012+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:23:36.246673+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:23:57.967252+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:26:23.435 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4377 8 365281 365273 0.651925 0 - 0.0299513 2026-02-23T14:26:23.974 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:26:23 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3334771877' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:26:24.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:26:23 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:26:23] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:26:24.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:26:23 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:26:23.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:26:24.435 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4378 8 365281 365273 0.651776 0 - 0.0299513 2026-02-23T14:26:25.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:26:24 trial068 ceph-mon[38805]: pgmap v2710: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:26:25.435 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4379 8 365281 365273 0.651627 0 - 0.0299513 2026-02-23T14:26:26.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:26:25 trial068 ceph-mon[38805]: osdmap e242: 4 total, 3 up, 4 in 2026-02-23T14:26:26.382 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:26:26.435 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T14:26:26.434480+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T14:26:26.435 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T14:26:26.435 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4380 8 365281 365273 0.651479 0 - 0.0299513 2026-02-23T14:26:26.501 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:26:26.604 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:26:26.602+0000 7fc059a56640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:26:26.604 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:26:26.602+0000 7fc059a56640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:26:26.713 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:26:26.714 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":242,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:26:24.603773+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:25:31.747822+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:23:36.246673+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:23:57.967252+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:26:26.714 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:26:26 trial068 ceph-mon[38805]: pgmap v2712: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:26:27.435 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4381 8 365281 365273 0.65133 0 - 0.0299513 2026-02-23T14:26:28.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:26:27 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3712039578' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:26:28.435 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4382 8 365281 365273 0.651181 0 - 0.0299513 2026-02-23T14:26:29.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:26:28 trial068 ceph-mon[38805]: pgmap v2713: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:26:29.435 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4383 8 365281 365273 0.651033 0 - 0.0299513 2026-02-23T14:26:29.751 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:26:29.871 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:26:29.973 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:26:29.971+0000 7f406153b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:26:29.973 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:26:29.971+0000 7f406153b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:26:30.082 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:26:30.082 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":243,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:26:29.596506+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:25:31.747822+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:23:36.246673+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:23:57.967252+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:26:30.435 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4384 8 365281 365273 0.650884 0 - 0.0299513 2026-02-23T14:26:31.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:26:30 trial068 ceph-mon[38805]: osdmap e243: 4 total, 3 up, 4 in 2026-02-23T14:26:31.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:26:30 trial068 ceph-mon[38805]: pgmap v2715: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:26:31.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:26:30 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3767680432' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:26:31.435 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4385 8 365281 365273 0.650736 0 - 0.0299513 2026-02-23T14:26:32.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:26:31 trial068 ceph-mon[38805]: 5.a scrub starts 2026-02-23T14:26:32.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:26:31 trial068 ceph-mon[38805]: 5.a scrub ok 2026-02-23T14:26:32.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:26:31 trial068 ceph-mon[38805]: pgmap v2716: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:26:32.435 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4386 8 365281 365273 0.650587 0 - 0.0299513 2026-02-23T14:26:33.121 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:26:33.238 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:26:33.344 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:26:33.342+0000 7fb5dc9d8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:26:33.344 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:26:33.342+0000 7fb5dc9d8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:26:33.435 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4387 8 365281 365273 0.650439 0 - 0.0299513 2026-02-23T14:26:33.453 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:26:33.454 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":243,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:26:29.596506+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:25:31.747822+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:23:36.246673+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:23:57.967252+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:26:33.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:26:33 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/98381658' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:26:34.266 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:26:33 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:26:33.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:26:34.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:26:33 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:26:33] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:26:34.435 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4388 8 365281 365273 0.650291 0 - 0.0299513 2026-02-23T14:26:34.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:26:34 trial068 ceph-mon[38805]: pgmap v2717: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:26:35.436 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4389 8 365281 365273 0.650143 0 - 0.0299513 2026-02-23T14:26:36.436 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4390 8 365281 365273 0.649995 0 - 0.0299513 2026-02-23T14:26:36.492 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:26:36.611 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:26:36.711 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:26:36.709+0000 7f0d5f792640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:26:36.711 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:26:36.709+0000 7f0d5f792640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:26:36.818 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:26:36.819 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":243,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:26:29.596506+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:25:31.747822+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:23:36.246673+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:23:57.967252+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:26:37.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:26:36 trial068 ceph-mon[38805]: pgmap v2718: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:26:37.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:26:36 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/4024671634' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:26:37.436 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4391 8 365281 365273 0.649847 0 - 0.0299513 2026-02-23T14:26:38.436 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4392 8 365281 365273 0.649699 0 - 0.0299513 2026-02-23T14:26:39.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:26:38 trial068 ceph-mon[38805]: pgmap v2719: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:26:39.436 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4393 8 365281 365273 0.649551 0 - 0.0299513 2026-02-23T14:26:39.856 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:26:39.883 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:26:39 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T14:26:39.884 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:26:39 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T14:26:39.971 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:26:40.083 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:26:40.081+0000 7f82265ae640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:26:40.083 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:26:40.081+0000 7f82265ae640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:26:40.194 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:26:40.194 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":243,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:26:29.596506+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:25:31.747822+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:23:36.246673+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:23:57.967252+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:26:40.436 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4394 8 365281 365273 0.649403 0 - 0.0299513 2026-02-23T14:26:41.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:26:40 trial068 ceph-mon[38805]: pgmap v2720: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:26:41.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:26:40 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3574791295' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:26:41.436 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4395 8 365281 365273 0.649255 0 - 0.0299513 2026-02-23T14:26:42.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:26:41 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T14:26:42.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:26:41 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T14:26:42.436 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4396 8 365281 365273 0.649107 0 - 0.0299513 2026-02-23T14:26:43.232 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:26:43.256 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:26:42 trial068 ceph-mon[38805]: pgmap v2721: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:26:43.350 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:26:43.436 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4397 8 365281 365273 0.64896 0 - 0.0299513 2026-02-23T14:26:43.450 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:26:43.448+0000 7f000096f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:26:43.450 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:26:43.448+0000 7f000096f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:26:43.561 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:26:43.561 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":243,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:26:29.596506+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:25:31.747822+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:23:36.246673+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:23:57.967252+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:26:44.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:26:43 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2668388603' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:26:44.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:26:43 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:26:43] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:26:44.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:26:43 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:26:43.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:26:44.436 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4398 8 365281 365273 0.648812 0 - 0.0299513 2026-02-23T14:26:45.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:26:44 trial068 ceph-mon[38805]: pgmap v2722: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:26:45.436 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4399 8 365281 365273 0.648665 0 - 0.0299513 2026-02-23T14:26:46.437 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T14:26:46.436158+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T14:26:46.437 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T14:26:46.437 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4400 8 365281 365273 0.648517 0 - 0.0299513 2026-02-23T14:26:46.599 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:26:46.718 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:26:46.817 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:26:46.815+0000 7fe142fc9640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:26:46.818 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:26:46.815+0000 7fe142fc9640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:26:46.928 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:26:46.928 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":243,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:26:29.596506+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:25:31.747822+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:23:36.246673+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:23:57.967252+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:26:47.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:26:46 trial068 ceph-mon[38805]: pgmap v2723: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:26:47.437 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4401 8 365281 365273 0.64837 0 - 0.0299513 2026-02-23T14:26:48.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:26:47 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1753628183' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:26:48.436 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4402 8 365281 365273 0.648223 0 - 0.0299513 2026-02-23T14:26:49.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:26:48 trial068 ceph-mon[38805]: pgmap v2724: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:26:49.437 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4403 8 365281 365273 0.648075 0 - 0.0299513 2026-02-23T14:26:49.966 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:26:50.084 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:26:50.183 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:26:50.181+0000 7ffa5f99d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:26:50.183 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:26:50.181+0000 7ffa5f99d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:26:50.294 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:26:50.294 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":243,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:26:29.596506+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:25:31.747822+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:23:36.246673+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:23:57.967252+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:26:50.437 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4404 8 365281 365273 0.647928 0 - 0.0299513 2026-02-23T14:26:51.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:26:50 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1278189113' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:26:51.437 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4405 8 365281 365273 0.647781 0 - 0.0299513 2026-02-23T14:26:52.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:26:51 trial068 ceph-mon[38805]: pgmap v2725: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:26:52.437 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4406 8 365281 365273 0.647634 0 - 0.0299513 2026-02-23T14:26:53.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:26:52 trial068 ceph-mon[38805]: 5.8 scrub starts 2026-02-23T14:26:53.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:26:52 trial068 ceph-mon[38805]: 5.8 scrub ok 2026-02-23T14:26:53.333 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:26:53.437 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4407 8 365281 365273 0.647487 0 - 0.0299513 2026-02-23T14:26:53.449 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:26:53.546 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:26:53.544+0000 7f068d86c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:26:53.546 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:26:53.544+0000 7f068d86c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:26:53.654 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:26:53.654 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":243,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:26:29.596506+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:25:31.747822+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:23:36.246673+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:23:57.967252+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:26:53.975 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:26:53 trial068 ceph-mon[38805]: pgmap v2726: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:26:54.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:26:53 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:26:53] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:26:54.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:26:53 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:26:53.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:26:54.437 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4408 8 365281 365273 0.64734 0 - 0.0299513 2026-02-23T14:26:55.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:26:54 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2557688573' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:26:55.437 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4409 8 365281 365273 0.647193 0 - 0.0299513 2026-02-23T14:26:56.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:26:55 trial068 ceph-mon[38805]: pgmap v2727: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:26:56.437 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4410 8 365281 365273 0.647047 0 - 0.0299513 2026-02-23T14:26:56.692 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:26:56.812 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:26:56.913 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:26:56.911+0000 7fa7b834f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:26:56.914 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:26:56.911+0000 7fa7b834f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:26:57.023 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:26:57.024 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":243,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:26:29.596506+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:25:31.747822+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:23:36.246673+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:23:57.967252+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:26:57.437 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4411 8 365281 365273 0.6469 0 - 0.0299513 2026-02-23T14:26:58.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:26:57 trial068 ceph-mon[38805]: pgmap v2728: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:26:58.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:26:57 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2494109659' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:26:58.437 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4412 8 365281 365273 0.646753 0 - 0.0299513 2026-02-23T14:26:59.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:26:58 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2026-02-23T14:26:59.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:26:58 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T14:26:59.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:26:58 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T14:26:59.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:26:58 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T14:26:59.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:26:58 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T14:26:59.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:26:58 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-23T14:26:59.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:26:58 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2026-02-23T14:26:59.095 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:26:58 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T14:26:59.095 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:26:58 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2026-02-23T14:26:59.437 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4413 8 365281 365273 0.646607 0 - 0.0299513 2026-02-23T14:27:00.061 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:27:00.085 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:26:59 trial068 ceph-mon[38805]: pgmap v2729: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:27:00.085 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:26:59 trial068 ceph-mon[38805]: pgmap v2730: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:27:00.085 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:26:59 trial068 ceph-mon[38805]: pgmap v2731: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:27:00.178 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:27:00.277 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:27:00.275+0000 7f92fa4e2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:27:00.277 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:27:00.275+0000 7f92fa4e2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:27:00.386 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:27:00.386 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":243,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:26:29.596506+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:25:31.747822+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:23:36.246673+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:23:57.967252+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:27:00.437 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4414 8 365281 365273 0.64646 0 - 0.0299513 2026-02-23T14:27:00.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:27:00 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1139381749' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:27:01.438 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4415 8 365281 365273 0.646314 0 - 0.0299513 2026-02-23T14:27:02.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:27:01 trial068 ceph-mon[38805]: pgmap v2732: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:27:02.438 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4416 8 365281 365273 0.646168 0 - 0.0299513 2026-02-23T14:27:03.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:27:02 trial068 ceph-mon[38805]: 5.5 scrub starts 2026-02-23T14:27:03.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:27:02 trial068 ceph-mon[38805]: 5.5 scrub ok 2026-02-23T14:27:03.423 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:27:03.438 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4417 8 365281 365273 0.646021 0 - 0.0299513 2026-02-23T14:27:03.540 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:27:03.639 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:27:03.637+0000 7fab3e0b7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:27:03.639 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:27:03.637+0000 7fab3e0b7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:27:03.749 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:27:03.749 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":243,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:26:29.596506+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:25:31.747822+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:23:36.246673+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:23:57.967252+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:27:03.749 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:27:03 trial068 ceph-mon[38805]: pgmap v2733: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:27:04.094 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:27:03 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:27:03] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:27:04.266 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:27:03 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:27:03.974Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:27:04.438 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4418 8 365281 365273 0.645875 0 - 0.0299513 2026-02-23T14:27:05.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:27:04 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2761123610' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:27:05.438 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4419 8 365281 365273 0.645729 0 - 0.0299513 2026-02-23T14:27:06.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:27:05 trial068 ceph-mon[38805]: pgmap v2734: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:27:06.438 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T14:27:06.437647+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T14:27:06.438 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T14:27:06.438 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4420 8 365281 365273 0.645583 0 - 0.0299513 2026-02-23T14:27:06.787 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:27:06.908 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:27:07.008 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:27:07.005+0000 7f689964b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:27:07.008 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:27:07.006+0000 7f689964b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:27:07.116 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:27:07.117 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":243,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:26:29.596506+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:25:31.747822+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:23:36.246673+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:23:57.967252+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:27:07.438 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4421 8 365281 365273 0.645437 0 - 0.0299513 2026-02-23T14:27:08.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:27:07 trial068 ceph-mon[38805]: pgmap v2735: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:27:08.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:27:07 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3944366237' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:27:08.438 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4422 8 365281 365273 0.645291 0 - 0.0299513 2026-02-23T14:27:09.438 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4423 8 365281 365273 0.645145 0 - 0.0299513 2026-02-23T14:27:10.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:27:09 trial068 ceph-mon[38805]: pgmap v2736: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:27:10.155 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:27:10.273 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:27:10.369 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:27:10.367+0000 7f6d6d8bd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:27:10.369 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:27:10.367+0000 7f6d6d8bd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:27:10.438 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4424 8 365281 365273 0.644999 0 - 0.0299513 2026-02-23T14:27:10.477 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:27:10.477 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":243,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:26:29.596506+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:25:31.747822+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:23:36.246673+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:23:57.967252+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:27:10.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:27:10 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3374800953' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:27:11.438 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4425 8 365281 365273 0.644853 0 - 0.0299513 2026-02-23T14:27:12.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:27:11 trial068 ceph-mon[38805]: pgmap v2737: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:27:12.438 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4426 8 365281 365273 0.644708 0 - 0.0299513 2026-02-23T14:27:13.439 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4427 8 365281 365273 0.644562 0 - 0.0299513 2026-02-23T14:27:13.514 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:27:13.631 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:27:13.728 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:27:13.726+0000 7fe067cd3640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:27:13.728 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:27:13.726+0000 7fe067cd3640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:27:13.839 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:27:13.839 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":243,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:26:29.596506+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:25:31.747822+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:23:36.246673+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:23:57.967252+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:27:13.839 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:27:13 trial068 ceph-mon[38805]: pgmap v2738: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:27:13.839 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:27:13 trial068 ceph-mon[38805]: 5.e scrub starts 2026-02-23T14:27:13.839 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:27:13 trial068 ceph-mon[38805]: 5.e scrub ok 2026-02-23T14:27:14.094 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:27:13 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:27:13] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:27:14.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:27:13 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:27:13.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:27:14.439 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4428 8 365281 365273 0.644416 0 - 0.0299513 2026-02-23T14:27:15.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:27:14 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2611281185' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:27:15.439 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4429 8 365281 365273 0.644271 0 - 0.0299513 2026-02-23T14:27:16.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:27:15 trial068 ceph-mon[38805]: pgmap v2739: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:27:16.439 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4430 8 365281 365273 0.644125 0 - 0.0299513 2026-02-23T14:27:16.876 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:27:16.993 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:27:17.097 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:27:17.094+0000 7feddb314640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:27:17.097 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:27:17.095+0000 7feddb314640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:27:17.208 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:27:17.208 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":243,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:26:29.596506+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:25:31.747822+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:23:36.246673+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:23:57.967252+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:27:17.439 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4431 8 365281 365273 0.64398 0 - 0.0299513 2026-02-23T14:27:18.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:27:17 trial068 ceph-mon[38805]: pgmap v2740: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:27:18.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:27:17 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/706339553' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:27:18.439 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4432 8 365281 365273 0.643835 0 - 0.0299513 2026-02-23T14:27:19.439 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4433 8 365281 365273 0.64369 0 - 0.0299513 2026-02-23T14:27:20.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:27:19 trial068 ceph-mon[38805]: pgmap v2741: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:27:20.248 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:27:20.364 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:27:20.439 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4434 8 365281 365273 0.643544 0 - 0.0299513 2026-02-23T14:27:20.464 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:27:20.462+0000 7f65dda7f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:27:20.464 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:27:20.462+0000 7f65dda7f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:27:20.573 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:27:20.573 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":243,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:26:29.596506+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:25:31.747822+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:23:36.246673+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:23:57.967252+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:27:20.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:27:20 trial068 ceph-mon[38805]: 5.7 scrub starts 2026-02-23T14:27:20.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:27:20 trial068 ceph-mon[38805]: 5.7 scrub ok 2026-02-23T14:27:20.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:27:20 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3609245092' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:27:21.439 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4435 8 365281 365273 0.643399 0 - 0.0299513 2026-02-23T14:27:22.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:27:21 trial068 ceph-mon[38805]: pgmap v2742: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:27:22.439 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4436 8 365281 365273 0.643254 0 - 0.0299513 2026-02-23T14:27:23.439 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4437 8 365281 365273 0.643109 0 - 0.0299513 2026-02-23T14:27:23.611 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:27:23.731 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:27:23.831 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:27:23.829+0000 7f9e88d02640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:27:23.831 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:27:23.829+0000 7f9e88d02640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:27:23.939 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:27:23 trial068 ceph-mon[38805]: pgmap v2743: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:27:23.939 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:27:23.939 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":243,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:26:29.596506+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:25:31.747822+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:23:36.246673+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:23:57.967252+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:27:24.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:27:23 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:27:23] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:27:24.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:27:23 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:27:23.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:27:24.439 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4438 8 365281 365273 0.642964 0 - 0.0299513 2026-02-23T14:27:25.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:27:24 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2129699610' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:27:25.440 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4439 8 365281 365273 0.64282 0 - 0.0299513 2026-02-23T14:27:26.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:27:25 trial068 ceph-mon[38805]: pgmap v2744: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:27:26.440 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T14:27:26.439398+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T14:27:26.440 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T14:27:26.440 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4440 8 365281 365273 0.642675 0 - 0.0299513 2026-02-23T14:27:26.975 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:27:27.093 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:27:27.196 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:27:27.193+0000 7ff685540640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:27:27.196 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:27:27.194+0000 7ff685540640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:27:27.303 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:27:27.303 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":243,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:26:29.596506+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:25:31.747822+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:23:36.246673+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:23:57.967252+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:27:27.440 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4441 8 365281 365273 0.64253 0 - 0.0299513 2026-02-23T14:27:28.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:27:27 trial068 ceph-mon[38805]: pgmap v2745: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:27:28.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:27:27 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2461105430' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:27:28.440 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4442 8 365281 365273 0.642385 0 - 0.0299513 2026-02-23T14:27:29.440 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4443 8 365281 365273 0.642241 0 - 0.0299513 2026-02-23T14:27:30.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:27:29 trial068 ceph-mon[38805]: pgmap v2746: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:27:30.341 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:27:30.440 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4444 8 365281 365273 0.642096 0 - 0.0299513 2026-02-23T14:27:30.461 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:27:30.559 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:27:30.557+0000 7f0d9b2c0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:27:30.559 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:27:30.557+0000 7f0d9b2c0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:27:30.667 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:27:30.668 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":243,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:26:29.596506+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:25:31.747822+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:23:36.246673+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:23:57.967252+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:27:31.440 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4445 8 365281 365273 0.641952 0 - 0.0299513 2026-02-23T14:27:32.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:27:31 trial068 ceph-mon[38805]: pgmap v2747: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:27:32.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:27:31 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/32483433' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:27:32.440 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4446 8 365281 365273 0.641807 0 - 0.0299513 2026-02-23T14:27:33.440 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4447 8 365281 365273 0.641663 0 - 0.0299513 2026-02-23T14:27:33.705 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:27:33.821 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:27:33.923 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:27:33.920+0000 7faac5077640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:27:33.923 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:27:33.921+0000 7faac5077640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:27:33.923 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:27:33 trial068 ceph-mon[38805]: pgmap v2748: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:27:34.034 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:27:34.034 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":243,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:26:29.596506+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:25:31.747822+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:23:36.246673+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:23:57.967252+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:27:34.266 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:27:33 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:27:33.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:27:34.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:27:33 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:27:33] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:27:34.440 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4448 8 365281 365273 0.641519 0 - 0.0299513 2026-02-23T14:27:35.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:27:34 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/780979483' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:27:35.440 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4449 8 365281 365273 0.641375 0 - 0.0299513 2026-02-23T14:27:36.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:27:35 trial068 ceph-mon[38805]: pgmap v2749: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:27:36.440 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4450 8 365281 365273 0.641231 0 - 0.0299513 2026-02-23T14:27:37.073 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:27:37.190 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:27:37.286 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:27:37.284+0000 7f3f41dd9640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:27:37.287 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:27:37.285+0000 7f3f41dd9640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:27:37.400 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:27:37.400 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":243,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:26:29.596506+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:25:31.747822+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:23:36.246673+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:23:57.967252+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:27:37.440 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4451 8 365281 365273 0.641086 0 - 0.0299513 2026-02-23T14:27:38.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:27:37 trial068 ceph-mon[38805]: pgmap v2750: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:27:38.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:27:37 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1169454120' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:27:38.441 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4452 8 365281 365273 0.640942 0 - 0.0299513 2026-02-23T14:27:39.441 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4453 8 365281 365273 0.640799 0 - 0.0299513 2026-02-23T14:27:40.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:27:39 trial068 ceph-mon[38805]: pgmap v2751: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:27:40.437 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:27:40.441 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4454 8 365281 365273 0.640655 0 - 0.0299513 2026-02-23T14:27:40.555 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:27:40.659 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:27:40.657+0000 7fd05f85d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:27:40.659 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:27:40.657+0000 7fd05f85d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:27:40.770 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:27:40.770 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":243,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:26:29.596506+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:25:31.747822+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:23:36.246673+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:23:57.967252+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:27:41.441 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4455 8 365281 365273 0.640511 0 - 0.0299513 2026-02-23T14:27:42.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:27:41 trial068 ceph-mon[38805]: pgmap v2752: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:27:42.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:27:41 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2778112081' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:27:42.441 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4456 8 365281 365273 0.640367 0 - 0.0299513 2026-02-23T14:27:43.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:27:42 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T14:27:43.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:27:42 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T14:27:43.441 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4457 8 365281 365273 0.640223 0 - 0.0299513 2026-02-23T14:27:43.809 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:27:43.929 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:27:43.952 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:27:43 trial068 ceph-mon[38805]: pgmap v2753: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:27:44.029 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:27:44.027+0000 7f212b16b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:27:44.029 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:27:44.027+0000 7f212b16b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:27:44.138 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:27:44.138 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":243,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:26:29.596506+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:25:31.747822+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:23:36.246673+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:23:57.967252+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:27:44.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:27:43 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:27:43] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:27:44.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:27:43 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:27:43.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:27:44.441 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4458 8 365281 365273 0.64008 0 - 0.0299513 2026-02-23T14:27:45.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:27:44 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3809753204' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:27:45.441 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4459 8 365281 365273 0.639936 0 - 0.0299513 2026-02-23T14:27:46.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:27:45 trial068 ceph-mon[38805]: pgmap v2754: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:27:46.441 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T14:27:46.440835+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T14:27:46.441 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T14:27:46.441 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4460 8 365281 365273 0.639793 0 - 0.0299513 2026-02-23T14:27:47.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:27:46 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T14:27:47.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:27:46 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T14:27:47.175 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:27:47.295 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:27:47.397 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:27:47.394+0000 7fecd08a2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:27:47.397 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:27:47.395+0000 7fecd08a2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:27:47.441 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4461 8 365281 365273 0.639649 0 - 0.0299513 2026-02-23T14:27:47.505 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:27:47.505 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":243,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:26:29.596506+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:25:31.747822+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:23:36.246673+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:23:57.967252+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:27:47.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:27:47 trial068 ceph-mon[38805]: pgmap v2755: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:27:47.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:27:47 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1858762926' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:27:48.441 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4462 8 365281 365273 0.639506 0 - 0.0299513 2026-02-23T14:27:49.441 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4463 8 365281 365273 0.639363 0 - 0.0299513 2026-02-23T14:27:50.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:27:49 trial068 ceph-mon[38805]: pgmap v2756: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:27:50.441 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4464 8 365281 365273 0.63922 0 - 0.0299513 2026-02-23T14:27:50.542 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:27:50.660 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:27:50.761 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:27:50.759+0000 7f995ee50640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:27:50.761 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:27:50.759+0000 7f995ee50640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:27:50.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:27:50 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T14:27:50.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:27:50 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T14:27:50.873 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:27:50.873 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":243,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:26:29.596506+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:25:31.747822+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:23:36.246673+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:23:57.967252+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:27:51.441 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4465 8 365281 365273 0.639076 0 - 0.0299513 2026-02-23T14:27:52.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:27:51 trial068 ceph-mon[38805]: pgmap v2757: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:27:52.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:27:51 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3715239245' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:27:52.441 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4466 8 365281 365273 0.638933 0 - 0.0299513 2026-02-23T14:27:53.442 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4467 8 365281 365273 0.63879 0 - 0.0299513 2026-02-23T14:27:53.913 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:27:53.937 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:27:53 trial068 ceph-mon[38805]: pgmap v2758: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:27:54.030 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:27:54.129 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:27:54.127+0000 7f15627fc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:27:54.130 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:27:54.128+0000 7f15627fc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:27:54.240 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:27:54.240 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":243,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:26:29.596506+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:25:31.747822+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:23:36.246673+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:23:57.967252+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:27:54.241 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:27:53 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:27:53] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:27:54.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:27:53 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:27:53.972Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:27:54.442 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4468 8 365281 365273 0.638647 0 - 0.0299513 2026-02-23T14:27:55.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:27:54 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/816672212' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:27:55.442 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4469 8 365281 365273 0.638504 0 - 0.0299513 2026-02-23T14:27:56.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:27:55 trial068 ceph-mon[38805]: pgmap v2759: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:27:56.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:27:55 trial068 ceph-mon[38805]: 5.a scrub starts 2026-02-23T14:27:56.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:27:55 trial068 ceph-mon[38805]: 5.a scrub ok 2026-02-23T14:27:56.442 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4470 8 365281 365273 0.638362 0 - 0.0299513 2026-02-23T14:27:57.280 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:27:57.397 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:27:57.442 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4471 8 365281 365273 0.638219 0 - 0.0299513 2026-02-23T14:27:57.497 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:27:57.495+0000 7fd50cbf0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:27:57.498 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:27:57.496+0000 7fd50cbf0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:27:57.608 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:27:57.608 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":243,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:26:29.596506+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:25:31.747822+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:23:36.246673+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:23:57.967252+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:27:58.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:27:57 trial068 ceph-mon[38805]: pgmap v2760: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:27:58.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:27:57 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2311464241' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:27:58.442 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4472 8 365281 365273 0.638076 0 - 0.0299513 2026-02-23T14:27:58.731 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:27:58 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2026-02-23T14:27:59.442 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4473 8 365281 365273 0.637933 0 - 0.0299513 2026-02-23T14:28:00.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:27:59 trial068 ceph-mon[38805]: pgmap v2761: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:28:00.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:27:59 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T14:28:00.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:27:59 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T14:28:00.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:27:59 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T14:28:00.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:27:59 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T14:28:00.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:27:59 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-23T14:28:00.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:27:59 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2026-02-23T14:28:00.095 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:27:59 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T14:28:00.095 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:27:59 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2026-02-23T14:28:00.442 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4474 8 365281 365273 0.637791 0 - 0.0299513 2026-02-23T14:28:00.648 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:28:00.766 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:28:00.866 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:28:00.864+0000 7fcabcd48640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:28:00.866 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:28:00.864+0000 7fcabcd48640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:28:00.975 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:28:00.975 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":243,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:26:29.596506+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:25:31.747822+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:23:36.246673+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:23:57.967252+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:28:00.976 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:28:00 trial068 ceph-mon[38805]: pgmap v2762: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:28:00.976 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:28:00 trial068 ceph-mon[38805]: pgmap v2763: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:28:01.442 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4475 8 365281 365273 0.637648 0 - 0.0299513 2026-02-23T14:28:02.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:28:01 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2934360188' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:28:02.442 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4476 8 365281 365273 0.637506 0 - 0.0299513 2026-02-23T14:28:03.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:28:02 trial068 ceph-mon[38805]: pgmap v2764: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:28:03.442 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4477 8 365281 365273 0.637363 0 - 0.0299513 2026-02-23T14:28:04.012 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:28:04.131 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:28:04.233 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:28:04.230+0000 7fe76e85d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:28:04.233 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:28:04.231+0000 7fe76e85d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:28:04.233 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:28:03 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:28:03] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:28:04.341 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:28:04.342 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":243,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:26:29.596506+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:25:31.747822+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:23:36.246673+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:23:57.967252+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:28:04.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:28:03 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:28:03.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:28:04.443 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4478 8 365281 365273 0.637221 0 - 0.0299513 2026-02-23T14:28:05.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:28:04 trial068 ceph-mon[38805]: pgmap v2765: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:28:05.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:28:04 trial068 ceph-mon[38805]: 5.8 scrub starts 2026-02-23T14:28:05.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:28:04 trial068 ceph-mon[38805]: 5.8 scrub ok 2026-02-23T14:28:05.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:28:04 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3629969134' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:28:05.443 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4479 8 365281 365273 0.637079 0 - 0.0299513 2026-02-23T14:28:06.443 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T14:28:06.442367+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T14:28:06.443 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T14:28:06.443 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4480 8 365281 365273 0.636937 0 - 0.0299513 2026-02-23T14:28:07.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:28:06 trial068 ceph-mon[38805]: pgmap v2766: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:28:07.380 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:28:07.443 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4481 8 365281 365273 0.636794 0 - 0.0299513 2026-02-23T14:28:07.496 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:28:07.600 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:28:07.598+0000 7f66f707c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:28:07.600 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:28:07.598+0000 7f66f707c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:28:07.710 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:28:07.710 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":243,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:26:29.596506+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:25:31.747822+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:23:36.246673+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:23:57.967252+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:28:08.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:28:07 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/806460662' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:28:08.443 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4482 8 365281 365273 0.636652 0 - 0.0299513 2026-02-23T14:28:09.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:28:08 trial068 ceph-mon[38805]: pgmap v2767: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:28:09.443 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4483 8 365281 365273 0.63651 0 - 0.0299513 2026-02-23T14:28:10.443 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4484 8 365281 365273 0.636368 0 - 0.0299513 2026-02-23T14:28:10.748 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:28:10.865 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:28:10.975 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:28:10.972+0000 7ff81c08a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:28:10.975 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:28:10.973+0000 7ff81c08a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:28:11.083 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:28:11.083 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":243,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:26:29.596506+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:25:31.747822+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:23:36.246673+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:23:57.967252+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:28:11.084 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:28:10 trial068 ceph-mon[38805]: pgmap v2768: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:28:11.443 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4485 8 365281 365273 0.636227 0 - 0.0299513 2026-02-23T14:28:12.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:28:11 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2111694067' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:28:12.443 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4486 8 365281 365273 0.636085 0 - 0.0299513 2026-02-23T14:28:13.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:28:12 trial068 ceph-mon[38805]: pgmap v2769: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:28:13.443 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4487 8 365281 365273 0.635943 0 - 0.0299513 2026-02-23T14:28:14.119 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:28:14.239 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:28:14.261 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:28:13 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:28:13] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:28:14.342 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:28:14.340+0000 7f151f2ee640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:28:14.342 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:28:14.340+0000 7f151f2ee640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:28:14.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:28:13 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:28:13.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:28:14.443 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4488 8 365281 365273 0.635801 0 - 0.0299513 2026-02-23T14:28:14.452 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:28:14.453 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":243,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:26:29.596506+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:25:31.747822+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:23:36.246673+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:23:57.967252+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:28:15.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:28:14 trial068 ceph-mon[38805]: pgmap v2770: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:28:15.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:28:14 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3363830987' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:28:15.443 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4489 8 365281 365273 0.63566 0 - 0.0299513 2026-02-23T14:28:16.443 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4490 8 365281 365273 0.635518 0 - 0.0299513 2026-02-23T14:28:17.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:28:16 trial068 ceph-mon[38805]: pgmap v2771: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:28:17.443 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4491 8 365281 365273 0.635377 0 - 0.0299513 2026-02-23T14:28:17.491 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:28:17.608 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:28:17.712 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:28:17.710+0000 7fe433a92640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:28:17.712 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:28:17.710+0000 7fe433a92640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:28:17.823 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:28:17.823 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":243,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:26:29.596506+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:25:31.747822+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:23:36.246673+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:23:57.967252+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:28:18.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:28:17 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/4129568646' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:28:18.444 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4492 8 365281 365273 0.635235 0 - 0.0299513 2026-02-23T14:28:19.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:28:18 trial068 ceph-mon[38805]: pgmap v2772: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:28:19.444 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4493 8 365281 365273 0.635094 0 - 0.0299513 2026-02-23T14:28:20.444 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4494 8 365281 365273 0.634952 0 - 0.0299513 2026-02-23T14:28:20.860 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:28:20.979 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:28:21.079 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:28:21.077+0000 7fd3ad54e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:28:21.080 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:28:21.077+0000 7fd3ad54e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:28:21.080 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:28:20 trial068 ceph-mon[38805]: pgmap v2773: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:28:21.189 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:28:21.189 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":243,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:26:29.596506+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:25:31.747822+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:23:36.246673+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:23:57.967252+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:28:21.444 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4495 8 365281 365273 0.634811 0 - 0.0299513 2026-02-23T14:28:22.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:28:21 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1479057821' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:28:22.444 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4496 8 365281 365273 0.63467 0 - 0.0299513 2026-02-23T14:28:23.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:28:22 trial068 ceph-mon[38805]: pgmap v2774: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:28:23.444 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4497 8 365281 365273 0.634529 0 - 0.0299513 2026-02-23T14:28:24.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:28:23 trial068 ceph-mon[38805]: pgmap v2775: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:28:24.094 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:28:23 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:28:23] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:28:24.241 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:28:24.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:28:23 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:28:23.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:28:24.359 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:28:24.444 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4498 8 365281 365273 0.634388 0 - 0.0299513 2026-02-23T14:28:24.456 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:28:24.454+0000 7fdb7b178640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:28:24.456 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:28:24.454+0000 7fdb7b178640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:28:24.566 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:28:24.566 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":243,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:26:29.596506+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:25:31.747822+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:23:36.246673+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:23:57.967252+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:28:25.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:28:24 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1728873795' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:28:25.444 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4499 8 365281 365273 0.634247 0 - 0.0299513 2026-02-23T14:28:26.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:28:25 trial068 ceph-mon[38805]: 5.e scrub starts 2026-02-23T14:28:26.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:28:25 trial068 ceph-mon[38805]: 5.e scrub ok 2026-02-23T14:28:26.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:28:25 trial068 ceph-mon[38805]: pgmap v2776: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:28:26.444 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T14:28:26.443918+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T14:28:26.444 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T14:28:26.444 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4500 8 365281 365273 0.634106 0 - 0.0299513 2026-02-23T14:28:27.444 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4501 8 365281 365273 0.633965 0 - 0.0299513 2026-02-23T14:28:27.605 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:28:27.721 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:28:27.818 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:28:27.816+0000 7f1cb4988640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:28:27.818 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:28:27.816+0000 7f1cb4988640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:28:27.927 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:28:27.927 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":243,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:26:29.596506+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:25:31.747822+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:23:36.246673+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:23:57.967252+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:28:28.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:28:28 trial068 ceph-mon[38805]: pgmap v2777: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:28:28.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:28:28 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1991420569' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:28:28.444 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4502 8 365281 365273 0.633824 0 - 0.0299513 2026-02-23T14:28:29.444 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4503 8 365281 365273 0.633683 0 - 0.0299513 2026-02-23T14:28:30.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:28:30 trial068 ceph-mon[38805]: pgmap v2778: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:28:30.444 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4504 8 365281 365273 0.633543 0 - 0.0299513 2026-02-23T14:28:30.964 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:28:31.082 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:28:31.188 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:28:31.185+0000 7f1f6f943640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:28:31.188 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:28:31.186+0000 7f1f6f943640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:28:31.297 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:28:31.298 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":243,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:26:29.596506+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:25:31.747822+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:23:36.246673+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:23:57.967252+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:28:31.445 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4505 8 365281 365273 0.633402 0 - 0.0299513 2026-02-23T14:28:32.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:28:32 trial068 ceph-mon[38805]: pgmap v2779: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:28:32.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:28:32 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2300739032' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:28:32.445 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4506 8 365281 365273 0.633261 0 - 0.0299513 2026-02-23T14:28:33.445 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4507 8 365281 365273 0.633121 0 - 0.0299513 2026-02-23T14:28:34.266 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:28:33 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:28:33.974Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:28:34.336 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:28:34.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:28:34 trial068 ceph-mon[38805]: pgmap v2780: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:28:34.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:28:34 trial068 ceph-mon[38805]: 5.5 scrub starts 2026-02-23T14:28:34.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:28:34 trial068 ceph-mon[38805]: 5.5 scrub ok 2026-02-23T14:28:34.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:28:33 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:28:33] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:28:34.445 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4508 8 365281 365273 0.63298 0 - 0.0299513 2026-02-23T14:28:34.457 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:28:34.565 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:28:34.563+0000 7f14202b2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:28:34.566 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:28:34.563+0000 7f14202b2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:28:34.675 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:28:34.676 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":243,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:26:29.596506+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:25:31.747822+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:23:36.246673+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:23:57.967252+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:28:35.445 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4509 8 365281 365273 0.63284 0 - 0.0299513 2026-02-23T14:28:36.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:28:35 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1671762169' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:28:36.445 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4510 8 365281 365273 0.6327 0 - 0.0299513 2026-02-23T14:28:37.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:28:36 trial068 ceph-mon[38805]: pgmap v2781: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:28:37.445 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4511 8 365281 365273 0.632559 0 - 0.0299513 2026-02-23T14:28:37.714 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:28:37.833 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:28:37.940 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:28:37.938+0000 7fa3c2939640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:28:37.941 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:28:37.938+0000 7fa3c2939640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:28:38.050 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:28:38.050 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":243,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:26:29.596506+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:25:31.747822+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:23:36.246673+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:23:57.967252+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:28:38.445 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4512 8 365281 365273 0.632419 0 - 0.0299513 2026-02-23T14:28:39.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:28:38 trial068 ceph-mon[38805]: pgmap v2782: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:28:39.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:28:38 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2022377398' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:28:39.445 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4513 8 365281 365273 0.632279 0 - 0.0299513 2026-02-23T14:28:40.445 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4514 8 365281 365273 0.632139 0 - 0.0299513 2026-02-23T14:28:41.089 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:28:41.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:28:40 trial068 ceph-mon[38805]: pgmap v2783: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:28:41.209 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:28:41.306 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:28:41.304+0000 7effa6b32640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:28:41.306 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:28:41.304+0000 7effa6b32640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:28:41.419 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:28:41.419 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":243,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:26:29.596506+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:25:31.747822+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:23:36.246673+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:23:57.967252+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:28:41.445 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4515 8 365281 365273 0.631999 0 - 0.0299513 2026-02-23T14:28:41.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:28:41 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/4135745446' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:28:42.445 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4516 8 365281 365273 0.631859 0 - 0.0299513 2026-02-23T14:28:43.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:28:42 trial068 ceph-mon[38805]: pgmap v2784: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:28:43.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:28:42 trial068 ceph-mon[38805]: 5.7 scrub starts 2026-02-23T14:28:43.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:28:42 trial068 ceph-mon[38805]: 5.7 scrub ok 2026-02-23T14:28:43.445 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4517 8 365281 365273 0.631719 0 - 0.0299513 2026-02-23T14:28:44.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:28:43 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:28:43] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:28:44.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:28:43 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:28:43.974Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:28:44.445 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4518 8 365281 365273 0.631579 0 - 0.0299513 2026-02-23T14:28:44.457 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:28:44.574 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:28:44.671 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:28:44.669+0000 7fbf146a7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:28:44.671 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:28:44.669+0000 7fbf146a7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:28:44.782 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:28:44.782 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":243,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:26:29.596506+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:25:31.747822+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:23:36.246673+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:23:57.967252+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:28:44.782 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:28:44 trial068 ceph-mon[38805]: pgmap v2785: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:28:45.446 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4519 8 365281 365273 0.63144 0 - 0.0299513 2026-02-23T14:28:46.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:28:45 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2046915050' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:28:46.446 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T14:28:46.445331+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T14:28:46.446 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T14:28:46.446 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4520 8 365281 365273 0.6313 0 - 0.0299513 2026-02-23T14:28:47.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:28:46 trial068 ceph-mon[38805]: pgmap v2786: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:28:47.446 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4521 8 365281 365273 0.63116 0 - 0.0299513 2026-02-23T14:28:47.819 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:28:47.935 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:28:48.037 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:28:48.035+0000 7f492f716640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:28:48.037 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:28:48.035+0000 7f492f716640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:28:48.147 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:28:48.147 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":243,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:26:29.596506+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:25:31.747822+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:23:36.246673+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:23:57.967252+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:28:48.446 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4522 8 365281 365273 0.631021 0 - 0.0299513 2026-02-23T14:28:49.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:28:48 trial068 ceph-mon[38805]: pgmap v2787: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:28:49.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:28:48 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3742179242' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:28:49.446 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4523 8 365281 365273 0.630881 0 - 0.0299513 2026-02-23T14:28:50.446 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4524 8 365281 365273 0.630742 0 - 0.0299513 2026-02-23T14:28:51.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:28:50 trial068 ceph-mon[38805]: pgmap v2788: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:28:51.191 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:28:51.309 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:28:51.409 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:28:51.407+0000 7fee9e3a0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:28:51.409 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:28:51.407+0000 7fee9e3a0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:28:51.446 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4525 8 365281 365273 0.630602 0 - 0.0299513 2026-02-23T14:28:51.517 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:28:51.518 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":243,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:26:29.596506+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:25:31.747822+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:23:36.246673+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:23:57.967252+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:28:51.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:28:51 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3990793270' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:28:52.446 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4526 8 365281 365273 0.630463 0 - 0.0299513 2026-02-23T14:28:53.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:28:52 trial068 ceph-mon[38805]: pgmap v2789: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:28:53.446 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4527 8 365281 365273 0.630324 0 - 0.0299513 2026-02-23T14:28:54.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:28:53 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:28:53] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:28:54.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:28:53 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:28:53.974Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:28:54.446 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4528 8 365281 365273 0.630185 0 - 0.0299513 2026-02-23T14:28:54.555 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:28:54.675 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:28:54.777 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:28:54.775+0000 7f95e34fc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:28:54.778 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:28:54.775+0000 7f95e34fc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:28:54.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:28:54 trial068 ceph-mon[38805]: pgmap v2790: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:28:54.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:28:54 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T14:28:54.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:28:54 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T14:28:54.887 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:28:54.887 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":243,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:26:29.596506+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:25:31.747822+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:23:36.246673+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:23:57.967252+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:28:55.447 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4529 8 365281 365273 0.630045 0 - 0.0299513 2026-02-23T14:28:56.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:28:55 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T14:28:56.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:28:55 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T14:28:56.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:28:55 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/623745665' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:28:56.447 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4530 8 365281 365273 0.629906 0 - 0.0299513 2026-02-23T14:28:57.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:28:56 trial068 ceph-mon[38805]: pgmap v2791: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:28:57.447 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4531 8 365281 365273 0.629767 0 - 0.0299513 2026-02-23T14:28:57.925 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:28:58.043 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:28:58.152 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:28:58.150+0000 7fe44d441640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:28:58.153 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:28:58.150+0000 7fe44d441640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:28:58.260 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:28:58.261 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":243,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:26:29.596506+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:25:31.747822+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:23:36.246673+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:23:57.967252+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:28:58.447 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4532 8 365281 365273 0.629628 0 - 0.0299513 2026-02-23T14:28:59.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:28:58 trial068 ceph-mon[38805]: pgmap v2792: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:28:59.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:28:58 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/4067113294' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:28:59.447 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4533 8 365281 365273 0.629489 0 - 0.0299513 2026-02-23T14:29:00.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:28:59 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2026-02-23T14:29:00.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:28:59 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-23T14:29:00.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:28:59 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2026-02-23T14:29:00.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:28:59 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T14:29:00.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:28:59 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2026-02-23T14:29:00.447 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4534 8 365281 365273 0.629351 0 - 0.0299513 2026-02-23T14:29:01.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:29:00 trial068 ceph-mon[38805]: pgmap v2793: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:29:01.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:29:00 trial068 ceph-mon[38805]: pgmap v2794: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:29:01.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:29:00 trial068 ceph-mon[38805]: pgmap v2795: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:29:01.298 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:29:01.415 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:29:01.447 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4535 8 365281 365273 0.629212 0 - 0.0299513 2026-02-23T14:29:01.514 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:29:01.512+0000 7f04396a9640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:29:01.515 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:29:01.512+0000 7f04396a9640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:29:01.623 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:29:01.623 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":244,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:29:00.639595+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:25:31.747822+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:27:49.594833+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:23:57.967252+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:29:02.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:29:01 trial068 ceph-mon[38805]: osdmap e244: 4 total, 3 up, 4 in 2026-02-23T14:29:02.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:29:01 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T14:29:02.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:29:01 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T14:29:02.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:29:01 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2909395553' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:29:02.447 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4536 8 365281 365273 0.629073 0 - 0.0299513 2026-02-23T14:29:03.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:29:02 trial068 ceph-mon[38805]: pgmap v2797: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:29:03.447 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4537 8 365281 365273 0.628935 0 - 0.0299513 2026-02-23T14:29:04.266 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:29:03 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:29:03.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:29:04.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:29:03 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:29:03] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:29:04.447 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4538 8 365281 365273 0.628796 0 - 0.0299513 2026-02-23T14:29:04.661 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:29:04.780 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:29:04.878 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:29:04.876+0000 7f1ba1895640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:29:04.878 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:29:04.876+0000 7f1ba1895640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:29:04.987 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:29:04.988 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":245,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:29:04.616780+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:25:31.747822+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:27:49.594833+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:23:57.967252+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:29:04.988 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:29:04 trial068 ceph-mon[38805]: pgmap v2798: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:29:04.988 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:29:04 trial068 ceph-mon[38805]: osdmap e245: 4 total, 3 up, 4 in 2026-02-23T14:29:05.447 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4539 8 365281 365273 0.628657 0 - 0.0299513 2026-02-23T14:29:06.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:29:05 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2171104489' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:29:06.447 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T14:29:06.446937+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T14:29:06.448 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T14:29:06.448 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4540 8 365281 365273 0.628519 0 - 0.0299513 2026-02-23T14:29:07.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:29:06 trial068 ceph-mon[38805]: pgmap v2800: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:29:07.447 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4541 8 365281 365273 0.628381 0 - 0.0299513 2026-02-23T14:29:08.027 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:29:08.146 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:29:08.250 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:29:08.248+0000 7f0606a29640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:29:08.250 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:29:08.248+0000 7f0606a29640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:29:08.359 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:29:08.359 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":245,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:29:04.616780+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:25:31.747822+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:27:49.594833+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:23:57.967252+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:29:08.447 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4542 8 365281 365273 0.628242 0 - 0.0299513 2026-02-23T14:29:09.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:29:08 trial068 ceph-mon[38805]: pgmap v2801: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:29:09.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:29:08 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2779773784' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:29:09.447 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4543 8 365281 365273 0.628104 0 - 0.0299513 2026-02-23T14:29:10.448 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4544 8 365281 365273 0.627966 0 - 0.0299513 2026-02-23T14:29:11.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:29:10 trial068 ceph-mon[38805]: pgmap v2802: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:29:11.396 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:29:11.448 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4545 8 365281 365273 0.627827 0 - 0.0299513 2026-02-23T14:29:11.514 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:29:11.617 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:29:11.615+0000 7f3034329640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:29:11.617 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:29:11.615+0000 7f3034329640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:29:11.728 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:29:11.728 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":245,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:29:04.616780+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:25:31.747822+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:27:49.594833+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:23:57.967252+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:29:12.448 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4546 8 365281 365273 0.627689 0 - 0.0299513 2026-02-23T14:29:13.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:29:12 trial068 ceph-mon[38805]: pgmap v2803: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:29:13.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:29:12 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2742245307' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:29:13.448 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4547 8 365281 365273 0.627551 0 - 0.0299513 2026-02-23T14:29:14.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:29:13 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:29:13] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:29:14.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:29:13 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:29:13.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:29:14.448 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4548 8 365281 365273 0.627413 0 - 0.0299513 2026-02-23T14:29:14.766 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:29:14.886 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:29:14.909 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:29:14 trial068 ceph-mon[38805]: pgmap v2804: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:29:14.995 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:29:14.993+0000 7f809143a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:29:14.995 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:29:14.993+0000 7f809143a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:29:15.104 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:29:15.104 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":245,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:29:04.616780+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:25:31.747822+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:27:49.594833+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:23:57.967252+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:29:15.448 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4549 8 365281 365273 0.627275 0 - 0.0299513 2026-02-23T14:29:16.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:29:15 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2754481405' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:29:16.448 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4550 8 365281 365273 0.627138 0 - 0.0299513 2026-02-23T14:29:17.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:29:16 trial068 ceph-mon[38805]: pgmap v2805: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:29:17.448 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4551 8 365281 365273 0.627 0 - 0.0299513 2026-02-23T14:29:18.141 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:29:18.260 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:29:18.357 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:29:18.355+0000 7ff43c0fe640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:29:18.358 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:29:18.356+0000 7ff43c0fe640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:29:18.448 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4552 8 365281 365273 0.626862 0 - 0.0299513 2026-02-23T14:29:18.466 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:29:18.466 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":245,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:29:04.616780+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:25:31.747822+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:27:49.594833+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:23:57.967252+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:29:18.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:29:18 trial068 ceph-mon[38805]: pgmap v2806: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:29:18.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:29:18 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/655298520' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:29:19.448 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4553 8 365281 365273 0.626724 0 - 0.0299513 2026-02-23T14:29:20.448 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4554 8 365281 365273 0.626587 0 - 0.0299513 2026-02-23T14:29:21.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:29:20 trial068 ceph-mon[38805]: pgmap v2807: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:29:21.448 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4555 8 365281 365273 0.626449 0 - 0.0299513 2026-02-23T14:29:21.503 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:29:21.621 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:29:21.719 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:29:21.716+0000 7fa166883640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:29:21.719 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:29:21.717+0000 7fa166883640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:29:21.826 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:29:21.826 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":245,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:29:04.616780+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:25:31.747822+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:27:49.594833+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:23:57.967252+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:29:22.448 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4556 8 365281 365273 0.626312 0 - 0.0299513 2026-02-23T14:29:23.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:29:22 trial068 ceph-mon[38805]: pgmap v2808: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:29:23.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:29:22 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/287812987' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:29:23.448 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4557 8 365281 365273 0.626174 0 - 0.0299513 2026-02-23T14:29:24.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:29:23 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:29:23] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:29:24.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:29:23 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:29:23.974Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:29:24.448 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4558 8 365281 365273 0.626037 0 - 0.0299513 2026-02-23T14:29:24.864 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:29:24.981 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:29:25.005 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:29:24 trial068 ceph-mon[38805]: pgmap v2809: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:29:25.082 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:29:25.079+0000 7f38129c7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:29:25.082 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:29:25.079+0000 7f38129c7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:29:25.190 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:29:25.190 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":245,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:29:04.616780+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:25:31.747822+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:27:49.594833+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:23:57.967252+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:29:25.449 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4559 8 365281 365273 0.6259 0 - 0.0299513 2026-02-23T14:29:26.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:29:25 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3753102772' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:29:26.449 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T14:29:26.448288+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T14:29:26.449 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T14:29:26.449 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4560 8 365281 365273 0.625762 0 - 0.0299513 2026-02-23T14:29:27.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:29:26 trial068 ceph-mon[38805]: pgmap v2810: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:29:27.449 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4561 8 365281 365273 0.625625 0 - 0.0299513 2026-02-23T14:29:28.227 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:29:28.346 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:29:28.443 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:29:28.441+0000 7f4d629c1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:29:28.444 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:29:28.441+0000 7f4d629c1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:29:28.449 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4562 8 365281 365273 0.625488 0 - 0.0299513 2026-02-23T14:29:28.552 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:29:28.552 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":245,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:29:04.616780+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:25:31.747822+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:27:49.594833+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:23:57.967252+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:29:28.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:29:28 trial068 ceph-mon[38805]: pgmap v2811: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:29:28.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:29:28 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/773511931' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:29:29.449 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4563 8 365281 365273 0.625351 0 - 0.0299513 2026-02-23T14:29:30.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:29:29 trial068 ceph-mon[38805]: 5.a scrub starts 2026-02-23T14:29:30.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:29:29 trial068 ceph-mon[38805]: 5.a scrub ok 2026-02-23T14:29:30.449 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4564 8 365281 365273 0.625214 0 - 0.0299513 2026-02-23T14:29:31.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:29:30 trial068 ceph-mon[38805]: pgmap v2812: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:29:31.449 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4565 8 365281 365273 0.625077 0 - 0.0299513 2026-02-23T14:29:31.604 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:29:31.724 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:29:31.828 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:29:31.826+0000 7f3d6439a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:29:31.828 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:29:31.826+0000 7f3d6439a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:29:31.938 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:29:31.939 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":245,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:29:04.616780+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:25:31.747822+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:27:49.594833+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:23:57.967252+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:29:32.449 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4566 8 365281 365273 0.62494 0 - 0.0299513 2026-02-23T14:29:33.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:29:32 trial068 ceph-mon[38805]: pgmap v2813: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:29:33.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:29:32 trial068 ceph-mon[38805]: 5.e scrub starts 2026-02-23T14:29:33.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:29:32 trial068 ceph-mon[38805]: 5.e scrub ok 2026-02-23T14:29:33.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:29:32 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1737579282' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:29:33.449 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4567 8 365281 365273 0.624803 0 - 0.0299513 2026-02-23T14:29:34.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:29:33 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:29:33] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:29:34.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:29:33 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:29:33.974Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:29:34.449 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4568 8 365281 365273 0.624666 0 - 0.0299513 2026-02-23T14:29:34.977 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:29:35.001 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:29:34 trial068 ceph-mon[38805]: pgmap v2814: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:29:35.094 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:29:35.192 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:29:35.190+0000 7ffac82c5640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:29:35.192 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:29:35.190+0000 7ffac82c5640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:29:35.301 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:29:35.301 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":245,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:29:04.616780+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:25:31.747822+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:27:49.594833+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:23:57.967252+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:29:35.449 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4569 8 365281 365273 0.62453 0 - 0.0299513 2026-02-23T14:29:36.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:29:35 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2625211862' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:29:36.449 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4570 8 365281 365273 0.624393 0 - 0.0299513 2026-02-23T14:29:37.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:29:36 trial068 ceph-mon[38805]: pgmap v2815: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:29:37.449 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4571 8 365281 365273 0.624256 0 - 0.0299513 2026-02-23T14:29:38.341 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:29:38.450 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4572 8 365281 365273 0.62412 0 - 0.0299513 2026-02-23T14:29:38.459 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:29:38.559 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:29:38.556+0000 7f78eb843640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:29:38.559 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:29:38.557+0000 7f78eb843640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:29:38.674 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:29:38.674 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":245,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:29:04.616780+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:25:31.747822+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:27:49.594833+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:23:57.967252+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:29:39.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:29:38 trial068 ceph-mon[38805]: pgmap v2816: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:29:39.449 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4573 8 365281 365273 0.623983 0 - 0.0299513 2026-02-23T14:29:40.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:29:39 trial068 ceph-mon[38805]: 5.8 scrub starts 2026-02-23T14:29:40.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:29:39 trial068 ceph-mon[38805]: 5.8 scrub ok 2026-02-23T14:29:40.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:29:39 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2906655876' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:29:40.450 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4574 8 365281 365273 0.623847 0 - 0.0299513 2026-02-23T14:29:41.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:29:40 trial068 ceph-mon[38805]: pgmap v2817: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:29:41.450 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4575 8 365281 365273 0.623711 0 - 0.0299513 2026-02-23T14:29:41.715 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:29:41.835 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:29:41.941 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:29:41.939+0000 7f7cedb9a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:29:41.941 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:29:41.939+0000 7f7cedb9a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:29:42.049 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:29:42.050 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":245,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:29:04.616780+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:25:31.747822+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:27:49.594833+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:23:57.967252+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:29:42.450 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4576 8 365281 365273 0.623574 0 - 0.0299513 2026-02-23T14:29:43.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:29:42 trial068 ceph-mon[38805]: pgmap v2818: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:29:43.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:29:42 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3297360539' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:29:43.450 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4577 8 365281 365273 0.623438 0 - 0.0299513 2026-02-23T14:29:44.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:29:43 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:29:43] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:29:44.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:29:43 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:29:43.974Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:29:44.450 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4578 8 365281 365273 0.623302 0 - 0.0299513 2026-02-23T14:29:45.085 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:29:45.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:29:44 trial068 ceph-mon[38805]: pgmap v2819: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:29:45.205 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:29:45.302 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:29:45.300+0000 7f2d81c51640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:29:45.302 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:29:45.300+0000 7f2d81c51640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:29:45.410 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:29:45.411 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":245,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:29:04.616780+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:25:31.747822+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:27:49.594833+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:23:57.967252+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:29:45.450 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4579 8 365281 365273 0.623166 0 - 0.0299513 2026-02-23T14:29:46.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:29:45 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3037139188' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:29:46.450 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T14:29:46.449644+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T14:29:46.450 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T14:29:46.450 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4580 8 365281 365273 0.62303 0 - 0.0299513 2026-02-23T14:29:47.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:29:46 trial068 ceph-mon[38805]: pgmap v2820: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:29:47.450 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4581 8 365281 365273 0.622894 0 - 0.0299513 2026-02-23T14:29:48.449 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:29:48.450 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4582 8 365281 365273 0.622758 0 - 0.0299513 2026-02-23T14:29:48.569 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:29:48.670 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:29:48.667+0000 7f2eda64f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:29:48.670 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:29:48.668+0000 7f2eda64f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:29:48.780 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:29:48.780 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":245,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:29:04.616780+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:25:31.747822+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:27:49.594833+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:23:57.967252+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:29:48.780 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:29:48 trial068 ceph-mon[38805]: pgmap v2821: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:29:49.450 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4583 8 365281 365273 0.622622 0 - 0.0299513 2026-02-23T14:29:50.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:29:49 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3812336877' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:29:50.450 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4584 8 365281 365273 0.622486 0 - 0.0299513 2026-02-23T14:29:51.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:29:50 trial068 ceph-mon[38805]: pgmap v2822: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:29:51.450 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4585 8 365281 365273 0.62235 0 - 0.0299513 2026-02-23T14:29:51.816 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:29:51.934 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:29:52.035 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:29:52.033+0000 7fb62c9a5640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:29:52.036 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:29:52.034+0000 7fb62c9a5640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:29:52.145 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:29:52.145 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":245,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:29:04.616780+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:25:31.747822+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:27:49.594833+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:23:57.967252+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:29:52.450 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4586 8 365281 365273 0.622215 0 - 0.0299513 2026-02-23T14:29:53.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:29:52 trial068 ceph-mon[38805]: pgmap v2823: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:29:53.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:29:52 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/390647437' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:29:53.450 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4587 8 365281 365273 0.622079 0 - 0.0299513 2026-02-23T14:29:54.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:29:53 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:29:53] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:29:54.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:29:53 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:29:53.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:29:54.450 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4588 8 365281 365273 0.621943 0 - 0.0299513 2026-02-23T14:29:55.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:29:54 trial068 ceph-mon[38805]: pgmap v2824: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:29:55.183 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:29:55.301 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:29:55.401 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:29:55.399+0000 7ff635fd2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:29:55.402 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:29:55.400+0000 7ff635fd2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:29:55.450 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4589 8 365281 365273 0.621808 0 - 0.0299513 2026-02-23T14:29:55.511 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:29:55.511 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":245,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:29:04.616780+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:25:31.747822+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:27:49.594833+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:23:57.967252+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:29:55.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:29:55 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3045462709' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:29:56.451 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4590 8 365281 365273 0.621672 0 - 0.0299513 2026-02-23T14:29:57.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:29:56 trial068 ceph-mon[38805]: pgmap v2825: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:29:57.451 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4591 8 365281 365273 0.621537 0 - 0.0299513 2026-02-23T14:29:58.451 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4592 8 365281 365273 0.621402 0 - 0.0299513 2026-02-23T14:29:58.550 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:29:58.666 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:29:58.766 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:29:58.764+0000 7fbe9d8ba640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:29:58.766 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:29:58.764+0000 7fbe9d8ba640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:29:58.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:29:58 trial068 ceph-mon[38805]: pgmap v2826: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:29:58.875 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:29:58.875 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":245,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:29:04.616780+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:25:31.747822+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:27:49.594833+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:23:57.967252+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:29:59.451 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4593 8 365281 365273 0.621266 0 - 0.0299513 2026-02-23T14:30:00.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:29:59 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2537970342' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:30:00.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:29:59 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2026-02-23T14:30:00.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:29:59 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-23T14:30:00.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:29:59 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2026-02-23T14:30:00.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:29:59 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T14:30:00.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:29:59 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2026-02-23T14:30:00.451 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4594 8 365281 365273 0.621131 0 - 0.0299513 2026-02-23T14:30:01.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:30:00 trial068 ceph-mon[38805]: pgmap v2827: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:30:01.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:30:00 trial068 ceph-mon[38805]: pgmap v2828: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:30:01.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:30:00 trial068 ceph-mon[38805]: pgmap v2829: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:30:01.451 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4595 8 365281 365273 0.620996 0 - 0.0299513 2026-02-23T14:30:01.915 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:30:02.033 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:30:02.136 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:30:02.134+0000 7ff2d4ee7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:30:02.136 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:30:02.134+0000 7ff2d4ee7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:30:02.247 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:30:02.247 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":245,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:29:04.616780+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:25:31.747822+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:27:49.594833+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:23:57.967252+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:30:02.451 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4596 8 365281 365273 0.620861 0 - 0.0299513 2026-02-23T14:30:03.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:30:02 trial068 ceph-mon[38805]: pgmap v2830: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:30:03.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:30:02 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T14:30:03.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:30:02 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T14:30:03.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:30:02 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/453844448' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:30:03.451 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4597 8 365281 365273 0.620726 0 - 0.0299513 2026-02-23T14:30:03.974 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:30:03 trial068 ceph-mon[38805]: 5.5 scrub starts 2026-02-23T14:30:03.974 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:30:03 trial068 ceph-mon[38805]: 5.5 scrub ok 2026-02-23T14:30:04.266 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:30:03 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:30:03.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:30:04.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:30:03 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:30:03] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:30:04.451 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4598 8 365281 365273 0.620591 0 - 0.0299513 2026-02-23T14:30:05.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:30:04 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T14:30:05.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:30:04 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T14:30:05.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:30:04 trial068 ceph-mon[38805]: pgmap v2831: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:30:05.289 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:30:05.409 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:30:05.451 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4599 8 365281 365273 0.620456 0 - 0.0299513 2026-02-23T14:30:05.509 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:30:05.507+0000 7f675bbff640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:30:05.510 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:30:05.508+0000 7f675bbff640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:30:05.618 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:30:05.618 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":245,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:29:04.616780+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:25:31.747822+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:27:49.594833+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:23:57.967252+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:30:06.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:30:05 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3161549071' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:30:06.451 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T14:30:06.451013+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T14:30:06.452 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T14:30:06.452 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4600 8 365281 365273 0.620321 0 - 0.0299513 2026-02-23T14:30:07.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:30:06 trial068 ceph-mon[38805]: pgmap v2832: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:30:07.451 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4601 8 365281 365273 0.620186 0 - 0.0299513 2026-02-23T14:30:08.452 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4602 8 365281 365273 0.620051 0 - 0.0299513 2026-02-23T14:30:08.657 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:30:08.776 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:30:08.878 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:30:08.875+0000 7fa3aecbe640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:30:08.878 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:30:08.876+0000 7fa3aecbe640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:30:08.988 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:30:08.988 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":245,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:29:04.616780+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:25:31.747822+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:27:49.594833+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:23:57.967252+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:30:08.989 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:30:08 trial068 ceph-mon[38805]: pgmap v2833: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:30:09.452 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4603 8 365281 365273 0.619917 0 - 0.0299513 2026-02-23T14:30:10.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:30:09 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1180393913' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:30:10.452 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4604 8 365281 365273 0.619782 0 - 0.0299513 2026-02-23T14:30:11.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:30:10 trial068 ceph-mon[38805]: pgmap v2834: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:30:11.099 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:30:10 trial068 ceph-mon[38805]: 5.7 scrub starts 2026-02-23T14:30:11.099 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:30:10 trial068 ceph-mon[38805]: 5.7 scrub ok 2026-02-23T14:30:11.452 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4605 8 365281 365273 0.619647 0 - 0.0299513 2026-02-23T14:30:12.026 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:30:12.145 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:30:12.248 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:30:12.245+0000 7f84f9462640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:30:12.248 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:30:12.246+0000 7f84f9462640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:30:12.356 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:30:12.356 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":245,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:29:04.616780+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:25:31.747822+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:27:49.594833+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:23:57.967252+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:30:12.452 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4606 8 365281 365273 0.619513 0 - 0.0299513 2026-02-23T14:30:13.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:30:12 trial068 ceph-mon[38805]: pgmap v2835: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:30:13.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:30:12 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1697774987' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:30:13.452 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4607 8 365281 365273 0.619378 0 - 0.0299513 2026-02-23T14:30:14.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:30:13 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:30:13] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:30:14.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:30:13 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:30:13.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:30:14.452 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4608 8 365281 365273 0.619244 0 - 0.0299513 2026-02-23T14:30:15.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:30:14 trial068 ceph-mon[38805]: pgmap v2836: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:30:15.393 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:30:15.452 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4609 8 365281 365273 0.61911 0 - 0.0299513 2026-02-23T14:30:15.510 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:30:15.617 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:30:15.615+0000 7f30203ee640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:30:15.618 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:30:15.615+0000 7f30203ee640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:30:15.728 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:30:15.728 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":245,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:29:04.616780+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:25:31.747822+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:27:49.594833+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:23:57.967252+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:30:16.452 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4610 8 365281 365273 0.618975 0 - 0.0299513 2026-02-23T14:30:17.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:30:16 trial068 ceph-mon[38805]: pgmap v2837: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:30:17.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:30:16 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/354875697' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:30:17.452 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4611 8 365281 365273 0.618841 0 - 0.0299513 2026-02-23T14:30:18.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:30:17 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T14:30:18.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:30:17 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T14:30:18.452 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4612 8 365281 365273 0.618707 0 - 0.0299513 2026-02-23T14:30:18.768 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:30:18.886 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:30:18.984 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:30:18.982+0000 7efe76cf0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:30:18.985 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:30:18.982+0000 7efe76cf0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:30:18.985 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:30:18 trial068 ceph-mon[38805]: pgmap v2838: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:30:18.985 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:30:18 trial068 ceph-mon[38805]: osdmap e246: 4 total, 3 up, 4 in 2026-02-23T14:30:19.093 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:30:19.094 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":246,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:30:17.705661+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:25:31.747822+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:27:49.594833+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:30:03.417272+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:30:19.452 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4613 8 365281 365273 0.618573 0 - 0.0299513 2026-02-23T14:30:20.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:30:19 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1116214786' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:30:20.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:30:19 trial068 ceph-mon[38805]: osdmap e247: 4 total, 3 up, 4 in 2026-02-23T14:30:20.452 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4614 8 365281 365273 0.618439 0 - 0.0299513 2026-02-23T14:30:21.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:30:20 trial068 ceph-mon[38805]: pgmap v2840: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:30:21.452 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4615 8 365281 365273 0.618305 0 - 0.0299513 2026-02-23T14:30:22.132 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:30:22.247 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:30:22.343 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:30:22.341+0000 7fa957b9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:30:22.344 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:30:22.341+0000 7fa957b9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:30:22.452 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:30:22.452 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":247,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:30:19.626954+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:25:31.747822+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:27:49.594833+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:30:03.417272+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:30:22.452 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4616 8 365281 365273 0.618171 0 - 0.0299513 2026-02-23T14:30:23.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:30:22 trial068 ceph-mon[38805]: pgmap v2842: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:30:23.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:30:22 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3749828847' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:30:23.452 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4617 8 365281 365273 0.618037 0 - 0.0299513 2026-02-23T14:30:24.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:30:23 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:30:23] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:30:24.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:30:23 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:30:23.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:30:24.453 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4618 8 365281 365273 0.617903 0 - 0.0299513 2026-02-23T14:30:25.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:30:24 trial068 ceph-mon[38805]: pgmap v2843: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:30:25.453 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4619 8 365281 365273 0.617769 0 - 0.0299513 2026-02-23T14:30:25.488 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:30:25.608 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:30:25.707 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:30:25.705+0000 7feb70498640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:30:25.707 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:30:25.705+0000 7feb70498640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:30:25.815 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:30:25.815 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":247,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:30:19.626954+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:25:31.747822+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:27:49.594833+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:30:03.417272+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:30:26.453 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T14:30:26.452394+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T14:30:26.453 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T14:30:26.453 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4620 8 365281 365273 0.617636 0 - 0.0299513 2026-02-23T14:30:27.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:30:26 trial068 ceph-mon[38805]: pgmap v2844: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:30:27.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:30:26 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2008386279' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:30:27.453 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4621 8 365281 365273 0.617502 0 - 0.0299513 2026-02-23T14:30:28.453 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4622 8 365281 365273 0.617368 0 - 0.0299513 2026-02-23T14:30:28.852 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:30:28.969 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:30:28.991 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:30:28 trial068 ceph-mon[38805]: pgmap v2845: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:30:29.066 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:30:29.064+0000 7f5865738640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:30:29.066 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:30:29.064+0000 7f5865738640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:30:29.176 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:30:29.177 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":247,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:30:19.626954+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:25:31.747822+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:27:49.594833+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:30:03.417272+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:30:29.453 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4623 8 365281 365273 0.617235 0 - 0.0299513 2026-02-23T14:30:30.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:30:29 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2717894654' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:30:30.453 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4624 8 365281 365273 0.617101 0 - 0.0299513 2026-02-23T14:30:31.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:30:30 trial068 ceph-mon[38805]: pgmap v2846: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:30:31.453 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4625 8 365281 365273 0.616968 0 - 0.0299513 2026-02-23T14:30:32.215 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:30:32.331 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:30:32.428 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:30:32.425+0000 7f31b9378640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:30:32.428 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:30:32.426+0000 7f31b9378640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:30:32.453 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4626 8 365281 365273 0.616834 0 - 0.0299513 2026-02-23T14:30:32.536 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:30:32.537 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":247,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:30:19.626954+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:25:31.747822+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:27:49.594833+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:30:03.417272+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:30:32.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:30:32 trial068 ceph-mon[38805]: pgmap v2847: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:30:32.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:30:32 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/643771838' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:30:33.453 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4627 8 365281 365273 0.616701 0 - 0.0299513 2026-02-23T14:30:34.266 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:30:33 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:30:33.974Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:30:34.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:30:33 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:30:33] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:30:34.453 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4628 8 365281 365273 0.616568 0 - 0.0299513 2026-02-23T14:30:35.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:30:34 trial068 ceph-mon[38805]: pgmap v2848: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:30:35.453 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4629 8 365281 365273 0.616435 0 - 0.0299513 2026-02-23T14:30:35.574 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:30:35.692 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:30:35.797 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:30:35.795+0000 7f421e7f7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:30:35.797 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:30:35.795+0000 7f421e7f7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:30:35.910 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:30:35.910 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":247,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:30:19.626954+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:25:31.747822+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:27:49.594833+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:30:03.417272+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:30:36.453 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4630 8 365281 365273 0.616302 0 - 0.0299513 2026-02-23T14:30:37.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:30:36 trial068 ceph-mon[38805]: pgmap v2849: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:30:37.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:30:36 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2260775690' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:30:37.453 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4631 8 365281 365273 0.616169 0 - 0.0299513 2026-02-23T14:30:38.454 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4632 8 365281 365273 0.616036 0 - 0.0299513 2026-02-23T14:30:38.951 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:30:38.974 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:30:38 trial068 ceph-mon[38805]: pgmap v2850: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:30:39.068 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:30:39.168 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:30:39.166+0000 7fc3d327b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:30:39.168 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:30:39.166+0000 7fc3d327b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:30:39.278 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:30:39.278 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":247,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:30:19.626954+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:25:31.747822+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:27:49.594833+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:30:03.417272+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:30:39.454 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4633 8 365281 365273 0.615903 0 - 0.0299513 2026-02-23T14:30:40.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:30:39 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3571444253' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:30:40.454 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4634 8 365281 365273 0.61577 0 - 0.0299513 2026-02-23T14:30:41.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:30:40 trial068 ceph-mon[38805]: pgmap v2851: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:30:41.454 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4635 8 365281 365273 0.615637 0 - 0.0299513 2026-02-23T14:30:42.319 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:30:42.437 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:30:42.454 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4636 8 365281 365273 0.615504 0 - 0.0299513 2026-02-23T14:30:42.536 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:30:42.534+0000 7f62000c0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:30:42.536 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:30:42.534+0000 7f62000c0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:30:42.644 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:30:42.645 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":247,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:30:19.626954+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:25:31.747822+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:27:49.594833+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:30:03.417272+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:30:43.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:30:42 trial068 ceph-mon[38805]: pgmap v2852: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:30:43.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:30:42 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/929184998' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:30:43.454 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4637 8 365281 365273 0.615371 0 - 0.0299513 2026-02-23T14:30:44.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:30:43 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:30:43] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:30:44.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:30:43 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:30:43.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:30:44.454 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4638 8 365281 365273 0.615239 0 - 0.0299513 2026-02-23T14:30:45.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:30:44 trial068 ceph-mon[38805]: pgmap v2853: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:30:45.454 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4639 8 365281 365273 0.615106 0 - 0.0299513 2026-02-23T14:30:45.683 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:30:45.801 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:30:45.897 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:30:45.895+0000 7f6530b85640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:30:45.897 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:30:45.895+0000 7f6530b85640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:30:46.007 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:30:46.007 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":247,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:30:19.626954+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:25:31.747822+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:27:49.594833+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:30:03.417272+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:30:46.454 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T14:30:46.453790+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T14:30:46.454 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T14:30:46.455 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4640 8 365281 365273 0.614973 0 - 0.0299513 2026-02-23T14:30:47.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:30:46 trial068 ceph-mon[38805]: pgmap v2854: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:30:47.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:30:46 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1188318961' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:30:47.454 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4641 8 365281 365273 0.614841 0 - 0.0299513 2026-02-23T14:30:48.454 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4642 8 365281 365273 0.614708 0 - 0.0299513 2026-02-23T14:30:49.045 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:30:49.071 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:30:48 trial068 ceph-mon[38805]: pgmap v2855: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:30:49.165 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:30:49.261 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:30:49.259+0000 7f754c425640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:30:49.262 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:30:49.260+0000 7f754c425640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:30:49.370 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:30:49.371 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":247,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:30:19.626954+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:25:31.747822+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:27:49.594833+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:30:03.417272+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:30:49.454 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4643 8 365281 365273 0.614576 0 - 0.0299513 2026-02-23T14:30:50.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:30:49 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2012204249' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:30:50.454 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4644 8 365281 365273 0.614444 0 - 0.0299513 2026-02-23T14:30:51.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:30:50 trial068 ceph-mon[38805]: pgmap v2856: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:30:51.454 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4645 8 365281 365273 0.614311 0 - 0.0299513 2026-02-23T14:30:52.410 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:30:52.454 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4646 8 365281 365273 0.614179 0 - 0.0299513 2026-02-23T14:30:52.527 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:30:52.627 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:30:52.625+0000 7f6d0d0fe640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:30:52.627 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:30:52.625+0000 7f6d0d0fe640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:30:52.738 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:30:52.738 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":247,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:30:19.626954+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:25:31.747822+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:27:49.594833+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:30:03.417272+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:30:53.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:30:52 trial068 ceph-mon[38805]: pgmap v2857: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:30:53.455 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4647 8 365281 365273 0.614047 0 - 0.0299513 2026-02-23T14:30:54.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:30:53 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/898379944' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:30:54.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:30:53 trial068 ceph-mon[38805]: 5.a scrub starts 2026-02-23T14:30:54.094 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:30:53 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:30:53] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:30:54.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:30:53 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:30:53.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:30:54.455 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4648 8 365281 365273 0.613915 0 - 0.0299513 2026-02-23T14:30:55.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:30:54 trial068 ceph-mon[38805]: 5.a scrub ok 2026-02-23T14:30:55.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:30:54 trial068 ceph-mon[38805]: pgmap v2858: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:30:55.455 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4649 8 365281 365273 0.613783 0 - 0.0299513 2026-02-23T14:30:55.777 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:30:55.895 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:30:56.000 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:30:55.997+0000 7ff6877c7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:30:56.000 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:30:55.998+0000 7ff6877c7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:30:56.109 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:30:56.109 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":247,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:30:19.626954+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:25:31.747822+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:27:49.594833+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:30:03.417272+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:30:56.455 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4650 8 365281 365273 0.613651 0 - 0.0299513 2026-02-23T14:30:57.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:30:56 trial068 ceph-mon[38805]: pgmap v2859: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:30:57.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:30:56 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/729341692' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:30:57.455 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4651 8 365281 365273 0.613519 0 - 0.0299513 2026-02-23T14:30:58.455 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4652 8 365281 365273 0.613387 0 - 0.0299513 2026-02-23T14:30:59.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:30:58 trial068 ceph-mon[38805]: pgmap v2860: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:30:59.148 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:30:59.266 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:30:59.368 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:30:59.366+0000 7f26d5241640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:30:59.368 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:30:59.366+0000 7f26d5241640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:30:59.455 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4653 8 365281 365273 0.613255 0 - 0.0299513 2026-02-23T14:30:59.478 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:30:59.478 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":247,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:30:19.626954+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:25:31.747822+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:27:49.594833+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:30:03.417272+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:31:00.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:30:59 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/929353505' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:31:00.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:30:59 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2026-02-23T14:31:00.455 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4654 8 365281 365273 0.613123 0 - 0.0299513 2026-02-23T14:31:01.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:31:00 trial068 ceph-mon[38805]: pgmap v2861: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:31:01.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:31:00 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-23T14:31:01.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:31:00 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2026-02-23T14:31:01.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:31:00 trial068 ceph-mon[38805]: pgmap v2862: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:31:01.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:31:00 trial068 ceph-mon[38805]: pgmap v2863: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:31:01.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:31:00 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T14:31:01.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:31:00 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2026-02-23T14:31:01.455 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4655 8 365281 365273 0.612992 0 - 0.0299513 2026-02-23T14:31:02.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:31:01 trial068 ceph-mon[38805]: pgmap v2864: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:31:02.455 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4656 8 365281 365273 0.61286 0 - 0.0299513 2026-02-23T14:31:02.516 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:31:02.636 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:31:02.743 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:31:02.741+0000 7efd23618640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:31:02.743 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:31:02.741+0000 7efd23618640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:31:02.850 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:31:02.850 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":247,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:30:19.626954+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:25:31.747822+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:27:49.594833+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:30:03.417272+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:31:03.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:31:02 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/458135440' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:31:03.455 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4657 8 365281 365273 0.612728 0 - 0.0299513 2026-02-23T14:31:04.265 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:31:03 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:31:03.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:31:04.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:31:03 trial068 ceph-mon[38805]: 5.e scrub starts 2026-02-23T14:31:04.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:31:03 trial068 ceph-mon[38805]: pgmap v2865: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:31:04.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:31:03 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:31:03] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:31:04.455 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4658 8 365281 365273 0.612597 0 - 0.0299513 2026-02-23T14:31:05.245 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:31:04 trial068 ceph-mon[38805]: 5.e scrub ok 2026-02-23T14:31:05.455 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4659 8 365281 365273 0.612465 0 - 0.0299513 2026-02-23T14:31:05.890 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:31:06.008 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:31:06.113 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:31:06.111+0000 7f912f068640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:31:06.113 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:31:06.111+0000 7f912f068640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:31:06.224 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:31:06.224 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":247,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:30:19.626954+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:25:31.747822+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:27:49.594833+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:30:03.417272+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:31:06.225 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:31:05 trial068 ceph-mon[38805]: pgmap v2866: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:31:06.455 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T14:31:06.455116+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T14:31:06.456 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T14:31:06.456 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4660 8 365281 365273 0.612334 0 - 0.0299513 2026-02-23T14:31:07.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:31:06 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/868528039' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:31:07.455 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4661 8 365281 365273 0.612203 0 - 0.0299513 2026-02-23T14:31:08.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:31:07 trial068 ceph-mon[38805]: 5.8 scrub starts 2026-02-23T14:31:08.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:31:07 trial068 ceph-mon[38805]: pgmap v2867: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:31:08.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:31:07 trial068 ceph-mon[38805]: 5.8 scrub ok 2026-02-23T14:31:08.456 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4662 8 365281 365273 0.612071 0 - 0.0299513 2026-02-23T14:31:09.261 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:31:09.378 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:31:09.456 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4663 8 365281 365273 0.61194 0 - 0.0299513 2026-02-23T14:31:09.474 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:31:09.472+0000 7f72250c3640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:31:09.474 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:31:09.472+0000 7f72250c3640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:31:09.582 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:31:09.583 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":247,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:30:19.626954+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:25:31.747822+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:27:49.594833+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:30:03.417272+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:31:09.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:31:09 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/4233201664' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:31:10.456 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4664 8 365281 365273 0.611809 0 - 0.0299513 2026-02-23T14:31:11.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:31:10 trial068 ceph-mon[38805]: pgmap v2868: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:31:11.456 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4665 8 365281 365273 0.611678 0 - 0.0299513 2026-02-23T14:31:12.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:31:11 trial068 ceph-mon[38805]: pgmap v2869: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:31:12.456 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4666 8 365281 365273 0.611547 0 - 0.0299513 2026-02-23T14:31:12.620 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:31:12.738 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:31:12.839 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:31:12.837+0000 7f6733729640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:31:12.839 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:31:12.837+0000 7f6733729640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:31:12.947 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:31:12.948 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":247,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:30:19.626954+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:25:31.747822+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:27:49.594833+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:30:03.417272+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:31:13.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:31:12 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1437436538' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:31:13.456 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4667 8 365281 365273 0.611416 0 - 0.0299513 2026-02-23T14:31:14.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:31:13 trial068 ceph-mon[38805]: pgmap v2870: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:31:14.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:31:13 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:31:13] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:31:14.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:31:13 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:31:13.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:31:14.456 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4668 8 365281 365273 0.611285 0 - 0.0299513 2026-02-23T14:31:15.456 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4669 8 365281 365273 0.611154 0 - 0.0299513 2026-02-23T14:31:15.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:31:15 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T14:31:15.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:31:15 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T14:31:15.985 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:31:16.106 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:31:16.212 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:31:16.210+0000 7f4578cd2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:31:16.212 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:31:16.211+0000 7f4578cd2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:31:16.322 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:31:16.323 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":247,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:30:19.626954+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:25:31.747822+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:27:49.594833+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:30:03.417272+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:31:16.323 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:31:16 trial068 ceph-mon[38805]: pgmap v2871: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:31:16.456 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4670 8 365281 365273 0.611023 0 - 0.0299513 2026-02-23T14:31:17.456 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4671 8 365281 365273 0.610892 0 - 0.0299513 2026-02-23T14:31:17.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:31:17 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3735770412' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:31:18.456 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4672 8 365281 365273 0.610761 0 - 0.0299513 2026-02-23T14:31:18.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:31:18 trial068 ceph-mon[38805]: pgmap v2872: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:31:19.359 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:31:19.456 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4673 8 365281 365273 0.610631 0 - 0.0299513 2026-02-23T14:31:19.476 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:31:19.575 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:31:19.573+0000 7f24fe2a6640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:31:19.575 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:31:19.573+0000 7f24fe2a6640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:31:19.685 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:31:19.685 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":247,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:30:19.626954+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:25:31.747822+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:27:49.594833+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:30:03.417272+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:31:20.457 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4674 8 365281 365273 0.6105 0 - 0.0299513 2026-02-23T14:31:21.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:31:20 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1780764591' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:31:21.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:31:20 trial068 ceph-mon[38805]: pgmap v2873: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:31:21.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:31:20 trial068 ceph-mon[38805]: 5.7 scrub starts 2026-02-23T14:31:21.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:31:20 trial068 ceph-mon[38805]: 5.7 scrub ok 2026-02-23T14:31:21.457 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4675 8 365281 365273 0.610369 0 - 0.0299513 2026-02-23T14:31:22.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:31:21 trial068 ceph-mon[38805]: pgmap v2874: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:31:22.457 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4676 8 365281 365273 0.610239 0 - 0.0299513 2026-02-23T14:31:22.724 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:31:22.840 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:31:22.943 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:31:22.941+0000 7f3e349d8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:31:22.943 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:31:22.941+0000 7f3e349d8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:31:23.052 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:31:23.053 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":247,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:30:19.626954+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:25:31.747822+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:27:49.594833+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:30:03.417272+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:31:23.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:31:23 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3156849329' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:31:23.457 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4677 8 365281 365273 0.610108 0 - 0.0299513 2026-02-23T14:31:24.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:31:24 trial068 ceph-mon[38805]: pgmap v2875: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:31:24.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:31:23 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:31:23] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:31:24.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:31:23 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:31:23.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:31:24.457 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4678 8 365281 365273 0.609978 0 - 0.0299513 2026-02-23T14:31:25.457 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4679 8 365281 365273 0.609848 0 - 0.0299513 2026-02-23T14:31:26.090 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:31:26.206 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:31:26.303 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:31:26.301+0000 7fb2c19c4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:31:26.304 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:31:26.302+0000 7fb2c19c4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:31:26.411 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:31:26.411 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":248,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:31:25.751583+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:30:16.829966+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:27:49.594833+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:30:03.417272+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:31:26.457 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T14:31:26.456751+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T14:31:26.457 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T14:31:26.458 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4680 8 365281 365273 0.609717 0 - 0.0299513 2026-02-23T14:31:27.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:31:26 trial068 ceph-mon[38805]: 5.5 scrub starts 2026-02-23T14:31:27.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:31:26 trial068 ceph-mon[38805]: osdmap e248: 4 total, 3 up, 4 in 2026-02-23T14:31:27.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:31:26 trial068 ceph-mon[38805]: pgmap v2877: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:31:27.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:31:26 trial068 ceph-mon[38805]: 5.5 scrub ok 2026-02-23T14:31:27.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:31:26 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2561956163' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:31:27.457 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4681 8 365281 365273 0.609587 0 - 0.0299513 2026-02-23T14:31:28.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:31:27 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T14:31:28.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:31:27 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T14:31:28.457 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4682 8 365281 365273 0.609457 0 - 0.0299513 2026-02-23T14:31:29.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:31:28 trial068 ceph-mon[38805]: pgmap v2878: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:31:29.448 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:31:29.457 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4683 8 365281 365273 0.609327 0 - 0.0299513 2026-02-23T14:31:29.566 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:31:29.672 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:31:29.670+0000 7f2580258640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:31:29.672 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:31:29.670+0000 7f2580258640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:31:29.783 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:31:29.783 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":249,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:31:29.636731+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:30:16.829966+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:27:49.594833+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:30:03.417272+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:31:30.457 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4684 8 365281 365273 0.609197 0 - 0.0299513 2026-02-23T14:31:31.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:31:30 trial068 ceph-mon[38805]: osdmap e249: 4 total, 3 up, 4 in 2026-02-23T14:31:31.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:31:30 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2928770212' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:31:31.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:31:30 trial068 ceph-mon[38805]: pgmap v2880: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:31:31.457 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4685 8 365281 365273 0.609066 0 - 0.0299513 2026-02-23T14:31:32.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:31:31 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T14:31:32.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:31:31 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T14:31:32.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:31:31 trial068 ceph-mon[38805]: pgmap v2881: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:31:32.457 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4686 8 365281 365273 0.608937 0 - 0.0299513 2026-02-23T14:31:32.818 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:31:32.936 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:31:33.037 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:31:33.034+0000 7f8d12c62640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:31:33.037 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:31:33.035+0000 7f8d12c62640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:31:33.146 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:31:33.146 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":249,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:31:29.636731+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:30:16.829966+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:27:49.594833+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:30:03.417272+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:31:33.458 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4687 8 365281 365273 0.608807 0 - 0.0299513 2026-02-23T14:31:33.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:31:33 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/4145049790' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:31:34.266 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:31:33 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:31:33.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:31:34.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:31:34 trial068 ceph-mon[38805]: pgmap v2882: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:31:34.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:31:33 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:31:33] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:31:34.458 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4688 8 365281 365273 0.608677 0 - 0.0299513 2026-02-23T14:31:35.458 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4689 8 365281 365273 0.608547 0 - 0.0299513 2026-02-23T14:31:36.146 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:31:35 trial068 ceph-mon[38805]: pgmap v2883: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:31:36.182 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:31:36.299 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:31:36.396 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:31:36.394+0000 7f99cf450640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:31:36.397 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:31:36.395+0000 7f99cf450640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:31:36.458 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4690 8 365281 365273 0.608417 0 - 0.0299513 2026-02-23T14:31:36.506 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:31:36.506 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":249,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:31:29.636731+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:30:16.829966+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:27:49.594833+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:30:03.417272+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:31:37.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:31:36 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1137023466' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:31:37.458 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4691 8 365281 365273 0.608287 0 - 0.0299513 2026-02-23T14:31:38.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:31:37 trial068 ceph-mon[38805]: pgmap v2884: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:31:38.458 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4692 8 365281 365273 0.608158 0 - 0.0299513 2026-02-23T14:31:39.458 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4693 8 365281 365273 0.608028 0 - 0.0299513 2026-02-23T14:31:39.544 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:31:39.662 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:31:39.762 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:31:39.760+0000 7fe5e3b26640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:31:39.762 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:31:39.760+0000 7fe5e3b26640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:31:39.871 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:31:39.872 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":249,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:31:29.636731+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:30:16.829966+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:27:49.594833+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:30:03.417272+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:31:40.458 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4694 8 365281 365273 0.607899 0 - 0.0299513 2026-02-23T14:31:41.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:31:40 trial068 ceph-mon[38805]: pgmap v2885: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:31:41.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:31:40 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/4199578060' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:31:41.458 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4695 8 365281 365273 0.607769 0 - 0.0299513 2026-02-23T14:31:42.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:31:41 trial068 ceph-mon[38805]: pgmap v2886: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:31:42.458 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4696 8 365281 365273 0.60764 0 - 0.0299513 2026-02-23T14:31:42.908 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:31:43.026 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:31:43.131 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:31:43.129+0000 7fb67e799640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:31:43.131 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:31:43.129+0000 7fb67e799640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:31:43.240 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:31:43.240 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":249,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:31:29.636731+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:30:16.829966+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:27:49.594833+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:30:03.417272+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:31:43.458 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4697 8 365281 365273 0.60751 0 - 0.0299513 2026-02-23T14:31:43.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:31:43 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/4160494421' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:31:44.292 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:31:43 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:31:43] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:31:44.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:31:43 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:31:43.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:31:44.458 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4698 8 365281 365273 0.607381 0 - 0.0299513 2026-02-23T14:31:44.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:31:44 trial068 ceph-mon[38805]: pgmap v2887: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:31:45.458 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4699 8 365281 365273 0.607252 0 - 0.0299513 2026-02-23T14:31:46.278 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:31:46.304 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:31:45 trial068 ceph-mon[38805]: pgmap v2888: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:31:46.400 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:31:46.458 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T14:31:46.458075+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T14:31:46.459 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T14:31:46.459 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4700 8 365281 365273 0.607123 0 - 0.0299513 2026-02-23T14:31:46.507 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:31:46.505+0000 7f74df705640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:31:46.507 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:31:46.505+0000 7f74df705640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:31:46.616 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:31:46.616 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":249,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:31:29.636731+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:30:16.829966+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:27:49.594833+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:30:03.417272+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:31:47.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:31:46 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/4109805714' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:31:47.458 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4701 8 365281 365273 0.606994 0 - 0.0299513 2026-02-23T14:31:48.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:31:47 trial068 ceph-mon[38805]: pgmap v2889: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:31:48.459 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4702 8 365281 365273 0.606864 0 - 0.0299513 2026-02-23T14:31:49.459 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4703 8 365281 365273 0.606735 0 - 0.0299513 2026-02-23T14:31:49.653 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:31:49.774 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:31:49.876 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:31:49.874+0000 7f120d67d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:31:49.876 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:31:49.874+0000 7f120d67d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:31:49.986 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:31:49.986 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":249,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:31:29.636731+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:30:16.829966+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:27:49.594833+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:30:03.417272+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:31:49.986 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:31:49 trial068 ceph-mon[38805]: pgmap v2890: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:31:50.459 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4704 8 365281 365273 0.606606 0 - 0.0299513 2026-02-23T14:31:51.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:31:50 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/4157728996' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:31:51.459 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4705 8 365281 365273 0.606477 0 - 0.0299513 2026-02-23T14:31:52.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:31:51 trial068 ceph-mon[38805]: pgmap v2891: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:31:52.459 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4706 8 365281 365273 0.606349 0 - 0.0299513 2026-02-23T14:31:53.022 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:31:53.140 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:31:53.243 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:31:53.241+0000 7fa34945e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:31:53.243 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:31:53.241+0000 7fa34945e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:31:53.353 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:31:53.353 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":249,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:31:29.636731+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:30:16.829966+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:27:49.594833+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:30:03.417272+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:31:53.459 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4707 8 365281 365273 0.60622 0 - 0.0299513 2026-02-23T14:31:53.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:31:53 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3424610652' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:31:54.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:31:53 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:31:53] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:31:54.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:31:53 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:31:53.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:31:54.459 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4708 8 365281 365273 0.606091 0 - 0.0299513 2026-02-23T14:31:54.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:31:54 trial068 ceph-mon[38805]: pgmap v2892: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:31:55.459 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4709 8 365281 365273 0.605962 0 - 0.0299513 2026-02-23T14:31:56.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:31:55 trial068 ceph-mon[38805]: pgmap v2893: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:31:56.405 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:31:56.459 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4710 8 365281 365273 0.605834 0 - 0.0299513 2026-02-23T14:31:56.524 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:31:56.625 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:31:56.622+0000 7fbc60f3d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:31:56.625 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:31:56.623+0000 7fbc60f3d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:31:56.735 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:31:56.736 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":249,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:31:29.636731+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:30:16.829966+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:27:49.594833+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:30:03.417272+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:31:57.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:31:56 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3060743379' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:31:57.459 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4711 8 365281 365273 0.605705 0 - 0.0299513 2026-02-23T14:31:58.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:31:57 trial068 ceph-mon[38805]: pgmap v2894: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:31:58.459 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4712 8 365281 365273 0.605577 0 - 0.0299513 2026-02-23T14:31:59.459 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4713 8 365281 365273 0.605448 0 - 0.0299513 2026-02-23T14:31:59.774 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:31:59.893 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:32:00.003 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:32:00.000+0000 7f0c5a80e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:32:00.003 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:32:00.001+0000 7f0c5a80e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:32:00.117 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:32:00.117 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":249,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:31:29.636731+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:30:16.829966+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:27:49.594833+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:30:03.417272+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:32:00.459 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4714 8 365281 365273 0.60532 0 - 0.0299513 2026-02-23T14:32:01.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:32:00 trial068 ceph-mon[38805]: pgmap v2895: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:32:01.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:32:00 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2026-02-23T14:32:01.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:32:00 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-23T14:32:01.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:32:00 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2026-02-23T14:32:01.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:32:00 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T14:32:01.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:32:00 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2026-02-23T14:32:01.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:32:00 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/4045167804' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:32:01.460 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4715 8 365281 365273 0.605191 0 - 0.0299513 2026-02-23T14:32:02.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:32:01 trial068 ceph-mon[38805]: pgmap v2896: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:32:02.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:32:01 trial068 ceph-mon[38805]: pgmap v2897: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:32:02.460 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4716 8 365281 365273 0.605063 0 - 0.0299513 2026-02-23T14:32:03.157 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:32:03.279 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:32:03.379 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:32:03.377+0000 7fc8b7b05640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:32:03.379 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:32:03.377+0000 7fc8b7b05640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:32:03.460 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4717 8 365281 365273 0.604935 0 - 0.0299513 2026-02-23T14:32:03.487 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:32:03.488 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":249,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:31:29.636731+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:30:16.829966+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:27:49.594833+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:30:03.417272+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:32:03.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:32:03 trial068 ceph-mon[38805]: pgmap v2898: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:32:03.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:32:03 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2697711652' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:32:04.266 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:32:03 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:32:03.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:32:04.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:32:03 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:32:03] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:32:04.460 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4718 8 365281 365273 0.604806 0 - 0.0299513 2026-02-23T14:32:05.460 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4719 8 365281 365273 0.604678 0 - 0.0299513 2026-02-23T14:32:06.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:32:05 trial068 ceph-mon[38805]: pgmap v2899: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:32:06.460 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T14:32:06.459539+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T14:32:06.460 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T14:32:06.460 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4720 8 365281 365273 0.60455 0 - 0.0299513 2026-02-23T14:32:06.524 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:32:06.641 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:32:06.741 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:32:06.738+0000 7fbbf3901640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:32:06.741 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:32:06.739+0000 7fbbf3901640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:32:06.854 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:32:06.854 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":249,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:31:29.636731+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:30:16.829966+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:27:49.594833+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:30:03.417272+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:32:07.460 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4721 8 365281 365273 0.604422 0 - 0.0299513 2026-02-23T14:32:08.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:32:07 trial068 ceph-mon[38805]: pgmap v2900: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:32:08.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:32:07 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1207009886' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:32:08.460 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4722 8 365281 365273 0.604294 0 - 0.0299513 2026-02-23T14:32:09.460 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4723 8 365281 365273 0.604166 0 - 0.0299513 2026-02-23T14:32:09.894 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:32:09.919 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:32:09 trial068 ceph-mon[38805]: pgmap v2901: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:32:10.012 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:32:10.112 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:32:10.110+0000 7f009cc96640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:32:10.112 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:32:10.110+0000 7f009cc96640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:32:10.221 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:32:10.222 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":249,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:31:29.636731+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:30:16.829966+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:27:49.594833+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:30:03.417272+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:32:10.460 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4724 8 365281 365273 0.604038 0 - 0.0299513 2026-02-23T14:32:11.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:32:10 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/253166628' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:32:11.460 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4725 8 365281 365273 0.60391 0 - 0.0299513 2026-02-23T14:32:12.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:32:11 trial068 ceph-mon[38805]: pgmap v2902: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:32:12.461 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4726 8 365281 365273 0.603783 0 - 0.0299513 2026-02-23T14:32:13.259 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:32:13.377 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:32:13.461 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4727 8 365281 365273 0.603655 0 - 0.0299513 2026-02-23T14:32:13.477 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:32:13.475+0000 7fc8974ef640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:32:13.477 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:32:13.475+0000 7fc8974ef640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:32:13.590 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:32:13.590 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":249,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:31:29.636731+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:30:16.829966+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:27:49.594833+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:30:03.417272+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:32:13.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:32:13 trial068 ceph-mon[38805]: pgmap v2903: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:32:13.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:32:13 trial068 ceph-mon[38805]: 5.a scrub starts 2026-02-23T14:32:13.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:32:13 trial068 ceph-mon[38805]: 5.a scrub ok 2026-02-23T14:32:13.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:32:13 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3402544491' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:32:14.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:32:13 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:32:13] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:32:14.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:32:13 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:32:13.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:32:14.461 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4728 8 365281 365273 0.603527 0 - 0.0299513 2026-02-23T14:32:15.461 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4729 8 365281 365273 0.6034 0 - 0.0299513 2026-02-23T14:32:16.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:32:15 trial068 ceph-mon[38805]: pgmap v2904: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:32:16.461 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4730 8 365281 365273 0.603272 0 - 0.0299513 2026-02-23T14:32:16.627 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:32:16.744 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:32:16.846 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:32:16.844+0000 7f12a78e1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:32:16.846 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:32:16.844+0000 7f12a78e1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:32:16.953 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:32:16.953 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":249,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:31:29.636731+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:30:16.829966+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:27:49.594833+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:30:03.417272+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:32:17.461 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4731 8 365281 365273 0.603144 0 - 0.0299513 2026-02-23T14:32:18.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:32:17 trial068 ceph-mon[38805]: pgmap v2905: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:32:18.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:32:17 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2544319458' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:32:18.461 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4732 8 365281 365273 0.603017 0 - 0.0299513 2026-02-23T14:32:19.461 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4733 8 365281 365273 0.60289 0 - 0.0299513 2026-02-23T14:32:19.991 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:32:20.016 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:32:19 trial068 ceph-mon[38805]: pgmap v2906: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:32:20.109 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:32:20.206 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:32:20.203+0000 7fbbbbcd9640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:32:20.206 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:32:20.204+0000 7fbbbbcd9640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:32:20.315 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:32:20.315 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":249,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:31:29.636731+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:30:16.829966+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:27:49.594833+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:30:03.417272+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:32:20.461 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4734 8 365281 365273 0.602762 0 - 0.0299513 2026-02-23T14:32:21.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:32:20 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1915406098' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:32:21.461 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4735 8 365281 365273 0.602635 0 - 0.0299513 2026-02-23T14:32:22.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:32:21 trial068 ceph-mon[38805]: pgmap v2907: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:32:22.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:32:21 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T14:32:22.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:32:21 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T14:32:22.461 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4736 8 365281 365273 0.602508 0 - 0.0299513 2026-02-23T14:32:23.352 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:32:23.461 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4737 8 365281 365273 0.602381 0 - 0.0299513 2026-02-23T14:32:23.469 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:32:23.571 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:32:23.568+0000 7efe6face640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:32:23.571 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:32:23.569+0000 7efe6face640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:32:23.683 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:32:23.683 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":249,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:31:29.636731+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:30:16.829966+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:27:49.594833+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:30:03.417272+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:32:23.975 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:32:23 trial068 ceph-mon[38805]: pgmap v2908: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:32:24.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:32:23 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:32:23] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:32:24.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:32:23 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:32:23.974Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:32:24.461 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4738 8 365281 365273 0.602253 0 - 0.0299513 2026-02-23T14:32:25.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:32:24 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/478282369' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:32:25.462 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4739 8 365281 365273 0.602126 0 - 0.0299513 2026-02-23T14:32:26.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:32:25 trial068 ceph-mon[38805]: pgmap v2909: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:32:26.462 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T14:32:26.461451+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T14:32:26.462 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T14:32:26.462 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4740 8 365281 365273 0.601999 0 - 0.0299513 2026-02-23T14:32:26.722 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:32:26.843 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:32:26.954 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:32:26.951+0000 7ff9c84a8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:32:26.954 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:32:26.952+0000 7ff9c84a8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:32:27.063 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:32:27.063 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":249,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:31:29.636731+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:30:16.829966+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:27:49.594833+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:30:03.417272+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:32:27.462 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4741 8 365281 365273 0.601872 0 - 0.0299513 2026-02-23T14:32:28.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:32:27 trial068 ceph-mon[38805]: pgmap v2910: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:32:28.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:32:27 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/576226761' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:32:28.462 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4742 8 365281 365273 0.601745 0 - 0.0299513 2026-02-23T14:32:29.462 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4743 8 365281 365273 0.601618 0 - 0.0299513 2026-02-23T14:32:30.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:32:29 trial068 ceph-mon[38805]: pgmap v2911: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:32:30.100 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:32:30.227 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:32:30.331 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:32:30.329+0000 7fb945469640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:32:30.331 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:32:30.329+0000 7fb945469640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:32:30.440 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:32:30.440 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":249,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:31:29.636731+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:30:16.829966+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:27:49.594833+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:30:03.417272+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:32:30.462 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4744 8 365281 365273 0.601492 0 - 0.0299513 2026-02-23T14:32:30.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:32:30 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3470731522' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:32:31.462 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4745 8 365281 365273 0.601365 0 - 0.0299513 2026-02-23T14:32:32.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:32:31 trial068 ceph-mon[38805]: pgmap v2912: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:32:32.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:32:31 trial068 ceph-mon[38805]: 5.8 scrub starts 2026-02-23T14:32:32.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:32:31 trial068 ceph-mon[38805]: 5.8 scrub ok 2026-02-23T14:32:32.462 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4746 8 365281 365273 0.601238 0 - 0.0299513 2026-02-23T14:32:33.462 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4747 8 365281 365273 0.601112 0 - 0.0299513 2026-02-23T14:32:33.478 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:32:33.595 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:32:33.699 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:32:33.697+0000 7fd0c9bc1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:32:33.699 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:32:33.697+0000 7fd0c9bc1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:32:33.809 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:32:33.810 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":249,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:31:29.636731+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:30:16.829966+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:27:49.594833+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:30:03.417272+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:32:33.810 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:32:33 trial068 ceph-mon[38805]: pgmap v2913: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:32:34.094 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:32:33 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:32:33] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:32:34.266 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:32:33 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:32:33.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:32:34.462 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4748 8 365281 365273 0.600985 0 - 0.0299513 2026-02-23T14:32:35.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:32:34 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3911618738' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:32:35.462 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4749 8 365281 365273 0.600858 0 - 0.0299513 2026-02-23T14:32:36.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:32:35 trial068 ceph-mon[38805]: pgmap v2914: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:32:36.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:32:35 trial068 ceph-mon[38805]: 5.e scrub starts 2026-02-23T14:32:36.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:32:35 trial068 ceph-mon[38805]: 5.e scrub ok 2026-02-23T14:32:36.462 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4750 8 365281 365273 0.600732 0 - 0.0299513 2026-02-23T14:32:36.846 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:32:36.964 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:32:37.064 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:32:37.062+0000 7f1daf58f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:32:37.064 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:32:37.062+0000 7f1daf58f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:32:37.173 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:32:37.173 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":249,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:31:29.636731+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:30:16.829966+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:27:49.594833+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:30:03.417272+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:32:37.463 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4751 8 365281 365273 0.600605 0 - 0.0299513 2026-02-23T14:32:38.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:32:37 trial068 ceph-mon[38805]: pgmap v2915: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:32:38.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:32:37 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1133742563' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:32:38.463 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4752 8 365281 365273 0.600479 0 - 0.0299513 2026-02-23T14:32:39.463 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4753 8 365281 365273 0.600353 0 - 0.0299513 2026-02-23T14:32:40.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:32:39 trial068 ceph-mon[38805]: pgmap v2916: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:32:40.211 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:32:40.328 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:32:40.434 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:32:40.431+0000 7f8ce4f3f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:32:40.434 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:32:40.431+0000 7f8ce4f3f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:32:40.463 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4754 8 365281 365273 0.600226 0 - 0.0299513 2026-02-23T14:32:40.544 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:32:40.545 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":249,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:31:29.636731+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:30:16.829966+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:27:49.594833+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:30:03.417272+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:32:40.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:32:40 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1303832512' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:32:41.463 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4755 8 365281 365273 0.6001 0 - 0.0299513 2026-02-23T14:32:42.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:32:41 trial068 ceph-mon[38805]: pgmap v2917: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:32:42.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:32:41 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T14:32:42.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:32:41 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T14:32:42.463 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4756 8 365281 365273 0.599974 0 - 0.0299513 2026-02-23T14:32:43.463 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4757 8 365281 365273 0.599848 0 - 0.0299513 2026-02-23T14:32:43.582 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:32:43.699 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:32:43.808 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:32:43.806+0000 7fc3e9c00640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:32:43.809 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:32:43.807+0000 7fc3e9c00640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:32:43.919 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:32:43.919 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":249,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:31:29.636731+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:30:16.829966+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:27:49.594833+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:30:03.417272+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:32:43.919 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:32:43 trial068 ceph-mon[38805]: pgmap v2918: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:32:44.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:32:43 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:32:43] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:32:44.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:32:43 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:32:43.972Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:32:44.463 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4758 8 365281 365273 0.599722 0 - 0.0299513 2026-02-23T14:32:45.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:32:44 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1118173792' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:32:45.463 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4759 8 365281 365273 0.599596 0 - 0.0299513 2026-02-23T14:32:46.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:32:45 trial068 ceph-mon[38805]: pgmap v2919: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:32:46.463 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T14:32:46.462832+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T14:32:46.463 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T14:32:46.463 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4760 8 365281 365273 0.59947 0 - 0.0299513 2026-02-23T14:32:46.955 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:32:47.073 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:32:47.174 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:32:47.172+0000 7f8815029640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:32:47.174 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:32:47.172+0000 7f8815029640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:32:47.283 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:32:47.284 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":249,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:31:29.636731+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:30:16.829966+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:27:49.594833+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:30:03.417272+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:32:47.463 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4761 8 365281 365273 0.599344 0 - 0.0299513 2026-02-23T14:32:48.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:32:47 trial068 ceph-mon[38805]: pgmap v2920: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:32:48.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:32:47 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/763486996' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:32:48.463 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4762 8 365281 365273 0.599218 0 - 0.0299513 2026-02-23T14:32:49.463 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4763 8 365281 365273 0.599092 0 - 0.0299513 2026-02-23T14:32:50.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:32:49 trial068 ceph-mon[38805]: pgmap v2921: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:32:50.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:32:49 trial068 ceph-mon[38805]: 5.5 scrub starts 2026-02-23T14:32:50.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:32:49 trial068 ceph-mon[38805]: 5.5 scrub ok 2026-02-23T14:32:50.321 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:32:50.438 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:32:50.463 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4764 8 365281 365273 0.598967 0 - 0.0299513 2026-02-23T14:32:50.545 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:32:50.543+0000 7f9a64135640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:32:50.545 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:32:50.543+0000 7f9a64135640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:32:50.654 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:32:50.654 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":249,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:31:29.636731+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:30:16.829966+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:27:49.594833+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:30:03.417272+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:32:51.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:32:50 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1947476993' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:32:51.463 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4765 8 365281 365273 0.598841 0 - 0.0299513 2026-02-23T14:32:52.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:32:51 trial068 ceph-mon[38805]: pgmap v2922: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:32:52.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:32:51 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T14:32:52.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:32:51 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T14:32:52.464 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4766 8 365281 365273 0.598715 0 - 0.0299513 2026-02-23T14:32:53.464 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4767 8 365281 365273 0.59859 0 - 0.0299513 2026-02-23T14:32:53.692 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:32:53.810 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:32:53.921 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:32:53.919+0000 7f0701ddc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:32:53.921 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:32:53.919+0000 7f0701ddc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:32:53.973 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:32:53 trial068 ceph-mon[38805]: pgmap v2923: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:32:54.030 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:32:54.031 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":249,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:31:29.636731+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:30:16.829966+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:27:49.594833+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:30:03.417272+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:32:54.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:32:53 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:32:53] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:32:54.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:32:53 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:32:53.972Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:32:54.464 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4768 8 365281 365273 0.598464 0 - 0.0299513 2026-02-23T14:32:55.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:32:54 trial068 ceph-mon[38805]: 5.7 scrub starts 2026-02-23T14:32:55.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:32:54 trial068 ceph-mon[38805]: 5.7 scrub ok 2026-02-23T14:32:55.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:32:54 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/435970645' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:32:55.464 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4769 8 365281 365273 0.598339 0 - 0.0299513 2026-02-23T14:32:56.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:32:55 trial068 ceph-mon[38805]: pgmap v2924: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:32:56.464 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4770 8 365281 365273 0.598213 0 - 0.0299513 2026-02-23T14:32:57.067 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:32:57.188 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:32:57.285 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:32:57.283+0000 7f05aa585640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:32:57.285 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:32:57.284+0000 7f05aa585640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:32:57.395 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:32:57.396 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":249,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:31:29.636731+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:30:16.829966+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:27:49.594833+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:30:03.417272+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:32:57.464 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4771 8 365281 365273 0.598088 0 - 0.0299513 2026-02-23T14:32:58.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:32:57 trial068 ceph-mon[38805]: pgmap v2925: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:32:58.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:32:57 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1162510274' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:32:58.464 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4772 8 365281 365273 0.597962 0 - 0.0299513 2026-02-23T14:32:59.464 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4773 8 365281 365273 0.597837 0 - 0.0299513 2026-02-23T14:33:00.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:32:59 trial068 ceph-mon[38805]: pgmap v2926: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:33:00.447 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:33:00.464 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4774 8 365281 365273 0.597712 0 - 0.0299513 2026-02-23T14:33:00.562 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:33:00.662 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:33:00.659+0000 7f99b65db640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:33:00.662 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:33:00.660+0000 7f99b65db640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:33:00.771 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:33:00.771 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":249,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:31:29.636731+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:30:16.829966+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:27:49.594833+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:30:03.417272+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:33:00.771 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:33:00 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2026-02-23T14:33:00.771 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:33:00 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-23T14:33:00.772 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:33:00 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2026-02-23T14:33:00.772 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:33:00 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T14:33:00.772 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:33:00 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2026-02-23T14:33:01.464 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4775 8 365281 365273 0.597587 0 - 0.0299513 2026-02-23T14:33:02.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:33:01 trial068 ceph-mon[38805]: pgmap v2927: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:33:02.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:33:01 trial068 ceph-mon[38805]: pgmap v2928: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:33:02.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:33:01 trial068 ceph-mon[38805]: pgmap v2929: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:33:02.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:33:01 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2447431584' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:33:02.464 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4776 8 365281 365273 0.597462 0 - 0.0299513 2026-02-23T14:33:03.464 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4777 8 365281 365273 0.597337 0 - 0.0299513 2026-02-23T14:33:03.823 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:33:03.943 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:33:03.963 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:33:03 trial068 ceph-mon[38805]: pgmap v2930: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:33:04.047 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:33:04.045+0000 7f2050766640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:33:04.048 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:33:04.046+0000 7f2050766640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:33:04.157 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:33:04.158 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":249,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:31:29.636731+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:30:16.829966+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:27:49.594833+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:30:03.417272+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:33:04.266 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:33:03 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:33:03.972Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:33:04.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:33:03 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:33:03] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:33:04.464 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4778 8 365281 365273 0.597212 0 - 0.0299513 2026-02-23T14:33:05.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:33:04 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/676309345' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:33:05.464 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4779 8 365281 365273 0.597087 0 - 0.0299513 2026-02-23T14:33:06.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:33:05 trial068 ceph-mon[38805]: pgmap v2931: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:33:06.464 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T14:33:06.464260+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T14:33:06.465 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T14:33:06.465 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4780 8 365281 365273 0.596962 0 - 0.0299513 2026-02-23T14:33:07.214 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:33:07.332 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:33:07.434 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:33:07.432+0000 7fd5ddeda640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:33:07.434 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:33:07.432+0000 7fd5ddeda640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:33:07.465 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4781 8 365281 365273 0.596837 0 - 0.0299513 2026-02-23T14:33:07.549 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:33:07.549 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":249,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:31:29.636731+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:30:16.829966+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:27:49.594833+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:30:03.417272+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:33:07.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:33:07 trial068 ceph-mon[38805]: pgmap v2932: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:33:07.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:33:07 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1212352510' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:33:08.465 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4782 8 365281 365273 0.596712 0 - 0.0299513 2026-02-23T14:33:09.465 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4783 8 365281 365273 0.596587 0 - 0.0299513 2026-02-23T14:33:10.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:33:09 trial068 ceph-mon[38805]: pgmap v2933: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:33:10.465 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4784 8 365281 365273 0.596463 0 - 0.0299513 2026-02-23T14:33:10.589 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:33:10.707 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:33:10.805 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:33:10.803+0000 7fa78a1d3640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:33:10.805 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:33:10.803+0000 7fa78a1d3640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:33:10.916 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:33:10.916 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":249,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:31:29.636731+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:30:16.829966+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:27:49.594833+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:30:03.417272+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:33:11.465 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4785 8 365281 365273 0.596338 0 - 0.0299513 2026-02-23T14:33:12.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:33:11 trial068 ceph-mon[38805]: pgmap v2934: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:33:12.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:33:11 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1618656431' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:33:12.465 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4786 8 365281 365273 0.596213 0 - 0.0299513 2026-02-23T14:33:13.465 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4787 8 365281 365273 0.596089 0 - 0.0299513 2026-02-23T14:33:13.953 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:33:13.973 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:33:13 trial068 ceph-mon[38805]: pgmap v2935: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:33:14.076 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:33:14.177 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:33:14.175+0000 7f6782203640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:33:14.177 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:33:14.175+0000 7f6782203640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:33:14.286 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:33:14.286 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":249,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:31:29.636731+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:30:16.829966+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:27:49.594833+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:30:03.417272+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:33:14.287 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:33:13 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:33:13] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:33:14.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:33:13 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:33:13.972Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:33:14.465 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4788 8 365281 365273 0.595964 0 - 0.0299513 2026-02-23T14:33:15.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:33:14 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2514776618' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:33:15.465 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4789 8 365281 365273 0.59584 0 - 0.0299513 2026-02-23T14:33:16.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:33:15 trial068 ceph-mon[38805]: 5.a scrub starts 2026-02-23T14:33:16.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:33:15 trial068 ceph-mon[38805]: 5.a scrub ok 2026-02-23T14:33:16.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:33:15 trial068 ceph-mon[38805]: pgmap v2936: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:33:16.465 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4790 8 365281 365273 0.595715 0 - 0.0299513 2026-02-23T14:33:17.324 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:33:17.441 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:33:17.465 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4791 8 365281 365273 0.595591 0 - 0.0299513 2026-02-23T14:33:17.543 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:33:17.541+0000 7f45d9078640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:33:17.544 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:33:17.542+0000 7f45d9078640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:33:17.652 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:33:17.652 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":249,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:31:29.636731+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:30:16.829966+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:27:49.594833+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:30:03.417272+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:33:18.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:33:17 trial068 ceph-mon[38805]: pgmap v2937: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:33:18.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:33:17 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2095703411' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:33:18.465 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4792 8 365281 365273 0.595467 0 - 0.0299513 2026-02-23T14:33:19.465 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4793 8 365281 365273 0.595343 0 - 0.0299513 2026-02-23T14:33:20.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:33:19 trial068 ceph-mon[38805]: pgmap v2938: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:33:20.465 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4794 8 365281 365273 0.595218 0 - 0.0299513 2026-02-23T14:33:20.690 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:33:20.809 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:33:20.908 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:33:20.905+0000 7f10bef8d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:33:20.908 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:33:20.906+0000 7f10bef8d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:33:21.015 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:33:21.016 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":249,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:31:29.636731+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:30:16.829966+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:27:49.594833+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:30:03.417272+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:33:21.466 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4795 8 365281 365273 0.595094 0 - 0.0299513 2026-02-23T14:33:22.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:33:21 trial068 ceph-mon[38805]: pgmap v2939: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:33:22.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:33:21 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3184159428' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:33:22.466 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4796 8 365281 365273 0.59497 0 - 0.0299513 2026-02-23T14:33:23.466 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4797 8 365281 365273 0.594846 0 - 0.0299513 2026-02-23T14:33:23.975 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:33:23 trial068 ceph-mon[38805]: pgmap v2940: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:33:24.053 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:33:24.172 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:33:24.229 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:33:23 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:33:23] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:33:24.276 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:33:24.274+0000 7fd3ae53b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:33:24.276 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:33:24.274+0000 7fd3ae53b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:33:24.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:33:23 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:33:23.974Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:33:24.386 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:33:24.387 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":249,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:31:29.636731+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:30:16.829966+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:27:49.594833+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:30:03.417272+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:33:24.466 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4798 8 365281 365273 0.594722 0 - 0.0299513 2026-02-23T14:33:25.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:33:24 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/746422596' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:33:25.466 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4799 8 365281 365273 0.594598 0 - 0.0299513 2026-02-23T14:33:26.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:33:25 trial068 ceph-mon[38805]: pgmap v2941: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:33:26.466 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T14:33:26.465678+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T14:33:26.466 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T14:33:26.466 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4800 8 365281 365273 0.594474 0 - 0.0299513 2026-02-23T14:33:27.423 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:33:27.466 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4801 8 365281 365273 0.59435 0 - 0.0299513 2026-02-23T14:33:27.542 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:33:27.645 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:33:27.642+0000 7f743110e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:33:27.645 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:33:27.643+0000 7f743110e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:33:27.755 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:33:27.756 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":249,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:31:29.636731+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:30:16.829966+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:27:49.594833+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:30:03.417272+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:33:28.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:33:27 trial068 ceph-mon[38805]: pgmap v2942: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:33:28.466 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4802 8 365281 365273 0.594227 0 - 0.0299513 2026-02-23T14:33:29.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:33:28 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3124443411' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:33:29.466 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4803 8 365281 365273 0.594103 0 - 0.0299513 2026-02-23T14:33:30.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:33:29 trial068 ceph-mon[38805]: pgmap v2943: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:33:30.466 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4804 8 365281 365273 0.593979 0 - 0.0299513 2026-02-23T14:33:30.796 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:33:30.912 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:33:31.013 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:33:31.011+0000 7f69253bc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:33:31.013 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:33:31.011+0000 7f69253bc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:33:31.122 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:33:31.123 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":249,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:31:29.636731+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:30:16.829966+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:27:49.594833+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:30:03.417272+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:33:31.466 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4805 8 365281 365273 0.593856 0 - 0.0299513 2026-02-23T14:33:32.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:33:31 trial068 ceph-mon[38805]: pgmap v2944: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:33:32.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:33:31 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/939681094' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:33:32.466 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4806 8 365281 365273 0.593732 0 - 0.0299513 2026-02-23T14:33:33.466 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4807 8 365281 365273 0.593609 0 - 0.0299513 2026-02-23T14:33:33.974 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:33:33 trial068 ceph-mon[38805]: pgmap v2945: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:33:34.160 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:33:34.266 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:33:33 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:33:33.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:33:34.279 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:33:34.301 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:33:33 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:33:33] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:33:34.376 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:33:34.374+0000 7f3492743640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:33:34.376 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:33:34.374+0000 7f3492743640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:33:34.466 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4808 8 365281 365273 0.593485 0 - 0.0299513 2026-02-23T14:33:34.487 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:33:34.488 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":249,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:31:29.636731+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:30:16.829966+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:27:49.594833+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:30:03.417272+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:33:35.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:33:34 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/96544913' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:33:35.467 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4809 8 365281 365273 0.593362 0 - 0.0299513 2026-02-23T14:33:36.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:33:35 trial068 ceph-mon[38805]: pgmap v2946: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:33:36.467 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4810 8 365281 365273 0.593238 0 - 0.0299513 2026-02-23T14:33:37.467 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4811 8 365281 365273 0.593115 0 - 0.0299513 2026-02-23T14:33:37.525 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:33:37.642 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:33:37.742 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:33:37.740+0000 7f424b6da640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:33:37.742 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:33:37.740+0000 7f424b6da640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:33:37.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:33:37 trial068 ceph-mon[38805]: pgmap v2947: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:33:37.851 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:33:37.851 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":249,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:31:29.636731+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:30:16.829966+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:27:49.594833+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:30:03.417272+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:33:38.467 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4812 8 365281 365273 0.592992 0 - 0.0299513 2026-02-23T14:33:39.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:33:38 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/50063991' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:33:39.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:33:38 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T14:33:39.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:33:38 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T14:33:39.467 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4813 8 365281 365273 0.592869 0 - 0.0299513 2026-02-23T14:33:40.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:33:39 trial068 ceph-mon[38805]: pgmap v2948: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:33:40.467 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4814 8 365281 365273 0.592745 0 - 0.0299513 2026-02-23T14:33:40.889 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:33:41.007 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:33:41.104 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:33:41.102+0000 7fd3e9663640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:33:41.105 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:33:41.103+0000 7fd3e9663640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:33:41.215 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:33:41.215 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":249,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:31:29.636731+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:30:16.829966+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:27:49.594833+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:30:03.417272+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:33:41.467 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4815 8 365281 365273 0.592622 0 - 0.0299513 2026-02-23T14:33:42.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:33:41 trial068 ceph-mon[38805]: pgmap v2949: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:33:42.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:33:41 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3192540683' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:33:42.467 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4816 8 365281 365273 0.592499 0 - 0.0299513 2026-02-23T14:33:43.467 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4817 8 365281 365273 0.592376 0 - 0.0299513 2026-02-23T14:33:44.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:33:43 trial068 ceph-mon[38805]: pgmap v2950: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:33:44.094 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:33:43 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:33:43] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:33:44.252 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:33:44.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:33:43 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:33:43.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:33:44.370 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:33:44.467 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4818 8 365281 365273 0.592253 0 - 0.0299513 2026-02-23T14:33:44.468 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:33:44.467+0000 7f1fa1bc6640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:33:44.469 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:33:44.467+0000 7f1fa1bc6640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:33:44.578 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:33:44.578 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":249,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:31:29.636731+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:30:16.829966+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:27:49.594833+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:30:03.417272+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:33:44.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:33:44 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T14:33:44.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:33:44 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T14:33:44.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:33:44 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1794484067' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:33:45.468 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4819 8 365281 365273 0.59213 0 - 0.0299513 2026-02-23T14:33:46.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:33:45 trial068 ceph-mon[38805]: 5.8 scrub starts 2026-02-23T14:33:46.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:33:45 trial068 ceph-mon[38805]: 5.8 scrub ok 2026-02-23T14:33:46.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:33:45 trial068 ceph-mon[38805]: pgmap v2951: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:33:46.468 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T14:33:46.467353+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T14:33:46.468 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T14:33:46.468 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4820 8 365281 365273 0.592008 0 - 0.0299513 2026-02-23T14:33:47.468 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4821 8 365281 365273 0.591885 0 - 0.0299513 2026-02-23T14:33:47.629 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:33:47.750 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:33:47.847 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:33:47.845+0000 7f3c991c1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:33:47.848 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:33:47.846+0000 7f3c991c1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:33:47.959 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:33:47.959 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":249,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:31:29.636731+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:30:16.829966+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:27:49.594833+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:30:03.417272+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:33:47.959 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:33:47 trial068 ceph-mon[38805]: pgmap v2952: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:33:48.468 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4822 8 365281 365273 0.591762 0 - 0.0299513 2026-02-23T14:33:49.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:33:48 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/564940896' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:33:49.468 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4823 8 365281 365273 0.591639 0 - 0.0299513 2026-02-23T14:33:50.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:33:49 trial068 ceph-mon[38805]: pgmap v2953: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:33:50.468 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4824 8 365281 365273 0.591517 0 - 0.0299513 2026-02-23T14:33:50.997 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:33:51.116 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:33:51.219 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:33:51.217+0000 7f0eb7a5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:33:51.219 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:33:51.217+0000 7f0eb7a5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:33:51.327 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:33:51.327 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":249,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:31:29.636731+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:30:16.829966+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:27:49.594833+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:30:03.417272+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:33:51.468 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4825 8 365281 365273 0.591394 0 - 0.0299513 2026-02-23T14:33:52.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:33:51 trial068 ceph-mon[38805]: pgmap v2954: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:33:52.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:33:51 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/145088798' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:33:52.468 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4826 8 365281 365273 0.591272 0 - 0.0299513 2026-02-23T14:33:53.468 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4827 8 365281 365273 0.591149 0 - 0.0299513 2026-02-23T14:33:54.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:33:53 trial068 ceph-mon[38805]: pgmap v2955: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:33:54.094 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:33:53 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:33:53] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:33:54.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:33:53 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:33:53.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:33:54.364 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:33:54.468 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4828 8 365281 365273 0.591027 0 - 0.0299513 2026-02-23T14:33:54.483 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:33:54.583 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:33:54.581+0000 7f08e8c27640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:33:54.583 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:33:54.581+0000 7f08e8c27640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:33:54.693 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:33:54.693 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":249,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:31:29.636731+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:30:16.829966+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:27:49.594833+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:30:03.417272+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:33:55.469 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4829 8 365281 365273 0.590904 0 - 0.0299513 2026-02-23T14:33:56.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:33:55 trial068 ceph-mon[38805]: pgmap v2956: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:33:56.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:33:55 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2780089494' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:33:56.469 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4830 8 365281 365273 0.590782 0 - 0.0299513 2026-02-23T14:33:57.469 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4831 8 365281 365273 0.59066 0 - 0.0299513 2026-02-23T14:33:57.732 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:33:57.851 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:33:57.909 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:33:57 trial068 ceph-mon[38805]: pgmap v2957: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:33:57.951 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:33:57.949+0000 7f6fd6219640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:33:57.951 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:33:57.950+0000 7f6fd6219640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:33:58.060 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:33:58.061 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":249,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:31:29.636731+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:30:16.829966+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:27:49.594833+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:30:03.417272+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:33:58.469 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4832 8 365281 365273 0.590537 0 - 0.0299513 2026-02-23T14:33:59.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:33:58 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1972484702' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:33:59.469 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4833 8 365281 365273 0.590415 0 - 0.0299513 2026-02-23T14:34:00.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:33:59 trial068 ceph-mon[38805]: pgmap v2958: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:34:00.469 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4834 8 365281 365273 0.590293 0 - 0.0299513 2026-02-23T14:34:01.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:34:00 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2026-02-23T14:34:01.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:34:00 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-23T14:34:01.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:34:00 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2026-02-23T14:34:01.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:34:00 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T14:34:01.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:34:00 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2026-02-23T14:34:01.098 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:34:01.215 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:34:01.313 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:34:01.311+0000 7f557c5f6640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:34:01.313 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:34:01.311+0000 7f557c5f6640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:34:01.422 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:34:01.422 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":249,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:31:29.636731+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:30:16.829966+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:27:49.594833+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:30:03.417272+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:34:01.469 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4835 8 365281 365273 0.590171 0 - 0.0299513 2026-02-23T14:34:01.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:34:01 trial068 ceph-mon[38805]: pgmap v2959: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:34:01.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:34:01 trial068 ceph-mon[38805]: pgmap v2960: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:34:01.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:34:01 trial068 ceph-mon[38805]: pgmap v2961: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:34:01.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:34:01 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2951311237' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:34:02.469 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4836 8 365281 365273 0.590049 0 - 0.0299513 2026-02-23T14:34:03.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:34:02 trial068 ceph-mon[38805]: osdmap e250: 4 total, 3 up, 4 in 2026-02-23T14:34:03.469 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4837 8 365281 365273 0.589927 0 - 0.0299513 2026-02-23T14:34:03.973 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:34:03 trial068 ceph-mon[38805]: pgmap v2963: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:34:04.266 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:34:03 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:34:03.972Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:34:04.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:34:03 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:34:03] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:34:04.459 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:34:04.469 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4838 8 365281 365273 0.589805 0 - 0.0299513 2026-02-23T14:34:04.578 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:34:04.682 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:34:04.680+0000 7f4e1a2db640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:34:04.682 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:34:04.680+0000 7f4e1a2db640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:34:04.792 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:34:04.792 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":251,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:34:04.656654+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:30:16.829966+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:33:43.438776+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:30:03.417272+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:34:04.793 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:34:04 trial068 ceph-mon[38805]: 5.e scrub starts 2026-02-23T14:34:04.793 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:34:04 trial068 ceph-mon[38805]: 5.e scrub ok 2026-02-23T14:34:04.793 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:34:04 trial068 ceph-mon[38805]: osdmap e251: 4 total, 3 up, 4 in 2026-02-23T14:34:05.469 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4839 8 365281 365273 0.589683 0 - 0.0299513 2026-02-23T14:34:06.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:34:05 trial068 ceph-mon[38805]: pgmap v2964: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:34:06.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:34:05 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2076566818' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:34:06.469 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T14:34:06.469141+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T14:34:06.470 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T14:34:06.470 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4840 8 365281 365273 0.589561 0 - 0.0299513 2026-02-23T14:34:07.470 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4841 8 365281 365273 0.589439 0 - 0.0299513 2026-02-23T14:34:07.830 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:34:07.949 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:34:07.970 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:34:07 trial068 ceph-mon[38805]: pgmap v2966: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:34:07.971 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:34:07 trial068 ceph-mon[38805]: 5.7 scrub starts 2026-02-23T14:34:07.971 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:34:07 trial068 ceph-mon[38805]: 5.7 scrub ok 2026-02-23T14:34:08.046 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:34:08.044+0000 7f666abcb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:34:08.047 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:34:08.045+0000 7f666abcb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:34:08.155 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:34:08.155 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":251,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:34:04.656654+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:30:16.829966+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:33:43.438776+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:30:03.417272+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:34:08.470 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4842 8 365281 365273 0.589318 0 - 0.0299513 2026-02-23T14:34:09.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:34:08 trial068 ceph-mon[38805]: 5.5 scrub starts 2026-02-23T14:34:09.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:34:08 trial068 ceph-mon[38805]: 5.5 scrub ok 2026-02-23T14:34:09.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:34:08 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/795458034' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:34:09.470 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4843 8 365281 365273 0.589196 0 - 0.0299513 2026-02-23T14:34:10.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:34:09 trial068 ceph-mon[38805]: pgmap v2967: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:34:10.470 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4844 8 365281 365273 0.589074 0 - 0.0299513 2026-02-23T14:34:11.191 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:34:11.310 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:34:11.414 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:34:11.412+0000 7f9b82fdf640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:34:11.414 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:34:11.412+0000 7f9b82fdf640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:34:11.470 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4845 8 365281 365273 0.588953 0 - 0.0299513 2026-02-23T14:34:11.524 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:34:11.524 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":251,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:34:04.656654+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:30:16.829966+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:33:43.438776+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:30:03.417272+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:34:11.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:34:11 trial068 ceph-mon[38805]: pgmap v2968: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:34:11.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:34:11 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1666774260' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:34:12.470 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4846 8 365281 365273 0.588831 0 - 0.0299513 2026-02-23T14:34:13.470 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4847 8 365281 365273 0.58871 0 - 0.0299513 2026-02-23T14:34:13.974 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:34:13 trial068 ceph-mon[38805]: pgmap v2969: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:34:14.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:34:13 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:34:13] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:34:14.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:34:13 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:34:13.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:34:14.470 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4848 8 365281 365273 0.588588 0 - 0.0299513 2026-02-23T14:34:14.562 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:34:14.680 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:34:14.779 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:34:14.777+0000 7fc0c09e5640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:34:14.779 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:34:14.777+0000 7fc0c09e5640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:34:14.890 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:34:14.891 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":251,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:34:04.656654+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:30:16.829966+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:33:43.438776+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:30:03.417272+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:34:15.470 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4849 8 365281 365273 0.588467 0 - 0.0299513 2026-02-23T14:34:16.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:34:15 trial068 ceph-mon[38805]: pgmap v2970: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:34:16.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:34:15 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T14:34:16.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:34:15 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T14:34:16.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:34:15 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/738596700' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:34:16.470 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4850 8 365281 365273 0.588346 0 - 0.0299513 2026-02-23T14:34:17.470 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4851 8 365281 365273 0.588224 0 - 0.0299513 2026-02-23T14:34:17.928 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:34:17.952 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:34:17 trial068 ceph-mon[38805]: pgmap v2971: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:34:18.046 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:34:18.147 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:34:18.144+0000 7fe1b5444640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:34:18.147 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:34:18.145+0000 7fe1b5444640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:34:18.255 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:34:18.255 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":251,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:34:04.656654+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:30:16.829966+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:33:43.438776+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:30:03.417272+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:34:18.470 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4852 8 365281 365273 0.588103 0 - 0.0299513 2026-02-23T14:34:19.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:34:18 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2688406777' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:34:19.470 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4853 8 365281 365273 0.587982 0 - 0.0299513 2026-02-23T14:34:20.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:34:19 trial068 ceph-mon[38805]: pgmap v2972: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:34:20.470 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4854 8 365281 365273 0.587861 0 - 0.0299513 2026-02-23T14:34:21.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:34:20 trial068 ceph-mon[38805]: 5.a scrub starts 2026-02-23T14:34:21.294 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:34:21.409 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:34:21.471 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4855 8 365281 365273 0.58774 0 - 0.0299513 2026-02-23T14:34:21.505 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:34:21.503+0000 7faf9ffbd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:34:21.505 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:34:21.503+0000 7faf9ffbd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:34:21.614 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:34:21.614 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":251,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:34:04.656654+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:30:16.829966+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:33:43.438776+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:30:03.417272+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:34:22.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:34:21 trial068 ceph-mon[38805]: 5.a scrub ok 2026-02-23T14:34:22.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:34:21 trial068 ceph-mon[38805]: pgmap v2973: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:34:22.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:34:21 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/4235401171' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:34:22.471 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4856 8 365281 365273 0.587619 0 - 0.0299513 2026-02-23T14:34:23.471 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4857 8 365281 365273 0.587498 0 - 0.0299513 2026-02-23T14:34:23.974 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:34:23 trial068 ceph-mon[38805]: pgmap v2974: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:34:24.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:34:23 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:34:23] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:34:24.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:34:23 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:34:23.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:34:24.471 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4858 8 365281 365273 0.587377 0 - 0.0299513 2026-02-23T14:34:24.651 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:34:24.769 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:34:24.870 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:34:24.868+0000 7fb465832640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:34:24.871 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:34:24.868+0000 7fb465832640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:34:24.979 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:34:24.979 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":251,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:34:04.656654+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:30:16.829966+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:33:43.438776+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:30:03.417272+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:34:25.471 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4859 8 365281 365273 0.587256 0 - 0.0299513 2026-02-23T14:34:26.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:34:25 trial068 ceph-mon[38805]: pgmap v2975: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:34:26.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:34:25 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3653010158' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:34:26.471 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T14:34:26.470534+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T14:34:26.471 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T14:34:26.471 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4860 8 365281 365273 0.587135 0 - 0.0299513 2026-02-23T14:34:27.471 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4861 8 365281 365273 0.587014 0 - 0.0299513 2026-02-23T14:34:28.015 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:34:28.041 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:34:27 trial068 ceph-mon[38805]: pgmap v2976: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:34:28.134 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:34:28.231 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:34:28.229+0000 7fd70d1f1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:34:28.231 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:34:28.229+0000 7fd70d1f1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:34:28.343 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:34:28.343 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":251,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:34:04.656654+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:30:16.829966+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:33:43.438776+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:30:03.417272+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:34:28.471 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4862 8 365281 365273 0.586894 0 - 0.0299513 2026-02-23T14:34:29.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:34:28 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1980862423' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:34:29.471 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4863 8 365281 365273 0.586773 0 - 0.0299513 2026-02-23T14:34:30.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:34:29 trial068 ceph-mon[38805]: pgmap v2977: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:34:30.471 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4864 8 365281 365273 0.586652 0 - 0.0299513 2026-02-23T14:34:31.381 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:34:31.471 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4865 8 365281 365273 0.586532 0 - 0.0299513 2026-02-23T14:34:31.499 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:34:31.600 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:34:31.598+0000 7f1abfb08640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:34:31.600 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:34:31.598+0000 7f1abfb08640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:34:31.709 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:34:31.709 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":251,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:34:04.656654+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:30:16.829966+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:33:43.438776+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:30:03.417272+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:34:32.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:34:31 trial068 ceph-mon[38805]: pgmap v2978: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:34:32.471 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4866 8 365281 365273 0.586411 0 - 0.0299513 2026-02-23T14:34:33.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:34:32 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/107170774' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:34:33.471 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4867 8 365281 365273 0.586291 0 - 0.0299513 2026-02-23T14:34:33.974 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:34:33 trial068 ceph-mon[38805]: pgmap v2979: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:34:34.266 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:34:33 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:34:33.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:34:34.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:34:33 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:34:33] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:34:34.471 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4868 8 365281 365273 0.58617 0 - 0.0299513 2026-02-23T14:34:34.747 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:34:34.863 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:34:34.967 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:34:34.965+0000 7fb0f6d3c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:34:34.967 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:34:34.965+0000 7fb0f6d3c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:34:35.078 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:34:35.078 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":251,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:34:04.656654+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:30:16.829966+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:33:43.438776+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:30:03.417272+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:34:35.472 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4869 8 365281 365273 0.58605 0 - 0.0299513 2026-02-23T14:34:36.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:34:35 trial068 ceph-mon[38805]: pgmap v2980: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:34:36.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:34:35 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2511506015' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:34:36.472 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4870 8 365281 365273 0.585929 0 - 0.0299513 2026-02-23T14:34:37.472 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4871 8 365281 365273 0.585809 0 - 0.0299513 2026-02-23T14:34:38.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:34:37 trial068 ceph-mon[38805]: pgmap v2981: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:34:38.116 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:34:38.232 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:34:38.331 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:34:38.329+0000 7febe8f1e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:34:38.331 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:34:38.329+0000 7febe8f1e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:34:38.440 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:34:38.440 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":251,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:34:04.656654+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:30:16.829966+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:33:43.438776+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:30:03.417272+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:34:38.472 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4872 8 365281 365273 0.585689 0 - 0.0299513 2026-02-23T14:34:39.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:34:38 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/815451292' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:34:39.472 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4873 8 365281 365273 0.585569 0 - 0.0299513 2026-02-23T14:34:40.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:34:39 trial068 ceph-mon[38805]: pgmap v2982: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:34:40.472 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4874 8 365281 365273 0.585449 0 - 0.0299513 2026-02-23T14:34:41.472 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4875 8 365281 365273 0.585329 0 - 0.0299513 2026-02-23T14:34:41.476 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:34:41.594 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:34:41.709 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:34:41.707+0000 7fb824051640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:34:41.709 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:34:41.707+0000 7fb824051640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:34:41.819 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:34:41.819 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":251,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:34:04.656654+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:30:16.829966+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:33:43.438776+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:30:03.417272+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:34:41.820 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:34:41 trial068 ceph-mon[38805]: pgmap v2983: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:34:42.472 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4876 8 365281 365273 0.585208 0 - 0.0299513 2026-02-23T14:34:43.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:34:42 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1840163457' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:34:43.472 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4877 8 365281 365273 0.585088 0 - 0.0299513 2026-02-23T14:34:43.974 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:34:43 trial068 ceph-mon[38805]: pgmap v2984: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:34:44.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:34:43 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:34:43] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:34:44.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:34:43 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:34:43.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:34:44.472 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4878 8 365281 365273 0.584969 0 - 0.0299513 2026-02-23T14:34:44.856 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:34:44.973 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:34:45.073 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:34:45.071+0000 7f980e104640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:34:45.073 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:34:45.071+0000 7f980e104640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:34:45.183 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:34:45.184 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":251,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:34:04.656654+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:30:16.829966+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:33:43.438776+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:30:03.417272+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:34:45.472 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4879 8 365281 365273 0.584849 0 - 0.0299513 2026-02-23T14:34:46.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:34:45 trial068 ceph-mon[38805]: pgmap v2985: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:34:46.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:34:45 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1268436263' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:34:46.472 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T14:34:46.472092+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T14:34:46.473 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T14:34:46.473 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4880 8 365281 365273 0.584729 0 - 0.0299513 2026-02-23T14:34:47.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:34:46 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T14:34:47.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:34:46 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T14:34:47.472 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4881 8 365281 365273 0.584609 0 - 0.0299513 2026-02-23T14:34:48.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:34:47 trial068 ceph-mon[38805]: pgmap v2986: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:34:48.222 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:34:48.339 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:34:48.446 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:34:48.444+0000 7fb8b7bb0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:34:48.446 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:34:48.444+0000 7fb8b7bb0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:34:48.472 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4882 8 365281 365273 0.584489 0 - 0.0299513 2026-02-23T14:34:48.556 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:34:48.557 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":251,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:34:04.656654+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:30:16.829966+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:33:43.438776+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:30:03.417272+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:34:48.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:34:48 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/917860272' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:34:49.472 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4883 8 365281 365273 0.58437 0 - 0.0299513 2026-02-23T14:34:50.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:34:49 trial068 ceph-mon[38805]: pgmap v2987: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:34:50.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:34:49 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T14:34:50.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:34:49 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T14:34:50.473 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4884 8 365281 365273 0.58425 0 - 0.0299513 2026-02-23T14:34:51.473 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4885 8 365281 365273 0.58413 0 - 0.0299513 2026-02-23T14:34:51.595 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:34:51.713 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:34:51.815 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:34:51.813+0000 7fc02b854640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:34:51.815 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:34:51.813+0000 7fc02b854640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:34:51.924 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:34:51.924 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":251,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:34:04.656654+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:30:16.829966+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:33:43.438776+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:30:03.417272+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:34:51.925 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:34:51 trial068 ceph-mon[38805]: pgmap v2988: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:34:52.473 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4886 8 365281 365273 0.584011 0 - 0.0299513 2026-02-23T14:34:53.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:34:52 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/813661107' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:34:53.473 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4887 8 365281 365273 0.583891 0 - 0.0299513 2026-02-23T14:34:53.974 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:34:53 trial068 ceph-mon[38805]: pgmap v2989: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:34:54.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:34:53 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:34:53] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:34:54.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:34:53 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:34:53.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:34:54.473 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4888 8 365281 365273 0.583772 0 - 0.0299513 2026-02-23T14:34:54.961 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:34:55.081 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:34:55.180 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:34:55.178+0000 7f4ce5f54640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:34:55.180 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:34:55.178+0000 7f4ce5f54640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:34:55.291 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:34:55.291 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":251,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:34:04.656654+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:30:16.829966+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:33:43.438776+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:30:03.417272+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:34:55.473 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4889 8 365281 365273 0.583652 0 - 0.0299513 2026-02-23T14:34:56.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:34:55 trial068 ceph-mon[38805]: pgmap v2990: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:34:56.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:34:55 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1546065440' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:34:56.473 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4890 8 365281 365273 0.583533 0 - 0.0299513 2026-02-23T14:34:57.473 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4891 8 365281 365273 0.583414 0 - 0.0299513 2026-02-23T14:34:58.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:34:57 trial068 ceph-mon[38805]: pgmap v2991: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:34:58.328 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:34:58.446 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:34:58.473 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4892 8 365281 365273 0.583294 0 - 0.0299513 2026-02-23T14:34:58.551 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:34:58.549+0000 7fc26c019640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:34:58.551 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:34:58.549+0000 7fc26c019640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:34:58.661 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:34:58.662 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":251,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:34:04.656654+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:30:16.829966+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:33:43.438776+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:30:03.417272+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:34:59.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:34:58 trial068 ceph-mon[38805]: 5.8 scrub starts 2026-02-23T14:34:59.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:34:58 trial068 ceph-mon[38805]: 5.8 scrub ok 2026-02-23T14:34:59.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:34:58 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1049516552' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:34:59.473 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4893 8 365281 365273 0.583175 0 - 0.0299513 2026-02-23T14:35:00.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:34:59 trial068 ceph-mon[38805]: pgmap v2992: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:35:00.473 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4894 8 365281 365273 0.583056 0 - 0.0299513 2026-02-23T14:35:01.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:35:00 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2026-02-23T14:35:01.474 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4895 8 365281 365273 0.582937 0 - 0.0299513 2026-02-23T14:35:01.699 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:35:01.815 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:35:01.915 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:35:01.913+0000 7f7cb241d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:35:01.915 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:35:01.913+0000 7f7cb241d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:35:02.023 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:35:02.023 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":251,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:34:04.656654+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:30:16.829966+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:33:43.438776+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:30:03.417272+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:35:02.024 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:35:01 trial068 ceph-mon[38805]: pgmap v2993: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:35:02.024 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:35:01 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-23T14:35:02.024 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:35:01 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2026-02-23T14:35:02.024 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:35:01 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T14:35:02.024 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:35:01 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2026-02-23T14:35:02.474 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4896 8 365281 365273 0.582818 0 - 0.0299513 2026-02-23T14:35:03.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:35:02 trial068 ceph-mon[38805]: pgmap v2994: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:35:03.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:35:02 trial068 ceph-mon[38805]: pgmap v2995: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:35:03.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:35:02 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/583453203' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:35:03.474 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4897 8 365281 365273 0.582699 0 - 0.0299513 2026-02-23T14:35:04.265 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:35:03 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:35:03.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:35:04.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:35:03 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:35:03] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:35:04.474 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4898 8 365281 365273 0.58258 0 - 0.0299513 2026-02-23T14:35:05.060 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:35:05.084 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:35:04 trial068 ceph-mon[38805]: pgmap v2996: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:35:05.179 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:35:05.287 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:35:05.285+0000 7fd4a9de6640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:35:05.288 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:35:05.286+0000 7fd4a9de6640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:35:05.396 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:35:05.396 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":251,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:34:04.656654+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:30:16.829966+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:33:43.438776+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:30:03.417272+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:35:05.474 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4899 8 365281 365273 0.582461 0 - 0.0299513 2026-02-23T14:35:06.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:35:05 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/593042030' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:35:06.474 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T14:35:06.473631+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T14:35:06.474 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T14:35:06.474 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4900 8 365281 365273 0.582342 0 - 0.0299513 2026-02-23T14:35:07.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:35:06 trial068 ceph-mon[38805]: pgmap v2997: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:35:07.474 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4901 8 365281 365273 0.582223 0 - 0.0299513 2026-02-23T14:35:08.434 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:35:08.474 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4902 8 365281 365273 0.582105 0 - 0.0299513 2026-02-23T14:35:08.555 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:35:08.660 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:35:08.658+0000 7f073f806640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:35:08.660 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:35:08.658+0000 7f073f806640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:35:08.770 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:35:08.771 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":251,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:34:04.656654+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:30:16.829966+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:33:43.438776+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:30:03.417272+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:35:09.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:35:08 trial068 ceph-mon[38805]: pgmap v2998: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:35:09.474 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4903 8 365281 365273 0.581986 0 - 0.0299513 2026-02-23T14:35:10.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:35:09 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1697333974' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:35:10.474 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4904 8 365281 365273 0.581867 0 - 0.0299513 2026-02-23T14:35:11.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:35:10 trial068 ceph-mon[38805]: pgmap v2999: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:35:11.474 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4905 8 365281 365273 0.581749 0 - 0.0299513 2026-02-23T14:35:11.810 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:35:11.926 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:35:12.024 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:35:12.022+0000 7fe39162e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:35:12.024 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:35:12.022+0000 7fe39162e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:35:12.134 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:35:12.135 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":251,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:34:04.656654+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:30:16.829966+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:33:43.438776+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:30:03.417272+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:35:12.474 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4906 8 365281 365273 0.58163 0 - 0.0299513 2026-02-23T14:35:13.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:35:12 trial068 ceph-mon[38805]: pgmap v3000: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:35:13.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:35:12 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/175799649' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:35:13.474 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4907 8 365281 365273 0.581511 0 - 0.0299513 2026-02-23T14:35:14.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:35:13 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:35:13] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:35:14.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:35:13 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:35:13.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:35:14.474 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4908 8 365281 365273 0.581393 0 - 0.0299513 2026-02-23T14:35:15.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:35:14 trial068 ceph-mon[38805]: pgmap v3001: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:35:15.172 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:35:15.288 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:35:15.397 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:35:15.395+0000 7f2eb597f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:35:15.397 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:35:15.396+0000 7f2eb597f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:35:15.474 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4909 8 365281 365273 0.581275 0 - 0.0299513 2026-02-23T14:35:15.507 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:35:15.507 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":251,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:34:04.656654+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:30:16.829966+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:33:43.438776+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:30:03.417272+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:35:15.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:35:15 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3968818627' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:35:16.475 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4910 8 365281 365273 0.581156 0 - 0.0299513 2026-02-23T14:35:17.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:35:16 trial068 ceph-mon[38805]: pgmap v3002: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:35:17.475 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4911 8 365281 365273 0.581038 0 - 0.0299513 2026-02-23T14:35:18.475 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4912 8 365281 365273 0.580919 0 - 0.0299513 2026-02-23T14:35:18.560 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:35:18.679 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:35:18.777 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:35:18.775+0000 7fe1c37f7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:35:18.777 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:35:18.775+0000 7fe1c37f7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:35:18.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:35:18 trial068 ceph-mon[38805]: pgmap v3003: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:35:18.884 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:35:18.885 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":251,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:34:04.656654+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:30:16.829966+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:33:43.438776+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:30:03.417272+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:35:19.475 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4913 8 365281 365273 0.580801 0 - 0.0299513 2026-02-23T14:35:20.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:35:19 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/792139109' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:35:20.475 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4914 8 365281 365273 0.580683 0 - 0.0299513 2026-02-23T14:35:21.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:35:20 trial068 ceph-mon[38805]: pgmap v3004: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:35:21.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:35:20 trial068 ceph-mon[38805]: osdmap e252: 4 total, 3 up, 4 in 2026-02-23T14:35:21.475 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4915 8 365281 365273 0.580565 0 - 0.0299513 2026-02-23T14:35:21.921 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:35:22.036 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:35:22.140 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:35:22.138+0000 7f495988c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:35:22.140 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:35:22.138+0000 7f495988c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:35:22.250 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:35:22.250 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":252,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:35:19.736230+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:30:16.829966+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:33:43.438776+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:34:48.677580+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:35:22.475 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4916 8 365281 365273 0.580447 0 - 0.0299513 2026-02-23T14:35:23.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:35:22 trial068 ceph-mon[38805]: pgmap v3006: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:35:23.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:35:22 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/783998819' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:35:23.475 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4917 8 365281 365273 0.580329 0 - 0.0299513 2026-02-23T14:35:24.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:35:23 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:35:23] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:35:24.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:35:23 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:35:23.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:35:24.475 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4918 8 365281 365273 0.580211 0 - 0.0299513 2026-02-23T14:35:25.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:35:24 trial068 ceph-mon[38805]: pgmap v3007: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:35:25.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:35:24 trial068 ceph-mon[38805]: osdmap e253: 4 total, 3 up, 4 in 2026-02-23T14:35:25.291 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:35:25.409 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:35:25.475 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4919 8 365281 365273 0.580093 0 - 0.0299513 2026-02-23T14:35:25.508 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:35:25.506+0000 7fad1b36b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:35:25.508 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:35:25.506+0000 7fad1b36b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:35:25.618 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:35:25.618 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":253,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:35:24.666678+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:30:16.829966+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:33:43.438776+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:34:48.677580+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:35:26.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:35:25 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/4178763433' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:35:26.475 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T14:35:26.474974+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T14:35:26.476 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T14:35:26.476 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4920 8 365281 365273 0.579975 0 - 0.0299513 2026-02-23T14:35:27.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:35:26 trial068 ceph-mon[38805]: pgmap v3009: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:35:27.475 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4921 8 365281 365273 0.579857 0 - 0.0299513 2026-02-23T14:35:28.475 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4922 8 365281 365273 0.579739 0 - 0.0299513 2026-02-23T14:35:28.654 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:35:28.770 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:35:28.869 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:35:28.866+0000 7f1f3aaaa640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:35:28.869 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:35:28.867+0000 7f1f3aaaa640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:35:28.978 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:35:28.978 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":253,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:35:24.666678+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:30:16.829966+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:33:43.438776+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:34:48.677580+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:35:28.978 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:35:28 trial068 ceph-mon[38805]: pgmap v3010: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:35:28.979 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:35:28 trial068 ceph-mon[38805]: 5.a scrub starts 2026-02-23T14:35:28.979 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:35:28 trial068 ceph-mon[38805]: 5.a scrub ok 2026-02-23T14:35:29.476 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4923 8 365281 365273 0.579621 0 - 0.0299513 2026-02-23T14:35:30.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:35:29 trial068 ceph-mon[38805]: 5.7 scrub starts 2026-02-23T14:35:30.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:35:29 trial068 ceph-mon[38805]: 5.7 scrub ok 2026-02-23T14:35:30.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:35:29 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/4134979180' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:35:30.476 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4924 8 365281 365273 0.579504 0 - 0.0299513 2026-02-23T14:35:31.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:35:30 trial068 ceph-mon[38805]: pgmap v3011: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:35:31.476 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4925 8 365281 365273 0.579386 0 - 0.0299513 2026-02-23T14:35:32.015 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:35:32.040 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:35:31 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T14:35:32.040 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:35:31 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T14:35:32.132 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:35:32.227 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:35:32.225+0000 7faf904eb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:35:32.228 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:35:32.226+0000 7faf904eb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:35:32.336 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:35:32.336 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":253,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:35:24.666678+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:30:16.829966+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:33:43.438776+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:34:48.677580+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:35:32.476 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4926 8 365281 365273 0.579268 0 - 0.0299513 2026-02-23T14:35:33.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:35:32 trial068 ceph-mon[38805]: pgmap v3012: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:35:33.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:35:32 trial068 ceph-mon[38805]: 5.5 scrub starts 2026-02-23T14:35:33.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:35:32 trial068 ceph-mon[38805]: 5.5 scrub ok 2026-02-23T14:35:33.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:35:32 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1173305659' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:35:33.476 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4927 8 365281 365273 0.579151 0 - 0.0299513 2026-02-23T14:35:34.266 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:35:33 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:35:33.974Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:35:34.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:35:33 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:35:33] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:35:34.476 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4928 8 365281 365273 0.579033 0 - 0.0299513 2026-02-23T14:35:35.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:35:34 trial068 ceph-mon[38805]: pgmap v3013: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:35:35.393 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:35:35.476 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4929 8 365281 365273 0.578916 0 - 0.0299513 2026-02-23T14:35:35.511 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:35:35.607 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:35:35.605+0000 7f78cd4e2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:35:35.607 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:35:35.605+0000 7f78cd4e2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:35:35.717 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:35:35.718 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":253,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:35:24.666678+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:30:16.829966+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:33:43.438776+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:34:48.677580+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:35:36.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:35:35 trial068 ceph-mon[38805]: 5.e scrub starts 2026-02-23T14:35:36.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:35:35 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3756747370' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:35:36.476 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4930 8 365281 365273 0.578799 0 - 0.0299513 2026-02-23T14:35:37.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:35:36 trial068 ceph-mon[38805]: pgmap v3014: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:35:37.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:35:36 trial068 ceph-mon[38805]: 5.e scrub ok 2026-02-23T14:35:37.476 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4931 8 365281 365273 0.578681 0 - 0.0299513 2026-02-23T14:35:38.476 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4932 8 365281 365273 0.578564 0 - 0.0299513 2026-02-23T14:35:38.755 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:35:38.872 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:35:38.968 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:35:38.966+0000 7f36af642640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:35:38.968 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:35:38.966+0000 7f36af642640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:35:39.076 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:35:39.076 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":253,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:35:24.666678+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:30:16.829966+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:33:43.438776+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:34:48.677580+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:35:39.077 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:35:38 trial068 ceph-mon[38805]: pgmap v3015: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:35:39.476 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4933 8 365281 365273 0.578447 0 - 0.0299513 2026-02-23T14:35:40.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:35:39 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3093727589' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:35:40.476 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4934 8 365281 365273 0.578329 0 - 0.0299513 2026-02-23T14:35:41.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:35:40 trial068 ceph-mon[38805]: pgmap v3016: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:35:41.476 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4935 8 365281 365273 0.578212 0 - 0.0299513 2026-02-23T14:35:42.113 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:35:42.232 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:35:42.339 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:35:42.337+0000 7f6ab6822640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:35:42.340 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:35:42.338+0000 7f6ab6822640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:35:42.448 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:35:42.448 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":253,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:35:24.666678+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:30:16.829966+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:33:43.438776+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:34:48.677580+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:35:42.476 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4936 8 365281 365273 0.578095 0 - 0.0299513 2026-02-23T14:35:43.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:35:42 trial068 ceph-mon[38805]: pgmap v3017: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:35:43.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:35:42 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/750309148' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:35:43.476 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4937 8 365281 365273 0.577978 0 - 0.0299513 2026-02-23T14:35:44.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:35:43 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:35:43] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:35:44.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:35:43 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:35:43.974Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:35:44.476 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4938 8 365281 365273 0.577861 0 - 0.0299513 2026-02-23T14:35:45.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:35:44 trial068 ceph-mon[38805]: pgmap v3018: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:35:45.476 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4939 8 365281 365273 0.577744 0 - 0.0299513 2026-02-23T14:35:45.485 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:35:45.603 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:35:45.709 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:35:45.707+0000 7f98c9c2a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:35:45.710 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:35:45.708+0000 7f98c9c2a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:35:45.819 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:35:45.819 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":253,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:35:24.666678+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:30:16.829966+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:33:43.438776+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:34:48.677580+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:35:46.477 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T14:35:46.476422+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T14:35:46.477 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T14:35:46.477 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4940 8 365281 365273 0.577627 0 - 0.0299513 2026-02-23T14:35:47.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:35:46 trial068 ceph-mon[38805]: pgmap v3019: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:35:47.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:35:46 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2275117021' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:35:47.477 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4941 8 365281 365273 0.57751 0 - 0.0299513 2026-02-23T14:35:48.477 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4942 8 365281 365273 0.577393 0 - 0.0299513 2026-02-23T14:35:48.858 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:35:48.980 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:35:49.024 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:35:48 trial068 ceph-mon[38805]: pgmap v3020: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:35:49.082 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:35:49.080+0000 7fca4954e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:35:49.082 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:35:49.080+0000 7fca4954e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:35:49.192 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:35:49.192 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":253,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:35:24.666678+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:30:16.829966+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:33:43.438776+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:34:48.677580+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:35:49.477 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4943 8 365281 365273 0.577276 0 - 0.0299513 2026-02-23T14:35:50.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:35:49 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/515979757' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:35:50.477 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4944 8 365281 365273 0.57716 0 - 0.0299513 2026-02-23T14:35:51.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:35:50 trial068 ceph-mon[38805]: pgmap v3021: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:35:51.477 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4945 8 365281 365273 0.577043 0 - 0.0299513 2026-02-23T14:35:52.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:35:51 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T14:35:52.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:35:51 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T14:35:52.230 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:35:52.348 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:35:52.447 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:35:52.445+0000 7f384c029640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:35:52.447 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:35:52.445+0000 7f384c029640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:35:52.477 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4946 8 365281 365273 0.576926 0 - 0.0299513 2026-02-23T14:35:52.558 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:35:52.559 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":253,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:35:24.666678+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:30:16.829966+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:33:43.438776+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:34:48.677580+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:35:52.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:35:52 trial068 ceph-mon[38805]: pgmap v3022: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:35:52.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:35:52 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2257190559' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:35:53.477 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4947 8 365281 365273 0.57681 0 - 0.0299513 2026-02-23T14:35:54.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:35:53 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:35:53] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:35:54.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:35:53 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:35:53.974Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:35:54.477 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4948 8 365281 365273 0.576693 0 - 0.0299513 2026-02-23T14:35:55.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:35:54 trial068 ceph-mon[38805]: pgmap v3023: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:35:55.477 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4949 8 365281 365273 0.576576 0 - 0.0299513 2026-02-23T14:35:55.596 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:35:55.713 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:35:55.811 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:35:55.809+0000 7f09c26b6640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:35:55.811 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:35:55.809+0000 7f09c26b6640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:35:55.920 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:35:55.921 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":253,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:35:24.666678+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:30:16.829966+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:33:43.438776+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:34:48.677580+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:35:56.477 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4950 8 365281 365273 0.57646 0 - 0.0299513 2026-02-23T14:35:57.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:35:56 trial068 ceph-mon[38805]: pgmap v3024: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:35:57.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:35:56 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2139006282' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:35:57.477 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4951 8 365281 365273 0.576344 0 - 0.0299513 2026-02-23T14:35:58.477 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4952 8 365281 365273 0.576227 0 - 0.0299513 2026-02-23T14:35:58.957 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:35:59.075 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:35:59.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:35:58 trial068 ceph-mon[38805]: pgmap v3025: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:35:59.175 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:35:59.173+0000 7f9ece750640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:35:59.175 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:35:59.174+0000 7f9ece750640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:35:59.284 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:35:59.285 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":253,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:35:24.666678+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:30:16.829966+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:33:43.438776+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:34:48.677580+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:35:59.478 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4953 8 365281 365273 0.576111 0 - 0.0299513 2026-02-23T14:36:00.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:35:59 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/538701094' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:36:00.478 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4954 8 365281 365273 0.575994 0 - 0.0299513 2026-02-23T14:36:01.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:36:00 trial068 ceph-mon[38805]: pgmap v3026: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:36:01.478 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4955 8 365281 365273 0.575878 0 - 0.0299513 2026-02-23T14:36:02.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:36:01 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T14:36:02.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:36:01 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T14:36:02.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:36:01 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2026-02-23T14:36:02.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:36:01 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-23T14:36:02.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:36:01 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2026-02-23T14:36:02.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:36:01 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T14:36:02.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:36:01 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2026-02-23T14:36:02.338 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:36:02.454 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:36:02.478 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4956 8 365281 365273 0.575762 0 - 0.0299513 2026-02-23T14:36:02.555 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:36:02.553+0000 7f37c360e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:36:02.556 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:36:02.554+0000 7f37c360e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:36:02.670 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:36:02.670 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":253,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:35:24.666678+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:30:16.829966+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:33:43.438776+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:34:48.677580+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:36:03.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:36:02 trial068 ceph-mon[38805]: pgmap v3027: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:36:03.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:36:02 trial068 ceph-mon[38805]: pgmap v3028: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:36:03.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:36:02 trial068 ceph-mon[38805]: pgmap v3029: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:36:03.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:36:02 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1411823860' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:36:03.478 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4957 8 365281 365273 0.575646 0 - 0.0299513 2026-02-23T14:36:04.266 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:36:03 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:36:03.972Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:36:04.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:36:03 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:36:03] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:36:04.478 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4958 8 365281 365273 0.57553 0 - 0.0299513 2026-02-23T14:36:05.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:36:04 trial068 ceph-mon[38805]: pgmap v3030: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:36:05.478 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4959 8 365281 365273 0.575414 0 - 0.0299513 2026-02-23T14:36:05.707 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:36:05.826 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:36:05.927 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:36:05.925+0000 7f62069f6640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:36:05.928 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:36:05.926+0000 7f62069f6640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:36:06.037 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:36:06.037 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":253,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:35:24.666678+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:30:16.829966+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:33:43.438776+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:34:48.677580+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:36:06.478 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T14:36:06.477780+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T14:36:06.479 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T14:36:06.479 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4960 8 365281 365273 0.575298 0 - 0.0299513 2026-02-23T14:36:07.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:36:06 trial068 ceph-mon[38805]: pgmap v3031: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:36:07.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:36:06 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2557772244' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:36:07.478 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4961 8 365281 365273 0.575182 0 - 0.0299513 2026-02-23T14:36:08.478 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4962 8 365281 365273 0.575066 0 - 0.0299513 2026-02-23T14:36:09.075 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:36:09.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:36:08 trial068 ceph-mon[38805]: pgmap v3032: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:36:09.194 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:36:09.296 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:36:09.294+0000 7fce73054640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:36:09.297 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:36:09.295+0000 7fce73054640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:36:09.406 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:36:09.406 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":253,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:35:24.666678+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:30:16.829966+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:33:43.438776+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:34:48.677580+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:36:09.478 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4963 8 365281 365273 0.57495 0 - 0.0299513 2026-02-23T14:36:10.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:36:09 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3865395897' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:36:10.478 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4964 8 365281 365273 0.574834 0 - 0.0299513 2026-02-23T14:36:11.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:36:10 trial068 ceph-mon[38805]: pgmap v3033: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:36:11.478 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4965 8 365281 365273 0.574718 0 - 0.0299513 2026-02-23T14:36:12.444 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:36:12.478 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4966 8 365281 365273 0.574603 0 - 0.0299513 2026-02-23T14:36:12.562 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:36:12.665 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:36:12.663+0000 7f3e301b2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:36:12.666 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:36:12.664+0000 7f3e301b2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:36:12.774 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:36:12.774 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":253,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:35:24.666678+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:30:16.829966+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:33:43.438776+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:34:48.677580+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:36:13.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:36:12 trial068 ceph-mon[38805]: pgmap v3034: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:36:13.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:36:12 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3385449433' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:36:13.478 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4967 8 365281 365273 0.574487 0 - 0.0299513 2026-02-23T14:36:14.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:36:13 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:36:13] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:36:14.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:36:13 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:36:13.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:36:14.478 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4968 8 365281 365273 0.574371 0 - 0.0299513 2026-02-23T14:36:15.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:36:14 trial068 ceph-mon[38805]: pgmap v3035: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:36:15.479 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4969 8 365281 365273 0.574256 0 - 0.0299513 2026-02-23T14:36:15.813 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:36:15.929 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:36:16.029 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:36:16.027+0000 7fab6c300640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:36:16.030 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:36:16.028+0000 7fab6c300640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:36:16.137 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:36:16.137 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":253,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:35:24.666678+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:30:16.829966+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:33:43.438776+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:34:48.677580+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:36:16.479 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4970 8 365281 365273 0.57414 0 - 0.0299513 2026-02-23T14:36:17.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:36:16 trial068 ceph-mon[38805]: 5.8 scrub starts 2026-02-23T14:36:17.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:36:16 trial068 ceph-mon[38805]: pgmap v3036: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:36:17.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:36:16 trial068 ceph-mon[38805]: 5.8 scrub ok 2026-02-23T14:36:17.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:36:16 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2172716199' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:36:17.479 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4971 8 365281 365273 0.574025 0 - 0.0299513 2026-02-23T14:36:18.479 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4972 8 365281 365273 0.573909 0 - 0.0299513 2026-02-23T14:36:19.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:36:18 trial068 ceph-mon[38805]: pgmap v3037: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:36:19.194 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:36:19.311 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:36:19.416 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:36:19.414+0000 7f30e2fee640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:36:19.416 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:36:19.414+0000 7f30e2fee640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:36:19.479 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4973 8 365281 365273 0.573794 0 - 0.0299513 2026-02-23T14:36:19.526 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:36:19.527 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":253,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:35:24.666678+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:30:16.829966+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:33:43.438776+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:34:48.677580+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:36:20.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:36:19 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1325139815' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:36:20.479 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4974 8 365281 365273 0.573678 0 - 0.0299513 2026-02-23T14:36:21.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:36:20 trial068 ceph-mon[38805]: pgmap v3038: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:36:21.479 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4975 8 365281 365273 0.573563 0 - 0.0299513 2026-02-23T14:36:22.479 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4976 8 365281 365273 0.573448 0 - 0.0299513 2026-02-23T14:36:22.565 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:36:22.684 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:36:22.783 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:36:22.781+0000 7f1499462640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:36:22.783 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:36:22.782+0000 7f1499462640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:36:22.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:36:22 trial068 ceph-mon[38805]: pgmap v3039: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:36:22.895 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:36:22.895 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":253,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:35:24.666678+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:30:16.829966+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:33:43.438776+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:34:48.677580+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:36:23.479 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4977 8 365281 365273 0.573333 0 - 0.0299513 2026-02-23T14:36:24.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:36:23 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2690094358' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:36:24.093 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:36:23 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:36:23] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:36:24.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:36:23 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:36:23.974Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:36:24.479 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4978 8 365281 365273 0.573218 0 - 0.0299513 2026-02-23T14:36:25.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:36:24 trial068 ceph-mon[38805]: pgmap v3040: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:36:25.479 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4979 8 365281 365273 0.573102 0 - 0.0299513 2026-02-23T14:36:25.933 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:36:26.051 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:36:26.151 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:36:26.149+0000 7fb1b18d0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:36:26.151 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:36:26.149+0000 7fb1b18d0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:36:26.259 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:36:26.259 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":253,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:35:24.666678+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:30:16.829966+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:33:43.438776+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:34:48.677580+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:36:26.479 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T14:36:26.479103+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T14:36:26.480 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T14:36:26.480 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4980 8 365281 365273 0.572987 0 - 0.0299513 2026-02-23T14:36:27.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:36:26 trial068 ceph-mon[38805]: pgmap v3041: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:36:27.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:36:26 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3678266765' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:36:27.479 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4981 8 365281 365273 0.572872 0 - 0.0299513 2026-02-23T14:36:28.479 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4982 8 365281 365273 0.572757 0 - 0.0299513 2026-02-23T14:36:29.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:36:28 trial068 ceph-mon[38805]: pgmap v3042: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:36:29.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:36:28 trial068 ceph-mon[38805]: osdmap e254: 4 total, 3 up, 4 in 2026-02-23T14:36:29.295 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:36:29.413 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:36:29.480 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4983 8 365281 365273 0.572642 0 - 0.0299513 2026-02-23T14:36:29.520 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:36:29.518+0000 7f3eafe03640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:36:29.521 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:36:29.519+0000 7f3eafe03640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:36:29.628 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:36:29.628 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":254,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:36:27.796745+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:35:30.456037+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:33:43.438776+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:34:48.677580+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:36:30.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:36:29 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3263714851' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:36:30.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:36:29 trial068 ceph-mon[38805]: osdmap e255: 4 total, 3 up, 4 in 2026-02-23T14:36:30.480 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4984 8 365281 365273 0.572527 0 - 0.0299513 2026-02-23T14:36:31.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:36:30 trial068 ceph-mon[38805]: pgmap v3044: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:36:31.480 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4985 8 365281 365273 0.572413 0 - 0.0299513 2026-02-23T14:36:32.480 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4986 8 365281 365273 0.572298 0 - 0.0299513 2026-02-23T14:36:32.666 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:36:32.786 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:36:32.889 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:36:32.887+0000 7f3acfd53640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:36:32.889 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:36:32.887+0000 7f3acfd53640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:36:32.998 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:36:32.999 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":255,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:36:29.675507+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:35:30.456037+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:33:43.438776+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:34:48.677580+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:36:32.999 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:36:32 trial068 ceph-mon[38805]: pgmap v3046: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:36:33.480 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4987 8 365281 365273 0.572183 0 - 0.0299513 2026-02-23T14:36:34.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:36:33 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3420736710' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:36:34.094 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:36:33 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:36:33] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:36:34.266 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:36:33 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:36:33.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:36:34.480 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4988 8 365281 365273 0.572068 0 - 0.0299513 2026-02-23T14:36:35.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:36:34 trial068 ceph-mon[38805]: pgmap v3047: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:36:35.480 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4989 8 365281 365273 0.571954 0 - 0.0299513 2026-02-23T14:36:36.036 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:36:36.154 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:36:36.255 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:36:36.253+0000 7f8c0cc98640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:36:36.255 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:36:36.253+0000 7f8c0cc98640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:36:36.365 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:36:36.365 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":255,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:36:29.675507+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:35:30.456037+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:33:43.438776+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:34:48.677580+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:36:36.480 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4990 8 365281 365273 0.571839 0 - 0.0299513 2026-02-23T14:36:37.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:36:36 trial068 ceph-mon[38805]: pgmap v3048: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:36:37.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:36:36 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2866493181' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:36:37.480 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4991 8 365281 365273 0.571724 0 - 0.0299513 2026-02-23T14:36:38.480 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4992 8 365281 365273 0.57161 0 - 0.0299513 2026-02-23T14:36:39.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:36:38 trial068 ceph-mon[38805]: pgmap v3049: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:36:39.402 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:36:39.480 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4993 8 365281 365273 0.571495 0 - 0.0299513 2026-02-23T14:36:39.523 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:36:39.625 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:36:39.624+0000 7f83c5ef2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:36:39.626 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:36:39.624+0000 7f83c5ef2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:36:39.736 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:36:39.736 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":255,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:36:29.675507+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:35:30.456037+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:33:43.438776+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:34:48.677580+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:36:40.480 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4994 8 365281 365273 0.571381 0 - 0.0299513 2026-02-23T14:36:41.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:36:40 trial068 ceph-mon[38805]: pgmap v3050: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:36:41.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:36:40 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2124495597' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:36:41.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:36:40 trial068 ceph-mon[38805]: 5.a scrub starts 2026-02-23T14:36:41.480 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4995 8 365281 365273 0.571267 0 - 0.0299513 2026-02-23T14:36:42.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:36:41 trial068 ceph-mon[38805]: 5.a scrub ok 2026-02-23T14:36:42.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:36:41 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T14:36:42.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:36:41 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T14:36:42.481 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4996 8 365281 365273 0.571152 0 - 0.0299513 2026-02-23T14:36:42.775 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:36:42.892 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:36:42.936 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:36:42 trial068 ceph-mon[38805]: pgmap v3051: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:36:42.990 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:36:42.988+0000 7fbb48045640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:36:42.990 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:36:42.988+0000 7fbb48045640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:36:43.098 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:36:43.099 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":255,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:36:29.675507+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:35:30.456037+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:33:43.438776+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:34:48.677580+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:36:43.481 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4997 8 365281 365273 0.571038 0 - 0.0299513 2026-02-23T14:36:43.975 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:36:43 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1820200013' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:36:44.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:36:43 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:36:43] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:36:44.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:36:43 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:36:43.974Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:36:44.481 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4998 8 365281 365273 0.570924 0 - 0.0299513 2026-02-23T14:36:45.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:36:44 trial068 ceph-mon[38805]: pgmap v3052: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:36:45.481 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 4999 8 365281 365273 0.57081 0 - 0.0299513 2026-02-23T14:36:46.136 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:36:46.255 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:36:46.352 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:36:46.349+0000 7f6396b0f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:36:46.352 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:36:46.350+0000 7f6396b0f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:36:46.460 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:36:46.460 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":255,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:36:29.675507+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:35:30.456037+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:33:43.438776+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:34:48.677580+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:36:46.481 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T14:36:46.480556+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T14:36:46.481 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T14:36:46.481 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5000 8 365281 365273 0.570695 0 - 0.0299513 2026-02-23T14:36:46.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:36:46 trial068 ceph-mon[38805]: pgmap v3053: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:36:46.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:36:46 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3633511654' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:36:47.481 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5001 8 365281 365273 0.570581 0 - 0.0299513 2026-02-23T14:36:48.481 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5002 8 365281 365273 0.570467 0 - 0.0299513 2026-02-23T14:36:49.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:36:48 trial068 ceph-mon[38805]: pgmap v3054: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:36:49.481 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5003 8 365281 365273 0.570353 0 - 0.0299513 2026-02-23T14:36:49.498 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:36:49.617 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:36:49.715 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:36:49.713+0000 7fab14b4e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:36:49.715 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:36:49.713+0000 7fab14b4e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:36:49.824 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:36:49.824 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":255,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:36:29.675507+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:35:30.456037+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:33:43.438776+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:34:48.677580+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:36:50.481 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5004 8 365281 365273 0.570239 0 - 0.0299513 2026-02-23T14:36:51.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:36:50 trial068 ceph-mon[38805]: pgmap v3055: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:36:51.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:36:50 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1966836589' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:36:51.481 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5005 8 365281 365273 0.570125 0 - 0.0299513 2026-02-23T14:36:52.481 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5006 8 365281 365273 0.570011 0 - 0.0299513 2026-02-23T14:36:52.862 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:36:52.978 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:36:53.000 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:36:52 trial068 ceph-mon[38805]: pgmap v3056: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:36:53.082 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:36:53.080+0000 7f5229273640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:36:53.082 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:36:53.080+0000 7f5229273640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:36:53.194 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:36:53.194 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":255,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:36:29.675507+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:35:30.456037+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:33:43.438776+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:34:48.677580+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:36:53.481 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5007 8 365281 365273 0.569898 0 - 0.0299513 2026-02-23T14:36:53.973 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:36:53 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/4102823051' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:36:54.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:36:53 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:36:53] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:36:54.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:36:53 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:36:53.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:36:54.481 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5008 8 365281 365273 0.569784 0 - 0.0299513 2026-02-23T14:36:55.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:36:54 trial068 ceph-mon[38805]: pgmap v3057: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:36:55.481 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5009 8 365281 365273 0.56967 0 - 0.0299513 2026-02-23T14:36:56.232 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:36:56.350 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:36:56.450 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:36:56.448+0000 7f4cafa6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:36:56.450 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:36:56.448+0000 7f4cafa6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:36:56.481 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5010 8 365281 365273 0.569556 0 - 0.0299513 2026-02-23T14:36:56.560 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:36:56.560 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":255,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:36:29.675507+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:35:30.456037+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:33:43.438776+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:34:48.677580+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:36:56.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:36:56 trial068 ceph-mon[38805]: pgmap v3058: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:36:56.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:36:56 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2415743654' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:36:57.482 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5011 8 365281 365273 0.569443 0 - 0.0299513 2026-02-23T14:36:58.482 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5012 8 365281 365273 0.569329 0 - 0.0299513 2026-02-23T14:36:59.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:36:58 trial068 ceph-mon[38805]: pgmap v3059: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:36:59.482 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5013 8 365281 365273 0.569215 0 - 0.0299513 2026-02-23T14:36:59.612 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:36:59.731 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:36:59.832 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:36:59.830+0000 7fcab5512640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:36:59.832 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:36:59.831+0000 7fcab5512640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:36:59.941 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:36:59.941 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":255,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:36:29.675507+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:35:30.456037+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:33:43.438776+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:34:48.677580+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:36:59.942 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:36:59 trial068 ceph-mon[38805]: 5.7 scrub starts 2026-02-23T14:36:59.942 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:36:59 trial068 ceph-mon[38805]: 5.7 scrub ok 2026-02-23T14:37:00.482 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5014 8 365281 365273 0.569102 0 - 0.0299513 2026-02-23T14:37:01.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:37:00 trial068 ceph-mon[38805]: pgmap v3060: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:37:01.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:37:00 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3567332453' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:37:01.482 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5015 8 365281 365273 0.568988 0 - 0.0299513 2026-02-23T14:37:02.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:37:01 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2026-02-23T14:37:02.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:37:01 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T14:37:02.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:37:01 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T14:37:02.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:37:01 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T14:37:02.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:37:01 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T14:37:02.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:37:01 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-23T14:37:02.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:37:01 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2026-02-23T14:37:02.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:37:01 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T14:37:02.095 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:37:01 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2026-02-23T14:37:02.482 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5016 8 365281 365273 0.568875 0 - 0.0299513 2026-02-23T14:37:02.978 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:37:03.003 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:37:02 trial068 ceph-mon[38805]: 5.5 scrub starts 2026-02-23T14:37:03.003 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:37:02 trial068 ceph-mon[38805]: 5.5 scrub ok 2026-02-23T14:37:03.003 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:37:02 trial068 ceph-mon[38805]: pgmap v3061: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:37:03.004 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:37:02 trial068 ceph-mon[38805]: pgmap v3062: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:37:03.004 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:37:02 trial068 ceph-mon[38805]: pgmap v3063: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:37:03.098 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:37:03.198 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:37:03.196+0000 7f6e54069640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:37:03.198 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:37:03.196+0000 7f6e54069640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:37:03.309 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:37:03.310 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":255,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:36:29.675507+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:35:30.456037+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:33:43.438776+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:34:48.677580+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:37:03.482 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5017 8 365281 365273 0.568762 0 - 0.0299513 2026-02-23T14:37:03.974 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:37:03 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/4019251261' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:37:04.266 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:37:03 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:37:03.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:37:04.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:37:03 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:37:03] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:37:04.482 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5018 8 365281 365273 0.568648 0 - 0.0299513 2026-02-23T14:37:05.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:37:04 trial068 ceph-mon[38805]: pgmap v3064: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:37:05.482 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5019 8 365281 365273 0.568535 0 - 0.0299513 2026-02-23T14:37:06.367 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:37:06.482 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T14:37:06.481951+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T14:37:06.482 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T14:37:06.482 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5020 8 365281 365273 0.568422 0 - 0.0299513 2026-02-23T14:37:06.488 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:37:06.591 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:37:06.589+0000 7fb82f879640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:37:06.591 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:37:06.590+0000 7fb82f879640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:37:06.700 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:37:06.700 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":255,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:36:29.675507+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:35:30.456037+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:33:43.438776+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:34:48.677580+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:37:07.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:37:06 trial068 ceph-mon[38805]: pgmap v3065: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:37:07.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:37:06 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/365467436' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:37:07.482 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5021 8 365281 365273 0.568309 0 - 0.0299513 2026-02-23T14:37:08.482 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5022 8 365281 365273 0.568195 0 - 0.0299513 2026-02-23T14:37:09.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:37:08 trial068 ceph-mon[38805]: pgmap v3066: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:37:09.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:37:08 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T14:37:09.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:37:08 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T14:37:09.483 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5023 8 365281 365273 0.568082 0 - 0.0299513 2026-02-23T14:37:09.738 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:37:09.853 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:37:09.955 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:37:09.953+0000 7fdb8689e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:37:09.955 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:37:09.953+0000 7fdb8689e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:37:09.967 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:37:09 trial068 ceph-mon[38805]: 5.e scrub starts 2026-02-23T14:37:09.967 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:37:09 trial068 ceph-mon[38805]: 5.e scrub ok 2026-02-23T14:37:10.065 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:37:10.065 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":255,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:36:29.675507+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:35:30.456037+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:33:43.438776+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:34:48.677580+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:37:10.483 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5024 8 365281 365273 0.567969 0 - 0.0299513 2026-02-23T14:37:11.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:37:10 trial068 ceph-mon[38805]: pgmap v3067: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:37:11.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:37:10 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3913275197' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:37:11.483 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5025 8 365281 365273 0.567856 0 - 0.0299513 2026-02-23T14:37:12.482 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5026 8 365281 365273 0.567743 0 - 0.0299513 2026-02-23T14:37:13.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:37:12 trial068 ceph-mon[38805]: pgmap v3068: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:37:13.117 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:37:13.232 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:37:13.331 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:37:13.329+0000 7efe668de640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:37:13.332 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:37:13.330+0000 7efe668de640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:37:13.440 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:37:13.441 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":255,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:36:29.675507+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:35:30.456037+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:33:43.438776+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:34:48.677580+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:37:13.482 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5027 8 365281 365273 0.56763 0 - 0.0299513 2026-02-23T14:37:13.973 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:37:13 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2435942186' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:37:14.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:37:13 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:37:13] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:37:14.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:37:13 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:37:13.972Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:37:14.482 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5028 8 365281 365273 0.567517 0 - 0.0299513 2026-02-23T14:37:15.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:37:14 trial068 ceph-mon[38805]: pgmap v3069: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:37:15.483 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5029 8 365281 365273 0.567404 0 - 0.0299513 2026-02-23T14:37:16.477 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:37:16.483 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5030 8 365281 365273 0.567292 0 - 0.0299513 2026-02-23T14:37:16.595 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:37:16.695 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:37:16.693+0000 7fdebef3c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:37:16.695 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:37:16.694+0000 7fdebef3c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:37:16.804 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:37:16.804 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":255,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:36:29.675507+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:35:30.456037+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:33:43.438776+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:34:48.677580+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:37:16.805 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:37:16 trial068 ceph-mon[38805]: pgmap v3070: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:37:17.483 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5031 8 365281 365273 0.567179 0 - 0.0299513 2026-02-23T14:37:18.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:37:17 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/550484033' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:37:18.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:37:17 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T14:37:18.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:37:17 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T14:37:18.483 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5032 8 365281 365273 0.567066 0 - 0.0299513 2026-02-23T14:37:19.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:37:18 trial068 ceph-mon[38805]: pgmap v3071: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:37:19.483 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5033 8 365281 365273 0.566954 0 - 0.0299513 2026-02-23T14:37:19.856 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:37:19.974 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:37:20.073 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:37:20.070+0000 7f673d641640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:37:20.073 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:37:20.071+0000 7f673d641640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:37:20.181 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:37:20.181 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":255,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:36:29.675507+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:35:30.456037+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:33:43.438776+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:34:48.677580+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:37:20.483 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5034 8 365281 365273 0.566841 0 - 0.0299513 2026-02-23T14:37:21.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:37:20 trial068 ceph-mon[38805]: pgmap v3072: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:37:21.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:37:20 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/400410569' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:37:21.483 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5035 8 365281 365273 0.566728 0 - 0.0299513 2026-02-23T14:37:22.483 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5036 8 365281 365273 0.566616 0 - 0.0299513 2026-02-23T14:37:23.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:37:22 trial068 ceph-mon[38805]: pgmap v3073: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:37:23.219 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:37:23.339 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:37:23.437 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:37:23.435+0000 7f3f61e8d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:37:23.437 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:37:23.435+0000 7f3f61e8d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:37:23.483 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5037 8 365281 365273 0.566503 0 - 0.0299513 2026-02-23T14:37:23.546 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:37:23.546 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":255,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:36:29.675507+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:35:30.456037+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:33:43.438776+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:34:48.677580+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:37:23.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:37:23 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1900696194' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:37:24.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:37:23 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:37:23] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:37:24.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:37:23 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:37:23.972Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:37:24.483 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5038 8 365281 365273 0.566391 0 - 0.0299513 2026-02-23T14:37:25.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:37:24 trial068 ceph-mon[38805]: pgmap v3074: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:37:25.483 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5039 8 365281 365273 0.566278 0 - 0.0299513 2026-02-23T14:37:26.483 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T14:37:26.483414+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T14:37:26.483 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T14:37:26.484 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5040 8 365281 365273 0.566166 0 - 0.0299513 2026-02-23T14:37:26.583 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:37:26.703 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:37:26.802 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:37:26.800+0000 7f8079107640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:37:26.802 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:37:26.800+0000 7f8079107640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:37:26.911 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:37:26.911 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":255,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:36:29.675507+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:35:30.456037+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:33:43.438776+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:34:48.677580+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:37:26.911 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:37:26 trial068 ceph-mon[38805]: pgmap v3075: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:37:27.484 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5041 8 365281 365273 0.566054 0 - 0.0299513 2026-02-23T14:37:28.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:37:27 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/4261870449' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:37:28.484 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5042 8 365281 365273 0.565941 0 - 0.0299513 2026-02-23T14:37:29.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:37:28 trial068 ceph-mon[38805]: pgmap v3076: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:37:29.484 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5043 8 365281 365273 0.565829 0 - 0.0299513 2026-02-23T14:37:29.948 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:37:30.067 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:37:30.168 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:37:30.166+0000 7f78ac17e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:37:30.168 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:37:30.166+0000 7f78ac17e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:37:30.277 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:37:30.277 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":255,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:36:29.675507+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:35:30.456037+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:33:43.438776+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:34:48.677580+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:37:30.484 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5044 8 365281 365273 0.565717 0 - 0.0299513 2026-02-23T14:37:31.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:37:30 trial068 ceph-mon[38805]: pgmap v3077: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:37:31.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:37:30 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/357838650' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:37:31.484 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5045 8 365281 365273 0.565605 0 - 0.0299513 2026-02-23T14:37:32.484 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5046 8 365281 365273 0.565493 0 - 0.0299513 2026-02-23T14:37:33.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:37:32 trial068 ceph-mon[38805]: pgmap v3078: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:37:33.315 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:37:33.434 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:37:33.484 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5047 8 365281 365273 0.565381 0 - 0.0299513 2026-02-23T14:37:33.533 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:37:33.531+0000 7f00aa911640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:37:33.533 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:37:33.531+0000 7f00aa911640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:37:33.642 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:37:33.643 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":255,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:36:29.675507+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:35:30.456037+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:33:43.438776+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:34:48.677580+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:37:33.974 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:37:33 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2526531949' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:37:34.266 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:37:33 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:37:33.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:37:34.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:37:33 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:37:33] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:37:34.485 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5048 8 365281 365273 0.565269 0 - 0.0299513 2026-02-23T14:37:35.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:37:34 trial068 ceph-mon[38805]: pgmap v3079: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:37:35.485 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5049 8 365281 365273 0.565157 0 - 0.0299513 2026-02-23T14:37:36.485 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5050 8 365281 365273 0.565045 0 - 0.0299513 2026-02-23T14:37:36.681 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:37:36.800 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:37:36.901 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:37:36.899+0000 7f10feb18640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:37:36.901 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:37:36.899+0000 7f10feb18640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:37:37.009 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:37:37.010 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":255,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:36:29.675507+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:35:30.456037+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:33:43.438776+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:34:48.677580+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:37:37.010 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:37:36 trial068 ceph-mon[38805]: pgmap v3080: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:37:37.485 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5051 8 365281 365273 0.564933 0 - 0.0299513 2026-02-23T14:37:38.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:37:37 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3969116842' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:37:38.485 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5052 8 365281 365273 0.564821 0 - 0.0299513 2026-02-23T14:37:39.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:37:38 trial068 ceph-mon[38805]: 5.8 scrub starts 2026-02-23T14:37:39.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:37:38 trial068 ceph-mon[38805]: 5.8 scrub ok 2026-02-23T14:37:39.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:37:38 trial068 ceph-mon[38805]: pgmap v3081: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:37:39.485 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5053 8 365281 365273 0.564709 0 - 0.0299513 2026-02-23T14:37:40.048 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:37:40.166 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:37:40.276 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:37:40.274+0000 7f9f8cf98640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:37:40.276 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:37:40.274+0000 7f9f8cf98640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:37:40.387 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:37:40.387 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":255,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:36:29.675507+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:35:30.456037+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:33:43.438776+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:34:48.677580+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:37:40.485 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5054 8 365281 365273 0.564598 0 - 0.0299513 2026-02-23T14:37:41.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:37:40 trial068 ceph-mon[38805]: pgmap v3082: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:37:41.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:37:40 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/625591616' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:37:41.485 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5055 8 365281 365273 0.564486 0 - 0.0299513 2026-02-23T14:37:42.485 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5056 8 365281 365273 0.564374 0 - 0.0299513 2026-02-23T14:37:43.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:37:42 trial068 ceph-mon[38805]: pgmap v3083: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:37:43.425 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:37:43.485 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5057 8 365281 365273 0.564263 0 - 0.0299513 2026-02-23T14:37:43.543 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:37:43.641 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:37:43.639+0000 7f9843520640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:37:43.641 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:37:43.639+0000 7f9843520640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:37:43.750 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:37:43.750 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":255,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:36:29.675507+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:35:30.456037+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:33:43.438776+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:34:48.677580+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:37:44.094 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:37:43 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:37:43] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:37:44.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:37:43 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:37:43.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:37:44.485 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5058 8 365281 365273 0.564151 0 - 0.0299513 2026-02-23T14:37:45.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:37:44 trial068 ceph-mon[38805]: pgmap v3084: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:37:45.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:37:44 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/681955389' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:37:45.485 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5059 8 365281 365273 0.56404 0 - 0.0299513 2026-02-23T14:37:46.485 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T14:37:46.485216+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T14:37:46.486 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T14:37:46.486 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5060 8 365281 365273 0.563928 0 - 0.0299513 2026-02-23T14:37:46.789 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:37:46.905 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:37:47.001 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:37:46 trial068 ceph-mon[38805]: pgmap v3085: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:37:47.002 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:37:46.999+0000 7f0a65780640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:37:47.002 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:37:47.000+0000 7f0a65780640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:37:47.111 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:37:47.111 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":255,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:36:29.675507+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:35:30.456037+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:33:43.438776+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:34:48.677580+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:37:47.486 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5061 8 365281 365273 0.563817 0 - 0.0299513 2026-02-23T14:37:48.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:37:47 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3320504316' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:37:48.486 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5062 8 365281 365273 0.563705 0 - 0.0299513 2026-02-23T14:37:49.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:37:48 trial068 ceph-mon[38805]: pgmap v3086: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:37:49.486 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5063 8 365281 365273 0.563594 0 - 0.0299513 2026-02-23T14:37:50.164 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:37:50.281 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:37:50.378 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:37:50.376+0000 7f88e2c01640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:37:50.378 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:37:50.376+0000 7f88e2c01640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:37:50.486 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5064 8 365281 365273 0.563483 0 - 0.0299513 2026-02-23T14:37:50.487 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:37:50.487 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":255,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:36:29.675507+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:35:30.456037+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:33:43.438776+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:34:48.677580+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:37:51.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:37:50 trial068 ceph-mon[38805]: pgmap v3087: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:37:51.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:37:50 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2189269840' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:37:51.486 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5065 8 365281 365273 0.563372 0 - 0.0299513 2026-02-23T14:37:52.486 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5066 8 365281 365273 0.56326 0 - 0.0299513 2026-02-23T14:37:53.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:37:52 trial068 ceph-mon[38805]: pgmap v3088: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:37:53.486 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5067 8 365281 365273 0.563149 0 - 0.0299513 2026-02-23T14:37:53.525 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:37:53.643 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:37:53.741 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:37:53.739+0000 7f41826e6640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:37:53.741 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:37:53.739+0000 7f41826e6640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:37:53.850 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:37:53.851 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":255,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:36:29.675507+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:35:30.456037+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:33:43.438776+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:34:48.677580+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:37:54.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:37:53 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:37:53] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:37:54.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:37:53 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:37:53.974Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:37:54.486 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5068 8 365281 365273 0.563038 0 - 0.0299513 2026-02-23T14:37:55.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:37:54 trial068 ceph-mon[38805]: pgmap v3089: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:37:55.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:37:54 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2103296230' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:37:55.486 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5069 8 365281 365273 0.562927 0 - 0.0299513 2026-02-23T14:37:56.487 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5070 8 365281 365273 0.562816 0 - 0.0299513 2026-02-23T14:37:56.887 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:37:57.007 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:37:57.029 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:37:56 trial068 ceph-mon[38805]: pgmap v3090: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:37:57.112 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:37:57.110+0000 7fdc4f2dd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:37:57.113 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:37:57.111+0000 7fdc4f2dd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:37:57.222 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:37:57.222 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":255,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:36:29.675507+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:35:30.456037+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:33:43.438776+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:34:48.677580+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:37:57.487 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5071 8 365281 365273 0.562705 0 - 0.0299513 2026-02-23T14:37:58.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:37:57 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T14:37:58.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:37:57 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T14:37:58.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:37:57 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/323378905' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:37:58.487 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5072 8 365281 365273 0.562594 0 - 0.0299513 2026-02-23T14:37:59.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:37:58 trial068 ceph-mon[38805]: pgmap v3091: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:37:59.487 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5073 8 365281 365273 0.562483 0 - 0.0299513 2026-02-23T14:38:00.260 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:38:00.375 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:38:00.477 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:38:00.475+0000 7fac29e8b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:38:00.477 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:38:00.475+0000 7fac29e8b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:38:00.487 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5074 8 365281 365273 0.562372 0 - 0.0299513 2026-02-23T14:38:00.587 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:38:00.587 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":255,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:36:29.675507+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:35:30.456037+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:33:43.438776+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:34:48.677580+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:38:00.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:38:00 trial068 ceph-mon[38805]: pgmap v3092: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:38:00.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:38:00 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2162066155' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:38:01.487 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5075 8 365281 365273 0.562261 0 - 0.0299513 2026-02-23T14:38:02.058 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:38:01 trial068 ceph-mon[38805]: 5.a scrub starts 2026-02-23T14:38:02.058 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:38:01 trial068 ceph-mon[38805]: 5.a scrub ok 2026-02-23T14:38:02.058 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:38:01 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2026-02-23T14:38:02.487 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5076 8 365281 365273 0.562151 0 - 0.0299513 2026-02-23T14:38:03.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:38:02 trial068 ceph-mon[38805]: pgmap v3093: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:38:03.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:38:02 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T14:38:03.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:38:02 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T14:38:03.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:38:02 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T14:38:03.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:38:02 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T14:38:03.095 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:38:02 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T14:38:03.095 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:38:02 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T14:38:03.095 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:38:02 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T14:38:03.095 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:38:02 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T14:38:03.487 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5077 8 365281 365273 0.56204 0 - 0.0299513 2026-02-23T14:38:03.646 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:38:03.764 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:38:03.865 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:38:03.862+0000 7f1c2b22d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:38:03.865 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:38:03.863+0000 7f1c2b22d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:38:03.972 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:38:03.973 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":255,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:36:29.675507+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:35:30.456037+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:33:43.438776+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:34:48.677580+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:38:04.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:38:03 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:38:03] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:38:04.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:38:03 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:38:03.972Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:38:04.487 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5078 8 365281 365273 0.561929 0 - 0.0299513 2026-02-23T14:38:04.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:38:04 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T14:38:04.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:38:04 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T14:38:04.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:38:04 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T14:38:04.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:38:04 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T14:38:04.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:38:04 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-23T14:38:04.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:38:04 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2026-02-23T14:38:04.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:38:04 trial068 ceph-mon[38805]: pgmap v3094: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:38:04.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:38:04 trial068 ceph-mon[38805]: pgmap v3095: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:38:04.845 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:38:04 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T14:38:04.845 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:38:04 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2026-02-23T14:38:04.845 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:38:04 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3178655973' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:38:05.487 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5079 8 365281 365273 0.561819 0 - 0.0299513 2026-02-23T14:38:06.487 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T14:38:06.487127+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T14:38:06.488 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T14:38:06.488 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5080 8 365281 365273 0.561708 0 - 0.0299513 2026-02-23T14:38:07.009 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:38:07.034 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:38:06 trial068 ceph-mon[38805]: pgmap v3096: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:38:07.126 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:38:07.228 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:38:07.226+0000 7fa37ee4f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:38:07.228 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:38:07.226+0000 7fa37ee4f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:38:07.335 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:38:07.336 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":255,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:36:29.675507+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:35:30.456037+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:33:43.438776+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:34:48.677580+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:38:07.487 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5081 8 365281 365273 0.561597 0 - 0.0299513 2026-02-23T14:38:08.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:38:07 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3795559134' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:38:08.487 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5082 8 365281 365273 0.561487 0 - 0.0299513 2026-02-23T14:38:09.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:38:08 trial068 ceph-mon[38805]: pgmap v3097: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:38:09.488 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5083 8 365281 365273 0.561376 0 - 0.0299513 2026-02-23T14:38:10.375 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:38:10.488 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5084 8 365281 365273 0.561266 0 - 0.0299513 2026-02-23T14:38:10.493 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:38:10.596 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:38:10.594+0000 7fdc94807640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:38:10.596 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:38:10.594+0000 7fdc94807640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:38:10.708 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:38:10.708 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":255,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:36:29.675507+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:35:30.456037+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:33:43.438776+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:34:48.677580+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:38:11.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:38:10 trial068 ceph-mon[38805]: pgmap v3098: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:38:11.488 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5085 8 365281 365273 0.561156 0 - 0.0299513 2026-02-23T14:38:12.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:38:11 trial068 ceph-mon[38805]: 5.5 scrub starts 2026-02-23T14:38:12.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:38:11 trial068 ceph-mon[38805]: 5.5 scrub ok 2026-02-23T14:38:12.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:38:11 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3693874883' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:38:12.488 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5086 8 365281 365273 0.561045 0 - 0.0299513 2026-02-23T14:38:13.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:38:12 trial068 ceph-mon[38805]: pgmap v3099: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:38:13.488 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5087 8 365281 365273 0.560935 0 - 0.0299513 2026-02-23T14:38:13.744 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:38:13.862 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:38:13.965 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:38:13.963+0000 7f8854a79640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:38:13.965 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:38:13.963+0000 7f8854a79640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:38:14.074 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:38:14.074 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":255,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:36:29.675507+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:35:30.456037+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:33:43.438776+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:34:48.677580+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:38:14.075 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:38:13 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:38:13] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:38:14.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:38:13 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:38:13.972Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:38:14.488 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5088 8 365281 365273 0.560825 0 - 0.0299513 2026-02-23T14:38:15.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:38:14 trial068 ceph-mon[38805]: pgmap v3100: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:38:15.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:38:14 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2142387466' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:38:15.488 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5089 8 365281 365273 0.560715 0 - 0.0299513 2026-02-23T14:38:16.488 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5090 8 365281 365273 0.560604 0 - 0.0299513 2026-02-23T14:38:17.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:38:16 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T14:38:17.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:38:16 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T14:38:17.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:38:16 trial068 ceph-mon[38805]: pgmap v3101: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:38:17.113 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:38:17.232 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:38:17.333 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:38:17.330+0000 7f399e589640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:38:17.333 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:38:17.331+0000 7f399e589640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:38:17.442 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:38:17.442 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":255,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:36:29.675507+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:35:30.456037+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:33:43.438776+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:34:48.677580+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:38:17.488 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5091 8 365281 365273 0.560494 0 - 0.0299513 2026-02-23T14:38:18.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:38:17 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3692432399' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:38:18.489 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5092 8 365281 365273 0.560384 0 - 0.0299513 2026-02-23T14:38:19.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:38:18 trial068 ceph-mon[38805]: pgmap v3102: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:38:19.489 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5093 8 365281 365273 0.560274 0 - 0.0299513 2026-02-23T14:38:20.479 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:38:20.488 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5094 8 365281 365273 0.560164 0 - 0.0299513 2026-02-23T14:38:20.595 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:38:20.694 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:38:20.692+0000 7f5b28961640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:38:20.694 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:38:20.692+0000 7f5b28961640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:38:20.808 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:38:20.808 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":255,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:36:29.675507+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:35:30.456037+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:33:43.438776+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:34:48.677580+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:38:20.808 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:38:20 trial068 ceph-mon[38805]: pgmap v3103: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:38:21.489 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5095 8 365281 365273 0.560054 0 - 0.0299513 2026-02-23T14:38:22.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:38:21 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/50845616' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:38:22.489 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5096 8 365281 365273 0.559944 0 - 0.0299513 2026-02-23T14:38:23.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:38:22 trial068 ceph-mon[38805]: pgmap v3104: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:38:23.489 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5097 8 365281 365273 0.559835 0 - 0.0299513 2026-02-23T14:38:23.845 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:38:23.962 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:38:23.974 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:38:23 trial068 ceph-mon[38805]: 5.7 scrub starts 2026-02-23T14:38:23.974 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:38:23 trial068 ceph-mon[38805]: 5.7 scrub ok 2026-02-23T14:38:24.061 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:38:24.059+0000 7fb15c250640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:38:24.062 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:38:24.060+0000 7fb15c250640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:38:24.170 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:38:24.171 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":255,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:36:29.675507+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:35:30.456037+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:33:43.438776+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:34:48.677580+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:38:24.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:38:23 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:38:23] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:38:24.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:38:23 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:38:23.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:38:24.489 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5098 8 365281 365273 0.559725 0 - 0.0299513 2026-02-23T14:38:25.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:38:24 trial068 ceph-mon[38805]: pgmap v3105: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:38:25.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:38:24 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/630361365' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:38:25.489 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5099 8 365281 365273 0.559615 0 - 0.0299513 2026-02-23T14:38:26.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:38:25 trial068 ceph-mon[38805]: 5.e scrub starts 2026-02-23T14:38:26.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:38:25 trial068 ceph-mon[38805]: 5.e scrub ok 2026-02-23T14:38:26.489 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T14:38:26.488912+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T14:38:26.489 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T14:38:26.490 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5100 8 365281 365273 0.559505 0 - 0.0299513 2026-02-23T14:38:27.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:38:26 trial068 ceph-mon[38805]: pgmap v3106: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:38:27.208 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:38:27.326 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:38:27.437 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:38:27.435+0000 7f8413462640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:38:27.437 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:38:27.435+0000 7f8413462640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:38:27.489 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5101 8 365281 365273 0.559396 0 - 0.0299513 2026-02-23T14:38:27.547 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:38:27.548 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":255,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:36:29.675507+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:35:30.456037+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:33:43.438776+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:34:48.677580+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:38:27.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:38:27 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2455196000' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:38:28.489 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5102 8 365281 365273 0.559286 0 - 0.0299513 2026-02-23T14:38:29.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:38:28 trial068 ceph-mon[38805]: pgmap v3107: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:38:29.489 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5103 8 365281 365273 0.559176 0 - 0.0299513 2026-02-23T14:38:30.490 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5104 8 365281 365273 0.559067 0 - 0.0299513 2026-02-23T14:38:30.587 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:38:30.704 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:38:30.811 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:38:30.809+0000 7f6896734640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:38:30.812 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:38:30.810+0000 7f6896734640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:38:30.920 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:38:30.920 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":255,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:36:29.675507+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:35:30.456037+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:33:43.438776+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:34:48.677580+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:38:30.921 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:38:30 trial068 ceph-mon[38805]: pgmap v3108: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:38:31.490 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5105 8 365281 365273 0.558957 0 - 0.0299513 2026-02-23T14:38:32.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:38:31 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3375842156' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:38:32.490 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5106 8 365281 365273 0.558848 0 - 0.0299513 2026-02-23T14:38:33.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:38:32 trial068 ceph-mon[38805]: pgmap v3109: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:38:33.490 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5107 8 365281 365273 0.558738 0 - 0.0299513 2026-02-23T14:38:33.957 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:38:34.075 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:38:34.173 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:38:34.171+0000 7f5d486e1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:38:34.173 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:38:34.171+0000 7f5d486e1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:38:34.266 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:38:33 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:38:33.972Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:38:34.282 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:38:34.282 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":255,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:36:29.675507+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:35:30.456037+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:33:43.438776+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:34:48.677580+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:38:34.283 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:38:33 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:38:33] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:38:34.490 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5108 8 365281 365273 0.558629 0 - 0.0299513 2026-02-23T14:38:35.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:38:34 trial068 ceph-mon[38805]: pgmap v3110: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:38:35.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:38:34 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2563990276' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:38:35.490 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5109 8 365281 365273 0.55852 0 - 0.0299513 2026-02-23T14:38:36.490 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5110 8 365281 365273 0.55841 0 - 0.0299513 2026-02-23T14:38:37.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:38:36 trial068 ceph-mon[38805]: pgmap v3111: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:38:37.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:38:36 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T14:38:37.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:38:36 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T14:38:37.320 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:38:37.437 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:38:37.490 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5111 8 365281 365273 0.558301 0 - 0.0299513 2026-02-23T14:38:37.536 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:38:37.534+0000 7f47f0d7d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:38:37.537 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:38:37.535+0000 7f47f0d7d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:38:37.645 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:38:37.646 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":255,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:36:29.675507+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:35:30.456037+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:33:43.438776+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:34:48.677580+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:38:38.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:38:37 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2596951110' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:38:38.490 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5112 8 365281 365273 0.558192 0 - 0.0299513 2026-02-23T14:38:39.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:38:38 trial068 ceph-mon[38805]: pgmap v3112: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:38:39.490 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5113 8 365281 365273 0.558083 0 - 0.0299513 2026-02-23T14:38:40.490 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5114 8 365281 365273 0.557974 0 - 0.0299513 2026-02-23T14:38:40.687 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:38:40.807 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:38:40.907 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:38:40.905+0000 7fd06857d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:38:40.907 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:38:40.905+0000 7fd06857d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:38:41.016 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:38:41.016 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":255,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:36:29.675507+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:35:30.456037+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:33:43.438776+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:34:48.677580+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:38:41.016 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:38:40 trial068 ceph-mon[38805]: pgmap v3113: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:38:41.490 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5115 8 365281 365273 0.557864 0 - 0.0299513 2026-02-23T14:38:42.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:38:41 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1297621971' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:38:42.490 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5116 8 365281 365273 0.557755 0 - 0.0299513 2026-02-23T14:38:43.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:38:42 trial068 ceph-mon[38805]: pgmap v3114: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:38:43.490 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5117 8 365281 365273 0.557646 0 - 0.0299513 2026-02-23T14:38:44.054 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:38:44.171 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:38:44.225 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:38:43 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:38:43] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:38:44.274 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:38:44.272+0000 7f0df7e43640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:38:44.274 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:38:44.272+0000 7f0df7e43640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:38:44.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:38:43 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:38:43.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:38:44.382 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:38:44.383 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":255,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:36:29.675507+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:35:30.456037+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:33:43.438776+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:34:48.677580+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:38:44.491 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5118 8 365281 365273 0.557537 0 - 0.0299513 2026-02-23T14:38:45.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:38:44 trial068 ceph-mon[38805]: pgmap v3115: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:38:45.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:38:44 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2559423665' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:38:45.491 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5119 8 365281 365273 0.557429 0 - 0.0299513 2026-02-23T14:38:46.491 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T14:38:46.490428+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T14:38:46.491 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T14:38:46.491 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5120 8 365281 365273 0.55732 0 - 0.0299513 2026-02-23T14:38:47.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:38:46 trial068 ceph-mon[38805]: pgmap v3116: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:38:47.420 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:38:47.491 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5121 8 365281 365273 0.557211 0 - 0.0299513 2026-02-23T14:38:47.536 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:38:47.639 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:38:47.637+0000 7f420c538640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:38:47.640 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:38:47.637+0000 7f420c538640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:38:47.752 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:38:47.752 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":255,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:36:29.675507+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:35:30.456037+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:33:43.438776+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:34:48.677580+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:38:48.491 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5122 8 365281 365273 0.557102 0 - 0.0299513 2026-02-23T14:38:49.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:38:48 trial068 ceph-mon[38805]: pgmap v3117: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:38:49.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:38:48 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2537368738' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:38:49.491 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5123 8 365281 365273 0.556993 0 - 0.0299513 2026-02-23T14:38:50.491 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5124 8 365281 365273 0.556885 0 - 0.0299513 2026-02-23T14:38:50.791 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:38:50.906 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:38:51.004 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:38:51.002+0000 7ff44376c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:38:51.005 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:38:50 trial068 ceph-mon[38805]: pgmap v3118: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:38:51.005 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:38:51.002+0000 7ff44376c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:38:51.115 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:38:51.115 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":255,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:36:29.675507+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:35:30.456037+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:33:43.438776+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:34:48.677580+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:38:51.491 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5125 8 365281 365273 0.556776 0 - 0.0299513 2026-02-23T14:38:52.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:38:51 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3036815823' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:38:52.491 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5126 8 365281 365273 0.556667 0 - 0.0299513 2026-02-23T14:38:53.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:38:52 trial068 ceph-mon[38805]: pgmap v3119: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:38:53.491 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5127 8 365281 365273 0.556559 0 - 0.0299513 2026-02-23T14:38:54.152 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:38:54.269 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:38:54.291 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:38:53 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:38:53] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:38:54.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:38:53 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:38:53.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:38:54.367 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:38:54.365+0000 7f9bdf129640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:38:54.367 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:38:54.365+0000 7f9bdf129640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:38:54.478 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:38:54.479 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":255,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:36:29.675507+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:35:30.456037+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:33:43.438776+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:34:48.677580+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:38:54.491 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5128 8 365281 365273 0.55645 0 - 0.0299513 2026-02-23T14:38:55.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:38:54 trial068 ceph-mon[38805]: pgmap v3120: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:38:55.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:38:54 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/505323024' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:38:55.491 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5129 8 365281 365273 0.556342 0 - 0.0299513 2026-02-23T14:38:56.492 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5130 8 365281 365273 0.556233 0 - 0.0299513 2026-02-23T14:38:57.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:38:56 trial068 ceph-mon[38805]: pgmap v3121: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:38:57.492 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5131 8 365281 365273 0.556125 0 - 0.0299513 2026-02-23T14:38:57.516 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:38:57.636 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:38:57.737 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:38:57.734+0000 7fcae693b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:38:57.737 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:38:57.735+0000 7fcae693b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:38:57.845 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:38:57.845 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":255,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:36:29.675507+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:35:30.456037+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:33:43.438776+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:34:48.677580+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:38:58.492 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5132 8 365281 365273 0.556016 0 - 0.0299513 2026-02-23T14:38:59.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:38:58 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T14:38:59.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:38:58 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T14:38:59.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:38:58 trial068 ceph-mon[38805]: pgmap v3122: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:38:59.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:38:58 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1645464079' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:38:59.492 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5133 8 365281 365273 0.555908 0 - 0.0299513 2026-02-23T14:39:00.492 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5134 8 365281 365273 0.5558 0 - 0.0299513 2026-02-23T14:39:00.883 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:39:01.000 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:39:01.023 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:39:00 trial068 ceph-mon[38805]: pgmap v3123: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:39:01.109 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:39:01.107+0000 7f68d8289640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:39:01.109 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:39:01.107+0000 7f68d8289640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:39:01.218 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:39:01.218 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":255,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:36:29.675507+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:35:30.456037+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:33:43.438776+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:34:48.677580+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:39:01.492 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5135 8 365281 365273 0.555692 0 - 0.0299513 2026-02-23T14:39:02.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:39:01 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1449037495' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:39:02.492 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5136 8 365281 365273 0.555583 0 - 0.0299513 2026-02-23T14:39:03.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:39:02 trial068 ceph-mon[38805]: pgmap v3124: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:39:03.493 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5137 8 365281 365273 0.555475 0 - 0.0299513 2026-02-23T14:39:04.094 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:39:03 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:39:03] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:39:04.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:39:03 trial068 ceph-mon[38805]: 5.8 scrub starts 2026-02-23T14:39:04.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:39:03 trial068 ceph-mon[38805]: 5.8 scrub ok 2026-02-23T14:39:04.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:39:03 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2026-02-23T14:39:04.095 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:39:03 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-23T14:39:04.095 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:39:03 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2026-02-23T14:39:04.095 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:39:03 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T14:39:04.095 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:39:03 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2026-02-23T14:39:04.256 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:39:04.266 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:39:03 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:39:03.972Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:39:04.369 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:39:04.473 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:39:04.471+0000 7f21bea36640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:39:04.473 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:39:04.471+0000 7f21bea36640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:39:04.492 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5138 8 365281 365273 0.555367 0 - 0.0299513 2026-02-23T14:39:04.582 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:39:04.582 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":256,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:39:03.753306+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:35:30.456037+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:38:14.929578+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:34:48.677580+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:39:04.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:39:04 trial068 ceph-mon[38805]: pgmap v3125: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:39:04.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:39:04 trial068 ceph-mon[38805]: pgmap v3126: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:39:04.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:39:04 trial068 ceph-mon[38805]: pgmap v3127: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:39:04.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:39:04 trial068 ceph-mon[38805]: osdmap e256: 4 total, 3 up, 4 in 2026-02-23T14:39:04.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:39:04 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1523577430' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:39:04.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:39:04 trial068 ceph-mon[38805]: osdmap e257: 4 total, 3 up, 4 in 2026-02-23T14:39:05.493 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5139 8 365281 365273 0.555259 0 - 0.0299513 2026-02-23T14:39:06.493 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T14:39:06.492467+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T14:39:06.493 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T14:39:06.493 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5140 8 365281 365273 0.555151 0 - 0.0299513 2026-02-23T14:39:07.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:39:06 trial068 ceph-mon[38805]: pgmap v3130: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:39:07.493 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5141 8 365281 365273 0.555043 0 - 0.0299513 2026-02-23T14:39:07.635 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:39:07.753 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:39:07.857 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:39:07.855+0000 7f9ea2c1a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:39:07.858 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:39:07.856+0000 7f9ea2c1a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:39:07.968 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:39:07.968 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":257,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:39:04.696493+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:35:30.456037+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:38:14.929578+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:34:48.677580+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:39:08.493 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5142 8 365281 365273 0.554935 0 - 0.0299513 2026-02-23T14:39:09.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:39:08 trial068 ceph-mon[38805]: pgmap v3131: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:39:09.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:39:08 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3531895480' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:39:09.493 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5143 8 365281 365273 0.554827 0 - 0.0299513 2026-02-23T14:39:10.493 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5144 8 365281 365273 0.554719 0 - 0.0299513 2026-02-23T14:39:11.005 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:39:11.031 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:39:10 trial068 ceph-mon[38805]: pgmap v3132: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:39:11.124 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:39:11.223 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:39:11.221+0000 7f666022e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:39:11.223 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:39:11.221+0000 7f666022e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:39:11.339 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:39:11.339 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":257,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:39:04.696493+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:35:30.456037+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:38:14.929578+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:34:48.677580+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:39:11.493 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5145 8 365281 365273 0.554612 0 - 0.0299513 2026-02-23T14:39:12.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:39:11 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/134861061' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:39:12.493 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5146 8 365281 365273 0.554504 0 - 0.0299513 2026-02-23T14:39:13.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:39:12 trial068 ceph-mon[38805]: pgmap v3133: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:39:13.493 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5147 8 365281 365273 0.554396 0 - 0.0299513 2026-02-23T14:39:14.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:39:13 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:39:13] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:39:14.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:39:13 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:39:13.974Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:39:14.377 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:39:14.493 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5148 8 365281 365273 0.554288 0 - 0.0299513 2026-02-23T14:39:14.495 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:39:14.595 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:39:14.593+0000 7fe92b456640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:39:14.595 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:39:14.593+0000 7fe92b456640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:39:14.708 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:39:14.708 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":257,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:39:04.696493+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:35:30.456037+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:38:14.929578+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:34:48.677580+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:39:15.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:39:14 trial068 ceph-mon[38805]: pgmap v3134: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:39:15.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:39:14 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2090225433' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:39:15.493 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5149 8 365281 365273 0.554181 0 - 0.0299513 2026-02-23T14:39:16.493 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5150 8 365281 365273 0.554073 0 - 0.0299513 2026-02-23T14:39:17.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:39:16 trial068 ceph-mon[38805]: pgmap v3135: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:39:17.493 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5151 8 365281 365273 0.553966 0 - 0.0299513 2026-02-23T14:39:17.746 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:39:17.862 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:39:17.961 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:39:17.959+0000 7f431ae81640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:39:17.961 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:39:17.959+0000 7f431ae81640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:39:18.071 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:39:18.071 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":257,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:39:04.696493+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:35:30.456037+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:38:14.929578+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:34:48.677580+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:39:18.494 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5152 8 365281 365273 0.553858 0 - 0.0299513 2026-02-23T14:39:19.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:39:18 trial068 ceph-mon[38805]: pgmap v3136: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:39:19.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:39:18 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/205941555' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:39:19.494 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5153 8 365281 365273 0.553751 0 - 0.0299513 2026-02-23T14:39:20.494 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5154 8 365281 365273 0.553643 0 - 0.0299513 2026-02-23T14:39:21.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:39:20 trial068 ceph-mon[38805]: pgmap v3137: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:39:21.109 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:39:21.224 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:39:21.335 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:39:21.333+0000 7f7cf9aea640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:39:21.336 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:39:21.333+0000 7f7cf9aea640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:39:21.447 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:39:21.447 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":257,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:39:04.696493+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:35:30.456037+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:38:14.929578+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:34:48.677580+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:39:21.494 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5155 8 365281 365273 0.553536 0 - 0.0299513 2026-02-23T14:39:22.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:39:21 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2353041352' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:39:22.494 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5156 8 365281 365273 0.553428 0 - 0.0299513 2026-02-23T14:39:23.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:39:22 trial068 ceph-mon[38805]: pgmap v3138: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:39:23.494 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5157 8 365281 365273 0.553321 0 - 0.0299513 2026-02-23T14:39:24.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:39:23 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:39:23] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:39:24.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:39:23 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:39:23.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:39:24.486 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:39:24.494 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5158 8 365281 365273 0.553214 0 - 0.0299513 2026-02-23T14:39:24.605 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:39:24.715 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:39:24.712+0000 7f8111620640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:39:24.715 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:39:24.713+0000 7f8111620640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:39:24.822 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:39:24.822 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":257,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:39:04.696493+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:35:30.456037+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:38:14.929578+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:34:48.677580+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:39:25.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:39:24 trial068 ceph-mon[38805]: pgmap v3139: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:39:25.494 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5159 8 365281 365273 0.553106 0 - 0.0299513 2026-02-23T14:39:26.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:39:25 trial068 ceph-mon[38805]: 5.7 scrub starts 2026-02-23T14:39:26.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:39:25 trial068 ceph-mon[38805]: 5.7 scrub ok 2026-02-23T14:39:26.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:39:25 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/301440863' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:39:26.494 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T14:39:26.493835+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T14:39:26.494 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T14:39:26.494 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5160 8 365281 365273 0.552999 0 - 0.0299513 2026-02-23T14:39:27.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:39:26 trial068 ceph-mon[38805]: pgmap v3140: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:39:27.494 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5161 8 365281 365273 0.552892 0 - 0.0299513 2026-02-23T14:39:27.861 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:39:27.980 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:39:28.081 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:39:28.079+0000 7fcb08431640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:39:28.081 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:39:28.080+0000 7fcb08431640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:39:28.190 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:39:28.190 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":257,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:39:04.696493+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:35:30.456037+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:38:14.929578+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:34:48.677580+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:39:28.494 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5162 8 365281 365273 0.552785 0 - 0.0299513 2026-02-23T14:39:29.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:39:28 trial068 ceph-mon[38805]: pgmap v3141: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:39:29.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:39:28 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2646982438' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:39:29.494 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5163 8 365281 365273 0.552678 0 - 0.0299513 2026-02-23T14:39:30.494 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5164 8 365281 365273 0.552571 0 - 0.0299513 2026-02-23T14:39:31.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:39:30 trial068 ceph-mon[38805]: pgmap v3142: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:39:31.228 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:39:31.347 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:39:31.443 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:39:31.441+0000 7f62b0621640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:39:31.443 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:39:31.442+0000 7f62b0621640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:39:31.494 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5165 8 365281 365273 0.552464 0 - 0.0299513 2026-02-23T14:39:31.554 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:39:31.555 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":257,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:39:04.696493+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:35:30.456037+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:38:14.929578+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:34:48.677580+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:39:31.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:39:31 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2577134927' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:39:32.494 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5166 8 365281 365273 0.552357 0 - 0.0299513 2026-02-23T14:39:33.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:39:32 trial068 ceph-mon[38805]: pgmap v3143: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:39:33.495 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5167 8 365281 365273 0.55225 0 - 0.0299513 2026-02-23T14:39:34.266 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:39:33 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:39:33.974Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:39:34.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:39:33 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:39:33] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:39:34.495 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5168 8 365281 365273 0.552143 0 - 0.0299513 2026-02-23T14:39:34.592 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:39:34.711 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:39:34.813 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:39:34.811+0000 7f5609a70640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:39:34.814 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:39:34.812+0000 7f5609a70640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:39:34.922 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:39:34.922 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":257,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:39:04.696493+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:35:30.456037+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:38:14.929578+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:34:48.677580+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:39:34.922 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:39:34 trial068 ceph-mon[38805]: 5.a scrub starts 2026-02-23T14:39:34.923 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:39:34 trial068 ceph-mon[38805]: 5.a scrub ok 2026-02-23T14:39:34.923 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:39:34 trial068 ceph-mon[38805]: pgmap v3144: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:39:35.495 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5169 8 365281 365273 0.552036 0 - 0.0299513 2026-02-23T14:39:36.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:39:35 trial068 ceph-mon[38805]: 5.5 scrub starts 2026-02-23T14:39:36.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:39:35 trial068 ceph-mon[38805]: 5.5 scrub ok 2026-02-23T14:39:36.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:39:35 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2667002179' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:39:36.495 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5170 8 365281 365273 0.55193 0 - 0.0299513 2026-02-23T14:39:37.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:39:36 trial068 ceph-mon[38805]: pgmap v3145: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:39:37.495 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5171 8 365281 365273 0.551823 0 - 0.0299513 2026-02-23T14:39:37.960 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:39:38.077 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:39:38.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:39:37 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T14:39:38.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:39:37 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T14:39:38.177 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:39:38.175+0000 7ff0ff4c3640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:39:38.177 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:39:38.175+0000 7ff0ff4c3640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:39:38.285 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:39:38.285 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":257,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:39:04.696493+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:35:30.456037+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:38:14.929578+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:34:48.677580+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:39:38.495 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5172 8 365281 365273 0.551716 0 - 0.0299513 2026-02-23T14:39:39.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:39:38 trial068 ceph-mon[38805]: pgmap v3146: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:39:39.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:39:38 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/892163646' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:39:39.495 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5173 8 365281 365273 0.55161 0 - 0.0299513 2026-02-23T14:39:40.495 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5174 8 365281 365273 0.551503 0 - 0.0299513 2026-02-23T14:39:41.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:39:40 trial068 ceph-mon[38805]: pgmap v3147: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:39:41.321 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:39:41.440 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:39:41.495 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5175 8 365281 365273 0.551396 0 - 0.0299513 2026-02-23T14:39:41.540 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:39:41.538+0000 7f5fba753640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:39:41.540 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:39:41.538+0000 7f5fba753640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:39:41.648 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:39:41.649 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":257,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:39:04.696493+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:35:30.456037+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:38:14.929578+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:34:48.677580+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:39:42.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:39:41 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3964026202' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:39:42.495 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5176 8 365281 365273 0.55129 0 - 0.0299513 2026-02-23T14:39:43.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:39:42 trial068 ceph-mon[38805]: pgmap v3148: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:39:43.495 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5177 8 365281 365273 0.551183 0 - 0.0299513 2026-02-23T14:39:44.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:39:43 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:39:43] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:39:44.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:39:43 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:39:43.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:39:44.495 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5178 8 365281 365273 0.551077 0 - 0.0299513 2026-02-23T14:39:44.686 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:39:44.805 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:39:44.904 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:39:44.902+0000 7f1a11260640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:39:44.905 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:39:44.903+0000 7f1a11260640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:39:45.013 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:39:45.013 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":257,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:39:04.696493+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:35:30.456037+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:38:14.929578+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:34:48.677580+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:39:45.013 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:39:44 trial068 ceph-mon[38805]: pgmap v3149: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:39:45.496 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5179 8 365281 365273 0.550971 0 - 0.0299513 2026-02-23T14:39:46.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:39:45 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3923206980' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:39:46.496 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T14:39:46.495372+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T14:39:46.496 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T14:39:46.496 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5180 8 365281 365273 0.550864 0 - 0.0299513 2026-02-23T14:39:47.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:39:46 trial068 ceph-mon[38805]: pgmap v3150: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:39:47.496 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5181 8 365281 365273 0.550758 0 - 0.0299513 2026-02-23T14:39:48.050 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:39:48.167 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:39:48.270 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:39:48.269+0000 7fe49cb92640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:39:48.271 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:39:48.269+0000 7fe49cb92640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:39:48.379 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:39:48.379 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":257,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:39:04.696493+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:35:30.456037+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:38:14.929578+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:34:48.677580+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:39:48.496 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5182 8 365281 365273 0.550652 0 - 0.0299513 2026-02-23T14:39:49.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:39:48 trial068 ceph-mon[38805]: pgmap v3151: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:39:49.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:39:48 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2219789121' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:39:49.496 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5183 8 365281 365273 0.550545 0 - 0.0299513 2026-02-23T14:39:50.496 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5184 8 365281 365273 0.550439 0 - 0.0299513 2026-02-23T14:39:51.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:39:50 trial068 ceph-mon[38805]: 5.e scrub starts 2026-02-23T14:39:51.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:39:50 trial068 ceph-mon[38805]: 5.e scrub ok 2026-02-23T14:39:51.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:39:50 trial068 ceph-mon[38805]: pgmap v3152: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:39:51.416 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:39:51.496 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5185 8 365281 365273 0.550333 0 - 0.0299513 2026-02-23T14:39:51.533 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:39:51.632 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:39:51.630+0000 7f42f3802640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:39:51.632 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:39:51.630+0000 7f42f3802640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:39:51.740 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:39:51.740 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":257,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:39:04.696493+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:35:30.456037+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:38:14.929578+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:34:48.677580+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:39:52.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:39:51 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1675952499' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:39:52.496 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5186 8 365281 365273 0.550227 0 - 0.0299513 2026-02-23T14:39:53.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:39:52 trial068 ceph-mon[38805]: pgmap v3153: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:39:53.496 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5187 8 365281 365273 0.550121 0 - 0.0299513 2026-02-23T14:39:54.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:39:53 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:39:53] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:39:54.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:39:53 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:39:53.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:39:54.496 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5188 8 365281 365273 0.550015 0 - 0.0299513 2026-02-23T14:39:54.777 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:39:54.894 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:39:55.001 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:39:54.999+0000 7f5823fff640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:39:55.002 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:39:55.000+0000 7f5823fff640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:39:55.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:39:54 trial068 ceph-mon[38805]: pgmap v3154: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:39:55.110 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:39:55.111 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":257,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:39:04.696493+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:35:30.456037+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:38:14.929578+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:34:48.677580+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:39:55.496 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5189 8 365281 365273 0.549909 0 - 0.0299513 2026-02-23T14:39:56.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:39:55 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1589326719' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:39:56.496 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5190 8 365281 365273 0.549803 0 - 0.0299513 2026-02-23T14:39:57.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:39:56 trial068 ceph-mon[38805]: pgmap v3155: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:39:57.496 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5191 8 365281 365273 0.549697 0 - 0.0299513 2026-02-23T14:39:58.148 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:39:58.265 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:39:58.366 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:39:58.364+0000 7f7cdc177640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:39:58.367 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:39:58.365+0000 7f7cdc177640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:39:58.476 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:39:58.476 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":257,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:39:04.696493+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:35:30.456037+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:38:14.929578+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:34:48.677580+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:39:58.496 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5192 8 365281 365273 0.549591 0 - 0.0299513 2026-02-23T14:39:59.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:39:58 trial068 ceph-mon[38805]: pgmap v3156: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:39:59.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:39:58 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1844847481' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:39:59.496 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5193 8 365281 365273 0.549485 0 - 0.0299513 2026-02-23T14:40:00.497 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5194 8 365281 365273 0.549379 0 - 0.0299513 2026-02-23T14:40:01.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:40:00 trial068 ceph-mon[38805]: pgmap v3157: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:40:01.497 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5195 8 365281 365273 0.549274 0 - 0.0299513 2026-02-23T14:40:01.513 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:40:01.634 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:40:01.734 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:40:01.732+0000 7f1aada61640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:40:01.735 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:40:01.733+0000 7f1aada61640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:40:01.843 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:40:01.843 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":257,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:39:04.696493+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:35:30.456037+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:38:14.929578+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:34:48.677580+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:40:02.497 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5196 8 365281 365273 0.549168 0 - 0.0299513 2026-02-23T14:40:03.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:40:02 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T14:40:03.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:40:02 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T14:40:03.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:40:02 trial068 ceph-mon[38805]: pgmap v3158: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:40:03.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:40:02 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/4085242386' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:40:03.497 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5197 8 365281 365273 0.549062 0 - 0.0299513 2026-02-23T14:40:04.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:40:03 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2026-02-23T14:40:04.094 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:40:03 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:40:03] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:40:04.266 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:40:03 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:40:03.972Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:40:04.497 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5198 8 365281 365273 0.548957 0 - 0.0299513 2026-02-23T14:40:04.881 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:40:04.997 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:40:05.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:40:04 trial068 ceph-mon[38805]: pgmap v3159: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:40:05.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:40:04 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-23T14:40:05.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:40:04 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2026-02-23T14:40:05.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:40:04 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T14:40:05.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:40:04 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2026-02-23T14:40:05.101 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:40:05.099+0000 7f9a4166c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:40:05.101 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:40:05.099+0000 7f9a4166c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:40:05.212 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:40:05.212 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":257,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:39:04.696493+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:35:30.456037+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:38:14.929578+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:34:48.677580+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:40:05.497 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5199 8 365281 365273 0.548851 0 - 0.0299513 2026-02-23T14:40:06.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:40:05 trial068 ceph-mon[38805]: pgmap v3160: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:40:06.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:40:05 trial068 ceph-mon[38805]: pgmap v3161: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:40:06.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:40:05 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2759063963' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:40:06.497 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T14:40:06.496870+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T14:40:06.497 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T14:40:06.497 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5200 8 365281 365273 0.548745 0 - 0.0299513 2026-02-23T14:40:07.497 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5201 8 365281 365273 0.54864 0 - 0.0299513 2026-02-23T14:40:08.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:40:07 trial068 ceph-mon[38805]: pgmap v3162: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:40:08.248 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:40:08.368 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:40:08.469 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:40:08.467+0000 7fb357435640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:40:08.469 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:40:08.467+0000 7fb357435640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:40:08.497 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5202 8 365281 365273 0.548534 0 - 0.0299513 2026-02-23T14:40:08.578 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:40:08.578 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":257,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:39:04.696493+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:35:30.456037+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:38:14.929578+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:34:48.677580+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:40:08.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:40:08 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/4091009584' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:40:09.497 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5203 8 365281 365273 0.548429 0 - 0.0299513 2026-02-23T14:40:10.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:40:09 trial068 ceph-mon[38805]: pgmap v3163: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:40:10.497 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5204 8 365281 365273 0.548324 0 - 0.0299513 2026-02-23T14:40:11.498 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5205 8 365281 365273 0.548218 0 - 0.0299513 2026-02-23T14:40:11.614 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:40:11.732 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:40:11.830 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:40:11.828+0000 7efddc6e8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:40:11.830 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:40:11.828+0000 7efddc6e8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:40:11.939 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:40:11.939 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":257,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:39:04.696493+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:35:30.456037+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:38:14.929578+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:34:48.677580+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:40:11.939 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:40:11 trial068 ceph-mon[38805]: pgmap v3164: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:40:12.498 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5206 8 365281 365273 0.548113 0 - 0.0299513 2026-02-23T14:40:13.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:40:12 trial068 ceph-mon[38805]: 5.8 scrub starts 2026-02-23T14:40:13.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:40:12 trial068 ceph-mon[38805]: 5.8 scrub ok 2026-02-23T14:40:13.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:40:12 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/639156574' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:40:13.498 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5207 8 365281 365273 0.548008 0 - 0.0299513 2026-02-23T14:40:14.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:40:13 trial068 ceph-mon[38805]: pgmap v3165: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:40:14.094 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:40:13 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:40:13] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:40:14.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:40:13 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:40:13.974Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:40:14.498 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5208 8 365281 365273 0.547903 0 - 0.0299513 2026-02-23T14:40:14.975 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:40:15.095 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:40:15.196 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:40:15.194+0000 7f66adfb2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:40:15.196 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:40:15.194+0000 7f66adfb2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:40:15.309 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:40:15.310 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":257,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:39:04.696493+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:35:30.456037+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:38:14.929578+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:34:48.677580+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:40:15.498 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5209 8 365281 365273 0.547797 0 - 0.0299513 2026-02-23T14:40:16.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:40:15 trial068 ceph-mon[38805]: pgmap v3166: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:40:16.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:40:15 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3352994306' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:40:16.498 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5210 8 365281 365273 0.547692 0 - 0.0299513 2026-02-23T14:40:17.498 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5211 8 365281 365273 0.547587 0 - 0.0299513 2026-02-23T14:40:18.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:40:17 trial068 ceph-mon[38805]: pgmap v3167: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:40:18.347 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:40:18.466 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:40:18.498 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5212 8 365281 365273 0.547482 0 - 0.0299513 2026-02-23T14:40:18.567 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:40:18.565+0000 7f281c454640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:40:18.568 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:40:18.566+0000 7f281c454640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:40:18.677 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:40:18.677 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":257,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:39:04.696493+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:35:30.456037+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:38:14.929578+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:34:48.677580+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:40:19.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:40:18 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1863121740' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:40:19.498 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5213 8 365281 365273 0.547377 0 - 0.0299513 2026-02-23T14:40:20.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:40:19 trial068 ceph-mon[38805]: pgmap v3168: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:40:20.498 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5214 8 365281 365273 0.547272 0 - 0.0299513 2026-02-23T14:40:21.498 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5215 8 365281 365273 0.547167 0 - 0.0299513 2026-02-23T14:40:21.714 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:40:21.832 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:40:21.933 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:40:21.932+0000 7f56ef96a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:40:21.934 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:40:21.932+0000 7f56ef96a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:40:22.043 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:40:22.044 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":258,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:40:20.827101+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:35:30.456037+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:38:14.929578+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:40:01.279839+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:40:22.044 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:40:21 trial068 ceph-mon[38805]: pgmap v3169: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:40:22.044 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:40:21 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T14:40:22.044 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:40:21 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T14:40:22.044 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:40:21 trial068 ceph-mon[38805]: osdmap e258: 4 total, 3 up, 4 in 2026-02-23T14:40:22.498 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5216 8 365281 365273 0.547062 0 - 0.0299513 2026-02-23T14:40:23.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:40:22 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/27909122' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:40:23.498 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5217 8 365281 365273 0.546957 0 - 0.0299513 2026-02-23T14:40:24.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:40:23 trial068 ceph-mon[38805]: pgmap v3171: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:40:24.094 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:40:23 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:40:23] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:40:24.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:40:23 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:40:23.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:40:24.498 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5218 8 365281 365273 0.546852 0 - 0.0299513 2026-02-23T14:40:25.081 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:40:25.204 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:40:25.305 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:40:25.303+0000 7f779fcb3640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:40:25.305 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:40:25.303+0000 7f779fcb3640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:40:25.414 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:40:25.415 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":259,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:40:24.706209+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:35:30.456037+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:38:14.929578+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:40:01.279839+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:40:25.499 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5219 8 365281 365273 0.546748 0 - 0.0299513 2026-02-23T14:40:26.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:40:25 trial068 ceph-mon[38805]: pgmap v3172: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:40:26.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:40:25 trial068 ceph-mon[38805]: osdmap e259: 4 total, 3 up, 4 in 2026-02-23T14:40:26.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:40:25 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3505205143' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:40:26.499 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T14:40:26.498501+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T14:40:26.499 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T14:40:26.499 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5220 8 365281 365273 0.546643 0 - 0.0299513 2026-02-23T14:40:27.499 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5221 8 365281 365273 0.546538 0 - 0.0299513 2026-02-23T14:40:28.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:40:27 trial068 ceph-mon[38805]: pgmap v3174: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:40:28.453 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:40:28.499 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5222 8 365281 365273 0.546434 0 - 0.0299513 2026-02-23T14:40:28.572 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:40:28.671 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:40:28.669+0000 7f24735dd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:40:28.672 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:40:28.670+0000 7f24735dd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:40:28.780 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:40:28.780 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":259,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:40:24.706209+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:35:30.456037+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:38:14.929578+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:40:01.279839+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:40:29.499 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5223 8 365281 365273 0.546329 0 - 0.0299513 2026-02-23T14:40:30.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:40:29 trial068 ceph-mon[38805]: pgmap v3175: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:40:30.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:40:29 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3948724015' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:40:30.499 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5224 8 365281 365273 0.546224 0 - 0.0299513 2026-02-23T14:40:31.499 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5225 8 365281 365273 0.54612 0 - 0.0299513 2026-02-23T14:40:31.819 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:40:31.937 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:40:31.986 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:40:31 trial068 ceph-mon[38805]: pgmap v3176: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:40:32.045 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:40:32.043+0000 7f2c66aa4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:40:32.045 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:40:32.043+0000 7f2c66aa4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:40:32.155 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:40:32.155 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":259,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:40:24.706209+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:35:30.456037+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:38:14.929578+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:40:01.279839+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:40:32.499 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5226 8 365281 365273 0.546015 0 - 0.0299513 2026-02-23T14:40:33.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:40:32 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/517195582' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:40:33.499 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5227 8 365281 365273 0.545911 0 - 0.0299513 2026-02-23T14:40:33.975 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:40:33 trial068 ceph-mon[38805]: pgmap v3177: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:40:34.266 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:40:33 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:40:33.974Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:40:34.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:40:33 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:40:33] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:40:34.499 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5228 8 365281 365273 0.545806 0 - 0.0299513 2026-02-23T14:40:35.194 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:40:35.312 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:40:35.419 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:40:35.417+0000 7fda35c85640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:40:35.420 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:40:35.417+0000 7fda35c85640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:40:35.499 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5229 8 365281 365273 0.545702 0 - 0.0299513 2026-02-23T14:40:35.530 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:40:35.530 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":259,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:40:24.706209+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:35:30.456037+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:38:14.929578+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:40:01.279839+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:40:35.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:40:35 trial068 ceph-mon[38805]: pgmap v3178: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:40:35.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:40:35 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3529773893' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:40:36.499 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5230 8 365281 365273 0.545598 0 - 0.0299513 2026-02-23T14:40:37.500 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5231 8 365281 365273 0.545493 0 - 0.0299513 2026-02-23T14:40:38.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:40:37 trial068 ceph-mon[38805]: pgmap v3179: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:40:38.500 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5232 8 365281 365273 0.545389 0 - 0.0299513 2026-02-23T14:40:38.569 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:40:38.687 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:40:38.788 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:40:38.786+0000 7f88d4caa640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:40:38.789 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:40:38.786+0000 7f88d4caa640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:40:38.900 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:40:38.900 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":259,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:40:24.706209+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:35:30.456037+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:38:14.929578+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:40:01.279839+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:40:38.901 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:40:38 trial068 ceph-mon[38805]: 5.5 scrub starts 2026-02-23T14:40:38.901 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:40:38 trial068 ceph-mon[38805]: 5.5 scrub ok 2026-02-23T14:40:39.500 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5233 8 365281 365273 0.545285 0 - 0.0299513 2026-02-23T14:40:40.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:40:39 trial068 ceph-mon[38805]: pgmap v3180: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:40:40.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:40:39 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1176553203' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:40:40.500 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5234 8 365281 365273 0.545181 0 - 0.0299513 2026-02-23T14:40:41.500 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5235 8 365281 365273 0.545077 0 - 0.0299513 2026-02-23T14:40:41.939 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:40:42.058 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:40:42.080 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:40:41 trial068 ceph-mon[38805]: pgmap v3181: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:40:42.156 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:40:42.154+0000 7fe7dd985640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:40:42.157 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:40:42.155+0000 7fe7dd985640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:40:42.268 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:40:42.268 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":259,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:40:24.706209+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:35:30.456037+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:38:14.929578+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:40:01.279839+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:40:42.500 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5236 8 365281 365273 0.544973 0 - 0.0299513 2026-02-23T14:40:43.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:40:42 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3575555084' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:40:43.500 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5237 8 365281 365273 0.544869 0 - 0.0299513 2026-02-23T14:40:43.974 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:40:43 trial068 ceph-mon[38805]: pgmap v3182: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:40:44.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:40:43 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:40:43] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:40:44.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:40:43 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:40:43.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:40:44.500 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5238 8 365281 365273 0.544764 0 - 0.0299513 2026-02-23T14:40:45.307 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:40:45.427 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:40:45.500 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5239 8 365281 365273 0.544661 0 - 0.0299513 2026-02-23T14:40:45.527 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:40:45.525+0000 7fa0c3049640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:40:45.527 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:40:45.526+0000 7fa0c3049640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:40:45.640 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:40:45.640 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":259,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:40:24.706209+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:35:30.456037+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:38:14.929578+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:40:01.279839+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:40:46.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:40:45 trial068 ceph-mon[38805]: pgmap v3183: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:40:46.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:40:45 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2560794922' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:40:46.500 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T14:40:46.499906+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T14:40:46.500 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T14:40:46.500 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5240 8 365281 365273 0.544557 0 - 0.0299513 2026-02-23T14:40:47.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:40:46 trial068 ceph-mon[38805]: 5.7 scrub starts 2026-02-23T14:40:47.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:40:46 trial068 ceph-mon[38805]: 5.7 scrub ok 2026-02-23T14:40:47.500 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5241 8 365281 365273 0.544453 0 - 0.0299513 2026-02-23T14:40:48.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:40:47 trial068 ceph-mon[38805]: pgmap v3184: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:40:48.500 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5242 8 365281 365273 0.544349 0 - 0.0299513 2026-02-23T14:40:48.680 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:40:48.796 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:40:48.896 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:40:48.895+0000 7fe901dab640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:40:48.897 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:40:48.895+0000 7fe901dab640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:40:49.009 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:40:49.009 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":259,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:40:24.706209+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:35:30.456037+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:38:14.929578+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:40:01.279839+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:40:49.500 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5243 8 365281 365273 0.544245 0 - 0.0299513 2026-02-23T14:40:50.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:40:49 trial068 ceph-mon[38805]: pgmap v3185: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:40:50.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:40:49 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3461531751' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:40:50.500 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5244 8 365281 365273 0.544141 0 - 0.0299513 2026-02-23T14:40:51.500 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5245 8 365281 365273 0.544037 0 - 0.0299513 2026-02-23T14:40:52.047 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:40:52.072 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:40:51 trial068 ceph-mon[38805]: pgmap v3186: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:40:52.167 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:40:52.263 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:40:52.261+0000 7fd93b02f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:40:52.264 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:40:52.262+0000 7fd93b02f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:40:52.373 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:40:52.373 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":259,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:40:24.706209+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:35:30.456037+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:38:14.929578+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:40:01.279839+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:40:52.501 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5246 8 365281 365273 0.543934 0 - 0.0299513 2026-02-23T14:40:53.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:40:52 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/4154029225' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:40:53.501 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5247 8 365281 365273 0.54383 0 - 0.0299513 2026-02-23T14:40:54.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:40:53 trial068 ceph-mon[38805]: pgmap v3187: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:40:54.094 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:40:53 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:40:53] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:40:54.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:40:53 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:40:53.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:40:54.501 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5248 8 365281 365273 0.543726 0 - 0.0299513 2026-02-23T14:40:55.411 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:40:55.501 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5249 8 365281 365273 0.543623 0 - 0.0299513 2026-02-23T14:40:55.530 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:40:55.634 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:40:55.632+0000 7f2c97b5b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:40:55.634 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:40:55.633+0000 7f2c97b5b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:40:55.745 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:40:55.745 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":259,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:40:24.706209+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:35:30.456037+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:38:14.929578+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:40:01.279839+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:40:56.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:40:55 trial068 ceph-mon[38805]: pgmap v3188: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:40:56.501 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5250 8 365281 365273 0.543519 0 - 0.0299513 2026-02-23T14:40:57.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:40:56 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1782883472' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:40:57.501 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5251 8 365281 365273 0.543416 0 - 0.0299513 2026-02-23T14:40:58.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:40:57 trial068 ceph-mon[38805]: pgmap v3189: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:40:58.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:40:57 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T14:40:58.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:40:57 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T14:40:58.501 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5252 8 365281 365273 0.543312 0 - 0.0299513 2026-02-23T14:40:58.803 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:40:58.921 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:40:59.018 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:40:59.016+0000 7fbb3a0b9640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:40:59.018 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:40:59.017+0000 7fbb3a0b9640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:40:59.128 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:40:59.129 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":259,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:40:24.706209+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:35:30.456037+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:38:14.929578+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:40:01.279839+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:40:59.501 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5253 8 365281 365273 0.543209 0 - 0.0299513 2026-02-23T14:41:00.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:40:59 trial068 ceph-mon[38805]: pgmap v3190: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:41:00.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:40:59 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/127649047' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:41:00.501 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5254 8 365281 365273 0.543106 0 - 0.0299513 2026-02-23T14:41:01.501 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5255 8 365281 365273 0.543002 0 - 0.0299513 2026-02-23T14:41:02.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:41:01 trial068 ceph-mon[38805]: pgmap v3191: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:41:02.166 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:41:02.283 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:41:02.383 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:41:02.381+0000 7f1ad6217640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:41:02.383 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:41:02.381+0000 7f1ad6217640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:41:02.498 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:41:02.498 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":259,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:40:24.706209+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:35:30.456037+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:38:14.929578+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:40:01.279839+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:41:02.501 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5256 8 365281 365273 0.542899 0 - 0.0299513 2026-02-23T14:41:02.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:41:02 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1766513299' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:41:03.501 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5257 8 365281 365273 0.542796 0 - 0.0299513 2026-02-23T14:41:04.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:41:03 trial068 ceph-mon[38805]: pgmap v3192: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:41:04.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:41:03 trial068 ceph-mon[38805]: 5.a scrub starts 2026-02-23T14:41:04.094 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:41:03 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:41:03] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:41:04.266 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:41:03 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:41:03.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:41:04.501 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5258 8 365281 365273 0.542692 0 - 0.0299513 2026-02-23T14:41:05.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:41:04 trial068 ceph-mon[38805]: 5.a scrub ok 2026-02-23T14:41:05.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:41:04 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2026-02-23T14:41:05.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:41:04 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-23T14:41:05.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:41:04 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2026-02-23T14:41:05.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:41:04 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T14:41:05.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:41:04 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2026-02-23T14:41:05.501 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5259 8 365281 365273 0.542589 0 - 0.0299513 2026-02-23T14:41:05.536 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:41:05.655 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:41:05.756 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:41:05.754+0000 7f87d8a0d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:41:05.756 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:41:05.754+0000 7f87d8a0d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:41:05.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:41:05 trial068 ceph-mon[38805]: pgmap v3193: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:41:05.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:41:05 trial068 ceph-mon[38805]: pgmap v3194: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:41:05.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:41:05 trial068 ceph-mon[38805]: pgmap v3195: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:41:05.864 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:41:05.864 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":259,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:40:24.706209+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:35:30.456037+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:38:14.929578+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:40:01.279839+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:41:06.502 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T14:41:06.501334+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T14:41:06.502 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T14:41:06.502 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5260 8 365281 365273 0.542486 0 - 0.0299513 2026-02-23T14:41:07.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:41:06 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2583278300' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:41:07.502 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5261 8 365281 365273 0.542383 0 - 0.0299513 2026-02-23T14:41:08.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:41:07 trial068 ceph-mon[38805]: pgmap v3196: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:41:08.502 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5262 8 365281 365273 0.54228 0 - 0.0299513 2026-02-23T14:41:08.901 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:41:09.020 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:41:09.123 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:41:09.121+0000 7f8418264640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:41:09.124 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:41:09.122+0000 7f8418264640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:41:09.235 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:41:09.235 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":259,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:40:24.706209+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:35:30.456037+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:38:14.929578+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:40:01.279839+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:41:09.502 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5263 8 365281 365273 0.542177 0 - 0.0299513 2026-02-23T14:41:10.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:41:09 trial068 ceph-mon[38805]: pgmap v3197: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:41:10.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:41:09 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/4113495586' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:41:10.502 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5264 8 365281 365273 0.542074 0 - 0.0299513 2026-02-23T14:41:11.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:41:10 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T14:41:11.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:41:10 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T14:41:11.502 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5265 8 365281 365273 0.541971 0 - 0.0299513 2026-02-23T14:41:12.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:41:11 trial068 ceph-mon[38805]: pgmap v3198: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:41:12.273 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:41:12.392 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:41:12.489 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:41:12.487+0000 7f57d7dff640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:41:12.489 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:41:12.487+0000 7f57d7dff640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:41:12.502 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5266 8 365281 365273 0.541868 0 - 0.0299513 2026-02-23T14:41:12.598 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:41:12.598 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":259,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:40:24.706209+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:35:30.456037+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:38:14.929578+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:40:01.279839+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:41:13.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:41:12 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2796854984' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:41:13.502 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5267 8 365281 365273 0.541765 0 - 0.0299513 2026-02-23T14:41:14.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:41:13 trial068 ceph-mon[38805]: pgmap v3199: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:41:14.094 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:41:13 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:41:13] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:41:14.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:41:13 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:41:13.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:41:14.502 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5268 8 365281 365273 0.541662 0 - 0.0299513 2026-02-23T14:41:15.502 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5269 8 365281 365273 0.541559 0 - 0.0299513 2026-02-23T14:41:15.636 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:41:15.755 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:41:15.855 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:41:15.853+0000 7f2195d63640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:41:15.856 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:41:15.854+0000 7f2195d63640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:41:15.965 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:41:15.965 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":259,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:40:24.706209+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:35:30.456037+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:38:14.929578+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:40:01.279839+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:41:15.965 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:41:15 trial068 ceph-mon[38805]: pgmap v3200: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:41:16.502 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5270 8 365281 365273 0.541457 0 - 0.0299513 2026-02-23T14:41:17.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:41:16 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/4168641711' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:41:17.502 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5271 8 365281 365273 0.541354 0 - 0.0299513 2026-02-23T14:41:18.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:41:17 trial068 ceph-mon[38805]: pgmap v3201: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:41:18.502 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5272 8 365281 365273 0.541251 0 - 0.0299513 2026-02-23T14:41:19.004 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:41:19.029 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:41:18 trial068 ceph-mon[38805]: 5.e scrub starts 2026-02-23T14:41:19.029 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:41:18 trial068 ceph-mon[38805]: 5.e scrub ok 2026-02-23T14:41:19.121 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:41:19.222 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:41:19.220+0000 7fe61ace2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:41:19.222 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:41:19.220+0000 7fe61ace2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:41:19.332 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:41:19.332 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":259,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:40:24.706209+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:35:30.456037+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:38:14.929578+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:40:01.279839+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:41:19.503 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5273 8 365281 365273 0.541149 0 - 0.0299513 2026-02-23T14:41:20.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:41:19 trial068 ceph-mon[38805]: pgmap v3202: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:41:20.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:41:19 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1181006944' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:41:20.503 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5274 8 365281 365273 0.541046 0 - 0.0299513 2026-02-23T14:41:21.503 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5275 8 365281 365273 0.540943 0 - 0.0299513 2026-02-23T14:41:22.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:41:21 trial068 ceph-mon[38805]: pgmap v3203: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:41:22.368 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:41:22.487 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:41:22.503 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5276 8 365281 365273 0.540841 0 - 0.0299513 2026-02-23T14:41:22.592 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:41:22.590+0000 7f8acb7ed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:41:22.592 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:41:22.590+0000 7f8acb7ed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:41:22.702 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:41:22.702 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":259,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:40:24.706209+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:35:30.456037+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:38:14.929578+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:40:01.279839+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:41:23.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:41:22 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1343263935' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:41:23.503 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5277 8 365281 365273 0.540738 0 - 0.0299513 2026-02-23T14:41:24.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:41:23 trial068 ceph-mon[38805]: pgmap v3204: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:41:24.094 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:41:23 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:41:23] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:41:24.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:41:23 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:41:23.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:41:24.503 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5278 8 365281 365273 0.540636 0 - 0.0299513 2026-02-23T14:41:25.503 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5279 8 365281 365273 0.540534 0 - 0.0299513 2026-02-23T14:41:25.739 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:41:25.860 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:41:25.969 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:41:25.967+0000 7fe6421b7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:41:25.970 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:41:25.968+0000 7fe6421b7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:41:26.078 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:41:26.078 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":259,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:40:24.706209+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:35:30.456037+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:38:14.929578+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:40:01.279839+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:41:26.079 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:41:25 trial068 ceph-mon[38805]: pgmap v3205: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:41:26.503 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T14:41:26.502865+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T14:41:26.503 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T14:41:26.504 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5280 8 365281 365273 0.540431 0 - 0.0299513 2026-02-23T14:41:27.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:41:26 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3507085642' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:41:27.503 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5281 8 365281 365273 0.540329 0 - 0.0299513 2026-02-23T14:41:28.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:41:27 trial068 ceph-mon[38805]: pgmap v3206: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:41:28.503 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5282 8 365281 365273 0.540227 0 - 0.0299513 2026-02-23T14:41:29.135 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:41:29.253 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:41:29.349 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:41:29.347+0000 7f9ee2b72640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:41:29.349 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:41:29.347+0000 7f9ee2b72640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:41:29.458 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:41:29.458 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":260,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:41:28.769975+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:40:20.346927+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:38:14.929578+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:40:01.279839+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:41:29.503 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5283 8 365281 365273 0.540124 0 - 0.0299513 2026-02-23T14:41:30.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:41:29 trial068 ceph-mon[38805]: pgmap v3207: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:41:30.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:41:29 trial068 ceph-mon[38805]: osdmap e260: 4 total, 3 up, 4 in 2026-02-23T14:41:30.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:41:29 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/327528152' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:41:30.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:41:29 trial068 ceph-mon[38805]: osdmap e261: 4 total, 3 up, 4 in 2026-02-23T14:41:30.503 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5284 8 365281 365273 0.540022 0 - 0.0299513 2026-02-23T14:41:31.504 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5285 8 365281 365273 0.53992 0 - 0.0299513 2026-02-23T14:41:32.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:41:31 trial068 ceph-mon[38805]: pgmap v3210: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:41:32.495 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:41:32.503 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5286 8 365281 365273 0.539818 0 - 0.0299513 2026-02-23T14:41:32.614 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:41:32.718 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:41:32.716+0000 7f7c3beac640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:41:32.718 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:41:32.717+0000 7f7c3beac640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:41:32.829 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:41:32.829 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":261,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:41:29.716652+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:40:20.346927+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:38:14.929578+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:40:01.279839+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:41:33.504 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5287 8 365281 365273 0.539716 0 - 0.0299513 2026-02-23T14:41:34.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:41:33 trial068 ceph-mon[38805]: pgmap v3211: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:41:34.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:41:33 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1688614099' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:41:34.094 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:41:33 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:41:33] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:41:34.266 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:41:33 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:41:33.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:41:34.504 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5288 8 365281 365273 0.539614 0 - 0.0299513 2026-02-23T14:41:35.504 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5289 8 365281 365273 0.539512 0 - 0.0299513 2026-02-23T14:41:35.867 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:41:35.983 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:41:36.031 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:41:35 trial068 ceph-mon[38805]: pgmap v3212: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:41:36.090 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:41:36.088+0000 7f22994b4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:41:36.090 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:41:36.088+0000 7f22994b4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:41:36.200 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:41:36.201 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":261,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:41:29.716652+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:40:20.346927+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:38:14.929578+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:40:01.279839+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:41:36.504 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5290 8 365281 365273 0.53941 0 - 0.0299513 2026-02-23T14:41:37.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:41:36 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T14:41:37.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:41:36 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T14:41:37.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:41:36 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1828828385' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:41:37.504 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5291 8 365281 365273 0.539308 0 - 0.0299513 2026-02-23T14:41:38.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:41:37 trial068 ceph-mon[38805]: pgmap v3213: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:41:38.504 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5292 8 365281 365273 0.539206 0 - 0.0299513 2026-02-23T14:41:39.239 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:41:39.356 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:41:39.458 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:41:39.456+0000 7f62befeb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:41:39.458 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:41:39.457+0000 7f62befeb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:41:39.504 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5293 8 365281 365273 0.539104 0 - 0.0299513 2026-02-23T14:41:39.568 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:41:39.568 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":261,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:41:29.716652+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:40:20.346927+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:38:14.929578+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:40:01.279839+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:41:39.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:41:39 trial068 ceph-mon[38805]: pgmap v3214: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:41:39.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:41:39 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/4240973589' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:41:40.504 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5294 8 365281 365273 0.539002 0 - 0.0299513 2026-02-23T14:41:41.504 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5295 8 365281 365273 0.5389 0 - 0.0299513 2026-02-23T14:41:42.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:41:41 trial068 ceph-mon[38805]: pgmap v3215: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:41:42.504 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5296 8 365281 365273 0.538798 0 - 0.0299513 2026-02-23T14:41:42.606 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:41:42.724 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:41:42.827 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:41:42.825+0000 7f1187ef9640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:41:42.827 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:41:42.825+0000 7f1187ef9640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:41:42.935 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:41:42.935 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":261,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:41:29.716652+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:40:20.346927+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:38:14.929578+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:40:01.279839+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:41:42.936 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:41:42 trial068 ceph-mon[38805]: 5.5 scrub starts 2026-02-23T14:41:42.936 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:41:42 trial068 ceph-mon[38805]: 5.5 scrub ok 2026-02-23T14:41:43.504 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5297 8 365281 365273 0.538697 0 - 0.0299513 2026-02-23T14:41:44.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:41:43 trial068 ceph-mon[38805]: pgmap v3216: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:41:44.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:41:43 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/367757444' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:41:44.094 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:41:43 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:41:43] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:41:44.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:41:43 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:41:43.974Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:41:44.504 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5298 8 365281 365273 0.538595 0 - 0.0299513 2026-02-23T14:41:45.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:41:44 trial068 ceph-mon[38805]: 5.8 scrub starts 2026-02-23T14:41:45.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:41:44 trial068 ceph-mon[38805]: 5.8 scrub ok 2026-02-23T14:41:45.504 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5299 8 365281 365273 0.538493 0 - 0.0299513 2026-02-23T14:41:45.973 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:41:46.088 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:41:46.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:41:45 trial068 ceph-mon[38805]: pgmap v3217: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:41:46.192 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:41:46.190+0000 7fd887084640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:41:46.192 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:41:46.190+0000 7fd887084640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:41:46.301 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:41:46.301 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":261,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:41:29.716652+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:40:20.346927+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:38:14.929578+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:40:01.279839+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:41:46.505 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T14:41:46.504316+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T14:41:46.505 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T14:41:46.505 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5300 8 365281 365273 0.538392 0 - 0.0299513 2026-02-23T14:41:47.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:41:46 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/63262680' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:41:47.505 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5301 8 365281 365273 0.53829 0 - 0.0299513 2026-02-23T14:41:48.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:41:47 trial068 ceph-mon[38805]: pgmap v3218: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:41:48.505 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5302 8 365281 365273 0.538189 0 - 0.0299513 2026-02-23T14:41:49.337 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:41:49.457 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:41:49.505 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5303 8 365281 365273 0.538087 0 - 0.0299513 2026-02-23T14:41:49.552 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:41:49.551+0000 7f6bea2c4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:41:49.553 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:41:49.551+0000 7f6bea2c4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:41:49.665 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:41:49.665 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":261,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:41:29.716652+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:40:20.346927+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:38:14.929578+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:40:01.279839+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:41:50.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:41:49 trial068 ceph-mon[38805]: pgmap v3219: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:41:50.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:41:49 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/499077493' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:41:50.505 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5304 8 365281 365273 0.537986 0 - 0.0299513 2026-02-23T14:41:51.505 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5305 8 365281 365273 0.537884 0 - 0.0299513 2026-02-23T14:41:52.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:41:51 trial068 ceph-mon[38805]: pgmap v3220: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:41:52.505 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5306 8 365281 365273 0.537783 0 - 0.0299513 2026-02-23T14:41:52.702 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:41:52.822 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:41:52.920 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:41:52.918+0000 7faeecc30640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:41:52.920 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:41:52.918+0000 7faeecc30640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:41:53.029 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:41:53.029 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":261,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:41:29.716652+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:40:20.346927+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:38:14.929578+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:40:01.279839+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:41:53.505 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5307 8 365281 365273 0.537682 0 - 0.0299513 2026-02-23T14:41:54.094 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:41:53 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:41:53] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:41:54.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:41:53 trial068 ceph-mon[38805]: pgmap v3221: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:41:54.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:41:53 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3593579766' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:41:54.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:41:53 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:41:53.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:41:54.505 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5308 8 365281 365273 0.53758 0 - 0.0299513 2026-02-23T14:41:55.505 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5309 8 365281 365273 0.537479 0 - 0.0299513 2026-02-23T14:41:56.066 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:41:56.091 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:41:55 trial068 ceph-mon[38805]: pgmap v3222: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:41:56.184 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:41:56.288 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:41:56.286+0000 7fdc57253640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:41:56.288 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:41:56.286+0000 7fdc57253640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:41:56.396 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:41:56.397 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":261,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:41:29.716652+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:40:20.346927+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:38:14.929578+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:40:01.279839+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:41:56.505 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5310 8 365281 365273 0.537378 0 - 0.0299513 2026-02-23T14:41:57.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:41:56 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3607380462' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:41:57.505 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5311 8 365281 365273 0.537277 0 - 0.0299513 2026-02-23T14:41:58.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:41:57 trial068 ceph-mon[38805]: pgmap v3223: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:41:58.506 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5312 8 365281 365273 0.537176 0 - 0.0299513 2026-02-23T14:41:59.433 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:41:59.506 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5313 8 365281 365273 0.537074 0 - 0.0299513 2026-02-23T14:41:59.551 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:41:59.649 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:41:59.647+0000 7f5a26518640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:41:59.649 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:41:59.648+0000 7f5a26518640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:41:59.760 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:41:59.760 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":261,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:41:29.716652+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:40:20.346927+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:38:14.929578+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:40:01.279839+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:42:00.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:41:59 trial068 ceph-mon[38805]: pgmap v3224: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:42:00.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:41:59 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2724792359' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:42:00.506 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5314 8 365281 365273 0.536973 0 - 0.0299513 2026-02-23T14:42:01.506 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5315 8 365281 365273 0.536872 0 - 0.0299513 2026-02-23T14:42:02.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:42:01 trial068 ceph-mon[38805]: pgmap v3225: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:42:02.506 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5316 8 365281 365273 0.536771 0 - 0.0299513 2026-02-23T14:42:02.799 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:42:02.916 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:42:03.018 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:42:03.016+0000 7f32ab1b2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:42:03.018 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:42:03.016+0000 7f32ab1b2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:42:03.127 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:42:03.128 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":261,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:41:29.716652+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:40:20.346927+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:38:14.929578+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:40:01.279839+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:42:03.506 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5317 8 365281 365273 0.53667 0 - 0.0299513 2026-02-23T14:42:03.975 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:42:03 trial068 ceph-mon[38805]: pgmap v3226: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:42:03.975 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:42:03 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3298475068' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:42:04.265 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:42:03 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:42:03.974Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:42:04.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:42:03 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:42:03] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:42:04.506 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5318 8 365281 365273 0.536569 0 - 0.0299513 2026-02-23T14:42:05.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:42:04 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2026-02-23T14:42:05.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:42:04 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-23T14:42:05.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:42:04 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2026-02-23T14:42:05.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:42:04 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T14:42:05.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:42:04 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2026-02-23T14:42:05.506 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5319 8 365281 365273 0.536469 0 - 0.0299513 2026-02-23T14:42:06.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:42:05 trial068 ceph-mon[38805]: pgmap v3227: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:42:06.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:42:05 trial068 ceph-mon[38805]: pgmap v3228: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:42:06.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:42:05 trial068 ceph-mon[38805]: pgmap v3229: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:42:06.166 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:42:06.282 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:42:06.391 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:42:06.389+0000 7f91a7848640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:42:06.392 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:42:06.390+0000 7f91a7848640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:42:06.499 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:42:06.499 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":261,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:41:29.716652+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:40:20.346927+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:38:14.929578+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:40:01.279839+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:42:06.506 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T14:42:06.505970+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T14:42:06.506 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T14:42:06.506 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5320 8 365281 365273 0.536368 0 - 0.0299513 2026-02-23T14:42:07.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:42:06 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1270925200' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:42:07.506 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5321 8 365281 365273 0.536267 0 - 0.0299513 2026-02-23T14:42:08.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:42:07 trial068 ceph-mon[38805]: pgmap v3230: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:42:08.506 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5322 8 365281 365273 0.536166 0 - 0.0299513 2026-02-23T14:42:09.506 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5323 8 365281 365273 0.536065 0 - 0.0299513 2026-02-23T14:42:09.539 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:42:09.657 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:42:09.754 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:42:09.752+0000 7facb6b36640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:42:09.754 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:42:09.752+0000 7facb6b36640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:42:09.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:42:09 trial068 ceph-mon[38805]: pgmap v3231: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:42:09.863 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:42:09.863 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":261,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:41:29.716652+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:40:20.346927+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:38:14.929578+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:40:01.279839+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:42:10.507 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5324 8 365281 365273 0.535965 0 - 0.0299513 2026-02-23T14:42:11.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:42:10 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3165334120' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:42:11.507 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5325 8 365281 365273 0.535864 0 - 0.0299513 2026-02-23T14:42:12.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:42:11 trial068 ceph-mon[38805]: pgmap v3232: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:42:12.507 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5326 8 365281 365273 0.535764 0 - 0.0299513 2026-02-23T14:42:12.900 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:42:13.018 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:42:13.068 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:42:12 trial068 ceph-mon[38805]: 5.a scrub starts 2026-02-23T14:42:13.068 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:42:12 trial068 ceph-mon[38805]: 5.a scrub ok 2026-02-23T14:42:13.068 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:42:12 trial068 ceph-mon[38805]: 5.7 scrub starts 2026-02-23T14:42:13.068 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:42:12 trial068 ceph-mon[38805]: 5.7 scrub ok 2026-02-23T14:42:13.114 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:42:13.112+0000 7f4b14935640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:42:13.114 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:42:13.113+0000 7f4b14935640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:42:13.223 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:42:13.223 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":261,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:41:29.716652+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:40:20.346927+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:38:14.929578+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:40:01.279839+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:42:13.507 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5327 8 365281 365273 0.535663 0 - 0.0299513 2026-02-23T14:42:14.094 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:42:13 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:42:13] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:42:14.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:42:13 trial068 ceph-mon[38805]: pgmap v3233: 17 pgs: 1 active+clean+scrubbing, 1 active+recovery_unfound+undersized+degraded, 4 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:42:14.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:42:13 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T14:42:14.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:42:13 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T14:42:14.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:42:13 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1892623665' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:42:14.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:42:13 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:42:13.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:42:14.507 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5328 8 365281 365273 0.535562 0 - 0.0299513 2026-02-23T14:42:15.507 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5329 8 365281 365273 0.535462 0 - 0.0299513 2026-02-23T14:42:16.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:42:15 trial068 ceph-mon[38805]: pgmap v3234: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 1 active+clean+scrubbing, 4 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:42:16.260 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:42:16.380 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:42:16.479 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:42:16.477+0000 7f33f84a9640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:42:16.479 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:42:16.477+0000 7f33f84a9640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:42:16.507 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5330 8 365281 365273 0.535361 0 - 0.0299513 2026-02-23T14:42:16.587 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:42:16.587 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":261,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:41:29.716652+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:40:20.346927+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:38:14.929578+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:40:01.279839+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:42:16.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:42:16 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1407754179' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:42:17.507 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5331 8 365281 365273 0.535261 0 - 0.0299513 2026-02-23T14:42:18.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:42:17 trial068 ceph-mon[38805]: pgmap v3235: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:42:18.507 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5332 8 365281 365273 0.535161 0 - 0.0299513 2026-02-23T14:42:19.507 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5333 8 365281 365273 0.53506 0 - 0.0299513 2026-02-23T14:42:19.624 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:42:19.741 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:42:19.843 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:42:19.841+0000 7fcaffea0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:42:19.843 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:42:19.841+0000 7fcaffea0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:42:19.951 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:42:19.952 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":261,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:41:29.716652+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:40:20.346927+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:38:14.929578+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:40:01.279839+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:42:19.952 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:42:19 trial068 ceph-mon[38805]: pgmap v3236: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:42:20.507 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5334 8 365281 365273 0.53496 0 - 0.0299513 2026-02-23T14:42:21.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:42:20 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3041740959' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:42:21.507 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5335 8 365281 365273 0.53486 0 - 0.0299513 2026-02-23T14:42:22.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:42:21 trial068 ceph-mon[38805]: pgmap v3237: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:42:22.507 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5336 8 365281 365273 0.534759 0 - 0.0299513 2026-02-23T14:42:22.988 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:42:23.107 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:42:23.211 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:42:23.209+0000 7fad07341640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:42:23.212 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:42:23.210+0000 7fad07341640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:42:23.322 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:42:23.322 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":261,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:41:29.716652+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:40:20.346927+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:38:14.929578+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:40:01.279839+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:42:23.507 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5337 8 365281 365273 0.534659 0 - 0.0299513 2026-02-23T14:42:24.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:42:23 trial068 ceph-mon[38805]: pgmap v3238: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:42:24.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:42:23 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/902373462' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:42:24.094 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:42:23 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:42:23] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:42:24.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:42:23 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:42:23.974Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:42:24.507 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5338 8 365281 365273 0.534559 0 - 0.0299513 2026-02-23T14:42:25.508 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5339 8 365281 365273 0.534459 0 - 0.0299513 2026-02-23T14:42:26.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:42:25 trial068 ceph-mon[38805]: pgmap v3239: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:42:26.359 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:42:26.478 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:42:26.508 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T14:42:26.507433+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T14:42:26.508 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T14:42:26.508 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5340 8 365281 365273 0.534359 0 - 0.0299513 2026-02-23T14:42:26.578 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:42:26.576+0000 7ff0cd840640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:42:26.578 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:42:26.576+0000 7ff0cd840640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:42:26.688 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:42:26.688 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":261,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:41:29.716652+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:40:20.346927+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:38:14.929578+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:40:01.279839+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:42:27.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:42:26 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T14:42:27.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:42:26 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T14:42:27.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:42:26 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2814687387' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:42:27.508 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5341 8 365281 365273 0.534259 0 - 0.0299513 2026-02-23T14:42:28.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:42:27 trial068 ceph-mon[38805]: pgmap v3240: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:42:28.508 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5342 8 365281 365273 0.534159 0 - 0.0299513 2026-02-23T14:42:29.508 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5343 8 365281 365273 0.534059 0 - 0.0299513 2026-02-23T14:42:29.725 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:42:29.845 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:42:29.951 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:42:29.949+0000 7fddd92d2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:42:29.951 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:42:29.949+0000 7fddd92d2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:42:30.060 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:42:30.060 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":261,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:41:29.716652+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:40:20.346927+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:38:14.929578+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:40:01.279839+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:42:30.060 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:42:29 trial068 ceph-mon[38805]: pgmap v3241: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:42:30.508 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5344 8 365281 365273 0.533959 0 - 0.0299513 2026-02-23T14:42:31.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:42:30 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1418091649' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:42:31.508 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5345 8 365281 365273 0.533859 0 - 0.0299513 2026-02-23T14:42:32.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:42:31 trial068 ceph-mon[38805]: pgmap v3242: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:42:32.508 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5346 8 365281 365273 0.533759 0 - 0.0299513 2026-02-23T14:42:33.099 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:42:33.217 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:42:33.319 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:42:33.317+0000 7fc798e21640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:42:33.320 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:42:33.318+0000 7fc798e21640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:42:33.430 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:42:33.430 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":261,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:41:29.716652+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:40:20.346927+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:38:14.929578+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:40:01.279839+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:42:33.508 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5347 8 365281 365273 0.533659 0 - 0.0299513 2026-02-23T14:42:34.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:42:33 trial068 ceph-mon[38805]: pgmap v3243: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:42:34.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:42:33 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3651949594' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:42:34.094 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:42:33 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:42:33] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:42:34.265 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:42:33 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:42:33.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:42:34.508 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5348 8 365281 365273 0.53356 0 - 0.0299513 2026-02-23T14:42:35.509 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5349 8 365281 365273 0.53346 0 - 0.0299513 2026-02-23T14:42:36.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:42:35 trial068 ceph-mon[38805]: pgmap v3244: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:42:36.468 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:42:36.509 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5350 8 365281 365273 0.53336 0 - 0.0299513 2026-02-23T14:42:36.587 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:42:36.687 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:42:36.685+0000 7f3e86e00640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:42:36.687 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:42:36.685+0000 7f3e86e00640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:42:36.800 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:42:36.800 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":261,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:41:29.716652+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:40:20.346927+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:38:14.929578+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:40:01.279839+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:42:37.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:42:36 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1286265086' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:42:37.509 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5351 8 365281 365273 0.53326 0 - 0.0299513 2026-02-23T14:42:38.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:42:37 trial068 ceph-mon[38805]: pgmap v3245: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:42:38.509 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5352 8 365281 365273 0.533161 0 - 0.0299513 2026-02-23T14:42:39.509 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5353 8 365281 365273 0.533061 0 - 0.0299513 2026-02-23T14:42:39.860 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:42:39.884 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:42:39 trial068 ceph-mon[38805]: pgmap v3246: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:42:39.978 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:42:40.077 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:42:40.075+0000 7ffbab0a5640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:42:40.078 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:42:40.076+0000 7ffbab0a5640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:42:40.187 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:42:40.188 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":261,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:41:29.716652+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:40:20.346927+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:38:14.929578+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:40:01.279839+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:42:40.509 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5354 8 365281 365273 0.532962 0 - 0.0299513 2026-02-23T14:42:41.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:42:40 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1071284121' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:42:41.509 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5355 8 365281 365273 0.532862 0 - 0.0299513 2026-02-23T14:42:42.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:42:41 trial068 ceph-mon[38805]: pgmap v3247: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:42:42.509 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5356 8 365281 365273 0.532763 0 - 0.0299513 2026-02-23T14:42:43.226 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:42:43.346 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:42:43.444 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:42:43.442+0000 7fce79516640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:42:43.444 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:42:43.442+0000 7fce79516640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:42:43.509 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5357 8 365281 365273 0.532663 0 - 0.0299513 2026-02-23T14:42:43.561 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:42:43.562 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":261,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:41:29.716652+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:40:20.346927+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:38:14.929578+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:40:01.279839+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:42:44.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:42:43 trial068 ceph-mon[38805]: pgmap v3248: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:42:44.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:42:43 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2413319220' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:42:44.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:42:43 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:42:43] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:42:44.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:42:43 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:42:43.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:42:44.509 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5358 8 365281 365273 0.532564 0 - 0.0299513 2026-02-23T14:42:45.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:42:44 trial068 ceph-mon[38805]: 5.e scrub starts 2026-02-23T14:42:45.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:42:44 trial068 ceph-mon[38805]: 5.e scrub ok 2026-02-23T14:42:45.509 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5359 8 365281 365273 0.532464 0 - 0.0299513 2026-02-23T14:42:46.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:42:45 trial068 ceph-mon[38805]: pgmap v3249: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:42:46.509 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T14:42:46.509153+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T14:42:46.509 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T14:42:46.509 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5360 8 365281 365273 0.532365 0 - 0.0299513 2026-02-23T14:42:46.602 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:42:46.720 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:42:46.818 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:42:46.816+0000 7f7857c5b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:42:46.818 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:42:46.817+0000 7f7857c5b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:42:46.929 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:42:46.929 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":261,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:41:29.716652+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:40:20.346927+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:38:14.929578+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:40:01.279839+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:42:47.509 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5361 8 365281 365273 0.532266 0 - 0.0299513 2026-02-23T14:42:48.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:42:47 trial068 ceph-mon[38805]: pgmap v3250: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:42:48.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:42:47 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/459818236' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:42:48.509 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5362 8 365281 365273 0.532166 0 - 0.0299513 2026-02-23T14:42:49.510 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5363 8 365281 365273 0.532067 0 - 0.0299513 2026-02-23T14:42:49.967 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:42:50.085 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:42:50.106 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:42:49 trial068 ceph-mon[38805]: pgmap v3251: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:42:50.185 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:42:50.183+0000 7f22b09a3640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:42:50.185 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:42:50.183+0000 7f22b09a3640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:42:50.294 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:42:50.294 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":261,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:41:29.716652+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:40:20.346927+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:38:14.929578+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:40:01.279839+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:42:50.510 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5364 8 365281 365273 0.531968 0 - 0.0299513 2026-02-23T14:42:51.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:42:50 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/730894632' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:42:51.510 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5365 8 365281 365273 0.531869 0 - 0.0299513 2026-02-23T14:42:52.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:42:51 trial068 ceph-mon[38805]: pgmap v3252: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:42:52.510 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5366 8 365281 365273 0.53177 0 - 0.0299513 2026-02-23T14:42:53.330 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:42:53.450 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:42:53.510 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5367 8 365281 365273 0.531671 0 - 0.0299513 2026-02-23T14:42:53.551 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:42:53.549+0000 7fb516cfb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:42:53.551 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:42:53.549+0000 7fb516cfb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:42:53.660 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:42:53.660 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":261,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:41:29.716652+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:40:20.346927+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:38:14.929578+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:40:01.279839+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:42:53.974 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:42:53 trial068 ceph-mon[38805]: pgmap v3253: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:42:53.974 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:42:53 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2587125461' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:42:54.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:42:53 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:42:53] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:42:54.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:42:53 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:42:53.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:42:54.510 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5368 8 365281 365273 0.531572 0 - 0.0299513 2026-02-23T14:42:55.510 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5369 8 365281 365273 0.531473 0 - 0.0299513 2026-02-23T14:42:56.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:42:55 trial068 ceph-mon[38805]: pgmap v3254: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:42:56.510 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5370 8 365281 365273 0.531374 0 - 0.0299513 2026-02-23T14:42:56.698 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:42:56.817 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:42:56.921 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:42:56.919+0000 7fc3d6722640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:42:56.922 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:42:56.920+0000 7fc3d6722640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:42:57.030 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:42:57.030 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":261,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:41:29.716652+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:40:20.346927+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:38:14.929578+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:40:01.279839+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:42:57.510 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5371 8 365281 365273 0.531275 0 - 0.0299513 2026-02-23T14:42:58.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:42:57 trial068 ceph-mon[38805]: pgmap v3255: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:42:58.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:42:57 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/4128239025' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:42:58.510 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5372 8 365281 365273 0.531176 0 - 0.0299513 2026-02-23T14:42:59.510 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5373 8 365281 365273 0.531077 0 - 0.0299513 2026-02-23T14:43:00.068 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:43:00.186 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:43:00.208 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:42:59 trial068 ceph-mon[38805]: pgmap v3256: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:43:00.285 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:43:00.283+0000 7f697aa35640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:43:00.285 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:43:00.283+0000 7f697aa35640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:43:00.394 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:43:00.394 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":261,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:41:29.716652+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:40:20.346927+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:38:14.929578+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:40:01.279839+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:43:00.510 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5374 8 365281 365273 0.530978 0 - 0.0299513 2026-02-23T14:43:01.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:43:00 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/4210464989' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:43:01.510 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5375 8 365281 365273 0.530879 0 - 0.0299513 2026-02-23T14:43:02.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:43:01 trial068 ceph-mon[38805]: pgmap v3257: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:43:02.510 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5376 8 365281 365273 0.530781 0 - 0.0299513 2026-02-23T14:43:03.433 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:43:03.511 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5377 8 365281 365273 0.530682 0 - 0.0299513 2026-02-23T14:43:03.551 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:43:03.649 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:43:03.647+0000 7fe27c9c8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:43:03.649 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:43:03.647+0000 7fe27c9c8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:43:03.759 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:43:03.759 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":261,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:41:29.716652+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:40:20.346927+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:38:14.929578+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:40:01.279839+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:43:04.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:43:03 trial068 ceph-mon[38805]: pgmap v3258: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:43:04.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:43:03 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3135849483' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:43:04.094 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:43:03 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:43:03] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:43:04.266 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:43:03 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:43:03.974Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:43:04.511 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5378 8 365281 365273 0.530583 0 - 0.0299513 2026-02-23T14:43:05.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:43:04 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T14:43:05.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:43:04 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T14:43:05.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:43:04 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2026-02-23T14:43:05.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:43:04 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-23T14:43:05.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:43:04 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2026-02-23T14:43:05.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:43:04 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T14:43:05.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:43:04 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2026-02-23T14:43:05.511 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5379 8 365281 365273 0.530485 0 - 0.0299513 2026-02-23T14:43:06.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:43:05 trial068 ceph-mon[38805]: pgmap v3259: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:43:06.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:43:05 trial068 ceph-mon[38805]: pgmap v3260: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:43:06.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:43:05 trial068 ceph-mon[38805]: pgmap v3261: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:43:06.511 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T14:43:06.510595+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T14:43:06.511 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T14:43:06.511 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5380 8 365281 365273 0.530386 0 - 0.0299513 2026-02-23T14:43:06.796 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:43:06.916 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:43:07.017 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:43:07.014+0000 7fed482b2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:43:07.017 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:43:07.015+0000 7fed482b2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:43:07.128 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:43:07.128 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":261,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:41:29.716652+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:40:20.346927+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:38:14.929578+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:40:01.279839+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:43:07.511 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5381 8 365281 365273 0.530287 0 - 0.0299513 2026-02-23T14:43:08.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:43:07 trial068 ceph-mon[38805]: pgmap v3262: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:43:08.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:43:07 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3699843850' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:43:08.511 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5382 8 365281 365273 0.530189 0 - 0.0299513 2026-02-23T14:43:09.511 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5383 8 365281 365273 0.53009 0 - 0.0299513 2026-02-23T14:43:10.146 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:43:09 trial068 ceph-mon[38805]: pgmap v3263: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:43:10.166 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:43:10.282 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:43:10.381 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:43:10.379+0000 7f0b78087640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:43:10.381 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:43:10.379+0000 7f0b78087640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:43:10.488 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:43:10.488 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":261,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:41:29.716652+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:40:20.346927+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:38:14.929578+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:40:01.279839+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:43:10.511 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5384 8 365281 365273 0.529992 0 - 0.0299513 2026-02-23T14:43:11.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:43:10 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3264437214' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:43:11.511 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5385 8 365281 365273 0.529893 0 - 0.0299513 2026-02-23T14:43:12.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:43:11 trial068 ceph-mon[38805]: pgmap v3264: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:43:12.511 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5386 8 365281 365273 0.529795 0 - 0.0299513 2026-02-23T14:43:13.511 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5387 8 365281 365273 0.529697 0 - 0.0299513 2026-02-23T14:43:13.525 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:43:13.644 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:43:13.741 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:43:13.739+0000 7fd108319640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:43:13.742 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:43:13.739+0000 7fd108319640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:43:13.850 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:43:13.850 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":261,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:41:29.716652+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:40:20.346927+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:38:14.929578+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:40:01.279839+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:43:14.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:43:13 trial068 ceph-mon[38805]: pgmap v3265: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:43:14.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:43:13 trial068 ceph-mon[38805]: 5.5 scrub starts 2026-02-23T14:43:14.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:43:13 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1540180357' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:43:14.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:43:13 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:43:13] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:43:14.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:43:13 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:43:13.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:43:14.511 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5388 8 365281 365273 0.529598 0 - 0.0299513 2026-02-23T14:43:15.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:43:14 trial068 ceph-mon[38805]: 5.5 scrub ok 2026-02-23T14:43:15.512 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5389 8 365281 365273 0.5295 0 - 0.0299513 2026-02-23T14:43:16.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:43:15 trial068 ceph-mon[38805]: pgmap v3266: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:43:16.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:43:15 trial068 ceph-mon[38805]: 5.8 scrub starts 2026-02-23T14:43:16.512 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5390 8 365281 365273 0.529402 0 - 0.0299513 2026-02-23T14:43:16.909 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:43:17.026 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:43:17.126 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:43:17.124+0000 7fd01ed85640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:43:17.127 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:43:16 trial068 ceph-mon[38805]: 5.8 scrub ok 2026-02-23T14:43:17.127 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:43:17.125+0000 7fd01ed85640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:43:17.236 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:43:17.237 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":261,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:41:29.716652+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:40:20.346927+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:38:14.929578+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:40:01.279839+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:43:17.512 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5391 8 365281 365273 0.529304 0 - 0.0299513 2026-02-23T14:43:18.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:43:17 trial068 ceph-mon[38805]: pgmap v3267: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:43:18.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:43:17 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3938854091' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:43:18.512 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5392 8 365281 365273 0.529206 0 - 0.0299513 2026-02-23T14:43:19.512 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5393 8 365281 365273 0.529107 0 - 0.0299513 2026-02-23T14:43:20.274 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:43:20.300 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:43:19 trial068 ceph-mon[38805]: pgmap v3268: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:43:20.393 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:43:20.494 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:43:20.491+0000 7ff0a9f9c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:43:20.494 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:43:20.492+0000 7ff0a9f9c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:43:20.512 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5394 8 365281 365273 0.529009 0 - 0.0299513 2026-02-23T14:43:20.605 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:43:20.605 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":261,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:41:29.716652+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:40:20.346927+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:38:14.929578+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:40:01.279839+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:43:21.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:43:20 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/96663384' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:43:21.512 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5395 8 365281 365273 0.528911 0 - 0.0299513 2026-02-23T14:43:22.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:43:21 trial068 ceph-mon[38805]: pgmap v3269: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:43:22.512 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5396 8 365281 365273 0.528813 0 - 0.0299513 2026-02-23T14:43:23.512 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5397 8 365281 365273 0.528715 0 - 0.0299513 2026-02-23T14:43:23.642 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:43:23.759 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:43:23.860 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:43:23.858+0000 7fe9705d0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:43:23.860 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:43:23.858+0000 7fe9705d0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:43:23.969 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:43:23.969 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":261,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:41:29.716652+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:40:20.346927+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:38:14.929578+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:40:01.279839+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:43:23.970 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:43:23 trial068 ceph-mon[38805]: pgmap v3270: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:43:23.970 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:43:23 trial068 ceph-mon[38805]: 5.a scrub starts 2026-02-23T14:43:24.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:43:23 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:43:23] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:43:24.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:43:23 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:43:23.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:43:24.512 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5398 8 365281 365273 0.528617 0 - 0.0299513 2026-02-23T14:43:25.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:43:24 trial068 ceph-mon[38805]: 5.a scrub ok 2026-02-23T14:43:25.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:43:24 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2647716250' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:43:25.512 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5399 8 365281 365273 0.528519 0 - 0.0299513 2026-02-23T14:43:26.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:43:25 trial068 ceph-mon[38805]: pgmap v3271: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:43:26.512 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T14:43:26.512062+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T14:43:26.513 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T14:43:26.513 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5400 8 365281 365273 0.528422 0 - 0.0299513 2026-02-23T14:43:27.006 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:43:27.123 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:43:27.220 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:43:27.218+0000 7f6288f1e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:43:27.221 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:43:27.219+0000 7f6288f1e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:43:27.330 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:43:27.330 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":261,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:41:29.716652+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:40:20.346927+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:38:14.929578+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:40:01.279839+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:43:27.512 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5401 8 365281 365273 0.528324 0 - 0.0299513 2026-02-23T14:43:28.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:43:27 trial068 ceph-mon[38805]: pgmap v3272: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:43:28.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:43:27 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/539020759' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:43:28.512 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5402 8 365281 365273 0.528226 0 - 0.0299513 2026-02-23T14:43:29.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:43:28 trial068 ceph-mon[38805]: 5.7 scrub starts 2026-02-23T14:43:29.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:43:28 trial068 ceph-mon[38805]: 5.7 scrub ok 2026-02-23T14:43:29.513 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5403 8 365281 365273 0.528128 0 - 0.0299513 2026-02-23T14:43:30.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:43:29 trial068 ceph-mon[38805]: pgmap v3273: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:43:30.369 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:43:30.486 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:43:30.513 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5404 8 365281 365273 0.52803 0 - 0.0299513 2026-02-23T14:43:30.589 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:43:30.587+0000 7fb6ab7d3640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:43:30.589 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:43:30.587+0000 7fb6ab7d3640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:43:30.697 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:43:30.697 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":261,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:41:29.716652+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:40:20.346927+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:38:14.929578+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:40:01.279839+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:43:31.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:43:30 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/4234748113' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:43:31.512 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5405 8 365281 365273 0.527933 0 - 0.0299513 2026-02-23T14:43:32.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:43:31 trial068 ceph-mon[38805]: pgmap v3274: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:43:32.513 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5406 8 365281 365273 0.527835 0 - 0.0299513 2026-02-23T14:43:33.513 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5407 8 365281 365273 0.527737 0 - 0.0299513 2026-02-23T14:43:33.734 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:43:33.853 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:43:33.956 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:43:33.954+0000 7faedb7d2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:43:33.956 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:43:33.954+0000 7faedb7d2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:43:34.067 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:43:34.067 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":261,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:41:29.716652+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:40:20.346927+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:38:14.929578+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:40:01.279839+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:43:34.068 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:43:33 trial068 ceph-mon[38805]: pgmap v3275: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:43:34.068 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:43:33 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:43:33] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:43:34.266 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:43:33 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:43:33.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:43:34.513 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5408 8 365281 365273 0.52764 0 - 0.0299513 2026-02-23T14:43:35.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:43:34 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2487210792' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:43:35.513 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5409 8 365281 365273 0.527542 0 - 0.0299513 2026-02-23T14:43:36.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:43:35 trial068 ceph-mon[38805]: pgmap v3276: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:43:36.513 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5410 8 365281 365273 0.527445 0 - 0.0299513 2026-02-23T14:43:37.106 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:43:37.224 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:43:37.327 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:43:37.325+0000 7f7a39e35640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:43:37.327 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:43:37.325+0000 7f7a39e35640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:43:37.435 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:43:37.436 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":261,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:41:29.716652+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:40:20.346927+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:38:14.929578+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:40:01.279839+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:43:37.513 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5411 8 365281 365273 0.527347 0 - 0.0299513 2026-02-23T14:43:38.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:43:37 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T14:43:38.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:43:37 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T14:43:38.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:43:37 trial068 ceph-mon[38805]: pgmap v3277: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:43:38.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:43:37 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T14:43:38.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:43:37 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T14:43:38.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:43:37 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3104261852' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:43:38.513 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5412 8 365281 365273 0.52725 0 - 0.0299513 2026-02-23T14:43:39.513 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5413 8 365281 365273 0.527152 0 - 0.0299513 2026-02-23T14:43:40.146 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:43:39 trial068 ceph-mon[38805]: pgmap v3278: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:43:40.473 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:43:40.513 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5414 8 365281 365273 0.527055 0 - 0.0299513 2026-02-23T14:43:40.593 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:43:40.692 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:43:40.690+0000 7fd8ca0c3640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:43:40.693 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:43:40.690+0000 7fd8ca0c3640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:43:40.803 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:43:40.804 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":261,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:41:29.716652+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:40:20.346927+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:38:14.929578+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:40:01.279839+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:43:41.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:43:40 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2064392158' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:43:41.513 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5415 8 365281 365273 0.526958 0 - 0.0299513 2026-02-23T14:43:42.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:43:41 trial068 ceph-mon[38805]: pgmap v3279: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:43:42.514 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5416 8 365281 365273 0.52686 0 - 0.0299513 2026-02-23T14:43:43.514 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5417 8 365281 365273 0.526763 0 - 0.0299513 2026-02-23T14:43:43.839 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:43:43.959 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:43:44.058 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:43:44.056+0000 7fb5711ae640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:43:44.058 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:43:44.057+0000 7fb5711ae640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:43:44.167 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:43:44.167 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":261,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:41:29.716652+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:40:20.346927+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:38:14.929578+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:40:01.279839+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:43:44.167 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:43:43 trial068 ceph-mon[38805]: pgmap v3280: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:43:44.167 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:43:43 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:43:43] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:43:44.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:43:43 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:43:43.972Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:43:44.514 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5418 8 365281 365273 0.526666 0 - 0.0299513 2026-02-23T14:43:45.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:43:44 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/113794872' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:43:45.514 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5419 8 365281 365273 0.526569 0 - 0.0299513 2026-02-23T14:43:46.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:43:45 trial068 ceph-mon[38805]: pgmap v3281: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:43:46.514 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T14:43:46.513588+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T14:43:46.514 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T14:43:46.514 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5420 8 365281 365273 0.526472 0 - 0.0299513 2026-02-23T14:43:47.205 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:43:47.230 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:43:46 trial068 ceph-mon[38805]: pgmap v3282: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:43:47.323 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:43:47.425 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:43:47.423+0000 7fbb83088640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:43:47.425 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:43:47.423+0000 7fbb83088640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:43:47.514 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5421 8 365281 365273 0.526375 0 - 0.0299513 2026-02-23T14:43:47.534 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:43:47.535 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":261,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:41:29.716652+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:40:20.346927+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:38:14.929578+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:40:01.279839+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:43:48.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:43:47 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2809014752' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:43:48.514 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5422 8 365281 365273 0.526277 0 - 0.0299513 2026-02-23T14:43:49.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:43:48 trial068 ceph-mon[38805]: pgmap v3283: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:43:49.514 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5423 8 365281 365273 0.52618 0 - 0.0299513 2026-02-23T14:43:50.514 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5424 8 365281 365273 0.526083 0 - 0.0299513 2026-02-23T14:43:50.575 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:43:50.695 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:43:50.794 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:43:50.792+0000 7f442c132640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:43:50.794 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:43:50.792+0000 7f442c132640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:43:50.904 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:43:50.904 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":261,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:41:29.716652+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:40:20.346927+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:38:14.929578+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:40:01.279839+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:43:51.514 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5425 8 365281 365273 0.525986 0 - 0.0299513 2026-02-23T14:43:52.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:43:51 trial068 ceph-mon[38805]: pgmap v3284: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:43:52.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:43:51 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2009526598' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:43:52.515 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5426 8 365281 365273 0.52589 0 - 0.0299513 2026-02-23T14:43:53.515 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5427 8 365281 365273 0.525793 0 - 0.0299513 2026-02-23T14:43:53.941 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:43:54.061 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:43:54.083 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:43:53 trial068 ceph-mon[38805]: pgmap v3285: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:43:54.083 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:43:53 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:43:53] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:43:54.159 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:43:54.157+0000 7fa853fc0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:43:54.159 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:43:54.157+0000 7fa853fc0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:43:54.269 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:43:54.269 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":261,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:41:29.716652+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:40:20.346927+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:38:14.929578+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:40:01.279839+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:43:54.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:43:53 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:43:53.972Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:43:54.515 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5428 8 365281 365273 0.525696 0 - 0.0299513 2026-02-23T14:43:55.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:43:54 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/596773153' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:43:55.515 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5429 8 365281 365273 0.525599 0 - 0.0299513 2026-02-23T14:43:56.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:43:55 trial068 ceph-mon[38805]: pgmap v3286: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:43:56.515 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5430 8 365281 365273 0.525502 0 - 0.0299513 2026-02-23T14:43:57.324 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:43:57.441 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:43:57.515 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5431 8 365281 365273 0.525405 0 - 0.0299513 2026-02-23T14:43:57.544 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:43:57.542+0000 7fd70c55d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:43:57.544 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:43:57.543+0000 7fd70c55d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:43:57.654 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:43:57.655 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":261,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:41:29.716652+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:40:20.346927+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:38:14.929578+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:40:01.279839+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:43:58.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:43:57 trial068 ceph-mon[38805]: pgmap v3287: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:43:58.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:43:57 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3585664535' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:43:58.515 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5432 8 365281 365273 0.525309 0 - 0.0299513 2026-02-23T14:43:59.515 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5433 8 365281 365273 0.525212 0 - 0.0299513 2026-02-23T14:44:00.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:43:59 trial068 ceph-mon[38805]: pgmap v3288: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:44:00.515 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5434 8 365281 365273 0.525115 0 - 0.0299513 2026-02-23T14:44:00.691 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:44:00.809 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:44:00.906 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:44:00.904+0000 7fdfa0bbb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:44:00.907 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:44:00.905+0000 7fdfa0bbb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:44:01.014 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:44:01.015 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":261,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:41:29.716652+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:40:20.346927+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:38:14.929578+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:40:01.279839+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:44:01.515 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5435 8 365281 365273 0.525019 0 - 0.0299513 2026-02-23T14:44:02.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:44:01 trial068 ceph-mon[38805]: pgmap v3289: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:44:02.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:44:01 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/233524964' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:44:02.516 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5436 8 365281 365273 0.524922 0 - 0.0299513 2026-02-23T14:44:03.515 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5437 8 365281 365273 0.524826 0 - 0.0299513 2026-02-23T14:44:04.054 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:44:04.079 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:44:03 trial068 ceph-mon[38805]: pgmap v3290: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:44:04.080 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:44:03 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:44:03] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:44:04.172 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:44:04.266 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:44:03 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:44:03.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:44:04.272 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:44:04.270+0000 7f8c21f53640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:44:04.272 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:44:04.270+0000 7f8c21f53640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:44:04.381 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:44:04.381 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":261,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:41:29.716652+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:40:20.346927+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:38:14.929578+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:40:01.279839+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:44:04.516 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5438 8 365281 365273 0.524729 0 - 0.0299513 2026-02-23T14:44:05.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:44:04 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1161831004' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:44:05.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:44:04 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2026-02-23T14:44:05.516 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5439 8 365281 365273 0.524633 0 - 0.0299513 2026-02-23T14:44:06.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:44:05 trial068 ceph-mon[38805]: pgmap v3291: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:44:06.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:44:05 trial068 ceph-mon[38805]: osdmap e262: 4 total, 3 up, 4 in 2026-02-23T14:44:06.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:44:05 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-23T14:44:06.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:44:05 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2026-02-23T14:44:06.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:44:05 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T14:44:06.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:44:05 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2026-02-23T14:44:06.516 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T14:44:06.515486+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T14:44:06.516 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T14:44:06.516 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5440 8 365281 365273 0.524536 0 - 0.0299513 2026-02-23T14:44:07.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:44:06 trial068 ceph-mon[38805]: pgmap v3293: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:44:07.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:44:06 trial068 ceph-mon[38805]: pgmap v3294: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:44:07.419 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:44:07.516 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5441 8 365281 365273 0.52444 0 - 0.0299513 2026-02-23T14:44:07.536 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:44:07.642 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:44:07.640+0000 7f4b6251e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:44:07.642 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:44:07.640+0000 7f4b6251e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:44:07.751 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:44:07.752 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":262,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:44:04.796409+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:40:20.346927+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:43:36.356556+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:40:01.279839+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:44:08.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:44:07 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3341287932' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:44:08.516 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5442 8 365281 365273 0.524343 0 - 0.0299513 2026-02-23T14:44:09.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:44:08 trial068 ceph-mon[38805]: pgmap v3295: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:44:09.516 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5443 8 365281 365273 0.524247 0 - 0.0299513 2026-02-23T14:44:10.516 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5444 8 365281 365273 0.524151 0 - 0.0299513 2026-02-23T14:44:10.789 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:44:10.911 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:44:11.014 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:44:11.012+0000 7f6807367640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:44:11.015 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:44:10 trial068 ceph-mon[38805]: pgmap v3296: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:44:11.015 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:44:10 trial068 ceph-mon[38805]: osdmap e263: 4 total, 3 up, 4 in 2026-02-23T14:44:11.015 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:44:11.013+0000 7f6807367640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:44:11.126 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:44:11.126 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":263,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:44:09.737746+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:40:20.346927+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:43:36.356556+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:40:01.279839+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:44:11.516 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5445 8 365281 365273 0.524054 0 - 0.0299513 2026-02-23T14:44:12.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:44:11 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1587367057' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:44:12.516 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5446 8 365281 365273 0.523958 0 - 0.0299513 2026-02-23T14:44:13.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:44:12 trial068 ceph-mon[38805]: pgmap v3298: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:44:13.516 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5447 8 365281 365273 0.523862 0 - 0.0299513 2026-02-23T14:44:14.164 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:44:14.284 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:44:14.306 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:44:13 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:44:13] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:44:14.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:44:13 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:44:13.974Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:44:14.393 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:44:14.390+0000 7f3a2dd56640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:44:14.393 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:44:14.390+0000 7f3a2dd56640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:44:14.502 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:44:14.502 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":263,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:44:09.737746+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:40:20.346927+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:43:36.356556+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:40:01.279839+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:44:14.516 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5448 8 365281 365273 0.523766 0 - 0.0299513 2026-02-23T14:44:15.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:44:14 trial068 ceph-mon[38805]: pgmap v3299: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:44:15.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:44:14 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2013157740' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:44:15.516 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5449 8 365281 365273 0.52367 0 - 0.0299513 2026-02-23T14:44:16.516 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5450 8 365281 365273 0.523574 0 - 0.0299513 2026-02-23T14:44:17.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:44:16 trial068 ceph-mon[38805]: pgmap v3300: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:44:17.516 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5451 8 365281 365273 0.523478 0 - 0.0299513 2026-02-23T14:44:17.540 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:44:17.660 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:44:17.759 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:44:17.757+0000 7fc065e27640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:44:17.759 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:44:17.757+0000 7fc065e27640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:44:17.867 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:44:17.868 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":263,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:44:09.737746+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:40:20.346927+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:43:36.356556+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:40:01.279839+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:44:18.517 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5452 8 365281 365273 0.523382 0 - 0.0299513 2026-02-23T14:44:19.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:44:18 trial068 ceph-mon[38805]: pgmap v3301: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:44:19.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:44:18 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1305181851' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:44:19.517 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5453 8 365281 365273 0.523286 0 - 0.0299513 2026-02-23T14:44:20.517 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5454 8 365281 365273 0.52319 0 - 0.0299513 2026-02-23T14:44:20.920 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:44:21.039 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:44:21.061 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:44:20 trial068 ceph-mon[38805]: pgmap v3302: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:44:21.061 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:44:20 trial068 ceph-mon[38805]: 5.e scrub starts 2026-02-23T14:44:21.061 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:44:20 trial068 ceph-mon[38805]: 5.e scrub ok 2026-02-23T14:44:21.142 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:44:21.140+0000 7feddb40e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:44:21.142 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:44:21.140+0000 7feddb40e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:44:21.252 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:44:21.252 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":263,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:44:09.737746+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:40:20.346927+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:43:36.356556+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:40:01.279839+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:44:21.517 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5455 8 365281 365273 0.523094 0 - 0.0299513 2026-02-23T14:44:22.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:44:21 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2274834526' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:44:22.517 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5456 8 365281 365273 0.522998 0 - 0.0299513 2026-02-23T14:44:23.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:44:22 trial068 ceph-mon[38805]: pgmap v3303: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:44:23.517 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5457 8 365281 365273 0.522902 0 - 0.0299513 2026-02-23T14:44:24.290 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:44:24.314 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:44:23 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:44:23] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:44:24.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:44:23 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:44:23.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:44:24.406 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:44:24.505 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:44:24.503+0000 7fef82dac640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:44:24.505 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:44:24.503+0000 7fef82dac640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:44:24.517 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5458 8 365281 365273 0.522806 0 - 0.0299513 2026-02-23T14:44:24.616 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:44:24.616 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":263,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:44:09.737746+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:40:20.346927+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:43:36.356556+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:40:01.279839+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:44:25.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:44:24 trial068 ceph-mon[38805]: pgmap v3304: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:44:25.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:44:24 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2775846424' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:44:25.517 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5459 8 365281 365273 0.52271 0 - 0.0299513 2026-02-23T14:44:26.517 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T14:44:26.516995+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T14:44:26.518 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T14:44:26.518 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5460 8 365281 365273 0.522615 0 - 0.0299513 2026-02-23T14:44:27.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:44:26 trial068 ceph-mon[38805]: pgmap v3305: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:44:27.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:44:26 trial068 ceph-mon[38805]: 5.5 scrub starts 2026-02-23T14:44:27.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:44:26 trial068 ceph-mon[38805]: 5.5 scrub ok 2026-02-23T14:44:27.517 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5461 8 365281 365273 0.522519 0 - 0.0299513 2026-02-23T14:44:27.654 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:44:27.775 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:44:27.876 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:44:27.874+0000 7f9974bdc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:44:27.876 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:44:27.874+0000 7f9974bdc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:44:27.986 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:44:27.986 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":263,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:44:09.737746+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:40:20.346927+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:43:36.356556+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:40:01.279839+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:44:27.986 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:44:27 trial068 ceph-mon[38805]: 5.8 scrub starts 2026-02-23T14:44:27.986 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:44:27 trial068 ceph-mon[38805]: 5.8 scrub ok 2026-02-23T14:44:28.517 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5462 8 365281 365273 0.522423 0 - 0.0299513 2026-02-23T14:44:29.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:44:28 trial068 ceph-mon[38805]: pgmap v3306: 17 pgs: 1 active+clean+scrubbing, 1 active+recovery_unfound+undersized+degraded, 4 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:44:29.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:44:28 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/822416203' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:44:29.517 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5463 8 365281 365273 0.522328 0 - 0.0299513 2026-02-23T14:44:30.518 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5464 8 365281 365273 0.522232 0 - 0.0299513 2026-02-23T14:44:31.024 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:44:31.049 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:44:30 trial068 ceph-mon[38805]: pgmap v3307: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 1 active+clean+scrubbing, 4 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:44:31.142 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:44:31.244 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:44:31.242+0000 7fe5b4032640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:44:31.245 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:44:31.243+0000 7fe5b4032640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:44:31.353 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:44:31.353 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":263,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:44:09.737746+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:40:20.346927+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:43:36.356556+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:40:01.279839+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:44:31.518 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5465 8 365281 365273 0.522137 0 - 0.0299513 2026-02-23T14:44:32.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:44:31 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2563761366' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:44:32.518 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5466 8 365281 365273 0.522041 0 - 0.0299513 2026-02-23T14:44:33.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:44:32 trial068 ceph-mon[38805]: pgmap v3308: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:44:33.518 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5467 8 365281 365273 0.521946 0 - 0.0299513 2026-02-23T14:44:34.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:44:33 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T14:44:34.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:44:33 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T14:44:34.094 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:44:33 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:44:33] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:44:34.266 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:44:33 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:44:33.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:44:34.389 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:44:34.505 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:44:34.518 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5468 8 365281 365273 0.52185 0 - 0.0299513 2026-02-23T14:44:34.607 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:44:34.605+0000 7f3a3f719640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:44:34.607 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:44:34.605+0000 7f3a3f719640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:44:34.717 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:44:34.717 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":263,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:44:09.737746+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:40:20.346927+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:43:36.356556+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:40:01.279839+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:44:35.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:44:34 trial068 ceph-mon[38805]: pgmap v3309: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:44:35.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:44:34 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3698114528' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:44:35.518 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5469 8 365281 365273 0.521755 0 - 0.0299513 2026-02-23T14:44:36.518 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5470 8 365281 365273 0.521659 0 - 0.0299513 2026-02-23T14:44:37.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:44:36 trial068 ceph-mon[38805]: pgmap v3310: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:44:37.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:44:36 trial068 ceph-mon[38805]: 5.7 scrub starts 2026-02-23T14:44:37.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:44:36 trial068 ceph-mon[38805]: 5.7 scrub ok 2026-02-23T14:44:37.518 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5471 8 365281 365273 0.521564 0 - 0.0299513 2026-02-23T14:44:37.755 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:44:37.875 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:44:37.979 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:44:37.977+0000 7fcd0477e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:44:37.979 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:44:37.977+0000 7fcd0477e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:44:38.089 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:44:38.089 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":263,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:44:09.737746+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:40:20.346927+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:43:36.356556+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:40:01.279839+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:44:38.518 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5472 8 365281 365273 0.521469 0 - 0.0299513 2026-02-23T14:44:39.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:44:38 trial068 ceph-mon[38805]: pgmap v3311: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:44:39.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:44:38 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/521783178' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:44:39.518 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5473 8 365281 365273 0.521373 0 - 0.0299513 2026-02-23T14:44:40.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:44:39 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T14:44:40.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:44:39 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T14:44:40.518 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5474 8 365281 365273 0.521278 0 - 0.0299513 2026-02-23T14:44:41.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:44:40 trial068 ceph-mon[38805]: pgmap v3312: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:44:41.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:44:40 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T14:44:41.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:44:40 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T14:44:41.127 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:44:41.245 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:44:41.345 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:44:41.343+0000 7f72dd589640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:44:41.345 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:44:41.343+0000 7f72dd589640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:44:41.457 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:44:41.458 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":263,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:44:09.737746+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:40:20.346927+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:43:36.356556+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:40:01.279839+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:44:41.518 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5475 8 365281 365273 0.521183 0 - 0.0299513 2026-02-23T14:44:42.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:44:41 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/229492688' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:44:42.519 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5476 8 365281 365273 0.521088 0 - 0.0299513 2026-02-23T14:44:43.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:44:42 trial068 ceph-mon[38805]: pgmap v3313: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:44:43.519 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5477 8 365281 365273 0.520993 0 - 0.0299513 2026-02-23T14:44:44.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:44:43 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:44:43] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:44:44.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:44:43 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:44:43.974Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:44:44.495 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:44:44.518 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5478 8 365281 365273 0.520897 0 - 0.0299513 2026-02-23T14:44:44.613 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:44:44.714 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:44:44.712+0000 7f49d17bc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:44:44.714 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:44:44.712+0000 7f49d17bc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:44:44.824 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:44:44.824 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":263,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:44:09.737746+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:40:20.346927+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:43:36.356556+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:40:01.279839+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:44:45.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:44:44 trial068 ceph-mon[38805]: pgmap v3314: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:44:45.519 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5479 8 365281 365273 0.520802 0 - 0.0299513 2026-02-23T14:44:46.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:44:45 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3992483026' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:44:46.519 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T14:44:46.518508+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T14:44:46.519 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T14:44:46.519 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5480 8 365281 365273 0.520707 0 - 0.0299513 2026-02-23T14:44:47.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:44:46 trial068 ceph-mon[38805]: pgmap v3315: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:44:47.519 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5481 8 365281 365273 0.520612 0 - 0.0299513 2026-02-23T14:44:47.862 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:44:47.982 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:44:48.081 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:44:48.079+0000 7efdc551b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:44:48.081 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:44:48.079+0000 7efdc551b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:44:48.189 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:44:48.189 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":263,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:44:09.737746+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:40:20.346927+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:43:36.356556+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:40:01.279839+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:44:48.519 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5482 8 365281 365273 0.520517 0 - 0.0299513 2026-02-23T14:44:49.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:44:48 trial068 ceph-mon[38805]: pgmap v3316: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:44:49.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:44:48 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2166950439' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:44:49.519 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5483 8 365281 365273 0.520422 0 - 0.0299513 2026-02-23T14:44:50.519 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5484 8 365281 365273 0.520328 0 - 0.0299513 2026-02-23T14:44:51.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:44:50 trial068 ceph-mon[38805]: pgmap v3317: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:44:51.227 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:44:51.346 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:44:51.446 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:44:51.444+0000 7f986b3f2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:44:51.446 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:44:51.444+0000 7f986b3f2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:44:51.519 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5485 8 365281 365273 0.520233 0 - 0.0299513 2026-02-23T14:44:51.555 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:44:51.556 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":263,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:44:09.737746+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:40:20.346927+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:43:36.356556+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:40:01.279839+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:44:51.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:44:51 trial068 ceph-mon[38805]: 5.a scrub starts 2026-02-23T14:44:51.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:44:51 trial068 ceph-mon[38805]: 5.a scrub ok 2026-02-23T14:44:51.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:44:51 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1006525747' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:44:52.519 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5486 8 365281 365273 0.520138 0 - 0.0299513 2026-02-23T14:44:53.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:44:52 trial068 ceph-mon[38805]: pgmap v3318: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:44:53.519 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5487 8 365281 365273 0.520043 0 - 0.0299513 2026-02-23T14:44:54.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:44:53 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:44:53] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:44:54.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:44:53 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:44:53.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:44:54.519 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5488 8 365281 365273 0.519948 0 - 0.0299513 2026-02-23T14:44:54.593 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:44:54.711 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:44:54.812 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:44:54.810+0000 7f53b547d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:44:54.812 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:44:54.810+0000 7f53b547d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:44:54.921 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:44:54.922 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":263,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:44:09.737746+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:40:20.346927+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:43:36.356556+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:40:01.279839+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:44:54.922 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:44:54 trial068 ceph-mon[38805]: pgmap v3319: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:44:55.519 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5489 8 365281 365273 0.519854 0 - 0.0299513 2026-02-23T14:44:56.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:44:55 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3251136496' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:44:56.519 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5490 8 365281 365273 0.519759 0 - 0.0299513 2026-02-23T14:44:57.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:44:56 trial068 ceph-mon[38805]: pgmap v3320: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:44:57.519 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5491 8 365281 365273 0.519664 0 - 0.0299513 2026-02-23T14:44:57.959 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:44:58.078 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:44:58.175 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:44:58.173+0000 7f92b7adf640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:44:58.176 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:44:58.174+0000 7f92b7adf640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:44:58.285 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:44:58.285 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":263,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:44:09.737746+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:40:20.346927+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:43:36.356556+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:40:01.279839+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:44:58.519 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5492 8 365281 365273 0.51957 0 - 0.0299513 2026-02-23T14:44:59.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:44:58 trial068 ceph-mon[38805]: pgmap v3321: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:44:59.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:44:58 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1629708587' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:44:59.520 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5493 8 365281 365273 0.519475 0 - 0.0299513 2026-02-23T14:45:00.520 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5494 8 365281 365273 0.51938 0 - 0.0299513 2026-02-23T14:45:01.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:45:00 trial068 ceph-mon[38805]: pgmap v3322: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:45:01.322 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:45:01.438 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:45:01.520 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5495 8 365281 365273 0.519286 0 - 0.0299513 2026-02-23T14:45:01.538 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:45:01.536+0000 7f898a4d8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:45:01.538 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:45:01.536+0000 7f898a4d8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:45:01.649 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:45:01.649 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":263,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:44:09.737746+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:40:20.346927+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:43:36.356556+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:40:01.279839+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:45:02.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:45:01 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1071959922' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:45:02.520 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5496 8 365281 365273 0.519191 0 - 0.0299513 2026-02-23T14:45:03.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:45:02 trial068 ceph-mon[38805]: pgmap v3323: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:45:03.520 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5497 8 365281 365273 0.519097 0 - 0.0299513 2026-02-23T14:45:04.266 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:45:03 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:45:03.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:45:04.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:45:03 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:45:03] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:45:04.520 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5498 8 365281 365273 0.519003 0 - 0.0299513 2026-02-23T14:45:04.686 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:45:04.806 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:45:04.910 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:45:04.908+0000 7f8d6409d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:45:04.910 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:45:04.909+0000 7f8d6409d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:45:05.020 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:45:05.020 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":263,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:44:09.737746+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:40:20.346927+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:43:36.356556+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:40:01.279839+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:45:05.020 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:45:04 trial068 ceph-mon[38805]: pgmap v3324: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:45:05.520 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5499 8 365281 365273 0.518908 0 - 0.0299513 2026-02-23T14:45:06.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:45:05 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3821090031' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:45:06.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:45:05 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2026-02-23T14:45:06.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:45:05 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-23T14:45:06.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:45:05 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2026-02-23T14:45:06.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:45:05 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T14:45:06.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:45:05 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2026-02-23T14:45:06.520 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T14:45:06.519934+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T14:45:06.520 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T14:45:06.520 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5500 8 365281 365273 0.518814 0 - 0.0299513 2026-02-23T14:45:07.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:45:06 trial068 ceph-mon[38805]: pgmap v3325: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:45:07.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:45:06 trial068 ceph-mon[38805]: pgmap v3326: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:45:07.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:45:06 trial068 ceph-mon[38805]: pgmap v3327: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:45:07.520 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5501 8 365281 365273 0.51872 0 - 0.0299513 2026-02-23T14:45:08.059 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:45:08.176 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:45:08.286 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:45:08.284+0000 7f4d9224a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:45:08.286 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:45:08.284+0000 7f4d9224a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:45:08.397 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:45:08.397 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":263,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:44:09.737746+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:40:20.346927+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:43:36.356556+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:40:01.279839+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:45:08.520 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5502 8 365281 365273 0.518625 0 - 0.0299513 2026-02-23T14:45:09.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:45:08 trial068 ceph-mon[38805]: pgmap v3328: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:45:09.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:45:08 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/296465164' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:45:09.520 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5503 8 365281 365273 0.518531 0 - 0.0299513 2026-02-23T14:45:10.520 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5504 8 365281 365273 0.518437 0 - 0.0299513 2026-02-23T14:45:11.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:45:10 trial068 ceph-mon[38805]: pgmap v3329: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:45:11.436 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:45:11.520 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5505 8 365281 365273 0.518343 0 - 0.0299513 2026-02-23T14:45:11.553 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:45:11.650 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:45:11.648+0000 7f2a19a9a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:45:11.651 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:45:11.649+0000 7f2a19a9a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:45:11.761 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:45:11.761 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":263,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:44:09.737746+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:40:20.346927+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:43:36.356556+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:40:01.279839+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:45:12.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:45:11 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/4088005734' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:45:12.521 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5506 8 365281 365273 0.518249 0 - 0.0299513 2026-02-23T14:45:13.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:45:12 trial068 ceph-mon[38805]: pgmap v3330: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:45:13.521 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5507 8 365281 365273 0.518154 0 - 0.0299513 2026-02-23T14:45:14.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:45:13 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:45:13] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:45:14.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:45:13 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:45:13.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:45:14.521 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5508 8 365281 365273 0.51806 0 - 0.0299513 2026-02-23T14:45:14.798 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:45:14.915 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:45:15.012 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:45:15.010+0000 7f52f7b15640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:45:15.012 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:45:15.010+0000 7f52f7b15640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:45:15.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:45:14 trial068 ceph-mon[38805]: pgmap v3331: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:45:15.124 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:45:15.124 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":263,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:44:09.737746+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:40:20.346927+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:43:36.356556+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:40:01.279839+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:45:15.521 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5509 8 365281 365273 0.517966 0 - 0.0299513 2026-02-23T14:45:16.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:45:15 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/908243329' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:45:16.521 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5510 8 365281 365273 0.517872 0 - 0.0299513 2026-02-23T14:45:17.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:45:16 trial068 ceph-mon[38805]: pgmap v3332: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:45:17.521 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5511 8 365281 365273 0.517778 0 - 0.0299513 2026-02-23T14:45:18.163 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:45:18.284 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:45:18.385 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:45:18.383+0000 7f1059d38640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:45:18.385 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:45:18.383+0000 7f1059d38640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:45:18.494 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:45:18.494 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":263,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:44:09.737746+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:40:20.346927+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:43:36.356556+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:40:01.279839+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:45:18.521 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5512 8 365281 365273 0.517684 0 - 0.0299513 2026-02-23T14:45:19.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:45:18 trial068 ceph-mon[38805]: pgmap v3333: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:45:19.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:45:18 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2034183250' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:45:19.521 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5513 8 365281 365273 0.517591 0 - 0.0299513 2026-02-23T14:45:20.521 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5514 8 365281 365273 0.517497 0 - 0.0299513 2026-02-23T14:45:21.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:45:20 trial068 ceph-mon[38805]: pgmap v3334: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:45:21.521 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5515 8 365281 365273 0.517403 0 - 0.0299513 2026-02-23T14:45:21.531 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:45:21.648 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:45:21.747 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:45:21.745+0000 7ff080dfc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:45:21.747 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:45:21.745+0000 7ff080dfc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:45:21.857 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:45:21.857 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":263,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:44:09.737746+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:40:20.346927+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:43:36.356556+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:40:01.279839+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:45:22.521 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5516 8 365281 365273 0.517309 0 - 0.0299513 2026-02-23T14:45:23.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:45:22 trial068 ceph-mon[38805]: pgmap v3335: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:45:23.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:45:22 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3520723946' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:45:23.521 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5517 8 365281 365273 0.517215 0 - 0.0299513 2026-02-23T14:45:24.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:45:23 trial068 ceph-mon[38805]: osdmap e264: 4 total, 3 up, 4 in 2026-02-23T14:45:24.094 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:45:23 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:45:23] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:45:24.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:45:23 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:45:23.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:45:24.521 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5518 8 365281 365273 0.517122 0 - 0.0299513 2026-02-23T14:45:24.894 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:45:25.015 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:45:25.069 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:45:24 trial068 ceph-mon[38805]: pgmap v3337: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:45:25.069 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:45:24 trial068 ceph-mon[38805]: osdmap e265: 4 total, 3 up, 4 in 2026-02-23T14:45:25.121 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:45:25.119+0000 7f4f2de7b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:45:25.121 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:45:25.119+0000 7f4f2de7b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:45:25.231 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:45:25.231 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":265,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:45:24.748279+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:40:20.346927+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:43:36.356556+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:44:40.015825+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:45:25.521 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5519 8 365281 365273 0.517028 0 - 0.0299513 2026-02-23T14:45:26.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:45:25 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1240800200' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:45:26.522 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T14:45:26.521283+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T14:45:26.522 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T14:45:26.522 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5520 8 365281 365273 0.516934 0 - 0.0299513 2026-02-23T14:45:27.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:45:26 trial068 ceph-mon[38805]: pgmap v3339: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:45:27.522 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5521 8 365281 365273 0.516841 0 - 0.0299513 2026-02-23T14:45:28.268 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:45:28.388 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:45:28.488 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:45:28.486+0000 7f185ddaf640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:45:28.488 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:45:28.486+0000 7f185ddaf640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:45:28.521 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5522 8 365281 365273 0.516747 0 - 0.0299513 2026-02-23T14:45:28.597 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:45:28.597 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":265,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:45:24.748279+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:40:20.346927+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:43:36.356556+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:44:40.015825+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:45:29.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:45:28 trial068 ceph-mon[38805]: pgmap v3340: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:45:29.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:45:28 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3950612025' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:45:29.522 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5523 8 365281 365273 0.516653 0 - 0.0299513 2026-02-23T14:45:30.522 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5524 8 365281 365273 0.51656 0 - 0.0299513 2026-02-23T14:45:31.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:45:30 trial068 ceph-mon[38805]: pgmap v3341: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:45:31.522 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5525 8 365281 365273 0.516466 0 - 0.0299513 2026-02-23T14:45:31.634 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:45:31.753 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:45:31.856 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:45:31.854+0000 7f9e4758c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:45:31.856 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:45:31.854+0000 7f9e4758c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:45:31.966 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:45:31.967 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":265,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:45:24.748279+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:40:20.346927+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:43:36.356556+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:44:40.015825+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:45:32.522 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5526 8 365281 365273 0.516373 0 - 0.0299513 2026-02-23T14:45:33.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:45:32 trial068 ceph-mon[38805]: pgmap v3342: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:45:33.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:45:32 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/219851624' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:45:33.522 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5527 8 365281 365273 0.516279 0 - 0.0299513 2026-02-23T14:45:34.266 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:45:33 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:45:33.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:45:34.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:45:33 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:45:33] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:45:34.522 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5528 8 365281 365273 0.516186 0 - 0.0299513 2026-02-23T14:45:35.004 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:45:35.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:45:34 trial068 ceph-mon[38805]: pgmap v3343: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:45:35.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:45:34 trial068 ceph-mon[38805]: 5.e scrub starts 2026-02-23T14:45:35.122 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:45:35.229 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:45:35.227+0000 7f326cdaf640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:45:35.229 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:45:35.227+0000 7f326cdaf640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:45:35.343 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:45:35.343 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":265,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:45:24.748279+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:40:20.346927+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:43:36.356556+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:44:40.015825+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:45:35.522 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5529 8 365281 365273 0.516093 0 - 0.0299513 2026-02-23T14:45:36.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:45:35 trial068 ceph-mon[38805]: 5.e scrub ok 2026-02-23T14:45:36.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:45:35 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2033491772' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:45:36.522 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5530 8 365281 365273 0.515999 0 - 0.0299513 2026-02-23T14:45:37.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:45:36 trial068 ceph-mon[38805]: 5.5 scrub starts 2026-02-23T14:45:37.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:45:36 trial068 ceph-mon[38805]: pgmap v3344: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:45:37.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:45:36 trial068 ceph-mon[38805]: 5.5 scrub ok 2026-02-23T14:45:37.522 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5531 8 365281 365273 0.515906 0 - 0.0299513 2026-02-23T14:45:38.380 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:45:38.497 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:45:38.523 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5532 8 365281 365273 0.515813 0 - 0.0299513 2026-02-23T14:45:38.597 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:45:38.595+0000 7f876d0d1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:45:38.597 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:45:38.595+0000 7f876d0d1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:45:38.706 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:45:38.707 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":265,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:45:24.748279+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:40:20.346927+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:43:36.356556+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:44:40.015825+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:45:39.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:45:38 trial068 ceph-mon[38805]: pgmap v3345: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:45:39.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:45:38 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2241732266' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:45:39.522 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5533 8 365281 365273 0.51572 0 - 0.0299513 2026-02-23T14:45:40.523 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5534 8 365281 365273 0.515626 0 - 0.0299513 2026-02-23T14:45:41.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:45:40 trial068 ceph-mon[38805]: pgmap v3346: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:45:41.523 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5535 8 365281 365273 0.515533 0 - 0.0299513 2026-02-23T14:45:41.744 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:45:41.863 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:45:41.960 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:45:41.958+0000 7f86f4674640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:45:41.961 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:45:41.959+0000 7f86f4674640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:45:42.069 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:45:42.069 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":265,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:45:24.748279+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:40:20.346927+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:43:36.356556+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:44:40.015825+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:45:42.523 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5536 8 365281 365273 0.51544 0 - 0.0299513 2026-02-23T14:45:43.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:45:42 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T14:45:43.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:45:42 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T14:45:43.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:45:42 trial068 ceph-mon[38805]: pgmap v3347: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:45:43.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:45:42 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2469334151' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:45:43.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:45:42 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T14:45:43.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:45:42 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T14:45:43.523 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5537 8 365281 365273 0.515347 0 - 0.0299513 2026-02-23T14:45:44.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:45:43 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:45:43] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:45:44.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:45:43 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:45:43.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:45:44.523 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5538 8 365281 365273 0.515254 0 - 0.0299513 2026-02-23T14:45:45.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:45:44 trial068 ceph-mon[38805]: pgmap v3348: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:45:45.106 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:45:45.221 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:45:45.330 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:45:45.328+0000 7fd90890b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:45:45.330 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:45:45.328+0000 7fd90890b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:45:45.440 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:45:45.440 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":265,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:45:24.748279+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:40:20.346927+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:43:36.356556+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:44:40.015825+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:45:45.523 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5539 8 365281 365273 0.515161 0 - 0.0299513 2026-02-23T14:45:46.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:45:45 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3521799081' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:45:46.523 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T14:45:46.522819+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T14:45:46.523 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T14:45:46.523 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5540 8 365281 365273 0.515068 0 - 0.0299513 2026-02-23T14:45:47.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:45:46 trial068 ceph-mon[38805]: pgmap v3349: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:45:47.523 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5541 8 365281 365273 0.514975 0 - 0.0299513 2026-02-23T14:45:48.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:45:47 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T14:45:48.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:45:47 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T14:45:48.478 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:45:48.523 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5542 8 365281 365273 0.514882 0 - 0.0299513 2026-02-23T14:45:48.598 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:45:48.697 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:45:48.695+0000 7fc9fabe3640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:45:48.698 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:45:48.696+0000 7fc9fabe3640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:45:48.810 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:45:48.810 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":265,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:45:24.748279+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:40:20.346927+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:43:36.356556+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:44:40.015825+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:45:49.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:45:48 trial068 ceph-mon[38805]: pgmap v3350: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:45:49.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:45:48 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2672604325' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:45:49.523 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5543 8 365281 365273 0.514789 0 - 0.0299513 2026-02-23T14:45:50.523 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5544 8 365281 365273 0.514696 0 - 0.0299513 2026-02-23T14:45:51.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:45:50 trial068 ceph-mon[38805]: pgmap v3351: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:45:51.523 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5545 8 365281 365273 0.514604 0 - 0.0299513 2026-02-23T14:45:51.867 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:45:51.986 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:45:52.086 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:45:52.084+0000 7f911d2ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:45:52.086 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:45:52.085+0000 7f911d2ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:45:52.198 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:45:52.198 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":265,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:45:24.748279+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:40:20.346927+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:43:36.356556+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:44:40.015825+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:45:52.524 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5546 8 365281 365273 0.514511 0 - 0.0299513 2026-02-23T14:45:53.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:45:52 trial068 ceph-mon[38805]: pgmap v3352: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:45:53.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:45:52 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3292816038' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:45:53.524 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5547 8 365281 365273 0.514418 0 - 0.0299513 2026-02-23T14:45:54.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:45:53 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:45:53] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:45:54.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:45:53 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:45:53.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:45:54.524 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5548 8 365281 365273 0.514325 0 - 0.0299513 2026-02-23T14:45:55.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:45:54 trial068 ceph-mon[38805]: pgmap v3353: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:45:55.249 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:45:55.367 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:45:55.465 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:45:55.463+0000 7eff89ad1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:45:55.466 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:45:55.463+0000 7eff89ad1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:45:55.524 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5549 8 365281 365273 0.514233 0 - 0.0299513 2026-02-23T14:45:55.577 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:45:55.577 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":265,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:45:24.748279+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:40:20.346927+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:43:36.356556+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:44:40.015825+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:45:56.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:45:55 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/495276872' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:45:56.524 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5550 8 365281 365273 0.51414 0 - 0.0299513 2026-02-23T14:45:57.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:45:56 trial068 ceph-mon[38805]: pgmap v3354: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:45:57.524 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5551 8 365281 365273 0.514047 0 - 0.0299513 2026-02-23T14:45:58.524 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5552 8 365281 365273 0.513955 0 - 0.0299513 2026-02-23T14:45:58.615 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:45:58.734 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:45:58.831 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:45:58.829+0000 7efd46540640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:45:58.831 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:45:58.829+0000 7efd46540640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:45:58.942 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:45:58 trial068 ceph-mon[38805]: 5.8 scrub starts 2026-02-23T14:45:58.942 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:45:58 trial068 ceph-mon[38805]: pgmap v3355: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:45:58.942 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:45:58 trial068 ceph-mon[38805]: 5.8 scrub ok 2026-02-23T14:45:58.942 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:45:58.942 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":265,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:45:24.748279+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:40:20.346927+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:43:36.356556+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:44:40.015825+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:45:59.524 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5553 8 365281 365273 0.513862 0 - 0.0299513 2026-02-23T14:46:00.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:45:59 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2535001656' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:46:00.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:45:59 trial068 ceph-mon[38805]: 5.7 scrub starts 2026-02-23T14:46:00.524 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5554 8 365281 365273 0.51377 0 - 0.0299513 2026-02-23T14:46:01.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:46:00 trial068 ceph-mon[38805]: pgmap v3356: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:46:01.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:46:00 trial068 ceph-mon[38805]: 5.7 scrub ok 2026-02-23T14:46:01.524 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5555 8 365281 365273 0.513677 0 - 0.0299513 2026-02-23T14:46:01.978 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:46:02.095 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:46:02.197 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:46:02.194+0000 7f932aeb3640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:46:02.197 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:46:02.195+0000 7f932aeb3640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:46:02.306 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:46:02.306 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":265,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:45:24.748279+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:40:20.346927+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:43:36.356556+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:44:40.015825+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:46:02.524 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5556 8 365281 365273 0.513585 0 - 0.0299513 2026-02-23T14:46:03.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:46:02 trial068 ceph-mon[38805]: pgmap v3357: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:46:03.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:46:02 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2114498589' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:46:03.524 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5557 8 365281 365273 0.513492 0 - 0.0299513 2026-02-23T14:46:04.266 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:46:03 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:46:03.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:46:04.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:46:03 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:46:03] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:46:04.524 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5558 8 365281 365273 0.5134 0 - 0.0299513 2026-02-23T14:46:05.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:46:04 trial068 ceph-mon[38805]: pgmap v3358: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:46:05.347 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:46:05.463 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:46:05.525 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5559 8 365281 365273 0.513308 0 - 0.0299513 2026-02-23T14:46:05.569 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:46:05.567+0000 7f39796e7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:46:05.569 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:46:05.568+0000 7f39796e7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:46:05.678 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:46:05.678 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":265,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:45:24.748279+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:40:20.346927+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:43:36.356556+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:44:40.015825+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:46:06.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:46:05 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2026-02-23T14:46:06.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:46:05 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-23T14:46:06.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:46:05 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2026-02-23T14:46:06.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:46:05 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T14:46:06.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:46:05 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2026-02-23T14:46:06.095 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:46:05 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/4276042602' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:46:06.525 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T14:46:06.524332+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T14:46:06.525 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T14:46:06.525 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5560 8 365281 365273 0.513215 0 - 0.0299513 2026-02-23T14:46:07.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:46:06 trial068 ceph-mon[38805]: 5.a scrub starts 2026-02-23T14:46:07.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:46:06 trial068 ceph-mon[38805]: 5.a scrub ok 2026-02-23T14:46:07.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:46:06 trial068 ceph-mon[38805]: pgmap v3359: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:46:07.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:46:06 trial068 ceph-mon[38805]: pgmap v3360: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:46:07.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:46:06 trial068 ceph-mon[38805]: pgmap v3361: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:46:07.525 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5561 8 365281 365273 0.513123 0 - 0.0299513 2026-02-23T14:46:08.525 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5562 8 365281 365273 0.513031 0 - 0.0299513 2026-02-23T14:46:08.715 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:46:08.832 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:46:08.933 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:46:08.931+0000 7effbe9a3640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:46:08.933 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:46:08.931+0000 7effbe9a3640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:46:09.042 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:46:09.043 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":265,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:45:24.748279+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:40:20.346927+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:43:36.356556+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:44:40.015825+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:46:09.043 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:46:08 trial068 ceph-mon[38805]: pgmap v3362: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:46:09.525 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5563 8 365281 365273 0.512938 0 - 0.0299513 2026-02-23T14:46:10.245 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:46:09 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3073286934' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:46:10.525 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5564 8 365281 365273 0.512846 0 - 0.0299513 2026-02-23T14:46:11.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:46:10 trial068 ceph-mon[38805]: pgmap v3363: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:46:11.525 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5565 8 365281 365273 0.512754 0 - 0.0299513 2026-02-23T14:46:12.082 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:46:12.199 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:46:12.296 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:46:12.294+0000 7f6327886640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:46:12.296 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:46:12.294+0000 7f6327886640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:46:12.407 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:46:12.407 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":265,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:45:24.748279+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:40:20.346927+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:43:36.356556+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:44:40.015825+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:46:12.525 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5566 8 365281 365273 0.512662 0 - 0.0299513 2026-02-23T14:46:13.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:46:12 trial068 ceph-mon[38805]: pgmap v3364: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:46:13.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:46:12 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1571998802' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:46:13.525 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5567 8 365281 365273 0.51257 0 - 0.0299513 2026-02-23T14:46:14.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:46:13 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:46:13] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:46:14.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:46:13 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:46:13.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:46:14.525 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5568 8 365281 365273 0.512478 0 - 0.0299513 2026-02-23T14:46:15.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:46:14 trial068 ceph-mon[38805]: pgmap v3365: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:46:15.445 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:46:15.525 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5569 8 365281 365273 0.512386 0 - 0.0299513 2026-02-23T14:46:15.565 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:46:15.668 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:46:15.666+0000 7f6edb0aa640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:46:15.668 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:46:15.666+0000 7f6edb0aa640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:46:15.777 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:46:15.777 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":265,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:45:24.748279+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:40:20.346927+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:43:36.356556+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:44:40.015825+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:46:16.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:46:15 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2754798890' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:46:16.525 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5570 8 365281 365273 0.512294 0 - 0.0299513 2026-02-23T14:46:17.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:46:16 trial068 ceph-mon[38805]: pgmap v3366: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:46:17.525 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5571 8 365281 365273 0.512202 0 - 0.0299513 2026-02-23T14:46:18.525 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5572 8 365281 365273 0.51211 0 - 0.0299513 2026-02-23T14:46:18.813 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:46:18.930 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:46:19.034 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:46:19.032+0000 7ff5dfa2e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:46:19.035 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:46:19.033+0000 7ff5dfa2e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:46:19.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:46:18 trial068 ceph-mon[38805]: pgmap v3367: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:46:19.145 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:46:19.145 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":265,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:45:24.748279+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:40:20.346927+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:43:36.356556+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:44:40.015825+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:46:19.525 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5573 8 365281 365273 0.512018 0 - 0.0299513 2026-02-23T14:46:20.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:46:19 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/4270200045' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:46:20.525 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5574 8 365281 365273 0.511926 0 - 0.0299513 2026-02-23T14:46:21.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:46:20 trial068 ceph-mon[38805]: pgmap v3368: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:46:21.525 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5575 8 365281 365273 0.511834 0 - 0.0299513 2026-02-23T14:46:22.183 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:46:22.298 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:46:22.400 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:46:22.398+0000 7faee3620640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:46:22.400 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:46:22.398+0000 7faee3620640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:46:22.509 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:46:22.509 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":265,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:45:24.748279+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:40:20.346927+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:43:36.356556+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:44:40.015825+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:46:22.526 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5576 8 365281 365273 0.511743 0 - 0.0299513 2026-02-23T14:46:23.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:46:22 trial068 ceph-mon[38805]: pgmap v3369: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:46:23.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:46:22 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3490462276' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:46:23.526 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5577 8 365281 365273 0.511651 0 - 0.0299513 2026-02-23T14:46:24.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:46:23 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:46:23] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:46:24.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:46:23 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:46:23.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:46:24.526 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5578 8 365281 365273 0.511559 0 - 0.0299513 2026-02-23T14:46:25.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:46:24 trial068 ceph-mon[38805]: pgmap v3370: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:46:25.526 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5579 8 365281 365273 0.511467 0 - 0.0299513 2026-02-23T14:46:25.545 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:46:25.664 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:46:25.761 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:46:25.760+0000 7f39362cf640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:46:25.762 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:46:25.760+0000 7f39362cf640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:46:25.872 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:46:25.872 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":265,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:45:24.748279+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:40:20.346927+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:43:36.356556+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:44:40.015825+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:46:26.526 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T14:46:26.525813+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T14:46:26.526 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T14:46:26.526 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5580 8 365281 365273 0.511376 0 - 0.0299513 2026-02-23T14:46:27.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:46:26 trial068 ceph-mon[38805]: pgmap v3371: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:46:27.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:46:26 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/435048' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:46:27.526 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5581 8 365281 365273 0.511284 0 - 0.0299513 2026-02-23T14:46:28.526 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5582 8 365281 365273 0.511192 0 - 0.0299513 2026-02-23T14:46:28.910 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:46:29.028 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:46:29.124 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:46:29.122+0000 7f3249ab9640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:46:29.125 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:46:28 trial068 ceph-mon[38805]: pgmap v3372: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:46:29.125 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:46:29.123+0000 7f3249ab9640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:46:29.235 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:46:29.235 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":265,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:45:24.748279+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:40:20.346927+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:43:36.356556+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:44:40.015825+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:46:29.526 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5583 8 365281 365273 0.511101 0 - 0.0299513 2026-02-23T14:46:30.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:46:29 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/4212044227' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:46:30.526 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5584 8 365281 365273 0.511009 0 - 0.0299513 2026-02-23T14:46:31.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:46:30 trial068 ceph-mon[38805]: pgmap v3373: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:46:31.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:46:30 trial068 ceph-mon[38805]: osdmap e266: 4 total, 3 up, 4 in 2026-02-23T14:46:31.526 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5585 8 365281 365273 0.510918 0 - 0.0299513 2026-02-23T14:46:32.273 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:46:32.392 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:46:32.496 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:46:32.494+0000 7fa6a68a0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:46:32.496 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:46:32.494+0000 7fa6a68a0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:46:32.526 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5586 8 365281 365273 0.510826 0 - 0.0299513 2026-02-23T14:46:32.606 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:46:32.606 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":266,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:46:29.865405+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:45:46.188265+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:43:36.356556+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:44:40.015825+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:46:33.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:46:32 trial068 ceph-mon[38805]: pgmap v3375: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:46:33.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:46:32 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2676274115' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:46:33.526 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5587 8 365281 365273 0.510735 0 - 0.0299513 2026-02-23T14:46:34.266 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:46:33 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:46:33.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:46:34.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:46:33 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:46:33] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:46:34.527 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5588 8 365281 365273 0.510644 0 - 0.0299513 2026-02-23T14:46:35.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:46:34 trial068 ceph-mon[38805]: pgmap v3376: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:46:35.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:46:34 trial068 ceph-mon[38805]: osdmap e267: 4 total, 3 up, 4 in 2026-02-23T14:46:35.527 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5589 8 365281 365273 0.510552 0 - 0.0299513 2026-02-23T14:46:35.643 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:46:35.762 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:46:35.862 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:46:35.860+0000 7f07fa62a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:46:35.862 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:46:35.860+0000 7f07fa62a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:46:35.974 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:46:35.974 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":267,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:46:34.758585+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:45:46.188265+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:43:36.356556+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:44:40.015825+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:46:36.527 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5590 8 365281 365273 0.510461 0 - 0.0299513 2026-02-23T14:46:37.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:46:36 trial068 ceph-mon[38805]: pgmap v3378: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:46:37.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:46:36 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/764024161' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:46:37.527 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5591 8 365281 365273 0.51037 0 - 0.0299513 2026-02-23T14:46:38.527 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5592 8 365281 365273 0.510278 0 - 0.0299513 2026-02-23T14:46:39.013 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:46:39.130 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:46:39.153 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:46:38 trial068 ceph-mon[38805]: pgmap v3379: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:46:39.231 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:46:39.229+0000 7ff40563b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:46:39.231 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:46:39.229+0000 7ff40563b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:46:39.340 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:46:39.340 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":267,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:46:34.758585+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:45:46.188265+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:43:36.356556+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:44:40.015825+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:46:39.527 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5593 8 365281 365273 0.510187 0 - 0.0299513 2026-02-23T14:46:40.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:46:39 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1501663183' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:46:40.527 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5594 8 365281 365273 0.510096 0 - 0.0299513 2026-02-23T14:46:41.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:46:40 trial068 ceph-mon[38805]: pgmap v3380: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:46:41.527 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5595 8 365281 365273 0.510005 0 - 0.0299513 2026-02-23T14:46:42.377 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:46:42.495 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:46:42.527 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5596 8 365281 365273 0.509914 0 - 0.0299513 2026-02-23T14:46:42.601 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:46:42.599+0000 7fd3d45e4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:46:42.601 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:46:42.599+0000 7fd3d45e4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:46:42.711 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:46:42.711 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":267,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:46:34.758585+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:45:46.188265+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:43:36.356556+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:44:40.015825+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:46:43.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:46:42 trial068 ceph-mon[38805]: pgmap v3381: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:46:43.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:46:42 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1484036732' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:46:43.527 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5597 8 365281 365273 0.509823 0 - 0.0299513 2026-02-23T14:46:44.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:46:43 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:46:43] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:46:44.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:46:43 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:46:43.974Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:46:44.527 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5598 8 365281 365273 0.509731 0 - 0.0299513 2026-02-23T14:46:45.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:46:44 trial068 ceph-mon[38805]: pgmap v3382: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:46:45.527 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5599 8 365281 365273 0.50964 0 - 0.0299513 2026-02-23T14:46:45.752 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:46:45.871 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:46:45.970 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:46:45.968+0000 7f7145986640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:46:45.970 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:46:45.968+0000 7f7145986640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:46:46.080 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:46:46.081 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":267,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:46:34.758585+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:45:46.188265+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:43:36.356556+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:44:40.015825+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:46:46.527 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T14:46:46.527247+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T14:46:46.528 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T14:46:46.528 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5600 8 365281 365273 0.509549 0 - 0.0299513 2026-02-23T14:46:47.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:46:46 trial068 ceph-mon[38805]: pgmap v3383: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:46:47.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:46:46 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3660039638' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:46:47.527 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5601 8 365281 365273 0.509458 0 - 0.0299513 2026-02-23T14:46:48.527 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5602 8 365281 365273 0.509367 0 - 0.0299513 2026-02-23T14:46:49.119 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:46:49.142 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:46:48 trial068 ceph-mon[38805]: 5.e scrub starts 2026-02-23T14:46:49.142 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:46:48 trial068 ceph-mon[38805]: 5.e scrub ok 2026-02-23T14:46:49.142 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:46:48 trial068 ceph-mon[38805]: pgmap v3384: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:46:49.237 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:46:49.337 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:46:49.335+0000 7f7602f0e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:46:49.338 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:46:49.336+0000 7f7602f0e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:46:49.449 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:46:49.449 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":267,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:46:34.758585+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:45:46.188265+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:43:36.356556+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:44:40.015825+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:46:49.528 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5603 8 365281 365273 0.509277 0 - 0.0299513 2026-02-23T14:46:50.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:46:49 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/348697924' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:46:50.528 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5604 8 365281 365273 0.509186 0 - 0.0299513 2026-02-23T14:46:51.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:46:50 trial068 ceph-mon[38805]: pgmap v3385: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:46:51.528 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5605 8 365281 365273 0.509095 0 - 0.0299513 2026-02-23T14:46:52.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:46:51 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T14:46:52.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:46:51 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T14:46:52.486 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:46:52.528 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5606 8 365281 365273 0.509004 0 - 0.0299513 2026-02-23T14:46:52.607 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:46:52.708 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:46:52.706+0000 7f5939958640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:46:52.708 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:46:52.706+0000 7f5939958640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:46:52.817 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:46:52.817 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":267,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:46:34.758585+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:45:46.188265+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:43:36.356556+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:44:40.015825+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:46:53.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:46:52 trial068 ceph-mon[38805]: pgmap v3386: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:46:53.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:46:52 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/856298628' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:46:53.528 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5607 8 365281 365273 0.508913 0 - 0.0299513 2026-02-23T14:46:54.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:46:53 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:46:53] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:46:54.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:46:53 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:46:53.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:46:54.528 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5608 8 365281 365273 0.508822 0 - 0.0299513 2026-02-23T14:46:55.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:46:54 trial068 ceph-mon[38805]: pgmap v3387: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:46:55.528 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5609 8 365281 365273 0.508732 0 - 0.0299513 2026-02-23T14:46:55.853 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:46:55.971 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:46:56.071 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:46:56.069+0000 7f57ad9a9640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:46:56.071 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:46:56.069+0000 7f57ad9a9640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:46:56.179 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:46:56.179 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":267,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:46:34.758585+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:45:46.188265+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:43:36.356556+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:44:40.015825+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:46:56.180 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:46:55 trial068 ceph-mon[38805]: 5.5 scrub starts 2026-02-23T14:46:56.180 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:46:55 trial068 ceph-mon[38805]: 5.5 scrub ok 2026-02-23T14:46:56.528 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5610 8 365281 365273 0.508641 0 - 0.0299513 2026-02-23T14:46:57.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:46:56 trial068 ceph-mon[38805]: pgmap v3388: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:46:57.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:46:56 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2181930021' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:46:57.528 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5611 8 365281 365273 0.50855 0 - 0.0299513 2026-02-23T14:46:58.528 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5612 8 365281 365273 0.50846 0 - 0.0299513 2026-02-23T14:46:59.216 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:46:59.240 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:46:58 trial068 ceph-mon[38805]: pgmap v3389: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:46:59.333 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:46:59.437 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:46:59.435+0000 7feae7933640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:46:59.437 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:46:59.435+0000 7feae7933640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:46:59.528 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5613 8 365281 365273 0.508369 0 - 0.0299513 2026-02-23T14:46:59.546 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:46:59.546 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":267,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:46:34.758585+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:45:46.188265+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:43:36.356556+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:44:40.015825+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:47:00.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:46:59 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/939140881' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:47:00.528 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5614 8 365281 365273 0.508279 0 - 0.0299513 2026-02-23T14:47:01.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:47:00 trial068 ceph-mon[38805]: pgmap v3390: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:47:01.529 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5615 8 365281 365273 0.508188 0 - 0.0299513 2026-02-23T14:47:02.529 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5616 8 365281 365273 0.508098 0 - 0.0299513 2026-02-23T14:47:02.582 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:47:02.700 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:47:02.798 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:47:02.796+0000 7fb3ce1d3640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:47:02.798 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:47:02.796+0000 7fb3ce1d3640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:47:02.906 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:47:02.907 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":267,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:46:34.758585+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:45:46.188265+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:43:36.356556+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:44:40.015825+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:47:03.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:47:02 trial068 ceph-mon[38805]: pgmap v3391: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:47:03.529 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5617 8 365281 365273 0.508007 0 - 0.0299513 2026-02-23T14:47:04.265 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:47:03 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:47:03.974Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:47:04.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:47:03 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3852521510' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:47:04.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:47:03 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:47:03] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:47:04.529 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5618 8 365281 365273 0.507917 0 - 0.0299513 2026-02-23T14:47:05.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:47:04 trial068 ceph-mon[38805]: pgmap v3392: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:47:05.529 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5619 8 365281 365273 0.507826 0 - 0.0299513 2026-02-23T14:47:05.959 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:47:06.074 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:47:06.181 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:47:06.179+0000 7f1b5165e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:47:06.181 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:47:05 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T14:47:06.182 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:47:05 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T14:47:06.182 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:47:05 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2026-02-23T14:47:06.182 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:47:05 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T14:47:06.182 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:47:05 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T14:47:06.182 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:47:05 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T14:47:06.182 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:47:05 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T14:47:06.182 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:47:06.179+0000 7f1b5165e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:47:06.290 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:47:06.290 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":267,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:46:34.758585+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:45:46.188265+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:43:36.356556+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:44:40.015825+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:47:06.529 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T14:47:06.528722+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T14:47:06.529 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T14:47:06.529 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5620 8 365281 365273 0.507736 0 - 0.0299513 2026-02-23T14:47:07.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:47:06 trial068 ceph-mon[38805]: pgmap v3393: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:47:07.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:47:06 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-23T14:47:07.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:47:06 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2026-02-23T14:47:07.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:47:06 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T14:47:07.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:47:06 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2026-02-23T14:47:07.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:47:06 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2291997177' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:47:07.529 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5621 8 365281 365273 0.507646 0 - 0.0299513 2026-02-23T14:47:08.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:47:07 trial068 ceph-mon[38805]: pgmap v3394: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:47:08.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:47:07 trial068 ceph-mon[38805]: pgmap v3395: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:47:08.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:47:07 trial068 ceph-mon[38805]: 5.7 scrub starts 2026-02-23T14:47:08.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:47:07 trial068 ceph-mon[38805]: 5.7 scrub ok 2026-02-23T14:47:08.529 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5622 8 365281 365273 0.507555 0 - 0.0299513 2026-02-23T14:47:09.328 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:47:09.448 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:47:09.529 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5623 8 365281 365273 0.507465 0 - 0.0299513 2026-02-23T14:47:09.559 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:47:09.557+0000 7f15c1075640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:47:09.559 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:47:09.557+0000 7f15c1075640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:47:09.668 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:47:09.668 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":267,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:46:34.758585+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:45:46.188265+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:43:36.356556+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:44:40.015825+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:47:10.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:47:09 trial068 ceph-mon[38805]: pgmap v3396: 17 pgs: 1 active+clean+scrubbing, 1 active+recovery_unfound+undersized+degraded, 4 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:47:10.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:47:09 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3307983506' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:47:10.529 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5624 8 365281 365273 0.507375 0 - 0.0299513 2026-02-23T14:47:11.529 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5625 8 365281 365273 0.507285 0 - 0.0299513 2026-02-23T14:47:12.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:47:11 trial068 ceph-mon[38805]: pgmap v3397: 17 pgs: 1 active+clean+scrubbing, 1 active+recovery_unfound+undersized+degraded, 4 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:47:12.529 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5626 8 365281 365273 0.507195 0 - 0.0299513 2026-02-23T14:47:12.706 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:47:12.824 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:47:12.929 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:47:12.927+0000 7f765e3a1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:47:12.930 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:47:12.927+0000 7f765e3a1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:47:13.036 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:47:13.036 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":267,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:46:34.758585+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:45:46.188265+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:43:36.356556+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:44:40.015825+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:47:13.037 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:47:12 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T14:47:13.037 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:47:12 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T14:47:13.529 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5627 8 365281 365273 0.507104 0 - 0.0299513 2026-02-23T14:47:14.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:47:13 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:47:13] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:47:14.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:47:13 trial068 ceph-mon[38805]: pgmap v3398: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:47:14.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:47:13 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3316394020' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:47:14.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:47:13 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:47:13.972Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:47:14.530 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5628 8 365281 365273 0.507014 0 - 0.0299513 2026-02-23T14:47:15.530 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5629 8 365281 365273 0.506924 0 - 0.0299513 2026-02-23T14:47:16.074 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:47:16.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:47:15 trial068 ceph-mon[38805]: pgmap v3399: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:47:16.194 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:47:16.291 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:47:16.289+0000 7fa63c2b0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:47:16.291 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:47:16.289+0000 7fa63c2b0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:47:16.400 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:47:16.400 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":267,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:46:34.758585+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:45:46.188265+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:43:36.356556+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:44:40.015825+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:47:16.530 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5630 8 365281 365273 0.506834 0 - 0.0299513 2026-02-23T14:47:17.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:47:16 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/792782522' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:47:17.530 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5631 8 365281 365273 0.506744 0 - 0.0299513 2026-02-23T14:47:18.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:47:17 trial068 ceph-mon[38805]: pgmap v3400: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:47:18.530 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5632 8 365281 365273 0.506654 0 - 0.0299513 2026-02-23T14:47:19.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:47:18 trial068 ceph-mon[38805]: 5.8 scrub starts 2026-02-23T14:47:19.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:47:18 trial068 ceph-mon[38805]: 5.8 scrub ok 2026-02-23T14:47:19.437 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:47:19.530 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5633 8 365281 365273 0.506564 0 - 0.0299513 2026-02-23T14:47:19.555 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:47:19.663 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:47:19.661+0000 7f2910b5b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:47:19.663 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:47:19.661+0000 7f2910b5b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:47:19.772 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:47:19.772 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":267,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:46:34.758585+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:45:46.188265+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:43:36.356556+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:44:40.015825+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:47:20.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:47:19 trial068 ceph-mon[38805]: pgmap v3401: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:47:20.530 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5634 8 365281 365273 0.506474 0 - 0.0299513 2026-02-23T14:47:21.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:47:20 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3102520029' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:47:21.530 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5635 8 365281 365273 0.506384 0 - 0.0299513 2026-02-23T14:47:22.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:47:21 trial068 ceph-mon[38805]: pgmap v3402: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:47:22.530 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5636 8 365281 365273 0.506295 0 - 0.0299513 2026-02-23T14:47:22.813 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:47:22.931 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:47:23.034 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:47:23.032+0000 7f4079164640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:47:23.034 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:47:23.032+0000 7f4079164640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:47:23.145 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:47:23.145 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":267,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:46:34.758585+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:45:46.188265+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:43:36.356556+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:44:40.015825+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:47:23.530 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5637 8 365281 365273 0.506205 0 - 0.0299513 2026-02-23T14:47:23.974 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:47:23 trial068 ceph-mon[38805]: pgmap v3403: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:47:23.974 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:47:23 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1488766009' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:47:24.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:47:23 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:47:23] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:47:24.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:47:23 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:47:23.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:47:24.530 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5638 8 365281 365273 0.506115 0 - 0.0299513 2026-02-23T14:47:25.530 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5639 8 365281 365273 0.506025 0 - 0.0299513 2026-02-23T14:47:26.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:47:25 trial068 ceph-mon[38805]: pgmap v3404: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:47:26.183 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:47:26.301 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:47:26.399 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:47:26.397+0000 7f4092c2a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:47:26.399 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:47:26.397+0000 7f4092c2a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:47:26.508 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:47:26.508 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":267,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:46:34.758585+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:45:46.188265+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:43:36.356556+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:44:40.015825+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:47:26.531 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T14:47:26.530165+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T14:47:26.531 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T14:47:26.531 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5640 8 365281 365273 0.505936 0 - 0.0299513 2026-02-23T14:47:27.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:47:26 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2225483016' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:47:27.531 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5641 8 365281 365273 0.505846 0 - 0.0299513 2026-02-23T14:47:28.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:47:27 trial068 ceph-mon[38805]: pgmap v3405: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:47:28.531 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5642 8 365281 365273 0.505756 0 - 0.0299513 2026-02-23T14:47:29.531 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5643 8 365281 365273 0.505667 0 - 0.0299513 2026-02-23T14:47:29.546 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:47:29.665 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:47:29.764 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:47:29.762+0000 7f2989a2e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:47:29.764 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:47:29.762+0000 7f2989a2e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:47:29.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:47:29 trial068 ceph-mon[38805]: pgmap v3406: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:47:29.873 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:47:29.873 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":267,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:46:34.758585+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:45:46.188265+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:43:36.356556+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:44:40.015825+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:47:30.531 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5644 8 365281 365273 0.505577 0 - 0.0299513 2026-02-23T14:47:31.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:47:30 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3909001423' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:47:31.531 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5645 8 365281 365273 0.505487 0 - 0.0299513 2026-02-23T14:47:32.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:47:31 trial068 ceph-mon[38805]: pgmap v3407: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:47:32.531 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5646 8 365281 365273 0.505398 0 - 0.0299513 2026-02-23T14:47:32.911 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:47:33.031 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:47:33.139 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:47:33.137+0000 7ff40ee8e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:47:33.139 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:47:33.137+0000 7ff40ee8e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:47:33.249 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:47:33.249 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":267,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:46:34.758585+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:45:46.188265+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:43:36.356556+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:44:40.015825+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:47:33.531 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5647 8 365281 365273 0.505308 0 - 0.0299513 2026-02-23T14:47:34.094 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:47:33 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:47:33] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:47:34.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:47:33 trial068 ceph-mon[38805]: pgmap v3408: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:47:34.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:47:33 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1484365136' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:47:34.266 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:47:33 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:47:33.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:47:34.531 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5648 8 365281 365273 0.505219 0 - 0.0299513 2026-02-23T14:47:35.531 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5649 8 365281 365273 0.505129 0 - 0.0299513 2026-02-23T14:47:36.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:47:35 trial068 ceph-mon[38805]: pgmap v3409: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:47:36.287 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:47:36.405 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:47:36.505 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:47:36.503+0000 7f647100a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:47:36.505 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:47:36.503+0000 7f647100a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:47:36.531 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5650 8 365281 365273 0.50504 0 - 0.0299513 2026-02-23T14:47:36.615 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:47:36.616 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":267,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:46:34.758585+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:45:46.188265+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:43:36.356556+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:44:40.015825+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:47:37.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:47:36 trial068 ceph-mon[38805]: 5.a scrub starts 2026-02-23T14:47:37.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:47:36 trial068 ceph-mon[38805]: 5.a scrub ok 2026-02-23T14:47:37.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:47:36 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2466829580' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:47:37.531 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5651 8 365281 365273 0.504951 0 - 0.0299513 2026-02-23T14:47:38.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:47:37 trial068 ceph-mon[38805]: pgmap v3410: 17 pgs: 1 active+clean+scrubbing, 1 active+recovery_unfound+undersized+degraded, 4 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:47:38.532 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5652 8 365281 365273 0.504861 0 - 0.0299513 2026-02-23T14:47:39.532 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5653 8 365281 365273 0.504772 0 - 0.0299513 2026-02-23T14:47:39.653 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:47:39.773 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:47:39.875 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:47:39.873+0000 7f83895ab640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:47:39.875 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:47:39.873+0000 7f83895ab640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:47:39.985 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:47:39.985 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":267,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:46:34.758585+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:45:46.188265+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:43:36.356556+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:44:40.015825+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:47:39.985 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:47:39 trial068 ceph-mon[38805]: pgmap v3411: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 1 active+clean+scrubbing, 4 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:47:40.532 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5654 8 365281 365273 0.504683 0 - 0.0299513 2026-02-23T14:47:41.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:47:40 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/709079229' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:47:41.532 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5655 8 365281 365273 0.504594 0 - 0.0299513 2026-02-23T14:47:42.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:47:41 trial068 ceph-mon[38805]: pgmap v3412: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 1 active+clean+scrubbing, 4 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:47:42.532 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5656 8 365281 365273 0.504504 0 - 0.0299513 2026-02-23T14:47:43.024 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:47:43.143 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:47:43.247 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:47:43.245+0000 7f64841b9640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:47:43.247 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:47:43.245+0000 7f64841b9640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:47:43.356 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:47:43.356 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":267,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:46:34.758585+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:45:46.188265+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:43:36.356556+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:44:40.015825+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:47:43.532 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5657 8 365281 365273 0.504415 0 - 0.0299513 2026-02-23T14:47:44.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:47:43 trial068 ceph-mon[38805]: pgmap v3413: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:47:44.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:47:43 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/318222197' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:47:44.094 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:47:43 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:47:43] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:47:44.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:47:43 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:47:43.974Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:47:44.532 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5658 8 365281 365273 0.504326 0 - 0.0299513 2026-02-23T14:47:45.532 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5659 8 365281 365273 0.504237 0 - 0.0299513 2026-02-23T14:47:46.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:47:45 trial068 ceph-mon[38805]: pgmap v3414: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:47:46.394 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:47:46.511 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:47:46.532 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T14:47:46.531953+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T14:47:46.532 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T14:47:46.532 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5660 8 365281 365273 0.504148 0 - 0.0299513 2026-02-23T14:47:46.609 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:47:46.607+0000 7fdf18198640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:47:46.610 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:47:46.608+0000 7fdf18198640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:47:46.719 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:47:46.719 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":267,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:46:34.758585+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:45:46.188265+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:43:36.356556+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:44:40.015825+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:47:47.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:47:46 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2590818482' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:47:47.532 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5661 8 365281 365273 0.504059 0 - 0.0299513 2026-02-23T14:47:48.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:47:47 trial068 ceph-mon[38805]: pgmap v3415: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:47:48.532 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5662 8 365281 365273 0.50397 0 - 0.0299513 2026-02-23T14:47:49.532 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5663 8 365281 365273 0.503881 0 - 0.0299513 2026-02-23T14:47:49.757 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:47:49.876 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:47:49.981 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:47:49.979+0000 7f2ff0182640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:47:49.981 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:47:49.979+0000 7f2ff0182640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:47:50.090 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:47:50.090 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":267,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:46:34.758585+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:45:46.188265+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:43:36.356556+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:44:40.015825+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:47:50.090 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:47:49 trial068 ceph-mon[38805]: pgmap v3416: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:47:50.532 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5664 8 365281 365273 0.503792 0 - 0.0299513 2026-02-23T14:47:51.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:47:50 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2608265743' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:47:51.532 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5665 8 365281 365273 0.503703 0 - 0.0299513 2026-02-23T14:47:52.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:47:51 trial068 ceph-mon[38805]: pgmap v3417: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:47:52.532 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5666 8 365281 365273 0.503614 0 - 0.0299513 2026-02-23T14:47:53.126 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:47:53.245 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:47:53.352 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:47:53.350+0000 7f9a05816640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:47:53.352 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:47:53.350+0000 7f9a05816640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:47:53.460 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:47:53.460 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":267,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:46:34.758585+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:45:46.188265+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:43:36.356556+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:44:40.015825+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:47:53.533 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5667 8 365281 365273 0.503525 0 - 0.0299513 2026-02-23T14:47:54.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:47:53 trial068 ceph-mon[38805]: pgmap v3418: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:47:54.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:47:53 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2666237143' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:47:54.094 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:47:53 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:47:53] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:47:54.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:47:53 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:47:53.974Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:47:54.533 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5668 8 365281 365273 0.503436 0 - 0.0299513 2026-02-23T14:47:55.533 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5669 8 365281 365273 0.503347 0 - 0.0299513 2026-02-23T14:47:56.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:47:55 trial068 ceph-mon[38805]: pgmap v3419: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:47:56.496 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:47:56.533 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5670 8 365281 365273 0.503259 0 - 0.0299513 2026-02-23T14:47:56.615 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:47:56.717 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:47:56.715+0000 7fdd3f4db640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:47:56.717 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:47:56.716+0000 7fdd3f4db640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:47:56.826 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:47:56.826 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":267,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:46:34.758585+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:45:46.188265+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:43:36.356556+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:44:40.015825+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:47:57.533 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5671 8 365281 365273 0.50317 0 - 0.0299513 2026-02-23T14:47:58.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:47:57 trial068 ceph-mon[38805]: pgmap v3420: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:47:58.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:47:57 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/4259377512' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:47:58.533 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5672 8 365281 365273 0.503081 0 - 0.0299513 2026-02-23T14:47:59.533 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5673 8 365281 365273 0.502993 0 - 0.0299513 2026-02-23T14:47:59.864 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:47:59.982 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:48:00.077 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:47:59 trial068 ceph-mon[38805]: pgmap v3421: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:48:00.078 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:48:00.075+0000 7fb96fb31640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:48:00.078 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:48:00.076+0000 7fb96fb31640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:48:00.187 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:48:00.187 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":267,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:46:34.758585+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:45:46.188265+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:43:36.356556+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:44:40.015825+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:48:00.533 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5674 8 365281 365273 0.502904 0 - 0.0299513 2026-02-23T14:48:01.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:48:00 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2611625265' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:48:01.533 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5675 8 365281 365273 0.502815 0 - 0.0299513 2026-02-23T14:48:02.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:48:01 trial068 ceph-mon[38805]: pgmap v3422: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:48:02.533 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5676 8 365281 365273 0.502727 0 - 0.0299513 2026-02-23T14:48:03.226 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:48:03.344 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:48:03.450 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:48:03.448+0000 7f0582e23640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:48:03.450 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:48:03.448+0000 7f0582e23640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:48:03.533 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5677 8 365281 365273 0.502638 0 - 0.0299513 2026-02-23T14:48:03.560 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:48:03.561 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":267,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:46:34.758585+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:45:46.188265+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:43:36.356556+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:44:40.015825+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:48:04.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:48:03 trial068 ceph-mon[38805]: pgmap v3423: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:48:04.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:48:03 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3583914483' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:48:04.094 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:48:03 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:48:03] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:48:04.266 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:48:03 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:48:03.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:48:04.533 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5678 8 365281 365273 0.50255 0 - 0.0299513 2026-02-23T14:48:05.533 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5679 8 365281 365273 0.502461 0 - 0.0299513 2026-02-23T14:48:06.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:48:05 trial068 ceph-mon[38805]: pgmap v3424: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:48:06.533 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T14:48:06.533369+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T14:48:06.534 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T14:48:06.534 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5680 8 365281 365273 0.502373 0 - 0.0299513 2026-02-23T14:48:06.599 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:48:06.725 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:48:06.834 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:48:06.832+0000 7f7dd2106640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:48:06.834 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:48:06.832+0000 7f7dd2106640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:48:06.942 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:48:06.942 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":267,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:46:34.758585+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:45:46.188265+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:43:36.356556+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:44:40.015825+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:48:07.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:48:06 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2026-02-23T14:48:07.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:48:06 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T14:48:07.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:48:06 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T14:48:07.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:48:06 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T14:48:07.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:48:06 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T14:48:07.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:48:06 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-23T14:48:07.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:48:06 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2026-02-23T14:48:07.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:48:06 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T14:48:07.095 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:48:06 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2026-02-23T14:48:07.534 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5681 8 365281 365273 0.502284 0 - 0.0299513 2026-02-23T14:48:08.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:48:07 trial068 ceph-mon[38805]: pgmap v3425: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:48:08.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:48:07 trial068 ceph-mon[38805]: 5.5 scrub starts 2026-02-23T14:48:08.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:48:07 trial068 ceph-mon[38805]: 5.5 scrub ok 2026-02-23T14:48:08.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:48:07 trial068 ceph-mon[38805]: pgmap v3426: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:48:08.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:48:07 trial068 ceph-mon[38805]: pgmap v3427: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:48:08.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:48:07 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3553255356' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:48:08.534 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5682 8 365281 365273 0.502196 0 - 0.0299513 2026-02-23T14:48:09.534 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5683 8 365281 365273 0.502107 0 - 0.0299513 2026-02-23T14:48:09.979 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:48:10.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:48:09 trial068 ceph-mon[38805]: pgmap v3428: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:48:10.097 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:48:10.194 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:48:10.193+0000 7fa932ad4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:48:10.195 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:48:10.193+0000 7fa932ad4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:48:10.305 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:48:10.305 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":267,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:46:34.758585+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:45:46.188265+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:43:36.356556+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:44:40.015825+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:48:10.534 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5684 8 365281 365273 0.502019 0 - 0.0299513 2026-02-23T14:48:11.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:48:10 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T14:48:11.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:48:10 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T14:48:11.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:48:10 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1710258313' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:48:11.534 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5685 8 365281 365273 0.501931 0 - 0.0299513 2026-02-23T14:48:12.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:48:11 trial068 ceph-mon[38805]: pgmap v3429: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:48:12.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:48:11 trial068 ceph-mon[38805]: 5.e scrub starts 2026-02-23T14:48:12.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:48:11 trial068 ceph-mon[38805]: 5.e scrub ok 2026-02-23T14:48:12.534 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5686 8 365281 365273 0.501843 0 - 0.0299513 2026-02-23T14:48:13.343 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:48:13.461 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:48:13.534 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5687 8 365281 365273 0.501754 0 - 0.0299513 2026-02-23T14:48:13.561 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:48:13.559+0000 7fbba7dbf640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:48:13.562 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:48:13.560+0000 7fbba7dbf640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:48:13.670 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:48:13.670 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":267,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:46:34.758585+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:45:46.188265+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:43:36.356556+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:44:40.015825+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:48:13.974 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:48:13 trial068 ceph-mon[38805]: pgmap v3430: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:48:13.975 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:48:13 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1479068610' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:48:14.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:48:13 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:48:13] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:48:14.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:48:13 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:48:13.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:48:14.534 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5688 8 365281 365273 0.501666 0 - 0.0299513 2026-02-23T14:48:15.534 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5689 8 365281 365273 0.501578 0 - 0.0299513 2026-02-23T14:48:16.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:48:15 trial068 ceph-mon[38805]: pgmap v3431: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:48:16.534 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5690 8 365281 365273 0.50149 0 - 0.0299513 2026-02-23T14:48:16.708 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:48:16.825 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:48:16.925 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:48:16.922+0000 7ff6bde8d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:48:16.925 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:48:16.923+0000 7ff6bde8d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:48:17.037 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:48:17.037 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":267,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:46:34.758585+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:45:46.188265+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:43:36.356556+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:44:40.015825+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:48:17.534 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5691 8 365281 365273 0.501402 0 - 0.0299513 2026-02-23T14:48:18.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:48:17 trial068 ceph-mon[38805]: pgmap v3432: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:48:18.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:48:17 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2574461745' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:48:18.534 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5692 8 365281 365273 0.501314 0 - 0.0299513 2026-02-23T14:48:19.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:48:18 trial068 ceph-mon[38805]: 5.7 scrub starts 2026-02-23T14:48:19.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:48:18 trial068 ceph-mon[38805]: 5.7 scrub ok 2026-02-23T14:48:19.534 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5693 8 365281 365273 0.501225 0 - 0.0299513 2026-02-23T14:48:20.075 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:48:20.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:48:19 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T14:48:20.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:48:19 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T14:48:20.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:48:19 trial068 ceph-mon[38805]: pgmap v3433: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:48:20.194 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:48:20.294 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:48:20.292+0000 7fdb84bbe640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:48:20.294 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:48:20.292+0000 7fdb84bbe640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:48:20.404 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:48:20.404 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":267,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:46:34.758585+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:45:46.188265+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:43:36.356556+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:44:40.015825+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:48:20.535 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5694 8 365281 365273 0.501137 0 - 0.0299513 2026-02-23T14:48:21.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:48:20 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3377952659' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:48:21.535 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5695 8 365281 365273 0.501049 0 - 0.0299513 2026-02-23T14:48:22.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:48:21 trial068 ceph-mon[38805]: pgmap v3434: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:48:22.535 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5696 8 365281 365273 0.500961 0 - 0.0299513 2026-02-23T14:48:23.442 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:48:23.535 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5697 8 365281 365273 0.500874 0 - 0.0299513 2026-02-23T14:48:23.561 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:48:23.656 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:48:23.654+0000 7f436eebd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:48:23.657 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:48:23.655+0000 7f436eebd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:48:23.766 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:48:23.766 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":267,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:46:34.758585+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:45:46.188265+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:43:36.356556+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:44:40.015825+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:48:24.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:48:23 trial068 ceph-mon[38805]: pgmap v3435: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:48:24.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:48:23 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1001186677' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:48:24.094 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:48:23 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:48:23] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:48:24.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:48:23 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:48:23.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:48:24.535 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5698 8 365281 365273 0.500786 0 - 0.0299513 2026-02-23T14:48:25.535 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5699 8 365281 365273 0.500698 0 - 0.0299513 2026-02-23T14:48:26.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:48:25 trial068 ceph-mon[38805]: pgmap v3436: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:48:26.535 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T14:48:26.534743+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T14:48:26.535 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T14:48:26.535 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5700 8 365281 365273 0.50061 0 - 0.0299513 2026-02-23T14:48:26.803 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:48:26.922 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:48:27.026 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:48:27.024+0000 7fdf884d2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:48:27.026 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:48:27.024+0000 7fdf884d2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:48:27.136 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:48:27.136 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":267,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:46:34.758585+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:45:46.188265+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:43:36.356556+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:44:40.015825+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:48:27.535 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5701 8 365281 365273 0.500522 0 - 0.0299513 2026-02-23T14:48:28.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:48:27 trial068 ceph-mon[38805]: pgmap v3437: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:48:28.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:48:27 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2994191437' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:48:28.535 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5702 8 365281 365273 0.500434 0 - 0.0299513 2026-02-23T14:48:29.535 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5703 8 365281 365273 0.500347 0 - 0.0299513 2026-02-23T14:48:30.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:48:29 trial068 ceph-mon[38805]: pgmap v3438: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:48:30.188 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:48:30.308 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:48:30.407 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:48:30.405+0000 7fe540960640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:48:30.407 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:48:30.405+0000 7fe540960640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:48:30.517 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:48:30.517 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":267,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:46:34.758585+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:45:46.188265+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:43:36.356556+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:44:40.015825+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:48:30.535 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5704 8 365281 365273 0.500259 0 - 0.0299513 2026-02-23T14:48:31.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:48:30 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/4091731969' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:48:31.535 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5705 8 365281 365273 0.500171 0 - 0.0299513 2026-02-23T14:48:32.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:48:31 trial068 ceph-mon[38805]: pgmap v3439: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:48:32.536 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5706 8 365281 365273 0.500084 0 - 0.0299513 2026-02-23T14:48:33.536 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5707 8 365281 365273 0.499996 0 - 0.0299513 2026-02-23T14:48:33.555 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:48:33.674 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:48:33.778 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:48:33.775+0000 7fa156598640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:48:33.778 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:48:33.776+0000 7fa156598640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:48:33.888 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:48:33.888 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":267,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:46:34.758585+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:45:46.188265+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:43:36.356556+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:44:40.015825+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:48:34.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:48:33 trial068 ceph-mon[38805]: pgmap v3440: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:48:34.094 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:48:33 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:48:33] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:48:34.264 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:48:33 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:48:33.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:48:34.536 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5708 8 365281 365273 0.499908 0 - 0.0299513 2026-02-23T14:48:35.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:48:34 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2275949630' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:48:35.536 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5709 8 365281 365273 0.499821 0 - 0.0299513 2026-02-23T14:48:36.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:48:35 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T14:48:36.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:48:35 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T14:48:36.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:48:35 trial068 ceph-mon[38805]: pgmap v3441: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:48:36.536 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5710 8 365281 365273 0.499733 0 - 0.0299513 2026-02-23T14:48:36.927 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:48:37.046 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:48:37.146 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:48:37.144+0000 7fc71fcfe640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:48:37.146 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:48:37.145+0000 7fc71fcfe640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:48:37.256 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:48:37.256 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":267,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:46:34.758585+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:45:46.188265+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:43:36.356556+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:44:40.015825+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:48:37.536 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5711 8 365281 365273 0.499646 0 - 0.0299513 2026-02-23T14:48:38.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:48:37 trial068 ceph-mon[38805]: pgmap v3442: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:48:38.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:48:37 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/4104757367' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:48:38.536 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5712 8 365281 365273 0.499558 0 - 0.0299513 2026-02-23T14:48:39.536 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5713 8 365281 365273 0.499471 0 - 0.0299513 2026-02-23T14:48:40.146 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:48:39 trial068 ceph-mon[38805]: pgmap v3443: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:48:40.296 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:48:40.413 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:48:40.512 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:48:40.510+0000 7fb6a6968640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:48:40.512 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:48:40.510+0000 7fb6a6968640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:48:40.536 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5714 8 365281 365273 0.499383 0 - 0.0299513 2026-02-23T14:48:40.623 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:48:40.623 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":267,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:46:34.758585+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:45:46.188265+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:43:36.356556+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:44:40.015825+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:48:41.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:48:40 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3405417616' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:48:41.536 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5715 8 365281 365273 0.499296 0 - 0.0299513 2026-02-23T14:48:42.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:48:41 trial068 ceph-mon[38805]: pgmap v3444: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:48:42.536 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5716 8 365281 365273 0.499209 0 - 0.0299513 2026-02-23T14:48:43.536 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5717 8 365281 365273 0.499121 0 - 0.0299513 2026-02-23T14:48:43.661 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:48:43.780 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:48:43.881 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:48:43.879+0000 7f96537b0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:48:43.881 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:48:43.879+0000 7f96537b0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:48:43.973 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:48:43 trial068 ceph-mon[38805]: pgmap v3445: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:48:43.990 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:48:43.990 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":267,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:46:34.758585+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:45:46.188265+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:43:36.356556+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:44:40.015825+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:48:44.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:48:43 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:48:43] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:48:44.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:48:43 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:48:43.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:48:44.536 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5718 8 365281 365273 0.499034 0 - 0.0299513 2026-02-23T14:48:45.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:48:44 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/4113737567' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:48:45.536 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5719 8 365281 365273 0.498947 0 - 0.0299513 2026-02-23T14:48:46.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:48:45 trial068 ceph-mon[38805]: pgmap v3446: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:48:46.537 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T14:48:46.536446+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T14:48:46.537 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T14:48:46.537 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5720 8 365281 365273 0.49886 0 - 0.0299513 2026-02-23T14:48:47.028 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:48:47.144 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:48:47.242 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:48:47.240+0000 7fea1174a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:48:47.242 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:48:47.240+0000 7fea1174a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:48:47.354 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:48:47.354 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":267,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:46:34.758585+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:45:46.188265+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:43:36.356556+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:44:40.015825+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:48:47.537 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5721 8 365281 365273 0.498772 0 - 0.0299513 2026-02-23T14:48:48.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:48:47 trial068 ceph-mon[38805]: pgmap v3447: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:48:48.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:48:47 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/258026522' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:48:48.537 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5722 8 365281 365273 0.498685 0 - 0.0299513 2026-02-23T14:48:49.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:48:48 trial068 ceph-mon[38805]: 5.a scrub starts 2026-02-23T14:48:49.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:48:48 trial068 ceph-mon[38805]: 5.a scrub ok 2026-02-23T14:48:49.537 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5723 8 365281 365273 0.498598 0 - 0.0299513 2026-02-23T14:48:50.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:48:49 trial068 ceph-mon[38805]: 5.8 scrub starts 2026-02-23T14:48:50.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:48:49 trial068 ceph-mon[38805]: pgmap v3448: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:48:50.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:48:49 trial068 ceph-mon[38805]: 5.8 scrub ok 2026-02-23T14:48:50.392 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:48:50.509 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:48:50.537 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5724 8 365281 365273 0.498511 0 - 0.0299513 2026-02-23T14:48:50.605 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:48:50.603+0000 7f2ecc080640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:48:50.605 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:48:50.603+0000 7f2ecc080640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:48:50.716 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:48:50.716 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":267,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:46:34.758585+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:45:46.188265+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:43:36.356556+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:44:40.015825+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:48:51.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:48:50 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/4017363464' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:48:51.537 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5725 8 365281 365273 0.498424 0 - 0.0299513 2026-02-23T14:48:52.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:48:51 trial068 ceph-mon[38805]: pgmap v3449: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:48:52.537 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5726 8 365281 365273 0.498337 0 - 0.0299513 2026-02-23T14:48:53.537 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5727 8 365281 365273 0.49825 0 - 0.0299513 2026-02-23T14:48:53.752 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:48:53.873 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:48:53.968 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:48:53.966+0000 7f584e400640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:48:53.968 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:48:53.966+0000 7f584e400640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:48:54.077 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:48:54.077 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":267,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:46:34.758585+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:45:46.188265+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:43:36.356556+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:44:40.015825+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:48:54.077 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:48:53 trial068 ceph-mon[38805]: pgmap v3450: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:48:54.078 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:48:53 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:48:53] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:48:54.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:48:53 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:48:53.972Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:48:54.537 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5728 8 365281 365273 0.498163 0 - 0.0299513 2026-02-23T14:48:55.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:48:54 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2236461424' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:48:55.537 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5729 8 365281 365273 0.498076 0 - 0.0299513 2026-02-23T14:48:56.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:48:55 trial068 ceph-mon[38805]: pgmap v3451: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:48:56.537 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5730 8 365281 365273 0.497989 0 - 0.0299513 2026-02-23T14:48:57.113 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:48:57.233 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:48:57.327 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:48:57.325+0000 7f385de8a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:48:57.328 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:48:57.326+0000 7f385de8a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:48:57.440 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:48:57.440 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":267,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:46:34.758585+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:45:46.188265+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:43:36.356556+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:44:40.015825+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:48:57.537 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5731 8 365281 365273 0.497902 0 - 0.0299513 2026-02-23T14:48:58.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:48:57 trial068 ceph-mon[38805]: pgmap v3452: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:48:58.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:48:57 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1917053419' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:48:58.538 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5732 8 365281 365273 0.497815 0 - 0.0299513 2026-02-23T14:48:59.538 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5733 8 365281 365273 0.497728 0 - 0.0299513 2026-02-23T14:49:00.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:48:59 trial068 ceph-mon[38805]: pgmap v3453: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:49:00.476 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:49:00.538 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5734 8 365281 365273 0.497642 0 - 0.0299513 2026-02-23T14:49:00.598 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:49:00.701 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:49:00.699+0000 7f055b6fb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:49:00.701 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:49:00.699+0000 7f055b6fb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:49:00.816 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:49:00.817 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":267,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:46:34.758585+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:45:46.188265+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:43:36.356556+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:44:40.015825+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:49:01.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:49:00 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2561605613' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:49:01.538 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5735 8 365281 365273 0.497555 0 - 0.0299513 2026-02-23T14:49:02.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:49:01 trial068 ceph-mon[38805]: pgmap v3454: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:49:02.538 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5736 8 365281 365273 0.497468 0 - 0.0299513 2026-02-23T14:49:03.538 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5737 8 365281 365273 0.497381 0 - 0.0299513 2026-02-23T14:49:03.854 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:49:03.971 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:49:04.069 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:49:04.067+0000 7fdfabde2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:49:04.070 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:49:04.068+0000 7fdfabde2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:49:04.178 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:49:04.178 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":267,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:46:34.758585+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:45:46.188265+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:43:36.356556+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:44:40.015825+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:49:04.179 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:49:03 trial068 ceph-mon[38805]: pgmap v3455: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:49:04.179 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:49:03 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:49:03] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:49:04.264 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:49:03 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:49:03.972Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:49:04.538 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5738 8 365281 365273 0.497295 0 - 0.0299513 2026-02-23T14:49:05.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:49:04 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1597860907' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:49:05.538 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5739 8 365281 365273 0.497208 0 - 0.0299513 2026-02-23T14:49:06.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:49:05 trial068 ceph-mon[38805]: pgmap v3456: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:49:06.538 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T14:49:06.537857+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T14:49:06.538 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T14:49:06.538 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5740 8 365281 365273 0.497121 0 - 0.0299513 2026-02-23T14:49:07.215 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:49:07.239 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:49:06 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2026-02-23T14:49:07.332 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:49:07.433 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:49:07.431+0000 7f8cbf8bc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:49:07.434 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:49:07.432+0000 7f8cbf8bc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:49:07.538 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5741 8 365281 365273 0.497035 0 - 0.0299513 2026-02-23T14:49:07.544 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:49:07.544 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":268,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:49:06.873080+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:45:46.188265+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:48:34.098389+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:44:40.015825+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:49:08.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:49:07 trial068 ceph-mon[38805]: pgmap v3457: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:49:08.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:49:07 trial068 ceph-mon[38805]: osdmap e268: 4 total, 3 up, 4 in 2026-02-23T14:49:08.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:49:07 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-23T14:49:08.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:49:07 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2026-02-23T14:49:08.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:49:07 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T14:49:08.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:49:07 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2026-02-23T14:49:08.345 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:49:07 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2897338327' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:49:08.538 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5742 8 365281 365273 0.496948 0 - 0.0299513 2026-02-23T14:49:09.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:49:08 trial068 ceph-mon[38805]: pgmap v3459: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:49:09.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:49:08 trial068 ceph-mon[38805]: pgmap v3460: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:49:09.538 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5743 8 365281 365273 0.496862 0 - 0.0299513 2026-02-23T14:49:10.538 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5744 8 365281 365273 0.496775 0 - 0.0299513 2026-02-23T14:49:10.582 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:49:10.699 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:49:10.803 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:49:10.801+0000 7fc005ee7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:49:10.803 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:49:10.801+0000 7fc005ee7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:49:10.912 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:49:10.912 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":269,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:49:09.781350+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:45:46.188265+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:48:34.098389+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:44:40.015825+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:49:10.913 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:49:10 trial068 ceph-mon[38805]: pgmap v3461: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:49:10.913 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:49:10 trial068 ceph-mon[38805]: osdmap e269: 4 total, 3 up, 4 in 2026-02-23T14:49:11.539 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5745 8 365281 365273 0.496689 0 - 0.0299513 2026-02-23T14:49:12.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:49:11 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/739316953' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:49:12.538 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5746 8 365281 365273 0.496602 0 - 0.0299513 2026-02-23T14:49:13.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:49:12 trial068 ceph-mon[38805]: pgmap v3463: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:49:13.539 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5747 8 365281 365273 0.496516 0 - 0.0299513 2026-02-23T14:49:13.948 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:49:14.067 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:49:14.088 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:49:13 trial068 ceph-mon[38805]: 5.5 scrub starts 2026-02-23T14:49:14.089 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:49:13 trial068 ceph-mon[38805]: 5.5 scrub ok 2026-02-23T14:49:14.089 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:49:13 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:49:13] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:49:14.163 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:49:14.161+0000 7fb4df5ac640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:49:14.163 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:49:14.161+0000 7fb4df5ac640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:49:14.272 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:49:14.273 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":269,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:49:09.781350+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:45:46.188265+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:48:34.098389+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:44:40.015825+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:49:14.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:49:13 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:49:13.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:49:14.539 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5748 8 365281 365273 0.496429 0 - 0.0299513 2026-02-23T14:49:15.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:49:14 trial068 ceph-mon[38805]: pgmap v3464: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:49:15.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:49:14 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3180418803' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:49:15.539 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5749 8 365281 365273 0.496343 0 - 0.0299513 2026-02-23T14:49:16.539 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5750 8 365281 365273 0.496257 0 - 0.0299513 2026-02-23T14:49:17.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:49:16 trial068 ceph-mon[38805]: pgmap v3465: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:49:17.310 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:49:17.428 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:49:17.528 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:49:17.526+0000 7fb799ada640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:49:17.529 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:49:17.527+0000 7fb799ada640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:49:17.539 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5751 8 365281 365273 0.496171 0 - 0.0299513 2026-02-23T14:49:17.636 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:49:17.636 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":269,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:49:09.781350+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:45:46.188265+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:48:34.098389+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:44:40.015825+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:49:18.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:49:17 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/4080759785' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:49:18.539 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5752 8 365281 365273 0.496084 0 - 0.0299513 2026-02-23T14:49:19.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:49:18 trial068 ceph-mon[38805]: pgmap v3466: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:49:19.539 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5753 8 365281 365273 0.495998 0 - 0.0299513 2026-02-23T14:49:20.539 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5754 8 365281 365273 0.495912 0 - 0.0299513 2026-02-23T14:49:20.673 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:49:20.791 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:49:20.887 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:49:20.885+0000 7fe7c5a1d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:49:20.888 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:49:20.886+0000 7fe7c5a1d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:49:20.999 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:49:20.999 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":269,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:49:09.781350+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:45:46.188265+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:48:34.098389+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:44:40.015825+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:49:20.999 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:49:20 trial068 ceph-mon[38805]: pgmap v3467: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:49:21.539 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5755 8 365281 365273 0.495826 0 - 0.0299513 2026-02-23T14:49:22.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:49:21 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1548348126' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:49:22.539 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5756 8 365281 365273 0.49574 0 - 0.0299513 2026-02-23T14:49:23.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:49:22 trial068 ceph-mon[38805]: pgmap v3468: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:49:23.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:49:22 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T14:49:23.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:49:22 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T14:49:23.539 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5757 8 365281 365273 0.495653 0 - 0.0299513 2026-02-23T14:49:24.036 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:49:24.153 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:49:24.250 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:49:23 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:49:23] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:49:24.252 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:49:24.250+0000 7fee6a05b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:49:24.252 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:49:24.250+0000 7fee6a05b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:49:24.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:49:23 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:49:23.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:49:24.364 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:49:24.365 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":269,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:49:09.781350+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:45:46.188265+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:48:34.098389+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:44:40.015825+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:49:24.539 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5758 8 365281 365273 0.495567 0 - 0.0299513 2026-02-23T14:49:25.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:49:24 trial068 ceph-mon[38805]: pgmap v3469: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:49:25.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:49:24 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/922153252' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:49:25.539 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5759 8 365281 365273 0.495481 0 - 0.0299513 2026-02-23T14:49:26.540 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T14:49:26.539311+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T14:49:26.540 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T14:49:26.540 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5760 8 365281 365273 0.495395 0 - 0.0299513 2026-02-23T14:49:27.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:49:26 trial068 ceph-mon[38805]: pgmap v3470: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:49:27.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:49:26 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T14:49:27.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:49:26 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T14:49:27.401 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:49:27.522 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:49:27.539 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5761 8 365281 365273 0.495309 0 - 0.0299513 2026-02-23T14:49:27.631 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:49:27.629+0000 7f975eb20640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:49:27.631 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:49:27.629+0000 7f975eb20640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:49:27.740 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:49:27.740 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":269,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:49:09.781350+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:45:46.188265+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:48:34.098389+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:44:40.015825+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:49:28.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:49:27 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/385392144' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:49:28.540 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5762 8 365281 365273 0.495223 0 - 0.0299513 2026-02-23T14:49:29.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:49:28 trial068 ceph-mon[38805]: pgmap v3471: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:49:29.540 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5763 8 365281 365273 0.495137 0 - 0.0299513 2026-02-23T14:49:30.540 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5764 8 365281 365273 0.495051 0 - 0.0299513 2026-02-23T14:49:30.778 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:49:30.898 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:49:30.997 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:49:30.995+0000 7f7069b78640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:49:30.997 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:49:30.995+0000 7f7069b78640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:49:31.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:49:30 trial068 ceph-mon[38805]: pgmap v3472: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:49:31.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:49:30 trial068 ceph-mon[38805]: 5.7 scrub starts 2026-02-23T14:49:31.106 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:49:31.106 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":269,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:49:09.781350+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:45:46.188265+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:48:34.098389+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:44:40.015825+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:49:31.540 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5765 8 365281 365273 0.494966 0 - 0.0299513 2026-02-23T14:49:32.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:49:31 trial068 ceph-mon[38805]: 5.7 scrub ok 2026-02-23T14:49:32.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:49:31 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/775278394' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:49:32.540 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5766 8 365281 365273 0.49488 0 - 0.0299513 2026-02-23T14:49:33.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:49:32 trial068 ceph-mon[38805]: pgmap v3473: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:49:33.540 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5767 8 365281 365273 0.494794 0 - 0.0299513 2026-02-23T14:49:34.142 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:49:34.263 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:49:34.266 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:49:33 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:49:33.974Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:49:34.285 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:49:33 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:49:33] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:49:34.366 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:49:34.364+0000 7f2f11936640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:49:34.367 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:49:34.365+0000 7f2f11936640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:49:34.477 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:49:34.477 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":269,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:49:09.781350+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:45:46.188265+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:48:34.098389+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:44:40.015825+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:49:34.540 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5768 8 365281 365273 0.494708 0 - 0.0299513 2026-02-23T14:49:35.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:49:34 trial068 ceph-mon[38805]: pgmap v3474: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:49:35.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:49:34 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2919779859' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:49:35.540 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5769 8 365281 365273 0.494622 0 - 0.0299513 2026-02-23T14:49:36.540 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5770 8 365281 365273 0.494537 0 - 0.0299513 2026-02-23T14:49:37.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:49:36 trial068 ceph-mon[38805]: pgmap v3475: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:49:37.515 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:49:37.540 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5771 8 365281 365273 0.494451 0 - 0.0299513 2026-02-23T14:49:37.634 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:49:37.736 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:49:37.734+0000 7fe3542a4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:49:37.736 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:49:37.734+0000 7fe3542a4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:49:37.844 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:49:37.844 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":269,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:49:09.781350+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:45:46.188265+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:48:34.098389+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:44:40.015825+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:49:38.541 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5772 8 365281 365273 0.494365 0 - 0.0299513 2026-02-23T14:49:39.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:49:38 trial068 ceph-mon[38805]: pgmap v3476: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:49:39.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:49:38 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3674982220' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:49:39.541 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5773 8 365281 365273 0.49428 0 - 0.0299513 2026-02-23T14:49:40.541 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5774 8 365281 365273 0.494194 0 - 0.0299513 2026-02-23T14:49:40.882 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:49:41.002 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:49:41.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:49:40 trial068 ceph-mon[38805]: pgmap v3477: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:49:41.101 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:49:41.099+0000 7f44eb9bf640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:49:41.101 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:49:41.099+0000 7f44eb9bf640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:49:41.215 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:49:41.215 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":269,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:49:09.781350+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:45:46.188265+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:48:34.098389+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:44:40.015825+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:49:41.541 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5775 8 365281 365273 0.494109 0 - 0.0299513 2026-02-23T14:49:42.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:49:41 trial068 ceph-mon[38805]: 5.e scrub starts 2026-02-23T14:49:42.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:49:41 trial068 ceph-mon[38805]: 5.e scrub ok 2026-02-23T14:49:42.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:49:41 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/4184717568' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:49:42.541 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5776 8 365281 365273 0.494023 0 - 0.0299513 2026-02-23T14:49:43.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:49:42 trial068 ceph-mon[38805]: pgmap v3478: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:49:43.541 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5777 8 365281 365273 0.493937 0 - 0.0299513 2026-02-23T14:49:44.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:49:43 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T14:49:44.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:49:43 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T14:49:44.094 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:49:43 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:49:43] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:49:44.255 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:49:44.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:49:43 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:49:43.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:49:44.372 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:49:44.473 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:49:44.472+0000 7ff1b7c4d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:49:44.474 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:49:44.472+0000 7ff1b7c4d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:49:44.541 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5778 8 365281 365273 0.493852 0 - 0.0299513 2026-02-23T14:49:44.584 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:49:44.584 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":269,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:49:09.781350+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:45:46.188265+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:48:34.098389+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:44:40.015825+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:49:44.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:49:44 trial068 ceph-mon[38805]: pgmap v3479: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:49:44.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:49:44 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/822396087' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:49:45.541 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5779 8 365281 365273 0.493767 0 - 0.0299513 2026-02-23T14:49:46.541 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T14:49:46.540859+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T14:49:46.541 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T14:49:46.541 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5780 8 365281 365273 0.493681 0 - 0.0299513 2026-02-23T14:49:47.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:49:46 trial068 ceph-mon[38805]: pgmap v3480: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:49:47.541 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5781 8 365281 365273 0.493596 0 - 0.0299513 2026-02-23T14:49:47.623 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:49:47.740 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:49:47.840 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:49:47.838+0000 7f0718c49640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:49:47.840 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:49:47.838+0000 7f0718c49640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:49:47.950 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:49:47.950 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":269,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:49:09.781350+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:45:46.188265+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:48:34.098389+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:44:40.015825+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:49:48.541 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5782 8 365281 365273 0.49351 0 - 0.0299513 2026-02-23T14:49:49.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:49:48 trial068 ceph-mon[38805]: pgmap v3481: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:49:49.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:49:48 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/17691902' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:49:49.541 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5783 8 365281 365273 0.493425 0 - 0.0299513 2026-02-23T14:49:50.541 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5784 8 365281 365273 0.49334 0 - 0.0299513 2026-02-23T14:49:50.987 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:49:51.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:49:50 trial068 ceph-mon[38805]: pgmap v3482: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:49:51.105 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:49:51.208 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:49:51.206+0000 7f59c876c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:49:51.208 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:49:51.206+0000 7f59c876c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:49:51.318 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:49:51.318 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":269,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:49:09.781350+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:45:46.188265+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:48:34.098389+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:44:40.015825+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:49:51.541 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5785 8 365281 365273 0.493254 0 - 0.0299513 2026-02-23T14:49:52.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:49:51 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3444664095' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:49:52.542 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5786 8 365281 365273 0.493169 0 - 0.0299513 2026-02-23T14:49:53.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:49:52 trial068 ceph-mon[38805]: pgmap v3483: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:49:53.542 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5787 8 365281 365273 0.493084 0 - 0.0299513 2026-02-23T14:49:54.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:49:53 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:49:53] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:49:54.354 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:49:54.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:49:53 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:49:53.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:49:54.472 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:49:54.542 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5788 8 365281 365273 0.492999 0 - 0.0299513 2026-02-23T14:49:54.573 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:49:54.571+0000 7f72f6e25640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:49:54.573 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:49:54.571+0000 7f72f6e25640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:49:54.681 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:49:54.681 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":269,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:49:09.781350+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:45:46.188265+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:48:34.098389+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:44:40.015825+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:49:55.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:49:54 trial068 ceph-mon[38805]: pgmap v3484: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:49:55.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:49:54 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3634711290' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:49:55.542 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5789 8 365281 365273 0.492914 0 - 0.0299513 2026-02-23T14:49:56.542 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5790 8 365281 365273 0.492828 0 - 0.0299513 2026-02-23T14:49:57.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:49:56 trial068 ceph-mon[38805]: pgmap v3485: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:49:57.542 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5791 8 365281 365273 0.492743 0 - 0.0299513 2026-02-23T14:49:57.720 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:49:57.839 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:49:57.937 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:49:57.935+0000 7fcd5135e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:49:57.937 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:49:57.935+0000 7fcd5135e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:49:58.046 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:49:58.047 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":269,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:49:09.781350+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:45:46.188265+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:48:34.098389+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:44:40.015825+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:49:58.047 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:49:57 trial068 ceph-mon[38805]: 5.8 scrub starts 2026-02-23T14:49:58.542 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5792 8 365281 365273 0.492658 0 - 0.0299513 2026-02-23T14:49:59.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:49:58 trial068 ceph-mon[38805]: pgmap v3486: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:49:59.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:49:58 trial068 ceph-mon[38805]: 5.8 scrub ok 2026-02-23T14:49:59.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:49:58 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1192378572' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:49:59.542 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5793 8 365281 365273 0.492573 0 - 0.0299513 2026-02-23T14:50:00.542 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5794 8 365281 365273 0.492488 0 - 0.0299513 2026-02-23T14:50:01.083 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:50:01.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:50:00 trial068 ceph-mon[38805]: pgmap v3487: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:50:01.200 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:50:01.302 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:50:01.300+0000 7f286ccd4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:50:01.302 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:50:01.300+0000 7f286ccd4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:50:01.413 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:50:01.413 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":269,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:49:09.781350+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:45:46.188265+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:48:34.098389+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:44:40.015825+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:50:01.542 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5795 8 365281 365273 0.492403 0 - 0.0299513 2026-02-23T14:50:02.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:50:01 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2220112743' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:50:02.542 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5796 8 365281 365273 0.492318 0 - 0.0299513 2026-02-23T14:50:03.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:50:02 trial068 ceph-mon[38805]: pgmap v3488: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:50:03.542 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5797 8 365281 365273 0.492233 0 - 0.0299513 2026-02-23T14:50:04.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:50:03 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:50:03] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:50:04.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:50:03 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:50:03.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:50:04.451 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:50:04.542 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5798 8 365281 365273 0.492148 0 - 0.0299513 2026-02-23T14:50:04.570 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:50:04.673 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:50:04.671+0000 7f58b7700640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:50:04.673 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:50:04.671+0000 7f58b7700640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:50:04.782 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:50:04.782 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":269,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:49:09.781350+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:45:46.188265+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:48:34.098389+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:44:40.015825+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:50:05.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:50:04 trial068 ceph-mon[38805]: pgmap v3489: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:50:05.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:50:04 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/249717765' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:50:05.543 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5799 8 365281 365273 0.492064 0 - 0.0299513 2026-02-23T14:50:06.543 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T14:50:06.542432+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T14:50:06.543 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T14:50:06.543 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5800 8 365281 365273 0.491979 0 - 0.0299513 2026-02-23T14:50:07.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:50:06 trial068 ceph-mon[38805]: pgmap v3490: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:50:07.543 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5801 8 365281 365273 0.491894 0 - 0.0299513 2026-02-23T14:50:07.818 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:50:07.935 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:50:08.035 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:50:08.032+0000 7f1f36206640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:50:08.035 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:50:08.032+0000 7f1f36206640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:50:08.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:50:07 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2026-02-23T14:50:08.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:50:07 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-23T14:50:08.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:50:07 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2026-02-23T14:50:08.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:50:07 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T14:50:08.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:50:07 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2026-02-23T14:50:08.143 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:50:08.143 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":269,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:49:09.781350+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:45:46.188265+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:48:34.098389+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:44:40.015825+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:50:08.543 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5802 8 365281 365273 0.491809 0 - 0.0299513 2026-02-23T14:50:09.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:50:08 trial068 ceph-mon[38805]: pgmap v3491: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:50:09.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:50:08 trial068 ceph-mon[38805]: pgmap v3492: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:50:09.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:50:08 trial068 ceph-mon[38805]: pgmap v3493: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:50:09.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:50:08 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3700820188' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:50:09.543 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5803 8 365281 365273 0.491724 0 - 0.0299513 2026-02-23T14:50:10.543 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5804 8 365281 365273 0.49164 0 - 0.0299513 2026-02-23T14:50:11.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:50:10 trial068 ceph-mon[38805]: pgmap v3494: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:50:11.182 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:50:11.301 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:50:11.401 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:50:11.399+0000 7f112eee0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:50:11.401 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:50:11.399+0000 7f112eee0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:50:11.509 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:50:11.509 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":269,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:49:09.781350+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:45:46.188265+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:48:34.098389+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:44:40.015825+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:50:11.543 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5805 8 365281 365273 0.491555 0 - 0.0299513 2026-02-23T14:50:12.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:50:11 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/880413305' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:50:12.543 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5806 8 365281 365273 0.49147 0 - 0.0299513 2026-02-23T14:50:13.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:50:12 trial068 ceph-mon[38805]: pgmap v3495: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:50:13.543 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5807 8 365281 365273 0.491386 0 - 0.0299513 2026-02-23T14:50:14.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:50:13 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:50:13] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:50:14.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:50:13 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:50:13.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:50:14.543 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5808 8 365281 365273 0.491301 0 - 0.0299513 2026-02-23T14:50:14.547 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:50:14.663 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:50:14.761 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:50:14.759+0000 7fcb229c2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:50:14.762 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:50:14.760+0000 7fcb229c2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:50:14.870 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:50:14.870 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":269,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:49:09.781350+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:45:46.188265+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:48:34.098389+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:44:40.015825+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:50:15.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:50:14 trial068 ceph-mon[38805]: pgmap v3496: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:50:15.543 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5809 8 365281 365273 0.491216 0 - 0.0299513 2026-02-23T14:50:16.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:50:15 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1483320905' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:50:16.543 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5810 8 365281 365273 0.491132 0 - 0.0299513 2026-02-23T14:50:17.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:50:16 trial068 ceph-mon[38805]: pgmap v3497: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:50:17.543 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5811 8 365281 365273 0.491047 0 - 0.0299513 2026-02-23T14:50:17.907 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:50:18.024 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:50:18.120 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:50:18.119+0000 7fa3b6ef1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:50:18.121 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:50:17 trial068 ceph-mon[38805]: 5.a scrub starts 2026-02-23T14:50:18.121 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:50:18.119+0000 7fa3b6ef1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:50:18.230 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:50:18.230 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":269,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:49:09.781350+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:45:46.188265+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:48:34.098389+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:44:40.015825+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:50:18.543 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5812 8 365281 365273 0.490963 0 - 0.0299513 2026-02-23T14:50:19.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:50:18 trial068 ceph-mon[38805]: 5.a scrub ok 2026-02-23T14:50:19.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:50:18 trial068 ceph-mon[38805]: pgmap v3498: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:50:19.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:50:18 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1239418617' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:50:19.544 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5813 8 365281 365273 0.490878 0 - 0.0299513 2026-02-23T14:50:20.544 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5814 8 365281 365273 0.490794 0 - 0.0299513 2026-02-23T14:50:21.284 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:50:21.309 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:50:20 trial068 ceph-mon[38805]: pgmap v3499: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:50:21.403 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:50:21.502 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:50:21.500+0000 7f94379e3640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:50:21.502 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:50:21.500+0000 7f94379e3640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:50:21.544 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5815 8 365281 365273 0.49071 0 - 0.0299513 2026-02-23T14:50:21.611 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:50:21.611 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":269,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:49:09.781350+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:45:46.188265+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:48:34.098389+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:44:40.015825+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:50:22.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:50:21 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2913426896' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:50:22.544 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5816 8 365281 365273 0.490625 0 - 0.0299513 2026-02-23T14:50:23.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:50:22 trial068 ceph-mon[38805]: pgmap v3500: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:50:23.544 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5817 8 365281 365273 0.490541 0 - 0.0299513 2026-02-23T14:50:24.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:50:23 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:50:23] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:50:24.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:50:23 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:50:23.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:50:24.544 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5818 8 365281 365273 0.490457 0 - 0.0299513 2026-02-23T14:50:24.649 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:50:24.768 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:50:24.867 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:50:24.865+0000 7f342958e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:50:24.867 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:50:24.865+0000 7f342958e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:50:24.977 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:50:24.978 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":271,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:50:24.791727+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:45:46.188265+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:48:34.098389+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:49:22.169310+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:50:24.978 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:50:24 trial068 ceph-mon[38805]: pgmap v3501: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:50:24.978 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:50:24 trial068 ceph-mon[38805]: osdmap e270: 4 total, 3 up, 4 in 2026-02-23T14:50:24.978 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:50:24 trial068 ceph-mon[38805]: osdmap e271: 4 total, 3 up, 4 in 2026-02-23T14:50:25.544 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5819 8 365281 365273 0.490372 0 - 0.0299513 2026-02-23T14:50:26.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:50:25 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/292813757' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:50:26.544 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T14:50:26.543975+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T14:50:26.544 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T14:50:26.545 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5820 8 365281 365273 0.490288 0 - 0.0299513 2026-02-23T14:50:27.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:50:26 trial068 ceph-mon[38805]: pgmap v3504: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:50:27.544 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5821 8 365281 365273 0.490204 0 - 0.0299513 2026-02-23T14:50:28.015 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:50:28.133 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:50:28.155 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:50:27 trial068 ceph-mon[38805]: 5.5 scrub starts 2026-02-23T14:50:28.155 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:50:27 trial068 ceph-mon[38805]: 5.5 scrub ok 2026-02-23T14:50:28.231 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:50:28.229+0000 7f8ed5504640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:50:28.231 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:50:28.229+0000 7f8ed5504640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:50:28.340 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:50:28.340 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":271,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:50:24.791727+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:45:46.188265+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:48:34.098389+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:49:22.169310+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:50:28.544 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5822 8 365281 365273 0.49012 0 - 0.0299513 2026-02-23T14:50:29.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:50:28 trial068 ceph-mon[38805]: pgmap v3505: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:50:29.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:50:28 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2807488700' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:50:29.544 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5823 8 365281 365273 0.490035 0 - 0.0299513 2026-02-23T14:50:30.545 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5824 8 365281 365273 0.489951 0 - 0.0299513 2026-02-23T14:50:31.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:50:30 trial068 ceph-mon[38805]: pgmap v3506: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:50:31.377 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:50:31.493 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:50:31.544 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5825 8 365281 365273 0.489867 0 - 0.0299513 2026-02-23T14:50:31.591 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:50:31.590+0000 7efe9894a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:50:31.592 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:50:31.590+0000 7efe9894a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:50:31.700 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:50:31.700 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":271,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:50:24.791727+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:45:46.188265+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:48:34.098389+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:49:22.169310+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:50:32.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:50:31 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3389728976' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:50:32.545 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5826 8 365281 365273 0.489783 0 - 0.0299513 2026-02-23T14:50:33.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:50:32 trial068 ceph-mon[38805]: pgmap v3507: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:50:33.545 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5827 8 365281 365273 0.489699 0 - 0.0299513 2026-02-23T14:50:34.266 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:50:33 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:50:33.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:50:34.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:50:33 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T14:50:34.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:50:33 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T14:50:34.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:50:33 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:50:33] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:50:34.545 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5828 8 365281 365273 0.489615 0 - 0.0299513 2026-02-23T14:50:34.737 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:50:34.855 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:50:34.958 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:50:34.955+0000 7fa1d513a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:50:34.958 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:50:34.956+0000 7fa1d513a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:50:35.065 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:50:35.066 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":271,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:50:24.791727+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:45:46.188265+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:48:34.098389+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:49:22.169310+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:50:35.066 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:50:34 trial068 ceph-mon[38805]: pgmap v3508: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:50:35.545 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5829 8 365281 365273 0.489531 0 - 0.0299513 2026-02-23T14:50:36.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:50:35 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3946260463' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:50:36.545 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5830 8 365281 365273 0.489447 0 - 0.0299513 2026-02-23T14:50:37.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:50:36 trial068 ceph-mon[38805]: pgmap v3509: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:50:37.545 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5831 8 365281 365273 0.489363 0 - 0.0299513 2026-02-23T14:50:38.105 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:50:38.129 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:50:37 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T14:50:38.130 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:50:37 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T14:50:38.224 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:50:38.319 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:50:38.317+0000 7f6c1b36d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:50:38.320 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:50:38.318+0000 7f6c1b36d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:50:38.429 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:50:38.430 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":271,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:50:24.791727+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:45:46.188265+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:48:34.098389+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:49:22.169310+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:50:38.545 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5832 8 365281 365273 0.489279 0 - 0.0299513 2026-02-23T14:50:39.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:50:38 trial068 ceph-mon[38805]: pgmap v3510: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:50:39.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:50:38 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2059097072' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:50:39.545 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5833 8 365281 365273 0.489195 0 - 0.0299513 2026-02-23T14:50:40.546 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5834 8 365281 365273 0.489111 0 - 0.0299513 2026-02-23T14:50:41.145 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:50:40 trial068 ceph-mon[38805]: pgmap v3511: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:50:41.468 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:50:41.546 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5835 8 365281 365273 0.489028 0 - 0.0299513 2026-02-23T14:50:41.586 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:50:41.686 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:50:41.684+0000 7f67d45bd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:50:41.686 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:50:41.684+0000 7f67d45bd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:50:41.794 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:50:41.794 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":271,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:50:24.791727+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:45:46.188265+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:48:34.098389+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:49:22.169310+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:50:42.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:50:41 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2593567025' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:50:42.546 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5836 8 365281 365273 0.488944 0 - 0.0299513 2026-02-23T14:50:43.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:50:42 trial068 ceph-mon[38805]: pgmap v3512: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:50:43.546 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5837 8 365281 365273 0.48886 0 - 0.0299513 2026-02-23T14:50:44.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:50:43 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:50:43] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:50:44.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:50:43 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:50:43.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:50:44.546 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5838 8 365281 365273 0.488776 0 - 0.0299513 2026-02-23T14:50:44.831 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:50:44.949 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:50:45.050 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:50:45.048+0000 7f8fd86cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:50:45.051 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:50:45.049+0000 7f8fd86cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:50:45.162 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:50:45.162 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":271,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:50:24.791727+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:45:46.188265+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:48:34.098389+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:49:22.169310+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:50:45.162 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:50:44 trial068 ceph-mon[38805]: pgmap v3513: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:50:45.546 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5839 8 365281 365273 0.488693 0 - 0.0299513 2026-02-23T14:50:46.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:50:45 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2549664683' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:50:46.546 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T14:50:46.545823+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T14:50:46.546 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T14:50:46.546 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5840 8 365281 365273 0.488609 0 - 0.0299513 2026-02-23T14:50:47.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:50:46 trial068 ceph-mon[38805]: pgmap v3514: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:50:47.546 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5841 8 365281 365273 0.488525 0 - 0.0299513 2026-02-23T14:50:48.200 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:50:48.317 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:50:48.420 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:50:48.418+0000 7f3b5559b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:50:48.420 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:50:48.418+0000 7f3b5559b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:50:48.531 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:50:48.531 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":271,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:50:24.791727+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:45:46.188265+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:48:34.098389+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:49:22.169310+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:50:48.546 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5842 8 365281 365273 0.488442 0 - 0.0299513 2026-02-23T14:50:49.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:50:48 trial068 ceph-mon[38805]: pgmap v3515: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:50:49.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:50:48 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/4110592352' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:50:49.546 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5843 8 365281 365273 0.488358 0 - 0.0299513 2026-02-23T14:50:50.546 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5844 8 365281 365273 0.488275 0 - 0.0299513 2026-02-23T14:50:51.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:50:50 trial068 ceph-mon[38805]: pgmap v3516: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:50:51.546 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5845 8 365281 365273 0.488191 0 - 0.0299513 2026-02-23T14:50:51.568 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:50:51.685 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:50:51.785 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:50:51.783+0000 7fc082156640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:50:51.785 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:50:51.783+0000 7fc082156640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:50:51.896 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:50:51.896 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":271,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:50:24.791727+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:45:46.188265+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:48:34.098389+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:49:22.169310+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:50:52.547 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5846 8 365281 365273 0.488108 0 - 0.0299513 2026-02-23T14:50:53.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:50:52 trial068 ceph-mon[38805]: pgmap v3517: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:50:53.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:50:52 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3886619680' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:50:53.546 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5847 8 365281 365273 0.488024 0 - 0.0299513 2026-02-23T14:50:54.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:50:53 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:50:53] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:50:54.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:50:53 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:50:53.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:50:54.546 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5848 8 365281 365273 0.487941 0 - 0.0299513 2026-02-23T14:50:54.935 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:50:55.056 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:50:55.157 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:50:55.155+0000 7f4c36a80640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:50:55.157 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:50:55.155+0000 7f4c36a80640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:50:55.157 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:50:54 trial068 ceph-mon[38805]: pgmap v3518: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:50:55.267 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:50:55.267 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":271,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:50:24.791727+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:45:46.188265+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:48:34.098389+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:49:22.169310+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:50:55.547 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5849 8 365281 365273 0.487857 0 - 0.0299513 2026-02-23T14:50:56.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:50:55 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3341510065' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:50:56.547 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5850 8 365281 365273 0.487774 0 - 0.0299513 2026-02-23T14:50:57.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:50:56 trial068 ceph-mon[38805]: pgmap v3519: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:50:57.547 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5851 8 365281 365273 0.48769 0 - 0.0299513 2026-02-23T14:50:58.304 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:50:58.422 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:50:58.524 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:50:58.522+0000 7f298978f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:50:58.525 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:50:58.523+0000 7f298978f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:50:58.547 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5852 8 365281 365273 0.487607 0 - 0.0299513 2026-02-23T14:50:58.635 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:50:58.635 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":271,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:50:24.791727+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:45:46.188265+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:48:34.098389+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:49:22.169310+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:50:59.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:50:58 trial068 ceph-mon[38805]: pgmap v3520: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:50:59.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:50:58 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2017268948' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:50:59.547 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5853 8 365281 365273 0.487524 0 - 0.0299513 2026-02-23T14:51:00.547 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5854 8 365281 365273 0.48744 0 - 0.0299513 2026-02-23T14:51:01.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:51:00 trial068 ceph-mon[38805]: pgmap v3521: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:51:01.547 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5855 8 365281 365273 0.487357 0 - 0.0299513 2026-02-23T14:51:01.672 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:51:01.790 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:51:01.898 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:51:01.896+0000 7f6100dc4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:51:01.898 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:51:01.896+0000 7f6100dc4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:51:02.006 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:51:02.006 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":271,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:50:24.791727+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:45:46.188265+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:48:34.098389+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:49:22.169310+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:51:02.007 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:51:01 trial068 ceph-mon[38805]: 5.e scrub starts 2026-02-23T14:51:02.007 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:51:01 trial068 ceph-mon[38805]: 5.e scrub ok 2026-02-23T14:51:02.547 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5856 8 365281 365273 0.487274 0 - 0.0299513 2026-02-23T14:51:03.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:51:02 trial068 ceph-mon[38805]: pgmap v3522: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:51:03.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:51:02 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2937754703' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:51:03.547 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5857 8 365281 365273 0.487191 0 - 0.0299513 2026-02-23T14:51:04.264 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:51:03 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:51:03.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:51:04.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:51:03 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:51:03] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:51:04.547 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5858 8 365281 365273 0.487108 0 - 0.0299513 2026-02-23T14:51:05.043 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:51:05.161 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:51:05.183 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:51:04 trial068 ceph-mon[38805]: pgmap v3523: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:51:05.262 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:51:05.260+0000 7efcb97a6640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:51:05.263 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:51:05.261+0000 7efcb97a6640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:51:05.372 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:51:05.372 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":271,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:50:24.791727+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:45:46.188265+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:48:34.098389+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:49:22.169310+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:51:05.547 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5859 8 365281 365273 0.487024 0 - 0.0299513 2026-02-23T14:51:06.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:51:05 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2053699080' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:51:06.548 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T14:51:06.547336+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T14:51:06.548 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T14:51:06.548 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5860 8 365281 365273 0.486941 0 - 0.0299513 2026-02-23T14:51:07.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:51:06 trial068 ceph-mon[38805]: pgmap v3524: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:51:07.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:51:06 trial068 ceph-mon[38805]: 5.7 scrub starts 2026-02-23T14:51:07.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:51:06 trial068 ceph-mon[38805]: 5.7 scrub ok 2026-02-23T14:51:07.548 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5861 8 365281 365273 0.486858 0 - 0.0299513 2026-02-23T14:51:08.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:51:07 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2026-02-23T14:51:08.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:51:07 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-23T14:51:08.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:51:07 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2026-02-23T14:51:08.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:51:07 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T14:51:08.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:51:07 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2026-02-23T14:51:08.409 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:51:08.526 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:51:08.548 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5862 8 365281 365273 0.486775 0 - 0.0299513 2026-02-23T14:51:08.629 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:51:08.627+0000 7f1981c8f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:51:08.629 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:51:08.627+0000 7f1981c8f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:51:08.741 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:51:08.741 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":271,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:50:24.791727+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:45:46.188265+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:48:34.098389+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:49:22.169310+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:51:09.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:51:08 trial068 ceph-mon[38805]: pgmap v3525: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:51:09.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:51:08 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T14:51:09.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:51:08 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T14:51:09.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:51:08 trial068 ceph-mon[38805]: pgmap v3526: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:51:09.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:51:08 trial068 ceph-mon[38805]: pgmap v3527: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:51:09.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:51:08 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3192742845' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:51:09.548 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5863 8 365281 365273 0.486692 0 - 0.0299513 2026-02-23T14:51:10.548 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5864 8 365281 365273 0.486609 0 - 0.0299513 2026-02-23T14:51:11.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:51:10 trial068 ceph-mon[38805]: pgmap v3528: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:51:11.548 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5865 8 365281 365273 0.486526 0 - 0.0299513 2026-02-23T14:51:11.779 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:51:11.897 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:51:11.997 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:51:11.995+0000 7f1a11a6e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:51:11.997 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:51:11.996+0000 7f1a11a6e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:51:12.107 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:51:12.107 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":271,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:50:24.791727+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:45:46.188265+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:48:34.098389+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:49:22.169310+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:51:12.548 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5866 8 365281 365273 0.486443 0 - 0.0299513 2026-02-23T14:51:13.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:51:12 trial068 ceph-mon[38805]: pgmap v3529: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:51:13.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:51:12 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/231148106' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:51:13.548 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5867 8 365281 365273 0.48636 0 - 0.0299513 2026-02-23T14:51:14.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:51:13 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:51:13] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:51:14.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:51:13 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:51:13.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:51:14.548 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5868 8 365281 365273 0.486278 0 - 0.0299513 2026-02-23T14:51:15.145 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:51:15.170 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:51:14 trial068 ceph-mon[38805]: pgmap v3530: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:51:15.262 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:51:15.362 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:51:15.360+0000 7f3239f50640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:51:15.362 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:51:15.360+0000 7f3239f50640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:51:15.473 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:51:15.473 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":271,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:50:24.791727+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:45:46.188265+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:48:34.098389+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:49:22.169310+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:51:15.548 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5869 8 365281 365273 0.486195 0 - 0.0299513 2026-02-23T14:51:16.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:51:15 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1705886296' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:51:16.548 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5870 8 365281 365273 0.486112 0 - 0.0299513 2026-02-23T14:51:17.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:51:16 trial068 ceph-mon[38805]: pgmap v3531: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:51:17.548 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5871 8 365281 365273 0.486029 0 - 0.0299513 2026-02-23T14:51:18.512 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:51:18.548 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5872 8 365281 365273 0.485946 0 - 0.0299513 2026-02-23T14:51:18.630 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:51:18.730 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:51:18.728+0000 7f30787ed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:51:18.730 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:51:18.729+0000 7f30787ed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:51:18.840 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:51:18.840 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":271,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:50:24.791727+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:45:46.188265+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:48:34.098389+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:49:22.169310+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:51:19.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:51:18 trial068 ceph-mon[38805]: pgmap v3532: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:51:19.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:51:18 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2883782954' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:51:19.549 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5873 8 365281 365273 0.485864 0 - 0.0299513 2026-02-23T14:51:20.549 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5874 8 365281 365273 0.485781 0 - 0.0299513 2026-02-23T14:51:21.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:51:20 trial068 ceph-mon[38805]: pgmap v3533: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:51:21.549 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5875 8 365281 365273 0.485698 0 - 0.0299513 2026-02-23T14:51:21.877 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:51:21.998 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:51:22.095 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:51:22.093+0000 7fd16c9a2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:51:22.095 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:51:22.093+0000 7fd16c9a2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:51:22.204 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:51:22.204 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":271,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:50:24.791727+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:45:46.188265+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:48:34.098389+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:49:22.169310+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:51:22.549 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5876 8 365281 365273 0.485615 0 - 0.0299513 2026-02-23T14:51:23.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:51:22 trial068 ceph-mon[38805]: pgmap v3534: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:51:23.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:51:22 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/108968552' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:51:23.549 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5877 8 365281 365273 0.485533 0 - 0.0299513 2026-02-23T14:51:24.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:51:23 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:51:23] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:51:24.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:51:23 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:51:23.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:51:24.549 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5878 8 365281 365273 0.48545 0 - 0.0299513 2026-02-23T14:51:25.242 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:51:25.266 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:51:24 trial068 ceph-mon[38805]: pgmap v3535: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:51:25.359 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:51:25.464 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:51:25.462+0000 7f24ee3af640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:51:25.464 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:51:25.462+0000 7f24ee3af640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:51:25.549 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5879 8 365281 365273 0.485368 0 - 0.0299513 2026-02-23T14:51:25.572 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:51:25.573 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":271,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:50:24.791727+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:45:46.188265+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:48:34.098389+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:49:22.169310+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:51:26.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:51:25 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3556940829' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:51:26.549 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T14:51:26.549093+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T14:51:26.550 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T14:51:26.550 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5880 8 365281 365273 0.485285 0 - 0.0299513 2026-02-23T14:51:27.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:51:26 trial068 ceph-mon[38805]: pgmap v3536: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:51:27.549 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5881 8 365281 365273 0.485203 0 - 0.0299513 2026-02-23T14:51:28.549 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5882 8 365281 365273 0.48512 0 - 0.0299513 2026-02-23T14:51:28.610 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:51:28.730 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:51:28.839 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:51:28.837+0000 7ff468e66640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:51:28.839 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:51:28.837+0000 7ff468e66640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:51:28.949 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:51:28.950 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":271,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:50:24.791727+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:45:46.188265+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:48:34.098389+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:49:22.169310+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:51:29.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:51:28 trial068 ceph-mon[38805]: pgmap v3537: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:51:29.549 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5883 8 365281 365273 0.485038 0 - 0.0299513 2026-02-23T14:51:30.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:51:29 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2900764339' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:51:30.550 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5884 8 365281 365273 0.484955 0 - 0.0299513 2026-02-23T14:51:31.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:51:30 trial068 ceph-mon[38805]: pgmap v3538: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:51:31.550 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5885 8 365281 365273 0.484873 0 - 0.0299513 2026-02-23T14:51:31.988 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:51:32.107 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:51:32.219 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:51:32.217+0000 7f983b501640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:51:32.219 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:51:32.217+0000 7f983b501640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:51:32.331 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:51:32.331 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":272,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:51:31.924721+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:50:36.055613+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:48:34.098389+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:49:22.169310+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:51:32.550 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5886 8 365281 365273 0.48479 0 - 0.0299513 2026-02-23T14:51:33.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:51:32 trial068 ceph-mon[38805]: pgmap v3539: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:51:33.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:51:32 trial068 ceph-mon[38805]: osdmap e272: 4 total, 3 up, 4 in 2026-02-23T14:51:33.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:51:32 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/4235742265' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:51:33.550 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5887 8 365281 365273 0.484708 0 - 0.0299513 2026-02-23T14:51:34.265 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:51:33 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:51:33.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:51:34.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:51:33 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:51:33] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:51:34.550 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5888 8 365281 365273 0.484626 0 - 0.0299513 2026-02-23T14:51:35.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:51:34 trial068 ceph-mon[38805]: pgmap v3541: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:51:35.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:51:34 trial068 ceph-mon[38805]: osdmap e273: 4 total, 3 up, 4 in 2026-02-23T14:51:35.368 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:51:35.484 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:51:35.550 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5889 8 365281 365273 0.484543 0 - 0.0299513 2026-02-23T14:51:35.583 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:51:35.582+0000 7fd9c0ee8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:51:35.584 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:51:35.582+0000 7fd9c0ee8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:51:35.693 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:51:35.693 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":273,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:51:34.800555+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:50:36.055613+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:48:34.098389+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:49:22.169310+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:51:36.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:51:35 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2300466545' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:51:36.550 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5890 8 365281 365273 0.484461 0 - 0.0299513 2026-02-23T14:51:37.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:51:36 trial068 ceph-mon[38805]: pgmap v3543: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:51:37.550 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5891 8 365281 365273 0.484379 0 - 0.0299513 2026-02-23T14:51:38.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:51:37 trial068 ceph-mon[38805]: pgmap v3544: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:51:38.550 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5892 8 365281 365273 0.484297 0 - 0.0299513 2026-02-23T14:51:38.731 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:51:38.850 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:51:38.953 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:51:38.951+0000 7fe558c03640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:51:38.954 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:51:38.952+0000 7fe558c03640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:51:39.063 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:51:39.063 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":273,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:51:34.800555+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:50:36.055613+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:48:34.098389+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:49:22.169310+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:51:39.064 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:51:38 trial068 ceph-mon[38805]: 5.8 scrub starts 2026-02-23T14:51:39.064 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:51:38 trial068 ceph-mon[38805]: 5.8 scrub ok 2026-02-23T14:51:39.550 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5893 8 365281 365273 0.484215 0 - 0.0299513 2026-02-23T14:51:40.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:51:39 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2066922902' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:51:40.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:51:39 trial068 ceph-mon[38805]: pgmap v3545: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:51:40.550 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5894 8 365281 365273 0.484132 0 - 0.0299513 2026-02-23T14:51:41.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:51:41 trial068 ceph-mon[38805]: 5.5 scrub starts 2026-02-23T14:51:41.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:51:41 trial068 ceph-mon[38805]: 5.5 scrub ok 2026-02-23T14:51:41.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:51:41 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T14:51:41.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:51:41 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T14:51:41.550 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5895 8 365281 365273 0.48405 0 - 0.0299513 2026-02-23T14:51:42.101 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:51:42.220 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:51:42.267 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:51:42 trial068 ceph-mon[38805]: pgmap v3546: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:51:42.319 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:51:42.316+0000 7f666d9d0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:51:42.319 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:51:42.317+0000 7f666d9d0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:51:42.427 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:51:42.428 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":273,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:51:34.800555+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:50:36.055613+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:48:34.098389+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:49:22.169310+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:51:42.551 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5896 8 365281 365273 0.483968 0 - 0.0299513 2026-02-23T14:51:43.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:51:43 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1027588617' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:51:43.550 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5897 8 365281 365273 0.483886 0 - 0.0299513 2026-02-23T14:51:44.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:51:44 trial068 ceph-mon[38805]: pgmap v3547: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:51:44.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:51:43 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:51:43] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:51:44.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:51:43 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:51:43.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:51:44.550 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5898 8 365281 365273 0.483804 0 - 0.0299513 2026-02-23T14:51:45.463 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:51:45.551 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5899 8 365281 365273 0.483722 0 - 0.0299513 2026-02-23T14:51:45.580 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:51:45.678 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:51:45.676+0000 7f127f210640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:51:45.678 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:51:45.677+0000 7f127f210640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:51:45.788 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:51:45.788 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":273,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:51:34.800555+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:50:36.055613+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:48:34.098389+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:49:22.169310+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:51:46.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:51:45 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T14:51:46.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:51:45 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T14:51:46.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:51:45 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2831619246' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:51:46.551 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T14:51:46.550627+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T14:51:46.551 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T14:51:46.551 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5900 8 365281 365273 0.48364 0 - 0.0299513 2026-02-23T14:51:47.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:51:46 trial068 ceph-mon[38805]: pgmap v3548: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:51:47.551 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5901 8 365281 365273 0.483558 0 - 0.0299513 2026-02-23T14:51:48.551 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5902 8 365281 365273 0.483476 0 - 0.0299513 2026-02-23T14:51:48.828 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:51:48.944 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:51:49.045 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:51:49.043+0000 7f53822d2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:51:49.045 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:51:49.043+0000 7f53822d2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:51:49.057 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:51:48 trial068 ceph-mon[38805]: pgmap v3549: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:51:49.153 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:51:49.153 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":273,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:51:34.800555+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:50:36.055613+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:48:34.098389+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:49:22.169310+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:51:49.551 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5903 8 365281 365273 0.483394 0 - 0.0299513 2026-02-23T14:51:50.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:51:49 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3869108003' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:51:50.551 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5904 8 365281 365273 0.483312 0 - 0.0299513 2026-02-23T14:51:51.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:51:50 trial068 ceph-mon[38805]: pgmap v3550: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:51:51.551 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5905 8 365281 365273 0.483231 0 - 0.0299513 2026-02-23T14:51:52.189 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:51:52.310 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:51:52.418 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:51:52.416+0000 7efc2c4c2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:51:52.418 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:51:52.416+0000 7efc2c4c2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:51:52.527 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:51:52.527 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":273,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:51:34.800555+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:50:36.055613+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:48:34.098389+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:49:22.169310+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:51:52.551 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5906 8 365281 365273 0.483149 0 - 0.0299513 2026-02-23T14:51:53.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:51:52 trial068 ceph-mon[38805]: pgmap v3551: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:51:53.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:51:52 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2940617973' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:51:53.551 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5907 8 365281 365273 0.483067 0 - 0.0299513 2026-02-23T14:51:54.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:51:53 trial068 ceph-mon[38805]: 5.a scrub starts 2026-02-23T14:51:54.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:51:53 trial068 ceph-mon[38805]: 5.a scrub ok 2026-02-23T14:51:54.094 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:51:53 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:51:53] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:51:54.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:51:53 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:51:53.974Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:51:54.551 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5908 8 365281 365273 0.482985 0 - 0.0299513 2026-02-23T14:51:55.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:51:54 trial068 ceph-mon[38805]: pgmap v3552: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:51:55.551 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5909 8 365281 365273 0.482903 0 - 0.0299513 2026-02-23T14:51:55.564 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:51:55.683 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:51:55.789 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:51:55.787+0000 7fbe3e80a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:51:55.789 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:51:55.787+0000 7fbe3e80a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:51:55.898 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:51:55.898 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":273,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:51:34.800555+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:50:36.055613+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:48:34.098389+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:49:22.169310+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:51:56.552 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5910 8 365281 365273 0.482822 0 - 0.0299513 2026-02-23T14:51:57.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:51:56 trial068 ceph-mon[38805]: pgmap v3553: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:51:57.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:51:56 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3344840139' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:51:57.552 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5911 8 365281 365273 0.48274 0 - 0.0299513 2026-02-23T14:51:58.552 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5912 8 365281 365273 0.482658 0 - 0.0299513 2026-02-23T14:51:58.934 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:51:59.051 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:51:59.073 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:51:58 trial068 ceph-mon[38805]: pgmap v3554: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:51:59.151 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:51:59.149+0000 7fd5f34b7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:51:59.151 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:51:59.149+0000 7fd5f34b7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:51:59.259 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:51:59.260 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":273,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:51:34.800555+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:50:36.055613+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:48:34.098389+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:49:22.169310+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:51:59.552 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5913 8 365281 365273 0.482577 0 - 0.0299513 2026-02-23T14:52:00.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:51:59 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2501252025' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:52:00.552 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5914 8 365281 365273 0.482495 0 - 0.0299513 2026-02-23T14:52:01.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:52:00 trial068 ceph-mon[38805]: pgmap v3555: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:52:01.552 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5915 8 365281 365273 0.482414 0 - 0.0299513 2026-02-23T14:52:02.295 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:52:02.414 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:52:02.514 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:52:02.512+0000 7fbcef7d3640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:52:02.514 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:52:02.512+0000 7fbcef7d3640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:52:02.552 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5916 8 365281 365273 0.482332 0 - 0.0299513 2026-02-23T14:52:02.623 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:52:02.623 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":273,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:51:34.800555+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:50:36.055613+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:48:34.098389+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:49:22.169310+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:52:03.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:52:02 trial068 ceph-mon[38805]: pgmap v3556: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:52:03.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:52:02 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/282540812' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:52:03.552 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5917 8 365281 365273 0.482251 0 - 0.0299513 2026-02-23T14:52:04.265 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:52:03 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:52:03.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:52:04.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:52:03 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:52:03] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:52:04.552 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5918 8 365281 365273 0.482169 0 - 0.0299513 2026-02-23T14:52:05.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:52:04 trial068 ceph-mon[38805]: pgmap v3557: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:52:05.552 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5919 8 365281 365273 0.482088 0 - 0.0299513 2026-02-23T14:52:05.660 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:52:05.778 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:52:05.880 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:52:05.877+0000 7f4f7ce0a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:52:05.880 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:52:05.878+0000 7f4f7ce0a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:52:05.987 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:52:05.988 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":273,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:51:34.800555+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:50:36.055613+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:48:34.098389+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:49:22.169310+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:52:06.552 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T14:52:06.552037+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T14:52:06.552 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T14:52:06.553 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5920 8 365281 365273 0.482006 0 - 0.0299513 2026-02-23T14:52:07.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:52:06 trial068 ceph-mon[38805]: pgmap v3558: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:52:07.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:52:06 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/4051192383' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:52:07.552 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5921 8 365281 365273 0.481925 0 - 0.0299513 2026-02-23T14:52:08.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:52:07 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2026-02-23T14:52:08.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:52:07 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-23T14:52:08.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:52:07 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2026-02-23T14:52:08.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:52:07 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T14:52:08.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:52:07 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2026-02-23T14:52:08.552 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5922 8 365281 365273 0.481843 0 - 0.0299513 2026-02-23T14:52:09.025 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:52:09.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:52:08 trial068 ceph-mon[38805]: pgmap v3559: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:52:09.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:52:08 trial068 ceph-mon[38805]: pgmap v3560: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:52:09.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:52:08 trial068 ceph-mon[38805]: pgmap v3561: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:52:09.141 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:52:09.238 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:52:09.236+0000 7f0c43f64640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:52:09.238 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:52:09.236+0000 7f0c43f64640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:52:09.351 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:52:09.352 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":273,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:51:34.800555+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:50:36.055613+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:48:34.098389+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:49:22.169310+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:52:09.552 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5923 8 365281 365273 0.481762 0 - 0.0299513 2026-02-23T14:52:10.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:52:09 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2192026947' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:52:10.553 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5924 8 365281 365273 0.481681 0 - 0.0299513 2026-02-23T14:52:11.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:52:10 trial068 ceph-mon[38805]: pgmap v3562: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:52:11.552 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5925 8 365281 365273 0.481599 0 - 0.0299513 2026-02-23T14:52:12.388 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:52:12.508 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:52:12.553 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5926 8 365281 365273 0.481518 0 - 0.0299513 2026-02-23T14:52:12.608 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:52:12.606+0000 7f690cdf3640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:52:12.608 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:52:12.607+0000 7f690cdf3640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:52:12.717 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:52:12.717 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":273,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:51:34.800555+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:50:36.055613+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:48:34.098389+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:49:22.169310+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:52:13.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:52:12 trial068 ceph-mon[38805]: pgmap v3563: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:52:13.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:52:12 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2413115299' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:52:13.553 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5927 8 365281 365273 0.481437 0 - 0.0299513 2026-02-23T14:52:14.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:52:13 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:52:13] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:52:14.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:52:13 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:52:13.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:52:14.553 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5928 8 365281 365273 0.481356 0 - 0.0299513 2026-02-23T14:52:15.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:52:14 trial068 ceph-mon[38805]: pgmap v3564: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:52:15.553 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5929 8 365281 365273 0.481274 0 - 0.0299513 2026-02-23T14:52:15.754 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:52:15.874 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:52:15.980 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:52:15.978+0000 7fb74f318640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:52:15.980 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:52:15.978+0000 7fb74f318640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:52:16.090 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:52:16.090 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":273,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:51:34.800555+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:50:36.055613+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:48:34.098389+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:49:22.169310+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:52:16.553 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5930 8 365281 365273 0.481193 0 - 0.0299513 2026-02-23T14:52:17.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:52:16 trial068 ceph-mon[38805]: pgmap v3565: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:52:17.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:52:16 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/22016322' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:52:17.553 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5931 8 365281 365273 0.481112 0 - 0.0299513 2026-02-23T14:52:18.553 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5932 8 365281 365273 0.481031 0 - 0.0299513 2026-02-23T14:52:19.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:52:18 trial068 ceph-mon[38805]: pgmap v3566: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:52:19.128 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:52:19.246 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:52:19.349 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:52:19.347+0000 7f776076e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:52:19.349 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:52:19.348+0000 7f776076e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:52:19.460 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:52:19.460 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":273,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:51:34.800555+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:50:36.055613+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:48:34.098389+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:49:22.169310+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:52:19.553 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5933 8 365281 365273 0.48095 0 - 0.0299513 2026-02-23T14:52:20.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:52:19 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3384873147' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:52:20.553 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5934 8 365281 365273 0.480869 0 - 0.0299513 2026-02-23T14:52:21.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:52:20 trial068 ceph-mon[38805]: pgmap v3567: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:52:21.553 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5935 8 365281 365273 0.480788 0 - 0.0299513 2026-02-23T14:52:22.501 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:52:22.553 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5936 8 365281 365273 0.480707 0 - 0.0299513 2026-02-23T14:52:22.617 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:52:22.719 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:52:22.717+0000 7ff37a794640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:52:22.719 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:52:22.717+0000 7ff37a794640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:52:22.829 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:52:22.829 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":273,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:51:34.800555+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:50:36.055613+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:48:34.098389+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:49:22.169310+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:52:23.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:52:22 trial068 ceph-mon[38805]: pgmap v3568: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:52:23.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:52:22 trial068 ceph-mon[38805]: 5.7 scrub starts 2026-02-23T14:52:23.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:52:22 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/4139352045' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:52:23.554 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5937 8 365281 365273 0.480626 0 - 0.0299513 2026-02-23T14:52:24.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:52:23 trial068 ceph-mon[38805]: 5.7 scrub ok 2026-02-23T14:52:24.094 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:52:23 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:52:23] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:52:24.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:52:23 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:52:23.974Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:52:24.553 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5938 8 365281 365273 0.480545 0 - 0.0299513 2026-02-23T14:52:25.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:52:24 trial068 ceph-mon[38805]: pgmap v3569: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:52:25.553 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5939 8 365281 365273 0.480464 0 - 0.0299513 2026-02-23T14:52:25.865 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:52:25.983 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:52:26.076 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:52:26.074+0000 7f6bc3035640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:52:26.076 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:52:26.075+0000 7f6bc3035640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:52:26.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:52:25 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T14:52:26.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:52:25 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T14:52:26.185 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:52:26.185 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":273,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:51:34.800555+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:50:36.055613+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:48:34.098389+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:49:22.169310+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:52:26.554 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T14:52:26.553482+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T14:52:26.554 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T14:52:26.554 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5940 8 365281 365273 0.480383 0 - 0.0299513 2026-02-23T14:52:27.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:52:26 trial068 ceph-mon[38805]: pgmap v3570: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:52:27.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:52:26 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/4180435736' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:52:27.554 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5941 8 365281 365273 0.480302 0 - 0.0299513 2026-02-23T14:52:28.554 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5942 8 365281 365273 0.480222 0 - 0.0299513 2026-02-23T14:52:29.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:52:28 trial068 ceph-mon[38805]: pgmap v3571: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:52:29.222 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:52:29.340 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:52:29.439 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:52:29.436+0000 7f93c01f6640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:52:29.439 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:52:29.437+0000 7f93c01f6640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:52:29.546 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:52:29.547 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":273,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:51:34.800555+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:50:36.055613+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:48:34.098389+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:49:22.169310+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:52:29.554 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5943 8 365281 365273 0.480141 0 - 0.0299513 2026-02-23T14:52:30.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:52:29 trial068 ceph-mon[38805]: 5.e scrub starts 2026-02-23T14:52:30.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:52:29 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3992401918' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:52:30.554 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5944 8 365281 365273 0.48006 0 - 0.0299513 2026-02-23T14:52:31.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:52:30 trial068 ceph-mon[38805]: 5.e scrub ok 2026-02-23T14:52:31.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:52:30 trial068 ceph-mon[38805]: pgmap v3572: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:52:31.554 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5945 8 365281 365273 0.479979 0 - 0.0299513 2026-02-23T14:52:32.554 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5946 8 365281 365273 0.479899 0 - 0.0299513 2026-02-23T14:52:32.583 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:52:32.699 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:52:32.801 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:52:32.799+0000 7f9ebec26640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:52:32.801 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:52:32.800+0000 7f9ebec26640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:52:32.911 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:52:32.912 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":273,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:51:34.800555+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:50:36.055613+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:48:34.098389+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:49:22.169310+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:52:32.912 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:52:32 trial068 ceph-mon[38805]: pgmap v3573: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:52:33.555 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5947 8 365281 365273 0.479818 0 - 0.0299513 2026-02-23T14:52:34.266 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:52:33 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:52:33.974Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:52:34.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:52:33 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2522536652' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:52:34.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:52:33 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:52:33] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:52:34.554 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5948 8 365281 365273 0.479737 0 - 0.0299513 2026-02-23T14:52:35.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:52:34 trial068 ceph-mon[38805]: pgmap v3574: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:52:35.554 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5949 8 365281 365273 0.479656 0 - 0.0299513 2026-02-23T14:52:35.950 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:52:36.069 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:52:36.177 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:52:36.175+0000 7f62e0772640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:52:36.177 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:52:36.175+0000 7f62e0772640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:52:36.289 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:52:36.289 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":273,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:51:34.800555+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:50:36.055613+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:48:34.098389+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:49:22.169310+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:52:36.554 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5950 8 365281 365273 0.479576 0 - 0.0299513 2026-02-23T14:52:37.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:52:36 trial068 ceph-mon[38805]: pgmap v3575: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:52:37.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:52:36 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1836529928' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:52:37.554 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5951 8 365281 365273 0.479495 0 - 0.0299513 2026-02-23T14:52:38.555 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5952 8 365281 365273 0.479415 0 - 0.0299513 2026-02-23T14:52:39.326 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:52:39.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:52:38 trial068 ceph-mon[38805]: pgmap v3576: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:52:39.445 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:52:39.547 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:52:39.546+0000 7fa753701640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:52:39.548 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:52:39.546+0000 7fa753701640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:52:39.554 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5953 8 365281 365273 0.479334 0 - 0.0299513 2026-02-23T14:52:39.657 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:52:39.657 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":273,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:51:34.800555+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:50:36.055613+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:48:34.098389+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:49:22.169310+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:52:40.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:52:39 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3379443490' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:52:40.555 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5954 8 365281 365273 0.479254 0 - 0.0299513 2026-02-23T14:52:41.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:52:40 trial068 ceph-mon[38805]: pgmap v3577: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:52:41.555 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5955 8 365281 365273 0.479173 0 - 0.0299513 2026-02-23T14:52:42.555 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5956 8 365281 365273 0.479093 0 - 0.0299513 2026-02-23T14:52:42.695 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:52:42.809 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:52:42.914 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:52:42.912+0000 7f4b0371c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:52:42.914 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:52:42.913+0000 7f4b0371c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:52:43.023 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:52:43.023 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":273,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:51:34.800555+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:50:36.055613+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:48:34.098389+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:49:22.169310+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:52:43.024 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:52:42 trial068 ceph-mon[38805]: pgmap v3578: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:52:43.555 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5957 8 365281 365273 0.479012 0 - 0.0299513 2026-02-23T14:52:44.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:52:43 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/4163343153' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:52:44.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:52:43 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:52:43] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:52:44.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:52:43 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:52:43.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:52:44.555 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5958 8 365281 365273 0.478932 0 - 0.0299513 2026-02-23T14:52:45.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:52:44 trial068 ceph-mon[38805]: pgmap v3579: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:52:45.555 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5959 8 365281 365273 0.478852 0 - 0.0299513 2026-02-23T14:52:46.061 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:52:46.177 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:52:46.283 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:52:46.281+0000 7f0de3fff640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:52:46.283 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:52:46.281+0000 7f0de3fff640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:52:46.391 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:52:46.391 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":273,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:51:34.800555+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:50:36.055613+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:48:34.098389+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:49:22.169310+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:52:46.555 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T14:52:46.554872+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T14:52:46.555 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T14:52:46.555 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5960 8 365281 365273 0.478771 0 - 0.0299513 2026-02-23T14:52:47.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:52:46 trial068 ceph-mon[38805]: pgmap v3580: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:52:47.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:52:46 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3087034867' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:52:47.555 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5961 8 365281 365273 0.478691 0 - 0.0299513 2026-02-23T14:52:48.555 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5962 8 365281 365273 0.478611 0 - 0.0299513 2026-02-23T14:52:49.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:52:48 trial068 ceph-mon[38805]: pgmap v3581: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:52:49.429 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:52:49.545 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:52:49.555 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5963 8 365281 365273 0.47853 0 - 0.0299513 2026-02-23T14:52:49.643 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:52:49.641+0000 7f3bb5cd5640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:52:49.643 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:52:49.641+0000 7f3bb5cd5640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:52:49.754 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:52:49.755 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":273,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:51:34.800555+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:50:36.055613+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:48:34.098389+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:49:22.169310+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:52:50.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:52:49 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3198499721' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:52:50.555 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5964 8 365281 365273 0.47845 0 - 0.0299513 2026-02-23T14:52:51.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:52:50 trial068 ceph-mon[38805]: pgmap v3582: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:52:51.555 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5965 8 365281 365273 0.47837 0 - 0.0299513 2026-02-23T14:52:52.555 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5966 8 365281 365273 0.47829 0 - 0.0299513 2026-02-23T14:52:52.791 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:52:52.912 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:52:53.018 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:52:53.016+0000 7f18758b0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:52:53.018 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:52:53.016+0000 7f18758b0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:52:53.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:52:52 trial068 ceph-mon[38805]: pgmap v3583: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:52:53.127 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:52:53.128 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":273,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:51:34.800555+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:50:36.055613+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:48:34.098389+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:49:22.169310+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:52:53.556 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5967 8 365281 365273 0.47821 0 - 0.0299513 2026-02-23T14:52:54.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:52:53 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2823577754' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:52:54.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:52:53 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:52:53] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:52:54.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:52:53 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:52:53.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:52:54.556 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5968 8 365281 365273 0.478129 0 - 0.0299513 2026-02-23T14:52:55.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:52:54 trial068 ceph-mon[38805]: pgmap v3584: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:52:55.556 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5969 8 365281 365273 0.478049 0 - 0.0299513 2026-02-23T14:52:56.164 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:52:56.283 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:52:56.382 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:52:56.380+0000 7f04ba66b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:52:56.382 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:52:56.380+0000 7f04ba66b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:52:56.492 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:52:56.492 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":273,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:51:34.800555+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:50:36.055613+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:48:34.098389+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:49:22.169310+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:52:56.556 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5970 8 365281 365273 0.477969 0 - 0.0299513 2026-02-23T14:52:57.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:52:56 trial068 ceph-mon[38805]: pgmap v3585: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:52:57.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:52:56 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2180919871' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:52:57.556 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5971 8 365281 365273 0.477889 0 - 0.0299513 2026-02-23T14:52:58.556 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5972 8 365281 365273 0.477809 0 - 0.0299513 2026-02-23T14:52:59.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:52:58 trial068 ceph-mon[38805]: pgmap v3586: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:52:59.529 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:52:59.556 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5973 8 365281 365273 0.477729 0 - 0.0299513 2026-02-23T14:52:59.648 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:52:59.746 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:52:59.744+0000 7f2f5d9dd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:52:59.746 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:52:59.744+0000 7f2f5d9dd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:52:59.861 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:52:59.861 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":273,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:51:34.800555+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:50:36.055613+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:48:34.098389+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:49:22.169310+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:53:00.556 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5974 8 365281 365273 0.477649 0 - 0.0299513 2026-02-23T14:53:01.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:53:00 trial068 ceph-mon[38805]: pgmap v3587: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:53:01.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:53:00 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/238868517' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:53:01.556 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5975 8 365281 365273 0.477569 0 - 0.0299513 2026-02-23T14:53:02.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:53:01 trial068 ceph-mon[38805]: 5.a scrub starts 2026-02-23T14:53:02.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:53:01 trial068 ceph-mon[38805]: 5.a scrub ok 2026-02-23T14:53:02.556 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5976 8 365281 365273 0.477489 0 - 0.0299513 2026-02-23T14:53:02.900 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:53:03.018 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:53:03.069 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:53:02 trial068 ceph-mon[38805]: pgmap v3588: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:53:03.115 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:53:03.113+0000 7f0f92fe8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:53:03.116 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:53:03.114+0000 7f0f92fe8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:53:03.225 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:53:03.225 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":273,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:51:34.800555+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:50:36.055613+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:48:34.098389+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:49:22.169310+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:53:03.556 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5977 8 365281 365273 0.477409 0 - 0.0299513 2026-02-23T14:53:04.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:53:03 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T14:53:04.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:53:03 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T14:53:04.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:53:03 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/790309452' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:53:04.094 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:53:03 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:53:03] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:53:04.265 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:53:03 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:53:03.972Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:53:04.556 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5978 8 365281 365273 0.47733 0 - 0.0299513 2026-02-23T14:53:05.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:53:04 trial068 ceph-mon[38805]: 5.8 scrub starts 2026-02-23T14:53:05.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:53:04 trial068 ceph-mon[38805]: 5.8 scrub ok 2026-02-23T14:53:05.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:53:04 trial068 ceph-mon[38805]: pgmap v3589: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:53:05.556 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5979 8 365281 365273 0.47725 0 - 0.0299513 2026-02-23T14:53:06.266 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:53:06.382 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:53:06.478 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:53:06.476+0000 7f2809ea9640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:53:06.478 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:53:06.476+0000 7f2809ea9640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:53:06.556 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T14:53:06.556312+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T14:53:06.557 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T14:53:06.557 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5980 8 365281 365273 0.47717 0 - 0.0299513 2026-02-23T14:53:06.590 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:53:06.590 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":273,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:51:34.800555+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:50:36.055613+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:48:34.098389+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:49:22.169310+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:53:06.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:53:06 trial068 ceph-mon[38805]: pgmap v3590: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:53:06.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:53:06 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/822026132' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:53:07.557 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5981 8 365281 365273 0.47709 0 - 0.0299513 2026-02-23T14:53:08.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:53:07 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2026-02-23T14:53:08.557 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5982 8 365281 365273 0.47701 0 - 0.0299513 2026-02-23T14:53:09.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:53:08 trial068 ceph-mon[38805]: pgmap v3591: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:53:09.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:53:08 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-23T14:53:09.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:53:08 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2026-02-23T14:53:09.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:53:08 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T14:53:09.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:53:08 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2026-02-23T14:53:09.557 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5983 8 365281 365273 0.476931 0 - 0.0299513 2026-02-23T14:53:09.629 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:53:09.747 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:53:09.848 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:53:09.846+0000 7fd734a1a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:53:09.848 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:53:09.846+0000 7fd734a1a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:53:09.958 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:53:09.958 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":273,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:51:34.800555+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:50:36.055613+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:48:34.098389+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:49:22.169310+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:53:09.958 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:53:09 trial068 ceph-mon[38805]: 5.5 scrub starts 2026-02-23T14:53:09.958 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:53:09 trial068 ceph-mon[38805]: pgmap v3592: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:53:09.959 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:53:09 trial068 ceph-mon[38805]: pgmap v3593: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:53:09.959 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:53:09 trial068 ceph-mon[38805]: 5.5 scrub ok 2026-02-23T14:53:09.959 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:53:09 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T14:53:09.959 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:53:09 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T14:53:10.557 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5984 8 365281 365273 0.476851 0 - 0.0299513 2026-02-23T14:53:11.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:53:10 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3794872185' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:53:11.557 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5985 8 365281 365273 0.476771 0 - 0.0299513 2026-02-23T14:53:12.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:53:11 trial068 ceph-mon[38805]: pgmap v3594: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:53:12.557 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5986 8 365281 365273 0.476692 0 - 0.0299513 2026-02-23T14:53:12.995 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:53:13.111 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:53:13.211 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:53:13.209+0000 7f917abe0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:53:13.211 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:53:13.209+0000 7f917abe0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:53:13.320 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:53:13.321 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":273,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:51:34.800555+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:50:36.055613+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:48:34.098389+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:49:22.169310+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:53:13.557 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5987 8 365281 365273 0.476612 0 - 0.0299513 2026-02-23T14:53:14.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:53:13 trial068 ceph-mon[38805]: pgmap v3595: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:53:14.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:53:13 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3294047370' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:53:14.094 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:53:13 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:53:13] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:53:14.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:53:13 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:53:13.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:53:14.557 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5988 8 365281 365273 0.476532 0 - 0.0299513 2026-02-23T14:53:15.557 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5989 8 365281 365273 0.476453 0 - 0.0299513 2026-02-23T14:53:16.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:53:15 trial068 ceph-mon[38805]: pgmap v3596: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:53:16.362 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:53:16.481 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:53:16.557 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5990 8 365281 365273 0.476373 0 - 0.0299513 2026-02-23T14:53:16.587 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:53:16.585+0000 7f97c7cf9640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:53:16.587 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:53:16.585+0000 7f97c7cf9640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:53:16.695 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:53:16.696 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":273,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:51:34.800555+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:50:36.055613+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:48:34.098389+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:49:22.169310+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:53:17.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:53:16 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/539745325' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:53:17.557 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5991 8 365281 365273 0.476294 0 - 0.0299513 2026-02-23T14:53:18.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:53:17 trial068 ceph-mon[38805]: pgmap v3597: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:53:18.557 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5992 8 365281 365273 0.476214 0 - 0.0299513 2026-02-23T14:53:19.557 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5993 8 365281 365273 0.476135 0 - 0.0299513 2026-02-23T14:53:19.733 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:53:19.852 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:53:19.949 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:53:19.947+0000 7f0b3fed1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:53:19.949 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:53:19.947+0000 7f0b3fed1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:53:20.057 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:53:20.058 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":273,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:51:34.800555+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:50:36.055613+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:48:34.098389+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:49:22.169310+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:53:20.058 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:53:19 trial068 ceph-mon[38805]: pgmap v3598: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:53:20.558 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5994 8 365281 365273 0.476055 0 - 0.0299513 2026-02-23T14:53:21.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:53:20 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3734477956' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:53:21.558 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5995 8 365281 365273 0.475976 0 - 0.0299513 2026-02-23T14:53:22.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:53:21 trial068 ceph-mon[38805]: pgmap v3599: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:53:22.558 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5996 8 365281 365273 0.475897 0 - 0.0299513 2026-02-23T14:53:23.095 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:53:23.214 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:53:23.313 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:53:23.311+0000 7ff4cb61a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:53:23.313 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:53:23.311+0000 7ff4cb61a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:53:23.427 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:53:23.427 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":273,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:51:34.800555+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:50:36.055613+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:48:34.098389+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:49:22.169310+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:53:23.558 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5997 8 365281 365273 0.475817 0 - 0.0299513 2026-02-23T14:53:24.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:53:23 trial068 ceph-mon[38805]: pgmap v3600: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:53:24.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:53:23 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3548246952' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:53:24.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:53:23 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:53:23] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:53:24.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:53:23 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:53:23.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:53:24.558 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5998 8 365281 365273 0.475738 0 - 0.0299513 2026-02-23T14:53:25.558 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 5999 8 365281 365273 0.475659 0 - 0.0299513 2026-02-23T14:53:26.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:53:25 trial068 ceph-mon[38805]: pgmap v3601: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:53:26.466 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:53:26.558 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T14:53:26.557837+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T14:53:26.558 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T14:53:26.558 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6000 8 365281 365273 0.475579 0 - 0.0299513 2026-02-23T14:53:26.584 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:53:26.690 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:53:26.688+0000 7fc43f3b8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:53:26.690 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:53:26.688+0000 7fc43f3b8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:53:26.799 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:53:26.800 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":273,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:51:34.800555+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:50:36.055613+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:48:34.098389+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:49:22.169310+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:53:27.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:53:26 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2495937031' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:53:27.558 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6001 8 365281 365273 0.4755 0 - 0.0299513 2026-02-23T14:53:28.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:53:27 trial068 ceph-mon[38805]: pgmap v3602: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:53:28.558 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6002 8 365281 365273 0.475421 0 - 0.0299513 2026-02-23T14:53:29.558 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6003 8 365281 365273 0.475342 0 - 0.0299513 2026-02-23T14:53:29.836 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:53:29.953 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:53:30.052 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:53:30.051+0000 7fe998e20640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:53:30.053 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:53:30.051+0000 7fe998e20640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:53:30.162 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:53:30.162 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":273,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:51:34.800555+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:50:36.055613+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:48:34.098389+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:49:22.169310+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:53:30.162 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:53:29 trial068 ceph-mon[38805]: pgmap v3603: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:53:30.558 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6004 8 365281 365273 0.475263 0 - 0.0299513 2026-02-23T14:53:31.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:53:30 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1298171301' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:53:31.558 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6005 8 365281 365273 0.475183 0 - 0.0299513 2026-02-23T14:53:32.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:53:31 trial068 ceph-mon[38805]: pgmap v3604: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:53:32.558 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6006 8 365281 365273 0.475104 0 - 0.0299513 2026-02-23T14:53:33.199 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:53:33.320 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:53:33.423 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:53:33.421+0000 7f0801207640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:53:33.423 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:53:33.421+0000 7f0801207640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:53:33.531 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:53:33.531 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":273,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:51:34.800555+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:50:36.055613+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:48:34.098389+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:49:22.169310+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:53:33.558 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6007 8 365281 365273 0.475025 0 - 0.0299513 2026-02-23T14:53:34.266 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:53:33 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:53:33.974Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:53:34.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:53:33 trial068 ceph-mon[38805]: pgmap v3605: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:53:34.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:53:33 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2364804890' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:53:34.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:53:33 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:53:33] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:53:34.559 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6008 8 365281 365273 0.474946 0 - 0.0299513 2026-02-23T14:53:35.559 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6009 8 365281 365273 0.474867 0 - 0.0299513 2026-02-23T14:53:36.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:53:35 trial068 ceph-mon[38805]: pgmap v3606: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:53:36.559 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6010 8 365281 365273 0.474788 0 - 0.0299513 2026-02-23T14:53:36.568 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:53:36.686 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:53:36.784 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:53:36.782+0000 7facb5b3d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:53:36.784 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:53:36.782+0000 7facb5b3d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:53:36.895 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:53:36.895 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":273,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:51:34.800555+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:50:36.055613+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:48:34.098389+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:49:22.169310+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:53:37.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:53:36 trial068 ceph-mon[38805]: 5.7 scrub starts 2026-02-23T14:53:37.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:53:36 trial068 ceph-mon[38805]: 5.7 scrub ok 2026-02-23T14:53:37.559 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6011 8 365281 365273 0.474709 0 - 0.0299513 2026-02-23T14:53:38.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:53:37 trial068 ceph-mon[38805]: pgmap v3607: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:53:38.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:53:37 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1914474822' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:53:38.559 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6012 8 365281 365273 0.47463 0 - 0.0299513 2026-02-23T14:53:39.559 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6013 8 365281 365273 0.474551 0 - 0.0299513 2026-02-23T14:53:39.946 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:53:40.064 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:53:40.113 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:53:39 trial068 ceph-mon[38805]: pgmap v3608: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:53:40.165 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:53:40.164+0000 7ff33692b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:53:40.166 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:53:40.164+0000 7ff33692b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:53:40.275 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:53:40.275 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":273,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:51:34.800555+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:50:36.055613+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:48:34.098389+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:49:22.169310+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:53:40.559 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6014 8 365281 365273 0.474472 0 - 0.0299513 2026-02-23T14:53:41.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:53:40 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T14:53:41.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:53:40 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T14:53:41.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:53:40 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1897059975' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:53:41.559 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6015 8 365281 365273 0.474393 0 - 0.0299513 2026-02-23T14:53:42.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:53:41 trial068 ceph-mon[38805]: pgmap v3609: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:53:42.559 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6016 8 365281 365273 0.474315 0 - 0.0299513 2026-02-23T14:53:43.244 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:53:42 trial068 ceph-mon[38805]: 5.e scrub starts 2026-02-23T14:53:43.245 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:53:42 trial068 ceph-mon[38805]: 5.e scrub ok 2026-02-23T14:53:43.313 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:53:43.433 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:53:43.536 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:53:43.534+0000 7f495c653640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:53:43.536 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:53:43.534+0000 7f495c653640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:53:43.559 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6017 8 365281 365273 0.474236 0 - 0.0299513 2026-02-23T14:53:43.645 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:53:43.645 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":273,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:51:34.800555+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:50:36.055613+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:48:34.098389+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:49:22.169310+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:53:44.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:53:43 trial068 ceph-mon[38805]: pgmap v3610: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:53:44.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:53:43 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2506599622' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:53:44.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:53:43 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:53:43] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:53:44.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:53:43 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:53:43.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:53:44.559 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6018 8 365281 365273 0.474157 0 - 0.0299513 2026-02-23T14:53:45.559 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6019 8 365281 365273 0.474078 0 - 0.0299513 2026-02-23T14:53:46.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:53:45 trial068 ceph-mon[38805]: pgmap v3611: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:53:46.559 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T14:53:46.559346+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T14:53:46.560 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T14:53:46.560 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6020 8 365281 365273 0.473999 0 - 0.0299513 2026-02-23T14:53:46.684 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:53:46.799 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:53:46.898 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:53:46.896+0000 7f08de74b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:53:46.898 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:53:46.896+0000 7f08de74b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:53:47.009 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:53:47.009 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":273,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:51:34.800555+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:50:36.055613+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:48:34.098389+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:49:22.169310+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:53:47.560 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6021 8 365281 365273 0.473921 0 - 0.0299513 2026-02-23T14:53:48.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:53:47 trial068 ceph-mon[38805]: pgmap v3612: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:53:48.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:53:47 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/259994178' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:53:48.560 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6022 8 365281 365273 0.473842 0 - 0.0299513 2026-02-23T14:53:49.560 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6023 8 365281 365273 0.473763 0 - 0.0299513 2026-02-23T14:53:50.063 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:53:50.181 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:53:50.204 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:53:49 trial068 ceph-mon[38805]: pgmap v3613: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:53:50.288 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:53:50.286+0000 7f187acfd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:53:50.288 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:53:50.286+0000 7f187acfd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:53:50.396 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:53:50.396 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":273,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:51:34.800555+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:50:36.055613+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:48:34.098389+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:49:22.169310+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:53:50.560 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6024 8 365281 365273 0.473685 0 - 0.0299513 2026-02-23T14:53:51.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:53:50 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/985537388' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:53:51.560 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6025 8 365281 365273 0.473606 0 - 0.0299513 2026-02-23T14:53:52.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:53:51 trial068 ceph-mon[38805]: pgmap v3614: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:53:52.560 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6026 8 365281 365273 0.473527 0 - 0.0299513 2026-02-23T14:53:53.434 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:53:53.551 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:53:53.560 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6027 8 365281 365273 0.473449 0 - 0.0299513 2026-02-23T14:53:53.649 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:53:53.648+0000 7ff2ebedc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:53:53.650 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:53:53.648+0000 7ff2ebedc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:53:53.759 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:53:53.759 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":273,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:51:34.800555+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:50:36.055613+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:48:34.098389+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:49:22.169310+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:53:54.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:53:53 trial068 ceph-mon[38805]: pgmap v3615: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:53:54.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:53:53 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/188704328' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:53:54.094 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:53:53 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:53:53] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:53:54.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:53:53 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:53:53.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:53:54.560 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6028 8 365281 365273 0.47337 0 - 0.0299513 2026-02-23T14:53:55.560 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6029 8 365281 365273 0.473292 0 - 0.0299513 2026-02-23T14:53:56.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:53:55 trial068 ceph-mon[38805]: pgmap v3616: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:53:56.560 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6030 8 365281 365273 0.473213 0 - 0.0299513 2026-02-23T14:53:56.797 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:53:56.917 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:53:57.023 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:53:57.021+0000 7fdb9d4ce640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:53:57.023 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:53:57.021+0000 7fdb9d4ce640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:53:57.133 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:53:57.133 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":273,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:51:34.800555+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:50:36.055613+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:48:34.098389+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:49:22.169310+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:53:57.560 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6031 8 365281 365273 0.473135 0 - 0.0299513 2026-02-23T14:53:58.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:53:57 trial068 ceph-mon[38805]: pgmap v3617: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:53:58.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:53:57 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2157915764' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:53:58.560 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6032 8 365281 365273 0.473056 0 - 0.0299513 2026-02-23T14:53:59.560 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6033 8 365281 365273 0.472978 0 - 0.0299513 2026-02-23T14:54:00.172 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:54:00.196 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:53:59 trial068 ceph-mon[38805]: pgmap v3618: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:54:00.293 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:54:00.390 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:54:00.388+0000 7fdb3a858640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:54:00.390 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:54:00.388+0000 7fdb3a858640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:54:00.500 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:54:00.500 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":273,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:51:34.800555+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:50:36.055613+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:48:34.098389+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:49:22.169310+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:54:00.560 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6034 8 365281 365273 0.4729 0 - 0.0299513 2026-02-23T14:54:01.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:54:00 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2249958195' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:54:01.561 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6035 8 365281 365273 0.472821 0 - 0.0299513 2026-02-23T14:54:02.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:54:01 trial068 ceph-mon[38805]: pgmap v3619: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:54:02.561 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6036 8 365281 365273 0.472743 0 - 0.0299513 2026-02-23T14:54:03.537 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:54:03.561 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6037 8 365281 365273 0.472665 0 - 0.0299513 2026-02-23T14:54:03.657 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:54:03.755 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:54:03.753+0000 7fe758a6a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:54:03.755 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:54:03.753+0000 7fe758a6a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:54:03.865 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:54:03.866 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":273,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:51:34.800555+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:50:36.055613+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:48:34.098389+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:49:22.169310+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:54:04.265 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:54:03 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:54:03.972Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:54:04.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:54:03 trial068 ceph-mon[38805]: pgmap v3620: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:54:04.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:54:03 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3705373935' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:54:04.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:54:03 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:54:03] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:54:04.561 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6038 8 365281 365273 0.472586 0 - 0.0299513 2026-02-23T14:54:05.561 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6039 8 365281 365273 0.472508 0 - 0.0299513 2026-02-23T14:54:06.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:54:05 trial068 ceph-mon[38805]: pgmap v3621: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:54:06.561 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T14:54:06.560727+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T14:54:06.561 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T14:54:06.561 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6040 8 365281 365273 0.47243 0 - 0.0299513 2026-02-23T14:54:06.924 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:54:07.043 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:54:07.144 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:54:07.142+0000 7f127b3c5640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:54:07.144 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:54:07.142+0000 7f127b3c5640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:54:07.253 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:54:07.254 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":273,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:51:34.800555+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:50:36.055613+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:48:34.098389+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:49:22.169310+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:54:07.561 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6041 8 365281 365273 0.472352 0 - 0.0299513 2026-02-23T14:54:08.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:54:07 trial068 ceph-mon[38805]: pgmap v3622: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:54:08.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:54:07 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3760205114' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:54:08.561 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6042 8 365281 365273 0.472274 0 - 0.0299513 2026-02-23T14:54:09.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:54:08 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2026-02-23T14:54:09.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:54:08 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-23T14:54:09.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:54:08 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2026-02-23T14:54:09.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:54:08 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T14:54:09.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:54:08 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2026-02-23T14:54:09.561 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6043 8 365281 365273 0.472195 0 - 0.0299513 2026-02-23T14:54:10.290 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:54:10.312 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:54:09 trial068 ceph-mon[38805]: pgmap v3623: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:54:10.312 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:54:09 trial068 ceph-mon[38805]: pgmap v3624: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:54:10.312 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:54:09 trial068 ceph-mon[38805]: pgmap v3625: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:54:10.312 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:54:09 trial068 ceph-mon[38805]: osdmap e274: 4 total, 3 up, 4 in 2026-02-23T14:54:10.312 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:54:09 trial068 ceph-mon[38805]: osdmap e275: 4 total, 3 up, 4 in 2026-02-23T14:54:10.405 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:54:10.505 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:54:10.503+0000 7f7c0e1e1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:54:10.505 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:54:10.503+0000 7f7c0e1e1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:54:10.561 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6044 8 365281 365273 0.472117 0 - 0.0299513 2026-02-23T14:54:10.614 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:54:10.614 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":275,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:54:09.822525+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:50:36.055613+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:53:39.251641+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:49:22.169310+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:54:11.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:54:10 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1129963352' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:54:11.561 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6045 8 365281 365273 0.472039 0 - 0.0299513 2026-02-23T14:54:12.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:54:11 trial068 ceph-mon[38805]: pgmap v3628: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:54:12.561 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6046 8 365281 365273 0.471961 0 - 0.0299513 2026-02-23T14:54:13.561 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6047 8 365281 365273 0.471883 0 - 0.0299513 2026-02-23T14:54:13.666 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:54:13.785 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:54:13.889 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:54:13.887+0000 7fcabefb6640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:54:13.889 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:54:13.887+0000 7fcabefb6640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:54:13.973 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:54:13 trial068 ceph-mon[38805]: 5.8 scrub starts 2026-02-23T14:54:13.973 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:54:13 trial068 ceph-mon[38805]: pgmap v3629: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:54:13.973 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:54:13 trial068 ceph-mon[38805]: 5.8 scrub ok 2026-02-23T14:54:14.000 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:54:14.000 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":275,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:54:09.822525+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:50:36.055613+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:53:39.251641+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:49:22.169310+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:54:14.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:54:13 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:54:13] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:54:14.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:54:13 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:54:13.972Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:54:14.561 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6048 8 365281 365273 0.471805 0 - 0.0299513 2026-02-23T14:54:15.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:54:14 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3333651973' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:54:15.562 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6049 8 365281 365273 0.471727 0 - 0.0299513 2026-02-23T14:54:16.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:54:15 trial068 ceph-mon[38805]: pgmap v3630: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:54:16.562 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6050 8 365281 365273 0.471649 0 - 0.0299513 2026-02-23T14:54:17.038 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:54:17.157 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:54:17.261 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:54:17.259+0000 7f6af9f0a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:54:17.261 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:54:17.259+0000 7f6af9f0a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:54:17.373 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:54:17.373 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":275,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:54:09.822525+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:50:36.055613+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:53:39.251641+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:49:22.169310+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:54:17.562 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6051 8 365281 365273 0.471571 0 - 0.0299513 2026-02-23T14:54:18.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:54:17 trial068 ceph-mon[38805]: pgmap v3631: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:54:18.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:54:17 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/282107498' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:54:18.562 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6052 8 365281 365273 0.471493 0 - 0.0299513 2026-02-23T14:54:19.562 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6053 8 365281 365273 0.471415 0 - 0.0299513 2026-02-23T14:54:20.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:54:19 trial068 ceph-mon[38805]: pgmap v3632: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:54:20.411 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:54:20.529 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:54:20.562 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6054 8 365281 365273 0.471337 0 - 0.0299513 2026-02-23T14:54:20.629 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:54:20.627+0000 7f17b1b43640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:54:20.630 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:54:20.628+0000 7f17b1b43640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:54:20.737 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:54:20.737 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":275,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:54:09.822525+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:50:36.055613+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:53:39.251641+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:49:22.169310+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:54:21.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:54:20 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2840185212' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:54:21.562 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6055 8 365281 365273 0.47126 0 - 0.0299513 2026-02-23T14:54:22.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:54:21 trial068 ceph-mon[38805]: pgmap v3633: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:54:22.562 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6056 8 365281 365273 0.471182 0 - 0.0299513 2026-02-23T14:54:23.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:54:22 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T14:54:23.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:54:22 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T14:54:23.562 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6057 8 365281 365273 0.471104 0 - 0.0299513 2026-02-23T14:54:23.793 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:54:23.911 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:54:24.014 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:54:24.012+0000 7f60b7526640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:54:24.015 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:54:24.013+0000 7f60b7526640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:54:24.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:54:23 trial068 ceph-mon[38805]: pgmap v3634: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:54:24.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:54:23 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T14:54:24.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:54:23 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T14:54:24.094 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:54:23 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:54:23] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:54:24.125 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:54:24.125 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":275,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:54:09.822525+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:50:36.055613+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:53:39.251641+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:49:22.169310+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:54:24.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:54:23 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:54:23.972Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:54:24.562 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6058 8 365281 365273 0.471026 0 - 0.0299513 2026-02-23T14:54:25.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:54:24 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1770040227' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:54:25.562 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6059 8 365281 365273 0.470948 0 - 0.0299513 2026-02-23T14:54:26.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:54:25 trial068 ceph-mon[38805]: pgmap v3635: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:54:26.562 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T14:54:26.562189+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T14:54:26.562 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T14:54:26.563 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6060 8 365281 365273 0.470871 0 - 0.0299513 2026-02-23T14:54:27.163 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:54:27.188 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:54:26 trial068 ceph-mon[38805]: 5.5 scrub starts 2026-02-23T14:54:27.188 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:54:26 trial068 ceph-mon[38805]: 5.5 scrub ok 2026-02-23T14:54:27.280 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:54:27.384 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:54:27.382+0000 7fcd9d67c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:54:27.384 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:54:27.382+0000 7fcd9d67c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:54:27.498 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:54:27.498 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":275,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:54:09.822525+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:50:36.055613+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:53:39.251641+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:49:22.169310+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:54:27.562 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6061 8 365281 365273 0.470793 0 - 0.0299513 2026-02-23T14:54:28.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:54:27 trial068 ceph-mon[38805]: pgmap v3636: 17 pgs: 1 active+clean+scrubbing, 1 active+recovery_unfound+undersized+degraded, 4 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:54:28.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:54:27 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1444164360' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:54:28.563 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6062 8 365281 365273 0.470715 0 - 0.0299513 2026-02-23T14:54:29.563 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6063 8 365281 365273 0.470638 0 - 0.0299513 2026-02-23T14:54:30.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:54:29 trial068 ceph-mon[38805]: pgmap v3637: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 1 active+clean+scrubbing, 4 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:54:30.538 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:54:30.563 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6064 8 365281 365273 0.47056 0 - 0.0299513 2026-02-23T14:54:30.654 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:54:30.752 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:54:30.750+0000 7f3c111f7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:54:30.752 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:54:30.750+0000 7f3c111f7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:54:30.861 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:54:30.862 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":275,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:54:09.822525+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:50:36.055613+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:53:39.251641+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:49:22.169310+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:54:31.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:54:30 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/777081907' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:54:31.563 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6065 8 365281 365273 0.470483 0 - 0.0299513 2026-02-23T14:54:32.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:54:31 trial068 ceph-mon[38805]: 5.a scrub starts 2026-02-23T14:54:32.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:54:31 trial068 ceph-mon[38805]: pgmap v3638: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 1 active+clean+scrubbing, 4 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:54:32.345 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:54:31 trial068 ceph-mon[38805]: 5.a scrub ok 2026-02-23T14:54:32.563 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6066 8 365281 365273 0.470405 0 - 0.0299513 2026-02-23T14:54:33.563 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6067 8 365281 365273 0.470327 0 - 0.0299513 2026-02-23T14:54:33.898 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:54:34.015 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:54:34.113 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:54:34.111+0000 7fe8769ea640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:54:34.113 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:54:34.111+0000 7fe8769ea640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:54:34.221 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:54:34.221 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":275,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:54:09.822525+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:50:36.055613+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:53:39.251641+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:49:22.169310+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:54:34.221 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:54:33 trial068 ceph-mon[38805]: pgmap v3639: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:54:34.222 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:54:33 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:54:33] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:54:34.266 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:54:33 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:54:33.972Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:54:34.563 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6068 8 365281 365273 0.47025 0 - 0.0299513 2026-02-23T14:54:35.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:54:34 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2002751303' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:54:35.563 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6069 8 365281 365273 0.470172 0 - 0.0299513 2026-02-23T14:54:36.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:54:35 trial068 ceph-mon[38805]: pgmap v3640: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:54:36.563 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6070 8 365281 365273 0.470095 0 - 0.0299513 2026-02-23T14:54:37.258 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:54:37.376 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:54:37.472 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:54:37.470+0000 7fdae8e54640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:54:37.472 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:54:37.470+0000 7fdae8e54640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:54:37.563 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6071 8 365281 365273 0.470018 0 - 0.0299513 2026-02-23T14:54:37.582 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:54:37.582 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":275,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:54:09.822525+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:50:36.055613+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:53:39.251641+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:49:22.169310+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:54:38.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:54:37 trial068 ceph-mon[38805]: pgmap v3641: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:54:38.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:54:37 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1835093513' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:54:38.563 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6072 8 365281 365273 0.46994 0 - 0.0299513 2026-02-23T14:54:39.563 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6073 8 365281 365273 0.469863 0 - 0.0299513 2026-02-23T14:54:40.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:54:39 trial068 ceph-mon[38805]: pgmap v3642: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:54:40.563 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6074 8 365281 365273 0.469785 0 - 0.0299513 2026-02-23T14:54:40.618 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:54:40.734 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:54:40.832 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:54:40.830+0000 7fca65b67640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:54:40.832 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:54:40.831+0000 7fca65b67640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:54:40.943 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:54:40.943 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":275,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:54:09.822525+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:50:36.055613+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:53:39.251641+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:49:22.169310+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:54:41.564 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6075 8 365281 365273 0.469708 0 - 0.0299513 2026-02-23T14:54:42.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:54:41 trial068 ceph-mon[38805]: pgmap v3643: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:54:42.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:54:41 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1043875057' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:54:42.564 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6076 8 365281 365273 0.469631 0 - 0.0299513 2026-02-23T14:54:43.564 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6077 8 365281 365273 0.469554 0 - 0.0299513 2026-02-23T14:54:43.979 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:54:44.097 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:54:44.195 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:54:44.193+0000 7f9c0860a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:54:44.195 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:54:44.193+0000 7f9c0860a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:54:44.196 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:54:43 trial068 ceph-mon[38805]: 5.7 scrub starts 2026-02-23T14:54:44.196 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:54:43 trial068 ceph-mon[38805]: pgmap v3644: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:54:44.196 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:54:43 trial068 ceph-mon[38805]: 5.7 scrub ok 2026-02-23T14:54:44.196 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:54:43 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:54:43] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:54:44.305 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:54:44.306 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":275,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:54:09.822525+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:50:36.055613+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:53:39.251641+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:49:22.169310+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:54:44.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:54:43 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:54:43.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:54:44.564 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6078 8 365281 365273 0.469476 0 - 0.0299513 2026-02-23T14:54:45.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:54:44 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2973365005' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:54:45.564 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6079 8 365281 365273 0.469399 0 - 0.0299513 2026-02-23T14:54:46.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:54:45 trial068 ceph-mon[38805]: pgmap v3645: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:54:46.564 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T14:54:46.563693+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T14:54:46.564 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T14:54:46.564 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6080 8 365281 365273 0.469322 0 - 0.0299513 2026-02-23T14:54:47.345 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:54:47.464 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:54:47.564 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6081 8 365281 365273 0.469245 0 - 0.0299513 2026-02-23T14:54:47.564 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:54:47.563+0000 7fa66cdc2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:54:47.565 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:54:47.563+0000 7fa66cdc2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:54:47.677 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:54:47.677 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":275,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:54:09.822525+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:50:36.055613+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:53:39.251641+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:49:22.169310+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:54:48.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:54:47 trial068 ceph-mon[38805]: pgmap v3646: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:54:48.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:54:47 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1974308270' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:54:48.564 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6082 8 365281 365273 0.469167 0 - 0.0299513 2026-02-23T14:54:49.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:54:48 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T14:54:49.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:54:48 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T14:54:49.565 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6083 8 365281 365273 0.46909 0 - 0.0299513 2026-02-23T14:54:50.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:54:49 trial068 ceph-mon[38805]: pgmap v3647: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:54:50.565 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6084 8 365281 365273 0.469013 0 - 0.0299513 2026-02-23T14:54:50.715 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:54:50.833 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:54:50.936 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:54:50.934+0000 7f788beaa640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:54:50.936 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:54:50.934+0000 7f788beaa640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:54:51.046 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:54:51.047 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":275,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:54:09.822525+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:50:36.055613+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:53:39.251641+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:49:22.169310+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:54:51.565 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6085 8 365281 365273 0.468936 0 - 0.0299513 2026-02-23T14:54:52.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:54:51 trial068 ceph-mon[38805]: pgmap v3648: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:54:52.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:54:51 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1065128667' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:54:52.565 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6086 8 365281 365273 0.468859 0 - 0.0299513 2026-02-23T14:54:53.565 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6087 8 365281 365273 0.468782 0 - 0.0299513 2026-02-23T14:54:54.085 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:54:54.202 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:54:54.224 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:54:53 trial068 ceph-mon[38805]: pgmap v3649: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:54:54.224 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:54:53 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:54:53] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:54:54.300 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:54:54.298+0000 7fd9dc2a1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:54:54.300 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:54:54.299+0000 7fd9dc2a1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:54:54.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:54:53 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:54:53.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:54:54.409 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:54:54.409 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":275,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:54:09.822525+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:50:36.055613+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:53:39.251641+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:49:22.169310+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:54:54.565 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6088 8 365281 365273 0.468705 0 - 0.0299513 2026-02-23T14:54:55.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:54:54 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/605679789' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:54:55.565 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6089 8 365281 365273 0.468628 0 - 0.0299513 2026-02-23T14:54:56.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:54:55 trial068 ceph-mon[38805]: pgmap v3650: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:54:56.565 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6090 8 365281 365273 0.468551 0 - 0.0299513 2026-02-23T14:54:57.449 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:54:57.565 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6091 8 365281 365273 0.468474 0 - 0.0299513 2026-02-23T14:54:57.567 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:54:57.670 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:54:57.668+0000 7f871d1a9640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:54:57.670 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:54:57.668+0000 7f871d1a9640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:54:57.779 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:54:57.779 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":275,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:54:09.822525+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:50:36.055613+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:53:39.251641+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:49:22.169310+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:54:58.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:54:57 trial068 ceph-mon[38805]: pgmap v3651: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:54:58.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:54:57 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1560273993' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:54:58.565 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6092 8 365281 365273 0.468397 0 - 0.0299513 2026-02-23T14:54:59.565 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6093 8 365281 365273 0.46832 0 - 0.0299513 2026-02-23T14:55:00.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:54:59 trial068 ceph-mon[38805]: pgmap v3652: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:55:00.565 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6094 8 365281 365273 0.468244 0 - 0.0299513 2026-02-23T14:55:00.815 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:55:00.933 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:55:01.033 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:55:01.031+0000 7f52e393c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:55:01.034 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:55:01.032+0000 7f52e393c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:55:01.142 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:55:01.142 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":275,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:54:09.822525+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:50:36.055613+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:53:39.251641+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:49:22.169310+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:55:01.566 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6095 8 365281 365273 0.468167 0 - 0.0299513 2026-02-23T14:55:02.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:55:01 trial068 ceph-mon[38805]: pgmap v3653: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:55:02.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:55:01 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2573603684' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:55:02.565 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6096 8 365281 365273 0.46809 0 - 0.0299513 2026-02-23T14:55:03.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:55:02 trial068 ceph-mon[38805]: 5.e scrub starts 2026-02-23T14:55:03.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:55:02 trial068 ceph-mon[38805]: 5.e scrub ok 2026-02-23T14:55:03.565 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6097 8 365281 365273 0.468013 0 - 0.0299513 2026-02-23T14:55:04.180 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:55:04.205 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:55:03 trial068 ceph-mon[38805]: pgmap v3654: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:55:04.205 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:55:03 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:55:03] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:55:04.265 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:55:03 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:55:03.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:55:04.298 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:55:04.407 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:55:04.405+0000 7f32489ac640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:55:04.407 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:55:04.405+0000 7f32489ac640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:55:04.516 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:55:04.516 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":275,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:54:09.822525+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:50:36.055613+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:53:39.251641+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:49:22.169310+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:55:04.566 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6098 8 365281 365273 0.467936 0 - 0.0299513 2026-02-23T14:55:05.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:55:04 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1944229683' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:55:05.566 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6099 8 365281 365273 0.46786 0 - 0.0299513 2026-02-23T14:55:06.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:55:05 trial068 ceph-mon[38805]: pgmap v3655: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:55:06.566 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T14:55:06.565647+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T14:55:06.566 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T14:55:06.566 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6100 8 365281 365273 0.467783 0 - 0.0299513 2026-02-23T14:55:07.553 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:55:07.566 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6101 8 365281 365273 0.467706 0 - 0.0299513 2026-02-23T14:55:07.670 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:55:07.772 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:55:07.770+0000 7f689b913640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:55:07.772 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:55:07.771+0000 7f689b913640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:55:07.883 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:55:07.883 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":275,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:54:09.822525+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:50:36.055613+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:53:39.251641+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:49:22.169310+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:55:08.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:55:07 trial068 ceph-mon[38805]: pgmap v3656: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:55:08.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:55:07 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3476152719' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:55:08.566 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6102 8 365281 365273 0.46763 0 - 0.0299513 2026-02-23T14:55:09.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:55:08 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2026-02-23T14:55:09.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:55:08 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-23T14:55:09.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:55:08 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2026-02-23T14:55:09.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:55:08 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T14:55:09.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:55:08 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2026-02-23T14:55:09.566 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6103 8 365281 365273 0.467553 0 - 0.0299513 2026-02-23T14:55:10.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:55:09 trial068 ceph-mon[38805]: pgmap v3657: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:55:10.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:55:09 trial068 ceph-mon[38805]: pgmap v3658: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:55:10.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:55:09 trial068 ceph-mon[38805]: pgmap v3659: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:55:10.566 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6104 8 365281 365273 0.467476 0 - 0.0299513 2026-02-23T14:55:10.935 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:55:11.052 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:55:11.161 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:55:11.158+0000 7f014182b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:55:11.161 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:55:11.159+0000 7f014182b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:55:11.270 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:55:11.270 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":275,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:54:09.822525+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:50:36.055613+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:53:39.251641+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:49:22.169310+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:55:11.271 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:55:10 trial068 ceph-mon[38805]: pgmap v3660: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:55:11.566 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6105 8 365281 365273 0.4674 0 - 0.0299513 2026-02-23T14:55:12.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:55:11 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2553491366' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:55:12.566 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6106 8 365281 365273 0.467323 0 - 0.0299513 2026-02-23T14:55:13.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:55:12 trial068 ceph-mon[38805]: pgmap v3661: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:55:13.566 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6107 8 365281 365273 0.467247 0 - 0.0299513 2026-02-23T14:55:14.244 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:55:13 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:55:13] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:55:14.307 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:55:14.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:55:13 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:55:13.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:55:14.427 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:55:14.533 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:55:14.531+0000 7fe2fc4a5640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:55:14.534 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:55:14.532+0000 7fe2fc4a5640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:55:14.566 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6108 8 365281 365273 0.46717 0 - 0.0299513 2026-02-23T14:55:14.645 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:55:14.645 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":275,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:54:09.822525+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:50:36.055613+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:53:39.251641+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:49:22.169310+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:55:15.566 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6109 8 365281 365273 0.467094 0 - 0.0299513 2026-02-23T14:55:16.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:55:15 trial068 ceph-mon[38805]: pgmap v3662: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:55:16.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:55:15 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2310882015' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:55:16.566 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6110 8 365281 365273 0.467017 0 - 0.0299513 2026-02-23T14:55:17.567 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6111 8 365281 365273 0.466941 0 - 0.0299513 2026-02-23T14:55:17.700 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:55:17.817 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:55:17.871 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:55:17 trial068 ceph-mon[38805]: pgmap v3663: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:55:17.916 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:55:17.914+0000 7f2f75ff5640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:55:17.916 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:55:17.914+0000 7f2f75ff5640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:55:18.024 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:55:18.024 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":275,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:54:09.822525+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:50:36.055613+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:53:39.251641+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:49:22.169310+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:55:18.567 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6112 8 365281 365273 0.466865 0 - 0.0299513 2026-02-23T14:55:19.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:55:18 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/666335039' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:55:19.567 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6113 8 365281 365273 0.466788 0 - 0.0299513 2026-02-23T14:55:20.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:55:19 trial068 ceph-mon[38805]: pgmap v3664: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:55:20.567 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6114 8 365281 365273 0.466712 0 - 0.0299513 2026-02-23T14:55:21.066 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:55:21.183 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:55:21.282 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:55:21.281+0000 7ffbdae8e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:55:21.283 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:55:21.281+0000 7ffbdae8e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:55:21.391 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:55:21.391 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":275,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:54:09.822525+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:50:36.055613+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:53:39.251641+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:49:22.169310+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:55:21.567 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6115 8 365281 365273 0.466636 0 - 0.0299513 2026-02-23T14:55:22.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:55:21 trial068 ceph-mon[38805]: pgmap v3665: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:55:22.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:55:21 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3675940897' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:55:22.567 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6116 8 365281 365273 0.466559 0 - 0.0299513 2026-02-23T14:55:23.567 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6117 8 365281 365273 0.466483 0 - 0.0299513 2026-02-23T14:55:24.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:55:23 trial068 ceph-mon[38805]: pgmap v3666: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:55:24.094 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:55:23 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:55:23] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:55:24.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:55:23 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:55:23.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:55:24.428 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:55:24.546 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:55:24.567 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6118 8 365281 365273 0.466407 0 - 0.0299513 2026-02-23T14:55:24.644 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:55:24.642+0000 7f49470f1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:55:24.645 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:55:24.643+0000 7f49470f1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:55:24.753 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:55:24.753 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":275,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:54:09.822525+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:50:36.055613+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:53:39.251641+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:49:22.169310+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:55:25.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:55:24 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3781517306' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:55:25.567 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6119 8 365281 365273 0.466331 0 - 0.0299513 2026-02-23T14:55:26.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:55:25 trial068 ceph-mon[38805]: pgmap v3667: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:55:26.567 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T14:55:26.567153+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T14:55:26.567 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T14:55:26.568 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6120 8 365281 365273 0.466254 0 - 0.0299513 2026-02-23T14:55:27.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:55:26 trial068 ceph-mon[38805]: osdmap e276: 4 total, 3 up, 4 in 2026-02-23T14:55:27.567 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6121 8 365281 365273 0.466178 0 - 0.0299513 2026-02-23T14:55:27.790 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:55:27.908 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:55:28.007 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:55:28.005+0000 7f4b942fc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:55:28.007 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:55:28.006+0000 7f4b942fc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:55:28.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:55:27 trial068 ceph-mon[38805]: pgmap v3669: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:55:28.116 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:55:28.116 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":276,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:55:25.837873+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:50:36.055613+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:53:39.251641+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:54:21.339477+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:55:28.567 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6122 8 365281 365273 0.466102 0 - 0.0299513 2026-02-23T14:55:29.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:55:28 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1359809659' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:55:29.568 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6123 8 365281 365273 0.466026 0 - 0.0299513 2026-02-23T14:55:30.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:55:29 trial068 ceph-mon[38805]: pgmap v3670: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:55:30.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:55:29 trial068 ceph-mon[38805]: osdmap e277: 4 total, 3 up, 4 in 2026-02-23T14:55:30.567 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6124 8 365281 365273 0.46595 0 - 0.0299513 2026-02-23T14:55:31.152 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:55:31.270 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:55:31.367 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:55:31.366+0000 7f62e0e08640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:55:31.368 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:55:31.366+0000 7f62e0e08640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:55:31.476 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:55:31.477 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":277,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:55:29.833972+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:50:36.055613+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:53:39.251641+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:54:21.339477+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:55:31.568 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6125 8 365281 365273 0.465874 0 - 0.0299513 2026-02-23T14:55:32.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:55:31 trial068 ceph-mon[38805]: pgmap v3672: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:55:32.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:55:31 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/561656290' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:55:32.568 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6126 8 365281 365273 0.465798 0 - 0.0299513 2026-02-23T14:55:33.568 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6127 8 365281 365273 0.465722 0 - 0.0299513 2026-02-23T14:55:34.265 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:55:33 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:55:33.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:55:34.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:55:33 trial068 ceph-mon[38805]: pgmap v3673: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:55:34.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:55:33 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:55:33] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:55:34.514 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:55:34.568 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6128 8 365281 365273 0.465646 0 - 0.0299513 2026-02-23T14:55:34.634 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:55:34.734 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:55:34.732+0000 7fed4f322640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:55:34.734 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:55:34.732+0000 7fed4f322640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:55:34.847 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:55:34.848 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":277,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:55:29.833972+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:50:36.055613+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:53:39.251641+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:54:21.339477+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:55:35.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:55:34 trial068 ceph-mon[38805]: 5.8 scrub starts 2026-02-23T14:55:35.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:55:34 trial068 ceph-mon[38805]: 5.8 scrub ok 2026-02-23T14:55:35.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:55:34 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3437519634' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:55:35.568 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6129 8 365281 365273 0.46557 0 - 0.0299513 2026-02-23T14:55:36.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:55:35 trial068 ceph-mon[38805]: pgmap v3674: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:55:36.568 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6130 8 365281 365273 0.465494 0 - 0.0299513 2026-02-23T14:55:37.568 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6131 8 365281 365273 0.465418 0 - 0.0299513 2026-02-23T14:55:37.901 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:55:38.020 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:55:38.127 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:55:38.125+0000 7f420922f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:55:38.127 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:55:37 trial068 ceph-mon[38805]: pgmap v3675: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:55:38.127 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:55:38.125+0000 7f420922f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:55:38.235 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:55:38.236 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":277,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:55:29.833972+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:50:36.055613+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:53:39.251641+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:54:21.339477+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:55:38.568 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6132 8 365281 365273 0.465342 0 - 0.0299513 2026-02-23T14:55:39.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:55:38 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/129491254' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:55:39.568 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6133 8 365281 365273 0.465266 0 - 0.0299513 2026-02-23T14:55:40.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:55:39 trial068 ceph-mon[38805]: pgmap v3676: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:55:40.568 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6134 8 365281 365273 0.46519 0 - 0.0299513 2026-02-23T14:55:41.274 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:55:41.393 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:55:41.495 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:55:41.493+0000 7f47c20f1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:55:41.495 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:55:41.493+0000 7f47c20f1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:55:41.568 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6135 8 365281 365273 0.465114 0 - 0.0299513 2026-02-23T14:55:41.605 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:55:41.606 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":277,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:55:29.833972+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:50:36.055613+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:53:39.251641+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:54:21.339477+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:55:42.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:55:41 trial068 ceph-mon[38805]: pgmap v3677: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:55:42.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:55:41 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/4023563194' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:55:42.568 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6136 8 365281 365273 0.465039 0 - 0.0299513 2026-02-23T14:55:43.569 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6137 8 365281 365273 0.464963 0 - 0.0299513 2026-02-23T14:55:44.146 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:55:43 trial068 ceph-mon[38805]: pgmap v3678: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:55:44.146 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:55:43 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:55:43] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:55:44.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:55:43 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:55:43.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:55:44.569 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6138 8 365281 365273 0.464887 0 - 0.0299513 2026-02-23T14:55:44.642 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:55:44.761 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:55:44.863 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:55:44.861+0000 7fe91ec20640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:55:44.863 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:55:44.861+0000 7fe91ec20640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:55:44.971 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:55:44.971 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":277,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:55:29.833972+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:50:36.055613+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:53:39.251641+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:54:21.339477+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:55:45.569 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6139 8 365281 365273 0.464811 0 - 0.0299513 2026-02-23T14:55:46.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:55:45 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T14:55:46.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:55:45 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T14:55:46.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:55:45 trial068 ceph-mon[38805]: pgmap v3679: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:55:46.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:55:45 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1135297811' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:55:46.569 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T14:55:46.568587+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T14:55:46.569 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T14:55:46.569 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6140 8 365281 365273 0.464736 0 - 0.0299513 2026-02-23T14:55:47.569 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6141 8 365281 365273 0.46466 0 - 0.0299513 2026-02-23T14:55:48.007 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:55:48.125 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:55:48.147 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:55:47 trial068 ceph-mon[38805]: pgmap v3680: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:55:48.148 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:55:47 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T14:55:48.148 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:55:47 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T14:55:48.230 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:55:48.228+0000 7f765e0b9640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:55:48.230 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:55:48.228+0000 7f765e0b9640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:55:48.340 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:55:48.341 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":277,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:55:29.833972+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:50:36.055613+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:53:39.251641+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:54:21.339477+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:55:48.569 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6142 8 365281 365273 0.464584 0 - 0.0299513 2026-02-23T14:55:49.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:55:48 trial068 ceph-mon[38805]: 5.a scrub starts 2026-02-23T14:55:49.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:55:48 trial068 ceph-mon[38805]: 5.a scrub ok 2026-02-23T14:55:49.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:55:48 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/823043260' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:55:49.569 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6143 8 365281 365273 0.464509 0 - 0.0299513 2026-02-23T14:55:50.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:55:49 trial068 ceph-mon[38805]: pgmap v3681: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:55:50.569 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6144 8 365281 365273 0.464433 0 - 0.0299513 2026-02-23T14:55:51.378 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:55:51.496 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:55:51.569 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6145 8 365281 365273 0.464357 0 - 0.0299513 2026-02-23T14:55:51.600 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:55:51.598+0000 7fe95d170640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:55:51.600 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:55:51.599+0000 7fe95d170640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:55:51.710 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:55:51.710 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":277,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:55:29.833972+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:50:36.055613+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:53:39.251641+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:54:21.339477+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:55:52.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:55:51 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T14:55:52.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:55:51 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T14:55:52.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:55:51 trial068 ceph-mon[38805]: pgmap v3682: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:55:52.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:55:51 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1021233603' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:55:52.569 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6146 8 365281 365273 0.464282 0 - 0.0299513 2026-02-23T14:55:53.569 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6147 8 365281 365273 0.464206 0 - 0.0299513 2026-02-23T14:55:54.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:55:53 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:55:53] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:55:54.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:55:53 trial068 ceph-mon[38805]: 5.5 scrub starts 2026-02-23T14:55:54.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:55:53 trial068 ceph-mon[38805]: 5.5 scrub ok 2026-02-23T14:55:54.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:55:53 trial068 ceph-mon[38805]: pgmap v3683: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:55:54.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:55:53 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:55:53.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:55:54.569 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6148 8 365281 365273 0.464131 0 - 0.0299513 2026-02-23T14:55:54.747 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:55:54.867 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:55:54.973 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:55:54.971+0000 7fe9bf71a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:55:54.973 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:55:54.971+0000 7fe9bf71a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:55:55.085 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:55:55.085 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":277,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:55:29.833972+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:50:36.055613+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:53:39.251641+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:54:21.339477+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:55:55.569 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6149 8 365281 365273 0.464055 0 - 0.0299513 2026-02-23T14:55:56.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:55:55 trial068 ceph-mon[38805]: pgmap v3684: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:55:56.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:55:55 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2634599690' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:55:56.569 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6150 8 365281 365273 0.46398 0 - 0.0299513 2026-02-23T14:55:57.570 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6151 8 365281 365273 0.463904 0 - 0.0299513 2026-02-23T14:55:58.123 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:55:58.146 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:55:57 trial068 ceph-mon[38805]: pgmap v3685: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:55:58.240 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:55:58.338 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:55:58.336+0000 7f7eaff55640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:55:58.338 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:55:58.336+0000 7f7eaff55640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:55:58.448 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:55:58.448 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":277,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:55:29.833972+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:50:36.055613+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:53:39.251641+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:54:21.339477+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:55:58.570 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6152 8 365281 365273 0.463829 0 - 0.0299513 2026-02-23T14:55:59.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:55:58 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/817207349' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:55:59.570 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6153 8 365281 365273 0.463754 0 - 0.0299513 2026-02-23T14:56:00.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:55:59 trial068 ceph-mon[38805]: pgmap v3686: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:56:00.570 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6154 8 365281 365273 0.463678 0 - 0.0299513 2026-02-23T14:56:01.485 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:56:01.570 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6155 8 365281 365273 0.463603 0 - 0.0299513 2026-02-23T14:56:01.603 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:56:01.702 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:56:01.700+0000 7f3d501ca640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:56:01.702 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:56:01.700+0000 7f3d501ca640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:56:01.810 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:56:01.810 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":277,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:55:29.833972+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:50:36.055613+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:53:39.251641+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:54:21.339477+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:56:02.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:56:01 trial068 ceph-mon[38805]: pgmap v3687: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:56:02.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:56:01 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/514341517' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:56:02.570 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6156 8 365281 365273 0.463528 0 - 0.0299513 2026-02-23T14:56:03.570 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6157 8 365281 365273 0.463452 0 - 0.0299513 2026-02-23T14:56:04.266 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:56:03 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:56:03.974Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:56:04.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:56:03 trial068 ceph-mon[38805]: pgmap v3688: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:56:04.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:56:03 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:56:03] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:56:04.570 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6158 8 365281 365273 0.463377 0 - 0.0299513 2026-02-23T14:56:04.848 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:56:04.967 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:56:05.064 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:56:05.062+0000 7fee6fef5640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:56:05.064 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:56:05.062+0000 7fee6fef5640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:56:05.173 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:56:05.174 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":277,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:55:29.833972+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:50:36.055613+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:53:39.251641+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:54:21.339477+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:56:05.570 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6159 8 365281 365273 0.463302 0 - 0.0299513 2026-02-23T14:56:06.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:56:05 trial068 ceph-mon[38805]: pgmap v3689: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:56:06.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:56:05 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1457453573' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:56:06.570 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T14:56:06.570154+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T14:56:06.571 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T14:56:06.571 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6160 8 365281 365273 0.463227 0 - 0.0299513 2026-02-23T14:56:07.570 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6161 8 365281 365273 0.463152 0 - 0.0299513 2026-02-23T14:56:08.226 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:56:08.250 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:56:07 trial068 ceph-mon[38805]: pgmap v3690: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:56:08.344 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:56:08.445 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:56:08.443+0000 7f9ff87bb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:56:08.445 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:56:08.443+0000 7f9ff87bb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:56:08.555 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:56:08.555 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":277,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:55:29.833972+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:50:36.055613+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:53:39.251641+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:54:21.339477+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:56:08.570 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6162 8 365281 365273 0.463076 0 - 0.0299513 2026-02-23T14:56:09.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:56:08 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/458703048' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:56:09.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:56:08 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2026-02-23T14:56:09.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:56:08 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-23T14:56:09.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:56:08 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2026-02-23T14:56:09.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:56:08 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T14:56:09.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:56:08 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2026-02-23T14:56:09.571 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6163 8 365281 365273 0.463001 0 - 0.0299513 2026-02-23T14:56:10.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:56:09 trial068 ceph-mon[38805]: pgmap v3691: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:56:10.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:56:09 trial068 ceph-mon[38805]: pgmap v3692: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:56:10.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:56:09 trial068 ceph-mon[38805]: pgmap v3693: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:56:10.571 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6164 8 365281 365273 0.462926 0 - 0.0299513 2026-02-23T14:56:11.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:56:10 trial068 ceph-mon[38805]: 5.7 scrub starts 2026-02-23T14:56:11.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:56:10 trial068 ceph-mon[38805]: 5.7 scrub ok 2026-02-23T14:56:11.571 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6165 8 365281 365273 0.462851 0 - 0.0299513 2026-02-23T14:56:11.593 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:56:11.710 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:56:11.811 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:56:11.809+0000 7f6ba205e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:56:11.811 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:56:11.809+0000 7f6ba205e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:56:11.921 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:56:11.921 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":277,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:55:29.833972+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:50:36.055613+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:53:39.251641+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:54:21.339477+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:56:12.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:56:11 trial068 ceph-mon[38805]: pgmap v3694: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:56:12.571 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6166 8 365281 365273 0.462776 0 - 0.0299513 2026-02-23T14:56:13.251 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:56:12 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2925059732' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:56:13.571 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6167 8 365281 365273 0.462701 0 - 0.0299513 2026-02-23T14:56:14.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:56:13 trial068 ceph-mon[38805]: pgmap v3695: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:56:14.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:56:13 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:56:13] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:56:14.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:56:13 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:56:13.972Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:56:14.571 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6168 8 365281 365273 0.462626 0 - 0.0299513 2026-02-23T14:56:14.959 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:56:15.079 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:56:15.179 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:56:15.177+0000 7fe829b52640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:56:15.179 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:56:15.177+0000 7fe829b52640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:56:15.290 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:56:15.291 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":277,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:55:29.833972+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:50:36.055613+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:53:39.251641+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:54:21.339477+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:56:15.571 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6169 8 365281 365273 0.462551 0 - 0.0299513 2026-02-23T14:56:16.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:56:15 trial068 ceph-mon[38805]: pgmap v3696: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:56:16.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:56:15 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2789298366' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:56:16.571 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6170 8 365281 365273 0.462476 0 - 0.0299513 2026-02-23T14:56:17.571 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6171 8 365281 365273 0.462401 0 - 0.0299513 2026-02-23T14:56:18.329 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:56:18.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:56:17 trial068 ceph-mon[38805]: pgmap v3697: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:56:18.448 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:56:18.549 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:56:18.547+0000 7fe7a91fa640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:56:18.550 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:56:18.548+0000 7fe7a91fa640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:56:18.571 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6172 8 365281 365273 0.462326 0 - 0.0299513 2026-02-23T14:56:18.657 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:56:18.658 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":277,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:55:29.833972+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:50:36.055613+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:53:39.251641+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:54:21.339477+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:56:19.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:56:18 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1547604734' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:56:19.571 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6173 8 365281 365273 0.462251 0 - 0.0299513 2026-02-23T14:56:20.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:56:19 trial068 ceph-mon[38805]: pgmap v3698: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:56:20.571 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6174 8 365281 365273 0.462176 0 - 0.0299513 2026-02-23T14:56:21.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:56:20 trial068 ceph-mon[38805]: 5.e scrub starts 2026-02-23T14:56:21.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:56:20 trial068 ceph-mon[38805]: 5.e scrub ok 2026-02-23T14:56:21.571 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6175 8 365281 365273 0.462101 0 - 0.0299513 2026-02-23T14:56:21.694 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:56:21.812 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:56:21.911 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:56:21.909+0000 7fdb00de1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:56:21.911 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:56:21.909+0000 7fdb00de1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:56:22.018 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:56:22.019 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":277,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:55:29.833972+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:50:36.055613+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:53:39.251641+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:54:21.339477+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:56:22.019 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:56:21 trial068 ceph-mon[38805]: pgmap v3699: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:56:22.572 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6176 8 365281 365273 0.462027 0 - 0.0299513 2026-02-23T14:56:23.297 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:56:22 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3950715411' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:56:23.572 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6177 8 365281 365273 0.461952 0 - 0.0299513 2026-02-23T14:56:24.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:56:23 trial068 ceph-mon[38805]: pgmap v3700: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:56:24.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:56:23 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:56:23] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:56:24.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:56:23 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:56:23.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:56:24.572 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6178 8 365281 365273 0.461877 0 - 0.0299513 2026-02-23T14:56:25.055 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:56:25.174 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:56:25.276 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:56:25.274+0000 7f8065832640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:56:25.277 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:56:25.275+0000 7f8065832640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:56:25.384 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:56:25.384 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":277,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:55:29.833972+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:50:36.055613+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:53:39.251641+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:54:21.339477+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:56:25.572 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6179 8 365281 365273 0.461802 0 - 0.0299513 2026-02-23T14:56:26.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:56:25 trial068 ceph-mon[38805]: pgmap v3701: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:56:26.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:56:25 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/981004859' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:56:26.572 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T14:56:26.571869+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T14:56:26.572 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T14:56:26.572 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6180 8 365281 365273 0.461728 0 - 0.0299513 2026-02-23T14:56:27.572 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6181 8 365281 365273 0.461653 0 - 0.0299513 2026-02-23T14:56:28.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:56:27 trial068 ceph-mon[38805]: pgmap v3702: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:56:28.437 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:56:28.555 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:56:28.572 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6182 8 365281 365273 0.461578 0 - 0.0299513 2026-02-23T14:56:28.663 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:56:28.661+0000 7fe31b46b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:56:28.663 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:56:28.662+0000 7fe31b46b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:56:28.774 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:56:28.774 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":277,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:55:29.833972+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:50:36.055613+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:53:39.251641+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:54:21.339477+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:56:29.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:56:28 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3570234503' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:56:29.572 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6183 8 365281 365273 0.461504 0 - 0.0299513 2026-02-23T14:56:30.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:56:29 trial068 ceph-mon[38805]: pgmap v3703: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:56:30.572 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6184 8 365281 365273 0.461429 0 - 0.0299513 2026-02-23T14:56:31.572 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6185 8 365281 365273 0.461354 0 - 0.0299513 2026-02-23T14:56:31.814 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:56:31.934 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:56:32.040 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:56:32.038+0000 7faaef9b6640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:56:32.040 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:56:32.038+0000 7faaef9b6640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:56:32.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:56:31 trial068 ceph-mon[38805]: pgmap v3704: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:56:32.149 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:56:32.149 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":277,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:55:29.833972+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:50:36.055613+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:53:39.251641+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:54:21.339477+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:56:32.573 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6186 8 365281 365273 0.46128 0 - 0.0299513 2026-02-23T14:56:33.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:56:32 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3704189274' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:56:33.572 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6187 8 365281 365273 0.461205 0 - 0.0299513 2026-02-23T14:56:34.266 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:56:33 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:56:33.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:56:34.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:56:33 trial068 ceph-mon[38805]: pgmap v3705: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:56:34.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:56:33 trial068 ceph-mon[38805]: osdmap e278: 4 total, 3 up, 4 in 2026-02-23T14:56:34.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:56:33 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:56:33] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:56:34.572 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6188 8 365281 365273 0.461131 0 - 0.0299513 2026-02-23T14:56:35.187 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:56:35.308 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:56:35.406 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:56:35.404+0000 7f6b53691640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:56:35.407 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:56:35.405+0000 7f6b53691640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:56:35.515 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:56:35.516 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":279,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:56:34.843079+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:55:46.930221+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:53:39.251641+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:54:21.339477+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:56:35.573 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6189 8 365281 365273 0.461056 0 - 0.0299513 2026-02-23T14:56:36.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:56:35 trial068 ceph-mon[38805]: pgmap v3707: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:56:36.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:56:35 trial068 ceph-mon[38805]: osdmap e279: 4 total, 3 up, 4 in 2026-02-23T14:56:36.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:56:35 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2627871044' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:56:36.573 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6190 8 365281 365273 0.460982 0 - 0.0299513 2026-02-23T14:56:37.573 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6191 8 365281 365273 0.460907 0 - 0.0299513 2026-02-23T14:56:38.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:56:37 trial068 ceph-mon[38805]: pgmap v3709: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:56:38.554 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:56:38.573 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6192 8 365281 365273 0.460833 0 - 0.0299513 2026-02-23T14:56:38.672 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:56:38.772 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:56:38.770+0000 7fbef39d6640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:56:38.772 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:56:38.770+0000 7fbef39d6640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:56:38.881 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:56:38.881 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":279,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:56:34.843079+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:55:46.930221+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:53:39.251641+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:54:21.339477+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:56:39.573 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6193 8 365281 365273 0.460758 0 - 0.0299513 2026-02-23T14:56:40.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:56:39 trial068 ceph-mon[38805]: pgmap v3710: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:56:40.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:56:39 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1439279275' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:56:40.573 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6194 8 365281 365273 0.460684 0 - 0.0299513 2026-02-23T14:56:41.573 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6195 8 365281 365273 0.46061 0 - 0.0299513 2026-02-23T14:56:41.918 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:56:42.037 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:56:42.101 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:56:41 trial068 ceph-mon[38805]: pgmap v3711: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:56:42.144 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:56:42.142+0000 7facfe295640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:56:42.144 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:56:42.142+0000 7facfe295640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:56:42.254 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:56:42.255 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":279,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:56:34.843079+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:55:46.930221+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:53:39.251641+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:54:21.339477+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:56:42.573 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6196 8 365281 365273 0.460535 0 - 0.0299513 2026-02-23T14:56:43.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:56:42 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/4100783335' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:56:43.573 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6197 8 365281 365273 0.460461 0 - 0.0299513 2026-02-23T14:56:44.245 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:56:43 trial068 ceph-mon[38805]: pgmap v3712: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:56:44.245 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:56:43 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:56:43] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:56:44.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:56:43 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:56:43.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:56:44.573 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6198 8 365281 365273 0.460387 0 - 0.0299513 2026-02-23T14:56:45.307 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:56:45.427 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:56:45.524 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:56:45.522+0000 7f35eb979640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:56:45.524 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:56:45.523+0000 7f35eb979640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:56:45.573 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6199 8 365281 365273 0.460312 0 - 0.0299513 2026-02-23T14:56:45.636 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:56:45.636 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":279,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:56:34.843079+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:55:46.930221+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:53:39.251641+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:54:21.339477+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:56:46.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:56:45 trial068 ceph-mon[38805]: pgmap v3713: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:56:46.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:56:45 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2179516336' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:56:46.573 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T14:56:46.573156+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T14:56:46.574 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T14:56:46.574 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6200 8 365281 365273 0.460238 0 - 0.0299513 2026-02-23T14:56:47.573 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6201 8 365281 365273 0.460164 0 - 0.0299513 2026-02-23T14:56:48.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:56:47 trial068 ceph-mon[38805]: pgmap v3714: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:56:48.574 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6202 8 365281 365273 0.46009 0 - 0.0299513 2026-02-23T14:56:48.693 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:56:48.813 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:56:48.912 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:56:48.910+0000 7f3a0317f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:56:48.912 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:56:48.910+0000 7f3a0317f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:56:49.021 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:56:49.022 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":279,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:56:34.843079+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:55:46.930221+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:53:39.251641+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:54:21.339477+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:56:49.022 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:56:48 trial068 ceph-mon[38805]: 5.8 scrub starts 2026-02-23T14:56:49.022 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:56:48 trial068 ceph-mon[38805]: 5.8 scrub ok 2026-02-23T14:56:49.574 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6203 8 365281 365273 0.460016 0 - 0.0299513 2026-02-23T14:56:50.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:56:49 trial068 ceph-mon[38805]: pgmap v3715: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:56:50.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:56:49 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2711789236' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:56:50.574 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6204 8 365281 365273 0.459941 0 - 0.0299513 2026-02-23T14:56:51.574 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6205 8 365281 365273 0.459867 0 - 0.0299513 2026-02-23T14:56:52.074 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:56:52.194 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:56:52.216 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:56:51 trial068 ceph-mon[38805]: pgmap v3716: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:56:52.291 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:56:52.290+0000 7f5355481640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:56:52.292 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:56:52.290+0000 7f5355481640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:56:52.400 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:56:52.400 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":279,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:56:34.843079+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:55:46.930221+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:53:39.251641+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:54:21.339477+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:56:52.574 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6206 8 365281 365273 0.459793 0 - 0.0299513 2026-02-23T14:56:53.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:56:52 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1006635358' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:56:53.574 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6207 8 365281 365273 0.459719 0 - 0.0299513 2026-02-23T14:56:54.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:56:53 trial068 ceph-mon[38805]: pgmap v3717: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:56:54.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:56:53 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:56:53] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:56:54.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:56:53 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:56:53.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:56:54.574 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6208 8 365281 365273 0.459645 0 - 0.0299513 2026-02-23T14:56:55.436 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:56:55.554 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:56:55.574 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6209 8 365281 365273 0.459571 0 - 0.0299513 2026-02-23T14:56:55.650 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:56:55.648+0000 7f917a8c6640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:56:55.650 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:56:55.648+0000 7f917a8c6640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:56:55.762 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:56:55.762 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":279,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:56:34.843079+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:55:46.930221+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:53:39.251641+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:54:21.339477+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:56:56.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:56:55 trial068 ceph-mon[38805]: pgmap v3718: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:56:56.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:56:55 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3037521411' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:56:56.574 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6210 8 365281 365273 0.459497 0 - 0.0299513 2026-02-23T14:56:57.574 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6211 8 365281 365273 0.459423 0 - 0.0299513 2026-02-23T14:56:58.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:56:57 trial068 ceph-mon[38805]: pgmap v3719: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:56:58.574 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6212 8 365281 365273 0.459349 0 - 0.0299513 2026-02-23T14:56:58.800 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:56:58.918 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:56:59.016 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:56:59.014+0000 7f78915be640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:56:59.016 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:56:59.015+0000 7f78915be640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:56:59.126 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:56:59.126 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":279,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:56:34.843079+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:55:46.930221+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:53:39.251641+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:54:21.339477+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:56:59.574 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6213 8 365281 365273 0.459275 0 - 0.0299513 2026-02-23T14:57:00.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:56:59 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T14:57:00.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:56:59 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T14:57:00.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:56:59 trial068 ceph-mon[38805]: pgmap v3720: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:57:00.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:56:59 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/369616637' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:57:00.574 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6214 8 365281 365273 0.459201 0 - 0.0299513 2026-02-23T14:57:01.574 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6215 8 365281 365273 0.459127 0 - 0.0299513 2026-02-23T14:57:02.179 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:57:02.202 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:57:01 trial068 ceph-mon[38805]: 5.a scrub starts 2026-02-23T14:57:02.202 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:57:01 trial068 ceph-mon[38805]: 5.a scrub ok 2026-02-23T14:57:02.202 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:57:01 trial068 ceph-mon[38805]: pgmap v3721: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:57:02.296 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:57:02.400 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:57:02.398+0000 7fd0b1834640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:57:02.400 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:57:02.398+0000 7fd0b1834640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:57:02.508 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:57:02.508 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":279,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:56:34.843079+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:55:46.930221+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:53:39.251641+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:54:21.339477+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:57:02.574 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6216 8 365281 365273 0.459054 0 - 0.0299513 2026-02-23T14:57:03.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:57:02 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T14:57:03.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:57:02 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T14:57:03.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:57:02 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/679815159' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:57:03.574 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6217 8 365281 365273 0.45898 0 - 0.0299513 2026-02-23T14:57:04.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:57:03 trial068 ceph-mon[38805]: pgmap v3722: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:57:04.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:57:03 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:57:03] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:57:04.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:57:03 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:57:03.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:57:04.574 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6218 8 365281 365273 0.458906 0 - 0.0299513 2026-02-23T14:57:05.546 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:57:05.574 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6219 8 365281 365273 0.458832 0 - 0.0299513 2026-02-23T14:57:05.663 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:57:05.767 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:57:05.765+0000 7f1775459640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:57:05.767 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:57:05.765+0000 7f1775459640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:57:05.878 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:57:05.878 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":279,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:56:34.843079+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:55:46.930221+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:53:39.251641+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:54:21.339477+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:57:06.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:57:05 trial068 ceph-mon[38805]: pgmap v3723: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:57:06.575 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T14:57:06.574455+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T14:57:06.575 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T14:57:06.575 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6220 8 365281 365273 0.458758 0 - 0.0299513 2026-02-23T14:57:07.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:57:06 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2453215584' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:57:07.575 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6221 8 365281 365273 0.458685 0 - 0.0299513 2026-02-23T14:57:08.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:57:07 trial068 ceph-mon[38805]: pgmap v3724: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:57:08.575 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6222 8 365281 365273 0.458611 0 - 0.0299513 2026-02-23T14:57:08.917 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:57:09.039 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:57:09.145 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:57:09.143+0000 7f2f78e73640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:57:09.146 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:57:08 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2026-02-23T14:57:09.146 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:57:09.144+0000 7f2f78e73640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:57:09.255 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:57:09.255 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":279,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:56:34.843079+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:55:46.930221+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:53:39.251641+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:54:21.339477+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:57:09.575 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6223 8 365281 365273 0.458537 0 - 0.0299513 2026-02-23T14:57:10.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:57:09 trial068 ceph-mon[38805]: pgmap v3725: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:57:10.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:57:09 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T14:57:10.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:57:09 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T14:57:10.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:57:09 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T14:57:10.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:57:09 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T14:57:10.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:57:09 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/82252299' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:57:10.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:57:09 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-23T14:57:10.345 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:57:09 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2026-02-23T14:57:10.345 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:57:09 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T14:57:10.345 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:57:09 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2026-02-23T14:57:10.575 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6224 8 365281 365273 0.458463 0 - 0.0299513 2026-02-23T14:57:11.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:57:10 trial068 ceph-mon[38805]: pgmap v3726: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:57:11.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:57:10 trial068 ceph-mon[38805]: pgmap v3727: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:57:11.575 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6225 8 365281 365273 0.45839 0 - 0.0299513 2026-02-23T14:57:12.295 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:57:12.413 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:57:12.515 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:57:12.513+0000 7f79cd3a3640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:57:12.515 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:57:12.513+0000 7f79cd3a3640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:57:12.575 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6226 8 365281 365273 0.458316 0 - 0.0299513 2026-02-23T14:57:12.626 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:57:12.626 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":279,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:56:34.843079+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:55:46.930221+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:53:39.251641+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:54:21.339477+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:57:13.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:57:12 trial068 ceph-mon[38805]: pgmap v3728: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:57:13.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:57:12 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3497190223' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:57:13.575 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6227 8 365281 365273 0.458243 0 - 0.0299513 2026-02-23T14:57:14.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:57:13 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T14:57:14.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:57:13 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T14:57:14.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:57:13 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:57:13] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:57:14.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:57:13 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:57:13.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:57:14.575 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6228 8 365281 365273 0.458169 0 - 0.0299513 2026-02-23T14:57:15.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:57:14 trial068 ceph-mon[38805]: pgmap v3729: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:57:15.575 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6229 8 365281 365273 0.458095 0 - 0.0299513 2026-02-23T14:57:15.664 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:57:15.781 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:57:15.885 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:57:15.883+0000 7f23fcd50640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:57:15.885 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:57:15.883+0000 7f23fcd50640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:57:15.994 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:57:15.995 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":279,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:56:34.843079+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:55:46.930221+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:53:39.251641+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:54:21.339477+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:57:16.575 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6230 8 365281 365273 0.458022 0 - 0.0299513 2026-02-23T14:57:17.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:57:16 trial068 ceph-mon[38805]: pgmap v3730: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:57:17.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:57:16 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2783456990' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:57:17.575 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6231 8 365281 365273 0.457948 0 - 0.0299513 2026-02-23T14:57:18.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:57:17 trial068 ceph-mon[38805]: 5.7 scrub starts 2026-02-23T14:57:18.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:57:17 trial068 ceph-mon[38805]: 5.7 scrub ok 2026-02-23T14:57:18.575 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6232 8 365281 365273 0.457875 0 - 0.0299513 2026-02-23T14:57:19.032 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:57:19.150 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:57:19.174 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:57:18 trial068 ceph-mon[38805]: pgmap v3731: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:57:19.255 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:57:19.253+0000 7ff8394c9640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:57:19.256 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:57:19.254+0000 7ff8394c9640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:57:19.364 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:57:19.364 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":279,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:56:34.843079+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:55:46.930221+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:53:39.251641+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:54:21.339477+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:57:19.576 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6233 8 365281 365273 0.457802 0 - 0.0299513 2026-02-23T14:57:20.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:57:19 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2150714413' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:57:20.576 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6234 8 365281 365273 0.457728 0 - 0.0299513 2026-02-23T14:57:21.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:57:20 trial068 ceph-mon[38805]: pgmap v3732: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:57:21.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:57:20 trial068 ceph-mon[38805]: 5.5 scrub starts 2026-02-23T14:57:21.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:57:20 trial068 ceph-mon[38805]: 5.5 scrub ok 2026-02-23T14:57:21.576 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6235 8 365281 365273 0.457655 0 - 0.0299513 2026-02-23T14:57:22.401 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:57:22.519 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:57:22.576 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6236 8 365281 365273 0.457581 0 - 0.0299513 2026-02-23T14:57:22.622 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:57:22.620+0000 7f4076a5a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:57:22.622 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:57:22.620+0000 7f4076a5a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:57:22.731 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:57:22.731 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":279,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:56:34.843079+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:55:46.930221+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:53:39.251641+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:54:21.339477+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:57:23.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:57:22 trial068 ceph-mon[38805]: pgmap v3733: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:57:23.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:57:22 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/90173587' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:57:23.576 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6237 8 365281 365273 0.457508 0 - 0.0299513 2026-02-23T14:57:24.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:57:23 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:57:23] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:57:24.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:57:23 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:57:23.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:57:24.576 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6238 8 365281 365273 0.457435 0 - 0.0299513 2026-02-23T14:57:25.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:57:24 trial068 ceph-mon[38805]: pgmap v3734: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:57:25.576 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6239 8 365281 365273 0.457361 0 - 0.0299513 2026-02-23T14:57:25.769 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:57:25.888 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:57:25.987 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:57:25.985+0000 7f5030e18640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:57:25.987 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:57:25.985+0000 7f5030e18640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:57:26.096 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:57:26.096 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":279,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:56:34.843079+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:55:46.930221+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:53:39.251641+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:54:21.339477+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:57:26.576 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T14:57:26.575684+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T14:57:26.576 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T14:57:26.576 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6240 8 365281 365273 0.457288 0 - 0.0299513 2026-02-23T14:57:27.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:57:26 trial068 ceph-mon[38805]: pgmap v3735: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:57:27.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:57:26 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1552289605' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:57:27.576 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6241 8 365281 365273 0.457215 0 - 0.0299513 2026-02-23T14:57:28.576 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6242 8 365281 365273 0.457141 0 - 0.0299513 2026-02-23T14:57:29.134 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:57:29.158 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:57:28 trial068 ceph-mon[38805]: pgmap v3736: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:57:29.252 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:57:29.355 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:57:29.353+0000 7fc80c689640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:57:29.355 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:57:29.353+0000 7fc80c689640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:57:29.464 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:57:29.465 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":279,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:56:34.843079+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:55:46.930221+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:53:39.251641+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:54:21.339477+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:57:29.576 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6243 8 365281 365273 0.457068 0 - 0.0299513 2026-02-23T14:57:30.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:57:29 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/4285065888' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:57:30.576 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6244 8 365281 365273 0.456995 0 - 0.0299513 2026-02-23T14:57:31.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:57:30 trial068 ceph-mon[38805]: pgmap v3737: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:57:31.576 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6245 8 365281 365273 0.456922 0 - 0.0299513 2026-02-23T14:57:32.501 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:57:32.576 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6246 8 365281 365273 0.456849 0 - 0.0299513 2026-02-23T14:57:32.619 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:57:32.715 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:57:32.713+0000 7f8a8132f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:57:32.715 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:57:32.714+0000 7f8a8132f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:57:32.825 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:57:32.825 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":279,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:56:34.843079+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:55:46.930221+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:53:39.251641+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:54:21.339477+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:57:33.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:57:32 trial068 ceph-mon[38805]: pgmap v3738: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:57:33.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:57:32 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1335342289' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:57:33.576 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6247 8 365281 365273 0.456776 0 - 0.0299513 2026-02-23T14:57:34.265 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:57:33 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:57:33.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:57:34.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:57:33 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:57:33] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:57:34.576 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6248 8 365281 365273 0.456702 0 - 0.0299513 2026-02-23T14:57:35.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:57:34 trial068 ceph-mon[38805]: pgmap v3739: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:57:35.576 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6249 8 365281 365273 0.456629 0 - 0.0299513 2026-02-23T14:57:35.862 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:57:35.979 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:57:36.079 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:57:36.077+0000 7f54f09b4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:57:36.079 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:57:36.077+0000 7f54f09b4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:57:36.189 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:57:36.190 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":279,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:56:34.843079+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:55:46.930221+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:53:39.251641+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:54:21.339477+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:57:36.576 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6250 8 365281 365273 0.456556 0 - 0.0299513 2026-02-23T14:57:37.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:57:36 trial068 ceph-mon[38805]: pgmap v3740: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:57:37.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:57:36 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1318287850' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:57:37.576 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6251 8 365281 365273 0.456483 0 - 0.0299513 2026-02-23T14:57:38.576 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6252 8 365281 365273 0.45641 0 - 0.0299513 2026-02-23T14:57:39.228 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:57:39.252 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:57:38 trial068 ceph-mon[38805]: pgmap v3741: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:57:39.344 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:57:39.443 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:57:39.441+0000 7f6d54bf4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:57:39.443 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:57:39.442+0000 7f6d54bf4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:57:39.553 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:57:39.553 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":279,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:56:34.843079+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:55:46.930221+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:53:39.251641+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:54:21.339477+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:57:39.577 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6253 8 365281 365273 0.456337 0 - 0.0299513 2026-02-23T14:57:40.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:57:39 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2412158251' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:57:40.577 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6254 8 365281 365273 0.456264 0 - 0.0299513 2026-02-23T14:57:41.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:57:40 trial068 ceph-mon[38805]: pgmap v3742: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:57:41.577 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6255 8 365281 365273 0.456191 0 - 0.0299513 2026-02-23T14:57:42.577 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6256 8 365281 365273 0.456118 0 - 0.0299513 2026-02-23T14:57:42.590 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:57:42.708 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:57:42.804 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:57:42.802+0000 7f0f8f959640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:57:42.804 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:57:42.802+0000 7f0f8f959640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:57:42.916 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:57:42.916 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":279,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:56:34.843079+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:55:46.930221+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:53:39.251641+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:54:21.339477+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:57:43.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:57:42 trial068 ceph-mon[38805]: pgmap v3743: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:57:43.577 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6257 8 365281 365273 0.456046 0 - 0.0299513 2026-02-23T14:57:44.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:57:43 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3390577263' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:57:44.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:57:43 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:57:43] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:57:44.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:57:43 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:57:43.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:57:44.577 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6258 8 365281 365273 0.455973 0 - 0.0299513 2026-02-23T14:57:45.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:57:44 trial068 ceph-mon[38805]: pgmap v3744: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:57:45.577 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6259 8 365281 365273 0.4559 0 - 0.0299513 2026-02-23T14:57:45.954 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:57:46.070 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:57:46.169 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:57:46.167+0000 7f7845aa7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:57:46.169 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:57:46.167+0000 7f7845aa7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:57:46.282 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:57:46.282 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":279,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:56:34.843079+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:55:46.930221+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:53:39.251641+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:54:21.339477+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:57:46.577 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T14:57:46.576847+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T14:57:46.577 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T14:57:46.577 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6260 8 365281 365273 0.455827 0 - 0.0299513 2026-02-23T14:57:47.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:57:46 trial068 ceph-mon[38805]: pgmap v3745: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:57:47.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:57:46 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3971351780' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:57:47.577 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6261 8 365281 365273 0.455754 0 - 0.0299513 2026-02-23T14:57:48.577 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6262 8 365281 365273 0.455681 0 - 0.0299513 2026-02-23T14:57:49.319 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:57:49.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:57:48 trial068 ceph-mon[38805]: pgmap v3746: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:57:49.436 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:57:49.538 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:57:49.536+0000 7f3141f54640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:57:49.538 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:57:49.536+0000 7f3141f54640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:57:49.577 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6263 8 365281 365273 0.455609 0 - 0.0299513 2026-02-23T14:57:49.648 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:57:49.648 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":279,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:56:34.843079+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:55:46.930221+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:53:39.251641+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:54:21.339477+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:57:50.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:57:49 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1599177855' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:57:50.577 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6264 8 365281 365273 0.455536 0 - 0.0299513 2026-02-23T14:57:51.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:57:50 trial068 ceph-mon[38805]: pgmap v3747: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:57:51.577 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6265 8 365281 365273 0.455463 0 - 0.0299513 2026-02-23T14:57:52.577 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6266 8 365281 365273 0.455391 0 - 0.0299513 2026-02-23T14:57:52.685 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:57:52.802 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:57:52.900 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:57:52.898+0000 7f4c0ba9b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:57:52.900 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:57:52.899+0000 7f4c0ba9b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:57:53.011 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:57:53.011 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":279,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:56:34.843079+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:55:46.930221+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:53:39.251641+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:54:21.339477+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:57:53.012 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:57:52 trial068 ceph-mon[38805]: pgmap v3748: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:57:53.577 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6267 8 365281 365273 0.455318 0 - 0.0299513 2026-02-23T14:57:54.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:57:53 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/305990082' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:57:54.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:57:53 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:57:53] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:57:54.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:57:53 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:57:53.974Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:57:54.577 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6268 8 365281 365273 0.455245 0 - 0.0299513 2026-02-23T14:57:55.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:57:54 trial068 ceph-mon[38805]: pgmap v3749: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:57:55.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:57:54 trial068 ceph-mon[38805]: 5.e scrub starts 2026-02-23T14:57:55.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:57:54 trial068 ceph-mon[38805]: 5.e scrub ok 2026-02-23T14:57:55.577 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6269 8 365281 365273 0.455173 0 - 0.0299513 2026-02-23T14:57:56.050 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:57:56.168 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:57:56.265 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:57:56.263+0000 7f1cdf585640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:57:56.265 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:57:56.263+0000 7f1cdf585640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:57:56.374 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:57:56.374 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":279,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:56:34.843079+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:55:46.930221+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:53:39.251641+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:54:21.339477+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:57:56.578 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6270 8 365281 365273 0.4551 0 - 0.0299513 2026-02-23T14:57:57.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:57:56 trial068 ceph-mon[38805]: pgmap v3750: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:57:57.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:57:56 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/292641251' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:57:57.578 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6271 8 365281 365273 0.455027 0 - 0.0299513 2026-02-23T14:57:58.578 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6272 8 365281 365273 0.454955 0 - 0.0299513 2026-02-23T14:57:59.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:57:58 trial068 ceph-mon[38805]: pgmap v3751: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:57:59.413 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:57:59.530 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:57:59.578 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6273 8 365281 365273 0.454882 0 - 0.0299513 2026-02-23T14:57:59.628 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:57:59.626+0000 7f7f51be2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:57:59.628 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:57:59.627+0000 7f7f51be2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:57:59.738 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:57:59.738 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":279,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:56:34.843079+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:55:46.930221+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:53:39.251641+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:54:21.339477+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:58:00.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:57:59 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2934400152' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:58:00.578 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6274 8 365281 365273 0.45481 0 - 0.0299513 2026-02-23T14:58:01.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:58:00 trial068 ceph-mon[38805]: pgmap v3752: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:58:01.578 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6275 8 365281 365273 0.454737 0 - 0.0299513 2026-02-23T14:58:02.578 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6276 8 365281 365273 0.454665 0 - 0.0299513 2026-02-23T14:58:02.775 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:58:02.894 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:58:02.992 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:58:02.990+0000 7fc73d8d3640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:58:02.993 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:58:02.991+0000 7fc73d8d3640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:58:03.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:58:02 trial068 ceph-mon[38805]: pgmap v3753: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:58:03.101 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:58:03.101 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":279,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:56:34.843079+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:55:46.930221+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:53:39.251641+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:54:21.339477+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:58:03.578 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6277 8 365281 365273 0.454592 0 - 0.0299513 2026-02-23T14:58:04.266 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:58:03 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:58:03.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:58:04.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:58:03 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/228531048' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:58:04.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:58:03 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:58:03] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:58:04.578 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6278 8 365281 365273 0.45452 0 - 0.0299513 2026-02-23T14:58:05.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:58:04 trial068 ceph-mon[38805]: pgmap v3754: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:58:05.578 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6279 8 365281 365273 0.454448 0 - 0.0299513 2026-02-23T14:58:06.140 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:58:06.259 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:58:06.282 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:58:05 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T14:58:06.283 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:58:05 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T14:58:06.371 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:58:06.369+0000 7f1e1045b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:58:06.371 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:58:06.369+0000 7f1e1045b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:58:06.479 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:58:06.479 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":279,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:56:34.843079+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:55:46.930221+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:53:39.251641+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:54:21.339477+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:58:06.578 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T14:58:06.577993+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T14:58:06.578 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T14:58:06.579 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6280 8 365281 365273 0.454375 0 - 0.0299513 2026-02-23T14:58:07.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:58:06 trial068 ceph-mon[38805]: pgmap v3755: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:58:07.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:58:06 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1947965441' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:58:07.578 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6281 8 365281 365273 0.454303 0 - 0.0299513 2026-02-23T14:58:08.578 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6282 8 365281 365273 0.454231 0 - 0.0299513 2026-02-23T14:58:09.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:58:08 trial068 ceph-mon[38805]: pgmap v3756: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:58:09.518 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:58:09.578 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6283 8 365281 365273 0.454158 0 - 0.0299513 2026-02-23T14:58:09.640 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:58:09.745 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:58:09.743+0000 7f352049a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:58:09.745 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:58:09.744+0000 7f352049a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:58:09.858 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:58:09.859 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":279,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:56:34.843079+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:55:46.930221+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:53:39.251641+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:54:21.339477+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:58:10.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:58:09 trial068 ceph-mon[38805]: 5.a scrub starts 2026-02-23T14:58:10.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:58:09 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2026-02-23T14:58:10.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:58:09 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T14:58:10.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:58:09 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T14:58:10.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:58:09 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T14:58:10.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:58:09 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T14:58:10.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:58:09 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/59492375' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:58:10.578 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6284 8 365281 365273 0.454086 0 - 0.0299513 2026-02-23T14:58:11.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:58:10 trial068 ceph-mon[38805]: 5.a scrub ok 2026-02-23T14:58:11.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:58:10 trial068 ceph-mon[38805]: pgmap v3757: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:58:11.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:58:10 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-23T14:58:11.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:58:10 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2026-02-23T14:58:11.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:58:10 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T14:58:11.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:58:10 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2026-02-23T14:58:11.579 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6285 8 365281 365273 0.454014 0 - 0.0299513 2026-02-23T14:58:12.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:58:11 trial068 ceph-mon[38805]: pgmap v3758: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:58:12.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:58:11 trial068 ceph-mon[38805]: pgmap v3759: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:58:12.579 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6286 8 365281 365273 0.453942 0 - 0.0299513 2026-02-23T14:58:12.899 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:58:13.018 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:58:13.134 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:58:13.132+0000 7fcd03454640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:58:13.134 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:58:13.132+0000 7fcd03454640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:58:13.242 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:58:13.242 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":279,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:56:34.843079+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:55:46.930221+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:53:39.251641+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:54:21.339477+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:58:13.579 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6287 8 365281 365273 0.453869 0 - 0.0299513 2026-02-23T14:58:14.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:58:13 trial068 ceph-mon[38805]: pgmap v3760: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:58:14.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:58:13 trial068 ceph-mon[38805]: 5.8 scrub starts 2026-02-23T14:58:14.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:58:13 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2103340739' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:58:14.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:58:13 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:58:13] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:58:14.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:58:13 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:58:13.972Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:58:14.579 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6288 8 365281 365273 0.453797 0 - 0.0299513 2026-02-23T14:58:15.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:58:14 trial068 ceph-mon[38805]: 5.8 scrub ok 2026-02-23T14:58:15.579 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6289 8 365281 365273 0.453725 0 - 0.0299513 2026-02-23T14:58:16.284 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:58:16.309 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:58:15 trial068 ceph-mon[38805]: pgmap v3761: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:58:16.402 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:58:16.503 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:58:16.501+0000 7f8af2fb6640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:58:16.504 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:58:16.502+0000 7f8af2fb6640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:58:16.579 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6290 8 365281 365273 0.453653 0 - 0.0299513 2026-02-23T14:58:16.612 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:58:16.612 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":279,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:56:34.843079+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:55:46.930221+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:53:39.251641+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:54:21.339477+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:58:17.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:58:16 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/84556675' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:58:17.579 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6291 8 365281 365273 0.453581 0 - 0.0299513 2026-02-23T14:58:18.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:58:17 trial068 ceph-mon[38805]: pgmap v3762: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:58:18.579 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6292 8 365281 365273 0.453509 0 - 0.0299513 2026-02-23T14:58:19.579 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6293 8 365281 365273 0.453437 0 - 0.0299513 2026-02-23T14:58:19.649 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:58:19.765 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:58:19.862 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:58:19.860+0000 7f4cb95f3640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:58:19.863 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:58:19.861+0000 7f4cb95f3640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:58:19.972 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:58:19.972 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":279,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:56:34.843079+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:55:46.930221+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:53:39.251641+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:54:21.339477+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:58:20.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:58:19 trial068 ceph-mon[38805]: pgmap v3763: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:58:20.579 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6294 8 365281 365273 0.453365 0 - 0.0299513 2026-02-23T14:58:21.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:58:20 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2936679857' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:58:21.579 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6295 8 365281 365273 0.453293 0 - 0.0299513 2026-02-23T14:58:22.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:58:21 trial068 ceph-mon[38805]: pgmap v3764: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:58:22.579 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6296 8 365281 365273 0.453221 0 - 0.0299513 2026-02-23T14:58:23.009 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:58:23.126 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:58:23.231 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:58:23.229+0000 7fba48461640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:58:23.231 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:58:23.229+0000 7fba48461640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:58:23.340 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:58:23.340 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":279,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:56:34.843079+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:55:46.930221+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:53:39.251641+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:54:21.339477+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:58:23.580 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6297 8 365281 365273 0.453149 0 - 0.0299513 2026-02-23T14:58:24.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:58:23 trial068 ceph-mon[38805]: pgmap v3765: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:58:24.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:58:23 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2346236163' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:58:24.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:58:23 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:58:23] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:58:24.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:58:23 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:58:23.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:58:24.580 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6298 8 365281 365273 0.453077 0 - 0.0299513 2026-02-23T14:58:25.580 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6299 8 365281 365273 0.453005 0 - 0.0299513 2026-02-23T14:58:26.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:58:25 trial068 ceph-mon[38805]: pgmap v3766: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:58:26.377 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:58:26.493 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:58:26.579 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T14:58:26.579522+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T14:58:26.580 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T14:58:26.580 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6300 8 365281 365273 0.452933 0 - 0.0299513 2026-02-23T14:58:26.591 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:58:26.589+0000 7fecf9192640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:58:26.591 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:58:26.590+0000 7fecf9192640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:58:26.700 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:58:26.701 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":279,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:56:34.843079+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:55:46.930221+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:53:39.251641+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:54:21.339477+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:58:27.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:58:26 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/573448241' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:58:27.580 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6301 8 365281 365273 0.452861 0 - 0.0299513 2026-02-23T14:58:28.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:58:27 trial068 ceph-mon[38805]: pgmap v3767: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:58:28.580 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6302 8 365281 365273 0.452789 0 - 0.0299513 2026-02-23T14:58:29.580 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6303 8 365281 365273 0.452717 0 - 0.0299513 2026-02-23T14:58:29.738 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:58:29.855 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:58:29.957 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:58:29.955+0000 7f9f3fb43640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:58:29.957 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:58:29.955+0000 7f9f3fb43640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:58:30.067 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:58:30.067 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":279,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:56:34.843079+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:55:46.930221+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:53:39.251641+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:54:21.339477+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:58:30.068 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:58:29 trial068 ceph-mon[38805]: pgmap v3768: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:58:30.068 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:58:29 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T14:58:30.068 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:58:29 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T14:58:30.580 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6304 8 365281 365273 0.452645 0 - 0.0299513 2026-02-23T14:58:31.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:58:30 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1932226257' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:58:31.580 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6305 8 365281 365273 0.452574 0 - 0.0299513 2026-02-23T14:58:32.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:58:31 trial068 ceph-mon[38805]: pgmap v3769: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:58:32.581 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6306 8 365281 365273 0.452502 0 - 0.0299513 2026-02-23T14:58:33.105 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:58:33.223 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:58:33.325 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:58:33.323+0000 7f93324ff640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:58:33.325 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:58:33.324+0000 7f93324ff640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:58:33.438 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:58:33.438 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":279,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:56:34.843079+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:55:46.930221+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:53:39.251641+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:54:21.339477+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:58:33.580 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6307 8 365281 365273 0.45243 0 - 0.0299513 2026-02-23T14:58:34.265 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:58:33 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:58:33.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:58:34.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:58:33 trial068 ceph-mon[38805]: pgmap v3770: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:58:34.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:58:33 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3013745900' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:58:34.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:58:33 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:58:33] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:58:34.581 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6308 8 365281 365273 0.452358 0 - 0.0299513 2026-02-23T14:58:35.581 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6309 8 365281 365273 0.452287 0 - 0.0299513 2026-02-23T14:58:36.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:58:35 trial068 ceph-mon[38805]: pgmap v3771: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:58:36.476 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:58:36.581 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6310 8 365281 365273 0.452215 0 - 0.0299513 2026-02-23T14:58:36.592 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:58:36.698 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:58:36.696+0000 7fca21c04640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:58:36.698 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:58:36.696+0000 7fca21c04640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:58:36.809 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:58:36.809 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":279,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:56:34.843079+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:55:46.930221+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:53:39.251641+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:54:21.339477+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:58:37.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:58:36 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2785017148' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:58:37.581 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6311 8 365281 365273 0.452143 0 - 0.0299513 2026-02-23T14:58:38.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:58:37 trial068 ceph-mon[38805]: pgmap v3772: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:58:38.581 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6312 8 365281 365273 0.452072 0 - 0.0299513 2026-02-23T14:58:39.581 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6313 8 365281 365273 0.452 0 - 0.0299513 2026-02-23T14:58:39.847 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:58:39.966 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:58:40.068 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:58:40.066+0000 7f0da4f6f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:58:40.068 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:58:40.066+0000 7f0da4f6f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:58:40.178 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:58:40.178 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":279,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:56:34.843079+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:55:46.930221+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:53:39.251641+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:54:21.339477+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:58:40.179 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:58:39 trial068 ceph-mon[38805]: pgmap v3773: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:58:40.581 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6314 8 365281 365273 0.451929 0 - 0.0299513 2026-02-23T14:58:41.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:58:40 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3803297616' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:58:41.581 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6315 8 365281 365273 0.451857 0 - 0.0299513 2026-02-23T14:58:42.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:58:41 trial068 ceph-mon[38805]: pgmap v3774: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:58:42.581 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6316 8 365281 365273 0.451785 0 - 0.0299513 2026-02-23T14:58:43.215 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:58:43.333 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:58:43.432 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:58:43.430+0000 7f743fe4b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:58:43.432 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:58:43.430+0000 7f743fe4b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:58:43.540 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:58:43.541 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":279,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:56:34.843079+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:55:46.930221+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:53:39.251641+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:54:21.339477+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:58:43.581 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6317 8 365281 365273 0.451714 0 - 0.0299513 2026-02-23T14:58:44.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:58:43 trial068 ceph-mon[38805]: pgmap v3775: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:58:44.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:58:43 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2174702414' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:58:44.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:58:43 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:58:43] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:58:44.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:58:43 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:58:43.972Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:58:44.581 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6318 8 365281 365273 0.451642 0 - 0.0299513 2026-02-23T14:58:45.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:58:44 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T14:58:45.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:58:44 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T14:58:45.581 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6319 8 365281 365273 0.451571 0 - 0.0299513 2026-02-23T14:58:46.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:58:45 trial068 ceph-mon[38805]: pgmap v3776: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:58:46.581 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T14:58:46.581241+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T14:58:46.582 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T14:58:46.582 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6320 8 365281 365273 0.4515 0 - 0.0299513 2026-02-23T14:58:46.595 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:58:46.714 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:58:46.817 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:58:46.815+0000 7f1df0679640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:58:46.817 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:58:46.815+0000 7f1df0679640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:58:46.925 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:58:46.925 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":279,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:56:34.843079+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:55:46.930221+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:53:39.251641+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:54:21.339477+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:58:47.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:58:46 trial068 ceph-mon[38805]: pgmap v3777: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:58:47.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:58:46 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1645874989' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:58:47.582 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6321 8 365281 365273 0.451428 0 - 0.0299513 2026-02-23T14:58:48.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:58:48 trial068 ceph-mon[38805]: 5.5 scrub starts 2026-02-23T14:58:48.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:58:48 trial068 ceph-mon[38805]: 5.5 scrub ok 2026-02-23T14:58:48.582 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6322 8 365281 365273 0.451357 0 - 0.0299513 2026-02-23T14:58:49.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:58:49 trial068 ceph-mon[38805]: 5.7 scrub starts 2026-02-23T14:58:49.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:58:49 trial068 ceph-mon[38805]: pgmap v3778: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:58:49.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:58:49 trial068 ceph-mon[38805]: 5.7 scrub ok 2026-02-23T14:58:49.582 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6323 8 365281 365273 0.451285 0 - 0.0299513 2026-02-23T14:58:49.963 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:58:50.079 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:58:50.175 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:58:50.173+0000 7f628cbf9640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:58:50.175 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:58:50.173+0000 7f628cbf9640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:58:50.285 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:58:50.286 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":279,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:56:34.843079+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:55:46.930221+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:53:39.251641+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:54:21.339477+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:58:50.582 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6324 8 365281 365273 0.451214 0 - 0.0299513 2026-02-23T14:58:51.582 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6325 8 365281 365273 0.451143 0 - 0.0299513 2026-02-23T14:58:51.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:58:51 trial068 ceph-mon[38805]: pgmap v3779: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:58:51.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:58:51 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/609082534' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:58:52.582 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6326 8 365281 365273 0.451071 0 - 0.0299513 2026-02-23T14:58:53.323 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:58:53.444 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:58:53.468 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:58:53 trial068 ceph-mon[38805]: pgmap v3780: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:58:53.546 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:58:53.544+0000 7fd579b25640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:58:53.546 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:58:53.544+0000 7fd579b25640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:58:53.582 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6327 8 365281 365273 0.451 0 - 0.0299513 2026-02-23T14:58:53.654 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:58:53.654 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":279,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:56:34.843079+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:55:46.930221+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:53:39.251641+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:54:21.339477+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:58:54.109 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:58:53 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:58:53] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:58:54.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:58:53 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:58:53.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:58:54.583 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6328 8 365281 365273 0.450929 0 - 0.0299513 2026-02-23T14:58:54.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:58:54 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1220251096' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:58:55.583 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6329 8 365281 365273 0.450857 0 - 0.0299513 2026-02-23T14:58:56.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:58:55 trial068 ceph-mon[38805]: pgmap v3781: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:58:56.583 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6330 8 365281 365273 0.450786 0 - 0.0299513 2026-02-23T14:58:56.690 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:58:56.808 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:58:56.906 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:58:56.904+0000 7fe2af1b9640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:58:56.906 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:58:56.904+0000 7fe2af1b9640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:58:57.015 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:58:57.015 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":279,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:56:34.843079+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:55:46.930221+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:53:39.251641+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:54:21.339477+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:58:57.583 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6331 8 365281 365273 0.450715 0 - 0.0299513 2026-02-23T14:58:58.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:58:57 trial068 ceph-mon[38805]: pgmap v3782: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:58:58.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:58:57 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3848248712' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:58:58.583 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6332 8 365281 365273 0.450644 0 - 0.0299513 2026-02-23T14:58:59.583 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6333 8 365281 365273 0.450573 0 - 0.0299513 2026-02-23T14:59:00.052 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:59:00.170 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:59:00.192 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:58:59 trial068 ceph-mon[38805]: pgmap v3783: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:59:00.193 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:58:59 trial068 ceph-mon[38805]: 5.e scrub starts 2026-02-23T14:59:00.193 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:58:59 trial068 ceph-mon[38805]: 5.e scrub ok 2026-02-23T14:59:00.269 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:59:00.267+0000 7f2d842a5640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:59:00.269 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:59:00.267+0000 7f2d842a5640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:59:00.378 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:59:00.379 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":279,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:56:34.843079+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:55:46.930221+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:53:39.251641+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:54:21.339477+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:59:00.583 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6334 8 365281 365273 0.450502 0 - 0.0299513 2026-02-23T14:59:01.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:59:00 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/217636822' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:59:01.583 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6335 8 365281 365273 0.45043 0 - 0.0299513 2026-02-23T14:59:02.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:59:01 trial068 ceph-mon[38805]: pgmap v3784: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:59:02.583 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6336 8 365281 365273 0.450359 0 - 0.0299513 2026-02-23T14:59:03.432 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:59:03.552 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:59:03.583 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6337 8 365281 365273 0.450288 0 - 0.0299513 2026-02-23T14:59:03.660 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:59:03.658+0000 7f72376b4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:59:03.660 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:59:03.658+0000 7f72376b4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:59:03.767 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:59:03.768 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":279,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:56:34.843079+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:55:46.930221+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:53:39.251641+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:54:21.339477+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:59:04.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:59:03 trial068 ceph-mon[38805]: pgmap v3785: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:59:04.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:59:03 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2171992905' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:59:04.095 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:59:03 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:59:03] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:59:04.266 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:59:03 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:59:03.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:59:04.583 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6338 8 365281 365273 0.450217 0 - 0.0299513 2026-02-23T14:59:05.583 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6339 8 365281 365273 0.450146 0 - 0.0299513 2026-02-23T14:59:06.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:59:05 trial068 ceph-mon[38805]: pgmap v3786: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:59:06.583 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T14:59:06.583114+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T14:59:06.584 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T14:59:06.584 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6340 8 365281 365273 0.450075 0 - 0.0299513 2026-02-23T14:59:06.806 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:59:06.925 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:59:07.029 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:59:07.027+0000 7f1ae3a9c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:59:07.030 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:59:07.028+0000 7f1ae3a9c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:59:07.139 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:59:07.140 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":279,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:56:34.843079+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:55:46.930221+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:53:39.251641+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:54:21.339477+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:59:07.583 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6341 8 365281 365273 0.450004 0 - 0.0299513 2026-02-23T14:59:08.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:59:07 trial068 ceph-mon[38805]: pgmap v3787: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:59:08.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:59:07 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/108525784' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:59:08.583 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6342 8 365281 365273 0.449933 0 - 0.0299513 2026-02-23T14:59:09.584 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6343 8 365281 365273 0.449862 0 - 0.0299513 2026-02-23T14:59:10.177 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:59:10.199 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:59:09 trial068 ceph-mon[38805]: pgmap v3788: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:59:10.294 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:59:10.398 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:59:10.395+0000 7f8ce22a9640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:59:10.398 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:59:10.396+0000 7f8ce22a9640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:59:10.507 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:59:10.507 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":280,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:59:09.873066+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:55:46.930221+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:58:43.617395+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:54:21.339477+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:59:10.584 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6344 8 365281 365273 0.449791 0 - 0.0299513 2026-02-23T14:59:11.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:59:10 trial068 ceph-mon[38805]: osdmap e280: 4 total, 3 up, 4 in 2026-02-23T14:59:11.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:59:10 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2026-02-23T14:59:11.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:59:10 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-23T14:59:11.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:59:10 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2026-02-23T14:59:11.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:59:10 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T14:59:11.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:59:10 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2026-02-23T14:59:11.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:59:10 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2831566698' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:59:11.584 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6345 8 365281 365273 0.449721 0 - 0.0299513 2026-02-23T14:59:12.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:59:11 trial068 ceph-mon[38805]: pgmap v3790: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:59:12.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:59:11 trial068 ceph-mon[38805]: pgmap v3791: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:59:12.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:59:11 trial068 ceph-mon[38805]: pgmap v3792: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:59:12.584 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6346 8 365281 365273 0.44965 0 - 0.0299513 2026-02-23T14:59:13.545 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:59:13.584 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6347 8 365281 365273 0.449579 0 - 0.0299513 2026-02-23T14:59:13.663 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:59:13.764 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:59:13.762+0000 7f7d255a2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:59:13.764 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:59:13.762+0000 7f7d255a2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:59:13.874 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:59:13.874 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":280,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:59:09.873066+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:55:46.930221+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:58:43.617395+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:54:21.339477+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:59:14.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:59:13 trial068 ceph-mon[38805]: pgmap v3793: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:59:14.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:59:13 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1919323672' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:59:14.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:59:13 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:59:13] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:59:14.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:59:13 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:59:13.972Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:59:14.584 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6348 8 365281 365273 0.449508 0 - 0.0299513 2026-02-23T14:59:15.584 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6349 8 365281 365273 0.449437 0 - 0.0299513 2026-02-23T14:59:16.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:59:15 trial068 ceph-mon[38805]: pgmap v3794: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:59:16.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:59:15 trial068 ceph-mon[38805]: osdmap e281: 4 total, 3 up, 4 in 2026-02-23T14:59:16.584 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6350 8 365281 365273 0.449366 0 - 0.0299513 2026-02-23T14:59:16.914 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:59:17.034 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:59:17.139 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:59:17.136+0000 7f35f8e30640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:59:17.139 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:59:17.137+0000 7f35f8e30640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:59:17.248 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:59:17.248 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":281,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:59:14.864729+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:55:46.930221+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:58:43.617395+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:54:21.339477+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:59:17.584 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6351 8 365281 365273 0.449296 0 - 0.0299513 2026-02-23T14:59:18.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:59:17 trial068 ceph-mon[38805]: pgmap v3796: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:59:18.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:59:17 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/246458925' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:59:18.584 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6352 8 365281 365273 0.449225 0 - 0.0299513 2026-02-23T14:59:19.584 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6353 8 365281 365273 0.449154 0 - 0.0299513 2026-02-23T14:59:20.287 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:59:20.311 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:59:19 trial068 ceph-mon[38805]: pgmap v3797: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:59:20.404 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:59:20.503 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:59:20.501+0000 7fbb76ecf640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:59:20.503 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:59:20.501+0000 7fbb76ecf640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:59:20.584 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6354 8 365281 365273 0.449084 0 - 0.0299513 2026-02-23T14:59:20.611 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:59:20.611 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":281,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:59:14.864729+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:55:46.930221+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:58:43.617395+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:54:21.339477+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:59:21.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:59:20 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/909100861' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:59:21.584 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6355 8 365281 365273 0.449013 0 - 0.0299513 2026-02-23T14:59:22.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:59:21 trial068 ceph-mon[38805]: pgmap v3798: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:59:22.585 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6356 8 365281 365273 0.448942 0 - 0.0299513 2026-02-23T14:59:23.585 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6357 8 365281 365273 0.448872 0 - 0.0299513 2026-02-23T14:59:23.649 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:59:23.768 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:59:23.866 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:59:23.863+0000 7f1f29662640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:59:23.866 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:59:23.864+0000 7f1f29662640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:59:23.973 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:59:23 trial068 ceph-mon[38805]: pgmap v3799: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:59:23.974 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:59:23.975 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":281,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:59:14.864729+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:55:46.930221+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:58:43.617395+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:54:21.339477+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:59:24.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:59:23 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:59:23] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:59:24.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:59:23 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:59:23.972Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:59:24.585 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6358 8 365281 365273 0.448801 0 - 0.0299513 2026-02-23T14:59:25.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:59:24 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3032817499' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:59:25.585 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6359 8 365281 365273 0.44873 0 - 0.0299513 2026-02-23T14:59:26.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:59:25 trial068 ceph-mon[38805]: pgmap v3800: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:59:26.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:59:25 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T14:59:26.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:59:25 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T14:59:26.585 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T14:59:26.584623+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T14:59:26.585 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T14:59:26.585 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6360 8 365281 365273 0.44866 0 - 0.0299513 2026-02-23T14:59:27.013 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:59:27.129 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:59:27.228 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:59:27.226+0000 7fde9d5cd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:59:27.228 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:59:27.226+0000 7fde9d5cd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:59:27.339 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:59:27.339 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":281,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:59:14.864729+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:55:46.930221+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:58:43.617395+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:54:21.339477+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:59:27.585 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6361 8 365281 365273 0.448589 0 - 0.0299513 2026-02-23T14:59:28.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:59:27 trial068 ceph-mon[38805]: pgmap v3801: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:59:28.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:59:27 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2967060359' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:59:28.585 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6362 8 365281 365273 0.448519 0 - 0.0299513 2026-02-23T14:59:29.585 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6363 8 365281 365273 0.448448 0 - 0.0299513 2026-02-23T14:59:30.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:59:29 trial068 ceph-mon[38805]: pgmap v3802: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:59:30.377 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:59:30.491 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:59:30.585 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6364 8 365281 365273 0.448378 0 - 0.0299513 2026-02-23T14:59:30.591 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:59:30.589+0000 7fc1cb3c9640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:59:30.591 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:59:30.590+0000 7fc1cb3c9640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:59:30.700 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:59:30.700 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":281,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:59:14.864729+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:55:46.930221+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:58:43.617395+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:54:21.339477+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:59:31.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:59:30 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/869089390' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:59:31.585 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6365 8 365281 365273 0.448307 0 - 0.0299513 2026-02-23T14:59:32.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:59:31 trial068 ceph-mon[38805]: pgmap v3803: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:59:32.585 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6366 8 365281 365273 0.448237 0 - 0.0299513 2026-02-23T14:59:33.585 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6367 8 365281 365273 0.448167 0 - 0.0299513 2026-02-23T14:59:33.738 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:59:33.857 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:59:33.956 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:59:33.954+0000 7f9b60dba640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:59:33.956 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:59:33.954+0000 7f9b60dba640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:59:34.065 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:59:34.065 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":281,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:59:14.864729+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:55:46.930221+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:58:43.617395+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:54:21.339477+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:59:34.066 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:59:33 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:59:33] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:59:34.066 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:59:33 trial068 ceph-mon[38805]: pgmap v3804: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:59:34.265 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:59:33 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:59:33.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:59:34.585 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6368 8 365281 365273 0.448096 0 - 0.0299513 2026-02-23T14:59:35.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:59:34 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1525920964' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:59:35.585 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6369 8 365281 365273 0.448026 0 - 0.0299513 2026-02-23T14:59:36.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:59:35 trial068 ceph-mon[38805]: pgmap v3805: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:59:36.586 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6370 8 365281 365273 0.447956 0 - 0.0299513 2026-02-23T14:59:37.103 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:59:37.223 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:59:37.329 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:59:37.327+0000 7fe80f0db640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:59:37.329 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:59:37.327+0000 7fe80f0db640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:59:37.440 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:59:37.440 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":281,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:59:14.864729+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:55:46.930221+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:58:43.617395+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:54:21.339477+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:59:37.586 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6371 8 365281 365273 0.447885 0 - 0.0299513 2026-02-23T14:59:38.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:59:37 trial068 ceph-mon[38805]: pgmap v3806: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:59:38.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:59:37 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T14:59:38.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:59:37 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T14:59:38.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:59:37 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/4182748109' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:59:38.586 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6372 8 365281 365273 0.447815 0 - 0.0299513 2026-02-23T14:59:39.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:59:38 trial068 ceph-mon[38805]: 5.8 scrub starts 2026-02-23T14:59:39.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:59:38 trial068 ceph-mon[38805]: 5.8 scrub ok 2026-02-23T14:59:39.586 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6373 8 365281 365273 0.447745 0 - 0.0299513 2026-02-23T14:59:40.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:59:39 trial068 ceph-mon[38805]: pgmap v3807: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:59:40.492 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:59:40.586 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6374 8 365281 365273 0.447674 0 - 0.0299513 2026-02-23T14:59:40.611 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:59:40.714 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:59:40.712+0000 7f827cd18640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:59:40.715 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:59:40.712+0000 7f827cd18640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:59:40.823 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:59:40.823 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":281,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:59:14.864729+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:55:46.930221+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:58:43.617395+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:54:21.339477+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:59:41.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:59:40 trial068 ceph-mon[38805]: 5.a scrub starts 2026-02-23T14:59:41.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:59:40 trial068 ceph-mon[38805]: 5.a scrub ok 2026-02-23T14:59:41.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:59:40 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3785459542' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:59:41.586 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6375 8 365281 365273 0.447604 0 - 0.0299513 2026-02-23T14:59:42.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:59:41 trial068 ceph-mon[38805]: pgmap v3808: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:59:42.586 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6376 8 365281 365273 0.447534 0 - 0.0299513 2026-02-23T14:59:43.586 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6377 8 365281 365273 0.447464 0 - 0.0299513 2026-02-23T14:59:43.860 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:59:43.978 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:59:44.074 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:59:44.072+0000 7f4a74394640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:59:44.074 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:59:44.073+0000 7f4a74394640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:59:44.146 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:59:43 trial068 ceph-mon[38805]: pgmap v3809: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:59:44.146 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:59:43 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:59:43] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:59:44.183 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:59:44.183 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":281,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:59:14.864729+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:55:46.930221+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:58:43.617395+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:54:21.339477+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:59:44.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:59:43 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:59:43.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:59:44.586 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6378 8 365281 365273 0.447394 0 - 0.0299513 2026-02-23T14:59:45.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:59:44 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1499431139' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:59:45.586 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6379 8 365281 365273 0.447324 0 - 0.0299513 2026-02-23T14:59:46.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:59:45 trial068 ceph-mon[38805]: pgmap v3810: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:59:46.586 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T14:59:46.586168+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T14:59:46.587 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T14:59:46.587 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6380 8 365281 365273 0.447253 0 - 0.0299513 2026-02-23T14:59:47.221 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:59:47.341 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:59:47.451 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:59:47.450+0000 7ff7e043f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:59:47.452 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:59:47.450+0000 7ff7e043f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:59:47.561 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:59:47.561 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":281,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:59:14.864729+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:55:46.930221+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:58:43.617395+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:54:21.339477+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:59:47.586 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6381 8 365281 365273 0.447183 0 - 0.0299513 2026-02-23T14:59:48.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:59:47 trial068 ceph-mon[38805]: pgmap v3811: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:59:48.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:59:47 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/379270350' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:59:48.586 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6382 8 365281 365273 0.447113 0 - 0.0299513 2026-02-23T14:59:49.587 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6383 8 365281 365273 0.447043 0 - 0.0299513 2026-02-23T14:59:50.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:59:49 trial068 ceph-mon[38805]: pgmap v3812: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:59:50.587 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6384 8 365281 365273 0.446973 0 - 0.0299513 2026-02-23T14:59:50.618 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:59:50.737 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:59:50.842 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:59:50.840+0000 7f7b79711640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:59:50.842 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:59:50.840+0000 7f7b79711640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:59:50.951 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:59:50.951 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":281,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:59:14.864729+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:55:46.930221+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:58:43.617395+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:54:21.339477+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:59:51.587 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6385 8 365281 365273 0.446903 0 - 0.0299513 2026-02-23T14:59:52.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:59:51 trial068 ceph-mon[38805]: pgmap v3813: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:59:52.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:59:51 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3100770761' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:59:52.587 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6386 8 365281 365273 0.446833 0 - 0.0299513 2026-02-23T14:59:53.587 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6387 8 365281 365273 0.446763 0 - 0.0299513 2026-02-23T14:59:53.992 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:59:54.110 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:59:54.155 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:59:53 trial068 ceph-mon[38805]: pgmap v3814: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:59:54.155 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 14:59:53 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:14:59:53] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T14:59:54.205 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:59:54.203+0000 7f1d7f85c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:59:54.205 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:59:54.203+0000 7f1d7f85c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:59:54.313 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:59:54.314 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":281,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:59:14.864729+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:55:46.930221+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:58:43.617395+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:54:21.339477+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:59:54.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 14:59:53 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T14:59:53.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T14:59:54.588 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6388 8 365281 365273 0.446693 0 - 0.0299513 2026-02-23T14:59:55.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:59:54 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2384990935' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:59:55.587 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6389 8 365281 365273 0.446623 0 - 0.0299513 2026-02-23T14:59:56.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:59:55 trial068 ceph-mon[38805]: pgmap v3815: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:59:56.587 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6390 8 365281 365273 0.446553 0 - 0.0299513 2026-02-23T14:59:57.352 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T14:59:57.469 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T14:59:57.568 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:59:57.566+0000 7f63fc89d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:59:57.568 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T14:59:57.566+0000 7f63fc89d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T14:59:57.587 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6391 8 365281 365273 0.446484 0 - 0.0299513 2026-02-23T14:59:57.676 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T14:59:57.677 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":281,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:59:14.864729+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:55:46.930221+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:58:43.617395+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:54:21.339477+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T14:59:58.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:59:57 trial068 ceph-mon[38805]: pgmap v3816: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T14:59:58.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:59:57 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3863705506' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T14:59:58.588 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6392 8 365281 365273 0.446414 0 - 0.0299513 2026-02-23T14:59:59.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:59:58 trial068 ceph-mon[38805]: 5.7 scrub starts 2026-02-23T14:59:59.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:59:58 trial068 ceph-mon[38805]: 5.7 scrub ok 2026-02-23T14:59:59.588 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6393 8 365281 365273 0.446344 0 - 0.0299513 2026-02-23T15:00:00.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 14:59:59 trial068 ceph-mon[38805]: pgmap v3817: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:00:00.588 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6394 8 365281 365273 0.446274 0 - 0.0299513 2026-02-23T15:00:00.715 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:00:00.833 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:00:00.931 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:00:00.928+0000 7fa260248640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:00:00.931 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:00:00.929+0000 7fa260248640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:00:01.038 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:00:01.038 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":281,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:59:14.864729+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:55:46.930221+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:58:43.617395+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:54:21.339477+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:00:01.588 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6395 8 365281 365273 0.446204 0 - 0.0299513 2026-02-23T15:00:02.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:00:01 trial068 ceph-mon[38805]: pgmap v3818: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:00:02.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:00:01 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1478197648' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:00:02.588 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6396 8 365281 365273 0.446135 0 - 0.0299513 2026-02-23T15:00:03.588 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6397 8 365281 365273 0.446065 0 - 0.0299513 2026-02-23T15:00:04.075 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:00:04.195 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:00:04.218 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:00:03 trial068 ceph-mon[38805]: pgmap v3819: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:00:04.219 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:00:03 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:00:03] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:00:04.264 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:00:03 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:00:03.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:00:04.295 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:00:04.293+0000 7f47b667f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:00:04.295 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:00:04.293+0000 7f47b667f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:00:04.405 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:00:04.405 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":281,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:59:14.864729+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:55:46.930221+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:58:43.617395+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:54:21.339477+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:00:04.588 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6398 8 365281 365273 0.445995 0 - 0.0299513 2026-02-23T15:00:05.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:00:04 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/4008181591' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:00:05.588 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6399 8 365281 365273 0.445925 0 - 0.0299513 2026-02-23T15:00:06.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:00:05 trial068 ceph-mon[38805]: pgmap v3820: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:00:06.588 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T15:00:06.587970+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T15:00:06.588 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T15:00:06.588 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6400 8 365281 365273 0.445856 0 - 0.0299513 2026-02-23T15:00:07.442 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:00:07.560 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:00:07.588 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6401 8 365281 365273 0.445786 0 - 0.0299513 2026-02-23T15:00:07.663 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:00:07.661+0000 7f7ff0a99640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:00:07.663 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:00:07.661+0000 7f7ff0a99640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:00:07.770 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:00:07.770 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":281,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:59:14.864729+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:55:46.930221+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:58:43.617395+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:54:21.339477+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:00:08.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:00:07 trial068 ceph-mon[38805]: pgmap v3821: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:00:08.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:00:07 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1595950029' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:00:08.588 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6402 8 365281 365273 0.445716 0 - 0.0299513 2026-02-23T15:00:09.588 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6403 8 365281 365273 0.445647 0 - 0.0299513 2026-02-23T15:00:10.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:00:09 trial068 ceph-mon[38805]: pgmap v3822: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:00:10.588 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6404 8 365281 365273 0.445577 0 - 0.0299513 2026-02-23T15:00:10.809 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:00:10.925 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:00:11.040 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:00:11.038+0000 7fadc0ad8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:00:11.040 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:00:11.038+0000 7fadc0ad8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:00:11.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:00:10 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T15:00:11.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:00:10 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T15:00:11.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:00:10 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2026-02-23T15:00:11.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:00:10 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-23T15:00:11.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:00:10 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2026-02-23T15:00:11.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:00:10 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T15:00:11.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:00:10 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2026-02-23T15:00:11.151 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:00:11.152 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":281,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:59:14.864729+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:55:46.930221+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:58:43.617395+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:54:21.339477+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:00:11.589 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6405 8 365281 365273 0.445508 0 - 0.0299513 2026-02-23T15:00:12.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:00:11 trial068 ceph-mon[38805]: pgmap v3823: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:00:12.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:00:11 trial068 ceph-mon[38805]: pgmap v3824: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:00:12.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:00:11 trial068 ceph-mon[38805]: pgmap v3825: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:00:12.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:00:11 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2159334582' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:00:12.589 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6406 8 365281 365273 0.445438 0 - 0.0299513 2026-02-23T15:00:13.589 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6407 8 365281 365273 0.445369 0 - 0.0299513 2026-02-23T15:00:14.193 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:00:14.218 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:00:13 trial068 ceph-mon[38805]: pgmap v3826: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:00:14.218 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:00:13 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:00:13] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:00:14.312 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:00:14.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:00:13 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:00:13.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:00:14.415 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:00:14.413+0000 7f05c3cf6640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:00:14.416 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:00:14.414+0000 7f05c3cf6640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:00:14.525 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:00:14.525 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":281,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:59:14.864729+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:55:46.930221+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:58:43.617395+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:54:21.339477+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:00:14.589 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6408 8 365281 365273 0.445299 0 - 0.0299513 2026-02-23T15:00:15.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:00:14 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1726530512' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:00:15.589 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6409 8 365281 365273 0.44523 0 - 0.0299513 2026-02-23T15:00:16.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:00:15 trial068 ceph-mon[38805]: pgmap v3827: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:00:16.589 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6410 8 365281 365273 0.44516 0 - 0.0299513 2026-02-23T15:00:17.576 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:00:17.589 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6411 8 365281 365273 0.445091 0 - 0.0299513 2026-02-23T15:00:17.698 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:00:17.805 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:00:17.803+0000 7f88e98bc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:00:17.805 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:00:17.803+0000 7f88e98bc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:00:17.914 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:00:17.914 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":281,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:59:14.864729+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:55:46.930221+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:58:43.617395+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:54:21.339477+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:00:18.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:00:17 trial068 ceph-mon[38805]: pgmap v3828: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:00:18.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:00:17 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/666597559' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:00:18.589 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6412 8 365281 365273 0.445021 0 - 0.0299513 2026-02-23T15:00:19.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:00:18 trial068 ceph-mon[38805]: 5.5 scrub starts 2026-02-23T15:00:19.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:00:18 trial068 ceph-mon[38805]: 5.5 scrub ok 2026-02-23T15:00:19.589 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6413 8 365281 365273 0.444952 0 - 0.0299513 2026-02-23T15:00:20.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:00:19 trial068 ceph-mon[38805]: pgmap v3829: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:00:20.589 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6414 8 365281 365273 0.444883 0 - 0.0299513 2026-02-23T15:00:20.953 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:00:21.071 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:00:21.175 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:00:21.173+0000 7f44b5b59640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:00:21.175 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:00:21.174+0000 7f44b5b59640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:00:21.287 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:00:21.287 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":281,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:59:14.864729+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:55:46.930221+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:58:43.617395+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:54:21.339477+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:00:21.589 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6415 8 365281 365273 0.444813 0 - 0.0299513 2026-02-23T15:00:22.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:00:21 trial068 ceph-mon[38805]: pgmap v3830: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:00:22.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:00:21 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2774660661' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:00:22.589 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6416 8 365281 365273 0.444744 0 - 0.0299513 2026-02-23T15:00:23.589 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6417 8 365281 365273 0.444675 0 - 0.0299513 2026-02-23T15:00:24.324 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:00:24.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:00:23 trial068 ceph-mon[38805]: pgmap v3831: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:00:24.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:00:23 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:00:23] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:00:24.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:00:23 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:00:23.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:00:24.441 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:00:24.544 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:00:24.542+0000 7f949839f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:00:24.544 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:00:24.542+0000 7f949839f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:00:24.589 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6418 8 365281 365273 0.444605 0 - 0.0299513 2026-02-23T15:00:24.655 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:00:24.655 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":281,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T14:59:14.864729+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:55:46.930221+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:58:43.617395+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:54:21.339477+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:00:25.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:00:24 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/448898707' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:00:25.589 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6419 8 365281 365273 0.444536 0 - 0.0299513 2026-02-23T15:00:26.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:00:25 trial068 ceph-mon[38805]: pgmap v3832: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:00:26.590 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T15:00:26.589361+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T15:00:26.590 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T15:00:26.590 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6420 8 365281 365273 0.444467 0 - 0.0299513 2026-02-23T15:00:27.590 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6421 8 365281 365273 0.444398 0 - 0.0299513 2026-02-23T15:00:27.693 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:00:27.811 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:00:27.916 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:00:27.914+0000 7f29ed053640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:00:27.916 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:00:27.914+0000 7f29ed053640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:00:28.029 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:00:28.029 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":282,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:00:26.932239+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:55:46.930221+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:58:43.617395+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:59:25.140122+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:00:28.030 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:00:27 trial068 ceph-mon[38805]: pgmap v3833: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:00:28.030 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:00:27 trial068 ceph-mon[38805]: osdmap e282: 4 total, 3 up, 4 in 2026-02-23T15:00:28.590 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6422 8 365281 365273 0.444328 0 - 0.0299513 2026-02-23T15:00:29.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:00:28 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2302542903' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:00:29.590 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6423 8 365281 365273 0.444259 0 - 0.0299513 2026-02-23T15:00:30.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:00:29 trial068 ceph-mon[38805]: pgmap v3835: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:00:30.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:00:29 trial068 ceph-mon[38805]: osdmap e283: 4 total, 3 up, 4 in 2026-02-23T15:00:30.590 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6424 8 365281 365273 0.44419 0 - 0.0299513 2026-02-23T15:00:31.067 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:00:31.184 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:00:31.279 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:00:31.277+0000 7ff7e7cc7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:00:31.279 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:00:31.277+0000 7ff7e7cc7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:00:31.387 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:00:31.387 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":283,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:00:29.876046+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:55:46.930221+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:58:43.617395+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:59:25.140122+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:00:31.590 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6425 8 365281 365273 0.444121 0 - 0.0299513 2026-02-23T15:00:32.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:00:31 trial068 ceph-mon[38805]: pgmap v3837: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:00:32.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:00:31 trial068 ceph-mon[38805]: 5.e scrub starts 2026-02-23T15:00:32.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:00:31 trial068 ceph-mon[38805]: 5.e scrub ok 2026-02-23T15:00:32.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:00:31 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/4103284881' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:00:32.590 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6426 8 365281 365273 0.444052 0 - 0.0299513 2026-02-23T15:00:33.590 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6427 8 365281 365273 0.443983 0 - 0.0299513 2026-02-23T15:00:34.266 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:00:33 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:00:33.974Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:00:34.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:00:33 trial068 ceph-mon[38805]: pgmap v3838: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:00:34.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:00:33 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:00:33] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:00:34.425 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:00:34.544 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:00:34.590 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6428 8 365281 365273 0.443914 0 - 0.0299513 2026-02-23T15:00:34.653 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:00:34.651+0000 7f406d956640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:00:34.654 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:00:34.652+0000 7f406d956640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:00:34.767 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:00:34.767 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":283,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:00:29.876046+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:55:46.930221+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:58:43.617395+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:59:25.140122+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:00:35.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:00:34 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2781198546' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:00:35.590 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6429 8 365281 365273 0.443845 0 - 0.0299513 2026-02-23T15:00:36.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:00:35 trial068 ceph-mon[38805]: pgmap v3839: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:00:36.590 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6430 8 365281 365273 0.443776 0 - 0.0299513 2026-02-23T15:00:37.590 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6431 8 365281 365273 0.443707 0 - 0.0299513 2026-02-23T15:00:37.805 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:00:37.921 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:00:38.018 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:00:38.016+0000 7f1ee4a92640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:00:38.018 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:00:38.016+0000 7f1ee4a92640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:00:38.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:00:37 trial068 ceph-mon[38805]: pgmap v3840: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:00:38.129 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:00:38.129 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":283,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:00:29.876046+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:55:46.930221+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:58:43.617395+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:59:25.140122+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:00:38.590 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6432 8 365281 365273 0.443638 0 - 0.0299513 2026-02-23T15:00:39.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:00:38 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1454492287' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:00:39.590 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6433 8 365281 365273 0.443569 0 - 0.0299513 2026-02-23T15:00:40.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:00:39 trial068 ceph-mon[38805]: pgmap v3841: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:00:40.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:00:39 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T15:00:40.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:00:39 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T15:00:40.591 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6434 8 365281 365273 0.4435 0 - 0.0299513 2026-02-23T15:00:41.167 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:00:41.285 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:00:41.383 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:00:41.381+0000 7fec237c1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:00:41.383 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:00:41.381+0000 7fec237c1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:00:41.491 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:00:41.492 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":283,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:00:29.876046+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:55:46.930221+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:58:43.617395+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:59:25.140122+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:00:41.591 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6435 8 365281 365273 0.443431 0 - 0.0299513 2026-02-23T15:00:42.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:00:41 trial068 ceph-mon[38805]: pgmap v3842: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:00:42.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:00:41 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3470586414' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:00:42.591 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6436 8 365281 365273 0.443362 0 - 0.0299513 2026-02-23T15:00:43.591 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6437 8 365281 365273 0.443293 0 - 0.0299513 2026-02-23T15:00:44.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:00:43 trial068 ceph-mon[38805]: pgmap v3843: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:00:44.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:00:43 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:00:43] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:00:44.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:00:43 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:00:43.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:00:44.529 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:00:44.591 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6438 8 365281 365273 0.443224 0 - 0.0299513 2026-02-23T15:00:44.648 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:00:44.749 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:00:44.747+0000 7fae5c03b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:00:44.749 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:00:44.747+0000 7fae5c03b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:00:44.858 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:00:44.858 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":283,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:00:29.876046+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:55:46.930221+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:58:43.617395+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:59:25.140122+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:00:45.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:00:44 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1448055464' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:00:45.591 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6439 8 365281 365273 0.443155 0 - 0.0299513 2026-02-23T15:00:46.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:00:45 trial068 ceph-mon[38805]: pgmap v3844: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:00:46.591 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T15:00:46.590776+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T15:00:46.591 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T15:00:46.591 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6440 8 365281 365273 0.443086 0 - 0.0299513 2026-02-23T15:00:47.591 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6441 8 365281 365273 0.443018 0 - 0.0299513 2026-02-23T15:00:47.894 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:00:48.015 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:00:48.116 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:00:48.114+0000 7fa15932a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:00:48.116 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:00:48.114+0000 7fa15932a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:00:48.225 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:00:48.226 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":283,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:00:29.876046+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:55:46.930221+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:58:43.617395+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:59:25.140122+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:00:48.226 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:00:47 trial068 ceph-mon[38805]: pgmap v3845: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:00:48.591 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6442 8 365281 365273 0.442949 0 - 0.0299513 2026-02-23T15:00:49.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:00:48 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1298693003' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:00:49.591 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6443 8 365281 365273 0.44288 0 - 0.0299513 2026-02-23T15:00:50.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:00:49 trial068 ceph-mon[38805]: pgmap v3846: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:00:50.591 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6444 8 365281 365273 0.442811 0 - 0.0299513 2026-02-23T15:00:51.263 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:00:51.382 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:00:51.477 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:00:51.475+0000 7eff5b9ea640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:00:51.477 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:00:51.475+0000 7eff5b9ea640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:00:51.585 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:00:51.585 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":283,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:00:29.876046+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:55:46.930221+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:58:43.617395+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:59:25.140122+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:00:51.591 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6445 8 365281 365273 0.442743 0 - 0.0299513 2026-02-23T15:00:52.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:00:51 trial068 ceph-mon[38805]: pgmap v3847: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:00:52.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:00:51 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/4054878510' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:00:52.591 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6446 8 365281 365273 0.442674 0 - 0.0299513 2026-02-23T15:00:53.592 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6447 8 365281 365273 0.442605 0 - 0.0299513 2026-02-23T15:00:54.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:00:53 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T15:00:54.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:00:53 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T15:00:54.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:00:53 trial068 ceph-mon[38805]: pgmap v3848: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:00:54.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:00:53 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:00:53] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:00:54.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:00:53 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:00:53.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:00:54.592 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6448 8 365281 365273 0.442537 0 - 0.0299513 2026-02-23T15:00:54.624 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:00:54.741 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:00:54.843 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:00:54.841+0000 7f9e34eac640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:00:54.844 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:00:54.841+0000 7f9e34eac640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:00:54.955 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:00:54.955 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":283,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:00:29.876046+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:55:46.930221+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:58:43.617395+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:59:25.140122+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:00:55.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:00:54 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/172756299' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:00:55.592 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6449 8 365281 365273 0.442468 0 - 0.0299513 2026-02-23T15:00:56.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:00:55 trial068 ceph-mon[38805]: pgmap v3849: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:00:56.592 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6450 8 365281 365273 0.4424 0 - 0.0299513 2026-02-23T15:00:57.592 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6451 8 365281 365273 0.442331 0 - 0.0299513 2026-02-23T15:00:57.995 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:00:58.111 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:00:58.212 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:00:58.209+0000 7f9abf930640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:00:58.212 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:00:58.210+0000 7f9abf930640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:00:58.212 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:00:57 trial068 ceph-mon[38805]: pgmap v3850: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:00:58.322 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:00:58.322 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":283,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:00:29.876046+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:55:46.930221+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:58:43.617395+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:59:25.140122+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:00:58.592 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6452 8 365281 365273 0.442262 0 - 0.0299513 2026-02-23T15:00:59.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:00:58 trial068 ceph-mon[38805]: 5.8 scrub starts 2026-02-23T15:00:59.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:00:58 trial068 ceph-mon[38805]: 5.8 scrub ok 2026-02-23T15:00:59.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:00:58 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1775134080' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:00:59.592 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6453 8 365281 365273 0.442194 0 - 0.0299513 2026-02-23T15:01:00.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:00:59 trial068 ceph-mon[38805]: pgmap v3851: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:01:00.592 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6454 8 365281 365273 0.442125 0 - 0.0299513 2026-02-23T15:01:01.358 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:01:01.476 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:01:01.576 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:01:01.574+0000 7f2c1f801640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:01:01.576 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:01:01.574+0000 7f2c1f801640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:01:01.592 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6455 8 365281 365273 0.442057 0 - 0.0299513 2026-02-23T15:01:01.685 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:01:01.686 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":283,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:00:29.876046+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:55:46.930221+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:58:43.617395+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:59:25.140122+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:01:02.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:01:01 trial068 ceph-mon[38805]: pgmap v3852: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:01:02.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:01:01 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3331527719' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:01:02.592 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6456 8 365281 365273 0.441988 0 - 0.0299513 2026-02-23T15:01:03.592 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6457 8 365281 365273 0.44192 0 - 0.0299513 2026-02-23T15:01:04.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:01:03 trial068 ceph-mon[38805]: pgmap v3853: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:01:04.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:01:03 trial068 ceph-mon[38805]: 5.a scrub starts 2026-02-23T15:01:04.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:01:03 trial068 ceph-mon[38805]: 5.a scrub ok 2026-02-23T15:01:04.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:01:03 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:01:03] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:01:04.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:01:03 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:01:03.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:01:04.592 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6458 8 365281 365273 0.441851 0 - 0.0299513 2026-02-23T15:01:04.723 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:01:04.840 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:01:04.944 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:01:04.942+0000 7f37231f2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:01:04.944 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:01:04.942+0000 7f37231f2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:01:05.056 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:01:05.056 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":283,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:00:29.876046+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:55:46.930221+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:58:43.617395+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:59:25.140122+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:01:05.593 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6459 8 365281 365273 0.441783 0 - 0.0299513 2026-02-23T15:01:06.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:01:05 trial068 ceph-mon[38805]: pgmap v3854: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:01:06.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:01:05 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/4043011773' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:01:06.592 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T15:01:06.592285+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T15:01:06.592 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T15:01:06.593 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6460 8 365281 365273 0.441715 0 - 0.0299513 2026-02-23T15:01:07.592 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6461 8 365281 365273 0.441646 0 - 0.0299513 2026-02-23T15:01:08.095 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:01:08.214 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:01:08.237 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:01:07 trial068 ceph-mon[38805]: pgmap v3855: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:01:08.315 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:01:08.313+0000 7fa5f468c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:01:08.315 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:01:08.313+0000 7fa5f468c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:01:08.424 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:01:08.424 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":283,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:00:29.876046+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:55:46.930221+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:58:43.617395+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:59:25.140122+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:01:08.592 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6462 8 365281 365273 0.441578 0 - 0.0299513 2026-02-23T15:01:09.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:01:08 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2194341816' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:01:09.592 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6463 8 365281 365273 0.44151 0 - 0.0299513 2026-02-23T15:01:10.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:01:09 trial068 ceph-mon[38805]: pgmap v3856: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:01:10.593 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6464 8 365281 365273 0.441441 0 - 0.0299513 2026-02-23T15:01:11.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:01:10 trial068 ceph-mon[38805]: pgmap v3857: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:01:11.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:01:10 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2026-02-23T15:01:11.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:01:10 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-23T15:01:11.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:01:10 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2026-02-23T15:01:11.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:01:10 trial068 ceph-mon[38805]: pgmap v3858: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:01:11.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:01:10 trial068 ceph-mon[38805]: pgmap v3859: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:01:11.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:01:10 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T15:01:11.345 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:01:10 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2026-02-23T15:01:11.461 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:01:11.579 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:01:11.593 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6465 8 365281 365273 0.441373 0 - 0.0299513 2026-02-23T15:01:11.679 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:01:11.677+0000 7f49ae0bc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:01:11.679 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:01:11.677+0000 7f49ae0bc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:01:11.789 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:01:11.790 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":283,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:00:29.876046+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:55:46.930221+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:58:43.617395+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:59:25.140122+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:01:12.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:01:11 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2000445851' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:01:12.593 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6466 8 365281 365273 0.441305 0 - 0.0299513 2026-02-23T15:01:13.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:01:12 trial068 ceph-mon[38805]: pgmap v3860: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:01:13.593 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6467 8 365281 365273 0.441237 0 - 0.0299513 2026-02-23T15:01:14.245 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:01:13 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:01:13] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:01:14.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:01:13 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:01:13.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:01:14.593 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6468 8 365281 365273 0.441168 0 - 0.0299513 2026-02-23T15:01:14.828 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:01:14.946 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:01:15.047 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:01:15.045+0000 7fc9e6ee5640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:01:15.047 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:01:15.045+0000 7fc9e6ee5640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:01:15.157 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:01:15.157 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":283,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:00:29.876046+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:55:46.930221+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:58:43.617395+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:59:25.140122+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:01:15.593 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6469 8 365281 365273 0.4411 0 - 0.0299513 2026-02-23T15:01:16.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:01:15 trial068 ceph-mon[38805]: pgmap v3861: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:01:16.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:01:15 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2234477157' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:01:16.593 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6470 8 365281 365273 0.441032 0 - 0.0299513 2026-02-23T15:01:17.593 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6471 8 365281 365273 0.440964 0 - 0.0299513 2026-02-23T15:01:18.193 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:01:18.219 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:01:17 trial068 ceph-mon[38805]: pgmap v3862: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:01:18.312 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:01:18.415 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:01:18.413+0000 7f865bd0a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:01:18.415 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:01:18.413+0000 7f865bd0a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:01:18.528 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:01:18.529 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":283,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:00:29.876046+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:55:46.930221+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:58:43.617395+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:59:25.140122+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:01:18.593 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6472 8 365281 365273 0.440896 0 - 0.0299513 2026-02-23T15:01:19.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:01:18 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/636989755' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:01:19.593 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6473 8 365281 365273 0.440828 0 - 0.0299513 2026-02-23T15:01:20.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:01:19 trial068 ceph-mon[38805]: pgmap v3863: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:01:20.593 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6474 8 365281 365273 0.44076 0 - 0.0299513 2026-02-23T15:01:21.567 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:01:21.593 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6475 8 365281 365273 0.440691 0 - 0.0299513 2026-02-23T15:01:21.684 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:01:21.786 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:01:21.784+0000 7fc1d0ddc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:01:21.786 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:01:21.785+0000 7fc1d0ddc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:01:21.895 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:01:21.895 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":283,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:00:29.876046+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:55:46.930221+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:58:43.617395+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:59:25.140122+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:01:22.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:01:21 trial068 ceph-mon[38805]: pgmap v3864: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:01:22.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:01:21 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1428359385' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:01:22.593 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6476 8 365281 365273 0.440623 0 - 0.0299513 2026-02-23T15:01:23.594 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6477 8 365281 365273 0.440555 0 - 0.0299513 2026-02-23T15:01:24.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:01:23 trial068 ceph-mon[38805]: pgmap v3865: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:01:24.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:01:23 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:01:23] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:01:24.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:01:23 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:01:23.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:01:24.594 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6478 8 365281 365273 0.440487 0 - 0.0299513 2026-02-23T15:01:24.931 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:01:25.050 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:01:25.157 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:01:25.155+0000 7fcae0819640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:01:25.157 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:01:25.156+0000 7fcae0819640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:01:25.266 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:01:25.266 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":283,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:00:29.876046+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:55:46.930221+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:58:43.617395+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:59:25.140122+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:01:25.594 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6479 8 365281 365273 0.440419 0 - 0.0299513 2026-02-23T15:01:26.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:01:25 trial068 ceph-mon[38805]: 5.5 scrub starts 2026-02-23T15:01:26.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:01:25 trial068 ceph-mon[38805]: 5.5 scrub ok 2026-02-23T15:01:26.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:01:25 trial068 ceph-mon[38805]: pgmap v3866: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:01:26.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:01:25 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2554184726' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:01:26.594 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T15:01:26.593489+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T15:01:26.594 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T15:01:26.594 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6480 8 365281 365273 0.440351 0 - 0.0299513 2026-02-23T15:01:27.594 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6481 8 365281 365273 0.440283 0 - 0.0299513 2026-02-23T15:01:28.303 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:01:28.328 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:01:27 trial068 ceph-mon[38805]: 5.7 scrub starts 2026-02-23T15:01:28.328 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:01:27 trial068 ceph-mon[38805]: 5.7 scrub ok 2026-02-23T15:01:28.329 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:01:27 trial068 ceph-mon[38805]: pgmap v3867: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:01:28.421 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:01:28.514 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:01:28.512+0000 7f26663d8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:01:28.514 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:01:28.512+0000 7f26663d8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:01:28.594 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6482 8 365281 365273 0.440216 0 - 0.0299513 2026-02-23T15:01:28.621 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:01:28.622 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":283,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:00:29.876046+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:55:46.930221+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:58:43.617395+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:59:25.140122+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:01:29.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:01:28 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T15:01:29.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:01:28 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T15:01:29.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:01:28 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2385996050' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:01:29.594 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6483 8 365281 365273 0.440148 0 - 0.0299513 2026-02-23T15:01:30.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:01:29 trial068 ceph-mon[38805]: pgmap v3868: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:01:30.594 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6484 8 365281 365273 0.44008 0 - 0.0299513 2026-02-23T15:01:31.594 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6485 8 365281 365273 0.440012 0 - 0.0299513 2026-02-23T15:01:31.659 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:01:31.777 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:01:31.878 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:01:31.876+0000 7f0ee3f58640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:01:31.879 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:01:31.877+0000 7f0ee3f58640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:01:31.989 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:01:31.989 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":283,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:00:29.876046+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:55:46.930221+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:58:43.617395+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:59:25.140122+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:01:31.989 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:01:31 trial068 ceph-mon[38805]: pgmap v3869: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:01:32.595 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6486 8 365281 365273 0.439944 0 - 0.0299513 2026-02-23T15:01:33.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:01:32 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2335393367' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:01:33.595 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6487 8 365281 365273 0.439876 0 - 0.0299513 2026-02-23T15:01:34.266 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:01:33 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:01:33.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:01:34.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:01:33 trial068 ceph-mon[38805]: pgmap v3870: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:01:34.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:01:33 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:01:33] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:01:34.595 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6488 8 365281 365273 0.439808 0 - 0.0299513 2026-02-23T15:01:35.027 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:01:35.144 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:01:35.165 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:01:34 trial068 ceph-mon[38805]: osdmap e284: 4 total, 3 up, 4 in 2026-02-23T15:01:35.165 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:01:34 trial068 ceph-mon[38805]: osdmap e285: 4 total, 3 up, 4 in 2026-02-23T15:01:35.250 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:01:35.248+0000 7fc7f7c0d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:01:35.250 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:01:35.248+0000 7fc7f7c0d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:01:35.358 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:01:35.358 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":285,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:01:34.884444+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:00:52.362313+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:58:43.617395+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:59:25.140122+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:01:35.595 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6489 8 365281 365273 0.439741 0 - 0.0299513 2026-02-23T15:01:36.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:01:35 trial068 ceph-mon[38805]: 5.e scrub starts 2026-02-23T15:01:36.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:01:35 trial068 ceph-mon[38805]: 5.e scrub ok 2026-02-23T15:01:36.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:01:35 trial068 ceph-mon[38805]: pgmap v3872: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:01:36.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:01:35 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/111342487' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:01:36.595 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6490 8 365281 365273 0.439673 0 - 0.0299513 2026-02-23T15:01:37.595 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6491 8 365281 365273 0.439605 0 - 0.0299513 2026-02-23T15:01:38.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:01:37 trial068 ceph-mon[38805]: pgmap v3874: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:01:38.396 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:01:38.513 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:01:38.595 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6492 8 365281 365273 0.439537 0 - 0.0299513 2026-02-23T15:01:38.615 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:01:38.613+0000 7f098c2a1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:01:38.615 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:01:38.613+0000 7f098c2a1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:01:38.724 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:01:38.724 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":285,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:01:34.884444+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:00:52.362313+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:58:43.617395+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:59:25.140122+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:01:39.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:01:38 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3899643575' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:01:39.595 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6493 8 365281 365273 0.43947 0 - 0.0299513 2026-02-23T15:01:40.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:01:39 trial068 ceph-mon[38805]: pgmap v3875: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:01:40.595 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6494 8 365281 365273 0.439402 0 - 0.0299513 2026-02-23T15:01:41.595 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6495 8 365281 365273 0.439334 0 - 0.0299513 2026-02-23T15:01:41.761 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:01:41.879 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:01:41.984 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:01:41.982+0000 7f71c61ee640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:01:41.985 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:01:41.983+0000 7f71c61ee640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:01:42.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:01:41 trial068 ceph-mon[38805]: pgmap v3876: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:01:42.094 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:01:42.095 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":285,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:01:34.884444+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:00:52.362313+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:58:43.617395+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:59:25.140122+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:01:42.595 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6496 8 365281 365273 0.439267 0 - 0.0299513 2026-02-23T15:01:43.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:01:42 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3966979108' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:01:43.595 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6497 8 365281 365273 0.439199 0 - 0.0299513 2026-02-23T15:01:44.245 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:01:43 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:01:43] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:01:44.245 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:01:43 trial068 ceph-mon[38805]: pgmap v3877: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:01:44.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:01:43 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:01:43.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:01:44.595 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6498 8 365281 365273 0.439132 0 - 0.0299513 2026-02-23T15:01:45.137 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:01:45.257 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:01:45.355 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:01:45.353+0000 7f65f81a8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:01:45.355 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:01:45.353+0000 7f65f81a8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:01:45.464 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:01:45.464 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":285,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:01:34.884444+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:00:52.362313+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:58:43.617395+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:59:25.140122+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:01:45.596 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6499 8 365281 365273 0.439064 0 - 0.0299513 2026-02-23T15:01:46.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:01:45 trial068 ceph-mon[38805]: pgmap v3878: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:01:46.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:01:45 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2340162667' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:01:46.595 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T15:01:46.595280+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T15:01:46.596 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T15:01:46.596 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6500 8 365281 365273 0.438996 0 - 0.0299513 2026-02-23T15:01:47.596 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6501 8 365281 365273 0.438929 0 - 0.0299513 2026-02-23T15:01:48.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:01:47 trial068 ceph-mon[38805]: pgmap v3879: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:01:48.501 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:01:48.596 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6502 8 365281 365273 0.438861 0 - 0.0299513 2026-02-23T15:01:48.620 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:01:48.718 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:01:48.716+0000 7fa41765e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:01:48.719 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:01:48.716+0000 7fa41765e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:01:48.827 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:01:48.828 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":285,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:01:34.884444+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:00:52.362313+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:58:43.617395+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:59:25.140122+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:01:49.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:01:48 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1977802957' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:01:49.596 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6503 8 365281 365273 0.438794 0 - 0.0299513 2026-02-23T15:01:50.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:01:49 trial068 ceph-mon[38805]: pgmap v3880: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:01:50.596 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6504 8 365281 365273 0.438726 0 - 0.0299513 2026-02-23T15:01:51.596 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6505 8 365281 365273 0.438659 0 - 0.0299513 2026-02-23T15:01:51.864 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:01:51.982 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:01:52.079 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:01:52.078+0000 7f9def016640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:01:52.079 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:01:52.078+0000 7f9def016640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:01:52.190 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:01:52.190 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":285,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:01:34.884444+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:00:52.362313+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:58:43.617395+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:59:25.140122+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:01:52.190 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:01:51 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T15:01:52.190 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:01:51 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T15:01:52.190 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:01:51 trial068 ceph-mon[38805]: pgmap v3881: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:01:52.596 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6506 8 365281 365273 0.438592 0 - 0.0299513 2026-02-23T15:01:53.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:01:52 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3707872730' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:01:53.596 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6507 8 365281 365273 0.438524 0 - 0.0299513 2026-02-23T15:01:54.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:01:53 trial068 ceph-mon[38805]: pgmap v3882: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:01:54.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:01:53 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:01:53] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:01:54.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:01:53 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:01:53.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:01:54.596 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6508 8 365281 365273 0.438457 0 - 0.0299513 2026-02-23T15:01:55.248 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:01:55.363 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:01:55.472 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:01:55.469+0000 7f18dee41640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:01:55.472 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:01:55.469+0000 7f18dee41640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:01:55.581 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:01:55.582 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":285,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:01:34.884444+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:00:52.362313+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:58:43.617395+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:59:25.140122+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:01:55.596 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6509 8 365281 365273 0.438389 0 - 0.0299513 2026-02-23T15:01:56.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:01:55 trial068 ceph-mon[38805]: pgmap v3883: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:01:56.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:01:55 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2867684991' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:01:56.596 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6510 8 365281 365273 0.438322 0 - 0.0299513 2026-02-23T15:01:57.596 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6511 8 365281 365273 0.438255 0 - 0.0299513 2026-02-23T15:01:58.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:01:57 trial068 ceph-mon[38805]: pgmap v3884: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:01:58.596 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6512 8 365281 365273 0.438187 0 - 0.0299513 2026-02-23T15:01:58.619 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:01:58.737 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:01:58.835 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:01:58.832+0000 7f48148ea640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:01:58.835 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:01:58.833+0000 7f48148ea640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:01:58.944 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:01:58.944 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":285,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:01:34.884444+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:00:52.362313+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:58:43.617395+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:59:25.140122+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:01:59.596 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6513 8 365281 365273 0.43812 0 - 0.0299513 2026-02-23T15:02:00.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:01:59 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T15:02:00.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:01:59 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T15:02:00.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:01:59 trial068 ceph-mon[38805]: pgmap v3885: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:02:00.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:01:59 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/213188983' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:02:00.597 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6514 8 365281 365273 0.438053 0 - 0.0299513 2026-02-23T15:02:01.597 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6515 8 365281 365273 0.437986 0 - 0.0299513 2026-02-23T15:02:01.998 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:02:02.115 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:02:02.216 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:02:02.214+0000 7f844965b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:02:02.217 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:02:01 trial068 ceph-mon[38805]: pgmap v3886: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:02:02.217 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:02:02.214+0000 7f844965b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:02:02.325 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:02:02.325 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":285,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:01:34.884444+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:00:52.362313+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:58:43.617395+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:59:25.140122+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:02:02.597 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6516 8 365281 365273 0.437918 0 - 0.0299513 2026-02-23T15:02:03.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:02:02 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3339849297' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:02:03.597 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6517 8 365281 365273 0.437851 0 - 0.0299513 2026-02-23T15:02:04.265 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:02:03 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:02:03.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:02:04.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:02:03 trial068 ceph-mon[38805]: pgmap v3887: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:02:04.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:02:03 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:02:03] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:02:04.597 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6518 8 365281 365273 0.437784 0 - 0.0299513 2026-02-23T15:02:05.377 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:02:05.496 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:02:05.597 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6519 8 365281 365273 0.437717 0 - 0.0299513 2026-02-23T15:02:05.598 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:02:05.595+0000 7f79e2321640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:02:05.598 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:02:05.596+0000 7f79e2321640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:02:05.707 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:02:05.707 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":285,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:01:34.884444+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:00:52.362313+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:58:43.617395+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:59:25.140122+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:02:06.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:02:05 trial068 ceph-mon[38805]: 5.a scrub starts 2026-02-23T15:02:06.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:02:05 trial068 ceph-mon[38805]: 5.a scrub ok 2026-02-23T15:02:06.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:02:05 trial068 ceph-mon[38805]: pgmap v3888: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:02:06.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:02:05 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1198663643' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:02:06.597 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T15:02:06.597102+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T15:02:06.598 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T15:02:06.598 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6520 8 365281 365273 0.43765 0 - 0.0299513 2026-02-23T15:02:07.597 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6521 8 365281 365273 0.437583 0 - 0.0299513 2026-02-23T15:02:08.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:02:07 trial068 ceph-mon[38805]: pgmap v3889: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:02:08.597 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6522 8 365281 365273 0.437516 0 - 0.0299513 2026-02-23T15:02:08.744 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:02:08.861 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:02:08.963 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:02:08.962+0000 7fc066db8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:02:08.964 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:02:08.962+0000 7fc066db8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:02:09.072 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:02:09.072 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":285,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:01:34.884444+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:00:52.362313+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:58:43.617395+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:59:25.140122+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:02:09.598 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6523 8 365281 365273 0.437449 0 - 0.0299513 2026-02-23T15:02:10.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:02:09 trial068 ceph-mon[38805]: pgmap v3890: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:02:10.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:02:09 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3337607244' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:02:10.597 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6524 8 365281 365273 0.437381 0 - 0.0299513 2026-02-23T15:02:11.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:02:10 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2026-02-23T15:02:11.598 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6525 8 365281 365273 0.437314 0 - 0.0299513 2026-02-23T15:02:12.109 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:02:12.225 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:02:12.248 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:02:11 trial068 ceph-mon[38805]: pgmap v3891: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:02:12.248 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:02:11 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-23T15:02:12.248 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:02:11 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2026-02-23T15:02:12.248 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:02:11 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T15:02:12.248 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:02:11 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2026-02-23T15:02:12.330 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:02:12.328+0000 7fca74d90640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:02:12.330 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:02:12.328+0000 7fca74d90640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:02:12.438 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:02:12.438 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":285,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:01:34.884444+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:00:52.362313+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:58:43.617395+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:59:25.140122+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:02:12.598 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6526 8 365281 365273 0.437247 0 - 0.0299513 2026-02-23T15:02:13.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:02:12 trial068 ceph-mon[38805]: pgmap v3892: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:02:13.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:02:12 trial068 ceph-mon[38805]: pgmap v3893: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:02:13.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:02:12 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2586881280' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:02:13.598 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6527 8 365281 365273 0.43718 0 - 0.0299513 2026-02-23T15:02:14.244 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:02:13 trial068 ceph-mon[38805]: 5.8 scrub starts 2026-02-23T15:02:14.244 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:02:13 trial068 ceph-mon[38805]: 5.8 scrub ok 2026-02-23T15:02:14.245 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:02:13 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:02:13] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:02:14.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:02:13 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:02:13.974Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:02:14.598 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6528 8 365281 365273 0.437113 0 - 0.0299513 2026-02-23T15:02:15.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:02:14 trial068 ceph-mon[38805]: pgmap v3894: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:02:15.477 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:02:15.594 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:02:15.598 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6529 8 365281 365273 0.437047 0 - 0.0299513 2026-02-23T15:02:15.696 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:02:15.694+0000 7f59c347b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:02:15.696 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:02:15.694+0000 7f59c347b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:02:15.805 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:02:15.806 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":285,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:01:34.884444+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:00:52.362313+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:58:43.617395+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:59:25.140122+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:02:16.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:02:15 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2747415383' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:02:16.598 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6530 8 365281 365273 0.43698 0 - 0.0299513 2026-02-23T15:02:17.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:02:16 trial068 ceph-mon[38805]: pgmap v3895: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:02:17.598 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6531 8 365281 365273 0.436913 0 - 0.0299513 2026-02-23T15:02:18.598 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6532 8 365281 365273 0.436846 0 - 0.0299513 2026-02-23T15:02:18.842 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:02:18.962 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:02:19.062 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:02:19.060+0000 7f955913b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:02:19.062 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:02:19.061+0000 7f955913b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:02:19.173 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:02:19.173 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":285,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:01:34.884444+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:00:52.362313+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:58:43.617395+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:59:25.140122+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:02:19.174 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:02:18 trial068 ceph-mon[38805]: pgmap v3896: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:02:19.598 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6533 8 365281 365273 0.436779 0 - 0.0299513 2026-02-23T15:02:20.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:02:19 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/204172846' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:02:20.598 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6534 8 365281 365273 0.436712 0 - 0.0299513 2026-02-23T15:02:21.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:02:20 trial068 ceph-mon[38805]: pgmap v3897: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:02:21.598 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6535 8 365281 365273 0.436645 0 - 0.0299513 2026-02-23T15:02:22.212 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:02:22.328 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:02:22.427 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:02:22.425+0000 7f67df023640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:02:22.427 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:02:22.425+0000 7f67df023640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:02:22.536 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:02:22.536 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":285,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:01:34.884444+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:00:52.362313+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:58:43.617395+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:59:25.140122+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:02:22.598 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6536 8 365281 365273 0.436578 0 - 0.0299513 2026-02-23T15:02:23.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:02:22 trial068 ceph-mon[38805]: pgmap v3898: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:02:23.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:02:22 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1586078118' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:02:23.598 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6537 8 365281 365273 0.436512 0 - 0.0299513 2026-02-23T15:02:24.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:02:23 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:02:23] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:02:24.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:02:23 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:02:23.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:02:24.598 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6538 8 365281 365273 0.436445 0 - 0.0299513 2026-02-23T15:02:25.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:02:24 trial068 ceph-mon[38805]: pgmap v3899: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:02:25.574 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:02:25.599 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6539 8 365281 365273 0.436378 0 - 0.0299513 2026-02-23T15:02:25.691 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:02:25.793 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:02:25.791+0000 7fe61d3e3640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:02:25.793 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:02:25.791+0000 7fe61d3e3640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:02:25.901 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:02:25.902 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":285,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:01:34.884444+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:00:52.362313+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:58:43.617395+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:59:25.140122+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:02:26.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:02:25 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2744095128' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:02:26.599 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T15:02:26.598595+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T15:02:26.599 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T15:02:26.599 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6540 8 365281 365273 0.436311 0 - 0.0299513 2026-02-23T15:02:27.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:02:26 trial068 ceph-mon[38805]: pgmap v3900: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:02:27.599 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6541 8 365281 365273 0.436245 0 - 0.0299513 2026-02-23T15:02:28.599 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6542 8 365281 365273 0.436178 0 - 0.0299513 2026-02-23T15:02:28.938 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:02:29.055 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:02:29.150 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:02:29.149+0000 7fdb79473640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:02:29.151 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:02:29.149+0000 7fdb79473640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:02:29.263 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:02:29.263 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":285,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:01:34.884444+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:00:52.362313+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:58:43.617395+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:59:25.140122+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:02:29.263 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:02:28 trial068 ceph-mon[38805]: pgmap v3901: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:02:29.599 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6543 8 365281 365273 0.436111 0 - 0.0299513 2026-02-23T15:02:30.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:02:29 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3557475709' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:02:30.599 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6544 8 365281 365273 0.436045 0 - 0.0299513 2026-02-23T15:02:31.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:02:30 trial068 ceph-mon[38805]: pgmap v3902: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:02:31.599 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6545 8 365281 365273 0.435978 0 - 0.0299513 2026-02-23T15:02:32.300 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:02:32.416 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:02:32.517 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:02:32.515+0000 7f9ba0b05640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:02:32.517 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:02:32.515+0000 7f9ba0b05640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:02:32.599 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6546 8 365281 365273 0.435912 0 - 0.0299513 2026-02-23T15:02:32.626 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:02:32.627 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":285,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:01:34.884444+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:00:52.362313+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:58:43.617395+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:59:25.140122+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:02:33.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:02:32 trial068 ceph-mon[38805]: pgmap v3903: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:02:33.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:02:32 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2043929204' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:02:33.599 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6547 8 365281 365273 0.435845 0 - 0.0299513 2026-02-23T15:02:34.266 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:02:33 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:02:33.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:02:34.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:02:33 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:02:33] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:02:34.599 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6548 8 365281 365273 0.435778 0 - 0.0299513 2026-02-23T15:02:35.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:02:34 trial068 ceph-mon[38805]: pgmap v3904: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:02:35.599 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6549 8 365281 365273 0.435712 0 - 0.0299513 2026-02-23T15:02:35.664 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:02:35.782 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:02:35.888 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:02:35.886+0000 7f2b8da5b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:02:35.888 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:02:35.886+0000 7f2b8da5b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:02:35.998 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:02:35.998 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":285,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:01:34.884444+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:00:52.362313+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:58:43.617395+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:59:25.140122+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:02:36.599 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6550 8 365281 365273 0.435645 0 - 0.0299513 2026-02-23T15:02:37.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:02:36 trial068 ceph-mon[38805]: pgmap v3905: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:02:37.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:02:36 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1432542928' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:02:37.599 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6551 8 365281 365273 0.435579 0 - 0.0299513 2026-02-23T15:02:38.600 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6552 8 365281 365273 0.435512 0 - 0.0299513 2026-02-23T15:02:39.034 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:02:39.154 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:02:39.257 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:02:39.255+0000 7f9b8e407640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:02:39.257 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:02:39.256+0000 7f9b8e407640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:02:39.258 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:02:38 trial068 ceph-mon[38805]: pgmap v3906: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:02:39.367 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:02:39.367 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":285,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:01:34.884444+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:00:52.362313+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:58:43.617395+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:59:25.140122+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:02:39.600 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6553 8 365281 365273 0.435446 0 - 0.0299513 2026-02-23T15:02:40.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:02:39 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3913101971' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:02:40.600 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6554 8 365281 365273 0.435379 0 - 0.0299513 2026-02-23T15:02:41.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:02:40 trial068 ceph-mon[38805]: pgmap v3907: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:02:41.600 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6555 8 365281 365273 0.435313 0 - 0.0299513 2026-02-23T15:02:42.405 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:02:42.524 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:02:42.600 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6556 8 365281 365273 0.435247 0 - 0.0299513 2026-02-23T15:02:42.632 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:02:42.631+0000 7f80b5831640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:02:42.633 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:02:42.631+0000 7f80b5831640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:02:42.744 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:02:42.745 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":285,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:01:34.884444+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:00:52.362313+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:58:43.617395+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:59:25.140122+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:02:43.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:02:42 trial068 ceph-mon[38805]: pgmap v3908: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:02:43.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:02:42 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/720174602' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:02:43.600 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6557 8 365281 365273 0.43518 0 - 0.0299513 2026-02-23T15:02:44.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:02:43 trial068 ceph-mon[38805]: 5.5 scrub starts 2026-02-23T15:02:44.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:02:43 trial068 ceph-mon[38805]: 5.5 scrub ok 2026-02-23T15:02:44.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:02:43 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:02:43] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:02:44.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:02:43 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:02:43.974Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:02:44.600 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6558 8 365281 365273 0.435114 0 - 0.0299513 2026-02-23T15:02:45.245 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:02:44 trial068 ceph-mon[38805]: pgmap v3909: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:02:45.600 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6559 8 365281 365273 0.435048 0 - 0.0299513 2026-02-23T15:02:45.784 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:02:45.903 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:02:46.003 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:02:46.001+0000 7f64de880640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:02:46.003 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:02:46.001+0000 7f64de880640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:02:46.112 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:02:46.113 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":285,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:01:34.884444+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:00:52.362313+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:58:43.617395+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:59:25.140122+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:02:46.600 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T15:02:46.600042+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T15:02:46.600 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T15:02:46.601 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6560 8 365281 365273 0.434981 0 - 0.0299513 2026-02-23T15:02:47.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:02:46 trial068 ceph-mon[38805]: pgmap v3910: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:02:47.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:02:46 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/500779416' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:02:47.600 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6561 8 365281 365273 0.434915 0 - 0.0299513 2026-02-23T15:02:48.600 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6562 8 365281 365273 0.434849 0 - 0.0299513 2026-02-23T15:02:49.150 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:02:49.268 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:02:49.290 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:02:48 trial068 ceph-mon[38805]: pgmap v3911: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:02:49.369 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:02:49.367+0000 7fbae770f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:02:49.370 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:02:49.368+0000 7fbae770f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:02:49.480 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:02:49.480 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":285,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:01:34.884444+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:00:52.362313+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:58:43.617395+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:59:25.140122+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:02:49.601 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6563 8 365281 365273 0.434782 0 - 0.0299513 2026-02-23T15:02:50.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:02:49 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2885123304' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:02:50.601 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6564 8 365281 365273 0.434716 0 - 0.0299513 2026-02-23T15:02:51.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:02:50 trial068 ceph-mon[38805]: pgmap v3912: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:02:51.601 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6565 8 365281 365273 0.43465 0 - 0.0299513 2026-02-23T15:02:52.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:02:51 trial068 ceph-mon[38805]: 5.7 scrub starts 2026-02-23T15:02:52.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:02:51 trial068 ceph-mon[38805]: 5.7 scrub ok 2026-02-23T15:02:52.518 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:02:52.601 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6566 8 365281 365273 0.434584 0 - 0.0299513 2026-02-23T15:02:52.635 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:02:52.736 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:02:52.734+0000 7f165da0e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:02:52.736 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:02:52.735+0000 7f165da0e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:02:52.848 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:02:52.848 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":285,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:01:34.884444+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:00:52.362313+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:58:43.617395+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:59:25.140122+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:02:53.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:02:52 trial068 ceph-mon[38805]: pgmap v3913: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:02:53.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:02:52 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/332588327' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:02:53.601 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6567 8 365281 365273 0.434518 0 - 0.0299513 2026-02-23T15:02:54.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:02:53 trial068 ceph-mon[38805]: pgmap v3914: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:02:54.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:02:53 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:02:53] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:02:54.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:02:53 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:02:53.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:02:54.601 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6568 8 365281 365273 0.434451 0 - 0.0299513 2026-02-23T15:02:55.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:02:54 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T15:02:55.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:02:54 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T15:02:55.601 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6569 8 365281 365273 0.434385 0 - 0.0299513 2026-02-23T15:02:55.885 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:02:56.003 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:02:56.102 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:02:56.100+0000 7f8c0e2a3640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:02:56.103 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:02:56.101+0000 7f8c0e2a3640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:02:56.212 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:02:56.212 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":285,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:01:34.884444+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:00:52.362313+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:58:43.617395+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:59:25.140122+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:02:56.212 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:02:55 trial068 ceph-mon[38805]: pgmap v3915: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:02:56.601 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6570 8 365281 365273 0.434319 0 - 0.0299513 2026-02-23T15:02:57.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:02:56 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3078000637' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:02:57.601 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6571 8 365281 365273 0.434253 0 - 0.0299513 2026-02-23T15:02:58.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:02:57 trial068 ceph-mon[38805]: pgmap v3916: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:02:58.601 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6572 8 365281 365273 0.434187 0 - 0.0299513 2026-02-23T15:02:59.250 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:02:59.275 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:02:58 trial068 ceph-mon[38805]: 5.e scrub starts 2026-02-23T15:02:59.276 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:02:58 trial068 ceph-mon[38805]: 5.e scrub ok 2026-02-23T15:02:59.369 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:02:59.469 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:02:59.467+0000 7fb6845c7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:02:59.469 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:02:59.467+0000 7fb6845c7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:02:59.577 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:02:59.578 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":285,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:01:34.884444+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:00:52.362313+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:58:43.617395+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:59:25.140122+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:02:59.601 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6573 8 365281 365273 0.434121 0 - 0.0299513 2026-02-23T15:03:00.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:02:59 trial068 ceph-mon[38805]: pgmap v3917: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:03:00.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:02:59 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2460706816' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:03:00.601 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6574 8 365281 365273 0.434055 0 - 0.0299513 2026-02-23T15:03:01.601 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6575 8 365281 365273 0.433989 0 - 0.0299513 2026-02-23T15:03:02.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:03:02 trial068 ceph-mon[38805]: pgmap v3918: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:03:02.601 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6576 8 365281 365273 0.433923 0 - 0.0299513 2026-02-23T15:03:02.614 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:03:02.735 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:03:02.835 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:03:02.833+0000 7f423571e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:03:02.836 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:03:02.834+0000 7f423571e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:03:02.946 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:03:02.946 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":285,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:01:34.884444+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:00:52.362313+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:58:43.617395+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:59:25.140122+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:03:03.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:03:03 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/235345250' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:03:03.602 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6577 8 365281 365273 0.433857 0 - 0.0299513 2026-02-23T15:03:04.265 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:03:03 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:03:03.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:03:04.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:03:04 trial068 ceph-mon[38805]: pgmap v3919: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:03:04.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:03:03 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:03:03] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:03:04.601 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6578 8 365281 365273 0.433791 0 - 0.0299513 2026-02-23T15:03:05.602 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6579 8 365281 365273 0.433725 0 - 0.0299513 2026-02-23T15:03:05.983 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:03:06.110 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:03:06.209 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:03:06.207+0000 7f75085dc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:03:06.209 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:03:06.207+0000 7f75085dc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:03:06.318 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:03:06.319 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":285,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:01:34.884444+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:00:52.362313+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:58:43.617395+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:59:25.140122+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:03:06.319 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:03:06 trial068 ceph-mon[38805]: pgmap v3920: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:03:06.602 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T15:03:06.601624+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T15:03:06.602 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T15:03:06.602 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6580 8 365281 365273 0.433659 0 - 0.0299513 2026-02-23T15:03:07.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:03:07 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2474592577' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:03:07.602 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6581 8 365281 365273 0.433593 0 - 0.0299513 2026-02-23T15:03:08.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:03:08 trial068 ceph-mon[38805]: pgmap v3921: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:03:08.602 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6582 8 365281 365273 0.433527 0 - 0.0299513 2026-02-23T15:03:09.357 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:03:09.476 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:03:09.576 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:03:09.574+0000 7fc99f359640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:03:09.576 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:03:09.575+0000 7fc99f359640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:03:09.602 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6583 8 365281 365273 0.433461 0 - 0.0299513 2026-02-23T15:03:09.685 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:03:09.686 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":285,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:01:34.884444+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:00:52.362313+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:58:43.617395+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:59:25.140122+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:03:10.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:03:10 trial068 ceph-mon[38805]: pgmap v3922: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:03:10.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:03:10 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2910121807' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:03:10.602 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6584 8 365281 365273 0.433396 0 - 0.0299513 2026-02-23T15:03:11.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:03:11 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2026-02-23T15:03:11.602 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6585 8 365281 365273 0.43333 0 - 0.0299513 2026-02-23T15:03:12.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:03:12 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T15:03:12.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:03:12 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T15:03:12.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:03:12 trial068 ceph-mon[38805]: pgmap v3923: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:03:12.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:03:12 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-23T15:03:12.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:03:12 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2026-02-23T15:03:12.595 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:03:12 trial068 ceph-mon[38805]: pgmap v3924: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:03:12.595 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:03:12 trial068 ceph-mon[38805]: pgmap v3925: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:03:12.595 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:03:12 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T15:03:12.595 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:03:12 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2026-02-23T15:03:12.602 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6586 8 365281 365273 0.433264 0 - 0.0299513 2026-02-23T15:03:12.725 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:03:12.844 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:03:12.952 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:03:12.950+0000 7f5b8546f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:03:12.952 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:03:12.950+0000 7f5b8546f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:03:13.065 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:03:13.066 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":285,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:01:34.884444+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:00:52.362313+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:58:43.617395+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:59:25.140122+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:03:13.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:03:13 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T15:03:13.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:03:13 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T15:03:13.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:03:13 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2632231256' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:03:13.602 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6587 8 365281 365273 0.433198 0 - 0.0299513 2026-02-23T15:03:14.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:03:14 trial068 ceph-mon[38805]: pgmap v3926: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:03:14.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:03:13 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:03:13] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:03:14.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:03:13 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:03:13.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:03:14.602 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6588 8 365281 365273 0.433132 0 - 0.0299513 2026-02-23T15:03:15.602 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6589 8 365281 365273 0.433067 0 - 0.0299513 2026-02-23T15:03:16.104 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:03:16.223 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:03:16.327 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:03:16.325+0000 7f5b54e0d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:03:16.327 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:03:16.325+0000 7f5b54e0d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:03:16.436 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:03:16.437 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":285,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:01:34.884444+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:00:52.362313+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:58:43.617395+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:59:25.140122+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:03:16.602 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6590 8 365281 365273 0.433001 0 - 0.0299513 2026-02-23T15:03:17.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:03:16 trial068 ceph-mon[38805]: pgmap v3927: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:03:17.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:03:16 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1173606094' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:03:17.602 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6591 8 365281 365273 0.432935 0 - 0.0299513 2026-02-23T15:03:18.602 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6592 8 365281 365273 0.43287 0 - 0.0299513 2026-02-23T15:03:19.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:03:18 trial068 ceph-mon[38805]: pgmap v3928: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:03:19.476 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:03:19.595 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:03:19.603 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6593 8 365281 365273 0.432804 0 - 0.0299513 2026-02-23T15:03:19.696 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:03:19.694+0000 7f381ae23640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:03:19.696 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:03:19.694+0000 7f381ae23640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:03:19.807 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:03:19.807 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":285,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:01:34.884444+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:00:52.362313+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:58:43.617395+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:59:25.140122+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:03:20.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:03:19 trial068 ceph-mon[38805]: 5.8 scrub starts 2026-02-23T15:03:20.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:03:19 trial068 ceph-mon[38805]: 5.8 scrub ok 2026-02-23T15:03:20.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:03:19 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3264914914' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:03:20.603 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6594 8 365281 365273 0.432738 0 - 0.0299513 2026-02-23T15:03:21.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:03:20 trial068 ceph-mon[38805]: pgmap v3929: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:03:21.603 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6595 8 365281 365273 0.432673 0 - 0.0299513 2026-02-23T15:03:22.603 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6596 8 365281 365273 0.432607 0 - 0.0299513 2026-02-23T15:03:22.844 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:03:22.962 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:03:23.061 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:03:23.059+0000 7f893943c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:03:23.061 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:03:23.059+0000 7f893943c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:03:23.170 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:03:23.170 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":285,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:01:34.884444+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:00:52.362313+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:58:43.617395+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:59:25.140122+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:03:23.170 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:03:22 trial068 ceph-mon[38805]: pgmap v3930: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:03:23.603 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6597 8 365281 365273 0.432542 0 - 0.0299513 2026-02-23T15:03:24.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:03:23 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/181783123' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:03:24.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:03:23 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:03:23] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:03:24.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:03:23 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:03:23.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:03:24.603 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6598 8 365281 365273 0.432476 0 - 0.0299513 2026-02-23T15:03:25.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:03:24 trial068 ceph-mon[38805]: pgmap v3931: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:03:25.603 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6599 8 365281 365273 0.43241 0 - 0.0299513 2026-02-23T15:03:26.222 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:03:26.340 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:03:26.446 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:03:26.444+0000 7f58ec273640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:03:26.446 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:03:26.444+0000 7f58ec273640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:03:26.556 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:03:26.556 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":285,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:01:34.884444+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:00:52.362313+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:58:43.617395+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:59:25.140122+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:03:26.603 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T15:03:26.603106+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T15:03:26.603 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T15:03:26.603 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6600 8 365281 365273 0.432345 0 - 0.0299513 2026-02-23T15:03:27.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:03:26 trial068 ceph-mon[38805]: pgmap v3932: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:03:27.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:03:26 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1756228699' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:03:27.603 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6601 8 365281 365273 0.432279 0 - 0.0299513 2026-02-23T15:03:28.603 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6602 8 365281 365273 0.432214 0 - 0.0299513 2026-02-23T15:03:29.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:03:28 trial068 ceph-mon[38805]: pgmap v3933: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:03:29.593 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:03:29.603 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6603 8 365281 365273 0.432149 0 - 0.0299513 2026-02-23T15:03:29.712 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:03:29.812 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:03:29.810+0000 7f149e92d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:03:29.812 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:03:29.810+0000 7f149e92d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:03:29.922 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:03:29.922 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":285,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:01:34.884444+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:00:52.362313+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:58:43.617395+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:59:25.140122+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:03:30.603 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6604 8 365281 365273 0.432083 0 - 0.0299513 2026-02-23T15:03:31.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:03:30 trial068 ceph-mon[38805]: pgmap v3934: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:03:31.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:03:30 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/290992602' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:03:31.603 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6605 8 365281 365273 0.432018 0 - 0.0299513 2026-02-23T15:03:32.604 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6606 8 365281 365273 0.431952 0 - 0.0299513 2026-02-23T15:03:32.959 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:03:33.077 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:03:33.177 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:03:33.175+0000 7f6938bdf640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:03:33.177 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:03:33.175+0000 7f6938bdf640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:03:33.177 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:03:32 trial068 ceph-mon[38805]: pgmap v3935: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:03:33.285 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:03:33.285 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":285,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:01:34.884444+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:00:52.362313+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:58:43.617395+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:59:25.140122+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:03:33.604 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6607 8 365281 365273 0.431887 0 - 0.0299513 2026-02-23T15:03:34.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:03:33 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/938026942' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:03:34.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:03:33 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:03:33] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:03:34.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:03:33 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:03:33.974Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:03:34.604 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6608 8 365281 365273 0.431822 0 - 0.0299513 2026-02-23T15:03:35.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:03:34 trial068 ceph-mon[38805]: pgmap v3936: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:03:35.604 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6609 8 365281 365273 0.431756 0 - 0.0299513 2026-02-23T15:03:36.323 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:03:36.442 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:03:36.544 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:03:36.542+0000 7f4313a08640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:03:36.544 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:03:36.542+0000 7f4313a08640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:03:36.604 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6610 8 365281 365273 0.431691 0 - 0.0299513 2026-02-23T15:03:36.652 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:03:36.652 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":285,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:01:34.884444+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:00:52.362313+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:58:43.617395+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:59:25.140122+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:03:37.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:03:36 trial068 ceph-mon[38805]: pgmap v3937: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:03:37.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:03:36 trial068 ceph-mon[38805]: 5.a scrub starts 2026-02-23T15:03:37.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:03:36 trial068 ceph-mon[38805]: 5.a scrub ok 2026-02-23T15:03:37.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:03:36 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/928474335' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:03:37.604 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6611 8 365281 365273 0.431626 0 - 0.0299513 2026-02-23T15:03:38.604 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6612 8 365281 365273 0.43156 0 - 0.0299513 2026-02-23T15:03:39.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:03:38 trial068 ceph-mon[38805]: pgmap v3938: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:03:39.604 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6613 8 365281 365273 0.431495 0 - 0.0299513 2026-02-23T15:03:39.691 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:03:39.807 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:03:39.905 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:03:39.903+0000 7fd75c2b7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:03:39.905 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:03:39.904+0000 7fd75c2b7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:03:40.015 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:03:40.016 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":285,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:01:34.884444+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:00:52.362313+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:58:43.617395+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:59:25.140122+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:03:40.604 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6614 8 365281 365273 0.43143 0 - 0.0299513 2026-02-23T15:03:41.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:03:40 trial068 ceph-mon[38805]: pgmap v3939: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:03:41.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:03:40 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/405527016' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:03:41.604 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6615 8 365281 365273 0.431365 0 - 0.0299513 2026-02-23T15:03:42.604 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6616 8 365281 365273 0.431299 0 - 0.0299513 2026-02-23T15:03:43.053 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:03:43.171 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:03:43.193 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:03:42 trial068 ceph-mon[38805]: pgmap v3940: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:03:43.270 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:03:43.269+0000 7fccfc6d0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:03:43.270 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:03:43.269+0000 7fccfc6d0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:03:43.378 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:03:43.378 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":285,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:01:34.884444+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:00:52.362313+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:58:43.617395+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:59:25.140122+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:03:43.604 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6617 8 365281 365273 0.431234 0 - 0.0299513 2026-02-23T15:03:44.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:03:43 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2093006679' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:03:44.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:03:43 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:03:43] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:03:44.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:03:43 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:03:43.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:03:44.604 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6618 8 365281 365273 0.431169 0 - 0.0299513 2026-02-23T15:03:45.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:03:44 trial068 ceph-mon[38805]: pgmap v3941: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:03:45.604 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6619 8 365281 365273 0.431104 0 - 0.0299513 2026-02-23T15:03:46.415 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:03:46.532 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:03:46.605 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T15:03:46.604549+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T15:03:46.605 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T15:03:46.605 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6620 8 365281 365273 0.431039 0 - 0.0299513 2026-02-23T15:03:46.634 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:03:46.632+0000 7f38bdfec640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:03:46.634 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:03:46.632+0000 7f38bdfec640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:03:46.744 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:03:46.745 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":285,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:01:34.884444+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:00:52.362313+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:58:43.617395+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:59:25.140122+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:03:47.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:03:46 trial068 ceph-mon[38805]: pgmap v3942: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:03:47.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:03:46 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3513309260' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:03:47.605 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6621 8 365281 365273 0.430974 0 - 0.0299513 2026-02-23T15:03:48.605 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6622 8 365281 365273 0.430909 0 - 0.0299513 2026-02-23T15:03:49.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:03:48 trial068 ceph-mon[38805]: pgmap v3943: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:03:49.605 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6623 8 365281 365273 0.430844 0 - 0.0299513 2026-02-23T15:03:49.783 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:03:49.901 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:03:50.010 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:03:50.008+0000 7f2decf2c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:03:50.010 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:03:50.008+0000 7f2decf2c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:03:50.122 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:03:50.122 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":285,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:01:34.884444+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:00:52.362313+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:58:43.617395+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:59:25.140122+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:03:50.605 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6624 8 365281 365273 0.430779 0 - 0.0299513 2026-02-23T15:03:51.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:03:50 trial068 ceph-mon[38805]: pgmap v3944: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:03:51.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:03:50 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3840140345' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:03:51.605 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6625 8 365281 365273 0.430714 0 - 0.0299513 2026-02-23T15:03:52.605 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6626 8 365281 365273 0.430648 0 - 0.0299513 2026-02-23T15:03:53.161 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:03:53.185 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:03:52 trial068 ceph-mon[38805]: pgmap v3945: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:03:53.278 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:03:53.382 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:03:53.380+0000 7f9969f97640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:03:53.382 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:03:53.381+0000 7f9969f97640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:03:53.492 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:03:53.492 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":285,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:01:34.884444+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:00:52.362313+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:58:43.617395+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:59:25.140122+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:03:53.605 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6627 8 365281 365273 0.430584 0 - 0.0299513 2026-02-23T15:03:54.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:03:53 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3671600676' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:03:54.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:03:53 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:03:53] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:03:54.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:03:53 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:03:53.972Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:03:54.605 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6628 8 365281 365273 0.430519 0 - 0.0299513 2026-02-23T15:03:55.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:03:54 trial068 ceph-mon[38805]: pgmap v3946: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:03:55.605 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6629 8 365281 365273 0.430454 0 - 0.0299513 2026-02-23T15:03:56.531 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:03:56.605 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6630 8 365281 365273 0.430389 0 - 0.0299513 2026-02-23T15:03:56.649 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:03:56.750 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:03:56.748+0000 7f7b749e5640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:03:56.750 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:03:56.748+0000 7f7b749e5640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:03:56.858 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:03:56.858 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":285,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:01:34.884444+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:00:52.362313+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:58:43.617395+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:59:25.140122+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:03:57.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:03:56 trial068 ceph-mon[38805]: pgmap v3947: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:03:57.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:03:56 trial068 ceph-mon[38805]: 5.5 scrub starts 2026-02-23T15:03:57.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:03:56 trial068 ceph-mon[38805]: 5.5 scrub ok 2026-02-23T15:03:57.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:03:56 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/458625514' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:03:57.605 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6631 8 365281 365273 0.430324 0 - 0.0299513 2026-02-23T15:03:58.605 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6632 8 365281 365273 0.430259 0 - 0.0299513 2026-02-23T15:03:59.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:03:58 trial068 ceph-mon[38805]: pgmap v3948: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:03:59.606 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6633 8 365281 365273 0.430194 0 - 0.0299513 2026-02-23T15:03:59.896 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:04:00.016 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:04:00.114 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:04:00.112+0000 7fcd0f667640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:04:00.114 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:04:00.112+0000 7fcd0f667640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:04:00.223 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:04:00.223 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":285,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:01:34.884444+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:00:52.362313+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:58:43.617395+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:59:25.140122+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:04:00.606 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6634 8 365281 365273 0.430129 0 - 0.0299513 2026-02-23T15:04:01.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:04:00 trial068 ceph-mon[38805]: pgmap v3949: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:04:01.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:04:00 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2654885906' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:04:01.606 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6635 8 365281 365273 0.430064 0 - 0.0299513 2026-02-23T15:04:02.606 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6636 8 365281 365273 0.43 0 - 0.0299513 2026-02-23T15:04:03.261 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:04:03.287 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:04:02 trial068 ceph-mon[38805]: pgmap v3950: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:04:03.379 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:04:03.483 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:04:03.482+0000 7f653537e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:04:03.484 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:04:03.482+0000 7f653537e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:04:03.594 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:04:03.594 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":285,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:01:34.884444+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:00:52.362313+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:58:43.617395+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:59:25.140122+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:04:03.606 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6637 8 365281 365273 0.429935 0 - 0.0299513 2026-02-23T15:04:04.265 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:04:03 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:04:03.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:04:04.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:04:03 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3468385172' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:04:04.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:04:03 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:04:03] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:04:04.606 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6638 8 365281 365273 0.42987 0 - 0.0299513 2026-02-23T15:04:05.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:04:04 trial068 ceph-mon[38805]: pgmap v3951: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:04:05.606 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6639 8 365281 365273 0.429805 0 - 0.0299513 2026-02-23T15:04:06.606 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T15:04:06.606318+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T15:04:06.607 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T15:04:06.607 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6640 8 365281 365273 0.42974 0 - 0.0299513 2026-02-23T15:04:06.631 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:04:06.748 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:04:06.853 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:04:06.851+0000 7f2673577640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:04:06.853 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:04:06.852+0000 7f2673577640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:04:06.963 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:04:06.964 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":285,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:01:34.884444+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:00:52.362313+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:58:43.617395+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:59:25.140122+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:04:07.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:04:06 trial068 ceph-mon[38805]: pgmap v3952: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:04:07.606 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6641 8 365281 365273 0.429676 0 - 0.0299513 2026-02-23T15:04:08.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:04:07 trial068 ceph-mon[38805]: 5.e scrub starts 2026-02-23T15:04:08.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:04:07 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2626351863' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:04:08.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:04:07 trial068 ceph-mon[38805]: 5.e scrub ok 2026-02-23T15:04:08.606 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6642 8 365281 365273 0.429611 0 - 0.0299513 2026-02-23T15:04:09.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:04:08 trial068 ceph-mon[38805]: pgmap v3953: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:04:09.607 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6643 8 365281 365273 0.429546 0 - 0.0299513 2026-02-23T15:04:10.002 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:04:10.117 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:04:10.216 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:04:10.214+0000 7f2d15bc4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:04:10.217 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:04:10.215+0000 7f2d15bc4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:04:10.326 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:04:10.326 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":285,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:01:34.884444+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:00:52.362313+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:58:43.617395+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:59:25.140122+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:04:10.607 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6644 8 365281 365273 0.429482 0 - 0.0299513 2026-02-23T15:04:11.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:04:10 trial068 ceph-mon[38805]: pgmap v3954: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:04:11.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:04:10 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1675154392' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:04:11.607 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6645 8 365281 365273 0.429417 0 - 0.0299513 2026-02-23T15:04:12.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:04:11 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2026-02-23T15:04:12.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:04:11 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-23T15:04:12.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:04:11 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2026-02-23T15:04:12.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:04:11 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T15:04:12.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:04:11 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2026-02-23T15:04:12.607 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6646 8 365281 365273 0.429353 0 - 0.0299513 2026-02-23T15:04:13.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:04:12 trial068 ceph-mon[38805]: pgmap v3955: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:04:13.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:04:12 trial068 ceph-mon[38805]: pgmap v3956: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:04:13.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:04:12 trial068 ceph-mon[38805]: pgmap v3957: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:04:13.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:04:12 trial068 ceph-mon[38805]: osdmap e286: 4 total, 3 up, 4 in 2026-02-23T15:04:13.362 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:04:13.481 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:04:13.580 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:04:13.578+0000 7f2c31667640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:04:13.580 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:04:13.579+0000 7f2c31667640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:04:13.607 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6647 8 365281 365273 0.429288 0 - 0.0299513 2026-02-23T15:04:13.692 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:04:13.692 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":286,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:04:11.944692+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:00:52.362313+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:02:53.966248+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:59:25.140122+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:04:14.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:04:13 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/152203750' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:04:14.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:04:13 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:04:13] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:04:14.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:04:13 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:04:13.972Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:04:14.607 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6648 8 365281 365273 0.429223 0 - 0.0299513 2026-02-23T15:04:15.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:04:14 trial068 ceph-mon[38805]: pgmap v3959: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:04:15.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:04:14 trial068 ceph-mon[38805]: osdmap e287: 4 total, 3 up, 4 in 2026-02-23T15:04:15.607 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6649 8 365281 365273 0.429159 0 - 0.0299513 2026-02-23T15:04:16.244 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:04:15 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T15:04:16.245 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:04:15 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T15:04:16.607 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6650 8 365281 365273 0.429094 0 - 0.0299513 2026-02-23T15:04:16.730 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:04:16.847 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:04:16.953 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:04:16.951+0000 7efc606c7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:04:16.953 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:04:16.951+0000 7efc606c7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:04:17.063 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:04:17.063 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":287,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:04:14.906283+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:00:52.362313+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:02:53.966248+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:59:25.140122+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:04:17.063 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:04:16 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T15:04:17.063 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:04:16 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T15:04:17.064 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:04:16 trial068 ceph-mon[38805]: pgmap v3961: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:04:17.064 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:04:16 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T15:04:17.064 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:04:16 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T15:04:17.607 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6651 8 365281 365273 0.42903 0 - 0.0299513 2026-02-23T15:04:18.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:04:17 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3292818558' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:04:18.607 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6652 8 365281 365273 0.428965 0 - 0.0299513 2026-02-23T15:04:19.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:04:18 trial068 ceph-mon[38805]: pgmap v3962: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:04:19.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:04:18 trial068 ceph-mon[38805]: 5.7 scrub starts 2026-02-23T15:04:19.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:04:18 trial068 ceph-mon[38805]: 5.7 scrub ok 2026-02-23T15:04:19.607 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6653 8 365281 365273 0.428901 0 - 0.0299513 2026-02-23T15:04:20.105 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:04:20.223 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:04:20.332 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:04:20.330+0000 7fa798e83640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:04:20.332 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:04:20.330+0000 7fa798e83640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:04:20.440 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:04:20.441 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":287,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:04:14.906283+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:00:52.362313+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:02:53.966248+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:59:25.140122+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:04:20.607 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6654 8 365281 365273 0.428836 0 - 0.0299513 2026-02-23T15:04:21.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:04:20 trial068 ceph-mon[38805]: pgmap v3963: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:04:21.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:04:20 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/573800338' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:04:21.608 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6655 8 365281 365273 0.428772 0 - 0.0299513 2026-02-23T15:04:22.608 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6656 8 365281 365273 0.428707 0 - 0.0299513 2026-02-23T15:04:23.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:04:22 trial068 ceph-mon[38805]: pgmap v3964: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:04:23.480 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:04:23.598 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:04:23.608 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6657 8 365281 365273 0.428643 0 - 0.0299513 2026-02-23T15:04:23.697 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:04:23.695+0000 7f69c897d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:04:23.697 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:04:23.695+0000 7f69c897d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:04:23.806 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:04:23.807 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":287,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:04:14.906283+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:00:52.362313+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:02:53.966248+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:59:25.140122+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:04:24.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:04:23 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2199456889' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:04:24.094 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:04:23 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:04:23] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:04:24.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:04:23 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:04:23.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:04:24.608 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6658 8 365281 365273 0.428579 0 - 0.0299513 2026-02-23T15:04:25.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:04:24 trial068 ceph-mon[38805]: pgmap v3965: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:04:25.608 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6659 8 365281 365273 0.428514 0 - 0.0299513 2026-02-23T15:04:26.608 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T15:04:26.607802+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T15:04:26.608 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T15:04:26.608 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6660 8 365281 365273 0.42845 0 - 0.0299513 2026-02-23T15:04:26.844 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:04:26.962 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:04:27.061 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:04:27.059+0000 7f9e4d64e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:04:27.061 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:04:27.059+0000 7f9e4d64e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:04:27.176 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:04:27.176 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":287,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:04:14.906283+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:00:52.362313+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:02:53.966248+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:59:25.140122+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:04:27.176 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:04:26 trial068 ceph-mon[38805]: pgmap v3966: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:04:27.608 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6661 8 365281 365273 0.428386 0 - 0.0299513 2026-02-23T15:04:28.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:04:27 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/678679707' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:04:28.608 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6662 8 365281 365273 0.428321 0 - 0.0299513 2026-02-23T15:04:29.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:04:28 trial068 ceph-mon[38805]: pgmap v3967: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:04:29.608 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6663 8 365281 365273 0.428257 0 - 0.0299513 2026-02-23T15:04:30.214 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:04:30.332 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:04:30.432 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:04:30.431+0000 7fd69fa59640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:04:30.432 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:04:30.431+0000 7fd69fa59640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:04:30.544 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:04:30.545 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":287,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:04:14.906283+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:00:52.362313+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:02:53.966248+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:59:25.140122+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:04:30.608 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6664 8 365281 365273 0.428193 0 - 0.0299513 2026-02-23T15:04:31.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:04:30 trial068 ceph-mon[38805]: pgmap v3968: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:04:31.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:04:30 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3796516775' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:04:31.608 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6665 8 365281 365273 0.428129 0 - 0.0299513 2026-02-23T15:04:32.609 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6666 8 365281 365273 0.428064 0 - 0.0299513 2026-02-23T15:04:33.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:04:32 trial068 ceph-mon[38805]: pgmap v3969: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:04:33.586 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:04:33.609 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6667 8 365281 365273 0.428 0 - 0.0299513 2026-02-23T15:04:33.705 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:04:33.808 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:04:33.806+0000 7f71fb702640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:04:33.808 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:04:33.806+0000 7f71fb702640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:04:33.919 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:04:33.919 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":287,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:04:14.906283+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:00:52.362313+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:02:53.966248+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:59:25.140122+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:04:34.264 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:04:33 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:04:33.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:04:34.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:04:33 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/498906997' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:04:34.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:04:33 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:04:33] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:04:34.609 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6668 8 365281 365273 0.427936 0 - 0.0299513 2026-02-23T15:04:35.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:04:34 trial068 ceph-mon[38805]: pgmap v3970: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:04:35.609 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6669 8 365281 365273 0.427872 0 - 0.0299513 2026-02-23T15:04:36.609 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6670 8 365281 365273 0.427808 0 - 0.0299513 2026-02-23T15:04:36.956 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:04:37.075 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:04:37.176 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:04:37.174+0000 7f168fe87640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:04:37.176 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:04:37.174+0000 7f168fe87640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:04:37.284 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:04:37.284 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":287,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:04:14.906283+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:00:52.362313+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:02:53.966248+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:59:25.140122+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:04:37.285 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:04:36 trial068 ceph-mon[38805]: pgmap v3971: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:04:37.609 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6671 8 365281 365273 0.427743 0 - 0.0299513 2026-02-23T15:04:38.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:04:37 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3743239616' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:04:38.609 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6672 8 365281 365273 0.427679 0 - 0.0299513 2026-02-23T15:04:39.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:04:38 trial068 ceph-mon[38805]: pgmap v3972: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:04:39.609 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6673 8 365281 365273 0.427615 0 - 0.0299513 2026-02-23T15:04:40.323 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:04:40.443 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:04:40.543 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:04:40.541+0000 7fb2fb577640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:04:40.543 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:04:40.541+0000 7fb2fb577640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:04:40.609 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6674 8 365281 365273 0.427551 0 - 0.0299513 2026-02-23T15:04:40.653 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:04:40.653 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":287,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:04:14.906283+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:00:52.362313+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:02:53.966248+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:59:25.140122+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:04:41.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:04:40 trial068 ceph-mon[38805]: pgmap v3973: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:04:41.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:04:40 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/4225648385' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:04:41.609 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6675 8 365281 365273 0.427487 0 - 0.0299513 2026-02-23T15:04:42.609 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6676 8 365281 365273 0.427423 0 - 0.0299513 2026-02-23T15:04:43.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:04:42 trial068 ceph-mon[38805]: pgmap v3974: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:04:43.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:04:42 trial068 ceph-mon[38805]: 5.a scrub starts 2026-02-23T15:04:43.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:04:42 trial068 ceph-mon[38805]: 5.a scrub ok 2026-02-23T15:04:43.609 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6677 8 365281 365273 0.427359 0 - 0.0299513 2026-02-23T15:04:43.692 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:04:43.811 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:04:43.914 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:04:43.912+0000 7f67ae938640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:04:43.914 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:04:43.912+0000 7f67ae938640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:04:44.025 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:04:44.025 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":287,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:04:14.906283+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:00:52.362313+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:02:53.966248+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:59:25.140122+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:04:44.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:04:43 trial068 ceph-mon[38805]: 5.8 scrub starts 2026-02-23T15:04:44.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:04:43 trial068 ceph-mon[38805]: 5.8 scrub ok 2026-02-23T15:04:44.345 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:04:43 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:04:43] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:04:44.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:04:43 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:04:43.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:04:44.610 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6678 8 365281 365273 0.427295 0 - 0.0299513 2026-02-23T15:04:45.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:04:44 trial068 ceph-mon[38805]: pgmap v3975: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:04:45.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:04:44 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2822346405' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:04:45.609 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6679 8 365281 365273 0.427231 0 - 0.0299513 2026-02-23T15:04:46.610 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T15:04:46.609425+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T15:04:46.610 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T15:04:46.610 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6680 8 365281 365273 0.427167 0 - 0.0299513 2026-02-23T15:04:47.082 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:04:47.198 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:04:47.249 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:04:46 trial068 ceph-mon[38805]: pgmap v3976: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:04:47.308 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:04:47.306+0000 7f47aaae8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:04:47.309 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:04:47.307+0000 7f47aaae8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:04:47.419 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:04:47.420 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":287,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:04:14.906283+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:00:52.362313+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:02:53.966248+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:59:25.140122+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:04:47.610 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6681 8 365281 365273 0.427103 0 - 0.0299513 2026-02-23T15:04:48.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:04:47 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1787331608' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:04:48.610 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6682 8 365281 365273 0.427039 0 - 0.0299513 2026-02-23T15:04:49.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:04:48 trial068 ceph-mon[38805]: pgmap v3977: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:04:49.610 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6683 8 365281 365273 0.426975 0 - 0.0299513 2026-02-23T15:04:50.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:04:49 trial068 ceph-mon[38805]: pgmap v3978: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:04:50.457 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:04:50.573 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:04:50.610 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6684 8 365281 365273 0.426912 0 - 0.0299513 2026-02-23T15:04:50.679 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:04:50.677+0000 7f04b2596640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:04:50.679 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:04:50.677+0000 7f04b2596640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:04:50.786 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:04:50.787 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":287,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:04:14.906283+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:00:52.362313+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:02:53.966248+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:59:25.140122+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:04:51.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:04:50 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2218006597' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:04:51.610 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6685 8 365281 365273 0.426848 0 - 0.0299513 2026-02-23T15:04:52.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:04:51 trial068 ceph-mon[38805]: pgmap v3979: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:04:52.610 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6686 8 365281 365273 0.426784 0 - 0.0299513 2026-02-23T15:04:53.610 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6687 8 365281 365273 0.42672 0 - 0.0299513 2026-02-23T15:04:53.824 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:04:53.944 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:04:54.048 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:04:54.046+0000 7fcf484d9640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:04:54.049 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:04:54.047+0000 7fcf484d9640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:04:54.156 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:04:54.156 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":287,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:04:14.906283+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:00:52.362313+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:02:53.966248+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:59:25.140122+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:04:54.157 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:04:53 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:04:53] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:04:54.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:04:53 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:04:53.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:04:54.610 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6688 8 365281 365273 0.426656 0 - 0.0299513 2026-02-23T15:04:55.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:04:54 trial068 ceph-mon[38805]: pgmap v3980: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:04:55.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:04:54 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3629250575' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:04:55.610 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6689 8 365281 365273 0.426592 0 - 0.0299513 2026-02-23T15:04:56.610 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6690 8 365281 365273 0.426529 0 - 0.0299513 2026-02-23T15:04:57.193 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:04:57.217 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:04:56 trial068 ceph-mon[38805]: pgmap v3981: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:04:57.309 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:04:57.407 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:04:57.405+0000 7fa9d14db640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:04:57.407 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:04:57.405+0000 7fa9d14db640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:04:57.516 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:04:57.517 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":287,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:04:14.906283+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:00:52.362313+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:02:53.966248+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:59:25.140122+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:04:57.610 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6691 8 365281 365273 0.426465 0 - 0.0299513 2026-02-23T15:04:58.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:04:57 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2123631403' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:04:58.610 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6692 8 365281 365273 0.426401 0 - 0.0299513 2026-02-23T15:04:59.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:04:58 trial068 ceph-mon[38805]: pgmap v3982: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:04:59.610 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6693 8 365281 365273 0.426337 0 - 0.0299513 2026-02-23T15:05:00.554 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:05:00.611 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6694 8 365281 365273 0.426274 0 - 0.0299513 2026-02-23T15:05:00.679 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:05:00.775 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:05:00.773+0000 7fbcb60d7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:05:00.775 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:05:00.773+0000 7fbcb60d7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:05:00.883 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:05:00.883 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":287,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:04:14.906283+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:00:52.362313+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:02:53.966248+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:59:25.140122+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:05:01.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:05:00 trial068 ceph-mon[38805]: pgmap v3983: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:05:01.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:05:00 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3808576853' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:05:01.611 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6695 8 365281 365273 0.42621 0 - 0.0299513 2026-02-23T15:05:02.611 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6696 8 365281 365273 0.426146 0 - 0.0299513 2026-02-23T15:05:03.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:05:02 trial068 ceph-mon[38805]: pgmap v3984: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:05:03.611 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6697 8 365281 365273 0.426083 0 - 0.0299513 2026-02-23T15:05:03.921 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:05:04.038 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:05:04.141 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:05:04.139+0000 7f1b42353640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:05:04.141 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:05:04.140+0000 7f1b42353640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:05:04.252 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:05:04.252 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":287,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:04:14.906283+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:00:52.362313+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:02:53.966248+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:59:25.140122+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:05:04.253 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:05:03 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:05:03] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:05:04.265 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:05:03 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:05:03.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:05:04.611 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6698 8 365281 365273 0.426019 0 - 0.0299513 2026-02-23T15:05:05.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:05:04 trial068 ceph-mon[38805]: pgmap v3985: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:05:05.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:05:04 trial068 ceph-mon[38805]: 5.5 scrub starts 2026-02-23T15:05:05.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:05:04 trial068 ceph-mon[38805]: 5.5 scrub ok 2026-02-23T15:05:05.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:05:04 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3727754074' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:05:05.611 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6699 8 365281 365273 0.425956 0 - 0.0299513 2026-02-23T15:05:06.611 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T15:05:06.610871+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T15:05:06.611 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T15:05:06.611 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6700 8 365281 365273 0.425892 0 - 0.0299513 2026-02-23T15:05:07.290 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:05:07.315 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:05:06 trial068 ceph-mon[38805]: pgmap v3986: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:05:07.408 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:05:07.509 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:05:07.507+0000 7f4ff4c8c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:05:07.509 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:05:07.508+0000 7f4ff4c8c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:05:07.611 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6701 8 365281 365273 0.425829 0 - 0.0299513 2026-02-23T15:05:07.619 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:05:07.619 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":287,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:04:14.906283+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:00:52.362313+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:02:53.966248+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:59:25.140122+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:05:08.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:05:07 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3618364896' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:05:08.611 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6702 8 365281 365273 0.425765 0 - 0.0299513 2026-02-23T15:05:09.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:05:08 trial068 ceph-mon[38805]: pgmap v3987: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:05:09.611 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6703 8 365281 365273 0.425701 0 - 0.0299513 2026-02-23T15:05:10.611 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6704 8 365281 365273 0.425638 0 - 0.0299513 2026-02-23T15:05:10.660 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:05:10.775 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:05:10.873 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:05:10.870+0000 7f78aa77d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:05:10.873 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:05:10.871+0000 7f78aa77d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:05:10.980 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:05:10.981 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":287,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:04:14.906283+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:00:52.362313+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:02:53.966248+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:59:25.140122+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:05:10.981 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:05:10 trial068 ceph-mon[38805]: pgmap v3988: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:05:11.612 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6705 8 365281 365273 0.425574 0 - 0.0299513 2026-02-23T15:05:12.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:05:11 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3368613685' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:05:12.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:05:11 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2026-02-23T15:05:12.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:05:11 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-23T15:05:12.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:05:11 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2026-02-23T15:05:12.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:05:11 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T15:05:12.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:05:11 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2026-02-23T15:05:12.611 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6706 8 365281 365273 0.425511 0 - 0.0299513 2026-02-23T15:05:13.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:05:12 trial068 ceph-mon[38805]: pgmap v3989: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:05:13.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:05:12 trial068 ceph-mon[38805]: pgmap v3990: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:05:13.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:05:12 trial068 ceph-mon[38805]: pgmap v3991: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:05:13.611 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6707 8 365281 365273 0.425448 0 - 0.0299513 2026-02-23T15:05:14.019 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:05:14.138 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:05:14.183 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:05:13 trial068 ceph-mon[38805]: 5.e scrub starts 2026-02-23T15:05:14.184 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:05:13 trial068 ceph-mon[38805]: 5.e scrub ok 2026-02-23T15:05:14.184 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:05:13 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:05:13] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:05:14.241 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:05:14.239+0000 7fe5d0323640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:05:14.241 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:05:14.239+0000 7fe5d0323640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:05:14.350 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:05:14.350 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":287,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:04:14.906283+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:00:52.362313+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:02:53.966248+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:59:25.140122+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:05:14.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:05:13 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:05:13.974Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:05:14.612 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6708 8 365281 365273 0.425384 0 - 0.0299513 2026-02-23T15:05:15.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:05:14 trial068 ceph-mon[38805]: pgmap v3992: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:05:15.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:05:14 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1882023969' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:05:15.612 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6709 8 365281 365273 0.425321 0 - 0.0299513 2026-02-23T15:05:16.612 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6710 8 365281 365273 0.425257 0 - 0.0299513 2026-02-23T15:05:17.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:05:16 trial068 ceph-mon[38805]: pgmap v3993: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:05:17.387 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:05:17.506 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:05:17.603 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:05:17.601+0000 7fe457188640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:05:17.603 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:05:17.602+0000 7fe457188640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:05:17.612 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6711 8 365281 365273 0.425194 0 - 0.0299513 2026-02-23T15:05:17.713 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:05:17.714 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":287,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:04:14.906283+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:00:52.362313+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:02:53.966248+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:59:25.140122+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:05:18.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:05:17 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1600693161' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:05:18.612 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6712 8 365281 365273 0.425131 0 - 0.0299513 2026-02-23T15:05:19.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:05:18 trial068 ceph-mon[38805]: pgmap v3994: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:05:19.612 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6713 8 365281 365273 0.425067 0 - 0.0299513 2026-02-23T15:05:20.612 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6714 8 365281 365273 0.425004 0 - 0.0299513 2026-02-23T15:05:20.752 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:05:20.871 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:05:20.973 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:05:20.971+0000 7fa55ffcf640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:05:20.973 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:05:20.971+0000 7fa55ffcf640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:05:21.081 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:05:21.082 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":287,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:04:14.906283+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:00:52.362313+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:02:53.966248+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:59:25.140122+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:05:21.082 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:05:20 trial068 ceph-mon[38805]: pgmap v3995: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:05:21.612 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6715 8 365281 365273 0.424941 0 - 0.0299513 2026-02-23T15:05:22.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:05:21 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1829886898' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:05:22.612 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6716 8 365281 365273 0.424877 0 - 0.0299513 2026-02-23T15:05:23.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:05:22 trial068 ceph-mon[38805]: pgmap v3996: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:05:23.612 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6717 8 365281 365273 0.424814 0 - 0.0299513 2026-02-23T15:05:24.119 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:05:24.235 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:05:24.257 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:05:23 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:05:23] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:05:24.333 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:05:24.331+0000 7feb62ae9640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:05:24.334 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:05:24.332+0000 7feb62ae9640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:05:24.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:05:23 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:05:23.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:05:24.443 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:05:24.443 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":287,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:04:14.906283+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:00:52.362313+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:02:53.966248+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:59:25.140122+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:05:24.612 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6718 8 365281 365273 0.424751 0 - 0.0299513 2026-02-23T15:05:25.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:05:24 trial068 ceph-mon[38805]: pgmap v3997: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:05:25.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:05:24 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1073006139' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:05:25.612 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6719 8 365281 365273 0.424688 0 - 0.0299513 2026-02-23T15:05:26.612 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T15:05:26.612298+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T15:05:26.613 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T15:05:26.613 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6720 8 365281 365273 0.424625 0 - 0.0299513 2026-02-23T15:05:27.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:05:26 trial068 ceph-mon[38805]: pgmap v3998: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:05:27.482 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:05:27.599 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:05:27.612 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6721 8 365281 365273 0.424561 0 - 0.0299513 2026-02-23T15:05:27.709 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:05:27.707+0000 7f735258e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:05:27.709 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:05:27.707+0000 7f735258e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:05:27.818 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:05:27.818 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":287,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:04:14.906283+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:00:52.362313+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:02:53.966248+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T14:59:25.140122+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:05:28.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:05:27 trial068 ceph-mon[38805]: 5.7 scrub starts 2026-02-23T15:05:28.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:05:27 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2141796818' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:05:28.613 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6722 8 365281 365273 0.424498 0 - 0.0299513 2026-02-23T15:05:29.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:05:28 trial068 ceph-mon[38805]: 5.7 scrub ok 2026-02-23T15:05:29.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:05:28 trial068 ceph-mon[38805]: pgmap v3999: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:05:29.613 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6723 8 365281 365273 0.424435 0 - 0.0299513 2026-02-23T15:05:30.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:05:29 trial068 ceph-mon[38805]: osdmap e288: 4 total, 3 up, 4 in 2026-02-23T15:05:30.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:05:29 trial068 ceph-mon[38805]: osdmap e289: 4 total, 3 up, 4 in 2026-02-23T15:05:30.613 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6724 8 365281 365273 0.424372 0 - 0.0299513 2026-02-23T15:05:30.855 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:05:30.971 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:05:31.078 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:05:31.076+0000 7f81d495a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:05:31.078 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:05:31.077+0000 7f81d495a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:05:31.189 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:05:31.190 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":289,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:05:29.915532+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:00:52.362313+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:02:53.966248+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:04:16.232725+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:05:31.190 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:05:30 trial068 ceph-mon[38805]: pgmap v4001: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:05:31.613 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6725 8 365281 365273 0.424309 0 - 0.0299513 2026-02-23T15:05:32.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:05:31 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1912900511' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:05:32.613 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6726 8 365281 365273 0.424246 0 - 0.0299513 2026-02-23T15:05:33.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:05:32 trial068 ceph-mon[38805]: pgmap v4003: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:05:33.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:05:32 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T15:05:33.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:05:32 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T15:05:33.613 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6727 8 365281 365273 0.424183 0 - 0.0299513 2026-02-23T15:05:34.227 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:05:34.252 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:05:33 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:05:33] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:05:34.265 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:05:33 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:05:33.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:05:34.344 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:05:34.445 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:05:34.443+0000 7ffa5a814640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:05:34.445 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:05:34.443+0000 7ffa5a814640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:05:34.553 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:05:34.553 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":289,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:05:29.915532+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:00:52.362313+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:02:53.966248+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:04:16.232725+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:05:34.613 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6728 8 365281 365273 0.42412 0 - 0.0299513 2026-02-23T15:05:35.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:05:34 trial068 ceph-mon[38805]: pgmap v4004: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:05:35.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:05:34 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2688707591' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:05:35.613 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6729 8 365281 365273 0.424057 0 - 0.0299513 2026-02-23T15:05:36.613 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6730 8 365281 365273 0.423994 0 - 0.0299513 2026-02-23T15:05:37.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:05:36 trial068 ceph-mon[38805]: pgmap v4005: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:05:37.591 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:05:37.613 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6731 8 365281 365273 0.423931 0 - 0.0299513 2026-02-23T15:05:37.709 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:05:37.816 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:05:37.814+0000 7f47fffff640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:05:37.816 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:05:37.814+0000 7f47fffff640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:05:37.924 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:05:37.924 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":289,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:05:29.915532+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:00:52.362313+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:02:53.966248+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:04:16.232725+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:05:38.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:05:37 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3313308275' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:05:38.613 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6732 8 365281 365273 0.423868 0 - 0.0299513 2026-02-23T15:05:39.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:05:38 trial068 ceph-mon[38805]: pgmap v4006: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:05:39.613 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6733 8 365281 365273 0.423805 0 - 0.0299513 2026-02-23T15:05:40.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:05:39 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T15:05:40.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:05:39 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T15:05:40.613 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6734 8 365281 365273 0.423742 0 - 0.0299513 2026-02-23T15:05:40.961 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:05:41.078 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:05:41.195 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:05:41.193+0000 7fbb60efc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:05:41.195 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:05:41.193+0000 7fbb60efc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:05:41.306 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:05:41.307 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":289,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:05:29.915532+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:00:52.362313+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:02:53.966248+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:04:16.232725+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:05:41.307 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:05:40 trial068 ceph-mon[38805]: pgmap v4007: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:05:41.613 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6735 8 365281 365273 0.423679 0 - 0.0299513 2026-02-23T15:05:42.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:05:41 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1472830227' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:05:42.614 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6736 8 365281 365273 0.423616 0 - 0.0299513 2026-02-23T15:05:43.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:05:42 trial068 ceph-mon[38805]: pgmap v4008: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:05:43.614 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6737 8 365281 365273 0.423553 0 - 0.0299513 2026-02-23T15:05:44.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:05:43 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:05:43] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:05:44.345 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:05:44.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:05:43 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:05:43.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:05:44.463 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:05:44.559 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:05:44.558+0000 7fcdd8098640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:05:44.560 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:05:44.558+0000 7fcdd8098640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:05:44.613 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6738 8 365281 365273 0.42349 0 - 0.0299513 2026-02-23T15:05:44.667 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:05:44.668 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":289,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:05:29.915532+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:00:52.362313+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:02:53.966248+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:04:16.232725+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:05:45.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:05:44 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T15:05:45.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:05:44 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T15:05:45.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:05:44 trial068 ceph-mon[38805]: pgmap v4009: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:05:45.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:05:44 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/405104810' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:05:45.614 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6739 8 365281 365273 0.423427 0 - 0.0299513 2026-02-23T15:05:46.614 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T15:05:46.613656+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T15:05:46.614 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T15:05:46.614 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6740 8 365281 365273 0.423365 0 - 0.0299513 2026-02-23T15:05:47.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:05:46 trial068 ceph-mon[38805]: pgmap v4010: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:05:47.614 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6741 8 365281 365273 0.423302 0 - 0.0299513 2026-02-23T15:05:47.705 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:05:47.822 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:05:47.919 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:05:47.918+0000 7f868a95a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:05:47.920 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:05:47.918+0000 7f868a95a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:05:48.029 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:05:48.029 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":289,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:05:29.915532+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:00:52.362313+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:02:53.966248+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:04:16.232725+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:05:48.614 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6742 8 365281 365273 0.423239 0 - 0.0299513 2026-02-23T15:05:49.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:05:48 trial068 ceph-mon[38805]: pgmap v4011: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:05:49.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:05:48 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3984467877' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:05:49.614 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6743 8 365281 365273 0.423176 0 - 0.0299513 2026-02-23T15:05:50.614 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6744 8 365281 365273 0.423113 0 - 0.0299513 2026-02-23T15:05:51.065 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:05:51.182 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:05:51.225 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:05:50 trial068 ceph-mon[38805]: pgmap v4012: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:05:51.284 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:05:51.282+0000 7f2bdef74640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:05:51.284 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:05:51.283+0000 7f2bdef74640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:05:51.394 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:05:51.394 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":289,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:05:29.915532+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:00:52.362313+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:02:53.966248+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:04:16.232725+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:05:51.614 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6745 8 365281 365273 0.423051 0 - 0.0299513 2026-02-23T15:05:52.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:05:51 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3514424718' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:05:52.614 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6746 8 365281 365273 0.422988 0 - 0.0299513 2026-02-23T15:05:53.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:05:52 trial068 ceph-mon[38805]: pgmap v4013: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:05:53.614 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6747 8 365281 365273 0.422925 0 - 0.0299513 2026-02-23T15:05:54.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:05:53 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:05:53] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:05:54.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:05:53 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:05:53.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:05:54.431 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:05:54.547 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:05:54.614 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6748 8 365281 365273 0.422863 0 - 0.0299513 2026-02-23T15:05:54.645 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:05:54.643+0000 7f6d4cd4f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:05:54.645 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:05:54.644+0000 7f6d4cd4f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:05:54.752 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:05:54.752 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":289,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:05:29.915532+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:00:52.362313+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:02:53.966248+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:04:16.232725+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:05:55.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:05:54 trial068 ceph-mon[38805]: pgmap v4014: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:05:55.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:05:54 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2110538648' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:05:55.614 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6749 8 365281 365273 0.4228 0 - 0.0299513 2026-02-23T15:05:56.615 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6750 8 365281 365273 0.422737 0 - 0.0299513 2026-02-23T15:05:57.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:05:56 trial068 ceph-mon[38805]: pgmap v4015: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:05:57.615 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6751 8 365281 365273 0.422675 0 - 0.0299513 2026-02-23T15:05:57.792 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:05:57.910 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:05:58.011 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:05:58.009+0000 7f100ecc2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:05:58.011 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:05:58.009+0000 7f100ecc2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:05:58.121 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:05:58.121 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":289,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:05:29.915532+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:00:52.362313+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:02:53.966248+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:04:16.232725+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:05:58.615 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6752 8 365281 365273 0.422612 0 - 0.0299513 2026-02-23T15:05:59.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:05:58 trial068 ceph-mon[38805]: pgmap v4016: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:05:59.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:05:58 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3337035635' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:05:59.615 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6753 8 365281 365273 0.42255 0 - 0.0299513 2026-02-23T15:06:00.615 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6754 8 365281 365273 0.422487 0 - 0.0299513 2026-02-23T15:06:01.159 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:06:01.284 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:06:01.307 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:06:00 trial068 ceph-mon[38805]: pgmap v4017: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:06:01.384 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:06:01.383+0000 7fc77699b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:06:01.385 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:06:01.383+0000 7fc77699b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:06:01.494 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:06:01.494 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":289,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:05:29.915532+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:00:52.362313+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:02:53.966248+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:04:16.232725+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:06:01.615 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6755 8 365281 365273 0.422424 0 - 0.0299513 2026-02-23T15:06:02.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:06:01 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3910791480' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:06:02.615 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6756 8 365281 365273 0.422362 0 - 0.0299513 2026-02-23T15:06:03.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:06:02 trial068 ceph-mon[38805]: pgmap v4018: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:06:03.615 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6757 8 365281 365273 0.422299 0 - 0.0299513 2026-02-23T15:06:04.265 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:06:03 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:06:03.974Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:06:04.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:06:03 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:06:03] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:06:04.531 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:06:04.615 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6758 8 365281 365273 0.422237 0 - 0.0299513 2026-02-23T15:06:04.650 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:06:04.749 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:06:04.747+0000 7f6570b4c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:06:04.749 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:06:04.747+0000 7f6570b4c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:06:04.866 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:06:04.866 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":289,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:05:29.915532+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:00:52.362313+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:02:53.966248+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:04:16.232725+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:06:05.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:06:04 trial068 ceph-mon[38805]: pgmap v4019: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:06:05.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:06:04 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3162297288' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:06:05.615 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6759 8 365281 365273 0.422174 0 - 0.0299513 2026-02-23T15:06:06.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:06:05 trial068 ceph-mon[38805]: 5.8 scrub starts 2026-02-23T15:06:06.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:06:05 trial068 ceph-mon[38805]: 5.8 scrub ok 2026-02-23T15:06:06.615 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T15:06:06.615153+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T15:06:06.615 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T15:06:06.616 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6760 8 365281 365273 0.422112 0 - 0.0299513 2026-02-23T15:06:07.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:06:06 trial068 ceph-mon[38805]: pgmap v4020: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:06:07.615 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6761 8 365281 365273 0.42205 0 - 0.0299513 2026-02-23T15:06:07.921 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:06:08.044 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:06:08.144 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:06:08.142+0000 7f8c52f84640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:06:08.145 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:06:08.143+0000 7f8c52f84640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:06:08.253 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:06:08.253 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":289,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:05:29.915532+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:00:52.362313+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:02:53.966248+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:04:16.232725+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:06:08.615 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6762 8 365281 365273 0.421987 0 - 0.0299513 2026-02-23T15:06:09.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:06:08 trial068 ceph-mon[38805]: pgmap v4021: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:06:09.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:06:08 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/541191429' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:06:09.615 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6763 8 365281 365273 0.421925 0 - 0.0299513 2026-02-23T15:06:10.616 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6764 8 365281 365273 0.421862 0 - 0.0299513 2026-02-23T15:06:11.292 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:06:11.316 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:06:10 trial068 ceph-mon[38805]: 5.a scrub starts 2026-02-23T15:06:11.316 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:06:10 trial068 ceph-mon[38805]: 5.a scrub ok 2026-02-23T15:06:11.317 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:06:10 trial068 ceph-mon[38805]: pgmap v4022: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:06:11.410 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:06:11.508 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:06:11.506+0000 7f27f1eea640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:06:11.508 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:06:11.506+0000 7f27f1eea640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:06:11.615 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6765 8 365281 365273 0.4218 0 - 0.0299513 2026-02-23T15:06:11.617 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:06:11.617 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":289,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:05:29.915532+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:00:52.362313+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:02:53.966248+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:04:16.232725+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:06:12.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:06:11 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/984892030' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:06:12.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:06:11 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2026-02-23T15:06:12.616 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6766 8 365281 365273 0.421738 0 - 0.0299513 2026-02-23T15:06:13.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:06:12 trial068 ceph-mon[38805]: pgmap v4023: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:06:13.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:06:12 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-23T15:06:13.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:06:12 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2026-02-23T15:06:13.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:06:12 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T15:06:13.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:06:12 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2026-02-23T15:06:13.616 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6767 8 365281 365273 0.421675 0 - 0.0299513 2026-02-23T15:06:14.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:06:13 trial068 ceph-mon[38805]: pgmap v4024: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:06:14.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:06:13 trial068 ceph-mon[38805]: pgmap v4025: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:06:14.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:06:13 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:06:13] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:06:14.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:06:13 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:06:13.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:06:14.615 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6768 8 365281 365273 0.421613 0 - 0.0299513 2026-02-23T15:06:14.654 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:06:14.770 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:06:14.875 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:06:14.874+0000 7f29aca88640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:06:14.875 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:06:14.874+0000 7f29aca88640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:06:14.984 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:06:14.984 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":289,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:05:29.915532+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:00:52.362313+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:02:53.966248+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:04:16.232725+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:06:15.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:06:14 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2768312140' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:06:15.616 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6769 8 365281 365273 0.421551 0 - 0.0299513 2026-02-23T15:06:16.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:06:15 trial068 ceph-mon[38805]: pgmap v4026: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:06:16.616 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6770 8 365281 365273 0.421488 0 - 0.0299513 2026-02-23T15:06:17.616 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6771 8 365281 365273 0.421426 0 - 0.0299513 2026-02-23T15:06:18.021 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:06:18.138 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:06:18.236 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:06:18.234+0000 7f6e07781640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:06:18.237 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:06:18.235+0000 7f6e07781640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:06:18.252 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:06:17 trial068 ceph-mon[38805]: pgmap v4027: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:06:18.346 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:06:18.346 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":289,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:05:29.915532+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:00:52.362313+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:02:53.966248+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:04:16.232725+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:06:18.616 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6772 8 365281 365273 0.421364 0 - 0.0299513 2026-02-23T15:06:19.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:06:18 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1660315225' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:06:19.616 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6773 8 365281 365273 0.421302 0 - 0.0299513 2026-02-23T15:06:20.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:06:19 trial068 ceph-mon[38805]: pgmap v4028: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:06:20.616 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6774 8 365281 365273 0.42124 0 - 0.0299513 2026-02-23T15:06:21.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:06:20 trial068 ceph-mon[38805]: pgmap v4029: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:06:21.383 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:06:21.502 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:06:21.604 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:06:21.601+0000 7f06e4678640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:06:21.604 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:06:21.602+0000 7f06e4678640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:06:21.616 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6775 8 365281 365273 0.421177 0 - 0.0299513 2026-02-23T15:06:21.714 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:06:21.714 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":289,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:05:29.915532+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:00:52.362313+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:02:53.966248+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:04:16.232725+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:06:22.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:06:21 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3971548213' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:06:22.616 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6776 8 365281 365273 0.421115 0 - 0.0299513 2026-02-23T15:06:23.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:06:22 trial068 ceph-mon[38805]: pgmap v4030: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:06:23.616 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6777 8 365281 365273 0.421053 0 - 0.0299513 2026-02-23T15:06:24.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:06:23 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:06:23] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:06:24.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:06:23 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:06:23.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:06:24.616 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6778 8 365281 365273 0.420991 0 - 0.0299513 2026-02-23T15:06:24.752 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:06:24.871 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:06:24.977 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:06:24.975+0000 7f1265dcb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:06:24.977 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:06:24.975+0000 7f1265dcb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:06:25.088 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:06:25.088 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":289,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:05:29.915532+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:00:52.362313+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:02:53.966248+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:04:16.232725+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:06:25.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:06:25 trial068 ceph-mon[38805]: pgmap v4031: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:06:25.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:06:25 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1723305671' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:06:25.616 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6779 8 365281 365273 0.420929 0 - 0.0299513 2026-02-23T15:06:26.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:06:26 trial068 ceph-mon[38805]: 5.5 scrub starts 2026-02-23T15:06:26.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:06:26 trial068 ceph-mon[38805]: 5.5 scrub ok 2026-02-23T15:06:26.616 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T15:06:26.616447+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T15:06:26.617 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T15:06:26.617 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6780 8 365281 365273 0.420867 0 - 0.0299513 2026-02-23T15:06:27.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:06:27 trial068 ceph-mon[38805]: pgmap v4032: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:06:27.616 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6781 8 365281 365273 0.420805 0 - 0.0299513 2026-02-23T15:06:28.125 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:06:28.243 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:06:28.346 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:06:28.344+0000 7f979791a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:06:28.346 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:06:28.344+0000 7f979791a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:06:28.453 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:06:28.454 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":289,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:05:29.915532+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:00:52.362313+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:02:53.966248+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:04:16.232725+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:06:28.454 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:06:28 trial068 ceph-mon[38805]: 5.e scrub starts 2026-02-23T15:06:28.617 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6782 8 365281 365273 0.420743 0 - 0.0299513 2026-02-23T15:06:29.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:06:29 trial068 ceph-mon[38805]: 5.e scrub ok 2026-02-23T15:06:29.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:06:29 trial068 ceph-mon[38805]: pgmap v4033: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:06:29.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:06:29 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/994163521' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:06:29.617 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6783 8 365281 365273 0.420681 0 - 0.0299513 2026-02-23T15:06:30.617 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6784 8 365281 365273 0.420619 0 - 0.0299513 2026-02-23T15:06:31.489 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:06:31.515 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:06:31 trial068 ceph-mon[38805]: pgmap v4034: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:06:31.607 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:06:31.617 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6785 8 365281 365273 0.420557 0 - 0.0299513 2026-02-23T15:06:31.706 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:06:31.704+0000 7f8a72a6a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:06:31.706 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:06:31.704+0000 7f8a72a6a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:06:31.814 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:06:31.814 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":289,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:05:29.915532+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:00:52.362313+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:02:53.966248+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:04:16.232725+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:06:32.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:06:32 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/705243889' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:06:32.617 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6786 8 365281 365273 0.420495 0 - 0.0299513 2026-02-23T15:06:33.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:06:33 trial068 ceph-mon[38805]: pgmap v4035: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:06:33.617 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6787 8 365281 365273 0.420433 0 - 0.0299513 2026-02-23T15:06:34.265 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:06:33 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:06:33.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:06:34.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:06:33 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:06:33] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:06:34.617 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6788 8 365281 365273 0.420371 0 - 0.0299513 2026-02-23T15:06:34.866 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:06:34.984 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:06:35.083 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:06:35.081+0000 7fc8253b5640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:06:35.083 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:06:35.081+0000 7fc8253b5640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:06:35.191 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:06:35.191 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":289,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:05:29.915532+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:00:52.362313+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:02:53.966248+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:04:16.232725+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:06:35.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:06:35 trial068 ceph-mon[38805]: 5.7 scrub starts 2026-02-23T15:06:35.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:06:35 trial068 ceph-mon[38805]: pgmap v4036: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:06:35.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:06:35 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2313481027' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:06:35.617 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6789 8 365281 365273 0.420309 0 - 0.0299513 2026-02-23T15:06:36.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:06:36 trial068 ceph-mon[38805]: 5.7 scrub ok 2026-02-23T15:06:36.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:06:36 trial068 ceph-mon[38805]: osdmap e290: 4 total, 3 up, 4 in 2026-02-23T15:06:36.617 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6790 8 365281 365273 0.420247 0 - 0.0299513 2026-02-23T15:06:37.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:06:37 trial068 ceph-mon[38805]: pgmap v4038: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:06:37.617 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6791 8 365281 365273 0.420185 0 - 0.0299513 2026-02-23T15:06:38.228 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:06:38.344 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:06:38.451 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:06:38.449+0000 7f7025302640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:06:38.452 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:06:38.450+0000 7f7025302640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:06:38.561 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:06:38.562 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":290,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:06:35.223408+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:05:38.714263+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:02:53.966248+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:04:16.232725+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:06:38.617 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6792 8 365281 365273 0.420123 0 - 0.0299513 2026-02-23T15:06:39.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:06:39 trial068 ceph-mon[38805]: pgmap v4039: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:06:39.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:06:39 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1263091924' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:06:39.617 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6793 8 365281 365273 0.420061 0 - 0.0299513 2026-02-23T15:06:40.617 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6794 8 365281 365273 0.42 0 - 0.0299513 2026-02-23T15:06:41.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:06:40 trial068 ceph-mon[38805]: osdmap e291: 4 total, 3 up, 4 in 2026-02-23T15:06:41.600 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:06:41.618 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6795 8 365281 365273 0.419938 0 - 0.0299513 2026-02-23T15:06:41.718 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:06:41.815 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:06:41.813+0000 7f177c2d9640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:06:41.815 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:06:41.814+0000 7f177c2d9640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:06:41.924 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:06:41.924 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":291,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:06:39.925670+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:05:38.714263+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:02:53.966248+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:04:16.232725+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:06:42.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:06:41 trial068 ceph-mon[38805]: pgmap v4041: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:06:42.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:06:41 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2015485704' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:06:42.618 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6796 8 365281 365273 0.419876 0 - 0.0299513 2026-02-23T15:06:43.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:06:42 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T15:06:43.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:06:42 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T15:06:43.618 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6797 8 365281 365273 0.419814 0 - 0.0299513 2026-02-23T15:06:44.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:06:43 trial068 ceph-mon[38805]: pgmap v4042: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:06:44.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:06:43 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:06:43] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:06:44.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:06:43 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:06:43.974Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:06:44.618 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6798 8 365281 365273 0.419752 0 - 0.0299513 2026-02-23T15:06:44.963 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:06:45.081 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:06:45.185 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:06:45.183+0000 7f052f8c1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:06:45.185 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:06:45.183+0000 7f052f8c1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:06:45.299 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:06:45.299 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":291,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:06:39.925670+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:05:38.714263+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:02:53.966248+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:04:16.232725+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:06:45.618 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6799 8 365281 365273 0.419691 0 - 0.0299513 2026-02-23T15:06:46.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:06:45 trial068 ceph-mon[38805]: pgmap v4043: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:06:46.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:06:45 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/438734230' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:06:46.618 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T15:06:46.617757+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T15:06:46.618 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T15:06:46.618 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6800 8 365281 365273 0.419629 0 - 0.0299513 2026-02-23T15:06:47.618 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6801 8 365281 365273 0.419567 0 - 0.0299513 2026-02-23T15:06:48.336 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:06:48.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:06:47 trial068 ceph-mon[38805]: pgmap v4044: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:06:48.454 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:06:48.556 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:06:48.554+0000 7f8549bb0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:06:48.556 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:06:48.555+0000 7f8549bb0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:06:48.618 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6802 8 365281 365273 0.419506 0 - 0.0299513 2026-02-23T15:06:48.665 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:06:48.665 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":291,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:06:39.925670+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:05:38.714263+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:02:53.966248+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:04:16.232725+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:06:49.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:06:48 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2136217776' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:06:49.618 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6803 8 365281 365273 0.419444 0 - 0.0299513 2026-02-23T15:06:50.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:06:49 trial068 ceph-mon[38805]: pgmap v4045: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:06:50.618 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6804 8 365281 365273 0.419382 0 - 0.0299513 2026-02-23T15:06:51.618 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6805 8 365281 365273 0.419321 0 - 0.0299513 2026-02-23T15:06:51.704 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:06:51.822 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:06:51.922 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:06:51.920+0000 7f9227184640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:06:51.922 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:06:51.920+0000 7f9227184640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:06:52.032 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:06:52.033 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":291,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:06:39.925670+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:05:38.714263+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:02:53.966248+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:04:16.232725+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:06:52.033 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:06:51 trial068 ceph-mon[38805]: pgmap v4046: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:06:52.619 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6806 8 365281 365273 0.419259 0 - 0.0299513 2026-02-23T15:06:53.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:06:52 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2039148880' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:06:53.619 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6807 8 365281 365273 0.419197 0 - 0.0299513 2026-02-23T15:06:54.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:06:53 trial068 ceph-mon[38805]: pgmap v4047: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:06:54.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:06:53 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:06:53] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:06:54.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:06:53 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:06:53.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:06:54.619 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6808 8 365281 365273 0.419136 0 - 0.0299513 2026-02-23T15:06:55.072 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:06:55.191 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:06:55.294 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:06:55.292+0000 7f821cb41640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:06:55.294 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:06:55.292+0000 7f821cb41640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:06:55.402 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:06:55.402 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":291,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:06:39.925670+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:05:38.714263+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:02:53.966248+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:04:16.232725+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:06:55.619 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6809 8 365281 365273 0.419074 0 - 0.0299513 2026-02-23T15:06:56.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:06:55 trial068 ceph-mon[38805]: pgmap v4048: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:06:56.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:06:55 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2622454498' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:06:56.619 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6810 8 365281 365273 0.419013 0 - 0.0299513 2026-02-23T15:06:57.619 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6811 8 365281 365273 0.418951 0 - 0.0299513 2026-02-23T15:06:58.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:06:57 trial068 ceph-mon[38805]: pgmap v4049: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:06:58.439 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:06:58.556 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:06:58.619 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6812 8 365281 365273 0.41889 0 - 0.0299513 2026-02-23T15:06:58.653 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:06:58.651+0000 7fdbadbbe640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:06:58.653 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:06:58.651+0000 7fdbadbbe640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:06:58.761 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:06:58.761 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":291,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:06:39.925670+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:05:38.714263+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:02:53.966248+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:04:16.232725+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:06:59.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:06:58 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1252955797' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:06:59.619 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6813 8 365281 365273 0.418828 0 - 0.0299513 2026-02-23T15:07:00.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:06:59 trial068 ceph-mon[38805]: pgmap v4050: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:07:00.619 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6814 8 365281 365273 0.418767 0 - 0.0299513 2026-02-23T15:07:01.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:07:00 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T15:07:01.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:07:00 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T15:07:01.619 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6815 8 365281 365273 0.418705 0 - 0.0299513 2026-02-23T15:07:01.800 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:07:01.919 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:07:02.025 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:07:02.023+0000 7efc68f32640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:07:02.025 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:07:02.023+0000 7efc68f32640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:07:02.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:07:01 trial068 ceph-mon[38805]: pgmap v4051: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:07:02.134 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:07:02.134 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":291,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:06:39.925670+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:05:38.714263+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:02:53.966248+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:04:16.232725+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:07:02.619 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6816 8 365281 365273 0.418644 0 - 0.0299513 2026-02-23T15:07:03.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:07:02 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/750059700' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:07:03.619 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6817 8 365281 365273 0.418583 0 - 0.0299513 2026-02-23T15:07:04.265 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:07:03 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:07:03.972Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:07:04.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:07:03 trial068 ceph-mon[38805]: pgmap v4052: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:07:04.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:07:03 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:07:03] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:07:04.619 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6818 8 365281 365273 0.418521 0 - 0.0299513 2026-02-23T15:07:05.173 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:07:05.291 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:07:05.390 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:07:05.388+0000 7f974874e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:07:05.390 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:07:05.388+0000 7f974874e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:07:05.500 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:07:05.500 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":291,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:06:39.925670+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:05:38.714263+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:02:53.966248+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:04:16.232725+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:07:05.619 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6819 8 365281 365273 0.41846 0 - 0.0299513 2026-02-23T15:07:06.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:07:05 trial068 ceph-mon[38805]: pgmap v4053: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:07:06.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:07:05 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/634750642' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:07:06.619 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T15:07:06.619347+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T15:07:06.620 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T15:07:06.620 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6820 8 365281 365273 0.418398 0 - 0.0299513 2026-02-23T15:07:07.619 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6821 8 365281 365273 0.418337 0 - 0.0299513 2026-02-23T15:07:08.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:07:07 trial068 ceph-mon[38805]: pgmap v4054: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:07:08.539 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:07:08.619 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6822 8 365281 365273 0.418276 0 - 0.0299513 2026-02-23T15:07:08.655 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:07:08.753 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:07:08.751+0000 7f55fd8c1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:07:08.753 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:07:08.751+0000 7f55fd8c1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:07:08.862 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:07:08.862 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":291,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:06:39.925670+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:05:38.714263+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:02:53.966248+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:04:16.232725+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:07:09.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:07:08 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2679898951' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:07:09.620 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6823 8 365281 365273 0.418214 0 - 0.0299513 2026-02-23T15:07:10.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:07:09 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T15:07:10.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:07:09 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T15:07:10.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:07:09 trial068 ceph-mon[38805]: pgmap v4055: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:07:10.620 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6824 8 365281 365273 0.418153 0 - 0.0299513 2026-02-23T15:07:11.620 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6825 8 365281 365273 0.418092 0 - 0.0299513 2026-02-23T15:07:11.899 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:07:12.018 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:07:12.118 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:07:12.116+0000 7efd4a766640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:07:12.118 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:07:12.116+0000 7efd4a766640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:07:12.230 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:07:12.231 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":291,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:06:39.925670+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:05:38.714263+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:02:53.966248+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:04:16.232725+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:07:12.231 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:07:11 trial068 ceph-mon[38805]: pgmap v4056: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:07:12.620 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6826 8 365281 365273 0.418031 0 - 0.0299513 2026-02-23T15:07:13.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:07:12 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2026-02-23T15:07:13.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:07:12 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/777091406' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:07:13.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:07:12 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T15:07:13.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:07:12 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T15:07:13.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:07:12 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T15:07:13.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:07:12 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T15:07:13.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:07:12 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-23T15:07:13.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:07:12 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2026-02-23T15:07:13.345 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:07:12 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T15:07:13.345 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:07:12 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2026-02-23T15:07:13.620 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6827 8 365281 365273 0.417969 0 - 0.0299513 2026-02-23T15:07:14.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:07:13 trial068 ceph-mon[38805]: pgmap v4057: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:07:14.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:07:13 trial068 ceph-mon[38805]: pgmap v4058: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:07:14.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:07:13 trial068 ceph-mon[38805]: pgmap v4059: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:07:14.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:07:13 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:07:13] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:07:14.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:07:13 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:07:13.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:07:14.620 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6828 8 365281 365273 0.417908 0 - 0.0299513 2026-02-23T15:07:15.270 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:07:15.387 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:07:15.492 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:07:15.490+0000 7f959b61c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:07:15.492 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:07:15.490+0000 7f959b61c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:07:15.602 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:07:15.603 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":291,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:06:39.925670+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:05:38.714263+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:02:53.966248+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:04:16.232725+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:07:15.620 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6829 8 365281 365273 0.417847 0 - 0.0299513 2026-02-23T15:07:16.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:07:15 trial068 ceph-mon[38805]: pgmap v4060: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:07:16.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:07:15 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1978213550' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:07:16.620 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6830 8 365281 365273 0.417786 0 - 0.0299513 2026-02-23T15:07:17.620 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6831 8 365281 365273 0.417725 0 - 0.0299513 2026-02-23T15:07:18.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:07:17 trial068 ceph-mon[38805]: pgmap v4061: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:07:18.620 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6832 8 365281 365273 0.417664 0 - 0.0299513 2026-02-23T15:07:18.640 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:07:18.760 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:07:18.864 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:07:18.862+0000 7f1c8483b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:07:18.864 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:07:18.862+0000 7f1c8483b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:07:18.974 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:07:18.974 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":291,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:06:39.925670+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:05:38.714263+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:02:53.966248+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:04:16.232725+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:07:19.620 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6833 8 365281 365273 0.417602 0 - 0.0299513 2026-02-23T15:07:20.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:07:19 trial068 ceph-mon[38805]: pgmap v4062: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:07:20.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:07:19 trial068 ceph-mon[38805]: 5.a scrub starts 2026-02-23T15:07:20.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:07:19 trial068 ceph-mon[38805]: 5.a scrub ok 2026-02-23T15:07:20.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:07:19 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1859518379' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:07:20.620 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6834 8 365281 365273 0.417541 0 - 0.0299513 2026-02-23T15:07:21.620 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6835 8 365281 365273 0.41748 0 - 0.0299513 2026-02-23T15:07:22.013 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:07:22.129 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:07:22.228 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:07:22.226+0000 7f73046fe640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:07:22.228 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:07:22.226+0000 7f73046fe640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:07:22.229 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:07:21 trial068 ceph-mon[38805]: pgmap v4063: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:07:22.229 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:07:21 trial068 ceph-mon[38805]: 5.8 scrub starts 2026-02-23T15:07:22.229 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:07:21 trial068 ceph-mon[38805]: 5.8 scrub ok 2026-02-23T15:07:22.337 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:07:22.338 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":291,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:06:39.925670+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:05:38.714263+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:02:53.966248+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:04:16.232725+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:07:22.620 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6836 8 365281 365273 0.417419 0 - 0.0299513 2026-02-23T15:07:23.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:07:22 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/268583097' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:07:23.620 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6837 8 365281 365273 0.417358 0 - 0.0299513 2026-02-23T15:07:24.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:07:23 trial068 ceph-mon[38805]: pgmap v4064: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:07:24.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:07:23 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:07:23] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:07:24.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:07:23 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:07:23.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:07:24.620 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6838 8 365281 365273 0.417297 0 - 0.0299513 2026-02-23T15:07:25.392 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:07:25.511 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:07:25.613 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:07:25.612+0000 7f3bb1c90640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:07:25.614 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:07:25.612+0000 7f3bb1c90640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:07:25.621 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6839 8 365281 365273 0.417236 0 - 0.0299513 2026-02-23T15:07:25.726 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:07:25.726 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":291,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:06:39.925670+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:05:38.714263+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:02:53.966248+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:04:16.232725+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:07:26.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:07:25 trial068 ceph-mon[38805]: pgmap v4065: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:07:26.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:07:25 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/127330928' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:07:26.621 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T15:07:26.620655+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T15:07:26.621 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T15:07:26.621 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6840 8 365281 365273 0.417175 0 - 0.0299513 2026-02-23T15:07:27.621 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6841 8 365281 365273 0.417114 0 - 0.0299513 2026-02-23T15:07:28.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:07:27 trial068 ceph-mon[38805]: pgmap v4066: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:07:28.620 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6842 8 365281 365273 0.417053 0 - 0.0299513 2026-02-23T15:07:28.781 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:07:28.896 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:07:28.996 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:07:28.994+0000 7f0af2573640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:07:28.996 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:07:28.995+0000 7f0af2573640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:07:29.105 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:07:29.105 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":291,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:06:39.925670+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:05:38.714263+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:02:53.966248+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:04:16.232725+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:07:29.621 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6843 8 365281 365273 0.416992 0 - 0.0299513 2026-02-23T15:07:30.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:07:29 trial068 ceph-mon[38805]: pgmap v4067: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:07:30.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:07:29 trial068 ceph-mon[38805]: 5.5 scrub starts 2026-02-23T15:07:30.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:07:29 trial068 ceph-mon[38805]: 5.5 scrub ok 2026-02-23T15:07:30.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:07:29 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/655947999' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:07:30.621 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6844 8 365281 365273 0.416931 0 - 0.0299513 2026-02-23T15:07:31.621 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6845 8 365281 365273 0.41687 0 - 0.0299513 2026-02-23T15:07:32.144 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:07:32.261 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:07:32.284 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:07:31 trial068 ceph-mon[38805]: pgmap v4068: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:07:32.370 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:07:32.368+0000 7f277e8f2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:07:32.370 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:07:32.369+0000 7f277e8f2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:07:32.479 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:07:32.479 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":291,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:06:39.925670+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:05:38.714263+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:02:53.966248+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:04:16.232725+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:07:32.621 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6846 8 365281 365273 0.416809 0 - 0.0299513 2026-02-23T15:07:33.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:07:32 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1755382063' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:07:33.621 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6847 8 365281 365273 0.416749 0 - 0.0299513 2026-02-23T15:07:34.265 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:07:33 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:07:33.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:07:34.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:07:33 trial068 ceph-mon[38805]: pgmap v4069: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:07:34.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:07:33 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:07:33] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:07:34.621 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6848 8 365281 365273 0.416688 0 - 0.0299513 2026-02-23T15:07:35.515 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:07:35.621 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6849 8 365281 365273 0.416627 0 - 0.0299513 2026-02-23T15:07:35.634 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:07:35.742 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:07:35.740+0000 7f332d376640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:07:35.742 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:07:35.740+0000 7f332d376640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:07:35.851 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:07:35.851 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":291,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:06:39.925670+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:05:38.714263+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:02:53.966248+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:04:16.232725+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:07:36.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:07:35 trial068 ceph-mon[38805]: pgmap v4070: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:07:36.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:07:35 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2844232954' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:07:36.621 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6850 8 365281 365273 0.416566 0 - 0.0299513 2026-02-23T15:07:37.621 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6851 8 365281 365273 0.416505 0 - 0.0299513 2026-02-23T15:07:38.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:07:37 trial068 ceph-mon[38805]: pgmap v4071: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:07:38.621 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6852 8 365281 365273 0.416444 0 - 0.0299513 2026-02-23T15:07:38.887 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:07:39.004 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:07:39.102 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:07:39.100+0000 7fa704167640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:07:39.102 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:07:39.100+0000 7fa704167640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:07:39.214 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:07:39.214 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":291,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:06:39.925670+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:05:38.714263+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:02:53.966248+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:04:16.232725+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:07:39.621 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6853 8 365281 365273 0.416384 0 - 0.0299513 2026-02-23T15:07:40.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:07:39 trial068 ceph-mon[38805]: pgmap v4072: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:07:40.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:07:39 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/988956420' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:07:40.622 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6854 8 365281 365273 0.416323 0 - 0.0299513 2026-02-23T15:07:41.622 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6855 8 365281 365273 0.416262 0 - 0.0299513 2026-02-23T15:07:42.252 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:07:42.277 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:07:41 trial068 ceph-mon[38805]: pgmap v4073: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:07:42.372 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:07:42.470 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:07:42.468+0000 7fcb0b664640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:07:42.471 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:07:42.469+0000 7fcb0b664640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:07:42.578 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:07:42.578 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":291,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:06:39.925670+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:05:38.714263+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:02:53.966248+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:04:16.232725+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:07:42.622 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6856 8 365281 365273 0.416201 0 - 0.0299513 2026-02-23T15:07:43.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:07:42 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/4217115674' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:07:43.622 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6857 8 365281 365273 0.416141 0 - 0.0299513 2026-02-23T15:07:44.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:07:43 trial068 ceph-mon[38805]: pgmap v4074: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:07:44.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:07:43 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:07:43] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:07:44.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:07:43 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:07:43.974Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:07:44.622 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6858 8 365281 365273 0.41608 0 - 0.0299513 2026-02-23T15:07:45.616 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:07:45.622 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6859 8 365281 365273 0.416019 0 - 0.0299513 2026-02-23T15:07:45.734 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:07:45.836 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:07:45.834+0000 7fb7b7622640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:07:45.836 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:07:45.834+0000 7fb7b7622640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:07:45.945 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:07:45.945 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":291,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:06:39.925670+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:05:38.714263+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:02:53.966248+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:04:16.232725+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:07:46.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:07:45 trial068 ceph-mon[38805]: pgmap v4075: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:07:46.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:07:45 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1900759639' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:07:46.622 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T15:07:46.621911+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T15:07:46.622 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T15:07:46.622 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6860 8 365281 365273 0.415959 0 - 0.0299513 2026-02-23T15:07:47.622 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6861 8 365281 365273 0.415898 0 - 0.0299513 2026-02-23T15:07:48.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:07:47 trial068 ceph-mon[38805]: pgmap v4076: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:07:48.622 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6862 8 365281 365273 0.415838 0 - 0.0299513 2026-02-23T15:07:48.982 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:07:49.101 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:07:49.198 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:07:49.196+0000 7fa509383640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:07:49.198 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:07:49.197+0000 7fa509383640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:07:49.309 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:07:49.309 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":291,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:06:39.925670+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:05:38.714263+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:02:53.966248+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:04:16.232725+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:07:49.622 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6863 8 365281 365273 0.415777 0 - 0.0299513 2026-02-23T15:07:50.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:07:49 trial068 ceph-mon[38805]: pgmap v4077: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:07:50.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:07:49 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3439945885' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:07:50.622 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6864 8 365281 365273 0.415716 0 - 0.0299513 2026-02-23T15:07:51.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:07:50 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T15:07:51.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:07:50 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T15:07:51.622 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6865 8 365281 365273 0.415656 0 - 0.0299513 2026-02-23T15:07:52.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:07:51 trial068 ceph-mon[38805]: pgmap v4078: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:07:52.346 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:07:52.461 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:07:52.571 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:07:52.569+0000 7f47bde7c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:07:52.571 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:07:52.569+0000 7f47bde7c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:07:52.622 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6866 8 365281 365273 0.415595 0 - 0.0299513 2026-02-23T15:07:52.681 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:07:52.681 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":291,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:06:39.925670+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:05:38.714263+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:02:53.966248+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:04:16.232725+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:07:53.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:07:52 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3220436833' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:07:53.622 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6867 8 365281 365273 0.415535 0 - 0.0299513 2026-02-23T15:07:54.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:07:53 trial068 ceph-mon[38805]: pgmap v4079: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:07:54.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:07:53 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:07:53] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:07:54.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:07:53 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:07:53.974Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:07:54.622 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6868 8 365281 365273 0.415474 0 - 0.0299513 2026-02-23T15:07:55.622 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6869 8 365281 365273 0.415414 0 - 0.0299513 2026-02-23T15:07:55.719 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:07:55.837 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:07:55.939 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:07:55.938+0000 7f8703414640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:07:55.940 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:07:55.938+0000 7f8703414640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:07:56.049 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:07:56.050 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":291,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:06:39.925670+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:05:38.714263+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:02:53.966248+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:04:16.232725+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:07:56.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:07:55 trial068 ceph-mon[38805]: pgmap v4080: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:07:56.623 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6870 8 365281 365273 0.415353 0 - 0.0299513 2026-02-23T15:07:57.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:07:56 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/615989166' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:07:57.623 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6871 8 365281 365273 0.415293 0 - 0.0299513 2026-02-23T15:07:58.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:07:57 trial068 ceph-mon[38805]: pgmap v4081: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:07:58.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:07:57 trial068 ceph-mon[38805]: 5.e scrub starts 2026-02-23T15:07:58.623 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6872 8 365281 365273 0.415232 0 - 0.0299513 2026-02-23T15:07:59.087 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:07:59.204 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:07:59.253 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:07:59 trial068 ceph-mon[38805]: 5.e scrub ok 2026-02-23T15:07:59.254 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:07:59 trial068 ceph-mon[38805]: 5.7 scrub starts 2026-02-23T15:07:59.305 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:07:59.303+0000 7fc62088b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:07:59.305 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:07:59.304+0000 7fc62088b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:07:59.414 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:07:59.414 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":291,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:06:39.925670+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:05:38.714263+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:02:53.966248+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:04:16.232725+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:07:59.623 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6873 8 365281 365273 0.415172 0 - 0.0299513 2026-02-23T15:08:00.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:08:00 trial068 ceph-mon[38805]: 5.7 scrub ok 2026-02-23T15:08:00.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:08:00 trial068 ceph-mon[38805]: pgmap v4082: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:08:00.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:08:00 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1486819528' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:08:00.623 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6874 8 365281 365273 0.415112 0 - 0.0299513 2026-02-23T15:08:01.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:08:01 trial068 ceph-mon[38805]: pgmap v4083: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:08:01.623 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6875 8 365281 365273 0.415051 0 - 0.0299513 2026-02-23T15:08:02.453 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:08:02.572 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:08:02.623 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6876 8 365281 365273 0.414991 0 - 0.0299513 2026-02-23T15:08:02.672 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:08:02.670+0000 7fed56483640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:08:02.672 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:08:02.671+0000 7fed56483640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:08:02.784 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:08:02.784 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":291,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:06:39.925670+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:05:38.714263+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:02:53.966248+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:04:16.232725+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:08:03.623 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6877 8 365281 365273 0.414931 0 - 0.0299513 2026-02-23T15:08:03.974 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:08:03 trial068 ceph-mon[38805]: pgmap v4084: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:08:03.974 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:08:03 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/4225969977' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:08:04.265 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:08:03 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:08:03.974Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:08:04.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:08:03 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:08:03] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:08:04.623 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6878 8 365281 365273 0.41487 0 - 0.0299513 2026-02-23T15:08:05.623 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6879 8 365281 365273 0.41481 0 - 0.0299513 2026-02-23T15:08:05.820 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:08:05.938 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:08:05.960 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:08:05 trial068 ceph-mon[38805]: pgmap v4085: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:08:06.035 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:08:06.033+0000 7f78aa0fd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:08:06.035 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:08:06.033+0000 7f78aa0fd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:08:06.144 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:08:06.144 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":291,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:06:39.925670+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:05:38.714263+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:02:53.966248+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:04:16.232725+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:08:06.623 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T15:08:06.623307+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T15:08:06.624 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T15:08:06.624 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6880 8 365281 365273 0.41475 0 - 0.0299513 2026-02-23T15:08:07.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:08:06 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/161132833' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:08:07.623 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6881 8 365281 365273 0.414689 0 - 0.0299513 2026-02-23T15:08:08.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:08:07 trial068 ceph-mon[38805]: pgmap v4086: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:08:08.623 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6882 8 365281 365273 0.414629 0 - 0.0299513 2026-02-23T15:08:09.182 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:08:09.301 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:08:09.398 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:08:09.396+0000 7f3a3557d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:08:09.398 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:08:09.396+0000 7f3a3557d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:08:09.508 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:08:09.508 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":291,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:06:39.925670+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:05:38.714263+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:02:53.966248+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:04:16.232725+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:08:09.623 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6883 8 365281 365273 0.414569 0 - 0.0299513 2026-02-23T15:08:09.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:08:09 trial068 ceph-mon[38805]: pgmap v4087: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:08:09.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:08:09 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1684383889' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:08:10.624 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6884 8 365281 365273 0.414509 0 - 0.0299513 2026-02-23T15:08:11.624 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6885 8 365281 365273 0.414448 0 - 0.0299513 2026-02-23T15:08:12.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:08:11 trial068 ceph-mon[38805]: pgmap v4088: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:08:12.546 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:08:12.624 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6886 8 365281 365273 0.414388 0 - 0.0299513 2026-02-23T15:08:12.666 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:08:12.776 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:08:12.774+0000 7f0e152b9640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:08:12.776 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:08:12.774+0000 7f0e152b9640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:08:12.827 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:08:12 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2026-02-23T15:08:12.888 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:08:12.889 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":291,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:06:39.925670+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:05:38.714263+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:02:53.966248+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:04:16.232725+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:08:13.624 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6887 8 365281 365273 0.414328 0 - 0.0299513 2026-02-23T15:08:13.973 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:08:13 trial068 ceph-mon[38805]: pgmap v4089: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:08:13.973 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:08:13 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3868049979' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:08:13.973 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:08:13 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T15:08:13.974 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:08:13 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T15:08:13.974 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:08:13 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T15:08:13.974 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:08:13 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T15:08:13.974 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:08:13 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T15:08:13.974 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:08:13 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T15:08:13.974 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:08:13 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T15:08:13.974 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:08:13 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T15:08:14.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:08:13 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:08:13] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:08:14.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:08:13 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:08:13.972Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:08:14.624 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6888 8 365281 365273 0.414268 0 - 0.0299513 2026-02-23T15:08:15.624 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6889 8 365281 365273 0.414208 0 - 0.0299513 2026-02-23T15:08:15.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:08:15 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T15:08:15.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:08:15 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T15:08:15.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:08:15 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T15:08:15.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:08:15 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T15:08:15.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:08:15 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-23T15:08:15.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:08:15 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2026-02-23T15:08:15.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:08:15 trial068 ceph-mon[38805]: pgmap v4090: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:08:15.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:08:15 trial068 ceph-mon[38805]: pgmap v4091: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:08:15.845 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:08:15 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T15:08:15.845 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:08:15 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2026-02-23T15:08:15.928 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:08:16.049 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:08:16.150 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:08:16.148+0000 7f8411576640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:08:16.150 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:08:16.148+0000 7f8411576640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:08:16.258 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:08:16.258 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":291,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:06:39.925670+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:05:38.714263+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:02:53.966248+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:04:16.232725+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:08:16.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:08:16 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2251144082' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:08:16.624 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6890 8 365281 365273 0.414148 0 - 0.0299513 2026-02-23T15:08:17.624 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6891 8 365281 365273 0.414088 0 - 0.0299513 2026-02-23T15:08:17.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:08:17 trial068 ceph-mon[38805]: pgmap v4092: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:08:18.624 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6892 8 365281 365273 0.414027 0 - 0.0299513 2026-02-23T15:08:18.744 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:08:18 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T15:08:18.744 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:08:18 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T15:08:19.296 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:08:19.415 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:08:19.514 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:08:19.513+0000 7f881c8a7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:08:19.515 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:08:19.513+0000 7f881c8a7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:08:19.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:08:19 trial068 ceph-mon[38805]: pgmap v4093: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:08:19.624 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6893 8 365281 365273 0.413967 0 - 0.0299513 2026-02-23T15:08:19.625 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:08:19.625 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":291,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:06:39.925670+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:05:38.714263+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:02:53.966248+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:04:16.232725+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:08:20.624 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6894 8 365281 365273 0.413907 0 - 0.0299513 2026-02-23T15:08:20.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:08:20 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2520372627' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:08:21.625 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6895 8 365281 365273 0.413847 0 - 0.0299513 2026-02-23T15:08:22.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:08:21 trial068 ceph-mon[38805]: pgmap v4094: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:08:22.624 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6896 8 365281 365273 0.413787 0 - 0.0299513 2026-02-23T15:08:22.661 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:08:22.778 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:08:22.879 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:08:22.877+0000 7fc76b5f2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:08:22.879 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:08:22.878+0000 7fc76b5f2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:08:22.992 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:08:22.992 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":291,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:06:39.925670+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:05:38.714263+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:02:53.966248+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:04:16.232725+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:08:23.625 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6897 8 365281 365273 0.413727 0 - 0.0299513 2026-02-23T15:08:24.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:08:23 trial068 ceph-mon[38805]: pgmap v4095: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:08:24.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:08:23 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1656247891' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:08:24.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:08:23 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:08:23] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:08:24.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:08:23 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:08:23.972Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:08:24.625 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6898 8 365281 365273 0.413667 0 - 0.0299513 2026-02-23T15:08:25.625 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6899 8 365281 365273 0.413607 0 - 0.0299513 2026-02-23T15:08:26.030 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:08:26.149 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:08:26.196 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:08:25 trial068 ceph-mon[38805]: pgmap v4096: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:08:26.254 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:08:26.253+0000 7f9517519640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:08:26.254 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:08:26.253+0000 7f9517519640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:08:26.365 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:08:26.365 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":291,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:06:39.925670+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:05:38.714263+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:02:53.966248+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:04:16.232725+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:08:26.625 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T15:08:26.624662+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T15:08:26.625 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T15:08:26.625 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6900 8 365281 365273 0.413547 0 - 0.0299513 2026-02-23T15:08:27.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:08:26 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2379905887' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:08:27.625 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6901 8 365281 365273 0.413488 0 - 0.0299513 2026-02-23T15:08:28.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:08:27 trial068 ceph-mon[38805]: pgmap v4097: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:08:28.625 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6902 8 365281 365273 0.413428 0 - 0.0299513 2026-02-23T15:08:29.402 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:08:29.519 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:08:29.625 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6903 8 365281 365273 0.413368 0 - 0.0299513 2026-02-23T15:08:29.628 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:08:29.627+0000 7fe1c3398640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:08:29.629 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:08:29.627+0000 7fe1c3398640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:08:29.737 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:08:29.738 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":291,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:06:39.925670+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:05:38.714263+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:02:53.966248+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:04:16.232725+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:08:30.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:08:29 trial068 ceph-mon[38805]: pgmap v4098: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:08:30.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:08:29 trial068 ceph-mon[38805]: 5.a scrub starts 2026-02-23T15:08:30.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:08:29 trial068 ceph-mon[38805]: 5.a scrub ok 2026-02-23T15:08:30.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:08:29 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3374841447' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:08:30.625 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6904 8 365281 365273 0.413308 0 - 0.0299513 2026-02-23T15:08:31.625 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6905 8 365281 365273 0.413248 0 - 0.0299513 2026-02-23T15:08:32.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:08:31 trial068 ceph-mon[38805]: pgmap v4099: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:08:32.625 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6906 8 365281 365273 0.413188 0 - 0.0299513 2026-02-23T15:08:32.774 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:08:32.891 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:08:32.986 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:08:32.985+0000 7ffb2ff88640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:08:32.987 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:08:32.985+0000 7ffb2ff88640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:08:33.096 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:08:33.096 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":291,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:06:39.925670+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:05:38.714263+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:02:53.966248+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:04:16.232725+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:08:33.625 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6907 8 365281 365273 0.413128 0 - 0.0299513 2026-02-23T15:08:34.263 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:08:33 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:08:33.972Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:08:34.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:08:33 trial068 ceph-mon[38805]: pgmap v4100: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:08:34.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:08:33 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1169350307' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:08:34.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:08:33 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:08:33] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:08:34.625 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6908 8 365281 365273 0.413069 0 - 0.0299513 2026-02-23T15:08:35.625 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6909 8 365281 365273 0.413009 0 - 0.0299513 2026-02-23T15:08:36.135 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:08:36.253 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:08:36.274 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:08:35 trial068 ceph-mon[38805]: pgmap v4101: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:08:36.350 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:08:36.348+0000 7fde8b8a7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:08:36.350 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:08:36.348+0000 7fde8b8a7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:08:36.459 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:08:36.459 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":291,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:06:39.925670+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:05:38.714263+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:02:53.966248+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:04:16.232725+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:08:36.625 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6910 8 365281 365273 0.412949 0 - 0.0299513 2026-02-23T15:08:37.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:08:36 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/632354623' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:08:37.625 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6911 8 365281 365273 0.412889 0 - 0.0299513 2026-02-23T15:08:38.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:08:37 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T15:08:38.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:08:37 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T15:08:38.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:08:37 trial068 ceph-mon[38805]: pgmap v4102: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:08:38.625 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6912 8 365281 365273 0.412829 0 - 0.0299513 2026-02-23T15:08:39.497 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:08:39.614 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:08:39.625 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6913 8 365281 365273 0.41277 0 - 0.0299513 2026-02-23T15:08:39.715 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:08:39.713+0000 7f08393c3640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:08:39.715 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:08:39.713+0000 7f08393c3640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:08:39.822 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:08:39.823 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":291,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:06:39.925670+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:05:38.714263+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:02:53.966248+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:04:16.232725+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:08:40.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:08:39 trial068 ceph-mon[38805]: pgmap v4103: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:08:40.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:08:39 trial068 ceph-mon[38805]: 5.8 scrub starts 2026-02-23T15:08:40.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:08:39 trial068 ceph-mon[38805]: 5.8 scrub ok 2026-02-23T15:08:40.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:08:39 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3240275168' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:08:40.626 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6914 8 365281 365273 0.41271 0 - 0.0299513 2026-02-23T15:08:41.626 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6915 8 365281 365273 0.41265 0 - 0.0299513 2026-02-23T15:08:42.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:08:41 trial068 ceph-mon[38805]: pgmap v4104: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:08:42.626 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6916 8 365281 365273 0.412591 0 - 0.0299513 2026-02-23T15:08:42.859 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:08:42.976 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:08:43.074 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:08:43.072+0000 7f9544072640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:08:43.074 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:08:43.072+0000 7f9544072640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:08:43.183 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:08:43.183 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":291,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:06:39.925670+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:05:38.714263+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:02:53.966248+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:04:16.232725+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:08:43.626 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6917 8 365281 365273 0.412531 0 - 0.0299513 2026-02-23T15:08:44.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:08:43 trial068 ceph-mon[38805]: pgmap v4105: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:08:44.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:08:43 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/786626627' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:08:44.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:08:43 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:08:43] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:08:44.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:08:43 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:08:43.972Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:08:44.626 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6918 8 365281 365273 0.412471 0 - 0.0299513 2026-02-23T15:08:45.626 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6919 8 365281 365273 0.412412 0 - 0.0299513 2026-02-23T15:08:46.220 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:08:46.243 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:08:45 trial068 ceph-mon[38805]: pgmap v4106: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:08:46.337 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:08:46.441 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:08:46.440+0000 7f2438831640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:08:46.441 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:08:46.440+0000 7f2438831640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:08:46.549 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:08:46.549 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":291,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:06:39.925670+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:05:38.714263+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:02:53.966248+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:04:16.232725+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:08:46.626 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T15:08:46.626006+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T15:08:46.626 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T15:08:46.626 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6920 8 365281 365273 0.412352 0 - 0.0299513 2026-02-23T15:08:47.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:08:46 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2563413357' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:08:47.626 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6921 8 365281 365273 0.412293 0 - 0.0299513 2026-02-23T15:08:48.244 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:08:47 trial068 ceph-mon[38805]: pgmap v4107: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:08:48.626 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6922 8 365281 365273 0.412233 0 - 0.0299513 2026-02-23T15:08:49.586 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:08:49.626 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6923 8 365281 365273 0.412174 0 - 0.0299513 2026-02-23T15:08:49.706 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:08:49.813 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:08:49.811+0000 7fc61fa3b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:08:49.813 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:08:49.811+0000 7fc61fa3b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:08:49.925 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:08:49.926 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":291,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:06:39.925670+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:05:38.714263+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:02:53.966248+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:04:16.232725+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:08:50.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:08:49 trial068 ceph-mon[38805]: pgmap v4108: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:08:50.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:08:49 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1225781625' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:08:50.626 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6924 8 365281 365273 0.412114 0 - 0.0299513 2026-02-23T15:08:51.626 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6925 8 365281 365273 0.412055 0 - 0.0299513 2026-02-23T15:08:52.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:08:51 trial068 ceph-mon[38805]: pgmap v4109: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:08:52.626 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6926 8 365281 365273 0.411995 0 - 0.0299513 2026-02-23T15:08:52.966 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:08:53.085 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:08:53.185 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:08:53.183+0000 7fbb1e3d3640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:08:53.185 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:08:53.184+0000 7fbb1e3d3640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:08:53.293 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:08:53.294 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":291,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:06:39.925670+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:05:38.714263+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:02:53.966248+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:04:16.232725+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:08:53.626 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6927 8 365281 365273 0.411936 0 - 0.0299513 2026-02-23T15:08:54.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:08:53 trial068 ceph-mon[38805]: pgmap v4110: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:08:54.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:08:53 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/979213495' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:08:54.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:08:53 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:08:53] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:08:54.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:08:53 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:08:53.972Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:08:54.626 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6928 8 365281 365273 0.411876 0 - 0.0299513 2026-02-23T15:08:55.626 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6929 8 365281 365273 0.411817 0 - 0.0299513 2026-02-23T15:08:56.331 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:08:56.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:08:55 trial068 ceph-mon[38805]: pgmap v4111: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:08:56.449 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:08:56.556 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:08:56.554+0000 7f36be4c4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:08:56.556 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:08:56.555+0000 7f36be4c4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:08:56.626 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6930 8 365281 365273 0.411757 0 - 0.0299513 2026-02-23T15:08:56.668 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:08:56.668 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":291,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:06:39.925670+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:05:38.714263+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:02:53.966248+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:04:16.232725+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:08:57.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:08:56 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/532807172' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:08:57.627 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6931 8 365281 365273 0.411698 0 - 0.0299513 2026-02-23T15:08:58.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:08:57 trial068 ceph-mon[38805]: pgmap v4112: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:08:58.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:08:57 trial068 ceph-mon[38805]: 5.5 scrub starts 2026-02-23T15:08:58.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:08:57 trial068 ceph-mon[38805]: 5.5 scrub ok 2026-02-23T15:08:58.627 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6932 8 365281 365273 0.411638 0 - 0.0299513 2026-02-23T15:08:59.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:08:58 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T15:08:59.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:08:58 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T15:08:59.627 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6933 8 365281 365273 0.411579 0 - 0.0299513 2026-02-23T15:08:59.708 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:08:59.827 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:08:59.932 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:08:59.930+0000 7f2c73915640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:08:59.932 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:08:59.930+0000 7f2c73915640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:09:00.042 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:09:00.042 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":291,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:06:39.925670+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:05:38.714263+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:02:53.966248+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:04:16.232725+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:09:00.042 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:08:59 trial068 ceph-mon[38805]: pgmap v4113: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:09:00.627 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6934 8 365281 365273 0.41152 0 - 0.0299513 2026-02-23T15:09:01.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:09:00 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/844149648' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:09:01.627 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6935 8 365281 365273 0.41146 0 - 0.0299513 2026-02-23T15:09:02.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:09:01 trial068 ceph-mon[38805]: pgmap v4114: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:09:02.627 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6936 8 365281 365273 0.411401 0 - 0.0299513 2026-02-23T15:09:03.078 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:09:03.196 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:09:03.304 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:09:03.302+0000 7f9d5df50640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:09:03.304 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:09:03.302+0000 7f9d5df50640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:09:03.414 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:09:03.414 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":291,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:06:39.925670+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:05:38.714263+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:02:53.966248+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:04:16.232725+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:09:03.627 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6937 8 365281 365273 0.411342 0 - 0.0299513 2026-02-23T15:09:04.265 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:09:03 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:09:03.974Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:09:04.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:09:03 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:09:03] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:09:04.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:09:03 trial068 ceph-mon[38805]: pgmap v4115: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:09:04.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:09:03 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2247738' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:09:04.627 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6938 8 365281 365273 0.411282 0 - 0.0299513 2026-02-23T15:09:05.627 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6939 8 365281 365273 0.411223 0 - 0.0299513 2026-02-23T15:09:06.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:09:05 trial068 ceph-mon[38805]: pgmap v4116: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:09:06.454 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:09:06.577 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:09:06.627 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T15:09:06.627218+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T15:09:06.627 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T15:09:06.627 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6940 8 365281 365273 0.411164 0 - 0.0299513 2026-02-23T15:09:06.680 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:09:06.678+0000 7f0592176640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:09:06.680 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:09:06.679+0000 7f0592176640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:09:06.791 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:09:06.791 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":291,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:06:39.925670+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:05:38.714263+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:02:53.966248+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:04:16.232725+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:09:07.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:09:06 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2395715262' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:09:07.627 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6941 8 365281 365273 0.411105 0 - 0.0299513 2026-02-23T15:09:08.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:09:07 trial068 ceph-mon[38805]: pgmap v4117: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:09:08.627 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6942 8 365281 365273 0.411045 0 - 0.0299513 2026-02-23T15:09:09.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:09:08 trial068 ceph-mon[38805]: 5.e scrub starts 2026-02-23T15:09:09.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:09:08 trial068 ceph-mon[38805]: 5.e scrub ok 2026-02-23T15:09:09.627 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6943 8 365281 365273 0.410986 0 - 0.0299513 2026-02-23T15:09:09.828 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:09:09.947 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:09:10.049 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:09:10.047+0000 7f47285e5640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:09:10.049 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:09:10.047+0000 7f47285e5640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:09:10.158 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:09:10.158 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":291,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:06:39.925670+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:05:38.714263+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:02:53.966248+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:04:16.232725+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:09:10.158 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:09:09 trial068 ceph-mon[38805]: pgmap v4118: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:09:10.628 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6944 8 365281 365273 0.410927 0 - 0.0299513 2026-02-23T15:09:11.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:09:10 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1073988999' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:09:11.627 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6945 8 365281 365273 0.410868 0 - 0.0299513 2026-02-23T15:09:12.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:09:11 trial068 ceph-mon[38805]: pgmap v4119: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:09:12.628 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6946 8 365281 365273 0.410809 0 - 0.0299513 2026-02-23T15:09:13.195 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:09:13.315 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:09:13.421 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:09:13.420+0000 7f4cc5a79640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:09:13.422 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:09:13.420+0000 7f4cc5a79640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:09:13.531 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:09:13.531 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":292,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:09:12.988860+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:05:38.714263+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:08:36.021494+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:04:16.232725+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:09:13.628 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6947 8 365281 365273 0.41075 0 - 0.0299513 2026-02-23T15:09:14.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:09:13 trial068 ceph-mon[38805]: pgmap v4120: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:09:14.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:09:13 trial068 ceph-mon[38805]: osdmap e292: 4 total, 3 up, 4 in 2026-02-23T15:09:14.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:09:13 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/468107230' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:09:14.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:09:13 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:09:13] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:09:14.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:09:13 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:09:13.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:09:14.628 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6948 8 365281 365273 0.410691 0 - 0.0299513 2026-02-23T15:09:15.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:09:14 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2026-02-23T15:09:15.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:09:14 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-23T15:09:15.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:09:14 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2026-02-23T15:09:15.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:09:14 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T15:09:15.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:09:14 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2026-02-23T15:09:15.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:09:14 trial068 ceph-mon[38805]: osdmap e293: 4 total, 3 up, 4 in 2026-02-23T15:09:15.628 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6949 8 365281 365273 0.410631 0 - 0.0299513 2026-02-23T15:09:16.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:09:15 trial068 ceph-mon[38805]: pgmap v4122: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:09:16.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:09:15 trial068 ceph-mon[38805]: pgmap v4123: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:09:16.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:09:15 trial068 ceph-mon[38805]: pgmap v4124: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:09:16.586 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:09:16.628 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6950 8 365281 365273 0.410572 0 - 0.0299513 2026-02-23T15:09:16.704 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:09:16.804 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:09:16.802+0000 7ff08c9ab640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:09:16.804 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:09:16.802+0000 7ff08c9ab640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:09:16.914 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:09:16.914 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":293,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:09:14.947760+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:05:38.714263+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:08:36.021494+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:04:16.232725+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:09:17.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:09:16 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1263230559' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:09:17.628 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6951 8 365281 365273 0.410513 0 - 0.0299513 2026-02-23T15:09:18.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:09:17 trial068 ceph-mon[38805]: pgmap v4126: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:09:18.628 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6952 8 365281 365273 0.410454 0 - 0.0299513 2026-02-23T15:09:19.628 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6953 8 365281 365273 0.410395 0 - 0.0299513 2026-02-23T15:09:19.951 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:09:20.074 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:09:20.181 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:09:20.179+0000 7f7da0ff2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:09:20.181 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:09:20.179+0000 7f7da0ff2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:09:20.290 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:09:20.290 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":293,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:09:14.947760+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:05:38.714263+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:08:36.021494+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:04:16.232725+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:09:20.290 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:09:19 trial068 ceph-mon[38805]: pgmap v4127: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:09:20.291 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:09:19 trial068 ceph-mon[38805]: 5.7 scrub starts 2026-02-23T15:09:20.291 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:09:19 trial068 ceph-mon[38805]: 5.7 scrub ok 2026-02-23T15:09:20.628 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6954 8 365281 365273 0.410336 0 - 0.0299513 2026-02-23T15:09:21.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:09:21 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3079118192' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:09:21.628 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6955 8 365281 365273 0.410277 0 - 0.0299513 2026-02-23T15:09:22.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:09:22 trial068 ceph-mon[38805]: pgmap v4128: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:09:22.628 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6956 8 365281 365273 0.410218 0 - 0.0299513 2026-02-23T15:09:23.327 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:09:23.447 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:09:23.548 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:09:23.546+0000 7fa0cdf87640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:09:23.548 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:09:23.546+0000 7fa0cdf87640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:09:23.628 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6957 8 365281 365273 0.410159 0 - 0.0299513 2026-02-23T15:09:23.656 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:09:23.656 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":293,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:09:14.947760+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:05:38.714263+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:08:36.021494+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:04:16.232725+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:09:24.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:09:24 trial068 ceph-mon[38805]: pgmap v4129: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:09:24.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:09:24 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2567757151' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:09:24.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:09:23 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:09:23] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:09:24.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:09:23 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:09:23.974Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:09:24.628 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6958 8 365281 365273 0.4101 0 - 0.0299513 2026-02-23T15:09:25.629 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6959 8 365281 365273 0.410041 0 - 0.0299513 2026-02-23T15:09:26.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:09:26 trial068 ceph-mon[38805]: pgmap v4130: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:09:26.629 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T15:09:26.628700+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T15:09:26.629 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T15:09:26.629 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6960 8 365281 365273 0.409982 0 - 0.0299513 2026-02-23T15:09:26.693 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:09:26.813 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:09:26.911 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:09:26.909+0000 7fb599eb3640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:09:26.911 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:09:26.909+0000 7fb599eb3640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:09:27.020 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:09:27.020 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":293,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:09:14.947760+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:05:38.714263+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:08:36.021494+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:04:16.232725+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:09:27.629 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6961 8 365281 365273 0.409924 0 - 0.0299513 2026-02-23T15:09:28.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:09:28 trial068 ceph-mon[38805]: pgmap v4131: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:09:28.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:09:28 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/4279258937' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:09:28.629 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6962 8 365281 365273 0.409865 0 - 0.0299513 2026-02-23T15:09:29.629 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6963 8 365281 365273 0.409806 0 - 0.0299513 2026-02-23T15:09:30.073 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:09:30.193 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:09:30.292 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:09:30.290+0000 7eff2f01c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:09:30.292 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:09:30.290+0000 7eff2f01c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:09:30.292 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:09:30 trial068 ceph-mon[38805]: pgmap v4132: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:09:30.401 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:09:30.402 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":293,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:09:14.947760+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:05:38.714263+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:08:36.021494+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:04:16.232725+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:09:30.629 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6964 8 365281 365273 0.409747 0 - 0.0299513 2026-02-23T15:09:31.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:09:31 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1498929749' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:09:31.629 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6965 8 365281 365273 0.409688 0 - 0.0299513 2026-02-23T15:09:32.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:09:32 trial068 ceph-mon[38805]: pgmap v4133: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:09:32.629 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6966 8 365281 365273 0.409629 0 - 0.0299513 2026-02-23T15:09:33.440 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:09:33.559 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:09:33.629 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6967 8 365281 365273 0.409571 0 - 0.0299513 2026-02-23T15:09:33.656 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:09:33.655+0000 7fe4f5259640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:09:33.656 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:09:33.655+0000 7fe4f5259640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:09:33.765 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:09:33.766 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":293,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:09:14.947760+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:05:38.714263+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:08:36.021494+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:04:16.232725+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:09:34.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:09:34 trial068 ceph-mon[38805]: pgmap v4134: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:09:34.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:09:34 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/699030200' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:09:34.094 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:09:33 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:09:33] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:09:34.265 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:09:33 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:09:33.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:09:34.629 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6968 8 365281 365273 0.409512 0 - 0.0299513 2026-02-23T15:09:35.629 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6969 8 365281 365273 0.409453 0 - 0.0299513 2026-02-23T15:09:36.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:09:36 trial068 ceph-mon[38805]: pgmap v4135: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:09:36.630 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6970 8 365281 365273 0.409394 0 - 0.0299513 2026-02-23T15:09:36.804 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:09:36.923 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:09:37.024 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:09:37.023+0000 7f5d5c62e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:09:37.025 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:09:37.023+0000 7f5d5c62e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:09:37.135 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:09:37.135 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":293,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:09:14.947760+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:05:38.714263+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:08:36.021494+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:04:16.232725+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:09:37.630 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6971 8 365281 365273 0.409335 0 - 0.0299513 2026-02-23T15:09:38.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:09:38 trial068 ceph-mon[38805]: pgmap v4136: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:09:38.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:09:38 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T15:09:38.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:09:38 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T15:09:38.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:09:38 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1880259279' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:09:38.630 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6972 8 365281 365273 0.409277 0 - 0.0299513 2026-02-23T15:09:39.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:09:39 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T15:09:39.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:09:39 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T15:09:39.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:09:39 trial068 ceph-mon[38805]: pgmap v4137: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:09:39.630 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6973 8 365281 365273 0.409218 0 - 0.0299513 2026-02-23T15:09:40.172 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:09:40.288 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:09:40.390 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:09:40.388+0000 7f52baa65640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:09:40.390 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:09:40.388+0000 7f52baa65640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:09:40.500 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:09:40.500 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":293,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:09:14.947760+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:05:38.714263+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:08:36.021494+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:04:16.232725+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:09:40.630 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6974 8 365281 365273 0.409159 0 - 0.0299513 2026-02-23T15:09:41.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:09:40 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1870454492' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:09:41.630 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6975 8 365281 365273 0.409101 0 - 0.0299513 2026-02-23T15:09:42.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:09:41 trial068 ceph-mon[38805]: pgmap v4138: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:09:42.630 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6976 8 365281 365273 0.409042 0 - 0.0299513 2026-02-23T15:09:43.536 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:09:43.630 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6977 8 365281 365273 0.408983 0 - 0.0299513 2026-02-23T15:09:43.653 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:09:43.750 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:09:43.749+0000 7fd1840a0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:09:43.751 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:09:43.749+0000 7fd1840a0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:09:43.859 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:09:43.859 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":293,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:09:14.947760+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:05:38.714263+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:08:36.021494+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:04:16.232725+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:09:44.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:09:43 trial068 ceph-mon[38805]: pgmap v4139: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:09:44.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:09:43 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1547394131' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:09:44.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:09:43 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:09:43] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:09:44.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:09:43 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:09:43.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:09:44.630 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6978 8 365281 365273 0.408925 0 - 0.0299513 2026-02-23T15:09:45.630 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6979 8 365281 365273 0.408866 0 - 0.0299513 2026-02-23T15:09:46.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:09:45 trial068 ceph-mon[38805]: 5.8 scrub starts 2026-02-23T15:09:46.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:09:45 trial068 ceph-mon[38805]: 5.8 scrub ok 2026-02-23T15:09:46.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:09:45 trial068 ceph-mon[38805]: pgmap v4140: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:09:46.630 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T15:09:46.630330+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T15:09:46.630 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T15:09:46.631 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6980 8 365281 365273 0.408808 0 - 0.0299513 2026-02-23T15:09:46.897 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:09:47.015 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:09:47.117 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:09:47.115+0000 7f2f79232640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:09:47.117 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:09:47.115+0000 7f2f79232640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:09:47.227 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:09:47.227 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":293,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:09:14.947760+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:05:38.714263+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:08:36.021494+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:04:16.232725+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:09:47.227 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:09:46 trial068 ceph-mon[38805]: 5.a scrub starts 2026-02-23T15:09:47.227 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:09:46 trial068 ceph-mon[38805]: 5.a scrub ok 2026-02-23T15:09:47.631 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6981 8 365281 365273 0.408749 0 - 0.0299513 2026-02-23T15:09:48.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:09:47 trial068 ceph-mon[38805]: pgmap v4141: 17 pgs: 1 active+clean+scrubbing, 1 active+recovery_unfound+undersized+degraded, 4 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:09:48.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:09:47 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/385937730' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:09:48.630 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6982 8 365281 365273 0.408691 0 - 0.0299513 2026-02-23T15:09:49.631 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6983 8 365281 365273 0.408632 0 - 0.0299513 2026-02-23T15:09:50.264 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:09:50.288 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:09:49 trial068 ceph-mon[38805]: pgmap v4142: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 1 active+clean+scrubbing, 4 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:09:50.380 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:09:50.477 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:09:50.476+0000 7f014a294640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:09:50.478 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:09:50.476+0000 7f014a294640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:09:50.587 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:09:50.587 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":293,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:09:14.947760+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:05:38.714263+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:08:36.021494+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:04:16.232725+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:09:50.631 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6984 8 365281 365273 0.408574 0 - 0.0299513 2026-02-23T15:09:51.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:09:50 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3820764991' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:09:51.631 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6985 8 365281 365273 0.408515 0 - 0.0299513 2026-02-23T15:09:52.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:09:51 trial068 ceph-mon[38805]: pgmap v4143: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:09:52.631 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6986 8 365281 365273 0.408457 0 - 0.0299513 2026-02-23T15:09:53.625 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:09:53.631 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6987 8 365281 365273 0.408398 0 - 0.0299513 2026-02-23T15:09:53.744 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:09:53.842 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:09:53.840+0000 7f528d7a1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:09:53.842 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:09:53.840+0000 7f528d7a1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:09:53.949 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:09:53.950 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":293,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:09:14.947760+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:05:38.714263+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:08:36.021494+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:04:16.232725+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:09:54.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:09:53 trial068 ceph-mon[38805]: pgmap v4144: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:09:54.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:09:53 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1553613112' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:09:54.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:09:53 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:09:53] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:09:54.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:09:53 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:09:53.972Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:09:54.631 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6988 8 365281 365273 0.40834 0 - 0.0299513 2026-02-23T15:09:55.631 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6989 8 365281 365273 0.408281 0 - 0.0299513 2026-02-23T15:09:56.240 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:09:55 trial068 ceph-mon[38805]: pgmap v4145: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:09:56.631 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6990 8 365281 365273 0.408223 0 - 0.0299513 2026-02-23T15:09:56.987 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:09:57.105 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:09:57.215 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:09:57.214+0000 7f35adfc9640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:09:57.216 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:09:57.214+0000 7f35adfc9640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:09:57.329 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:09:57.329 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":293,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:09:14.947760+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:05:38.714263+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:08:36.021494+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:04:16.232725+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:09:57.632 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6991 8 365281 365273 0.408164 0 - 0.0299513 2026-02-23T15:09:58.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:09:57 trial068 ceph-mon[38805]: pgmap v4146: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:09:58.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:09:57 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3034662226' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:09:58.632 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6992 8 365281 365273 0.408106 0 - 0.0299513 2026-02-23T15:09:59.632 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6993 8 365281 365273 0.408048 0 - 0.0299513 2026-02-23T15:10:00.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:09:59 trial068 ceph-mon[38805]: 5.5 scrub starts 2026-02-23T15:10:00.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:09:59 trial068 ceph-mon[38805]: 5.5 scrub ok 2026-02-23T15:10:00.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:09:59 trial068 ceph-mon[38805]: pgmap v4147: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:10:00.367 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:10:00.486 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:10:00.592 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:10:00.590+0000 7f837bf02640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:10:00.592 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:10:00.590+0000 7f837bf02640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:10:00.632 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6994 8 365281 365273 0.407989 0 - 0.0299513 2026-02-23T15:10:00.699 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:10:00.699 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":293,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:09:14.947760+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:05:38.714263+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:08:36.021494+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:04:16.232725+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:10:01.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:10:00 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3340269047' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:10:01.632 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6995 8 365281 365273 0.407931 0 - 0.0299513 2026-02-23T15:10:02.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:10:01 trial068 ceph-mon[38805]: pgmap v4148: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:10:02.632 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6996 8 365281 365273 0.407873 0 - 0.0299513 2026-02-23T15:10:03.632 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6997 8 365281 365273 0.407814 0 - 0.0299513 2026-02-23T15:10:03.736 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:10:03.855 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:10:03.955 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:10:03.953+0000 7fc7da8d2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:10:03.955 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:10:03.953+0000 7fc7da8d2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:10:04.063 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:10:04.064 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":293,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:09:14.947760+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:05:38.714263+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:08:36.021494+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:04:16.232725+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:10:04.064 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:10:03 trial068 ceph-mon[38805]: pgmap v4149: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:10:04.064 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:10:03 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:10:03] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:10:04.265 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:10:03 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:10:03.972Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:10:04.632 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6998 8 365281 365273 0.407756 0 - 0.0299513 2026-02-23T15:10:05.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:10:04 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/276709170' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:10:05.632 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 6999 8 365281 365273 0.407698 0 - 0.0299513 2026-02-23T15:10:06.280 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:10:05 trial068 ceph-mon[38805]: pgmap v4150: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:10:06.632 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T15:10:06.632103+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T15:10:06.633 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T15:10:06.633 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7000 8 365281 365273 0.40764 0 - 0.0299513 2026-02-23T15:10:07.100 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:10:07.219 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:10:07.317 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:10:07.315+0000 7f8979ee2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:10:07.317 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:10:07.315+0000 7f8979ee2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:10:07.424 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:10:07.424 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":293,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:09:14.947760+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:05:38.714263+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:08:36.021494+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:04:16.232725+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:10:07.632 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7001 8 365281 365273 0.407581 0 - 0.0299513 2026-02-23T15:10:08.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:10:07 trial068 ceph-mon[38805]: pgmap v4151: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:10:08.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:10:07 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2419123485' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:10:08.632 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7002 8 365281 365273 0.407523 0 - 0.0299513 2026-02-23T15:10:09.632 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7003 8 365281 365273 0.407465 0 - 0.0299513 2026-02-23T15:10:10.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:10:09 trial068 ceph-mon[38805]: pgmap v4152: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:10:10.462 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:10:10.581 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:10:10.632 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7004 8 365281 365273 0.407407 0 - 0.0299513 2026-02-23T15:10:10.684 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:10:10.683+0000 7f56a3484640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:10:10.685 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:10:10.683+0000 7f56a3484640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:10:10.793 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:10:10.794 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":293,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:09:14.947760+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:05:38.714263+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:08:36.021494+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:04:16.232725+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:10:11.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:10:10 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/840680880' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:10:11.633 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7005 8 365281 365273 0.407349 0 - 0.0299513 2026-02-23T15:10:12.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:10:11 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T15:10:12.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:10:11 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T15:10:12.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:10:11 trial068 ceph-mon[38805]: pgmap v4153: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:10:12.633 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7006 8 365281 365273 0.407291 0 - 0.0299513 2026-02-23T15:10:13.633 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7007 8 365281 365273 0.407232 0 - 0.0299513 2026-02-23T15:10:13.832 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:10:13.950 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:10:14.052 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:10:14.050+0000 7ff995f1e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:10:14.052 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:10:14.051+0000 7ff995f1e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:10:14.163 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:10:14.163 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":293,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:09:14.947760+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:05:38.714263+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:08:36.021494+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:04:16.232725+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:10:14.163 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:10:13 trial068 ceph-mon[38805]: pgmap v4154: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:10:14.164 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:10:13 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:10:13] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:10:14.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:10:13 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:10:13.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:10:14.633 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7008 8 365281 365273 0.407174 0 - 0.0299513 2026-02-23T15:10:15.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:10:14 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2705928232' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:10:15.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:10:14 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2026-02-23T15:10:15.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:10:14 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-23T15:10:15.345 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:10:14 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2026-02-23T15:10:15.345 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:10:14 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T15:10:15.345 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:10:14 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2026-02-23T15:10:15.633 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7009 8 365281 365273 0.407116 0 - 0.0299513 2026-02-23T15:10:16.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:10:15 trial068 ceph-mon[38805]: pgmap v4155: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:10:16.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:10:15 trial068 ceph-mon[38805]: pgmap v4156: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:10:16.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:10:15 trial068 ceph-mon[38805]: pgmap v4157: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:10:16.633 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7010 8 365281 365273 0.407058 0 - 0.0299513 2026-02-23T15:10:17.201 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:10:17.319 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:10:17.420 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:10:17.418+0000 7f4f507f3640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:10:17.421 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:10:17.418+0000 7f4f507f3640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:10:17.531 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:10:17.531 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":293,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:09:14.947760+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:05:38.714263+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:08:36.021494+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:04:16.232725+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:10:17.633 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7011 8 365281 365273 0.407 0 - 0.0299513 2026-02-23T15:10:18.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:10:17 trial068 ceph-mon[38805]: pgmap v4158: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:10:18.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:10:17 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2780495813' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:10:18.633 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7012 8 365281 365273 0.406942 0 - 0.0299513 2026-02-23T15:10:19.633 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7013 8 365281 365273 0.406884 0 - 0.0299513 2026-02-23T15:10:20.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:10:19 trial068 ceph-mon[38805]: pgmap v4159: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:10:20.568 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:10:20.633 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7014 8 365281 365273 0.406826 0 - 0.0299513 2026-02-23T15:10:20.687 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:10:20.793 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:10:20.792+0000 7fa4a307d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:10:20.794 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:10:20.792+0000 7fa4a307d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:10:20.901 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:10:20.901 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":293,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:09:14.947760+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:05:38.714263+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:08:36.021494+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:04:16.232725+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:10:21.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:10:20 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3191856692' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:10:21.633 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7015 8 365281 365273 0.406768 0 - 0.0299513 2026-02-23T15:10:22.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:10:21 trial068 ceph-mon[38805]: pgmap v4160: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:10:22.633 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7016 8 365281 365273 0.40671 0 - 0.0299513 2026-02-23T15:10:23.634 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7017 8 365281 365273 0.406652 0 - 0.0299513 2026-02-23T15:10:23.939 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:10:24.061 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:10:24.162 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:10:24.160+0000 7fac6c219640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:10:24.163 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:10:24.161+0000 7fac6c219640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:10:24.272 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:10:24.273 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":293,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:09:14.947760+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:05:38.714263+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:08:36.021494+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:04:16.232725+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:10:24.273 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:10:23 trial068 ceph-mon[38805]: pgmap v4161: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:10:24.273 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:10:23 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:10:23] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:10:24.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:10:23 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:10:23.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:10:24.633 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7018 8 365281 365273 0.406594 0 - 0.0299513 2026-02-23T15:10:25.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:10:24 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/4154599690' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:10:25.634 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7019 8 365281 365273 0.406536 0 - 0.0299513 2026-02-23T15:10:26.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:10:25 trial068 ceph-mon[38805]: pgmap v4162: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:10:26.633 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T15:10:26.633529+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T15:10:26.634 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T15:10:26.634 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7020 8 365281 365273 0.406478 0 - 0.0299513 2026-02-23T15:10:27.309 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:10:27.333 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:10:26 trial068 ceph-mon[38805]: 5.7 scrub starts 2026-02-23T15:10:27.333 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:10:26 trial068 ceph-mon[38805]: 5.7 scrub ok 2026-02-23T15:10:27.428 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:10:27.525 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:10:27.523+0000 7fe3c6bd2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:10:27.525 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:10:27.524+0000 7fe3c6bd2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:10:27.632 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:10:27.633 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":293,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:09:14.947760+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:05:38.714263+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:08:36.021494+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:04:16.232725+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:10:27.633 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7021 8 365281 365273 0.40642 0 - 0.0299513 2026-02-23T15:10:28.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:10:27 trial068 ceph-mon[38805]: pgmap v4163: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:10:28.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:10:27 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2883149426' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:10:28.634 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7022 8 365281 365273 0.406363 0 - 0.0299513 2026-02-23T15:10:29.634 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7023 8 365281 365273 0.406305 0 - 0.0299513 2026-02-23T15:10:30.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:10:29 trial068 ceph-mon[38805]: pgmap v4164: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:10:30.634 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7024 8 365281 365273 0.406247 0 - 0.0299513 2026-02-23T15:10:30.670 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:10:30.786 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:10:30.885 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:10:30.883+0000 7f3c5e747640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:10:30.885 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:10:30.883+0000 7f3c5e747640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:10:30.994 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:10:30.994 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":294,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:10:30.958955+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:05:38.714263+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:08:36.021494+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:09:36.944612+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:10:31.634 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7025 8 365281 365273 0.406189 0 - 0.0299513 2026-02-23T15:10:32.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:10:31 trial068 ceph-mon[38805]: osdmap e294: 4 total, 3 up, 4 in 2026-02-23T15:10:32.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:10:31 trial068 ceph-mon[38805]: pgmap v4166: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:10:32.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:10:31 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/4068252872' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:10:32.634 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7026 8 365281 365273 0.406131 0 - 0.0299513 2026-02-23T15:10:33.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:10:33 trial068 ceph-mon[38805]: pgmap v4167: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:10:33.634 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7027 8 365281 365273 0.406073 0 - 0.0299513 2026-02-23T15:10:34.033 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:10:34.152 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:10:34.260 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:10:34.259+0000 7fc41c8f6640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:10:34.261 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:10:33 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:10:33] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:10:34.261 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:10:34.259+0000 7fc41c8f6640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:10:34.265 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:10:33 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:10:33.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:10:34.371 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:10:34.371 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":294,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:10:30.958955+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:05:38.714263+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:08:36.021494+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:09:36.944612+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:10:34.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:10:34 trial068 ceph-mon[38805]: 5.e scrub starts 2026-02-23T15:10:34.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:10:34 trial068 ceph-mon[38805]: 5.e scrub ok 2026-02-23T15:10:34.634 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7028 8 365281 365273 0.406016 0 - 0.0299513 2026-02-23T15:10:35.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:10:35 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3014686656' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:10:35.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:10:35 trial068 ceph-mon[38805]: osdmap e295: 4 total, 3 up, 4 in 2026-02-23T15:10:35.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:10:35 trial068 ceph-mon[38805]: pgmap v4169: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:10:35.634 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7029 8 365281 365273 0.405958 0 - 0.0299513 2026-02-23T15:10:36.634 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7030 8 365281 365273 0.4059 0 - 0.0299513 2026-02-23T15:10:37.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:10:37 trial068 ceph-mon[38805]: pgmap v4170: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:10:37.409 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:10:37.532 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:10:37.630 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:10:37.628+0000 7f87e7b2c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:10:37.630 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:10:37.628+0000 7f87e7b2c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:10:37.634 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7031 8 365281 365273 0.405842 0 - 0.0299513 2026-02-23T15:10:37.741 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:10:37.741 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":295,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:10:34.959286+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:05:38.714263+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:08:36.021494+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:09:36.944612+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:10:38.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:10:38 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/646960170' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:10:38.634 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7032 8 365281 365273 0.405785 0 - 0.0299513 2026-02-23T15:10:39.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:10:39 trial068 ceph-mon[38805]: pgmap v4171: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:10:39.634 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7033 8 365281 365273 0.405727 0 - 0.0299513 2026-02-23T15:10:40.634 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7034 8 365281 365273 0.405669 0 - 0.0299513 2026-02-23T15:10:40.778 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:10:40.898 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:10:40.999 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:10:40.997+0000 7f37ea843640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:10:40.999 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:10:40.997+0000 7f37ea843640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:10:41.109 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:10:41.109 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":295,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:10:34.959286+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:05:38.714263+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:08:36.021494+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:09:36.944612+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:10:41.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:10:41 trial068 ceph-mon[38805]: pgmap v4172: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:10:41.635 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7035 8 365281 365273 0.405612 0 - 0.0299513 2026-02-23T15:10:42.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:10:42 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1282990273' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:10:42.635 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7036 8 365281 365273 0.405554 0 - 0.0299513 2026-02-23T15:10:43.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:10:43 trial068 ceph-mon[38805]: pgmap v4173: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:10:43.635 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7037 8 365281 365273 0.405496 0 - 0.0299513 2026-02-23T15:10:44.146 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:10:44.265 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:10:44.288 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:10:43 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:10:43] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:10:44.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:10:43 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:10:43.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:10:44.366 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:10:44.364+0000 7fccb8129640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:10:44.366 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:10:44.364+0000 7fccb8129640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:10:44.475 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:10:44.476 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":295,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:10:34.959286+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:05:38.714263+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:08:36.021494+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:09:36.944612+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:10:44.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:10:44 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3470439825' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:10:44.635 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7038 8 365281 365273 0.405439 0 - 0.0299513 2026-02-23T15:10:45.635 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7039 8 365281 365273 0.405381 0 - 0.0299513 2026-02-23T15:10:46.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:10:45 trial068 ceph-mon[38805]: pgmap v4174: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:10:46.635 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T15:10:46.634828+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T15:10:46.635 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T15:10:46.635 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7040 8 365281 365273 0.405324 0 - 0.0299513 2026-02-23T15:10:47.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:10:47 trial068 ceph-mon[38805]: pgmap v4175: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:10:47.514 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:10:47.632 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:10:47.635 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7041 8 365281 365273 0.405266 0 - 0.0299513 2026-02-23T15:10:47.728 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:10:47.726+0000 7f3b5661f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:10:47.728 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:10:47.727+0000 7f3b5661f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:10:47.838 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:10:47.839 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":295,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:10:34.959286+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:05:38.714263+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:08:36.021494+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:09:36.944612+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:10:48.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:10:48 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3554012332' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:10:48.635 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7042 8 365281 365273 0.405208 0 - 0.0299513 2026-02-23T15:10:49.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:10:49 trial068 ceph-mon[38805]: pgmap v4176: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:10:49.635 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7043 8 365281 365273 0.405151 0 - 0.0299513 2026-02-23T15:10:50.635 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7044 8 365281 365273 0.405093 0 - 0.0299513 2026-02-23T15:10:50.875 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:10:50.992 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:10:51.095 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:10:51.093+0000 7fd8f0675640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:10:51.095 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:10:51.093+0000 7fd8f0675640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:10:51.206 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:10:51.206 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":295,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:10:34.959286+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:05:38.714263+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:08:36.021494+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:09:36.944612+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:10:51.206 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:10:51 trial068 ceph-mon[38805]: pgmap v4177: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:10:51.635 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7045 8 365281 365273 0.405036 0 - 0.0299513 2026-02-23T15:10:52.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:10:52 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2525307429' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:10:52.635 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7046 8 365281 365273 0.404978 0 - 0.0299513 2026-02-23T15:10:53.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:10:53 trial068 ceph-mon[38805]: pgmap v4178: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:10:53.635 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7047 8 365281 365273 0.404921 0 - 0.0299513 2026-02-23T15:10:54.245 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:10:54.269 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:10:54 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T15:10:54.269 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:10:54 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T15:10:54.269 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:10:53 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:10:53] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:10:54.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:10:53 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:10:53.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:10:54.361 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:10:54.461 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:10:54.459+0000 7f13aa2bc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:10:54.461 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:10:54.459+0000 7f13aa2bc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:10:54.571 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:10:54.571 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":295,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:10:34.959286+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:05:38.714263+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:08:36.021494+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:09:36.944612+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:10:54.635 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7048 8 365281 365273 0.404863 0 - 0.0299513 2026-02-23T15:10:55.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:10:55 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/320919036' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:10:55.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:10:55 trial068 ceph-mon[38805]: pgmap v4179: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:10:55.635 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7049 8 365281 365273 0.404806 0 - 0.0299513 2026-02-23T15:10:56.636 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7050 8 365281 365273 0.404749 0 - 0.0299513 2026-02-23T15:10:57.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:10:57 trial068 ceph-mon[38805]: pgmap v4180: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:10:57.610 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:10:57.635 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7051 8 365281 365273 0.404691 0 - 0.0299513 2026-02-23T15:10:57.727 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:10:57.824 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:10:57.822+0000 7fb35f2d7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:10:57.824 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:10:57.822+0000 7fb35f2d7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:10:57.934 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:10:57.934 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":295,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:10:34.959286+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:05:38.714263+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:08:36.021494+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:09:36.944612+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:10:58.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:10:58 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2273988927' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:10:58.636 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7052 8 365281 365273 0.404634 0 - 0.0299513 2026-02-23T15:10:59.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:10:59 trial068 ceph-mon[38805]: pgmap v4181: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:10:59.636 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7053 8 365281 365273 0.404576 0 - 0.0299513 2026-02-23T15:11:00.636 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7054 8 365281 365273 0.404519 0 - 0.0299513 2026-02-23T15:11:00.972 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:11:01.089 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:11:01.189 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:11:01.187+0000 7f87c3aee640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:11:01.189 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:11:01.187+0000 7f87c3aee640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:11:01.299 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:11:01.300 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":295,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:10:34.959286+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:05:38.714263+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:08:36.021494+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:09:36.944612+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:11:01.300 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:11:01 trial068 ceph-mon[38805]: 5.8 scrub starts 2026-02-23T15:11:01.300 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:11:01 trial068 ceph-mon[38805]: pgmap v4182: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:11:01.636 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7055 8 365281 365273 0.404462 0 - 0.0299513 2026-02-23T15:11:02.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:11:02 trial068 ceph-mon[38805]: 5.8 scrub ok 2026-02-23T15:11:02.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:11:02 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/415770596' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:11:02.636 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7056 8 365281 365273 0.404404 0 - 0.0299513 2026-02-23T15:11:03.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:11:03 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T15:11:03.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:11:03 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T15:11:03.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:11:03 trial068 ceph-mon[38805]: 5.a scrub starts 2026-02-23T15:11:03.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:11:03 trial068 ceph-mon[38805]: pgmap v4183: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:11:03.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:11:03 trial068 ceph-mon[38805]: 5.a scrub ok 2026-02-23T15:11:03.636 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7057 8 365281 365273 0.404347 0 - 0.0299513 2026-02-23T15:11:04.265 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:11:03 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:11:03.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:11:04.338 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:11:04.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:11:03 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:11:03] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:11:04.455 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:11:04.552 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:11:04.550+0000 7faf159a6640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:11:04.552 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:11:04.550+0000 7faf159a6640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:11:04.636 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7058 8 365281 365273 0.40429 0 - 0.0299513 2026-02-23T15:11:04.661 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:11:04.661 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":295,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:10:34.959286+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:05:38.714263+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:08:36.021494+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:09:36.944612+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:11:05.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:11:04 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2478181999' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:11:05.637 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7059 8 365281 365273 0.404233 0 - 0.0299513 2026-02-23T15:11:06.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:11:05 trial068 ceph-mon[38805]: pgmap v4184: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:11:06.636 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T15:11:06.636190+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T15:11:06.636 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T15:11:06.637 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7060 8 365281 365273 0.404175 0 - 0.0299513 2026-02-23T15:11:07.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:11:07 trial068 ceph-mon[38805]: pgmap v4185: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:11:07.636 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7061 8 365281 365273 0.404118 0 - 0.0299513 2026-02-23T15:11:07.699 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:11:07.817 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:11:07.919 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:11:07.917+0000 7fa760ea8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:11:07.919 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:11:07.918+0000 7fa760ea8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:11:08.027 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:11:08.027 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":295,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:10:34.959286+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:05:38.714263+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:08:36.021494+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:09:36.944612+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:11:08.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:11:08 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/537839799' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:11:08.636 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7062 8 365281 365273 0.404061 0 - 0.0299513 2026-02-23T15:11:09.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:11:09 trial068 ceph-mon[38805]: pgmap v4186: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:11:09.636 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7063 8 365281 365273 0.404004 0 - 0.0299513 2026-02-23T15:11:10.637 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7064 8 365281 365273 0.403946 0 - 0.0299513 2026-02-23T15:11:11.063 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:11:11.181 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:11:11.282 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:11:11.279+0000 7fcebe276640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:11:11.282 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:11:11.280+0000 7fcebe276640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:11:11.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:11:11 trial068 ceph-mon[38805]: pgmap v4187: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:11:11.389 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:11:11.390 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":295,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:10:34.959286+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:05:38.714263+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:08:36.021494+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:09:36.944612+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:11:11.637 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7065 8 365281 365273 0.403889 0 - 0.0299513 2026-02-23T15:11:12.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:11:12 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1858379601' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:11:12.636 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7066 8 365281 365273 0.403832 0 - 0.0299513 2026-02-23T15:11:13.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:11:13 trial068 ceph-mon[38805]: pgmap v4188: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:11:13.637 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7067 8 365281 365273 0.403775 0 - 0.0299513 2026-02-23T15:11:14.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:11:13 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:11:13] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:11:14.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:11:13 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:11:13.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:11:14.427 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:11:14.545 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:11:14.637 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7068 8 365281 365273 0.403718 0 - 0.0299513 2026-02-23T15:11:14.646 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:11:14.644+0000 7f186098d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:11:14.646 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:11:14.644+0000 7f186098d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:11:14.759 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:11:14.759 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":295,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:10:34.959286+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:05:38.714263+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:08:36.021494+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:09:36.944612+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:11:15.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:11:14 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/980808133' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:11:15.637 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7069 8 365281 365273 0.403661 0 - 0.0299513 2026-02-23T15:11:16.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:11:15 trial068 ceph-mon[38805]: pgmap v4189: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:11:16.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:11:15 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2026-02-23T15:11:16.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:11:15 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-23T15:11:16.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:11:15 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2026-02-23T15:11:16.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:11:15 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T15:11:16.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:11:15 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2026-02-23T15:11:16.637 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7070 8 365281 365273 0.403604 0 - 0.0299513 2026-02-23T15:11:17.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:11:16 trial068 ceph-mon[38805]: pgmap v4190: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:11:17.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:11:16 trial068 ceph-mon[38805]: pgmap v4191: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:11:17.637 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7071 8 365281 365273 0.403547 0 - 0.0299513 2026-02-23T15:11:17.796 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:11:17.915 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:11:18.018 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:11:18.016+0000 7fb29f89d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:11:18.018 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:11:18.016+0000 7fb29f89d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:11:18.127 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:11:18.127 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":295,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:10:34.959286+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:05:38.714263+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:08:36.021494+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:09:36.944612+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:11:18.637 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7072 8 365281 365273 0.40349 0 - 0.0299513 2026-02-23T15:11:19.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:11:18 trial068 ceph-mon[38805]: pgmap v4192: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:11:19.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:11:18 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/4208754374' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:11:19.637 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7073 8 365281 365273 0.403432 0 - 0.0299513 2026-02-23T15:11:20.637 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7074 8 365281 365273 0.403375 0 - 0.0299513 2026-02-23T15:11:21.165 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:11:21.281 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:11:21.302 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:11:20 trial068 ceph-mon[38805]: pgmap v4193: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:11:21.377 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:11:21.375+0000 7f3d46f76640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:11:21.377 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:11:21.376+0000 7f3d46f76640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:11:21.485 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:11:21.485 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":295,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:10:34.959286+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:05:38.714263+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:08:36.021494+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:09:36.944612+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:11:21.637 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7075 8 365281 365273 0.403318 0 - 0.0299513 2026-02-23T15:11:22.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:11:21 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2941180713' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:11:22.637 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7076 8 365281 365273 0.403261 0 - 0.0299513 2026-02-23T15:11:23.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:11:22 trial068 ceph-mon[38805]: pgmap v4194: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:11:23.637 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7077 8 365281 365273 0.403204 0 - 0.0299513 2026-02-23T15:11:24.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:11:23 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:11:23] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:11:24.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:11:23 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:11:23.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:11:24.522 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:11:24.637 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7078 8 365281 365273 0.403147 0 - 0.0299513 2026-02-23T15:11:24.639 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:11:24.744 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:11:24.742+0000 7ff9f67bd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:11:24.744 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:11:24.743+0000 7ff9f67bd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:11:24.853 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:11:24.854 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":295,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:10:34.959286+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:05:38.714263+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:08:36.021494+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:09:36.944612+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:11:25.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:11:24 trial068 ceph-mon[38805]: pgmap v4195: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:11:25.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:11:24 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/4223048717' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:11:25.638 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7079 8 365281 365273 0.403091 0 - 0.0299513 2026-02-23T15:11:26.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:11:25 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T15:11:26.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:11:25 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T15:11:26.638 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T15:11:26.637625+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T15:11:26.638 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T15:11:26.638 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7080 8 365281 365273 0.403034 0 - 0.0299513 2026-02-23T15:11:27.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:11:26 trial068 ceph-mon[38805]: pgmap v4196: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:11:27.638 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7081 8 365281 365273 0.402977 0 - 0.0299513 2026-02-23T15:11:27.892 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:11:28.012 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:11:28.114 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:11:28.113+0000 7f90b5eec640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:11:28.115 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:11:28.113+0000 7f90b5eec640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:11:28.223 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:11:28.224 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":295,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:10:34.959286+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:05:38.714263+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:08:36.021494+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:09:36.944612+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:11:28.638 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7082 8 365281 365273 0.40292 0 - 0.0299513 2026-02-23T15:11:29.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:11:28 trial068 ceph-mon[38805]: pgmap v4197: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:11:29.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:11:28 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3061535074' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:11:29.638 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7083 8 365281 365273 0.402863 0 - 0.0299513 2026-02-23T15:11:30.638 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7084 8 365281 365273 0.402806 0 - 0.0299513 2026-02-23T15:11:31.262 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:11:31.288 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:11:30 trial068 ceph-mon[38805]: pgmap v4198: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:11:31.381 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:11:31.488 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:11:31.486+0000 7f85449cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:11:31.488 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:11:31.487+0000 7f85449cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:11:31.599 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:11:31.600 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":295,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:10:34.959286+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:05:38.714263+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:08:36.021494+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:09:36.944612+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:11:31.638 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7085 8 365281 365273 0.402749 0 - 0.0299513 2026-02-23T15:11:32.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:11:31 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2579179638' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:11:32.638 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7086 8 365281 365273 0.402692 0 - 0.0299513 2026-02-23T15:11:33.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:11:32 trial068 ceph-mon[38805]: pgmap v4199: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:11:33.638 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7087 8 365281 365273 0.402636 0 - 0.0299513 2026-02-23T15:11:34.266 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:11:33 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:11:33.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:11:34.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:11:33 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:11:33] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:11:34.638 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:11:34.639 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7088 8 365281 365273 0.402579 0 - 0.0299513 2026-02-23T15:11:34.757 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:11:34.856 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:11:34.854+0000 7f177d9c7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:11:34.856 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:11:34.855+0000 7f177d9c7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:11:34.966 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:11:34.966 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":295,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:10:34.959286+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:05:38.714263+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:08:36.021494+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:09:36.944612+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:11:35.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:11:34 trial068 ceph-mon[38805]: pgmap v4200: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:11:35.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:11:34 trial068 ceph-mon[38805]: 5.5 scrub starts 2026-02-23T15:11:35.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:11:34 trial068 ceph-mon[38805]: 5.5 scrub ok 2026-02-23T15:11:35.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:11:34 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/288853026' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:11:35.638 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7089 8 365281 365273 0.402522 0 - 0.0299513 2026-02-23T15:11:36.638 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7090 8 365281 365273 0.402465 0 - 0.0299513 2026-02-23T15:11:37.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:11:36 trial068 ceph-mon[38805]: pgmap v4201: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:11:37.638 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7091 8 365281 365273 0.402408 0 - 0.0299513 2026-02-23T15:11:38.005 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:11:38.123 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:11:38.228 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:11:38.226+0000 7f79f46ce640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:11:38.228 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:11:38.227+0000 7f79f46ce640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:11:38.338 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:11:38.338 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":296,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:11:36.985886+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:11:24.652666+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:08:36.021494+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:09:36.944612+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:11:38.338 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:11:37 trial068 ceph-mon[38805]: osdmap e296: 4 total, 3 up, 4 in 2026-02-23T15:11:38.638 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7092 8 365281 365273 0.402352 0 - 0.0299513 2026-02-23T15:11:39.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:11:38 trial068 ceph-mon[38805]: pgmap v4203: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:11:39.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:11:38 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2189104400' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:11:39.638 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7093 8 365281 365273 0.402295 0 - 0.0299513 2026-02-23T15:11:40.638 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7094 8 365281 365273 0.402238 0 - 0.0299513 2026-02-23T15:11:41.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:11:40 trial068 ceph-mon[38805]: pgmap v4204: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:11:41.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:11:40 trial068 ceph-mon[38805]: osdmap e297: 4 total, 3 up, 4 in 2026-02-23T15:11:41.375 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:11:41.490 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:11:41.588 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:11:41.586+0000 7f81a0264640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:11:41.588 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:11:41.586+0000 7f81a0264640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:11:41.638 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7095 8 365281 365273 0.402182 0 - 0.0299513 2026-02-23T15:11:41.698 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:11:41.698 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":297,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:11:39.967396+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:11:24.652666+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:08:36.021494+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:09:36.944612+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:11:42.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:11:41 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2691353785' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:11:42.639 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7096 8 365281 365273 0.402125 0 - 0.0299513 2026-02-23T15:11:43.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:11:42 trial068 ceph-mon[38805]: pgmap v4206: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:11:43.639 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7097 8 365281 365273 0.402068 0 - 0.0299513 2026-02-23T15:11:44.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:11:43 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:11:43] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:11:44.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:11:43 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:11:43.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:11:44.639 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7098 8 365281 365273 0.402012 0 - 0.0299513 2026-02-23T15:11:44.736 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:11:44.854 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:11:44.960 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:11:44.958+0000 7f59a2ef6640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:11:44.960 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:11:44.958+0000 7f59a2ef6640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:11:45.070 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:11:45.070 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":297,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:11:39.967396+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:11:24.652666+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:08:36.021494+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:09:36.944612+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:11:45.070 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:11:44 trial068 ceph-mon[38805]: pgmap v4207: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:11:45.639 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7099 8 365281 365273 0.401955 0 - 0.0299513 2026-02-23T15:11:46.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:11:45 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2695005427' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:11:46.639 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T15:11:46.638995+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T15:11:46.639 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T15:11:46.639 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7100 8 365281 365273 0.401898 0 - 0.0299513 2026-02-23T15:11:47.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:11:46 trial068 ceph-mon[38805]: pgmap v4208: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:11:47.639 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7101 8 365281 365273 0.401842 0 - 0.0299513 2026-02-23T15:11:48.109 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:11:48.227 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:11:48.324 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:11:48.323+0000 7fc0a1dd6640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:11:48.325 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:11:48.323+0000 7fc0a1dd6640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:11:48.434 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:11:48.435 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":297,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:11:39.967396+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:11:24.652666+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:08:36.021494+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:09:36.944612+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:11:48.639 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7102 8 365281 365273 0.401785 0 - 0.0299513 2026-02-23T15:11:49.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:11:48 trial068 ceph-mon[38805]: pgmap v4209: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:11:49.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:11:48 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1664375761' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:11:49.639 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7103 8 365281 365273 0.401729 0 - 0.0299513 2026-02-23T15:11:50.639 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7104 8 365281 365273 0.401672 0 - 0.0299513 2026-02-23T15:11:51.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:11:50 trial068 ceph-mon[38805]: pgmap v4210: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:11:51.475 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:11:51.594 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:11:51.639 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7105 8 365281 365273 0.401615 0 - 0.0299513 2026-02-23T15:11:51.696 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:11:51.694+0000 7fbd229c9640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:11:51.696 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:11:51.695+0000 7fbd229c9640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:11:51.805 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:11:51.805 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":297,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:11:39.967396+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:11:24.652666+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:08:36.021494+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:09:36.944612+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:11:52.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:11:51 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/427234398' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:11:52.640 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7106 8 365281 365273 0.401559 0 - 0.0299513 2026-02-23T15:11:53.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:11:52 trial068 ceph-mon[38805]: pgmap v4211: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:11:53.640 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7107 8 365281 365273 0.401502 0 - 0.0299513 2026-02-23T15:11:54.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:11:53 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:11:53] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:11:54.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:11:53 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:11:53.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:11:54.640 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7108 8 365281 365273 0.401446 0 - 0.0299513 2026-02-23T15:11:54.844 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:11:54.960 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:11:55.066 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:11:55.064+0000 7f12f466f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:11:55.066 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:11:55.064+0000 7f12f466f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:11:55.175 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:11:55.175 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":297,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:11:39.967396+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:11:24.652666+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:08:36.021494+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:09:36.944612+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:11:55.175 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:11:54 trial068 ceph-mon[38805]: pgmap v4212: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:11:55.640 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7109 8 365281 365273 0.401389 0 - 0.0299513 2026-02-23T15:11:56.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:11:55 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/571128191' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:11:56.640 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7110 8 365281 365273 0.401333 0 - 0.0299513 2026-02-23T15:11:57.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:11:56 trial068 ceph-mon[38805]: pgmap v4213: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:11:57.640 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7111 8 365281 365273 0.401277 0 - 0.0299513 2026-02-23T15:11:58.212 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:11:58.333 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:11:58.434 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:11:58.433+0000 7fa0c43f4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:11:58.435 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:11:58.433+0000 7fa0c43f4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:11:58.545 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:11:58.545 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":297,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:11:39.967396+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:11:24.652666+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:08:36.021494+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:09:36.944612+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:11:58.640 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7112 8 365281 365273 0.40122 0 - 0.0299513 2026-02-23T15:11:59.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:11:58 trial068 ceph-mon[38805]: pgmap v4214: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:11:59.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:11:58 trial068 ceph-mon[38805]: 5.7 scrub starts 2026-02-23T15:11:59.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:11:58 trial068 ceph-mon[38805]: 5.7 scrub ok 2026-02-23T15:11:59.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:11:58 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1078162951' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:11:59.640 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7113 8 365281 365273 0.401164 0 - 0.0299513 2026-02-23T15:12:00.640 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7114 8 365281 365273 0.401107 0 - 0.0299513 2026-02-23T15:12:01.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:12:00 trial068 ceph-mon[38805]: pgmap v4215: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:12:01.583 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:12:01.640 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7115 8 365281 365273 0.401051 0 - 0.0299513 2026-02-23T15:12:01.701 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:12:01.801 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:12:01.800+0000 7f0e0ec71640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:12:01.802 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:12:01.800+0000 7f0e0ec71640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:12:01.913 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:12:01.913 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":297,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:11:39.967396+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:11:24.652666+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:08:36.021494+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:09:36.944612+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:12:02.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:12:01 trial068 ceph-mon[38805]: 5.e scrub starts 2026-02-23T15:12:02.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:12:01 trial068 ceph-mon[38805]: 5.e scrub ok 2026-02-23T15:12:02.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:12:01 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/339529824' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:12:02.640 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7116 8 365281 365273 0.400995 0 - 0.0299513 2026-02-23T15:12:03.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:12:02 trial068 ceph-mon[38805]: pgmap v4216: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:12:03.640 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7117 8 365281 365273 0.400938 0 - 0.0299513 2026-02-23T15:12:04.265 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:12:03 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:12:03.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:12:04.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:12:03 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:12:03] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:12:04.641 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7118 8 365281 365273 0.400882 0 - 0.0299513 2026-02-23T15:12:04.949 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:12:05.073 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:12:05.174 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:12:05.172+0000 7f8e34d49640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:12:05.174 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:12:05.172+0000 7f8e34d49640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:12:05.284 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:12:05.285 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":297,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:11:39.967396+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:11:24.652666+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:08:36.021494+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:09:36.944612+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:12:05.285 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:12:04 trial068 ceph-mon[38805]: pgmap v4217: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:12:05.641 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7119 8 365281 365273 0.400826 0 - 0.0299513 2026-02-23T15:12:06.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:12:05 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2708524959' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:12:06.641 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T15:12:06.640732+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T15:12:06.641 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T15:12:06.641 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7120 8 365281 365273 0.400769 0 - 0.0299513 2026-02-23T15:12:07.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:12:06 trial068 ceph-mon[38805]: pgmap v4218: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:12:07.641 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7121 8 365281 365273 0.400713 0 - 0.0299513 2026-02-23T15:12:08.322 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:12:08.441 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:12:08.543 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:12:08.541+0000 7fa1c37ab640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:12:08.543 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:12:08.541+0000 7fa1c37ab640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:12:08.641 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7122 8 365281 365273 0.400657 0 - 0.0299513 2026-02-23T15:12:08.654 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:12:08.654 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":297,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:11:39.967396+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:11:24.652666+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:08:36.021494+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:09:36.944612+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:12:09.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:12:08 trial068 ceph-mon[38805]: pgmap v4219: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:12:09.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:12:08 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3227452250' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:12:09.641 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7123 8 365281 365273 0.400601 0 - 0.0299513 2026-02-23T15:12:10.641 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7124 8 365281 365273 0.400544 0 - 0.0299513 2026-02-23T15:12:11.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:12:10 trial068 ceph-mon[38805]: pgmap v4220: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:12:11.641 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7125 8 365281 365273 0.400488 0 - 0.0299513 2026-02-23T15:12:11.692 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:12:11.810 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:12:11.907 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:12:11.905+0000 7f2ea329f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:12:11.907 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:12:11.905+0000 7f2ea329f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:12:12.016 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:12:12.017 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":297,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:11:39.967396+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:11:24.652666+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:08:36.021494+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:09:36.944612+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:12:12.641 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7126 8 365281 365273 0.400432 0 - 0.0299513 2026-02-23T15:12:13.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:12:13 trial068 ceph-mon[38805]: pgmap v4221: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:12:13.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:12:13 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/788937898' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:12:13.641 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7127 8 365281 365273 0.400376 0 - 0.0299513 2026-02-23T15:12:14.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:12:13 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:12:13] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:12:14.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:12:13 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:12:13.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:12:14.641 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7128 8 365281 365273 0.40032 0 - 0.0299513 2026-02-23T15:12:15.054 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:12:15.174 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:12:15.274 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:12:15.273+0000 7f38be37a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:12:15.275 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:12:15 trial068 ceph-mon[38805]: pgmap v4222: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:12:15.275 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:12:15.273+0000 7f38be37a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:12:15.385 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:12:15.386 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":297,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:11:39.967396+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:11:24.652666+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:08:36.021494+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:09:36.944612+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:12:15.641 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7129 8 365281 365273 0.400263 0 - 0.0299513 2026-02-23T15:12:16.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:12:16 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2026-02-23T15:12:16.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:12:16 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/4083518797' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:12:16.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:12:16 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-23T15:12:16.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:12:16 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2026-02-23T15:12:16.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:12:16 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T15:12:16.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:12:16 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2026-02-23T15:12:16.641 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7130 8 365281 365273 0.400207 0 - 0.0299513 2026-02-23T15:12:17.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:12:17 trial068 ceph-mon[38805]: pgmap v4223: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:12:17.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:12:17 trial068 ceph-mon[38805]: pgmap v4224: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:12:17.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:12:17 trial068 ceph-mon[38805]: pgmap v4225: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:12:17.642 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7131 8 365281 365273 0.400151 0 - 0.0299513 2026-02-23T15:12:18.427 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:12:18.546 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:12:18.642 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7132 8 365281 365273 0.400095 0 - 0.0299513 2026-02-23T15:12:18.652 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:12:18.651+0000 7f21edcf6640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:12:18.652 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:12:18.651+0000 7f21edcf6640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:12:18.761 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:12:18.761 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":297,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:11:39.967396+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:11:24.652666+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:08:36.021494+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:09:36.944612+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:12:19.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:12:19 trial068 ceph-mon[38805]: pgmap v4226: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:12:19.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:12:19 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3660625127' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:12:19.642 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7133 8 365281 365273 0.400039 0 - 0.0299513 2026-02-23T15:12:20.642 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7134 8 365281 365273 0.399983 0 - 0.0299513 2026-02-23T15:12:21.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:12:21 trial068 ceph-mon[38805]: pgmap v4227: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:12:21.642 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7135 8 365281 365273 0.399927 0 - 0.0299513 2026-02-23T15:12:21.799 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:12:21.917 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:12:22.019 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:12:22.017+0000 7fb8cc92c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:12:22.019 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:12:22.018+0000 7fb8cc92c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:12:22.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:12:22 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T15:12:22.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:12:22 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T15:12:22.131 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:12:22.131 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":297,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:11:39.967396+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:11:24.652666+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:08:36.021494+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:09:36.944612+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:12:22.642 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7136 8 365281 365273 0.399871 0 - 0.0299513 2026-02-23T15:12:23.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:12:23 trial068 ceph-mon[38805]: pgmap v4228: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:12:23.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:12:23 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T15:12:23.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:12:23 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T15:12:23.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:12:23 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1766175904' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:12:23.642 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7137 8 365281 365273 0.399815 0 - 0.0299513 2026-02-23T15:12:24.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:12:23 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:12:23] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:12:24.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:12:23 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:12:23.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:12:24.642 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7138 8 365281 365273 0.399759 0 - 0.0299513 2026-02-23T15:12:25.169 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:12:25.287 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:12:25.308 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:12:25 trial068 ceph-mon[38805]: pgmap v4229: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:12:25.391 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:12:25.389+0000 7f75806c5640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:12:25.391 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:12:25.390+0000 7f75806c5640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:12:25.500 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:12:25.501 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":297,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:11:39.967396+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:11:24.652666+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:08:36.021494+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:09:36.944612+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:12:25.642 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7139 8 365281 365273 0.399703 0 - 0.0299513 2026-02-23T15:12:26.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:12:26 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3197975227' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:12:26.642 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T15:12:26.642223+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T15:12:26.642 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T15:12:26.643 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7140 8 365281 365273 0.399647 0 - 0.0299513 2026-02-23T15:12:27.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:12:27 trial068 ceph-mon[38805]: pgmap v4230: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:12:27.642 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7141 8 365281 365273 0.399591 0 - 0.0299513 2026-02-23T15:12:28.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:12:28 trial068 ceph-mon[38805]: 5.a scrub starts 2026-02-23T15:12:28.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:12:28 trial068 ceph-mon[38805]: 5.a scrub ok 2026-02-23T15:12:28.538 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:12:28.642 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7142 8 365281 365273 0.399535 0 - 0.0299513 2026-02-23T15:12:28.659 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:12:28.754 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:12:28.753+0000 7ff20dda9640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:12:28.755 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:12:28.753+0000 7ff20dda9640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:12:28.861 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:12:28.862 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":297,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:11:39.967396+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:11:24.652666+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:08:36.021494+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:09:36.944612+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:12:29.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:12:29 trial068 ceph-mon[38805]: 5.8 scrub starts 2026-02-23T15:12:29.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:12:29 trial068 ceph-mon[38805]: 5.8 scrub ok 2026-02-23T15:12:29.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:12:29 trial068 ceph-mon[38805]: pgmap v4231: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:12:29.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:12:29 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1742774787' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:12:29.642 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7143 8 365281 365273 0.399479 0 - 0.0299513 2026-02-23T15:12:30.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:12:30 trial068 ceph-mon[38805]: pgmap v4232: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:12:30.643 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7144 8 365281 365273 0.399423 0 - 0.0299513 2026-02-23T15:12:31.643 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7145 8 365281 365273 0.399367 0 - 0.0299513 2026-02-23T15:12:31.899 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:12:32.016 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:12:32.116 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:12:32.115+0000 7fd0e01bf640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:12:32.117 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:12:32.115+0000 7fd0e01bf640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:12:32.227 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:12:32.227 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":297,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:11:39.967396+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:11:24.652666+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:08:36.021494+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:09:36.944612+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:12:32.643 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7146 8 365281 365273 0.399311 0 - 0.0299513 2026-02-23T15:12:32.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:12:32 trial068 ceph-mon[38805]: pgmap v4233: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:12:32.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:12:32 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/109692021' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:12:33.643 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7147 8 365281 365273 0.399255 0 - 0.0299513 2026-02-23T15:12:34.265 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:12:33 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:12:33.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:12:34.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:12:33 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:12:33] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:12:34.643 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7148 8 365281 365273 0.399199 0 - 0.0299513 2026-02-23T15:12:34.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:12:34 trial068 ceph-mon[38805]: pgmap v4234: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:12:35.265 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:12:35.382 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:12:35.490 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:12:35.488+0000 7fb87fa80640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:12:35.490 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:12:35.488+0000 7fb87fa80640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:12:35.598 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:12:35.598 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":297,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:11:39.967396+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:11:24.652666+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:08:36.021494+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:09:36.944612+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:12:35.643 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7149 8 365281 365273 0.399144 0 - 0.0299513 2026-02-23T15:12:36.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:12:35 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1646296011' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:12:36.643 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7150 8 365281 365273 0.399088 0 - 0.0299513 2026-02-23T15:12:37.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:12:36 trial068 ceph-mon[38805]: pgmap v4235: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:12:37.643 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7151 8 365281 365273 0.399032 0 - 0.0299513 2026-02-23T15:12:38.637 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:12:38.643 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7152 8 365281 365273 0.398976 0 - 0.0299513 2026-02-23T15:12:38.753 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:12:38.856 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:12:38.854+0000 7f56db3b3640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:12:38.856 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:12:38.854+0000 7f56db3b3640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:12:38.965 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:12:38.965 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":297,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:11:39.967396+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:11:24.652666+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:08:36.021494+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:09:36.944612+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:12:39.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:12:38 trial068 ceph-mon[38805]: pgmap v4236: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:12:39.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:12:38 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2797587868' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:12:39.643 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7153 8 365281 365273 0.39892 0 - 0.0299513 2026-02-23T15:12:40.643 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7154 8 365281 365273 0.398865 0 - 0.0299513 2026-02-23T15:12:41.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:12:40 trial068 ceph-mon[38805]: pgmap v4237: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:12:41.643 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7155 8 365281 365273 0.398809 0 - 0.0299513 2026-02-23T15:12:42.001 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:12:42.119 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:12:42.214 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:12:42.212+0000 7fe7b491e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:12:42.214 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:12:42.213+0000 7fe7b491e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:12:42.322 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:12:42.323 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":297,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:11:39.967396+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:11:24.652666+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:08:36.021494+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:09:36.944612+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:12:42.643 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7156 8 365281 365273 0.398753 0 - 0.0299513 2026-02-23T15:12:43.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:12:42 trial068 ceph-mon[38805]: pgmap v4238: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:12:43.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:12:42 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3893902251' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:12:43.644 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7157 8 365281 365273 0.398697 0 - 0.0299513 2026-02-23T15:12:44.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:12:43 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:12:43] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:12:44.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:12:43 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:12:43.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:12:44.644 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7158 8 365281 365273 0.398642 0 - 0.0299513 2026-02-23T15:12:45.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:12:44 trial068 ceph-mon[38805]: pgmap v4239: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:12:45.374 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:12:45.491 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:12:45.588 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:12:45.586+0000 7f516340a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:12:45.588 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:12:45.586+0000 7f516340a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:12:45.643 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7159 8 365281 365273 0.398586 0 - 0.0299513 2026-02-23T15:12:45.696 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:12:45.696 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":297,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:11:39.967396+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:11:24.652666+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:08:36.021494+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:09:36.944612+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:12:46.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:12:45 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T15:12:46.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:12:45 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T15:12:46.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:12:45 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3057564552' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:12:46.644 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T15:12:46.643710+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T15:12:46.644 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T15:12:46.644 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7160 8 365281 365273 0.39853 0 - 0.0299513 2026-02-23T15:12:47.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:12:46 trial068 ceph-mon[38805]: pgmap v4240: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:12:47.644 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7161 8 365281 365273 0.398475 0 - 0.0299513 2026-02-23T15:12:48.244 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:12:47 trial068 ceph-mon[38805]: 5.5 scrub starts 2026-02-23T15:12:48.244 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:12:47 trial068 ceph-mon[38805]: 5.5 scrub ok 2026-02-23T15:12:48.644 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7162 8 365281 365273 0.398419 0 - 0.0299513 2026-02-23T15:12:48.732 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:12:48.851 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:12:48.946 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:12:48.945+0000 7fb26f9ed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:12:48.947 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:12:48.945+0000 7fb26f9ed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:12:49.059 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:12:49.060 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":297,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:11:39.967396+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:11:24.652666+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:08:36.021494+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:09:36.944612+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:12:49.060 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:12:48 trial068 ceph-mon[38805]: pgmap v4241: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:12:49.644 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7163 8 365281 365273 0.398364 0 - 0.0299513 2026-02-23T15:12:50.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:12:49 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3741562766' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:12:50.644 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7164 8 365281 365273 0.398308 0 - 0.0299513 2026-02-23T15:12:51.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:12:50 trial068 ceph-mon[38805]: pgmap v4242: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:12:51.644 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7165 8 365281 365273 0.398252 0 - 0.0299513 2026-02-23T15:12:52.098 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:12:52.216 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:12:52.322 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:12:52.320+0000 7f9f88180640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:12:52.322 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:12:52.320+0000 7f9f88180640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:12:52.429 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:12:52.429 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":297,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:11:39.967396+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:11:24.652666+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:08:36.021494+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:09:36.944612+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:12:52.644 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7166 8 365281 365273 0.398197 0 - 0.0299513 2026-02-23T15:12:53.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:12:52 trial068 ceph-mon[38805]: pgmap v4243: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:12:53.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:12:52 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3874919756' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:12:53.644 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7167 8 365281 365273 0.398141 0 - 0.0299513 2026-02-23T15:12:54.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:12:53 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:12:53] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:12:54.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:12:53 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:12:53.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:12:54.644 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7168 8 365281 365273 0.398086 0 - 0.0299513 2026-02-23T15:12:55.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:12:54 trial068 ceph-mon[38805]: pgmap v4244: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:12:55.467 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:12:55.585 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:12:55.644 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7169 8 365281 365273 0.39803 0 - 0.0299513 2026-02-23T15:12:55.690 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:12:55.688+0000 7ff1cf36d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:12:55.690 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:12:55.688+0000 7ff1cf36d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:12:55.800 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:12:55.800 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":297,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:11:39.967396+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:11:24.652666+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:08:36.021494+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:09:36.944612+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:12:56.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:12:55 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/160021286' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:12:56.644 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7170 8 365281 365273 0.397975 0 - 0.0299513 2026-02-23T15:12:57.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:12:56 trial068 ceph-mon[38805]: pgmap v4245: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:12:57.644 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7171 8 365281 365273 0.397919 0 - 0.0299513 2026-02-23T15:12:58.644 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7172 8 365281 365273 0.397864 0 - 0.0299513 2026-02-23T15:12:58.838 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:12:58.956 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:12:59.052 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:12:59.050+0000 7f77639a9640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:12:59.052 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:12:59.050+0000 7f77639a9640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:12:59.164 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:12:59.164 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":297,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:11:39.967396+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:11:24.652666+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:08:36.021494+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:09:36.944612+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:12:59.165 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:12:58 trial068 ceph-mon[38805]: pgmap v4246: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:12:59.645 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7173 8 365281 365273 0.397808 0 - 0.0299513 2026-02-23T15:13:00.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:12:59 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3976326253' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:13:00.645 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7174 8 365281 365273 0.397753 0 - 0.0299513 2026-02-23T15:13:01.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:13:01 trial068 ceph-mon[38805]: pgmap v4247: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:13:01.645 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7175 8 365281 365273 0.397697 0 - 0.0299513 2026-02-23T15:13:02.203 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:13:02.319 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:13:02.420 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:13:02.419+0000 7fb39e996640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:13:02.421 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:13:02.419+0000 7fb39e996640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:13:02.532 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:13:02.532 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":297,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:11:39.967396+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:11:24.652666+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:08:36.021494+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:09:36.944612+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:13:02.645 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7176 8 365281 365273 0.397642 0 - 0.0299513 2026-02-23T15:13:03.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:13:03 trial068 ceph-mon[38805]: pgmap v4248: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:13:03.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:13:03 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2768781695' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:13:03.645 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7177 8 365281 365273 0.397586 0 - 0.0299513 2026-02-23T15:13:04.264 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:13:03 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:13:03.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:13:04.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:13:03 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:13:03] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:13:04.645 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7178 8 365281 365273 0.397531 0 - 0.0299513 2026-02-23T15:13:05.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:13:05 trial068 ceph-mon[38805]: 5.7 scrub starts 2026-02-23T15:13:05.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:13:05 trial068 ceph-mon[38805]: pgmap v4249: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:13:05.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:13:05 trial068 ceph-mon[38805]: 5.7 scrub ok 2026-02-23T15:13:05.570 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:13:05.645 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7179 8 365281 365273 0.397476 0 - 0.0299513 2026-02-23T15:13:05.690 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:13:05.790 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:13:05.788+0000 7f1d9ef45640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:13:05.790 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:13:05.788+0000 7f1d9ef45640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:13:05.898 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:13:05.898 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":297,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:11:39.967396+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:11:24.652666+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:08:36.021494+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:09:36.944612+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:13:06.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:13:06 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2314390622' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:13:06.645 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T15:13:06.645166+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T15:13:06.645 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T15:13:06.645 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7180 8 365281 365273 0.39742 0 - 0.0299513 2026-02-23T15:13:07.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:13:07 trial068 ceph-mon[38805]: pgmap v4250: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:13:07.645 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7181 8 365281 365273 0.397365 0 - 0.0299513 2026-02-23T15:13:08.645 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7182 8 365281 365273 0.39731 0 - 0.0299513 2026-02-23T15:13:08.934 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:13:09.053 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:13:09.153 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:13:09.151+0000 7fd4b3fe5640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:13:09.153 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:13:09.151+0000 7fd4b3fe5640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:13:09.260 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:13:09.261 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":297,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:11:39.967396+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:11:24.652666+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:08:36.021494+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:09:36.944612+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:13:09.261 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:13:09 trial068 ceph-mon[38805]: pgmap v4251: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:13:09.645 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7183 8 365281 365273 0.397254 0 - 0.0299513 2026-02-23T15:13:10.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:13:10 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/617656686' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:13:10.645 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7184 8 365281 365273 0.397199 0 - 0.0299513 2026-02-23T15:13:11.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:13:11 trial068 ceph-mon[38805]: pgmap v4252: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:13:11.645 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7185 8 365281 365273 0.397144 0 - 0.0299513 2026-02-23T15:13:12.298 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:13:12.414 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:13:12.510 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:13:12.509+0000 7fc5929cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:13:12.511 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:13:12.509+0000 7fc5929cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:13:12.620 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:13:12.620 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":297,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:11:39.967396+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:11:24.652666+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:08:36.021494+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:09:36.944612+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:13:12.646 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7186 8 365281 365273 0.397088 0 - 0.0299513 2026-02-23T15:13:13.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:13:13 trial068 ceph-mon[38805]: pgmap v4253: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:13:13.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:13:13 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1045640807' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:13:13.646 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7187 8 365281 365273 0.397033 0 - 0.0299513 2026-02-23T15:13:14.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:13:13 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:13:13] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:13:14.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:13:13 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:13:13.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:13:14.646 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7188 8 365281 365273 0.396978 0 - 0.0299513 2026-02-23T15:13:15.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:13:15 trial068 ceph-mon[38805]: pgmap v4254: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:13:15.646 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7189 8 365281 365273 0.396923 0 - 0.0299513 2026-02-23T15:13:15.658 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:13:15.775 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:13:15.873 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:13:15.871+0000 7f478b0b7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:13:15.873 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:13:15.871+0000 7f478b0b7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:13:15.985 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:13:15.985 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":297,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:11:39.967396+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:11:24.652666+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:08:36.021494+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:09:36.944612+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:13:16.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:13:16 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2026-02-23T15:13:16.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:13:16 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-23T15:13:16.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:13:16 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2026-02-23T15:13:16.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:13:16 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T15:13:16.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:13:16 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2026-02-23T15:13:16.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:13:16 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/630844986' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:13:16.646 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7190 8 365281 365273 0.396868 0 - 0.0299513 2026-02-23T15:13:17.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:13:17 trial068 ceph-mon[38805]: pgmap v4255: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:13:17.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:13:17 trial068 ceph-mon[38805]: pgmap v4256: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:13:17.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:13:17 trial068 ceph-mon[38805]: pgmap v4257: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:13:17.646 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7191 8 365281 365273 0.396812 0 - 0.0299513 2026-02-23T15:13:18.646 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7192 8 365281 365273 0.396757 0 - 0.0299513 2026-02-23T15:13:19.026 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:13:19.142 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:13:19.242 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:13:19.241+0000 7f4130f1a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:13:19.242 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:13:19.241+0000 7f4130f1a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:13:19.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:13:19 trial068 ceph-mon[38805]: pgmap v4258: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:13:19.353 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:13:19.353 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":297,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:11:39.967396+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:11:24.652666+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:08:36.021494+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:09:36.944612+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:13:19.646 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7193 8 365281 365273 0.396702 0 - 0.0299513 2026-02-23T15:13:20.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:13:20 trial068 ceph-mon[38805]: 5.e scrub starts 2026-02-23T15:13:20.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:13:20 trial068 ceph-mon[38805]: 5.e scrub ok 2026-02-23T15:13:20.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:13:20 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2765208839' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:13:20.646 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7194 8 365281 365273 0.396647 0 - 0.0299513 2026-02-23T15:13:21.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:13:21 trial068 ceph-mon[38805]: pgmap v4259: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:13:21.646 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7195 8 365281 365273 0.396592 0 - 0.0299513 2026-02-23T15:13:22.392 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:13:22.510 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:13:22.606 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:13:22.604+0000 7f06b7193640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:13:22.606 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:13:22.604+0000 7f06b7193640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:13:22.646 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7196 8 365281 365273 0.396537 0 - 0.0299513 2026-02-23T15:13:22.716 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:13:22.717 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":297,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:11:39.967396+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:11:24.652666+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:08:36.021494+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:09:36.944612+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:13:23.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:13:23 trial068 ceph-mon[38805]: pgmap v4260: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:13:23.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:13:23 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1626007737' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:13:23.647 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7197 8 365281 365273 0.396482 0 - 0.0299513 2026-02-23T15:13:24.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:13:23 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:13:23] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:13:24.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:13:23 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:13:23.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:13:24.647 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7198 8 365281 365273 0.396426 0 - 0.0299513 2026-02-23T15:13:25.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:13:25 trial068 ceph-mon[38805]: pgmap v4261: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:13:25.647 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7199 8 365281 365273 0.396371 0 - 0.0299513 2026-02-23T15:13:25.754 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:13:25.873 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:13:25.981 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:13:25.979+0000 7fd704c5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:13:25.981 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:13:25.980+0000 7fd704c5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:13:26.092 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:13:26.092 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":297,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:11:39.967396+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:11:24.652666+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:08:36.021494+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:09:36.944612+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:13:26.647 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T15:13:26.646902+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T15:13:26.647 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T15:13:26.647 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7200 8 365281 365273 0.396316 0 - 0.0299513 2026-02-23T15:13:27.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:13:27 trial068 ceph-mon[38805]: pgmap v4262: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:13:27.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:13:27 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/43305599' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:13:27.647 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7201 8 365281 365273 0.396261 0 - 0.0299513 2026-02-23T15:13:28.647 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7202 8 365281 365273 0.396206 0 - 0.0299513 2026-02-23T15:13:29.130 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:13:29.248 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:13:29.293 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:13:29 trial068 ceph-mon[38805]: pgmap v4263: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:13:29.351 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:13:29.349+0000 7fee16121640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:13:29.351 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:13:29.349+0000 7fee16121640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:13:29.460 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:13:29.460 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":297,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:11:39.967396+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:11:24.652666+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:08:36.021494+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:09:36.944612+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:13:29.647 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7203 8 365281 365273 0.396151 0 - 0.0299513 2026-02-23T15:13:30.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:13:30 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/492126836' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:13:30.647 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7204 8 365281 365273 0.396096 0 - 0.0299513 2026-02-23T15:13:31.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:13:31 trial068 ceph-mon[38805]: pgmap v4264: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:13:31.647 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7205 8 365281 365273 0.396041 0 - 0.0299513 2026-02-23T15:13:32.496 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:13:32.614 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:13:32.647 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7206 8 365281 365273 0.395986 0 - 0.0299513 2026-02-23T15:13:32.720 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:13:32.718+0000 7fa4560ae640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:13:32.720 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:13:32.718+0000 7fa4560ae640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:13:32.829 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:13:32.829 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":297,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:11:39.967396+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:11:24.652666+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:08:36.021494+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:09:36.944612+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:13:33.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:13:33 trial068 ceph-mon[38805]: pgmap v4265: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:13:33.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:13:33 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2669423291' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:13:33.647 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7207 8 365281 365273 0.395931 0 - 0.0299513 2026-02-23T15:13:34.265 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:13:33 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:13:33.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:13:34.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:13:34 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T15:13:34.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:13:34 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T15:13:34.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:13:33 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:13:33] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:13:34.647 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7208 8 365281 365273 0.395876 0 - 0.0299513 2026-02-23T15:13:35.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:13:35 trial068 ceph-mon[38805]: pgmap v4266: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:13:35.647 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7209 8 365281 365273 0.395822 0 - 0.0299513 2026-02-23T15:13:35.867 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:13:35.985 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:13:36.086 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:13:36.085+0000 7f6c489d1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:13:36.087 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:13:36.085+0000 7f6c489d1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:13:36.195 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:13:36.195 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":297,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:11:39.967396+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:11:24.652666+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:08:36.021494+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:09:36.944612+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:13:36.195 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:13:36 trial068 ceph-mon[38805]: pgmap v4267: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:13:36.648 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7210 8 365281 365273 0.395767 0 - 0.0299513 2026-02-23T15:13:37.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:13:37 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/923636466' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:13:37.648 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7211 8 365281 365273 0.395712 0 - 0.0299513 2026-02-23T15:13:38.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:13:38 trial068 ceph-mon[38805]: pgmap v4268: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:13:38.648 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7212 8 365281 365273 0.395657 0 - 0.0299513 2026-02-23T15:13:39.249 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:13:39.370 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:13:39.473 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:13:39.472+0000 7fecc7470640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:13:39.473 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:13:39.472+0000 7fecc7470640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:13:39.582 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:13:39.582 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":297,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:11:39.967396+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:11:24.652666+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:08:36.021494+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:09:36.944612+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:13:39.648 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7213 8 365281 365273 0.395602 0 - 0.0299513 2026-02-23T15:13:39.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:13:39 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1227672766' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:13:40.648 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7214 8 365281 365273 0.395547 0 - 0.0299513 2026-02-23T15:13:41.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:13:40 trial068 ceph-mon[38805]: pgmap v4269: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:13:41.648 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7215 8 365281 365273 0.395492 0 - 0.0299513 2026-02-23T15:13:42.619 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:13:42.648 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7216 8 365281 365273 0.395438 0 - 0.0299513 2026-02-23T15:13:42.740 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:13:42.839 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:13:42.837+0000 7fc10e499640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:13:42.839 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:13:42.838+0000 7fc10e499640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:13:42.949 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:13:42.949 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":297,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:11:39.967396+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:11:24.652666+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:08:36.021494+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:09:36.944612+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:13:42.949 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:13:42 trial068 ceph-mon[38805]: pgmap v4270: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:13:43.648 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7217 8 365281 365273 0.395383 0 - 0.0299513 2026-02-23T15:13:44.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:13:43 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2475069471' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:13:44.094 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:13:43 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:13:43] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:13:44.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:13:43 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:13:43.974Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:13:44.648 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7218 8 365281 365273 0.395328 0 - 0.0299513 2026-02-23T15:13:45.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:13:44 trial068 ceph-mon[38805]: pgmap v4271: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:13:45.648 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7219 8 365281 365273 0.395273 0 - 0.0299513 2026-02-23T15:13:45.987 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:13:46.105 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:13:46.203 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:13:46.202+0000 7f41fc10f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:13:46.204 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:13:46.202+0000 7f41fc10f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:13:46.316 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:13:46.317 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":297,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:11:39.967396+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:11:24.652666+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:08:36.021494+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:09:36.944612+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:13:46.648 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T15:13:46.648383+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T15:13:46.648 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T15:13:46.649 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7220 8 365281 365273 0.395219 0 - 0.0299513 2026-02-23T15:13:47.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:13:46 trial068 ceph-mon[38805]: pgmap v4272: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:13:47.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:13:46 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3157320447' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:13:47.648 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7221 8 365281 365273 0.395164 0 - 0.0299513 2026-02-23T15:13:48.648 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7222 8 365281 365273 0.395109 0 - 0.0299513 2026-02-23T15:13:49.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:13:48 trial068 ceph-mon[38805]: pgmap v4273: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:13:49.354 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:13:49.468 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:13:49.567 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:13:49.565+0000 7fa80a221640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:13:49.567 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:13:49.565+0000 7fa80a221640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:13:49.648 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7223 8 365281 365273 0.395054 0 - 0.0299513 2026-02-23T15:13:49.675 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:13:49.675 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":297,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:11:39.967396+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:11:24.652666+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:08:36.021494+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:09:36.944612+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:13:50.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:13:49 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3422652662' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:13:50.649 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7224 8 365281 365273 0.395 0 - 0.0299513 2026-02-23T15:13:51.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:13:50 trial068 ceph-mon[38805]: pgmap v4274: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:13:51.649 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7225 8 365281 365273 0.394945 0 - 0.0299513 2026-02-23T15:13:52.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:13:52 trial068 ceph-mon[38805]: 5.a scrub starts 2026-02-23T15:13:52.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:13:52 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T15:13:52.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:13:52 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T15:13:52.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:13:52 trial068 ceph-mon[38805]: 5.a scrub ok 2026-02-23T15:13:52.649 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7226 8 365281 365273 0.39489 0 - 0.0299513 2026-02-23T15:13:52.711 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:13:52.827 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:13:52.926 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:13:52.924+0000 7fbc42ac1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:13:52.926 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:13:52.924+0000 7fbc42ac1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:13:53.036 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:13:53.036 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":297,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:11:39.967396+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:11:24.652666+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:08:36.021494+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:09:36.944612+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:13:53.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:13:53 trial068 ceph-mon[38805]: 5.8 scrub starts 2026-02-23T15:13:53.345 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:13:53 trial068 ceph-mon[38805]: 5.8 scrub ok 2026-02-23T15:13:53.345 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:13:53 trial068 ceph-mon[38805]: pgmap v4275: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:13:53.649 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7227 8 365281 365273 0.394836 0 - 0.0299513 2026-02-23T15:13:54.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:13:54 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2610937659' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:13:54.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:13:53 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:13:53] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:13:54.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:13:53 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:13:53.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:13:54.649 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7228 8 365281 365273 0.394781 0 - 0.0299513 2026-02-23T15:13:55.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:13:55 trial068 ceph-mon[38805]: pgmap v4276: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:13:55.649 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7229 8 365281 365273 0.394726 0 - 0.0299513 2026-02-23T15:13:56.075 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:13:56.193 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:13:56.296 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:13:56.294+0000 7efc20866640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:13:56.296 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:13:56.294+0000 7efc20866640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:13:56.404 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:13:56.404 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":297,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:11:39.967396+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:11:24.652666+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:08:36.021494+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:09:36.944612+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:13:56.649 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7230 8 365281 365273 0.394672 0 - 0.0299513 2026-02-23T15:13:57.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:13:57 trial068 ceph-mon[38805]: pgmap v4277: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:13:57.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:13:57 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3746378356' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:13:57.649 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7231 8 365281 365273 0.394617 0 - 0.0299513 2026-02-23T15:13:58.649 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7232 8 365281 365273 0.394563 0 - 0.0299513 2026-02-23T15:13:59.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:13:59 trial068 ceph-mon[38805]: pgmap v4278: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:13:59.441 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:13:59.560 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:13:59.649 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7233 8 365281 365273 0.394508 0 - 0.0299513 2026-02-23T15:13:59.669 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:13:59.667+0000 7f9b1c2bb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:13:59.669 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:13:59.668+0000 7f9b1c2bb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:13:59.781 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:13:59.782 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":297,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:11:39.967396+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:11:24.652666+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:08:36.021494+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:09:36.944612+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:14:00.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:14:00 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3430068577' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:14:00.649 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7234 8 365281 365273 0.394454 0 - 0.0299513 2026-02-23T15:14:01.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:14:01 trial068 ceph-mon[38805]: pgmap v4279: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:14:01.650 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7235 8 365281 365273 0.394399 0 - 0.0299513 2026-02-23T15:14:02.650 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7236 8 365281 365273 0.394345 0 - 0.0299513 2026-02-23T15:14:02.818 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:14:02.937 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:14:03.038 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:14:03.037+0000 7fa440cb0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:14:03.039 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:14:03.037+0000 7fa440cb0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:14:03.147 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:14:03.148 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":297,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:11:39.967396+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:11:24.652666+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:08:36.021494+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:09:36.944612+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:14:03.148 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:14:03 trial068 ceph-mon[38805]: 5.5 scrub starts 2026-02-23T15:14:03.148 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:14:03 trial068 ceph-mon[38805]: 5.5 scrub ok 2026-02-23T15:14:03.148 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:14:03 trial068 ceph-mon[38805]: pgmap v4280: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:14:03.650 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7237 8 365281 365273 0.39429 0 - 0.0299513 2026-02-23T15:14:04.265 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:14:03 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:14:03.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:14:04.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:14:04 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3527823963' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:14:04.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:14:03 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:14:03] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:14:04.650 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7238 8 365281 365273 0.394236 0 - 0.0299513 2026-02-23T15:14:05.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:14:05 trial068 ceph-mon[38805]: pgmap v4281: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:14:05.650 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7239 8 365281 365273 0.394181 0 - 0.0299513 2026-02-23T15:14:06.185 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:14:06.301 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:14:06.398 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:14:06.396+0000 7fd6cae12640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:14:06.398 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:14:06.397+0000 7fd6cae12640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:14:06.507 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:14:06.507 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":297,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:11:39.967396+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:11:24.652666+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:08:36.021494+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:09:36.944612+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:14:06.650 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T15:14:06.650422+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T15:14:06.651 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T15:14:06.651 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7240 8 365281 365273 0.394127 0 - 0.0299513 2026-02-23T15:14:07.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:14:07 trial068 ceph-mon[38805]: pgmap v4282: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:14:07.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:14:07 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/222799158' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:14:07.651 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7241 8 365281 365273 0.394072 0 - 0.0299513 2026-02-23T15:14:08.651 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7242 8 365281 365273 0.394018 0 - 0.0299513 2026-02-23T15:14:09.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:14:09 trial068 ceph-mon[38805]: pgmap v4283: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:14:09.543 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:14:09.651 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7243 8 365281 365273 0.393963 0 - 0.0299513 2026-02-23T15:14:09.659 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:14:09.759 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:14:09.758+0000 7f4048321640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:14:09.760 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:14:09.758+0000 7f4048321640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:14:09.870 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:14:09.870 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":297,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:11:39.967396+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:11:24.652666+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:08:36.021494+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:09:36.944612+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:14:10.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:14:10 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2724111309' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:14:10.651 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7244 8 365281 365273 0.393909 0 - 0.0299513 2026-02-23T15:14:11.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:14:11 trial068 ceph-mon[38805]: pgmap v4284: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:14:11.651 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7245 8 365281 365273 0.393855 0 - 0.0299513 2026-02-23T15:14:12.651 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7246 8 365281 365273 0.3938 0 - 0.0299513 2026-02-23T15:14:12.907 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:14:13.026 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:14:13.130 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:14:13.129+0000 7fcf85f2b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:14:13.131 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:14:13.129+0000 7fcf85f2b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:14:13.241 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:14:13.241 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":297,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:11:39.967396+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:11:24.652666+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:08:36.021494+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:09:36.944612+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:14:13.241 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:14:13 trial068 ceph-mon[38805]: pgmap v4285: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:14:13.651 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7247 8 365281 365273 0.393746 0 - 0.0299513 2026-02-23T15:14:14.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:14:14 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/116409667' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:14:14.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:14:13 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:14:13] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:14:14.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:14:13 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:14:13.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:14:14.651 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7248 8 365281 365273 0.393692 0 - 0.0299513 2026-02-23T15:14:15.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:14:15 trial068 ceph-mon[38805]: pgmap v4286: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:14:15.651 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7249 8 365281 365273 0.393637 0 - 0.0299513 2026-02-23T15:14:16.281 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:14:16.306 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:14:16 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T15:14:16.306 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:14:16 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T15:14:16.306 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:14:16 trial068 ceph-mon[38805]: osdmap e298: 4 total, 3 up, 4 in 2026-02-23T15:14:16.306 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:14:16 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2026-02-23T15:14:16.306 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:14:16 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-23T15:14:16.306 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:14:16 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2026-02-23T15:14:16.306 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:14:16 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T15:14:16.307 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:14:16 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2026-02-23T15:14:16.400 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:14:16.502 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:14:16.500+0000 7fbaba98a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:14:16.502 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:14:16.500+0000 7fbaba98a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:14:16.609 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:14:16.609 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":298,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:14:15.024092+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:11:24.652666+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:13:32.603417+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:09:36.944612+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:14:16.651 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7250 8 365281 365273 0.393583 0 - 0.0299513 2026-02-23T15:14:17.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:14:17 trial068 ceph-mon[38805]: pgmap v4288: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:14:17.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:14:17 trial068 ceph-mon[38805]: pgmap v4289: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:14:17.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:14:17 trial068 ceph-mon[38805]: pgmap v4290: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:14:17.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:14:17 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2684797896' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:14:17.651 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7251 8 365281 365273 0.393529 0 - 0.0299513 2026-02-23T15:14:18.651 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7252 8 365281 365273 0.393475 0 - 0.0299513 2026-02-23T15:14:19.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:14:19 trial068 ceph-mon[38805]: pgmap v4291: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:14:19.648 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:14:19.652 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7253 8 365281 365273 0.39342 0 - 0.0299513 2026-02-23T15:14:19.766 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:14:19.871 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:14:19.869+0000 7fd9bdee7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:14:19.871 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:14:19.870+0000 7fd9bdee7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:14:19.980 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:14:19.980 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":298,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:14:15.024092+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:11:24.652666+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:13:32.603417+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:09:36.944612+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:14:20.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:14:20 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/677853739' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:14:20.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:14:20 trial068 ceph-mon[38805]: osdmap e299: 4 total, 3 up, 4 in 2026-02-23T15:14:20.652 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7254 8 365281 365273 0.393366 0 - 0.0299513 2026-02-23T15:14:21.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:14:21 trial068 ceph-mon[38805]: pgmap v4293: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:14:21.652 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7255 8 365281 365273 0.393312 0 - 0.0299513 2026-02-23T15:14:22.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:14:22 trial068 ceph-mon[38805]: pgmap v4294: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:14:22.652 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7256 8 365281 365273 0.393258 0 - 0.0299513 2026-02-23T15:14:23.018 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:14:23.137 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:14:23.238 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:14:23.236+0000 7f1af5187640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:14:23.238 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:14:23.236+0000 7f1af5187640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:14:23.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:14:23 trial068 ceph-mon[38805]: 5.7 scrub starts 2026-02-23T15:14:23.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:14:23 trial068 ceph-mon[38805]: 5.7 scrub ok 2026-02-23T15:14:23.348 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:14:23.348 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":299,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:14:19.987733+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:11:24.652666+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:13:32.603417+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:09:36.944612+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:14:23.652 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7257 8 365281 365273 0.393203 0 - 0.0299513 2026-02-23T15:14:24.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:14:24 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3055934368' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:14:24.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:14:24 trial068 ceph-mon[38805]: pgmap v4295: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:14:24.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:14:23 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:14:23] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:14:24.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:14:23 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:14:23.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:14:24.652 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7258 8 365281 365273 0.393149 0 - 0.0299513 2026-02-23T15:14:25.652 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7259 8 365281 365273 0.393095 0 - 0.0299513 2026-02-23T15:14:26.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:14:26 trial068 ceph-mon[38805]: pgmap v4296: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:14:26.386 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:14:26.504 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:14:26.601 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:14:26.599+0000 7f08f66dc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:14:26.601 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:14:26.599+0000 7f08f66dc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:14:26.652 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T15:14:26.652038+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T15:14:26.652 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T15:14:26.652 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7260 8 365281 365273 0.393041 0 - 0.0299513 2026-02-23T15:14:26.709 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:14:26.709 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":299,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:14:19.987733+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:11:24.652666+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:13:32.603417+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:09:36.944612+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:14:27.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:14:27 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3390345287' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:14:27.652 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7261 8 365281 365273 0.392987 0 - 0.0299513 2026-02-23T15:14:28.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:14:28 trial068 ceph-mon[38805]: pgmap v4297: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:14:28.652 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7262 8 365281 365273 0.392933 0 - 0.0299513 2026-02-23T15:14:29.652 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7263 8 365281 365273 0.392879 0 - 0.0299513 2026-02-23T15:14:29.746 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:14:29.866 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:14:29.967 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:14:29.965+0000 7f3a0f20a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:14:29.967 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:14:29.965+0000 7f3a0f20a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:14:30.075 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:14:30.076 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":299,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:14:19.987733+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:11:24.652666+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:13:32.603417+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:09:36.944612+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:14:30.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:14:30 trial068 ceph-mon[38805]: pgmap v4298: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:14:30.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:14:30 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/659733795' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:14:30.652 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7264 8 365281 365273 0.392825 0 - 0.0299513 2026-02-23T15:14:31.652 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7265 8 365281 365273 0.39277 0 - 0.0299513 2026-02-23T15:14:32.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:14:32 trial068 ceph-mon[38805]: pgmap v4299: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:14:32.653 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7266 8 365281 365273 0.392716 0 - 0.0299513 2026-02-23T15:14:33.113 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:14:33.232 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:14:33.332 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:14:33.331+0000 7f928fd38640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:14:33.332 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:14:33.331+0000 7f928fd38640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:14:33.443 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:14:33.444 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":299,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:14:19.987733+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:11:24.652666+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:13:32.603417+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:09:36.944612+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:14:33.653 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7267 8 365281 365273 0.392662 0 - 0.0299513 2026-02-23T15:14:33.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:14:33 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2868602793' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:14:34.264 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:14:33 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:14:33.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:14:34.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:14:33 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:14:33] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:14:34.653 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7268 8 365281 365273 0.392608 0 - 0.0299513 2026-02-23T15:14:34.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:14:34 trial068 ceph-mon[38805]: pgmap v4300: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:14:35.653 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7269 8 365281 365273 0.392554 0 - 0.0299513 2026-02-23T15:14:36.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:14:36 trial068 ceph-mon[38805]: pgmap v4301: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:14:36.496 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:14:36.615 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:14:36.653 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7270 8 365281 365273 0.3925 0 - 0.0299513 2026-02-23T15:14:36.718 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:14:36.716+0000 7f52eb959640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:14:36.718 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:14:36.716+0000 7f52eb959640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:14:36.832 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:14:36.832 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":299,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:14:19.987733+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:11:24.652666+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:13:32.603417+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:09:36.944612+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:14:37.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:14:37 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2918621724' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:14:37.653 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7271 8 365281 365273 0.392446 0 - 0.0299513 2026-02-23T15:14:38.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:14:38 trial068 ceph-mon[38805]: pgmap v4302: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:14:38.653 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7272 8 365281 365273 0.392392 0 - 0.0299513 2026-02-23T15:14:39.653 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7273 8 365281 365273 0.392338 0 - 0.0299513 2026-02-23T15:14:39.870 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:14:39.987 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:14:40.084 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:14:40.082+0000 7f5255cc9640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:14:40.084 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:14:40.083+0000 7f5255cc9640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:14:40.196 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:14:40.197 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":299,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:14:19.987733+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:11:24.652666+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:13:32.603417+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:09:36.944612+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:14:40.197 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:14:40 trial068 ceph-mon[38805]: pgmap v4303: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:14:40.653 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7274 8 365281 365273 0.392284 0 - 0.0299513 2026-02-23T15:14:41.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:14:41 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1732642615' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:14:41.653 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7275 8 365281 365273 0.392231 0 - 0.0299513 2026-02-23T15:14:42.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:14:42 trial068 ceph-mon[38805]: pgmap v4304: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:14:42.653 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7276 8 365281 365273 0.392177 0 - 0.0299513 2026-02-23T15:14:43.234 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:14:43.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:14:43 trial068 ceph-mon[38805]: 5.e scrub starts 2026-02-23T15:14:43.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:14:43 trial068 ceph-mon[38805]: 5.e scrub ok 2026-02-23T15:14:43.351 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:14:43.454 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:14:43.453+0000 7f226d9b2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:14:43.455 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:14:43.453+0000 7f226d9b2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:14:43.564 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:14:43.565 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":299,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:14:19.987733+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:11:24.652666+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:13:32.603417+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:09:36.944612+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:14:43.653 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7277 8 365281 365273 0.392123 0 - 0.0299513 2026-02-23T15:14:44.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:14:44 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1253710743' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:14:44.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:14:44 trial068 ceph-mon[38805]: pgmap v4305: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:14:44.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:14:43 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:14:43] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:14:44.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:14:43 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:14:43.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:14:44.654 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7278 8 365281 365273 0.392069 0 - 0.0299513 2026-02-23T15:14:45.654 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7279 8 365281 365273 0.392015 0 - 0.0299513 2026-02-23T15:14:46.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:14:46 trial068 ceph-mon[38805]: pgmap v4306: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:14:46.620 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:14:46.654 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T15:14:46.653730+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T15:14:46.654 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T15:14:46.654 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7280 8 365281 365273 0.391961 0 - 0.0299513 2026-02-23T15:14:46.739 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:14:46.838 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:14:46.836+0000 7f2c480bd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:14:46.838 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:14:46.837+0000 7f2c480bd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:14:46.948 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:14:46.948 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":299,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:14:19.987733+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:11:24.652666+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:13:32.603417+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:09:36.944612+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:14:47.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:14:47 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/578154560' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:14:47.654 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7281 8 365281 365273 0.391907 0 - 0.0299513 2026-02-23T15:14:48.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:14:48 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T15:14:48.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:14:48 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T15:14:48.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:14:48 trial068 ceph-mon[38805]: pgmap v4307: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:14:48.654 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7282 8 365281 365273 0.391854 0 - 0.0299513 2026-02-23T15:14:49.654 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7283 8 365281 365273 0.3918 0 - 0.0299513 2026-02-23T15:14:49.986 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:14:50.104 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:14:50.206 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:14:50.205+0000 7f922670f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:14:50.207 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:14:50.205+0000 7f922670f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:14:50.315 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:14:50.315 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":299,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:14:19.987733+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:11:24.652666+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:13:32.603417+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:09:36.944612+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:14:50.654 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7284 8 365281 365273 0.391746 0 - 0.0299513 2026-02-23T15:14:51.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:14:50 trial068 ceph-mon[38805]: pgmap v4308: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:14:51.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:14:50 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2431733532' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:14:51.654 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7285 8 365281 365273 0.391692 0 - 0.0299513 2026-02-23T15:14:52.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:14:52 trial068 ceph-mon[38805]: pgmap v4309: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:14:52.654 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7286 8 365281 365273 0.391638 0 - 0.0299513 2026-02-23T15:14:53.353 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:14:53.471 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:14:53.576 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:14:53.575+0000 7efe23737640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:14:53.577 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:14:53.575+0000 7efe23737640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:14:53.654 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7287 8 365281 365273 0.391585 0 - 0.0299513 2026-02-23T15:14:53.685 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:14:53.685 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":299,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:14:19.987733+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:11:24.652666+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:13:32.603417+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:09:36.944612+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:14:53.974 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:14:53 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2051114454' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:14:54.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:14:53 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:14:53] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:14:54.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:14:53 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:14:53.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:14:54.654 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7288 8 365281 365273 0.391531 0 - 0.0299513 2026-02-23T15:14:55.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:14:54 trial068 ceph-mon[38805]: pgmap v4310: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:14:55.654 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7289 8 365281 365273 0.391477 0 - 0.0299513 2026-02-23T15:14:56.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:14:56 trial068 ceph-mon[38805]: pgmap v4311: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:14:56.654 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7290 8 365281 365273 0.391424 0 - 0.0299513 2026-02-23T15:14:56.722 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:14:56.842 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:14:56.943 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:14:56.941+0000 7f5b38733640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:14:56.943 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:14:56.941+0000 7f5b38733640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:14:57.055 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:14:57.055 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":299,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:14:19.987733+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:11:24.652666+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:13:32.603417+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:09:36.944612+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:14:57.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:14:57 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2661590460' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:14:57.654 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7291 8 365281 365273 0.39137 0 - 0.0299513 2026-02-23T15:14:58.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:14:58 trial068 ceph-mon[38805]: pgmap v4312: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:14:58.654 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7292 8 365281 365273 0.391316 0 - 0.0299513 2026-02-23T15:14:59.655 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7293 8 365281 365273 0.391263 0 - 0.0299513 2026-02-23T15:15:00.092 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:15:00.210 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:15:00.309 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:15:00.308+0000 7fb46d7cd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:15:00.309 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:15:00.308+0000 7fb46d7cd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:15:00.418 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:15:00.418 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":299,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:14:19.987733+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:11:24.652666+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:13:32.603417+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:09:36.944612+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:15:00.655 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7294 8 365281 365273 0.391209 0 - 0.0299513 2026-02-23T15:15:01.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:15:01 trial068 ceph-mon[38805]: pgmap v4313: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:15:01.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:15:01 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/510120953' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:15:01.655 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7295 8 365281 365273 0.391155 0 - 0.0299513 2026-02-23T15:15:02.655 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7296 8 365281 365273 0.391102 0 - 0.0299513 2026-02-23T15:15:03.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:15:03 trial068 ceph-mon[38805]: pgmap v4314: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:15:03.456 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:15:03.575 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:15:03.655 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7297 8 365281 365273 0.391048 0 - 0.0299513 2026-02-23T15:15:03.673 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:15:03.672+0000 7f5228496640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:15:03.673 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:15:03.672+0000 7f5228496640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:15:03.784 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:15:03.784 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":299,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:14:19.987733+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:11:24.652666+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:13:32.603417+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:09:36.944612+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:15:04.084 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:15:03 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:15:03] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:15:04.265 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:15:03 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:15:03.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:15:04.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:15:04 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3917346165' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:15:04.655 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7298 8 365281 365273 0.390994 0 - 0.0299513 2026-02-23T15:15:05.655 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7299 8 365281 365273 0.390941 0 - 0.0299513 2026-02-23T15:15:06.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:15:05 trial068 ceph-mon[38805]: pgmap v4315: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:15:06.655 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T15:15:06.655256+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T15:15:06.655 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T15:15:06.656 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7300 8 365281 365273 0.390887 0 - 0.0299513 2026-02-23T15:15:06.822 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:15:06.938 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:15:07.039 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:15:07.037+0000 7f113c302640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:15:07.039 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:15:07.038+0000 7f113c302640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:15:07.152 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:15:07.152 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":299,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:14:19.987733+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:11:24.652666+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:13:32.603417+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:09:36.944612+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:15:07.655 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7301 8 365281 365273 0.390834 0 - 0.0299513 2026-02-23T15:15:08.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:15:07 trial068 ceph-mon[38805]: pgmap v4316: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:15:08.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:15:07 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/986085512' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:15:08.655 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7302 8 365281 365273 0.39078 0 - 0.0299513 2026-02-23T15:15:09.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:15:08 trial068 ceph-mon[38805]: 5.8 scrub starts 2026-02-23T15:15:09.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:15:08 trial068 ceph-mon[38805]: 5.8 scrub ok 2026-02-23T15:15:09.655 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7303 8 365281 365273 0.390727 0 - 0.0299513 2026-02-23T15:15:10.192 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:15:10.311 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:15:10.333 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:15:09 trial068 ceph-mon[38805]: pgmap v4317: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:15:10.415 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:15:10.413+0000 7f09bf5b0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:15:10.415 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:15:10.414+0000 7f09bf5b0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:15:10.526 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:15:10.526 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":299,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:14:19.987733+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:11:24.652666+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:13:32.603417+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:09:36.944612+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:15:10.655 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7304 8 365281 365273 0.390673 0 - 0.0299513 2026-02-23T15:15:11.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:15:11 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T15:15:11.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:15:11 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T15:15:11.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:15:11 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2701599858' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:15:11.656 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7305 8 365281 365273 0.39062 0 - 0.0299513 2026-02-23T15:15:12.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:15:12 trial068 ceph-mon[38805]: pgmap v4318: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:15:12.656 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7306 8 365281 365273 0.390566 0 - 0.0299513 2026-02-23T15:15:13.567 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:15:13.656 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7307 8 365281 365273 0.390513 0 - 0.0299513 2026-02-23T15:15:13.684 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:15:13.788 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:15:13.786+0000 7f357e9a3640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:15:13.788 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:15:13.786+0000 7f357e9a3640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:15:13.895 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:15:13.896 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":299,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:14:19.987733+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:11:24.652666+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:13:32.603417+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:09:36.944612+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:15:14.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:15:14 trial068 ceph-mon[38805]: pgmap v4319: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:15:14.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:15:14 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1729150872' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:15:14.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:15:13 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:15:13] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:15:14.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:15:13 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:15:13.972Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:15:14.656 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7308 8 365281 365273 0.390459 0 - 0.0299513 2026-02-23T15:15:15.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:15:15 trial068 ceph-mon[38805]: 5.a scrub starts 2026-02-23T15:15:15.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:15:15 trial068 ceph-mon[38805]: 5.a scrub ok 2026-02-23T15:15:15.656 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7309 8 365281 365273 0.390406 0 - 0.0299513 2026-02-23T15:15:16.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:15:16 trial068 ceph-mon[38805]: pgmap v4320: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:15:16.656 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7310 8 365281 365273 0.390353 0 - 0.0299513 2026-02-23T15:15:16.932 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:15:17.051 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:15:17.151 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:15:17.149+0000 7efc24746640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:15:17.151 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:15:17.149+0000 7efc24746640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:15:17.263 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:15:17.263 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":299,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:14:19.987733+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:11:24.652666+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:13:32.603417+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:09:36.944612+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:15:17.264 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:15:17 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2026-02-23T15:15:17.264 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:15:17 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-23T15:15:17.264 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:15:17 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2026-02-23T15:15:17.264 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:15:17 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T15:15:17.264 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:15:17 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2026-02-23T15:15:17.656 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7311 8 365281 365273 0.390299 0 - 0.0299513 2026-02-23T15:15:18.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:15:18 trial068 ceph-mon[38805]: pgmap v4321: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:15:18.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:15:18 trial068 ceph-mon[38805]: pgmap v4322: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:15:18.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:15:18 trial068 ceph-mon[38805]: pgmap v4323: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:15:18.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:15:18 trial068 ceph-mon[38805]: 5.5 scrub starts 2026-02-23T15:15:18.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:15:18 trial068 ceph-mon[38805]: 5.5 scrub ok 2026-02-23T15:15:18.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:15:18 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/4171350080' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:15:18.656 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7312 8 365281 365273 0.390246 0 - 0.0299513 2026-02-23T15:15:19.656 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7313 8 365281 365273 0.390192 0 - 0.0299513 2026-02-23T15:15:20.299 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:15:20.324 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:15:20 trial068 ceph-mon[38805]: pgmap v4324: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:15:20.416 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:15:20.510 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:15:20.508+0000 7fd24b5be640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:15:20.510 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:15:20.508+0000 7fd24b5be640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:15:20.619 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:15:20.619 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":299,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:14:19.987733+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:11:24.652666+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:13:32.603417+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:09:36.944612+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:15:20.657 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7314 8 365281 365273 0.390139 0 - 0.0299513 2026-02-23T15:15:21.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:15:21 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/671537783' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:15:21.656 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7315 8 365281 365273 0.390086 0 - 0.0299513 2026-02-23T15:15:22.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:15:22 trial068 ceph-mon[38805]: pgmap v4325: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:15:22.656 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7316 8 365281 365273 0.390032 0 - 0.0299513 2026-02-23T15:15:23.656 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7317 8 365281 365273 0.389979 0 - 0.0299513 2026-02-23T15:15:23.657 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:15:23.774 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:15:23.878 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:15:23.876+0000 7fd81e76e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:15:23.878 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:15:23.876+0000 7fd81e76e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:15:23.991 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:15:23.992 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":299,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:14:19.987733+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:11:24.652666+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:13:32.603417+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:09:36.944612+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:15:24.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:15:24 trial068 ceph-mon[38805]: pgmap v4326: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:15:24.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:15:24 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2182069374' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:15:24.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:15:23 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:15:23] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:15:24.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:15:23 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:15:23.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:15:24.657 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7318 8 365281 365273 0.389926 0 - 0.0299513 2026-02-23T15:15:25.657 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7319 8 365281 365273 0.389873 0 - 0.0299513 2026-02-23T15:15:26.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:15:26 trial068 ceph-mon[38805]: pgmap v4327: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:15:26.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:15:26 trial068 ceph-mon[38805]: 5.7 scrub starts 2026-02-23T15:15:26.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:15:26 trial068 ceph-mon[38805]: 5.7 scrub ok 2026-02-23T15:15:26.657 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T15:15:26.656771+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T15:15:26.657 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T15:15:26.657 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7320 8 365281 365273 0.389819 0 - 0.0299513 2026-02-23T15:15:27.029 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:15:27.147 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:15:27.247 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:15:27.246+0000 7fee35f97640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:15:27.248 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:15:27.246+0000 7fee35f97640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:15:27.357 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:15:27.357 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":299,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:14:19.987733+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:11:24.652666+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:13:32.603417+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:09:36.944612+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:15:27.657 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7321 8 365281 365273 0.389766 0 - 0.0299513 2026-02-23T15:15:28.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:15:28 trial068 ceph-mon[38805]: pgmap v4328: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:15:28.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:15:28 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1409069900' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:15:28.657 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7322 8 365281 365273 0.389713 0 - 0.0299513 2026-02-23T15:15:29.657 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7323 8 365281 365273 0.38966 0 - 0.0299513 2026-02-23T15:15:30.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:15:30 trial068 ceph-mon[38805]: pgmap v4329: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:15:30.394 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:15:30.512 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:15:30.610 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:15:30.608+0000 7fd703f1c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:15:30.610 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:15:30.609+0000 7fd703f1c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:15:30.657 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7324 8 365281 365273 0.389606 0 - 0.0299513 2026-02-23T15:15:30.722 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:15:30.722 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":299,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:14:19.987733+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:11:24.652666+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:13:32.603417+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:09:36.944612+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:15:31.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:15:31 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/13546021' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:15:31.657 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7325 8 365281 365273 0.389553 0 - 0.0299513 2026-02-23T15:15:32.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:15:32 trial068 ceph-mon[38805]: pgmap v4330: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:15:32.657 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7326 8 365281 365273 0.3895 0 - 0.0299513 2026-02-23T15:15:33.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:15:33 trial068 ceph-mon[38805]: osdmap e300: 4 total, 3 up, 4 in 2026-02-23T15:15:33.657 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7327 8 365281 365273 0.389447 0 - 0.0299513 2026-02-23T15:15:33.759 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:15:33.878 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:15:33.979 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:15:33.977+0000 7efec7221640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:15:33.979 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:15:33.977+0000 7efec7221640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:15:34.088 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:15:34.088 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":300,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:15:32.021517+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:11:24.652666+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:13:32.603417+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:15:09.310704+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:15:34.088 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:15:34 trial068 ceph-mon[38805]: pgmap v4332: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:15:34.089 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:15:33 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:15:33] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:15:34.265 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:15:33 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:15:33.972Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:15:34.657 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7328 8 365281 365273 0.389394 0 - 0.0299513 2026-02-23T15:15:35.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:15:35 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2128280752' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:15:35.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:15:35 trial068 ceph-mon[38805]: osdmap e301: 4 total, 3 up, 4 in 2026-02-23T15:15:35.657 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7329 8 365281 365273 0.389341 0 - 0.0299513 2026-02-23T15:15:36.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:15:36 trial068 ceph-mon[38805]: pgmap v4333: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:15:36.657 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7330 8 365281 365273 0.389288 0 - 0.0299513 2026-02-23T15:15:37.127 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:15:37.246 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:15:37.349 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:15:37.348+0000 7fe1b6b44640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:15:37.350 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:15:37.348+0000 7fe1b6b44640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:15:37.459 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:15:37.459 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":301,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:15:34.996693+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:11:24.652666+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:13:32.603417+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:15:09.310704+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:15:37.658 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7331 8 365281 365273 0.389234 0 - 0.0299513 2026-02-23T15:15:38.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:15:38 trial068 ceph-mon[38805]: pgmap v4335: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:15:38.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:15:38 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1061923102' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:15:38.658 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7332 8 365281 365273 0.389181 0 - 0.0299513 2026-02-23T15:15:39.658 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7333 8 365281 365273 0.389128 0 - 0.0299513 2026-02-23T15:15:40.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:15:40 trial068 ceph-mon[38805]: pgmap v4336: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:15:40.516 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:15:40.633 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:15:40.658 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7334 8 365281 365273 0.389075 0 - 0.0299513 2026-02-23T15:15:40.740 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:15:40.738+0000 7f3645884640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:15:40.740 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:15:40.738+0000 7f3645884640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:15:40.849 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:15:40.849 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":301,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:15:34.996693+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:11:24.652666+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:13:32.603417+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:15:09.310704+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:15:41.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:15:41 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2693843186' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:15:41.658 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7335 8 365281 365273 0.389022 0 - 0.0299513 2026-02-23T15:15:42.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:15:42 trial068 ceph-mon[38805]: pgmap v4337: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:15:42.658 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7336 8 365281 365273 0.388969 0 - 0.0299513 2026-02-23T15:15:43.658 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7337 8 365281 365273 0.388916 0 - 0.0299513 2026-02-23T15:15:43.888 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:15:44.004 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:15:44.105 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:15:44.103+0000 7fe122353640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:15:44.105 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:15:44.104+0000 7fe122353640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:15:44.216 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:15:44.216 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":301,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:15:34.996693+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:11:24.652666+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:13:32.603417+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:15:09.310704+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:15:44.217 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:15:44 trial068 ceph-mon[38805]: pgmap v4338: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:15:44.217 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:15:43 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:15:43] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:15:44.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:15:43 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:15:43.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:15:44.658 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7338 8 365281 365273 0.388863 0 - 0.0299513 2026-02-23T15:15:45.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:15:45 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T15:15:45.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:15:45 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T15:15:45.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:15:45 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3302017568' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:15:45.658 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7339 8 365281 365273 0.38881 0 - 0.0299513 2026-02-23T15:15:46.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:15:46 trial068 ceph-mon[38805]: pgmap v4339: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:15:46.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:15:46 trial068 ceph-mon[38805]: 5.e scrub starts 2026-02-23T15:15:46.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:15:46 trial068 ceph-mon[38805]: 5.e scrub ok 2026-02-23T15:15:46.658 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T15:15:46.658306+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T15:15:46.658 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T15:15:46.659 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7340 8 365281 365273 0.388757 0 - 0.0299513 2026-02-23T15:15:47.253 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:15:47.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:15:47 trial068 ceph-mon[38805]: pgmap v4340: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:15:47.372 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:15:47.470 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:15:47.469+0000 7fde1e96c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:15:47.470 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:15:47.469+0000 7fde1e96c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:15:47.580 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:15:47.581 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":301,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:15:34.996693+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:11:24.652666+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:13:32.603417+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:15:09.310704+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:15:47.658 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7341 8 365281 365273 0.388704 0 - 0.0299513 2026-02-23T15:15:48.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:15:48 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3160217528' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:15:48.658 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7342 8 365281 365273 0.388651 0 - 0.0299513 2026-02-23T15:15:49.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:15:49 trial068 ceph-mon[38805]: pgmap v4341: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:15:49.658 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7343 8 365281 365273 0.388598 0 - 0.0299513 2026-02-23T15:15:50.616 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:15:50.658 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7344 8 365281 365273 0.388545 0 - 0.0299513 2026-02-23T15:15:50.736 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:15:50.840 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:15:50.839+0000 7f7022298640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:15:50.840 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:15:50.839+0000 7f7022298640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:15:50.949 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:15:50.949 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":301,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:15:34.996693+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:11:24.652666+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:13:32.603417+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:15:09.310704+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:15:51.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:15:51 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2699599196' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:15:51.659 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7345 8 365281 365273 0.388493 0 - 0.0299513 2026-02-23T15:15:52.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:15:52 trial068 ceph-mon[38805]: pgmap v4342: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:15:52.659 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7346 8 365281 365273 0.38844 0 - 0.0299513 2026-02-23T15:15:53.659 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7347 8 365281 365273 0.388387 0 - 0.0299513 2026-02-23T15:15:53.987 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:15:54.103 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:15:54.204 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:15:54.202+0000 7f7d54e7f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:15:54.204 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:15:54.203+0000 7f7d54e7f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:15:54.313 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:15:54.313 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":301,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:15:34.996693+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:11:24.652666+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:13:32.603417+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:15:09.310704+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:15:54.313 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:15:54 trial068 ceph-mon[38805]: pgmap v4343: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:15:54.314 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:15:53 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:15:53] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:15:54.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:15:53 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:15:53.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:15:54.659 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7348 8 365281 365273 0.388334 0 - 0.0299513 2026-02-23T15:15:55.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:15:55 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2450587101' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:15:55.659 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7349 8 365281 365273 0.388281 0 - 0.0299513 2026-02-23T15:15:56.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:15:56 trial068 ceph-mon[38805]: pgmap v4344: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:15:56.659 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7350 8 365281 365273 0.388228 0 - 0.0299513 2026-02-23T15:15:57.366 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:15:57.483 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:15:57.580 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:15:57.578+0000 7fce84eb9640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:15:57.580 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:15:57.579+0000 7fce84eb9640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:15:57.659 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7351 8 365281 365273 0.388175 0 - 0.0299513 2026-02-23T15:15:57.694 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:15:57.694 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":301,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:15:34.996693+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:11:24.652666+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:13:32.603417+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:15:09.310704+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:15:58.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:15:58 trial068 ceph-mon[38805]: pgmap v4345: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:15:58.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:15:58 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/350729920' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:15:58.659 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7352 8 365281 365273 0.388123 0 - 0.0299513 2026-02-23T15:15:59.659 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7353 8 365281 365273 0.38807 0 - 0.0299513 2026-02-23T15:16:00.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:16:00 trial068 ceph-mon[38805]: pgmap v4346: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:16:00.659 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7354 8 365281 365273 0.388017 0 - 0.0299513 2026-02-23T15:16:00.731 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:16:00.847 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:16:00.948 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:16:00.946+0000 7f007586a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:16:00.948 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:16:00.947+0000 7f007586a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:16:01.057 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:16:01.057 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":301,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:15:34.996693+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:11:24.652666+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:13:32.603417+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:15:09.310704+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:16:01.659 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7355 8 365281 365273 0.387964 0 - 0.0299513 2026-02-23T15:16:02.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:16:02 trial068 ceph-mon[38805]: pgmap v4347: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:16:02.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:16:02 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2364234148' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:16:02.659 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7356 8 365281 365273 0.387912 0 - 0.0299513 2026-02-23T15:16:03.659 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7357 8 365281 365273 0.387859 0 - 0.0299513 2026-02-23T15:16:04.094 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:16:04.213 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:16:04.236 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:16:04 trial068 ceph-mon[38805]: pgmap v4348: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:16:04.236 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:16:03 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:16:03] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:16:04.265 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:16:03 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:16:03.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:16:04.316 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:16:04.314+0000 7fb4f65b6640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:16:04.316 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:16:04.314+0000 7fb4f65b6640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:16:04.425 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:16:04.425 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":301,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:15:34.996693+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:11:24.652666+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:13:32.603417+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:15:09.310704+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:16:04.659 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7358 8 365281 365273 0.387806 0 - 0.0299513 2026-02-23T15:16:05.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:16:05 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/147847830' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:16:05.659 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7359 8 365281 365273 0.387753 0 - 0.0299513 2026-02-23T15:16:06.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:16:06 trial068 ceph-mon[38805]: pgmap v4349: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:16:06.660 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T15:16:06.659640+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T15:16:06.660 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T15:16:06.660 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7360 8 365281 365273 0.387701 0 - 0.0299513 2026-02-23T15:16:07.477 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:16:07.595 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:16:07.660 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7361 8 365281 365273 0.387648 0 - 0.0299513 2026-02-23T15:16:07.695 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:16:07.693+0000 7f4d2acc0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:16:07.695 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:16:07.693+0000 7f4d2acc0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:16:07.805 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:16:07.805 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":301,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:15:34.996693+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:11:24.652666+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:13:32.603417+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:15:09.310704+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:16:08.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:16:08 trial068 ceph-mon[38805]: pgmap v4350: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:16:08.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:16:08 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3991130240' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:16:08.660 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7362 8 365281 365273 0.387595 0 - 0.0299513 2026-02-23T15:16:09.660 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7363 8 365281 365273 0.387543 0 - 0.0299513 2026-02-23T15:16:10.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:16:10 trial068 ceph-mon[38805]: pgmap v4351: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:16:10.660 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7364 8 365281 365273 0.38749 0 - 0.0299513 2026-02-23T15:16:10.842 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:16:10.959 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:16:11.060 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:16:11.058+0000 7f9bf0c25640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:16:11.060 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:16:11.058+0000 7f9bf0c25640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:16:11.168 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:16:11.168 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":301,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:15:34.996693+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:11:24.652666+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:13:32.603417+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:15:09.310704+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:16:11.660 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7365 8 365281 365273 0.387438 0 - 0.0299513 2026-02-23T15:16:12.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:16:12 trial068 ceph-mon[38805]: pgmap v4352: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:16:12.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:16:12 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T15:16:12.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:16:12 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T15:16:12.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:16:12 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2264111745' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:16:12.660 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7366 8 365281 365273 0.387385 0 - 0.0299513 2026-02-23T15:16:13.660 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7367 8 365281 365273 0.387332 0 - 0.0299513 2026-02-23T15:16:14.207 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:16:14.232 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:16:13 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:16:13] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:16:14.233 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:16:14 trial068 ceph-mon[38805]: pgmap v4353: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:16:14.326 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:16:14.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:16:13 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:16:13.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:16:14.427 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:16:14.425+0000 7f0bc4faa640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:16:14.427 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:16:14.425+0000 7f0bc4faa640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:16:14.537 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:16:14.537 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":301,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:15:34.996693+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:11:24.652666+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:13:32.603417+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:15:09.310704+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:16:14.660 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7368 8 365281 365273 0.38728 0 - 0.0299513 2026-02-23T15:16:15.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:16:15 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2791168263' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:16:15.660 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7369 8 365281 365273 0.387227 0 - 0.0299513 2026-02-23T15:16:16.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:16:16 trial068 ceph-mon[38805]: pgmap v4354: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:16:16.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:16:16 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T15:16:16.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:16:16 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T15:16:16.660 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7370 8 365281 365273 0.387175 0 - 0.0299513 2026-02-23T15:16:17.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:16:17 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2026-02-23T15:16:17.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:16:17 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-23T15:16:17.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:16:17 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2026-02-23T15:16:17.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:16:17 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T15:16:17.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:16:17 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2026-02-23T15:16:17.577 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:16:17.660 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7371 8 365281 365273 0.387122 0 - 0.0299513 2026-02-23T15:16:17.694 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:16:17.798 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:16:17.796+0000 7f40de840640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:16:17.798 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:16:17.796+0000 7f40de840640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:16:17.906 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:16:17.907 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":301,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:15:34.996693+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:11:24.652666+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:13:32.603417+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:15:09.310704+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:16:18.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:16:18 trial068 ceph-mon[38805]: pgmap v4355: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:16:18.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:16:18 trial068 ceph-mon[38805]: pgmap v4356: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:16:18.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:16:18 trial068 ceph-mon[38805]: pgmap v4357: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:16:18.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:16:18 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3459729972' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:16:18.660 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7372 8 365281 365273 0.38707 0 - 0.0299513 2026-02-23T15:16:19.661 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7373 8 365281 365273 0.387017 0 - 0.0299513 2026-02-23T15:16:20.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:16:20 trial068 ceph-mon[38805]: pgmap v4358: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:16:20.661 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7374 8 365281 365273 0.386965 0 - 0.0299513 2026-02-23T15:16:20.944 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:16:21.061 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:16:21.156 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:16:21.154+0000 7f0116ddc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:16:21.156 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:16:21.155+0000 7f0116ddc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:16:21.265 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:16:21.265 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":301,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:15:34.996693+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:11:24.652666+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:13:32.603417+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:15:09.310704+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:16:21.266 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:16:21 trial068 ceph-mon[38805]: 5.a scrub starts 2026-02-23T15:16:21.266 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:16:21 trial068 ceph-mon[38805]: 5.a scrub ok 2026-02-23T15:16:21.661 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7375 8 365281 365273 0.386912 0 - 0.0299513 2026-02-23T15:16:22.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:16:22 trial068 ceph-mon[38805]: pgmap v4359: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:16:22.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:16:22 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/61873397' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:16:22.661 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7376 8 365281 365273 0.38686 0 - 0.0299513 2026-02-23T15:16:23.661 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7377 8 365281 365273 0.386807 0 - 0.0299513 2026-02-23T15:16:24.302 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:16:24.326 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:16:24 trial068 ceph-mon[38805]: pgmap v4360: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:16:24.327 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:16:23 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:16:23] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:16:24.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:16:23 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:16:23.982Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:16:24.419 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:16:24.525 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:16:24.523+0000 7fcd353d8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:16:24.525 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:16:24.524+0000 7fcd353d8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:16:24.639 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:16:24.639 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":301,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:15:34.996693+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:11:24.652666+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:13:32.603417+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:15:09.310704+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:16:24.661 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7378 8 365281 365273 0.386755 0 - 0.0299513 2026-02-23T15:16:25.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:16:25 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/4239291997' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:16:25.661 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7379 8 365281 365273 0.386702 0 - 0.0299513 2026-02-23T15:16:26.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:16:26 trial068 ceph-mon[38805]: pgmap v4361: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:16:26.661 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T15:16:26.661071+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T15:16:26.661 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T15:16:26.661 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7380 8 365281 365273 0.38665 0 - 0.0299513 2026-02-23T15:16:27.661 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7381 8 365281 365273 0.386598 0 - 0.0299513 2026-02-23T15:16:27.678 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:16:27.797 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:16:27.893 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:16:27.892+0000 7faa69116640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:16:27.894 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:16:27.892+0000 7faa69116640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:16:28.002 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:16:28.002 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":301,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:15:34.996693+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:11:24.652666+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:13:32.603417+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:15:09.310704+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:16:28.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:16:28 trial068 ceph-mon[38805]: pgmap v4362: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:16:28.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:16:28 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/583615744' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:16:28.661 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7382 8 365281 365273 0.386545 0 - 0.0299513 2026-02-23T15:16:29.661 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7383 8 365281 365273 0.386493 0 - 0.0299513 2026-02-23T15:16:30.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:16:30 trial068 ceph-mon[38805]: pgmap v4363: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:16:30.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:16:30 trial068 ceph-mon[38805]: 5.7 scrub starts 2026-02-23T15:16:30.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:16:30 trial068 ceph-mon[38805]: 5.7 scrub ok 2026-02-23T15:16:30.661 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7384 8 365281 365273 0.386441 0 - 0.0299513 2026-02-23T15:16:31.040 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:16:31.160 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:16:31.259 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:16:31.257+0000 7f87ccec9640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:16:31.259 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:16:31.258+0000 7f87ccec9640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:16:31.368 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:16:31.368 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":301,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:15:34.996693+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:11:24.652666+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:13:32.603417+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:15:09.310704+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:16:31.661 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7385 8 365281 365273 0.386388 0 - 0.0299513 2026-02-23T15:16:32.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:16:32 trial068 ceph-mon[38805]: pgmap v4364: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:16:32.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:16:32 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/345270408' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:16:32.661 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7386 8 365281 365273 0.386336 0 - 0.0299513 2026-02-23T15:16:33.661 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7387 8 365281 365273 0.386284 0 - 0.0299513 2026-02-23T15:16:34.265 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:16:33 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:16:33.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:16:34.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:16:34 trial068 ceph-mon[38805]: pgmap v4365: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:16:34.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:16:33 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:16:33] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:16:34.406 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:16:34.531 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:16:34.639 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:16:34.637+0000 7f143f310640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:16:34.639 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:16:34.637+0000 7f143f310640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:16:34.661 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7388 8 365281 365273 0.386231 0 - 0.0299513 2026-02-23T15:16:34.746 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:16:34.747 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":301,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:15:34.996693+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:11:24.652666+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:13:32.603417+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:15:09.310704+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:16:35.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:16:35 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2333153750' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:16:35.661 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7389 8 365281 365273 0.386179 0 - 0.0299513 2026-02-23T15:16:36.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:16:36 trial068 ceph-mon[38805]: 5.8 scrub starts 2026-02-23T15:16:36.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:16:36 trial068 ceph-mon[38805]: 5.8 scrub ok 2026-02-23T15:16:36.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:16:36 trial068 ceph-mon[38805]: pgmap v4366: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:16:36.661 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7390 8 365281 365273 0.386127 0 - 0.0299513 2026-02-23T15:16:37.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:16:37 trial068 ceph-mon[38805]: pgmap v4367: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:16:37.661 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7391 8 365281 365273 0.386075 0 - 0.0299513 2026-02-23T15:16:37.783 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:16:37.901 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:16:37.997 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:16:37.995+0000 7f1a9be30640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:16:37.997 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:16:37.995+0000 7f1a9be30640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:16:38.106 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:16:38.106 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":301,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:15:34.996693+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:11:24.652666+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:13:32.603417+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:15:09.310704+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:16:38.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:16:38 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2158709679' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:16:38.661 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7392 8 365281 365273 0.386022 0 - 0.0299513 2026-02-23T15:16:39.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:16:39 trial068 ceph-mon[38805]: pgmap v4368: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:16:39.661 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7393 8 365281 365273 0.38597 0 - 0.0299513 2026-02-23T15:16:40.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:16:40 trial068 ceph-mon[38805]: osdmap e302: 4 total, 3 up, 4 in 2026-02-23T15:16:40.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:16:40 trial068 ceph-mon[38805]: osdmap e303: 4 total, 3 up, 4 in 2026-02-23T15:16:40.662 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7394 8 365281 365273 0.385918 0 - 0.0299513 2026-02-23T15:16:41.159 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:16:41.276 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:16:41.373 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:16:41.372+0000 7f2ef8711640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:16:41.373 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:16:41.372+0000 7f2ef8711640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:16:41.482 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:16:41.482 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":303,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:16:40.005952+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:15:43.826304+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:13:32.603417+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:15:09.310704+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:16:41.483 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:16:41 trial068 ceph-mon[38805]: pgmap v4371: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:16:41.662 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7395 8 365281 365273 0.385866 0 - 0.0299513 2026-02-23T15:16:42.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:16:42 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/523308235' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:16:42.662 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7396 8 365281 365273 0.385814 0 - 0.0299513 2026-02-23T15:16:43.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:16:43 trial068 ceph-mon[38805]: pgmap v4372: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:16:43.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:16:43 trial068 ceph-mon[38805]: 5.5 scrub starts 2026-02-23T15:16:43.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:16:43 trial068 ceph-mon[38805]: 5.5 scrub ok 2026-02-23T15:16:43.662 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7397 8 365281 365273 0.385761 0 - 0.0299513 2026-02-23T15:16:44.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:16:43 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:16:43] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:16:44.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:16:43 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:16:43.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:16:44.520 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:16:44.637 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:16:44.662 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7398 8 365281 365273 0.385709 0 - 0.0299513 2026-02-23T15:16:44.736 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:16:44.734+0000 7f6f71dc6640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:16:44.736 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:16:44.734+0000 7f6f71dc6640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:16:44.846 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:16:44.846 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":303,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:16:40.005952+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:15:43.826304+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:13:32.603417+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:15:09.310704+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:16:45.662 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7399 8 365281 365273 0.385657 0 - 0.0299513 2026-02-23T15:16:46.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:16:45 trial068 ceph-mon[38805]: pgmap v4373: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:16:46.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:16:45 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1908286431' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:16:46.662 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T15:16:46.662105+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T15:16:46.662 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T15:16:46.662 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7400 8 365281 365273 0.385605 0 - 0.0299513 2026-02-23T15:16:47.662 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7401 8 365281 365273 0.385553 0 - 0.0299513 2026-02-23T15:16:47.883 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:16:47.908 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:16:47 trial068 ceph-mon[38805]: pgmap v4374: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:16:48.002 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:16:48.103 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:16:48.101+0000 7f7f3b930640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:16:48.103 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:16:48.102+0000 7f7f3b930640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:16:48.213 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:16:48.213 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":303,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:16:40.005952+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:15:43.826304+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:13:32.603417+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:15:09.310704+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:16:48.662 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7402 8 365281 365273 0.385501 0 - 0.0299513 2026-02-23T15:16:49.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:16:48 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2962523305' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:16:49.662 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7403 8 365281 365273 0.385449 0 - 0.0299513 2026-02-23T15:16:50.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:16:49 trial068 ceph-mon[38805]: pgmap v4375: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:16:50.662 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7404 8 365281 365273 0.385397 0 - 0.0299513 2026-02-23T15:16:51.250 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:16:51.370 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:16:51.467 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:16:51.465+0000 7f1a36c5a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:16:51.467 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:16:51.466+0000 7f1a36c5a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:16:51.576 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:16:51.577 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":303,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:16:40.005952+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:15:43.826304+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:13:32.603417+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:15:09.310704+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:16:51.662 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7405 8 365281 365273 0.385345 0 - 0.0299513 2026-02-23T15:16:52.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:16:52 trial068 ceph-mon[38805]: pgmap v4376: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:16:52.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:16:52 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T15:16:52.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:16:52 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T15:16:52.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:16:52 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1101050869' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:16:52.662 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7406 8 365281 365273 0.385293 0 - 0.0299513 2026-02-23T15:16:53.662 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7407 8 365281 365273 0.385241 0 - 0.0299513 2026-02-23T15:16:54.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:16:54 trial068 ceph-mon[38805]: pgmap v4377: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:16:54.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:16:53 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:16:53] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:16:54.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:16:53 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:16:53.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:16:54.613 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:16:54.662 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7408 8 365281 365273 0.385189 0 - 0.0299513 2026-02-23T15:16:54.732 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:16:54.834 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:16:54.833+0000 7fc8a6f2b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:16:54.835 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:16:54.833+0000 7fc8a6f2b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:16:54.943 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:16:54.944 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":303,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:16:40.005952+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:15:43.826304+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:13:32.603417+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:15:09.310704+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:16:55.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:16:55 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/4261051487' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:16:55.663 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7409 8 365281 365273 0.385137 0 - 0.0299513 2026-02-23T15:16:56.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:16:56 trial068 ceph-mon[38805]: pgmap v4378: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:16:56.663 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7410 8 365281 365273 0.385085 0 - 0.0299513 2026-02-23T15:16:57.663 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7411 8 365281 365273 0.385033 0 - 0.0299513 2026-02-23T15:16:57.980 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:16:58.100 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:16:58.202 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:16:58.200+0000 7fd910904640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:16:58.202 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:16:58.200+0000 7fd910904640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:16:58.310 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:16:58.311 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":303,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:16:40.005952+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:15:43.826304+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:13:32.603417+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:15:09.310704+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:16:58.311 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:16:58 trial068 ceph-mon[38805]: pgmap v4379: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:16:58.663 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7412 8 365281 365273 0.384981 0 - 0.0299513 2026-02-23T15:16:59.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:16:59 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2693661713' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:16:59.663 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7413 8 365281 365273 0.384929 0 - 0.0299513 2026-02-23T15:17:00.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:17:00 trial068 ceph-mon[38805]: pgmap v4380: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:17:00.663 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7414 8 365281 365273 0.384877 0 - 0.0299513 2026-02-23T15:17:01.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:17:01 trial068 ceph-mon[38805]: 5.e scrub starts 2026-02-23T15:17:01.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:17:01 trial068 ceph-mon[38805]: 5.e scrub ok 2026-02-23T15:17:01.348 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:17:01.467 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:17:01.563 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:17:01.561+0000 7ff986366640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:17:01.563 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:17:01.561+0000 7ff986366640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:17:01.663 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7415 8 365281 365273 0.384825 0 - 0.0299513 2026-02-23T15:17:01.671 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:17:01.672 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":303,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:16:40.005952+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:15:43.826304+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:13:32.603417+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:15:09.310704+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:17:02.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:17:02 trial068 ceph-mon[38805]: pgmap v4381: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:17:02.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:17:02 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/12196936' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:17:02.663 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7416 8 365281 365273 0.384773 0 - 0.0299513 2026-02-23T15:17:03.663 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7417 8 365281 365273 0.384721 0 - 0.0299513 2026-02-23T15:17:04.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:17:04 trial068 ceph-mon[38805]: pgmap v4382: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:17:04.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:17:03 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:17:03] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:17:04.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:17:03 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:17:03.974Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:17:04.663 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7418 8 365281 365273 0.384669 0 - 0.0299513 2026-02-23T15:17:04.710 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:17:04.827 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:17:04.924 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:17:04.922+0000 7f35daba9640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:17:04.924 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:17:04.923+0000 7f35daba9640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:17:05.032 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:17:05.032 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":303,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:16:40.005952+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:15:43.826304+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:13:32.603417+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:15:09.310704+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:17:05.663 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7419 8 365281 365273 0.384618 0 - 0.0299513 2026-02-23T15:17:06.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:17:06 trial068 ceph-mon[38805]: pgmap v4383: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:17:06.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:17:06 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1628410129' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:17:06.663 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T15:17:06.663477+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T15:17:06.663 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T15:17:06.664 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7420 8 365281 365273 0.384566 0 - 0.0299513 2026-02-23T15:17:07.663 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7421 8 365281 365273 0.384514 0 - 0.0299513 2026-02-23T15:17:08.069 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:17:08.186 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:17:08.284 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:17:08.282+0000 7f0f44468640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:17:08.284 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:17:08.283+0000 7f0f44468640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:17:08.284 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:17:08 trial068 ceph-mon[38805]: pgmap v4384: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:17:08.393 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:17:08.393 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":303,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:16:40.005952+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:15:43.826304+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:13:32.603417+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:15:09.310704+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:17:08.663 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7422 8 365281 365273 0.384462 0 - 0.0299513 2026-02-23T15:17:09.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:17:09 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3852693613' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:17:09.663 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7423 8 365281 365273 0.38441 0 - 0.0299513 2026-02-23T15:17:10.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:17:10 trial068 ceph-mon[38805]: pgmap v4385: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:17:10.664 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7424 8 365281 365273 0.384359 0 - 0.0299513 2026-02-23T15:17:11.429 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:17:11.547 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:17:11.650 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:17:11.649+0000 7f53bf427640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:17:11.651 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:17:11.649+0000 7f53bf427640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:17:11.664 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7425 8 365281 365273 0.384307 0 - 0.0299513 2026-02-23T15:17:11.761 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:17:11.762 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":303,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:16:40.005952+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:15:43.826304+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:13:32.603417+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:15:09.310704+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:17:12.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:17:12 trial068 ceph-mon[38805]: pgmap v4386: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:17:12.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:17:12 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/613643259' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:17:12.664 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7426 8 365281 365273 0.384255 0 - 0.0299513 2026-02-23T15:17:13.664 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7427 8 365281 365273 0.384203 0 - 0.0299513 2026-02-23T15:17:14.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:17:14 trial068 ceph-mon[38805]: pgmap v4387: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:17:14.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:17:13 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:17:13] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:17:14.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:17:13 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:17:13.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:17:14.664 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7428 8 365281 365273 0.384152 0 - 0.0299513 2026-02-23T15:17:14.800 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:17:14.916 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:17:15.016 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:17:15.015+0000 7fdd1c479640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:17:15.016 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:17:15.015+0000 7fdd1c479640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:17:15.125 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:17:15.126 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":303,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:16:40.005952+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:15:43.826304+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:13:32.603417+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:15:09.310704+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:17:15.664 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7429 8 365281 365273 0.3841 0 - 0.0299513 2026-02-23T15:17:16.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:17:16 trial068 ceph-mon[38805]: pgmap v4388: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:17:16.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:17:16 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2744006444' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:17:16.664 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7430 8 365281 365273 0.384048 0 - 0.0299513 2026-02-23T15:17:17.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:17:17 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2026-02-23T15:17:17.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:17:17 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T15:17:17.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:17:17 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T15:17:17.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:17:17 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T15:17:17.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:17:17 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T15:17:17.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:17:17 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-23T15:17:17.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:17:17 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2026-02-23T15:17:17.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:17:17 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T15:17:17.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:17:17 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2026-02-23T15:17:17.664 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7431 8 365281 365273 0.383996 0 - 0.0299513 2026-02-23T15:17:18.163 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:17:18.281 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:17:18.303 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:17:18 trial068 ceph-mon[38805]: pgmap v4389: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:17:18.303 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:17:18 trial068 ceph-mon[38805]: pgmap v4390: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:17:18.303 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:17:18 trial068 ceph-mon[38805]: pgmap v4391: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:17:18.383 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:17:18.382+0000 7fb62830c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:17:18.383 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:17:18.382+0000 7fb62830c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:17:18.496 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:17:18.496 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":303,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:16:40.005952+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:15:43.826304+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:13:32.603417+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:15:09.310704+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:17:18.664 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7432 8 365281 365273 0.383945 0 - 0.0299513 2026-02-23T15:17:19.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:17:19 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2096596841' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:17:19.664 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7433 8 365281 365273 0.383893 0 - 0.0299513 2026-02-23T15:17:20.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:17:20 trial068 ceph-mon[38805]: pgmap v4392: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:17:20.664 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7434 8 365281 365273 0.383842 0 - 0.0299513 2026-02-23T15:17:21.534 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:17:21.652 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:17:21.664 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7435 8 365281 365273 0.38379 0 - 0.0299513 2026-02-23T15:17:21.750 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:17:21.749+0000 7f816fea4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:17:21.750 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:17:21.749+0000 7f816fea4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:17:21.860 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:17:21.860 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":303,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:16:40.005952+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:15:43.826304+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:13:32.603417+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:15:09.310704+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:17:22.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:17:22 trial068 ceph-mon[38805]: pgmap v4393: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:17:22.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:17:22 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/826440483' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:17:22.664 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7436 8 365281 365273 0.383738 0 - 0.0299513 2026-02-23T15:17:23.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:17:23 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T15:17:23.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:17:23 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T15:17:23.664 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7437 8 365281 365273 0.383687 0 - 0.0299513 2026-02-23T15:17:24.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:17:24 trial068 ceph-mon[38805]: pgmap v4394: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:17:24.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:17:23 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:17:23] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:17:24.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:17:23 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:17:23.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:17:24.665 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7438 8 365281 365273 0.383635 0 - 0.0299513 2026-02-23T15:17:24.896 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:17:25.014 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:17:25.115 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:17:25.114+0000 7f69bf6fe640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:17:25.115 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:17:25.114+0000 7f69bf6fe640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:17:25.224 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:17:25.224 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":303,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:16:40.005952+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:15:43.826304+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:13:32.603417+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:15:09.310704+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:17:25.665 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7439 8 365281 365273 0.383584 0 - 0.0299513 2026-02-23T15:17:26.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:17:26 trial068 ceph-mon[38805]: pgmap v4395: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:17:26.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:17:26 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/96150271' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:17:26.665 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T15:17:26.664942+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T15:17:26.665 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T15:17:26.665 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7440 8 365281 365273 0.383532 0 - 0.0299513 2026-02-23T15:17:27.665 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7441 8 365281 365273 0.38348 0 - 0.0299513 2026-02-23T15:17:28.261 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:17:28.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:17:28 trial068 ceph-mon[38805]: pgmap v4396: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:17:28.380 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:17:28.484 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:17:28.482+0000 7fc26798e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:17:28.484 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:17:28.483+0000 7fc26798e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:17:28.594 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:17:28.594 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":303,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:16:40.005952+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:15:43.826304+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:13:32.603417+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:15:09.310704+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:17:28.665 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7442 8 365281 365273 0.383429 0 - 0.0299513 2026-02-23T15:17:29.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:17:29 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3027048629' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:17:29.665 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7443 8 365281 365273 0.383377 0 - 0.0299513 2026-02-23T15:17:30.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:17:30 trial068 ceph-mon[38805]: pgmap v4397: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:17:30.665 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7444 8 365281 365273 0.383326 0 - 0.0299513 2026-02-23T15:17:31.630 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:17:31.665 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7445 8 365281 365273 0.383274 0 - 0.0299513 2026-02-23T15:17:31.747 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:17:31.846 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:17:31.845+0000 7facd162e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:17:31.847 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:17:31.845+0000 7facd162e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:17:31.957 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:17:31.957 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":303,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:16:40.005952+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:15:43.826304+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:13:32.603417+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:15:09.310704+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:17:32.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:17:32 trial068 ceph-mon[38805]: 5.a scrub starts 2026-02-23T15:17:32.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:17:32 trial068 ceph-mon[38805]: 5.a scrub ok 2026-02-23T15:17:32.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:17:32 trial068 ceph-mon[38805]: pgmap v4398: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:17:32.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:17:32 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1547382559' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:17:32.665 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7446 8 365281 365273 0.383223 0 - 0.0299513 2026-02-23T15:17:33.665 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7447 8 365281 365273 0.383171 0 - 0.0299513 2026-02-23T15:17:34.265 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:17:33 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:17:33.974Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:17:34.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:17:34 trial068 ceph-mon[38805]: pgmap v4399: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:17:34.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:17:33 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:17:33] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:17:34.665 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7448 8 365281 365273 0.38312 0 - 0.0299513 2026-02-23T15:17:35.010 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:17:35.129 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:17:35.237 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:17:35.235+0000 7f6e6fadb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:17:35.237 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:17:35.236+0000 7f6e6fadb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:17:35.347 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:17:35.347 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":303,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:16:40.005952+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:15:43.826304+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:13:32.603417+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:15:09.310704+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:17:35.666 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7449 8 365281 365273 0.383069 0 - 0.0299513 2026-02-23T15:17:36.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:17:36 trial068 ceph-mon[38805]: pgmap v4400: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:17:36.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:17:36 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3185510139' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:17:36.666 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7450 8 365281 365273 0.383017 0 - 0.0299513 2026-02-23T15:17:37.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:17:37 trial068 ceph-mon[38805]: pgmap v4401: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:17:37.666 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7451 8 365281 365273 0.382966 0 - 0.0299513 2026-02-23T15:17:38.387 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:17:38.505 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:17:38.607 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:17:38.606+0000 7f262b3c7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:17:38.608 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:17:38.606+0000 7f262b3c7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:17:38.666 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7452 8 365281 365273 0.382914 0 - 0.0299513 2026-02-23T15:17:38.715 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:17:38.716 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":303,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:16:40.005952+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:15:43.826304+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:13:32.603417+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:15:09.310704+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:17:39.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:17:38 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/17038698' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:17:39.666 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7453 8 365281 365273 0.382863 0 - 0.0299513 2026-02-23T15:17:40.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:17:39 trial068 ceph-mon[38805]: pgmap v4402: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:17:40.666 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7454 8 365281 365273 0.382812 0 - 0.0299513 2026-02-23T15:17:41.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:17:41 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T15:17:41.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:17:41 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T15:17:41.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:17:41 trial068 ceph-mon[38805]: 5.7 scrub starts 2026-02-23T15:17:41.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:17:41 trial068 ceph-mon[38805]: 5.7 scrub ok 2026-02-23T15:17:41.666 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7455 8 365281 365273 0.38276 0 - 0.0299513 2026-02-23T15:17:41.752 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:17:41.869 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:17:41.969 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:17:41.967+0000 7f2e110a4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:17:41.969 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:17:41.967+0000 7f2e110a4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:17:42.076 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:17:42.077 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":303,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:16:40.005952+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:15:43.826304+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:13:32.603417+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:15:09.310704+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:17:42.077 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:17:42 trial068 ceph-mon[38805]: pgmap v4403: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:17:42.666 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7456 8 365281 365273 0.382709 0 - 0.0299513 2026-02-23T15:17:43.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:17:43 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1348357045' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:17:43.666 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7457 8 365281 365273 0.382658 0 - 0.0299513 2026-02-23T15:17:44.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:17:44 trial068 ceph-mon[38805]: pgmap v4404: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:17:44.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:17:43 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:17:43] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:17:44.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:17:43 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:17:43.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:17:44.666 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7458 8 365281 365273 0.382606 0 - 0.0299513 2026-02-23T15:17:45.114 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:17:45.233 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:17:45.329 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:17:45.328+0000 7f574735c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:17:45.330 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:17:45.328+0000 7f574735c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:17:45.440 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:17:45.440 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":303,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:16:40.005952+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:15:43.826304+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:13:32.603417+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:15:09.310704+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:17:45.666 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7459 8 365281 365273 0.382555 0 - 0.0299513 2026-02-23T15:17:46.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:17:46 trial068 ceph-mon[38805]: pgmap v4405: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:17:46.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:17:46 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/359715383' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:17:46.666 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T15:17:46.666464+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T15:17:46.667 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T15:17:46.667 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7460 8 365281 365273 0.382504 0 - 0.0299513 2026-02-23T15:17:47.666 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7461 8 365281 365273 0.382452 0 - 0.0299513 2026-02-23T15:17:48.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:17:48 trial068 ceph-mon[38805]: pgmap v4406: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:17:48.476 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:17:48.594 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:17:48.667 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7462 8 365281 365273 0.382401 0 - 0.0299513 2026-02-23T15:17:48.693 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:17:48.691+0000 7faf6f508640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:17:48.693 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:17:48.692+0000 7faf6f508640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:17:48.801 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:17:48.801 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":303,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:16:40.005952+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:15:43.826304+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:13:32.603417+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:15:09.310704+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:17:49.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:17:49 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/66499049' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:17:49.667 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7463 8 365281 365273 0.38235 0 - 0.0299513 2026-02-23T15:17:50.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:17:50 trial068 ceph-mon[38805]: pgmap v4407: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:17:50.667 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7464 8 365281 365273 0.382299 0 - 0.0299513 2026-02-23T15:17:51.667 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7465 8 365281 365273 0.382248 0 - 0.0299513 2026-02-23T15:17:51.841 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:17:51.957 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:17:52.056 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:17:52.054+0000 7f782f4d3640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:17:52.056 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:17:52.054+0000 7f782f4d3640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:17:52.145 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:17:52 trial068 ceph-mon[38805]: pgmap v4408: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:17:52.167 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:17:52.167 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":303,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:16:40.005952+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:15:43.826304+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:13:32.603417+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:15:09.310704+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:17:52.667 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7466 8 365281 365273 0.382196 0 - 0.0299513 2026-02-23T15:17:53.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:17:53 trial068 ceph-mon[38805]: 5.8 scrub starts 2026-02-23T15:17:53.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:17:53 trial068 ceph-mon[38805]: 5.8 scrub ok 2026-02-23T15:17:53.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:17:53 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2164614500' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:17:53.667 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7467 8 365281 365273 0.382145 0 - 0.0299513 2026-02-23T15:17:54.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:17:54 trial068 ceph-mon[38805]: pgmap v4409: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:17:54.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:17:53 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:17:53] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:17:54.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:17:53 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:17:53.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:17:54.667 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7468 8 365281 365273 0.382094 0 - 0.0299513 2026-02-23T15:17:55.205 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:17:55.323 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:17:55.422 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:17:55.420+0000 7f6c92ec3640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:17:55.422 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:17:55.420+0000 7f6c92ec3640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:17:55.530 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:17:55.530 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":303,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:16:40.005952+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:15:43.826304+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:13:32.603417+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:15:09.310704+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:17:55.667 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7469 8 365281 365273 0.382043 0 - 0.0299513 2026-02-23T15:17:56.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:17:56 trial068 ceph-mon[38805]: pgmap v4410: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:17:56.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:17:56 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3875694123' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:17:56.667 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7470 8 365281 365273 0.381992 0 - 0.0299513 2026-02-23T15:17:57.667 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7471 8 365281 365273 0.381941 0 - 0.0299513 2026-02-23T15:17:58.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:17:58 trial068 ceph-mon[38805]: pgmap v4411: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:17:58.568 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:17:58.667 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7472 8 365281 365273 0.381889 0 - 0.0299513 2026-02-23T15:17:58.690 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:17:58.791 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:17:58.789+0000 7f9890ccc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:17:58.791 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:17:58.789+0000 7f9890ccc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:17:58.900 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:17:58.900 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":303,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:16:40.005952+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:15:43.826304+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:13:32.603417+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:15:09.310704+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:17:59.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:17:59 trial068 ceph-mon[38805]: 5.5 scrub starts 2026-02-23T15:17:59.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:17:59 trial068 ceph-mon[38805]: 5.5 scrub ok 2026-02-23T15:17:59.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:17:59 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/4026813554' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:17:59.667 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7473 8 365281 365273 0.381838 0 - 0.0299513 2026-02-23T15:18:00.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:18:00 trial068 ceph-mon[38805]: pgmap v4412: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:18:00.667 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7474 8 365281 365273 0.381787 0 - 0.0299513 2026-02-23T15:18:01.667 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7475 8 365281 365273 0.381736 0 - 0.0299513 2026-02-23T15:18:01.937 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:18:02.054 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:18:02.162 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:18:02.161+0000 7f3662270640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:18:02.162 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:18:02.161+0000 7f3662270640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:18:02.271 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:18:02.272 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":303,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:16:40.005952+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:15:43.826304+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:13:32.603417+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:15:09.310704+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:18:02.272 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:18:02 trial068 ceph-mon[38805]: pgmap v4413: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:18:02.667 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7476 8 365281 365273 0.381685 0 - 0.0299513 2026-02-23T15:18:03.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:18:03 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1701704130' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:18:03.668 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7477 8 365281 365273 0.381634 0 - 0.0299513 2026-02-23T15:18:04.265 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:18:03 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:18:03.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:18:04.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:18:04 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T15:18:04.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:18:04 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T15:18:04.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:18:04 trial068 ceph-mon[38805]: pgmap v4414: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:18:04.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:18:03 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:18:03] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:18:04.668 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7478 8 365281 365273 0.381583 0 - 0.0299513 2026-02-23T15:18:05.308 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:18:05.427 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:18:05.526 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:18:05.525+0000 7f7502be5640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:18:05.526 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:18:05.525+0000 7f7502be5640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:18:05.635 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:18:05.635 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":303,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:16:40.005952+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:15:43.826304+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:13:32.603417+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:15:09.310704+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:18:05.668 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7479 8 365281 365273 0.381532 0 - 0.0299513 2026-02-23T15:18:06.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:18:06 trial068 ceph-mon[38805]: pgmap v4415: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:18:06.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:18:06 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/872572881' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:18:06.668 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T15:18:06.667967+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T15:18:06.668 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T15:18:06.668 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7480 8 365281 365273 0.381481 0 - 0.0299513 2026-02-23T15:18:07.668 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7481 8 365281 365273 0.38143 0 - 0.0299513 2026-02-23T15:18:08.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:18:08 trial068 ceph-mon[38805]: pgmap v4416: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:18:08.668 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7482 8 365281 365273 0.381379 0 - 0.0299513 2026-02-23T15:18:08.673 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:18:08.793 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:18:08.897 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:18:08.895+0000 7f89d38c9640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:18:08.897 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:18:08.895+0000 7f89d38c9640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:18:09.005 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:18:09.006 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":303,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:16:40.005952+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:15:43.826304+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:13:32.603417+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:15:09.310704+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:18:09.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:18:09 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2671614040' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:18:09.668 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7483 8 365281 365273 0.381328 0 - 0.0299513 2026-02-23T15:18:10.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:18:10 trial068 ceph-mon[38805]: pgmap v4417: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:18:10.668 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7484 8 365281 365273 0.381277 0 - 0.0299513 2026-02-23T15:18:11.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:18:11 trial068 ceph-mon[38805]: pgmap v4418: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:18:11.668 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7485 8 365281 365273 0.381226 0 - 0.0299513 2026-02-23T15:18:12.042 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:18:12.161 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:18:12.268 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:18:12.266+0000 7f53a59d6640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:18:12.268 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:18:12.267+0000 7f53a59d6640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:18:12.376 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:18:12.376 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":303,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:16:40.005952+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:15:43.826304+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:13:32.603417+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:15:09.310704+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:18:12.668 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7486 8 365281 365273 0.381175 0 - 0.0299513 2026-02-23T15:18:12.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:18:12 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2476714062' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:18:13.668 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7487 8 365281 365273 0.381124 0 - 0.0299513 2026-02-23T15:18:13.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:18:13 trial068 ceph-mon[38805]: pgmap v4419: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:18:14.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:18:13 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:18:13] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:18:14.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:18:13 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:18:13.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:18:14.668 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7488 8 365281 365273 0.381073 0 - 0.0299513 2026-02-23T15:18:15.429 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:18:15.548 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:18:15.647 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:18:15.645+0000 7fded4096640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:18:15.647 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:18:15.645+0000 7fded4096640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:18:15.668 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7489 8 365281 365273 0.381023 0 - 0.0299513 2026-02-23T15:18:15.759 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:18:15.759 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":303,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:16:40.005952+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:15:43.826304+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:13:32.603417+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:15:09.310704+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:18:16.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:18:16 trial068 ceph-mon[38805]: pgmap v4420: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:18:16.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:18:16 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3982205243' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:18:16.669 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7490 8 365281 365273 0.380972 0 - 0.0299513 2026-02-23T15:18:17.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:18:17 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2026-02-23T15:18:17.669 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7491 8 365281 365273 0.380921 0 - 0.0299513 2026-02-23T15:18:18.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:18:18 trial068 ceph-mon[38805]: pgmap v4421: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:18:18.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:18:18 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T15:18:18.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:18:18 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T15:18:18.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:18:18 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T15:18:18.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:18:18 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T15:18:18.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:18:18 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-23T15:18:18.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:18:18 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2026-02-23T15:18:18.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:18:18 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T15:18:18.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:18:18 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2026-02-23T15:18:18.669 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7492 8 365281 365273 0.38087 0 - 0.0299513 2026-02-23T15:18:18.799 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:18:18.919 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:18:19.021 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:18:19.020+0000 7f9e04b6f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:18:19.021 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:18:19.020+0000 7f9e04b6f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:18:19.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:18:19 trial068 ceph-mon[38805]: pgmap v4422: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:18:19.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:18:19 trial068 ceph-mon[38805]: pgmap v4423: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:18:19.130 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:18:19.130 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":303,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:16:40.005952+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:15:43.826304+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:13:32.603417+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:15:09.310704+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:18:19.669 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7493 8 365281 365273 0.380819 0 - 0.0299513 2026-02-23T15:18:20.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:18:20 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/525427943' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:18:20.669 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7494 8 365281 365273 0.380768 0 - 0.0299513 2026-02-23T15:18:21.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:18:21 trial068 ceph-mon[38805]: pgmap v4424: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:18:21.669 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7495 8 365281 365273 0.380718 0 - 0.0299513 2026-02-23T15:18:22.169 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:18:22.285 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:18:22.384 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:18:22.382+0000 7f74ae86f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:18:22.384 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:18:22.382+0000 7f74ae86f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:18:22.493 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:18:22.493 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":303,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:16:40.005952+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:15:43.826304+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:13:32.603417+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:15:09.310704+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:18:22.669 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7496 8 365281 365273 0.380667 0 - 0.0299513 2026-02-23T15:18:23.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:18:23 trial068 ceph-mon[38805]: pgmap v4425: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:18:23.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:18:23 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3140016267' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:18:23.669 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7497 8 365281 365273 0.380616 0 - 0.0299513 2026-02-23T15:18:24.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:18:23 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:18:23] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:18:24.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:18:23 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:18:23.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:18:24.669 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7498 8 365281 365273 0.380565 0 - 0.0299513 2026-02-23T15:18:25.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:18:25 trial068 ceph-mon[38805]: pgmap v4426: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:18:25.530 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:18:25.649 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:18:25.669 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7499 8 365281 365273 0.380514 0 - 0.0299513 2026-02-23T15:18:25.749 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:18:25.747+0000 7fcafbf2f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:18:25.749 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:18:25.747+0000 7fcafbf2f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:18:25.859 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:18:25.859 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":303,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:16:40.005952+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:15:43.826304+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:13:32.603417+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:15:09.310704+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:18:26.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:18:26 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2590983397' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:18:26.669 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T15:18:26.669404+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T15:18:26.669 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T15:18:26.670 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7500 8 365281 365273 0.380464 0 - 0.0299513 2026-02-23T15:18:27.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:18:27 trial068 ceph-mon[38805]: pgmap v4427: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:18:27.669 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7501 8 365281 365273 0.380413 0 - 0.0299513 2026-02-23T15:18:28.669 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7502 8 365281 365273 0.380362 0 - 0.0299513 2026-02-23T15:18:28.897 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:18:29.015 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:18:29.115 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:18:29.113+0000 7feb74916640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:18:29.115 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:18:29.113+0000 7feb74916640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:18:29.224 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:18:29.224 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":303,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:16:40.005952+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:15:43.826304+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:13:32.603417+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:15:09.310704+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:18:29.225 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:18:29 trial068 ceph-mon[38805]: pgmap v4428: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:18:29.669 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7503 8 365281 365273 0.380312 0 - 0.0299513 2026-02-23T15:18:30.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:18:30 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1314028703' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:18:30.670 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7504 8 365281 365273 0.380261 0 - 0.0299513 2026-02-23T15:18:31.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:18:31 trial068 ceph-mon[38805]: pgmap v4429: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:18:31.670 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7505 8 365281 365273 0.38021 0 - 0.0299513 2026-02-23T15:18:32.262 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:18:32.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:18:32 trial068 ceph-mon[38805]: 5.e scrub starts 2026-02-23T15:18:32.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:18:32 trial068 ceph-mon[38805]: 5.e scrub ok 2026-02-23T15:18:32.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:18:32 trial068 ceph-mon[38805]: pgmap v4430: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:18:32.381 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:18:32.488 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:18:32.487+0000 7f5c2989b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:18:32.489 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:18:32.487+0000 7f5c2989b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:18:32.597 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:18:32.597 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":303,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:16:40.005952+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:15:43.826304+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:13:32.603417+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:15:09.310704+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:18:32.670 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7506 8 365281 365273 0.38016 0 - 0.0299513 2026-02-23T15:18:33.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:18:33 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2985601207' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:18:33.670 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7507 8 365281 365273 0.380109 0 - 0.0299513 2026-02-23T15:18:34.265 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:18:33 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:18:33.972Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:18:34.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:18:34 trial068 ceph-mon[38805]: pgmap v4431: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:18:34.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:18:33 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:18:33] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:18:34.670 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7508 8 365281 365273 0.380058 0 - 0.0299513 2026-02-23T15:18:35.635 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:18:35.670 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7509 8 365281 365273 0.380008 0 - 0.0299513 2026-02-23T15:18:35.752 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:18:35.852 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:18:35.850+0000 7ff9bf403640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:18:35.852 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:18:35.851+0000 7ff9bf403640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:18:35.960 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:18:35.960 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":303,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:16:40.005952+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:15:43.826304+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:13:32.603417+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:15:09.310704+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:18:36.670 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7510 8 365281 365273 0.379957 0 - 0.0299513 2026-02-23T15:18:37.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:18:36 trial068 ceph-mon[38805]: pgmap v4432: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:18:37.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:18:36 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2072012287' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:18:37.670 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7511 8 365281 365273 0.379907 0 - 0.0299513 2026-02-23T15:18:38.670 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7512 8 365281 365273 0.379856 0 - 0.0299513 2026-02-23T15:18:38.999 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:18:39.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:18:38 trial068 ceph-mon[38805]: pgmap v4433: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:18:39.118 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:18:39.215 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:18:39.214+0000 7f6416ac0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:18:39.215 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:18:39.214+0000 7f6416ac0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:18:39.325 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:18:39.325 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":303,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:16:40.005952+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:15:43.826304+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:13:32.603417+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:15:09.310704+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:18:39.670 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7513 8 365281 365273 0.379805 0 - 0.0299513 2026-02-23T15:18:40.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:18:39 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/26056192' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:18:40.670 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7514 8 365281 365273 0.379755 0 - 0.0299513 2026-02-23T15:18:41.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:18:41 trial068 ceph-mon[38805]: pgmap v4434: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:18:41.670 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7515 8 365281 365273 0.379704 0 - 0.0299513 2026-02-23T15:18:42.365 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:18:42.483 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:18:42.588 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:18:42.587+0000 7f7a91ce8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:18:42.588 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:18:42.587+0000 7f7a91ce8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:18:42.670 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7516 8 365281 365273 0.379654 0 - 0.0299513 2026-02-23T15:18:42.696 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:18:42.696 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":303,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:16:40.005952+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:15:43.826304+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:13:32.603417+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:15:09.310704+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:18:43.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:18:43 trial068 ceph-mon[38805]: pgmap v4435: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:18:43.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:18:43 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1561493436' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:18:43.671 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7517 8 365281 365273 0.379603 0 - 0.0299513 2026-02-23T15:18:44.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:18:43 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:18:43] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:18:44.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:18:43 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:18:43.974Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:18:44.671 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7518 8 365281 365273 0.379553 0 - 0.0299513 2026-02-23T15:18:45.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:18:45 trial068 ceph-mon[38805]: 5.a scrub starts 2026-02-23T15:18:45.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:18:45 trial068 ceph-mon[38805]: 5.a scrub ok 2026-02-23T15:18:45.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:18:45 trial068 ceph-mon[38805]: pgmap v4436: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:18:45.671 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7519 8 365281 365273 0.379502 0 - 0.0299513 2026-02-23T15:18:45.748 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:18:45.869 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:18:45.965 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:18:45.963+0000 7f05435c2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:18:45.965 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:18:45.964+0000 7f05435c2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:18:46.073 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:18:46.074 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":303,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:16:40.005952+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:15:43.826304+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:13:32.603417+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:15:09.310704+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:18:46.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:18:46 trial068 ceph-mon[38805]: 5.7 scrub starts 2026-02-23T15:18:46.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:18:46 trial068 ceph-mon[38805]: 5.7 scrub ok 2026-02-23T15:18:46.671 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T15:18:46.670934+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T15:18:46.671 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T15:18:46.671 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7520 8 365281 365273 0.379452 0 - 0.0299513 2026-02-23T15:18:47.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:18:47 trial068 ceph-mon[38805]: pgmap v4437: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:18:47.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:18:47 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3314653427' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:18:47.671 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7521 8 365281 365273 0.379401 0 - 0.0299513 2026-02-23T15:18:48.671 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7522 8 365281 365273 0.379351 0 - 0.0299513 2026-02-23T15:18:49.111 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:18:49.227 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:18:49.280 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:18:49 trial068 ceph-mon[38805]: pgmap v4438: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:18:49.326 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:18:49.324+0000 7ff29e6d6640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:18:49.326 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:18:49.324+0000 7ff29e6d6640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:18:49.435 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:18:49.435 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":303,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:16:40.005952+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:15:43.826304+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:13:32.603417+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:15:09.310704+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:18:49.671 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7523 8 365281 365273 0.379301 0 - 0.0299513 2026-02-23T15:18:50.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:18:50 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3371418394' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:18:50.671 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7524 8 365281 365273 0.37925 0 - 0.0299513 2026-02-23T15:18:51.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:18:51 trial068 ceph-mon[38805]: pgmap v4439: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:18:51.671 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7525 8 365281 365273 0.3792 0 - 0.0299513 2026-02-23T15:18:52.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:18:52 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T15:18:52.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:18:52 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T15:18:52.487 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:18:52.603 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:18:52.671 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7526 8 365281 365273 0.379149 0 - 0.0299513 2026-02-23T15:18:52.706 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:18:52.704+0000 7f03f9ae3640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:18:52.706 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:18:52.705+0000 7f03f9ae3640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:18:52.815 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:18:52.815 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":303,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:16:40.005952+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:15:43.826304+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:13:32.603417+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:15:09.310704+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:18:53.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:18:53 trial068 ceph-mon[38805]: pgmap v4440: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:18:53.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:18:53 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1249169140' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:18:53.671 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7527 8 365281 365273 0.379099 0 - 0.0299513 2026-02-23T15:18:54.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:18:53 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:18:53] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:18:54.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:18:53 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:18:53.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:18:54.671 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7528 8 365281 365273 0.379049 0 - 0.0299513 2026-02-23T15:18:55.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:18:55 trial068 ceph-mon[38805]: pgmap v4441: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:18:55.671 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7529 8 365281 365273 0.378998 0 - 0.0299513 2026-02-23T15:18:55.867 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:18:55.984 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:18:56.086 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:18:56.084+0000 7f06f6b5b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:18:56.086 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:18:56.084+0000 7f06f6b5b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:18:56.194 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:18:56.194 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":303,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:16:40.005952+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:15:43.826304+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:13:32.603417+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:15:09.310704+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:18:56.672 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7530 8 365281 365273 0.378948 0 - 0.0299513 2026-02-23T15:18:57.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:18:57 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T15:18:57.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:18:57 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T15:18:57.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:18:57 trial068 ceph-mon[38805]: pgmap v4442: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:18:57.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:18:57 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2266102022' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:18:57.672 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7531 8 365281 365273 0.378898 0 - 0.0299513 2026-02-23T15:18:58.672 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7532 8 365281 365273 0.378847 0 - 0.0299513 2026-02-23T15:18:59.233 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:18:59.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:18:59 trial068 ceph-mon[38805]: pgmap v4443: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:18:59.351 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:18:59.445 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:18:59.444+0000 7f05455bc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:18:59.445 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:18:59.444+0000 7f05455bc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:18:59.555 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:18:59.556 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":303,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:16:40.005952+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:15:43.826304+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:13:32.603417+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:15:09.310704+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:18:59.672 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7533 8 365281 365273 0.378797 0 - 0.0299513 2026-02-23T15:19:00.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:19:00 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/986648914' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:19:00.672 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7534 8 365281 365273 0.378747 0 - 0.0299513 2026-02-23T15:19:01.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:19:01 trial068 ceph-mon[38805]: pgmap v4444: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:19:01.672 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7535 8 365281 365273 0.378696 0 - 0.0299513 2026-02-23T15:19:02.593 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:19:02.672 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7536 8 365281 365273 0.378646 0 - 0.0299513 2026-02-23T15:19:02.713 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:19:02.812 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:19:02.810+0000 7f9bcf837640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:19:02.812 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:19:02.811+0000 7f9bcf837640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:19:02.920 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:19:02.921 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":303,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:16:40.005952+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:15:43.826304+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:13:32.603417+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:15:09.310704+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:19:03.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:19:03 trial068 ceph-mon[38805]: pgmap v4445: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:19:03.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:19:03 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3479732257' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:19:03.672 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7537 8 365281 365273 0.378596 0 - 0.0299513 2026-02-23T15:19:04.265 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:19:03 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:19:03.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:19:04.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:19:03 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:19:03] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:19:04.672 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7538 8 365281 365273 0.378546 0 - 0.0299513 2026-02-23T15:19:05.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:19:05 trial068 ceph-mon[38805]: pgmap v4446: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:19:05.672 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7539 8 365281 365273 0.378496 0 - 0.0299513 2026-02-23T15:19:05.957 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:19:06.078 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:19:06.182 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:19:06.180+0000 7f1e0bac6640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:19:06.182 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:19:06.181+0000 7f1e0bac6640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:19:06.291 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:19:06.292 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":303,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:16:40.005952+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:15:43.826304+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:13:32.603417+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:15:09.310704+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:19:06.672 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T15:19:06.672440+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T15:19:06.673 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T15:19:06.673 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7540 8 365281 365273 0.378445 0 - 0.0299513 2026-02-23T15:19:07.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:19:07 trial068 ceph-mon[38805]: pgmap v4447: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:19:07.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:19:07 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1064456335' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:19:07.672 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7541 8 365281 365273 0.378395 0 - 0.0299513 2026-02-23T15:19:08.672 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7542 8 365281 365273 0.378345 0 - 0.0299513 2026-02-23T15:19:09.329 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:19:09.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:19:09 trial068 ceph-mon[38805]: pgmap v4448: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:19:09.446 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:19:09.543 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:19:09.541+0000 7f785852b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:19:09.543 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:19:09.542+0000 7f785852b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:19:09.654 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:19:09.654 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":303,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:16:40.005952+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:15:43.826304+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:13:32.603417+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:15:09.310704+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:19:09.673 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7543 8 365281 365273 0.378295 0 - 0.0299513 2026-02-23T15:19:10.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:19:10 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/86151148' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:19:10.673 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7544 8 365281 365273 0.378245 0 - 0.0299513 2026-02-23T15:19:11.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:19:11 trial068 ceph-mon[38805]: pgmap v4449: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:19:11.673 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7545 8 365281 365273 0.378195 0 - 0.0299513 2026-02-23T15:19:12.673 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7546 8 365281 365273 0.378144 0 - 0.0299513 2026-02-23T15:19:12.693 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:19:12.812 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:19:12.912 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:19:12.910+0000 7f670f8fc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:19:12.912 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:19:12.911+0000 7f670f8fc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:19:13.024 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:19:13.024 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":303,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:16:40.005952+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:15:43.826304+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:13:32.603417+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:15:09.310704+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:19:13.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:19:13 trial068 ceph-mon[38805]: pgmap v4450: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:19:13.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:19:13 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/4291985618' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:19:13.673 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7547 8 365281 365273 0.378094 0 - 0.0299513 2026-02-23T15:19:14.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:19:13 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:19:13] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:19:14.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:19:13 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:19:13.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:19:14.673 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7548 8 365281 365273 0.378044 0 - 0.0299513 2026-02-23T15:19:15.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:19:15 trial068 ceph-mon[38805]: pgmap v4451: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:19:15.673 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7549 8 365281 365273 0.377994 0 - 0.0299513 2026-02-23T15:19:16.063 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:19:16.180 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:19:16.280 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:19:16.278+0000 7fba2d550640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:19:16.280 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:19:16.278+0000 7fba2d550640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:19:16.391 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:19:16.391 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":303,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:16:40.005952+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:15:43.826304+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:13:32.603417+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:15:09.310704+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:19:16.673 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7550 8 365281 365273 0.377944 0 - 0.0299513 2026-02-23T15:19:17.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:19:17 trial068 ceph-mon[38805]: pgmap v4452: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:19:17.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:19:17 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3722783387' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:19:17.673 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7551 8 365281 365273 0.377894 0 - 0.0299513 2026-02-23T15:19:18.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:19:18 trial068 ceph-mon[38805]: osdmap e304: 4 total, 3 up, 4 in 2026-02-23T15:19:18.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:19:18 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2026-02-23T15:19:18.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:19:18 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-23T15:19:18.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:19:18 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2026-02-23T15:19:18.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:19:18 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T15:19:18.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:19:18 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2026-02-23T15:19:18.673 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7552 8 365281 365273 0.377844 0 - 0.0299513 2026-02-23T15:19:19.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:19:19 trial068 ceph-mon[38805]: pgmap v4454: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:19:19.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:19:19 trial068 ceph-mon[38805]: pgmap v4455: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:19:19.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:19:19 trial068 ceph-mon[38805]: pgmap v4456: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:19:19.430 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:19:19.549 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:19:19.656 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:19:19.654+0000 7f5928cd1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:19:19.656 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:19:19.655+0000 7f5928cd1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:19:19.673 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7553 8 365281 365273 0.377794 0 - 0.0299513 2026-02-23T15:19:19.764 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:19:19.765 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":304,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:19:17.045991+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:15:43.826304+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:18:55.148231+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:15:09.310704+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:19:20.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:19:20 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3049313701' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:19:20.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:19:20 trial068 ceph-mon[38805]: pgmap v4457: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:19:20.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:19:20 trial068 ceph-mon[38805]: osdmap e305: 4 total, 3 up, 4 in 2026-02-23T15:19:20.673 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7554 8 365281 365273 0.377744 0 - 0.0299513 2026-02-23T15:19:21.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:19:21 trial068 ceph-mon[38805]: 5.5 scrub starts 2026-02-23T15:19:21.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:19:21 trial068 ceph-mon[38805]: 5.5 scrub ok 2026-02-23T15:19:21.673 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7555 8 365281 365273 0.377694 0 - 0.0299513 2026-02-23T15:19:22.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:19:22 trial068 ceph-mon[38805]: 5.8 scrub starts 2026-02-23T15:19:22.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:19:22 trial068 ceph-mon[38805]: 5.8 scrub ok 2026-02-23T15:19:22.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:19:22 trial068 ceph-mon[38805]: pgmap v4459: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:19:22.673 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7556 8 365281 365273 0.377644 0 - 0.0299513 2026-02-23T15:19:22.801 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:19:22.917 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:19:23.015 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:19:23.014+0000 7f7375482640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:19:23.015 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:19:23.014+0000 7f7375482640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:19:23.126 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:19:23.127 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":305,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:19:20.028204+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:15:43.826304+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:18:55.148231+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:15:09.310704+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:19:23.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:19:23 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3423105204' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:19:23.673 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7557 8 365281 365273 0.377594 0 - 0.0299513 2026-02-23T15:19:24.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:19:24 trial068 ceph-mon[38805]: pgmap v4460: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:19:24.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:19:23 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:19:23] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:19:24.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:19:23 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:19:23.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:19:24.673 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7558 8 365281 365273 0.377544 0 - 0.0299513 2026-02-23T15:19:25.674 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7559 8 365281 365273 0.377494 0 - 0.0299513 2026-02-23T15:19:26.164 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:19:26.282 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:19:26.381 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:19:26.380+0000 7fcfd2b13640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:19:26.382 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:19:26 trial068 ceph-mon[38805]: pgmap v4461: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:19:26.382 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:19:26.380+0000 7fcfd2b13640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:19:26.490 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:19:26.490 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":305,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:19:20.028204+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:15:43.826304+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:18:55.148231+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:15:09.310704+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:19:26.674 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T15:19:26.673830+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T15:19:26.674 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T15:19:26.674 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7560 8 365281 365273 0.377444 0 - 0.0299513 2026-02-23T15:19:27.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:19:27 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/278538788' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:19:27.674 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7561 8 365281 365273 0.377394 0 - 0.0299513 2026-02-23T15:19:28.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:19:28 trial068 ceph-mon[38805]: pgmap v4462: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:19:28.674 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7562 8 365281 365273 0.377344 0 - 0.0299513 2026-02-23T15:19:29.529 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:19:29.554 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:19:29 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T15:19:29.554 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:19:29 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T15:19:29.646 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:19:29.674 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7563 8 365281 365273 0.377294 0 - 0.0299513 2026-02-23T15:19:29.743 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:19:29.742+0000 7fa465e8b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:19:29.743 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:19:29.742+0000 7fa465e8b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:19:29.853 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:19:29.853 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":305,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:19:20.028204+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:15:43.826304+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:18:55.148231+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:15:09.310704+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:19:30.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:19:30 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3861733997' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:19:30.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:19:30 trial068 ceph-mon[38805]: pgmap v4463: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:19:30.674 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7564 8 365281 365273 0.377245 0 - 0.0299513 2026-02-23T15:19:31.674 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7565 8 365281 365273 0.377195 0 - 0.0299513 2026-02-23T15:19:32.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:19:32 trial068 ceph-mon[38805]: pgmap v4464: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:19:32.674 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7566 8 365281 365273 0.377145 0 - 0.0299513 2026-02-23T15:19:32.890 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:19:33.009 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:19:33.112 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:19:33.110+0000 7ff2d3b25640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:19:33.112 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:19:33.111+0000 7ff2d3b25640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:19:33.220 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:19:33.220 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":305,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:19:20.028204+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:15:43.826304+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:18:55.148231+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:15:09.310704+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:19:33.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:19:33 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3167204357' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:19:33.674 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7567 8 365281 365273 0.377095 0 - 0.0299513 2026-02-23T15:19:34.265 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:19:33 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:19:33.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:19:34.272 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:19:33 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:19:33] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:19:34.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:19:34 trial068 ceph-mon[38805]: pgmap v4465: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:19:34.674 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7568 8 365281 365273 0.377045 0 - 0.0299513 2026-02-23T15:19:35.674 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7569 8 365281 365273 0.376995 0 - 0.0299513 2026-02-23T15:19:36.259 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:19:36.376 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:19:36.398 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:19:36 trial068 ceph-mon[38805]: pgmap v4466: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:19:36.476 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:19:36.474+0000 7f03a32ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:19:36.476 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:19:36.474+0000 7f03a32ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:19:36.585 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:19:36.585 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":305,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:19:20.028204+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:15:43.826304+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:18:55.148231+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:15:09.310704+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:19:36.674 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7570 8 365281 365273 0.376946 0 - 0.0299513 2026-02-23T15:19:37.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:19:37 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3549853633' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:19:37.675 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7571 8 365281 365273 0.376896 0 - 0.0299513 2026-02-23T15:19:38.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:19:38 trial068 ceph-mon[38805]: pgmap v4467: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:19:38.675 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7572 8 365281 365273 0.376846 0 - 0.0299513 2026-02-23T15:19:39.623 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:19:39.675 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7573 8 365281 365273 0.376796 0 - 0.0299513 2026-02-23T15:19:39.740 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:19:39.839 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:19:39.837+0000 7fc5ae76b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:19:39.839 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:19:39.838+0000 7fc5ae76b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:19:39.948 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:19:39.948 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":305,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:19:20.028204+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:15:43.826304+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:18:55.148231+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:15:09.310704+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:19:40.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:19:39 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3801347406' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:19:40.675 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7574 8 365281 365273 0.376746 0 - 0.0299513 2026-02-23T15:19:41.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:19:40 trial068 ceph-mon[38805]: pgmap v4468: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:19:41.675 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7575 8 365281 365273 0.376697 0 - 0.0299513 2026-02-23T15:19:42.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:19:42 trial068 ceph-mon[38805]: pgmap v4469: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:19:42.675 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7576 8 365281 365273 0.376647 0 - 0.0299513 2026-02-23T15:19:42.988 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:19:43.105 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:19:43.208 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:19:43.207+0000 7f00f4f00640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:19:43.208 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:19:43.207+0000 7f00f4f00640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:19:43.316 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:19:43.316 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":305,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:19:20.028204+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:15:43.826304+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:18:55.148231+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:15:09.310704+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:19:43.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:19:43 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2995409483' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:19:43.675 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7577 8 365281 365273 0.376597 0 - 0.0299513 2026-02-23T15:19:44.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:19:43 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:19:43] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:19:44.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:19:43 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:19:43.974Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:19:44.675 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7578 8 365281 365273 0.376548 0 - 0.0299513 2026-02-23T15:19:44.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:19:44 trial068 ceph-mon[38805]: pgmap v4470: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:19:45.675 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7579 8 365281 365273 0.376498 0 - 0.0299513 2026-02-23T15:19:46.353 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:19:46.472 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:19:46.576 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:19:46.575+0000 7fe9a8746640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:19:46.576 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:19:46.575+0000 7fe9a8746640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:19:46.675 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T15:19:46.675268+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T15:19:46.675 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T15:19:46.675 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7580 8 365281 365273 0.376448 0 - 0.0299513 2026-02-23T15:19:46.685 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:19:46.685 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":305,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:19:20.028204+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:15:43.826304+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:18:55.148231+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:15:09.310704+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:19:47.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:19:47 trial068 ceph-mon[38805]: pgmap v4471: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:19:47.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:19:47 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/365457321' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:19:47.675 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7581 8 365281 365273 0.376399 0 - 0.0299513 2026-02-23T15:19:48.675 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7582 8 365281 365273 0.376349 0 - 0.0299513 2026-02-23T15:19:49.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:19:49 trial068 ceph-mon[38805]: pgmap v4472: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:19:49.675 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7583 8 365281 365273 0.376299 0 - 0.0299513 2026-02-23T15:19:49.723 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:19:49.840 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:19:49.942 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:19:49.940+0000 7f57904f8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:19:49.942 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:19:49.941+0000 7f57904f8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:19:50.052 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:19:50.053 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":305,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:19:20.028204+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:15:43.826304+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:18:55.148231+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:15:09.310704+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:19:50.675 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7584 8 365281 365273 0.37625 0 - 0.0299513 2026-02-23T15:19:51.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:19:51 trial068 ceph-mon[38805]: pgmap v4473: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:19:51.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:19:51 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3659913445' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:19:51.675 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7585 8 365281 365273 0.3762 0 - 0.0299513 2026-02-23T15:19:52.676 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7586 8 365281 365273 0.376151 0 - 0.0299513 2026-02-23T15:19:53.092 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:19:53.207 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:19:53.310 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:19:53.308+0000 7fd29d8a8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:19:53.310 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:19:53 trial068 ceph-mon[38805]: pgmap v4474: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:19:53.311 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:19:53.309+0000 7fd29d8a8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:19:53.419 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:19:53.419 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":305,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:19:20.028204+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:15:43.826304+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:18:55.148231+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:15:09.310704+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:19:53.676 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7587 8 365281 365273 0.376101 0 - 0.0299513 2026-02-23T15:19:54.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:19:54 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1123446983' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:19:54.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:19:53 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:19:53] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:19:54.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:19:53 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:19:53.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:19:54.676 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7588 8 365281 365273 0.376051 0 - 0.0299513 2026-02-23T15:19:55.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:19:55 trial068 ceph-mon[38805]: pgmap v4475: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:19:55.676 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7589 8 365281 365273 0.376002 0 - 0.0299513 2026-02-23T15:19:56.456 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:19:56.577 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:19:56.676 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7590 8 365281 365273 0.375952 0 - 0.0299513 2026-02-23T15:19:56.676 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:19:56.675+0000 7fd3b9f6a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:19:56.677 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:19:56.675+0000 7fd3b9f6a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:19:56.785 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:19:56.786 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":305,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:19:20.028204+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:15:43.826304+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:18:55.148231+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:15:09.310704+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:19:57.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:19:57 trial068 ceph-mon[38805]: 5.a scrub starts 2026-02-23T15:19:57.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:19:57 trial068 ceph-mon[38805]: 5.a scrub ok 2026-02-23T15:19:57.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:19:57 trial068 ceph-mon[38805]: pgmap v4476: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:19:57.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:19:57 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1832324955' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:19:57.676 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7591 8 365281 365273 0.375903 0 - 0.0299513 2026-02-23T15:19:58.676 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7592 8 365281 365273 0.375853 0 - 0.0299513 2026-02-23T15:19:59.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:19:59 trial068 ceph-mon[38805]: pgmap v4477: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:19:59.676 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7593 8 365281 365273 0.375804 0 - 0.0299513 2026-02-23T15:19:59.841 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:19:59.960 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:20:00.059 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:20:00.058+0000 7f98d6b12640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:20:00.060 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:20:00.058+0000 7f98d6b12640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:20:00.170 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:20:00.170 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":305,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:19:20.028204+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:15:43.826304+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:18:55.148231+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:15:09.310704+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:20:00.676 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7594 8 365281 365273 0.375754 0 - 0.0299513 2026-02-23T15:20:01.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:20:01 trial068 ceph-mon[38805]: pgmap v4478: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:20:01.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:20:01 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/981328419' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:20:01.676 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7595 8 365281 365273 0.375705 0 - 0.0299513 2026-02-23T15:20:02.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:20:02 trial068 ceph-mon[38805]: 5.7 scrub starts 2026-02-23T15:20:02.676 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7596 8 365281 365273 0.375655 0 - 0.0299513 2026-02-23T15:20:03.207 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:20:03.326 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:20:03.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:20:03 trial068 ceph-mon[38805]: 5.7 scrub ok 2026-02-23T15:20:03.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:20:03 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T15:20:03.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:20:03 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T15:20:03.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:20:03 trial068 ceph-mon[38805]: pgmap v4479: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:20:03.423 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:20:03.421+0000 7fe8bdcb4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:20:03.423 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:20:03.422+0000 7fe8bdcb4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:20:03.533 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:20:03.533 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":305,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:19:20.028204+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:15:43.826304+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:18:55.148231+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:15:09.310704+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:20:03.676 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7597 8 365281 365273 0.375606 0 - 0.0299513 2026-02-23T15:20:04.264 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:20:03 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:20:03.974Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:20:04.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:20:04 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1571689608' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:20:04.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:20:03 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:20:03] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:20:04.676 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7598 8 365281 365273 0.375556 0 - 0.0299513 2026-02-23T15:20:05.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:20:05 trial068 ceph-mon[38805]: pgmap v4480: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:20:05.677 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7599 8 365281 365273 0.375507 0 - 0.0299513 2026-02-23T15:20:06.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:20:06 trial068 ceph-mon[38805]: 5.e scrub starts 2026-02-23T15:20:06.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:20:06 trial068 ceph-mon[38805]: pgmap v4481: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:20:06.570 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:20:06.677 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T15:20:06.676824+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T15:20:06.677 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T15:20:06.677 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7600 8 365281 365273 0.375458 0 - 0.0299513 2026-02-23T15:20:06.688 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:20:06.790 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:20:06.788+0000 7f922d652640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:20:06.790 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:20:06.789+0000 7f922d652640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:20:06.898 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:20:06.898 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":305,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:19:20.028204+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:15:43.826304+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:18:55.148231+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:15:09.310704+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:20:07.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:20:07 trial068 ceph-mon[38805]: 5.e scrub ok 2026-02-23T15:20:07.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:20:07 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/4238960465' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:20:07.677 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7601 8 365281 365273 0.375408 0 - 0.0299513 2026-02-23T15:20:08.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:20:08 trial068 ceph-mon[38805]: pgmap v4482: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:20:08.677 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7602 8 365281 365273 0.375359 0 - 0.0299513 2026-02-23T15:20:09.677 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7603 8 365281 365273 0.375309 0 - 0.0299513 2026-02-23T15:20:09.935 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:20:10.052 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:20:10.154 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:20:10.152+0000 7fbf4899c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:20:10.154 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:20:10.152+0000 7fbf4899c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:20:10.263 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:20:10.263 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":305,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:19:20.028204+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:15:43.826304+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:18:55.148231+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:15:09.310704+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:20:10.677 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7604 8 365281 365273 0.37526 0 - 0.0299513 2026-02-23T15:20:11.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:20:11 trial068 ceph-mon[38805]: pgmap v4483: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:20:11.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:20:11 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1034378383' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:20:11.677 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7605 8 365281 365273 0.375211 0 - 0.0299513 2026-02-23T15:20:12.677 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7606 8 365281 365273 0.375161 0 - 0.0299513 2026-02-23T15:20:13.317 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:20:13.342 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:20:13 trial068 ceph-mon[38805]: pgmap v4484: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:20:13.435 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:20:13.534 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:20:13.532+0000 7fefd8cd8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:20:13.534 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:20:13.532+0000 7fefd8cd8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:20:13.645 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:20:13.645 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":305,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:19:20.028204+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:15:43.826304+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:18:55.148231+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:15:09.310704+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:20:13.677 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7607 8 365281 365273 0.375112 0 - 0.0299513 2026-02-23T15:20:14.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:20:14 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2339836932' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:20:14.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:20:13 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:20:13] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:20:14.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:20:13 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:20:13.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:20:14.677 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7608 8 365281 365273 0.375063 0 - 0.0299513 2026-02-23T15:20:15.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:20:15 trial068 ceph-mon[38805]: pgmap v4485: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:20:15.677 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7609 8 365281 365273 0.375014 0 - 0.0299513 2026-02-23T15:20:16.677 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7610 8 365281 365273 0.374964 0 - 0.0299513 2026-02-23T15:20:16.683 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:20:16.802 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:20:16.904 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:20:16.903+0000 7fe71ee3d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:20:16.904 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:20:16.903+0000 7fe71ee3d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:20:17.014 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:20:17.014 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":305,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:19:20.028204+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:15:43.826304+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:18:55.148231+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:15:09.310704+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:20:17.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:20:17 trial068 ceph-mon[38805]: pgmap v4486: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:20:17.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:20:17 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2246772361' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:20:17.677 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7611 8 365281 365273 0.374915 0 - 0.0299513 2026-02-23T15:20:18.677 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7612 8 365281 365273 0.374866 0 - 0.0299513 2026-02-23T15:20:19.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:20:19 trial068 ceph-mon[38805]: pgmap v4487: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:20:19.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:20:19 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2026-02-23T15:20:19.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:20:19 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-23T15:20:19.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:20:19 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2026-02-23T15:20:19.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:20:19 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T15:20:19.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:20:19 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2026-02-23T15:20:19.678 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7613 8 365281 365273 0.374816 0 - 0.0299513 2026-02-23T15:20:20.051 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:20:20.167 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:20:20.270 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:20:20.268+0000 7f7d60372640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:20:20.270 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:20:20.268+0000 7f7d60372640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:20:20.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:20:20 trial068 ceph-mon[38805]: pgmap v4488: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:20:20.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:20:20 trial068 ceph-mon[38805]: pgmap v4489: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:20:20.379 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:20:20.379 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":305,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:19:20.028204+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:15:43.826304+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:18:55.148231+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:15:09.310704+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:20:20.678 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7614 8 365281 365273 0.374767 0 - 0.0299513 2026-02-23T15:20:21.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:20:21 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2460486749' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:20:21.678 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7615 8 365281 365273 0.374718 0 - 0.0299513 2026-02-23T15:20:22.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:20:22 trial068 ceph-mon[38805]: pgmap v4490: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:20:22.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:20:22 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T15:20:22.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:20:22 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T15:20:22.678 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7616 8 365281 365273 0.374669 0 - 0.0299513 2026-02-23T15:20:23.416 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:20:23.536 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:20:23.637 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:20:23.635+0000 7f0d71bd4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:20:23.637 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:20:23.635+0000 7f0d71bd4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:20:23.678 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7617 8 365281 365273 0.37462 0 - 0.0299513 2026-02-23T15:20:23.746 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:20:23.746 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":305,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:19:20.028204+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:15:43.826304+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:18:55.148231+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:15:09.310704+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:20:24.051 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:20:23 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:20:23] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:20:24.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:20:24 trial068 ceph-mon[38805]: pgmap v4491: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:20:24.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:20:24 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/245388358' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:20:24.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:20:23 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:20:23.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:20:24.678 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7618 8 365281 365273 0.37457 0 - 0.0299513 2026-02-23T15:20:25.678 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7619 8 365281 365273 0.374521 0 - 0.0299513 2026-02-23T15:20:26.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:20:26 trial068 ceph-mon[38805]: pgmap v4492: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:20:26.678 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T15:20:26.678308+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T15:20:26.678 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T15:20:26.679 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7620 8 365281 365273 0.374472 0 - 0.0299513 2026-02-23T15:20:26.783 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:20:26.902 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:20:27.003 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:20:27.002+0000 7f9642d45640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:20:27.004 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:20:27.002+0000 7f9642d45640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:20:27.111 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:20:27.111 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":305,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:19:20.028204+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:15:43.826304+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:18:55.148231+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:15:09.310704+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:20:27.678 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7621 8 365281 365273 0.374423 0 - 0.0299513 2026-02-23T15:20:28.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:20:28 trial068 ceph-mon[38805]: pgmap v4493: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:20:28.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:20:28 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/14046545' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:20:28.678 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7622 8 365281 365273 0.374374 0 - 0.0299513 2026-02-23T15:20:29.678 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7623 8 365281 365273 0.374325 0 - 0.0299513 2026-02-23T15:20:30.149 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:20:30.266 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:20:30.307 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:20:30 trial068 ceph-mon[38805]: pgmap v4494: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:20:30.365 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:20:30.363+0000 7f0b4b2ff640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:20:30.365 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:20:30.363+0000 7f0b4b2ff640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:20:30.474 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:20:30.475 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":305,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:19:20.028204+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:15:43.826304+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:18:55.148231+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:15:09.310704+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:20:30.678 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7624 8 365281 365273 0.374276 0 - 0.0299513 2026-02-23T15:20:31.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:20:31 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/4192034958' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:20:31.679 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7625 8 365281 365273 0.374227 0 - 0.0299513 2026-02-23T15:20:32.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:20:32 trial068 ceph-mon[38805]: pgmap v4495: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:20:32.679 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7626 8 365281 365273 0.374178 0 - 0.0299513 2026-02-23T15:20:33.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:20:33 trial068 ceph-mon[38805]: pgmap v4496: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:20:33.514 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:20:33.632 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:20:33.679 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7627 8 365281 365273 0.374128 0 - 0.0299513 2026-02-23T15:20:33.736 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:20:33.734+0000 7fdef7bfa640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:20:33.736 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:20:33.735+0000 7fdef7bfa640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:20:33.847 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:20:33.847 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":305,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:19:20.028204+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:15:43.826304+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:18:55.148231+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:15:09.310704+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:20:34.265 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:20:33 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:20:33.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:20:34.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:20:34 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3947442484' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:20:34.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:20:33 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:20:33] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:20:34.679 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7628 8 365281 365273 0.374079 0 - 0.0299513 2026-02-23T15:20:35.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:20:35 trial068 ceph-mon[38805]: osdmap e306: 4 total, 3 up, 4 in 2026-02-23T15:20:35.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:20:35 trial068 ceph-mon[38805]: pgmap v4498: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:20:35.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:20:35 trial068 ceph-mon[38805]: osdmap e307: 4 total, 3 up, 4 in 2026-02-23T15:20:35.679 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7629 8 365281 365273 0.37403 0 - 0.0299513 2026-02-23T15:20:36.679 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7630 8 365281 365273 0.373981 0 - 0.0299513 2026-02-23T15:20:36.884 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:20:37.003 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:20:37.106 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:20:37.104+0000 7f8e40a61640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:20:37.106 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:20:37.104+0000 7f8e40a61640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:20:37.213 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:20:37.214 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":307,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:20:35.037565+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:15:43.826304+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:18:55.148231+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:20:01.787973+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:20:37.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:20:37 trial068 ceph-mon[38805]: pgmap v4500: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:20:37.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:20:37 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1588273139' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:20:37.679 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7631 8 365281 365273 0.373932 0 - 0.0299513 2026-02-23T15:20:38.679 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7632 8 365281 365273 0.373883 0 - 0.0299513 2026-02-23T15:20:39.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:20:39 trial068 ceph-mon[38805]: pgmap v4501: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:20:39.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:20:39 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T15:20:39.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:20:39 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T15:20:39.679 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7633 8 365281 365273 0.373834 0 - 0.0299513 2026-02-23T15:20:40.253 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:20:40.370 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:20:40.472 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:20:40.470+0000 7fa6c40ae640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:20:40.472 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:20:40.471+0000 7fa6c40ae640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:20:40.583 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:20:40.583 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":307,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:20:35.037565+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:15:43.826304+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:18:55.148231+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:20:01.787973+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:20:40.679 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7634 8 365281 365273 0.373785 0 - 0.0299513 2026-02-23T15:20:41.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:20:41 trial068 ceph-mon[38805]: pgmap v4502: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:20:41.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:20:41 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1091485887' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:20:41.679 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7635 8 365281 365273 0.373736 0 - 0.0299513 2026-02-23T15:20:42.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:20:42 trial068 ceph-mon[38805]: 5.8 scrub starts 2026-02-23T15:20:42.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:20:42 trial068 ceph-mon[38805]: 5.8 scrub ok 2026-02-23T15:20:42.679 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7636 8 365281 365273 0.373688 0 - 0.0299513 2026-02-23T15:20:43.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:20:43 trial068 ceph-mon[38805]: pgmap v4503: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:20:43.620 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:20:43.679 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7637 8 365281 365273 0.373639 0 - 0.0299513 2026-02-23T15:20:43.737 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:20:43.839 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:20:43.837+0000 7f6a1dabf640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:20:43.839 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:20:43.838+0000 7f6a1dabf640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:20:43.948 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:20:43.949 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":307,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:20:35.037565+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:15:43.826304+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:18:55.148231+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:20:01.787973+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:20:44.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:20:43 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:20:43] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:20:44.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:20:43 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:20:43.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:20:44.679 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7638 8 365281 365273 0.37359 0 - 0.0299513 2026-02-23T15:20:44.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:20:44 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3980337244' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:20:45.679 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7639 8 365281 365273 0.373541 0 - 0.0299513 2026-02-23T15:20:45.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:20:45 trial068 ceph-mon[38805]: pgmap v4504: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:20:46.679 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T15:20:46.679587+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T15:20:46.680 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T15:20:46.680 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7640 8 365281 365273 0.373492 0 - 0.0299513 2026-02-23T15:20:46.985 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:20:47.103 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:20:47.207 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:20:47.205+0000 7fea5d49d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:20:47.207 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:20:47.206+0000 7fea5d49d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:20:47.317 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:20:47.317 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":307,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:20:35.037565+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:15:43.826304+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:18:55.148231+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:20:01.787973+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:20:47.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:20:47 trial068 ceph-mon[38805]: pgmap v4505: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:20:47.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:20:47 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3865150513' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:20:47.679 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7641 8 365281 365273 0.373443 0 - 0.0299513 2026-02-23T15:20:48.680 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7642 8 365281 365273 0.373394 0 - 0.0299513 2026-02-23T15:20:49.680 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7643 8 365281 365273 0.373345 0 - 0.0299513 2026-02-23T15:20:49.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:20:49 trial068 ceph-mon[38805]: pgmap v4506: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:20:50.354 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:20:50.472 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:20:50.572 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:20:50.571+0000 7f8b08878640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:20:50.573 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:20:50.571+0000 7f8b08878640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:20:50.680 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7644 8 365281 365273 0.373296 0 - 0.0299513 2026-02-23T15:20:50.681 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:20:50.681 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":307,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:20:35.037565+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:15:43.826304+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:18:55.148231+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:20:01.787973+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:20:51.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:20:51 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1595054399' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:20:51.680 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7645 8 365281 365273 0.373248 0 - 0.0299513 2026-02-23T15:20:52.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:20:52 trial068 ceph-mon[38805]: pgmap v4507: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:20:52.680 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7646 8 365281 365273 0.373199 0 - 0.0299513 2026-02-23T15:20:53.680 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7647 8 365281 365273 0.37315 0 - 0.0299513 2026-02-23T15:20:53.719 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:20:53.839 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:20:53.937 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:20:53.936+0000 7f8cddf02640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:20:53.938 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:20:53.936+0000 7f8cddf02640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:20:54.045 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:20:54.045 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":307,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:20:35.037565+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:15:43.826304+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:18:55.148231+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:20:01.787973+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:20:54.045 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:20:53 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:20:53] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:20:54.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:20:54 trial068 ceph-mon[38805]: pgmap v4508: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:20:54.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:20:53 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:20:53.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:20:54.680 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7648 8 365281 365273 0.373101 0 - 0.0299513 2026-02-23T15:20:55.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:20:55 trial068 ceph-mon[38805]: 5.5 scrub starts 2026-02-23T15:20:55.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:20:55 trial068 ceph-mon[38805]: 5.5 scrub ok 2026-02-23T15:20:55.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:20:55 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/853461208' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:20:55.680 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7649 8 365281 365273 0.373052 0 - 0.0299513 2026-02-23T15:20:56.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:20:56 trial068 ceph-mon[38805]: pgmap v4509: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:20:56.680 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7650 8 365281 365273 0.373004 0 - 0.0299513 2026-02-23T15:20:57.082 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:20:57.202 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:20:57.300 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:20:57.298+0000 7fb112b4e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:20:57.300 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:20:57.299+0000 7fb112b4e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:20:57.409 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:20:57.410 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":307,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:20:35.037565+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:15:43.826304+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:18:55.148231+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:20:01.787973+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:20:57.680 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7651 8 365281 365273 0.372955 0 - 0.0299513 2026-02-23T15:20:58.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:20:58 trial068 ceph-mon[38805]: pgmap v4510: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:20:58.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:20:58 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/236654819' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:20:58.680 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7652 8 365281 365273 0.372906 0 - 0.0299513 2026-02-23T15:20:59.680 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7653 8 365281 365273 0.372857 0 - 0.0299513 2026-02-23T15:21:00.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:21:00 trial068 ceph-mon[38805]: pgmap v4511: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:21:00.462 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:21:00.581 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:21:00.681 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7654 8 365281 365273 0.372809 0 - 0.0299513 2026-02-23T15:21:00.681 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:21:00.679+0000 7f1d84497640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:21:00.681 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:21:00.679+0000 7f1d84497640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:21:00.790 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:21:00.790 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":307,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:20:35.037565+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:15:43.826304+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:18:55.148231+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:20:01.787973+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:21:01.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:21:01 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/304537376' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:21:01.680 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7655 8 365281 365273 0.37276 0 - 0.0299513 2026-02-23T15:21:02.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:21:02 trial068 ceph-mon[38805]: pgmap v4512: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:21:02.681 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7656 8 365281 365273 0.372711 0 - 0.0299513 2026-02-23T15:21:03.681 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7657 8 365281 365273 0.372663 0 - 0.0299513 2026-02-23T15:21:03.828 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:21:03.947 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:21:04.051 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:21:04.050+0000 7f4d4622c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:21:04.052 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:21:04.050+0000 7f4d4622c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:21:04.161 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:21:04.162 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":307,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:20:35.037565+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:15:43.826304+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:18:55.148231+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:20:01.787973+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:21:04.162 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:21:04 trial068 ceph-mon[38805]: pgmap v4513: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:21:04.162 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:21:03 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:21:03] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:21:04.265 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:21:03 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:21:03.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:21:04.681 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7658 8 365281 365273 0.372614 0 - 0.0299513 2026-02-23T15:21:05.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:21:05 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3536604530' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:21:05.681 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7659 8 365281 365273 0.372565 0 - 0.0299513 2026-02-23T15:21:06.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:21:06 trial068 ceph-mon[38805]: pgmap v4514: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:21:06.681 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T15:21:06.680929+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T15:21:06.681 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T15:21:06.681 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7660 8 365281 365273 0.372517 0 - 0.0299513 2026-02-23T15:21:07.200 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:21:07.319 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:21:07.340 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:21:07 trial068 ceph-mon[38805]: pgmap v4515: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:21:07.420 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:21:07.419+0000 7fb578dcf640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:21:07.420 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:21:07.419+0000 7fb578dcf640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:21:07.528 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:21:07.529 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":307,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:20:35.037565+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:15:43.826304+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:18:55.148231+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:20:01.787973+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:21:07.681 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7661 8 365281 365273 0.372468 0 - 0.0299513 2026-02-23T15:21:08.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:21:08 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1918587449' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:21:08.681 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7662 8 365281 365273 0.372419 0 - 0.0299513 2026-02-23T15:21:09.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:21:09 trial068 ceph-mon[38805]: pgmap v4516: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:21:09.681 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7663 8 365281 365273 0.372371 0 - 0.0299513 2026-02-23T15:21:10.566 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:21:10.681 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7664 8 365281 365273 0.372322 0 - 0.0299513 2026-02-23T15:21:10.684 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:21:10.787 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:21:10.785+0000 7ff11cc91640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:21:10.787 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:21:10.785+0000 7ff11cc91640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:21:10.899 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:21:10.900 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":307,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:20:35.037565+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:15:43.826304+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:18:55.148231+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:20:01.787973+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:21:11.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:21:11 trial068 ceph-mon[38805]: 5.a scrub starts 2026-02-23T15:21:11.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:21:11 trial068 ceph-mon[38805]: 5.a scrub ok 2026-02-23T15:21:11.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:21:11 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/4234947209' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:21:11.681 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7665 8 365281 365273 0.372274 0 - 0.0299513 2026-02-23T15:21:12.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:21:12 trial068 ceph-mon[38805]: pgmap v4517: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:21:12.681 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7666 8 365281 365273 0.372225 0 - 0.0299513 2026-02-23T15:21:13.681 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7667 8 365281 365273 0.372177 0 - 0.0299513 2026-02-23T15:21:13.937 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:21:14.057 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:21:14.160 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:21:14.158+0000 7f201c6b0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:21:14.160 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:21:14.159+0000 7f201c6b0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:21:14.271 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:21:14.271 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":307,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:20:35.037565+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:15:43.826304+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:18:55.148231+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:20:01.787973+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:21:14.271 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:21:14 trial068 ceph-mon[38805]: pgmap v4518: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:21:14.272 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:21:13 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:21:13] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:21:14.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:21:13 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:21:13.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:21:14.681 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7668 8 365281 365273 0.372128 0 - 0.0299513 2026-02-23T15:21:15.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:21:15 trial068 ceph-mon[38805]: 5.e scrub starts 2026-02-23T15:21:15.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:21:15 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/4230890835' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:21:15.681 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7669 8 365281 365273 0.37208 0 - 0.0299513 2026-02-23T15:21:16.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:21:16 trial068 ceph-mon[38805]: pgmap v4519: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:21:16.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:21:16 trial068 ceph-mon[38805]: 5.e scrub ok 2026-02-23T15:21:16.681 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7670 8 365281 365273 0.372031 0 - 0.0299513 2026-02-23T15:21:17.325 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:21:17.444 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:21:17.545 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:21:17.543+0000 7f8e00b23640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:21:17.545 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:21:17.543+0000 7f8e00b23640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:21:17.655 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:21:17.655 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":307,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:20:35.037565+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:15:43.826304+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:18:55.148231+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:20:01.787973+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:21:17.682 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7671 8 365281 365273 0.371983 0 - 0.0299513 2026-02-23T15:21:18.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:21:18 trial068 ceph-mon[38805]: pgmap v4520: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:21:18.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:21:18 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2679054579' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:21:18.682 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7672 8 365281 365273 0.371934 0 - 0.0299513 2026-02-23T15:21:19.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:21:19 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2026-02-23T15:21:19.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:21:19 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-23T15:21:19.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:21:19 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2026-02-23T15:21:19.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:21:19 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T15:21:19.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:21:19 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2026-02-23T15:21:19.682 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7673 8 365281 365273 0.371886 0 - 0.0299513 2026-02-23T15:21:20.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:21:20 trial068 ceph-mon[38805]: pgmap v4521: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:21:20.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:21:20 trial068 ceph-mon[38805]: pgmap v4522: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:21:20.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:21:20 trial068 ceph-mon[38805]: pgmap v4523: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:21:20.682 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7674 8 365281 365273 0.371837 0 - 0.0299513 2026-02-23T15:21:20.691 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:21:20.808 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:21:20.905 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:21:20.903+0000 7f4b37118640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:21:20.905 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:21:20.903+0000 7f4b37118640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:21:21.013 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:21:21.013 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":307,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:20:35.037565+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:15:43.826304+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:18:55.148231+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:20:01.787973+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:21:21.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:21:21 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1446613761' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:21:21.682 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7675 8 365281 365273 0.371789 0 - 0.0299513 2026-02-23T15:21:22.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:21:22 trial068 ceph-mon[38805]: pgmap v4524: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:21:22.682 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7676 8 365281 365273 0.37174 0 - 0.0299513 2026-02-23T15:21:23.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:21:23 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T15:21:23.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:21:23 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T15:21:23.682 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7677 8 365281 365273 0.371692 0 - 0.0299513 2026-02-23T15:21:24.050 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:21:24.171 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:21:24.224 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:21:24 trial068 ceph-mon[38805]: pgmap v4525: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:21:24.225 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:21:23 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:21:23] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:21:24.273 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:21:24.271+0000 7fcdaa2c4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:21:24.273 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:21:24.271+0000 7fcdaa2c4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:21:24.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:21:23 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:21:23.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:21:24.381 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:21:24.382 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":307,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:20:35.037565+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:15:43.826304+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:18:55.148231+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:20:01.787973+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:21:24.682 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7678 8 365281 365273 0.371643 0 - 0.0299513 2026-02-23T15:21:25.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:21:25 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T15:21:25.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:21:25 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T15:21:25.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:21:25 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/324301361' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:21:25.682 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7679 8 365281 365273 0.371595 0 - 0.0299513 2026-02-23T15:21:26.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:21:26 trial068 ceph-mon[38805]: pgmap v4526: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:21:26.682 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T15:21:26.682361+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T15:21:26.682 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T15:21:26.683 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7680 8 365281 365273 0.371547 0 - 0.0299513 2026-02-23T15:21:27.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:21:27 trial068 ceph-mon[38805]: pgmap v4527: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:21:27.419 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:21:27.536 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:21:27.638 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:21:27.636+0000 7fa0e2525640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:21:27.638 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:21:27.637+0000 7fa0e2525640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:21:27.682 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7681 8 365281 365273 0.371498 0 - 0.0299513 2026-02-23T15:21:27.749 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:21:27.749 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":307,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:20:35.037565+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:15:43.826304+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:18:55.148231+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:20:01.787973+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:21:28.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:21:28 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2882835575' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:21:28.682 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7682 8 365281 365273 0.37145 0 - 0.0299513 2026-02-23T15:21:29.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:21:29 trial068 ceph-mon[38805]: pgmap v4528: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:21:29.682 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7683 8 365281 365273 0.371402 0 - 0.0299513 2026-02-23T15:21:30.683 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7684 8 365281 365273 0.371353 0 - 0.0299513 2026-02-23T15:21:30.786 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:21:30.902 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:21:31.003 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:21:31.001+0000 7f4b30616640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:21:31.003 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:21:31.002+0000 7f4b30616640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:21:31.112 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:21:31.112 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":307,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:20:35.037565+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:15:43.826304+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:18:55.148231+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:20:01.787973+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:21:31.683 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7685 8 365281 365273 0.371305 0 - 0.0299513 2026-02-23T15:21:32.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:21:32 trial068 ceph-mon[38805]: pgmap v4529: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:21:32.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:21:32 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/753909209' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:21:32.683 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7686 8 365281 365273 0.371257 0 - 0.0299513 2026-02-23T15:21:33.683 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7687 8 365281 365273 0.371208 0 - 0.0299513 2026-02-23T15:21:34.163 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:21:34.265 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:21:33 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:21:33.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:21:34.280 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:21:34.302 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:21:34 trial068 ceph-mon[38805]: pgmap v4530: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:21:34.302 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:21:33 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:21:33] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:21:34.380 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:21:34.378+0000 7ff9edcc3640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:21:34.380 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:21:34.378+0000 7ff9edcc3640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:21:34.488 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:21:34.489 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":307,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:20:35.037565+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:15:43.826304+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:18:55.148231+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:20:01.787973+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:21:34.683 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7688 8 365281 365273 0.37116 0 - 0.0299513 2026-02-23T15:21:35.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:21:35 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2699617221' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:21:35.683 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7689 8 365281 365273 0.371112 0 - 0.0299513 2026-02-23T15:21:36.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:21:36 trial068 ceph-mon[38805]: pgmap v4531: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:21:36.683 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7690 8 365281 365273 0.371063 0 - 0.0299513 2026-02-23T15:21:37.526 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:21:37.643 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:21:37.683 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7691 8 365281 365273 0.371015 0 - 0.0299513 2026-02-23T15:21:37.741 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:21:37.739+0000 7fabb85e5640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:21:37.741 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:21:37.740+0000 7fabb85e5640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:21:37.851 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:21:37.851 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":307,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:20:35.037565+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:15:43.826304+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:18:55.148231+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:20:01.787973+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:21:38.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:21:38 trial068 ceph-mon[38805]: pgmap v4532: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:21:38.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:21:38 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2058658361' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:21:38.683 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7692 8 365281 365273 0.370967 0 - 0.0299513 2026-02-23T15:21:39.683 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7693 8 365281 365273 0.370919 0 - 0.0299513 2026-02-23T15:21:40.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:21:40 trial068 ceph-mon[38805]: pgmap v4533: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:21:40.683 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7694 8 365281 365273 0.370871 0 - 0.0299513 2026-02-23T15:21:40.889 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:21:41.007 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:21:41.106 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:21:41.104+0000 7f017a255640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:21:41.106 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:21:41.104+0000 7f017a255640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:21:41.217 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:21:41.217 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":308,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:21:40.053381+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:20:38.684224+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:18:55.148231+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:20:01.787973+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:21:41.217 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:21:41 trial068 ceph-mon[38805]: osdmap e308: 4 total, 3 up, 4 in 2026-02-23T15:21:41.683 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7695 8 365281 365273 0.370822 0 - 0.0299513 2026-02-23T15:21:42.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:21:42 trial068 ceph-mon[38805]: pgmap v4535: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:21:42.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:21:42 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2885742235' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:21:42.683 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7696 8 365281 365273 0.370774 0 - 0.0299513 2026-02-23T15:21:43.683 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7697 8 365281 365273 0.370726 0 - 0.0299513 2026-02-23T15:21:44.255 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:21:44.279 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:21:44 trial068 ceph-mon[38805]: pgmap v4536: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:21:44.280 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:21:43 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:21:43] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:21:44.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:21:43 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:21:43.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:21:44.371 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:21:44.473 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:21:44.471+0000 7fabc97fc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:21:44.473 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:21:44.471+0000 7fabc97fc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:21:44.581 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:21:44.581 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":308,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:21:40.053381+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:20:38.684224+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:18:55.148231+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:20:01.787973+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:21:44.683 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7698 8 365281 365273 0.370678 0 - 0.0299513 2026-02-23T15:21:45.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:21:45 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1639242261' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:21:45.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:21:45 trial068 ceph-mon[38805]: osdmap e309: 4 total, 3 up, 4 in 2026-02-23T15:21:45.684 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7699 8 365281 365273 0.37063 0 - 0.0299513 2026-02-23T15:21:46.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:21:46 trial068 ceph-mon[38805]: pgmap v4537: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:21:46.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:21:46 trial068 ceph-mon[38805]: 5.7 scrub starts 2026-02-23T15:21:46.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:21:46 trial068 ceph-mon[38805]: 5.7 scrub ok 2026-02-23T15:21:46.684 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T15:21:46.683766+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T15:21:46.684 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T15:21:46.684 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7700 8 365281 365273 0.370582 0 - 0.0299513 2026-02-23T15:21:47.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:21:47 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T15:21:47.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:21:47 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T15:21:47.620 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:21:47.684 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7701 8 365281 365273 0.370533 0 - 0.0299513 2026-02-23T15:21:47.739 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:21:47.837 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:21:47.836+0000 7ff714afe640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:21:47.838 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:21:47.836+0000 7ff714afe640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:21:47.947 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:21:47.948 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":309,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:21:45.047022+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:20:38.684224+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:18:55.148231+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:20:01.787973+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:21:48.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:21:48 trial068 ceph-mon[38805]: pgmap v4539: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:21:48.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:21:48 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/4102846261' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:21:48.684 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7702 8 365281 365273 0.370485 0 - 0.0299513 2026-02-23T15:21:49.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:21:49 trial068 ceph-mon[38805]: pgmap v4540: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:21:49.684 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7703 8 365281 365273 0.370437 0 - 0.0299513 2026-02-23T15:21:50.684 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7704 8 365281 365273 0.370389 0 - 0.0299513 2026-02-23T15:21:50.984 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:21:51.105 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:21:51.206 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:21:51.204+0000 7fb5be26c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:21:51.206 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:21:51.204+0000 7fb5be26c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:21:51.316 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:21:51.316 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":309,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:21:45.047022+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:20:38.684224+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:18:55.148231+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:20:01.787973+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:21:51.684 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7705 8 365281 365273 0.370341 0 - 0.0299513 2026-02-23T15:21:52.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:21:52 trial068 ceph-mon[38805]: pgmap v4541: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:21:52.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:21:52 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/817865273' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:21:52.684 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7706 8 365281 365273 0.370293 0 - 0.0299513 2026-02-23T15:21:53.684 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7707 8 365281 365273 0.370245 0 - 0.0299513 2026-02-23T15:21:54.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:21:54 trial068 ceph-mon[38805]: pgmap v4542: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:21:54.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:21:53 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:21:53] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:21:54.354 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:21:54.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:21:53 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:21:53.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:21:54.471 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:21:54.570 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:21:54.569+0000 7fb158aec640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:21:54.570 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:21:54.569+0000 7fb158aec640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:21:54.679 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:21:54.679 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":309,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:21:45.047022+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:20:38.684224+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:18:55.148231+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:20:01.787973+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:21:54.684 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7708 8 365281 365273 0.370197 0 - 0.0299513 2026-02-23T15:21:55.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:21:55 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2546971379' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:21:55.684 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7709 8 365281 365273 0.370149 0 - 0.0299513 2026-02-23T15:21:56.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:21:56 trial068 ceph-mon[38805]: pgmap v4543: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:21:56.684 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7710 8 365281 365273 0.370101 0 - 0.0299513 2026-02-23T15:21:57.684 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7711 8 365281 365273 0.370053 0 - 0.0299513 2026-02-23T15:21:57.716 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:21:57.833 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:21:57.935 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:21:57.934+0000 7f060719f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:21:57.935 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:21:57.934+0000 7f060719f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:21:58.044 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:21:58.044 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":309,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:21:45.047022+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:20:38.684224+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:18:55.148231+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:20:01.787973+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:21:58.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:21:58 trial068 ceph-mon[38805]: pgmap v4544: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:21:58.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:21:58 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/195008340' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:21:58.684 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7712 8 365281 365273 0.370005 0 - 0.0299513 2026-02-23T15:21:59.684 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7713 8 365281 365273 0.369957 0 - 0.0299513 2026-02-23T15:22:00.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:22:00 trial068 ceph-mon[38805]: pgmap v4545: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:22:00.684 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7714 8 365281 365273 0.369909 0 - 0.0299513 2026-02-23T15:22:01.081 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:22:01.199 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:22:01.306 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:22:01.304+0000 7ff1b8d1c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:22:01.306 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:22:01.305+0000 7ff1b8d1c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:22:01.417 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:22:01.417 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":309,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:21:45.047022+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:20:38.684224+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:18:55.148231+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:20:01.787973+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:22:01.685 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7715 8 365281 365273 0.369861 0 - 0.0299513 2026-02-23T15:22:02.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:22:02 trial068 ceph-mon[38805]: pgmap v4546: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:22:02.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:22:02 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3406467495' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:22:02.685 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7716 8 365281 365273 0.369813 0 - 0.0299513 2026-02-23T15:22:03.685 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7717 8 365281 365273 0.369765 0 - 0.0299513 2026-02-23T15:22:04.265 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:22:03 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:22:03.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:22:04.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:22:04 trial068 ceph-mon[38805]: pgmap v4547: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:22:04.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:22:03 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:22:03] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:22:04.453 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:22:04.571 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:22:04.668 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:22:04.666+0000 7fc8352c0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:22:04.668 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:22:04.667+0000 7fc8352c0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:22:04.685 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7718 8 365281 365273 0.369717 0 - 0.0299513 2026-02-23T15:22:04.776 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:22:04.777 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":309,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:21:45.047022+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:20:38.684224+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:18:55.148231+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:20:01.787973+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:22:05.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:22:05 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1348376997' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:22:05.685 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7719 8 365281 365273 0.369669 0 - 0.0299513 2026-02-23T15:22:06.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:22:06 trial068 ceph-mon[38805]: pgmap v4548: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:22:06.685 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T15:22:06.685561+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T15:22:06.686 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T15:22:06.686 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7720 8 365281 365273 0.369622 0 - 0.0299513 2026-02-23T15:22:07.686 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7721 8 365281 365273 0.369574 0 - 0.0299513 2026-02-23T15:22:07.814 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:22:07.930 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:22:08.029 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:22:08.027+0000 7fa1cea0e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:22:08.029 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:22:08.027+0000 7fa1cea0e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:22:08.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:22:08 trial068 ceph-mon[38805]: pgmap v4549: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:22:08.140 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:22:08.141 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":309,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:21:45.047022+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:20:38.684224+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:18:55.148231+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:20:01.787973+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:22:08.686 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7722 8 365281 365273 0.369526 0 - 0.0299513 2026-02-23T15:22:09.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:22:09 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3133623468' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:22:09.686 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7723 8 365281 365273 0.369478 0 - 0.0299513 2026-02-23T15:22:10.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:22:10 trial068 ceph-mon[38805]: pgmap v4550: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:22:10.686 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7724 8 365281 365273 0.36943 0 - 0.0299513 2026-02-23T15:22:11.177 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:22:11.292 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:22:11.331 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:22:11 trial068 ceph-mon[38805]: 5.8 scrub starts 2026-02-23T15:22:11.331 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:22:11 trial068 ceph-mon[38805]: 5.8 scrub ok 2026-02-23T15:22:11.332 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:22:11 trial068 ceph-mon[38805]: pgmap v4551: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:22:11.389 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:22:11.387+0000 7f798f639640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:22:11.389 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:22:11.388+0000 7f798f639640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:22:11.499 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:22:11.499 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":309,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:21:45.047022+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:20:38.684224+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:18:55.148231+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:20:01.787973+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:22:11.686 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7725 8 365281 365273 0.369382 0 - 0.0299513 2026-02-23T15:22:12.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:22:12 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3749799990' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:22:12.686 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7726 8 365281 365273 0.369334 0 - 0.0299513 2026-02-23T15:22:13.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:22:13 trial068 ceph-mon[38805]: pgmap v4552: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:22:13.686 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7727 8 365281 365273 0.369287 0 - 0.0299513 2026-02-23T15:22:14.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:22:13 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:22:13] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:22:14.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:22:13 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:22:13.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:22:14.535 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:22:14.654 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:22:14.686 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7728 8 365281 365273 0.369239 0 - 0.0299513 2026-02-23T15:22:14.753 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:22:14.751+0000 7ff7aa576640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:22:14.753 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:22:14.751+0000 7ff7aa576640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:22:14.861 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:22:14.861 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":309,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:21:45.047022+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:20:38.684224+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:18:55.148231+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:20:01.787973+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:22:15.686 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7729 8 365281 365273 0.369191 0 - 0.0299513 2026-02-23T15:22:16.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:22:15 trial068 ceph-mon[38805]: pgmap v4553: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:22:16.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:22:15 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3443268762' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:22:16.686 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7730 8 365281 365273 0.369143 0 - 0.0299513 2026-02-23T15:22:17.686 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7731 8 365281 365273 0.369096 0 - 0.0299513 2026-02-23T15:22:17.897 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:22:17.921 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:22:17 trial068 ceph-mon[38805]: pgmap v4554: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:22:18.014 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:22:18.113 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:22:18.112+0000 7fad2f75f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:22:18.113 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:22:18.112+0000 7fad2f75f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:22:18.226 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:22:18.226 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":309,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:21:45.047022+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:20:38.684224+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:18:55.148231+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:20:01.787973+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:22:18.686 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7732 8 365281 365273 0.369048 0 - 0.0299513 2026-02-23T15:22:19.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:22:18 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1754519945' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:22:19.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:22:18 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2026-02-23T15:22:19.687 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7733 8 365281 365273 0.369 0 - 0.0299513 2026-02-23T15:22:20.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:22:19 trial068 ceph-mon[38805]: pgmap v4555: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:22:20.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:22:19 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-23T15:22:20.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:22:19 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2026-02-23T15:22:20.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:22:19 trial068 ceph-mon[38805]: pgmap v4556: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:22:20.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:22:19 trial068 ceph-mon[38805]: pgmap v4557: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:22:20.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:22:19 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T15:22:20.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:22:19 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2026-02-23T15:22:20.686 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7734 8 365281 365273 0.368952 0 - 0.0299513 2026-02-23T15:22:21.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:22:20 trial068 ceph-mon[38805]: 5.5 scrub starts 2026-02-23T15:22:21.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:22:20 trial068 ceph-mon[38805]: 5.5 scrub ok 2026-02-23T15:22:21.265 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:22:21.382 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:22:21.488 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:22:21.487+0000 7f1fed71b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:22:21.489 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:22:21.487+0000 7f1fed71b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:22:21.600 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:22:21.601 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":309,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:21:45.047022+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:20:38.684224+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:18:55.148231+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:20:01.787973+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:22:21.686 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7735 8 365281 365273 0.368905 0 - 0.0299513 2026-02-23T15:22:22.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:22:21 trial068 ceph-mon[38805]: pgmap v4558: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:22:22.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:22:21 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2011972316' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:22:22.687 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7736 8 365281 365273 0.368857 0 - 0.0299513 2026-02-23T15:22:23.687 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7737 8 365281 365273 0.368809 0 - 0.0299513 2026-02-23T15:22:24.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:22:23 trial068 ceph-mon[38805]: pgmap v4559: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:22:24.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:22:23 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:22:23] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:22:24.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:22:23 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:22:23.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:22:24.639 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:22:24.687 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7738 8 365281 365273 0.368762 0 - 0.0299513 2026-02-23T15:22:24.759 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:22:24.855 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:22:24.854+0000 7f016a43b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:22:24.856 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:22:24.854+0000 7f016a43b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:22:24.964 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:22:24.964 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":309,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:21:45.047022+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:20:38.684224+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:18:55.148231+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:20:01.787973+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:22:24.964 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:22:24 trial068 ceph-mon[38805]: 5.e scrub starts 2026-02-23T15:22:24.965 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:22:24 trial068 ceph-mon[38805]: 5.e scrub ok 2026-02-23T15:22:25.687 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7739 8 365281 365273 0.368714 0 - 0.0299513 2026-02-23T15:22:26.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:22:25 trial068 ceph-mon[38805]: pgmap v4560: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:22:26.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:22:25 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3736526351' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:22:26.687 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T15:22:26.686974+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T15:22:26.687 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T15:22:26.687 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7740 8 365281 365273 0.368666 0 - 0.0299513 2026-02-23T15:22:27.687 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7741 8 365281 365273 0.368619 0 - 0.0299513 2026-02-23T15:22:28.016 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:22:28.133 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:22:28.155 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:22:27 trial068 ceph-mon[38805]: pgmap v4561: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:22:28.236 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:22:28.234+0000 7ff096a88640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:22:28.236 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:22:28.234+0000 7ff096a88640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:22:28.344 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:22:28.344 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":309,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:21:45.047022+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:20:38.684224+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:18:55.148231+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:20:01.787973+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:22:28.687 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7742 8 365281 365273 0.368571 0 - 0.0299513 2026-02-23T15:22:29.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:22:28 trial068 ceph-mon[38805]: 5.a scrub starts 2026-02-23T15:22:29.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:22:28 trial068 ceph-mon[38805]: 5.a scrub ok 2026-02-23T15:22:29.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:22:28 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/479532488' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:22:29.687 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7743 8 365281 365273 0.368524 0 - 0.0299513 2026-02-23T15:22:30.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:22:29 trial068 ceph-mon[38805]: pgmap v4562: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:22:30.687 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7744 8 365281 365273 0.368476 0 - 0.0299513 2026-02-23T15:22:31.382 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:22:31.500 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:22:31.594 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:22:31.593+0000 7f3b9745a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:22:31.594 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:22:31.593+0000 7f3b9745a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:22:31.687 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7745 8 365281 365273 0.368428 0 - 0.0299513 2026-02-23T15:22:31.705 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:22:31.706 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":309,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:21:45.047022+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:20:38.684224+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:18:55.148231+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:20:01.787973+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:22:32.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:22:32 trial068 ceph-mon[38805]: pgmap v4563: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:22:32.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:22:32 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3457183958' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:22:32.687 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7746 8 365281 365273 0.368381 0 - 0.0299513 2026-02-23T15:22:33.687 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7747 8 365281 365273 0.368333 0 - 0.0299513 2026-02-23T15:22:34.265 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:22:33 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:22:33.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:22:34.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:22:34 trial068 ceph-mon[38805]: pgmap v4564: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:22:34.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:22:33 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:22:33] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:22:34.687 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7748 8 365281 365273 0.368286 0 - 0.0299513 2026-02-23T15:22:34.744 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:22:34.864 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:22:34.960 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:22:34.958+0000 7fb73d67e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:22:34.960 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:22:34.958+0000 7fb73d67e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:22:35.069 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:22:35.070 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":309,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:21:45.047022+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:20:38.684224+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:18:55.148231+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:20:01.787973+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:22:35.688 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7749 8 365281 365273 0.368238 0 - 0.0299513 2026-02-23T15:22:36.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:22:36 trial068 ceph-mon[38805]: pgmap v4565: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:22:36.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:22:36 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/607098289' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:22:36.688 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7750 8 365281 365273 0.368191 0 - 0.0299513 2026-02-23T15:22:37.688 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7751 8 365281 365273 0.368143 0 - 0.0299513 2026-02-23T15:22:38.109 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:22:38.224 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:22:38.320 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:22:38.319+0000 7f1f48d25640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:22:38.321 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:22:38 trial068 ceph-mon[38805]: pgmap v4566: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:22:38.321 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:22:38.319+0000 7f1f48d25640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:22:38.429 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:22:38.430 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":309,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:21:45.047022+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:20:38.684224+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:18:55.148231+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:20:01.787973+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:22:38.688 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7752 8 365281 365273 0.368096 0 - 0.0299513 2026-02-23T15:22:39.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:22:39 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3765435846' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:22:39.688 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7753 8 365281 365273 0.368048 0 - 0.0299513 2026-02-23T15:22:40.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:22:40 trial068 ceph-mon[38805]: pgmap v4567: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:22:40.688 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7754 8 365281 365273 0.368001 0 - 0.0299513 2026-02-23T15:22:41.467 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:22:41.584 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:22:41.688 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7755 8 365281 365273 0.367953 0 - 0.0299513 2026-02-23T15:22:41.693 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:22:41.692+0000 7fdbd2d43640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:22:41.693 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:22:41.692+0000 7fdbd2d43640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:22:41.804 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:22:41.804 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":309,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:21:45.047022+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:20:38.684224+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:18:55.148231+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:20:01.787973+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:22:42.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:22:42 trial068 ceph-mon[38805]: pgmap v4568: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:22:42.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:22:42 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2489463207' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:22:42.688 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7756 8 365281 365273 0.367906 0 - 0.0299513 2026-02-23T15:22:43.688 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7757 8 365281 365273 0.367858 0 - 0.0299513 2026-02-23T15:22:44.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:22:44 trial068 ceph-mon[38805]: pgmap v4569: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:22:44.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:22:43 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:22:43] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:22:44.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:22:43 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:22:43.974Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:22:44.688 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7758 8 365281 365273 0.367811 0 - 0.0299513 2026-02-23T15:22:44.841 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:22:44.958 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:22:45.064 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:22:45.062+0000 7f88fec18640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:22:45.064 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:22:45.063+0000 7f88fec18640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:22:45.173 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:22:45.173 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":309,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:21:45.047022+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:20:38.684224+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:18:55.148231+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:20:01.787973+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:22:45.688 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7759 8 365281 365273 0.367764 0 - 0.0299513 2026-02-23T15:22:46.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:22:46 trial068 ceph-mon[38805]: pgmap v4570: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:22:46.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:22:46 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/4036792219' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:22:46.688 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T15:22:46.688319+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T15:22:46.688 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T15:22:46.688 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7760 8 365281 365273 0.367716 0 - 0.0299513 2026-02-23T15:22:47.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:22:47 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T15:22:47.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:22:47 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T15:22:47.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:22:47 trial068 ceph-mon[38805]: pgmap v4571: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:22:47.688 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7761 8 365281 365273 0.367669 0 - 0.0299513 2026-02-23T15:22:48.213 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:22:48.329 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:22:48.428 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:22:48.427+0000 7f0faf68f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:22:48.428 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:22:48.427+0000 7f0faf68f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:22:48.538 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:22:48.539 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":309,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:21:45.047022+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:20:38.684224+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:18:55.148231+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:20:01.787973+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:22:48.688 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7762 8 365281 365273 0.367622 0 - 0.0299513 2026-02-23T15:22:48.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:22:48 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3862208544' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:22:49.688 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7763 8 365281 365273 0.367574 0 - 0.0299513 2026-02-23T15:22:49.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:22:49 trial068 ceph-mon[38805]: pgmap v4572: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:22:49.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:22:49 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T15:22:49.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:22:49 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T15:22:50.688 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7764 8 365281 365273 0.367527 0 - 0.0299513 2026-02-23T15:22:51.577 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:22:51.689 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7765 8 365281 365273 0.367479 0 - 0.0299513 2026-02-23T15:22:51.694 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:22:51.791 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:22:51.790+0000 7f7abba12640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:22:51.791 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:22:51.790+0000 7f7abba12640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:22:51.901 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:22:51.901 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":309,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:21:45.047022+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:20:38.684224+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:18:55.148231+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:20:01.787973+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:22:52.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:22:52 trial068 ceph-mon[38805]: pgmap v4573: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:22:52.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:22:52 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3908555269' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:22:52.689 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7766 8 365281 365273 0.367432 0 - 0.0299513 2026-02-23T15:22:53.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:22:53 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T15:22:53.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:22:53 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T15:22:53.689 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7767 8 365281 365273 0.367385 0 - 0.0299513 2026-02-23T15:22:54.244 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:22:54 trial068 ceph-mon[38805]: pgmap v4574: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:22:54.245 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:22:53 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:22:53] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:22:54.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:22:53 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:22:53.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:22:54.689 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7768 8 365281 365273 0.367338 0 - 0.0299513 2026-02-23T15:22:54.938 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:22:55.058 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:22:55.153 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:22:55.151+0000 7f67252ca640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:22:55.153 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:22:55.151+0000 7f67252ca640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:22:55.261 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:22:55.262 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":309,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:21:45.047022+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:20:38.684224+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:18:55.148231+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:20:01.787973+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:22:55.689 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7769 8 365281 365273 0.36729 0 - 0.0299513 2026-02-23T15:22:56.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:22:56 trial068 ceph-mon[38805]: pgmap v4575: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:22:56.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:22:56 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2858978150' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:22:56.689 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7770 8 365281 365273 0.367243 0 - 0.0299513 2026-02-23T15:22:57.689 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7771 8 365281 365273 0.367196 0 - 0.0299513 2026-02-23T15:22:58.297 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:22:58.322 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:22:58 trial068 ceph-mon[38805]: pgmap v4576: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:22:58.415 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:22:58.512 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:22:58.510+0000 7f6203725640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:22:58.512 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:22:58.511+0000 7f6203725640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:22:58.621 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:22:58.621 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":309,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:21:45.047022+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:20:38.684224+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:18:55.148231+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:20:01.787973+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:22:58.689 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7772 8 365281 365273 0.367149 0 - 0.0299513 2026-02-23T15:22:59.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:22:59 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/853136488' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:22:59.689 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7773 8 365281 365273 0.367101 0 - 0.0299513 2026-02-23T15:23:00.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:23:00 trial068 ceph-mon[38805]: pgmap v4577: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:23:00.689 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7774 8 365281 365273 0.367054 0 - 0.0299513 2026-02-23T15:23:01.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:23:01 trial068 ceph-mon[38805]: 5.7 scrub starts 2026-02-23T15:23:01.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:23:01 trial068 ceph-mon[38805]: 5.7 scrub ok 2026-02-23T15:23:01.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:23:01 trial068 ceph-mon[38805]: pgmap v4578: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:23:01.658 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:23:01.689 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7775 8 365281 365273 0.367007 0 - 0.0299513 2026-02-23T15:23:01.778 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:23:01.877 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:23:01.875+0000 7f3de3a1d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:23:01.877 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:23:01.875+0000 7f3de3a1d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:23:01.986 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:23:01.986 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":309,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:21:45.047022+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:20:38.684224+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:18:55.148231+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:20:01.787973+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:23:02.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:23:02 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3607267153' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:23:02.689 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7776 8 365281 365273 0.36696 0 - 0.0299513 2026-02-23T15:23:03.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:23:03 trial068 ceph-mon[38805]: pgmap v4579: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:23:03.689 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7777 8 365281 365273 0.366912 0 - 0.0299513 2026-02-23T15:23:04.265 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:23:03 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:23:03.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:23:04.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:23:03 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:23:03] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:23:04.689 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7778 8 365281 365273 0.366865 0 - 0.0299513 2026-02-23T15:23:05.024 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:23:05.142 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:23:05.248 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:23:05.246+0000 7fc861707640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:23:05.248 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:23:05.246+0000 7fc861707640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:23:05.357 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:23:05.357 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":309,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:21:45.047022+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:20:38.684224+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:18:55.148231+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:20:01.787973+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:23:05.689 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7779 8 365281 365273 0.366818 0 - 0.0299513 2026-02-23T15:23:06.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:23:05 trial068 ceph-mon[38805]: pgmap v4580: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:23:06.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:23:05 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/225269058' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:23:06.689 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T15:23:06.689609+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T15:23:06.690 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T15:23:06.690 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7780 8 365281 365273 0.366771 0 - 0.0299513 2026-02-23T15:23:07.689 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7781 8 365281 365273 0.366724 0 - 0.0299513 2026-02-23T15:23:08.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:23:07 trial068 ceph-mon[38805]: pgmap v4581: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:23:08.396 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:23:08.512 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:23:08.609 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:23:08.608+0000 7fc026437640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:23:08.610 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:23:08.608+0000 7fc026437640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:23:08.689 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7782 8 365281 365273 0.366677 0 - 0.0299513 2026-02-23T15:23:08.719 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:23:08.719 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":309,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:21:45.047022+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:20:38.684224+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:18:55.148231+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:20:01.787973+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:23:09.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:23:08 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/644298681' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:23:09.690 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7783 8 365281 365273 0.36663 0 - 0.0299513 2026-02-23T15:23:10.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:23:09 trial068 ceph-mon[38805]: pgmap v4582: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:23:10.690 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7784 8 365281 365273 0.366583 0 - 0.0299513 2026-02-23T15:23:11.690 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7785 8 365281 365273 0.366535 0 - 0.0299513 2026-02-23T15:23:11.771 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:23:11.891 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:23:11.993 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:23:11.991+0000 7efd2e054640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:23:11.993 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:23:11.992+0000 7efd2e054640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:23:12.102 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:23:12.102 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":309,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:21:45.047022+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:20:38.684224+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:18:55.148231+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:20:01.787973+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:23:12.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:23:12 trial068 ceph-mon[38805]: pgmap v4583: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:23:12.690 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7786 8 365281 365273 0.366488 0 - 0.0299513 2026-02-23T15:23:13.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:23:13 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3010909815' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:23:13.690 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7787 8 365281 365273 0.366441 0 - 0.0299513 2026-02-23T15:23:14.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:23:14 trial068 ceph-mon[38805]: pgmap v4584: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:23:14.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:23:13 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:23:13] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:23:14.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:23:13 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:23:13.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:23:14.690 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7788 8 365281 365273 0.366394 0 - 0.0299513 2026-02-23T15:23:15.140 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:23:15.259 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:23:15.360 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:23:15.359+0000 7fdf1b76a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:23:15.360 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:23:15.359+0000 7fdf1b76a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:23:15.469 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:23:15.469 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":309,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:21:45.047022+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:20:38.684224+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:18:55.148231+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:20:01.787973+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:23:15.690 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7789 8 365281 365273 0.366347 0 - 0.0299513 2026-02-23T15:23:16.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:23:16 trial068 ceph-mon[38805]: pgmap v4585: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:23:16.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:23:16 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/65151639' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:23:16.690 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7790 8 365281 365273 0.3663 0 - 0.0299513 2026-02-23T15:23:17.690 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7791 8 365281 365273 0.366253 0 - 0.0299513 2026-02-23T15:23:18.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:23:18 trial068 ceph-mon[38805]: pgmap v4586: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:23:18.505 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:23:18.623 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:23:18.690 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7792 8 365281 365273 0.366206 0 - 0.0299513 2026-02-23T15:23:18.726 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:23:18.725+0000 7fee777a2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:23:18.727 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:23:18.725+0000 7fee777a2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:23:18.836 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:23:18.836 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":309,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:21:45.047022+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:20:38.684224+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:18:55.148231+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:20:01.787973+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:23:19.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:23:19 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2809632356' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:23:19.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:23:19 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2026-02-23T15:23:19.690 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7793 8 365281 365273 0.366159 0 - 0.0299513 2026-02-23T15:23:20.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:23:20 trial068 ceph-mon[38805]: pgmap v4587: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:23:20.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:23:20 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-23T15:23:20.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:23:20 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2026-02-23T15:23:20.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:23:20 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T15:23:20.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:23:20 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2026-02-23T15:23:20.690 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7794 8 365281 365273 0.366112 0 - 0.0299513 2026-02-23T15:23:21.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:23:21 trial068 ceph-mon[38805]: pgmap v4588: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:23:21.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:23:21 trial068 ceph-mon[38805]: pgmap v4589: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:23:21.690 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7795 8 365281 365273 0.366065 0 - 0.0299513 2026-02-23T15:23:21.893 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:23:22.011 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:23:22.116 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:23:22.114+0000 7f58dbb1a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:23:22.116 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:23:22.114+0000 7f58dbb1a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:23:22.224 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:23:22.225 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":309,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:21:45.047022+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:20:38.684224+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:18:55.148231+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:20:01.787973+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:23:22.691 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7796 8 365281 365273 0.366018 0 - 0.0299513 2026-02-23T15:23:23.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:23:23 trial068 ceph-mon[38805]: pgmap v4590: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:23:23.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:23:23 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/217698110' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:23:23.691 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7797 8 365281 365273 0.365971 0 - 0.0299513 2026-02-23T15:23:24.244 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:23:24 trial068 ceph-mon[38805]: pgmap v4591: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:23:24.245 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:23:23 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:23:23] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:23:24.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:23:23 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:23:23.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:23:24.691 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7798 8 365281 365273 0.365924 0 - 0.0299513 2026-02-23T15:23:25.263 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:23:25.381 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:23:25.480 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:23:25.478+0000 7f09727d0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:23:25.480 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:23:25.478+0000 7f09727d0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:23:25.589 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:23:25.589 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":309,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:21:45.047022+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:20:38.684224+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:18:55.148231+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:20:01.787973+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:23:25.691 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7799 8 365281 365273 0.365877 0 - 0.0299513 2026-02-23T15:23:26.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:23:26 trial068 ceph-mon[38805]: pgmap v4592: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:23:26.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:23:26 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/4123981714' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:23:26.691 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T15:23:26.690908+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T15:23:26.691 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T15:23:26.691 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7800 8 365281 365273 0.365831 0 - 0.0299513 2026-02-23T15:23:27.691 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7801 8 365281 365273 0.365784 0 - 0.0299513 2026-02-23T15:23:28.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:23:28 trial068 ceph-mon[38805]: pgmap v4593: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:23:28.626 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:23:28.691 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7802 8 365281 365273 0.365737 0 - 0.0299513 2026-02-23T15:23:28.745 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:23:28.842 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:23:28.841+0000 7f021d741640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:23:28.842 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:23:28.841+0000 7f021d741640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:23:28.953 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:23:28.954 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":309,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:21:45.047022+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:20:38.684224+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:18:55.148231+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:20:01.787973+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:23:29.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:23:29 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3950031254' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:23:29.691 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7803 8 365281 365273 0.36569 0 - 0.0299513 2026-02-23T15:23:30.691 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7804 8 365281 365273 0.365643 0 - 0.0299513 2026-02-23T15:23:31.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:23:31 trial068 ceph-mon[38805]: pgmap v4594: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:23:31.691 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7805 8 365281 365273 0.365596 0 - 0.0299513 2026-02-23T15:23:31.991 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:23:32.111 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:23:32.214 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:23:32.212+0000 7f023e7ec640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:23:32.214 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:23:32.212+0000 7f023e7ec640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:23:32.324 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:23:32.324 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":309,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:21:45.047022+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:20:38.684224+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:18:55.148231+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:20:01.787973+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:23:32.691 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7806 8 365281 365273 0.365549 0 - 0.0299513 2026-02-23T15:23:33.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:23:33 trial068 ceph-mon[38805]: pgmap v4595: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:23:33.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:23:33 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3950292169' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:23:33.692 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7807 8 365281 365273 0.365503 0 - 0.0299513 2026-02-23T15:23:34.265 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:23:33 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:23:33.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:23:34.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:23:33 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:23:33] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:23:34.692 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7808 8 365281 365273 0.365456 0 - 0.0299513 2026-02-23T15:23:35.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:23:35 trial068 ceph-mon[38805]: pgmap v4596: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:23:35.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:23:35 trial068 ceph-mon[38805]: 5.8 scrub starts 2026-02-23T15:23:35.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:23:35 trial068 ceph-mon[38805]: 5.8 scrub ok 2026-02-23T15:23:35.360 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:23:35.475 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:23:35.576 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:23:35.574+0000 7fcefbebc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:23:35.576 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:23:35.574+0000 7fcefbebc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:23:35.684 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:23:35.684 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":309,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:21:45.047022+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:20:38.684224+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:18:55.148231+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:20:01.787973+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:23:35.691 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7809 8 365281 365273 0.365409 0 - 0.0299513 2026-02-23T15:23:36.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:23:36 trial068 ceph-mon[38805]: 5.e scrub starts 2026-02-23T15:23:36.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:23:36 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2886836098' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:23:36.692 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7810 8 365281 365273 0.365362 0 - 0.0299513 2026-02-23T15:23:37.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:23:37 trial068 ceph-mon[38805]: pgmap v4597: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:23:37.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:23:37 trial068 ceph-mon[38805]: 5.e scrub ok 2026-02-23T15:23:37.692 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7811 8 365281 365273 0.365315 0 - 0.0299513 2026-02-23T15:23:38.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:23:38 trial068 ceph-mon[38805]: pgmap v4598: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:23:38.692 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7812 8 365281 365273 0.365269 0 - 0.0299513 2026-02-23T15:23:38.723 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:23:38.838 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:23:38.943 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:23:38.941+0000 7fad87836640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:23:38.943 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:23:38.942+0000 7fad87836640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:23:39.052 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:23:39.052 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":309,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:21:45.047022+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:20:38.684224+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:18:55.148231+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:20:01.787973+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:23:39.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:23:39 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3374365227' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:23:39.692 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7813 8 365281 365273 0.365222 0 - 0.0299513 2026-02-23T15:23:40.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:23:40 trial068 ceph-mon[38805]: pgmap v4599: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:23:40.692 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7814 8 365281 365273 0.365175 0 - 0.0299513 2026-02-23T15:23:41.692 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7815 8 365281 365273 0.365128 0 - 0.0299513 2026-02-23T15:23:42.089 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:23:42.208 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:23:42.307 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:23:42.305+0000 7f653c800640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:23:42.307 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:23:42.306+0000 7f653c800640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:23:42.415 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:23:42.415 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":309,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:21:45.047022+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:20:38.684224+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:18:55.148231+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:20:01.787973+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:23:42.415 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:23:42 trial068 ceph-mon[38805]: pgmap v4600: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:23:42.692 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7816 8 365281 365273 0.365082 0 - 0.0299513 2026-02-23T15:23:43.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:23:43 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3199743430' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:23:43.692 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7817 8 365281 365273 0.365035 0 - 0.0299513 2026-02-23T15:23:44.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:23:44 trial068 ceph-mon[38805]: pgmap v4601: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:23:44.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:23:43 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:23:43] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:23:44.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:23:43 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:23:43.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:23:44.692 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7818 8 365281 365273 0.364988 0 - 0.0299513 2026-02-23T15:23:45.454 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:23:45.573 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:23:45.677 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:23:45.675+0000 7fa622f95640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:23:45.677 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:23:45.676+0000 7fa622f95640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:23:45.692 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7819 8 365281 365273 0.364942 0 - 0.0299513 2026-02-23T15:23:45.787 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:23:45.787 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":309,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:21:45.047022+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:20:38.684224+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:18:55.148231+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:20:01.787973+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:23:46.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:23:46 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/15652823' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:23:46.692 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T15:23:46.692595+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T15:23:46.693 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T15:23:46.693 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7820 8 365281 365273 0.364895 0 - 0.0299513 2026-02-23T15:23:47.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:23:47 trial068 ceph-mon[38805]: pgmap v4602: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:23:47.692 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7821 8 365281 365273 0.364848 0 - 0.0299513 2026-02-23T15:23:48.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:23:48 trial068 ceph-mon[38805]: 5.5 scrub starts 2026-02-23T15:23:48.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:23:48 trial068 ceph-mon[38805]: 5.5 scrub ok 2026-02-23T15:23:48.693 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7822 8 365281 365273 0.364802 0 - 0.0299513 2026-02-23T15:23:48.823 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:23:48.941 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:23:49.048 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:23:49.047+0000 7fea2f4a2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:23:49.048 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:23:49.047+0000 7fea2f4a2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:23:49.157 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:23:49.157 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":309,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:21:45.047022+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:20:38.684224+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:18:55.148231+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:20:01.787973+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:23:49.157 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:23:49 trial068 ceph-mon[38805]: pgmap v4603: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:23:49.693 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7823 8 365281 365273 0.364755 0 - 0.0299513 2026-02-23T15:23:50.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:23:50 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2761806541' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:23:50.693 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7824 8 365281 365273 0.364708 0 - 0.0299513 2026-02-23T15:23:51.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:23:51 trial068 ceph-mon[38805]: pgmap v4604: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:23:51.693 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7825 8 365281 365273 0.364662 0 - 0.0299513 2026-02-23T15:23:52.193 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:23:52.312 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:23:52.334 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:23:52 trial068 ceph-mon[38805]: pgmap v4605: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:23:52.415 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:23:52.413+0000 7f11c1045640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:23:52.415 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:23:52.413+0000 7f11c1045640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:23:52.524 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:23:52.524 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":309,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:21:45.047022+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:20:38.684224+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:18:55.148231+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:20:01.787973+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:23:52.693 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7826 8 365281 365273 0.364615 0 - 0.0299513 2026-02-23T15:23:53.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:23:53 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/662417215' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:23:53.693 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7827 8 365281 365273 0.364569 0 - 0.0299513 2026-02-23T15:23:54.244 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:23:54 trial068 ceph-mon[38805]: pgmap v4606: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:23:54.244 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:23:53 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:23:53] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:23:54.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:23:53 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:23:53.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:23:54.693 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7828 8 365281 365273 0.364522 0 - 0.0299513 2026-02-23T15:23:55.563 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:23:55.682 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:23:55.693 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7829 8 365281 365273 0.364475 0 - 0.0299513 2026-02-23T15:23:55.779 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:23:55.777+0000 7f0750a54640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:23:55.779 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:23:55.778+0000 7f0750a54640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:23:55.889 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:23:55.889 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":309,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:21:45.047022+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:20:38.684224+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:18:55.148231+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:20:01.787973+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:23:56.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:23:56 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1590511444' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:23:56.693 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7830 8 365281 365273 0.364429 0 - 0.0299513 2026-02-23T15:23:57.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:23:57 trial068 ceph-mon[38805]: pgmap v4607: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:23:57.693 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7831 8 365281 365273 0.364382 0 - 0.0299513 2026-02-23T15:23:58.693 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7832 8 365281 365273 0.364336 0 - 0.0299513 2026-02-23T15:23:58.926 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:23:59.044 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:23:59.142 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:23:59.140+0000 7f8b4639a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:23:59.142 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:23:59.140+0000 7f8b4639a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:23:59.251 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:23:59.251 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":309,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:21:45.047022+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:20:38.684224+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:18:55.148231+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:20:01.787973+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:23:59.251 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:23:59 trial068 ceph-mon[38805]: pgmap v4608: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:23:59.251 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:23:59 trial068 ceph-mon[38805]: 5.a scrub starts 2026-02-23T15:23:59.251 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:23:59 trial068 ceph-mon[38805]: 5.a scrub ok 2026-02-23T15:23:59.693 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7833 8 365281 365273 0.364289 0 - 0.0299513 2026-02-23T15:24:00.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:24:00 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2199840443' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:24:00.693 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7834 8 365281 365273 0.364243 0 - 0.0299513 2026-02-23T15:24:01.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:24:01 trial068 ceph-mon[38805]: pgmap v4609: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:24:01.693 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7835 8 365281 365273 0.364196 0 - 0.0299513 2026-02-23T15:24:02.289 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:24:02.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:24:02 trial068 ceph-mon[38805]: pgmap v4610: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:24:02.408 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:24:02.511 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:24:02.510+0000 7fcfea1a6640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:24:02.512 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:24:02.510+0000 7fcfea1a6640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:24:02.619 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:24:02.620 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":309,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:21:45.047022+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:20:38.684224+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:18:55.148231+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:20:01.787973+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:24:02.693 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7836 8 365281 365273 0.36415 0 - 0.0299513 2026-02-23T15:24:03.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:24:03 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/546631711' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:24:03.694 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7837 8 365281 365273 0.364103 0 - 0.0299513 2026-02-23T15:24:04.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:24:04 trial068 ceph-mon[38805]: pgmap v4611: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:24:04.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:24:03 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:24:03] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:24:04.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:24:03 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:24:03.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:24:04.694 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7838 8 365281 365273 0.364057 0 - 0.0299513 2026-02-23T15:24:05.657 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:24:05.694 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7839 8 365281 365273 0.364011 0 - 0.0299513 2026-02-23T15:24:05.775 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:24:05.873 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:24:05.871+0000 7f7ae8cf9640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:24:05.873 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:24:05.871+0000 7f7ae8cf9640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:24:05.983 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:24:05.983 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":309,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:21:45.047022+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:20:38.684224+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:18:55.148231+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:20:01.787973+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:24:06.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:24:06 trial068 ceph-mon[38805]: pgmap v4612: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:24:06.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:24:06 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/4239274948' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:24:06.694 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T15:24:06.693898+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T15:24:06.694 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T15:24:06.694 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7840 8 365281 365273 0.363964 0 - 0.0299513 2026-02-23T15:24:07.694 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7841 8 365281 365273 0.363918 0 - 0.0299513 2026-02-23T15:24:08.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:24:08 trial068 ceph-mon[38805]: pgmap v4613: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:24:08.694 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7842 8 365281 365273 0.363871 0 - 0.0299513 2026-02-23T15:24:09.020 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:24:09.139 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:24:09.243 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:24:09.242+0000 7f7ae52d1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:24:09.243 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:24:09.242+0000 7f7ae52d1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:24:09.354 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:24:09.354 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":309,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:21:45.047022+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:20:38.684224+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:18:55.148231+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:20:01.787973+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:24:09.694 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7843 8 365281 365273 0.363825 0 - 0.0299513 2026-02-23T15:24:10.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:24:10 trial068 ceph-mon[38805]: pgmap v4614: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:24:10.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:24:10 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/36376587' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:24:10.694 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7844 8 365281 365273 0.363778 0 - 0.0299513 2026-02-23T15:24:11.694 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7845 8 365281 365273 0.363732 0 - 0.0299513 2026-02-23T15:24:12.392 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:24:12.416 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:24:12 trial068 ceph-mon[38805]: pgmap v4615: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:24:12.510 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:24:12.611 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:24:12.610+0000 7f7a74d48640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:24:12.612 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:24:12.610+0000 7f7a74d48640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:24:12.694 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7846 8 365281 365273 0.363686 0 - 0.0299513 2026-02-23T15:24:12.720 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:24:12.720 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":309,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:21:45.047022+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:20:38.684224+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:18:55.148231+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:20:01.787973+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:24:13.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:24:13 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T15:24:13.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:24:13 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T15:24:13.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:24:13 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2175247178' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:24:13.694 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7847 8 365281 365273 0.363639 0 - 0.0299513 2026-02-23T15:24:14.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:24:14 trial068 ceph-mon[38805]: pgmap v4616: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:24:14.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:24:13 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:24:13] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:24:14.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:24:13 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:24:13.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:24:14.694 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7848 8 365281 365273 0.363593 0 - 0.0299513 2026-02-23T15:24:15.694 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7849 8 365281 365273 0.363547 0 - 0.0299513 2026-02-23T15:24:15.758 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:24:15.876 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:24:15.974 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:24:15.972+0000 7f7cfdcec640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:24:15.974 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:24:15.972+0000 7f7cfdcec640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:24:16.082 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:24:16.082 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":309,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:21:45.047022+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:20:38.684224+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:18:55.148231+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:20:01.787973+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:24:16.695 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7850 8 365281 365273 0.3635 0 - 0.0299513 2026-02-23T15:24:17.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:24:17 trial068 ceph-mon[38805]: pgmap v4617: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:24:17.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:24:17 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T15:24:17.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:24:17 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T15:24:17.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:24:17 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2827505015' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:24:17.695 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7851 8 365281 365273 0.363454 0 - 0.0299513 2026-02-23T15:24:18.695 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7852 8 365281 365273 0.363408 0 - 0.0299513 2026-02-23T15:24:19.120 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:24:19.234 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:24:19.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:24:19 trial068 ceph-mon[38805]: pgmap v4618: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:24:19.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:24:19 trial068 ceph-mon[38805]: osdmap e310: 4 total, 3 up, 4 in 2026-02-23T15:24:19.345 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:24:19.344+0000 7fb61241a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:24:19.345 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:24:19.344+0000 7fb61241a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:24:19.455 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:24:19.455 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":310,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:24:18.070348+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:20:38.684224+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:24:15.482052+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:20:01.787973+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:24:19.695 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7853 8 365281 365273 0.363362 0 - 0.0299513 2026-02-23T15:24:20.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:24:20 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T15:24:20.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:24:20 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T15:24:20.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:24:20 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2026-02-23T15:24:20.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:24:20 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-23T15:24:20.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:24:20 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2026-02-23T15:24:20.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:24:20 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T15:24:20.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:24:20 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2026-02-23T15:24:20.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:24:20 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1606660242' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:24:20.695 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7854 8 365281 365273 0.363315 0 - 0.0299513 2026-02-23T15:24:21.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:24:21 trial068 ceph-mon[38805]: pgmap v4620: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:24:21.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:24:21 trial068 ceph-mon[38805]: pgmap v4621: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:24:21.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:24:21 trial068 ceph-mon[38805]: pgmap v4622: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:24:21.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:24:21 trial068 ceph-mon[38805]: osdmap e311: 4 total, 3 up, 4 in 2026-02-23T15:24:21.695 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7855 8 365281 365273 0.363269 0 - 0.0299513 2026-02-23T15:24:22.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:24:22 trial068 ceph-mon[38805]: pgmap v4624: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:24:22.491 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:24:22.609 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:24:22.695 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7856 8 365281 365273 0.363223 0 - 0.0299513 2026-02-23T15:24:22.707 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:24:22.705+0000 7f80d6530640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:24:22.707 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:24:22.706+0000 7f80d6530640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:24:22.816 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:24:22.816 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":311,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:24:20.068033+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:20:38.684224+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:24:15.482052+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:20:01.787973+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:24:23.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:24:23 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3788693824' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:24:23.695 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7857 8 365281 365273 0.363177 0 - 0.0299513 2026-02-23T15:24:24.244 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:24:24 trial068 ceph-mon[38805]: pgmap v4625: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:24:24.245 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:24:23 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:24:23] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:24:24.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:24:23 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:24:23.974Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:24:24.695 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7858 8 365281 365273 0.36313 0 - 0.0299513 2026-02-23T15:24:25.695 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7859 8 365281 365273 0.363084 0 - 0.0299513 2026-02-23T15:24:25.853 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:24:25.972 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:24:26.076 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:24:26.074+0000 7f1fad376640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:24:26.076 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:24:26.074+0000 7f1fad376640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:24:26.185 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:24:26.185 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":311,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:24:20.068033+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:20:38.684224+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:24:15.482052+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:20:01.787973+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:24:26.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:24:26 trial068 ceph-mon[38805]: pgmap v4626: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:24:26.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:24:26 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1031333068' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:24:26.695 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T15:24:26.695312+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T15:24:26.695 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T15:24:26.695 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7860 8 365281 365273 0.363038 0 - 0.0299513 2026-02-23T15:24:27.695 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7861 8 365281 365273 0.362992 0 - 0.0299513 2026-02-23T15:24:28.695 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7862 8 365281 365273 0.362946 0 - 0.0299513 2026-02-23T15:24:28.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:24:28 trial068 ceph-mon[38805]: pgmap v4627: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:24:28.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:24:28 trial068 ceph-mon[38805]: 5.7 scrub starts 2026-02-23T15:24:29.223 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:24:29.341 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:24:29.446 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:24:29.444+0000 7f38ff128640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:24:29.446 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:24:29.445+0000 7f38ff128640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:24:29.556 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:24:29.556 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":311,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:24:20.068033+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:20:38.684224+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:24:15.482052+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:20:01.787973+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:24:29.557 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:24:29 trial068 ceph-mon[38805]: 5.7 scrub ok 2026-02-23T15:24:29.695 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7863 8 365281 365273 0.362899 0 - 0.0299513 2026-02-23T15:24:30.695 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7864 8 365281 365273 0.362853 0 - 0.0299513 2026-02-23T15:24:30.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:24:30 trial068 ceph-mon[38805]: pgmap v4628: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:24:30.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:24:30 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3589489290' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:24:31.695 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7865 8 365281 365273 0.362807 0 - 0.0299513 2026-02-23T15:24:32.593 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:24:32.695 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7866 8 365281 365273 0.362761 0 - 0.0299513 2026-02-23T15:24:32.712 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:24:32.735 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:24:32 trial068 ceph-mon[38805]: pgmap v4629: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:24:32.818 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:24:32.816+0000 7f03c6521640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:24:32.818 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:24:32.817+0000 7f03c6521640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:24:32.927 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:24:32.927 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":311,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:24:20.068033+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:20:38.684224+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:24:15.482052+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:20:01.787973+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:24:33.696 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7867 8 365281 365273 0.362715 0 - 0.0299513 2026-02-23T15:24:33.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:24:33 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/760519472' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:24:34.265 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:24:33 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:24:33.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:24:34.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:24:33 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:24:33] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:24:34.696 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7868 8 365281 365273 0.362669 0 - 0.0299513 2026-02-23T15:24:34.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:24:34 trial068 ceph-mon[38805]: pgmap v4630: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:24:35.696 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7869 8 365281 365273 0.362623 0 - 0.0299513 2026-02-23T15:24:35.965 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:24:36.084 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:24:36.182 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:24:36.180+0000 7f1e59555640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:24:36.182 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:24:36.181+0000 7f1e59555640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:24:36.292 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:24:36.292 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":311,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:24:20.068033+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:20:38.684224+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:24:15.482052+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:20:01.787973+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:24:36.696 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7870 8 365281 365273 0.362577 0 - 0.0299513 2026-02-23T15:24:37.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:24:37 trial068 ceph-mon[38805]: pgmap v4631: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:24:37.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:24:37 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/978299564' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:24:37.696 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7871 8 365281 365273 0.362531 0 - 0.0299513 2026-02-23T15:24:38.696 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7872 8 365281 365273 0.362485 0 - 0.0299513 2026-02-23T15:24:39.330 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:24:39.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:24:39 trial068 ceph-mon[38805]: pgmap v4632: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:24:39.449 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:24:39.560 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:24:39.559+0000 7f312645d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:24:39.561 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:24:39.559+0000 7f312645d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:24:39.672 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:24:39.672 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":311,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:24:20.068033+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:20:38.684224+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:24:15.482052+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:20:01.787973+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:24:39.696 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7873 8 365281 365273 0.362439 0 - 0.0299513 2026-02-23T15:24:40.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:24:40 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/50745040' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:24:40.696 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7874 8 365281 365273 0.362392 0 - 0.0299513 2026-02-23T15:24:41.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:24:41 trial068 ceph-mon[38805]: pgmap v4633: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:24:41.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:24:41 trial068 ceph-mon[38805]: 5.8 scrub starts 2026-02-23T15:24:41.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:24:41 trial068 ceph-mon[38805]: 5.8 scrub ok 2026-02-23T15:24:41.696 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7875 8 365281 365273 0.362346 0 - 0.0299513 2026-02-23T15:24:42.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:24:42 trial068 ceph-mon[38805]: pgmap v4634: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:24:42.696 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7876 8 365281 365273 0.3623 0 - 0.0299513 2026-02-23T15:24:42.709 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:24:42.827 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:24:42.932 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:24:42.930+0000 7f2485436640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:24:42.932 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:24:42.930+0000 7f2485436640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:24:43.040 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:24:43.040 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":311,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:24:20.068033+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:20:38.684224+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:24:15.482052+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:20:01.787973+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:24:43.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:24:43 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1458446963' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:24:43.696 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7877 8 365281 365273 0.362254 0 - 0.0299513 2026-02-23T15:24:44.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:24:44 trial068 ceph-mon[38805]: pgmap v4635: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:24:44.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:24:43 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:24:43] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:24:44.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:24:43 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:24:43.974Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:24:44.696 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7878 8 365281 365273 0.362208 0 - 0.0299513 2026-02-23T15:24:45.696 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7879 8 365281 365273 0.362163 0 - 0.0299513 2026-02-23T15:24:46.079 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:24:46.196 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:24:46.295 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:24:46.293+0000 7f85bbf8c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:24:46.295 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:24:46.294+0000 7f85bbf8c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:24:46.405 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:24:46.405 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":311,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:24:20.068033+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:20:38.684224+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:24:15.482052+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:20:01.787973+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:24:46.696 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T15:24:46.696530+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T15:24:46.696 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T15:24:46.697 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7880 8 365281 365273 0.362117 0 - 0.0299513 2026-02-23T15:24:47.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:24:47 trial068 ceph-mon[38805]: pgmap v4636: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:24:47.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:24:47 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2715406395' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:24:47.697 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7881 8 365281 365273 0.362071 0 - 0.0299513 2026-02-23T15:24:48.697 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7882 8 365281 365273 0.362025 0 - 0.0299513 2026-02-23T15:24:49.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:24:49 trial068 ceph-mon[38805]: pgmap v4637: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:24:49.442 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:24:49.559 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:24:49.658 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:24:49.657+0000 7f736a17d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:24:49.658 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:24:49.657+0000 7f736a17d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:24:49.697 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7883 8 365281 365273 0.361979 0 - 0.0299513 2026-02-23T15:24:49.769 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:24:49.770 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":311,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:24:20.068033+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:20:38.684224+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:24:15.482052+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:20:01.787973+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:24:50.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:24:50 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2454372784' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:24:50.697 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7884 8 365281 365273 0.361933 0 - 0.0299513 2026-02-23T15:24:51.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:24:51 trial068 ceph-mon[38805]: pgmap v4638: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:24:51.697 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7885 8 365281 365273 0.361887 0 - 0.0299513 2026-02-23T15:24:52.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:24:52 trial068 ceph-mon[38805]: pgmap v4639: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:24:52.697 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7886 8 365281 365273 0.361841 0 - 0.0299513 2026-02-23T15:24:52.807 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:24:52.925 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:24:53.022 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:24:53.020+0000 7fc15e203640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:24:53.022 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:24:53.021+0000 7fc15e203640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:24:53.131 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:24:53.131 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":311,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:24:20.068033+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:20:38.684224+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:24:15.482052+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:20:01.787973+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:24:53.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:24:53 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/947686031' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:24:53.697 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7887 8 365281 365273 0.361795 0 - 0.0299513 2026-02-23T15:24:54.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:24:54 trial068 ceph-mon[38805]: pgmap v4640: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:24:54.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:24:53 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:24:53] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:24:54.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:24:53 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:24:53.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:24:54.697 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7888 8 365281 365273 0.361749 0 - 0.0299513 2026-02-23T15:24:55.697 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7889 8 365281 365273 0.361703 0 - 0.0299513 2026-02-23T15:24:56.168 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:24:56.286 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:24:56.392 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:24:56.390+0000 7f66fb3ac640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:24:56.393 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:24:56.391+0000 7f66fb3ac640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:24:56.502 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:24:56.502 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":311,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:24:20.068033+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:20:38.684224+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:24:15.482052+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:20:01.787973+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:24:56.697 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7890 8 365281 365273 0.361658 0 - 0.0299513 2026-02-23T15:24:57.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:24:57 trial068 ceph-mon[38805]: pgmap v4641: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:24:57.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:24:57 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/735939107' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:24:57.697 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7891 8 365281 365273 0.361612 0 - 0.0299513 2026-02-23T15:24:58.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:24:58 trial068 ceph-mon[38805]: pgmap v4642: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:24:58.697 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7892 8 365281 365273 0.361566 0 - 0.0299513 2026-02-23T15:24:59.540 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:24:59.660 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:24:59.697 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7893 8 365281 365273 0.36152 0 - 0.0299513 2026-02-23T15:24:59.758 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:24:59.757+0000 7fa635795640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:24:59.759 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:24:59.757+0000 7fa635795640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:24:59.867 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:24:59.868 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":311,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:24:20.068033+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:20:38.684224+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:24:15.482052+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:20:01.787973+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:25:00.698 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7894 8 365281 365273 0.361474 0 - 0.0299513 2026-02-23T15:25:00.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:25:00 trial068 ceph-mon[38805]: pgmap v4643: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:25:00.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:25:00 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/672770044' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:25:00.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:25:00 trial068 ceph-mon[38805]: 5.e scrub starts 2026-02-23T15:25:01.697 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7895 8 365281 365273 0.361429 0 - 0.0299513 2026-02-23T15:25:01.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:25:01 trial068 ceph-mon[38805]: 5.e scrub ok 2026-02-23T15:25:02.697 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7896 8 365281 365273 0.361383 0 - 0.0299513 2026-02-23T15:25:02.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:25:02 trial068 ceph-mon[38805]: pgmap v4644: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:25:02.905 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:25:03.024 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:25:03.124 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:25:03.122+0000 7facf95f0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:25:03.124 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:25:03.123+0000 7facf95f0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:25:03.235 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:25:03.235 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":311,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:24:20.068033+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:20:38.684224+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:24:15.482052+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:20:01.787973+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:25:03.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:25:03 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2689272286' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:25:03.697 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7897 8 365281 365273 0.361337 0 - 0.0299513 2026-02-23T15:25:04.265 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:25:03 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:25:03.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:25:04.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:25:03 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:25:03] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:25:04.698 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7898 8 365281 365273 0.361291 0 - 0.0299513 2026-02-23T15:25:04.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:25:04 trial068 ceph-mon[38805]: pgmap v4645: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:25:05.698 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7899 8 365281 365273 0.361246 0 - 0.0299513 2026-02-23T15:25:06.273 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:25:06.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:25:06 trial068 ceph-mon[38805]: pgmap v4646: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:25:06.392 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:25:06.491 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:25:06.489+0000 7f4fcd420640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:25:06.491 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:25:06.489+0000 7f4fcd420640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:25:06.598 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:25:06.598 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":311,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:24:20.068033+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:20:38.684224+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:24:15.482052+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:20:01.787973+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:25:06.698 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T15:25:06.697905+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T15:25:06.698 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T15:25:06.698 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7900 8 365281 365273 0.3612 0 - 0.0299513 2026-02-23T15:25:07.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:25:07 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2924069917' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:25:07.698 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7901 8 365281 365273 0.361154 0 - 0.0299513 2026-02-23T15:25:08.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:25:08 trial068 ceph-mon[38805]: pgmap v4647: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:25:08.698 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7902 8 365281 365273 0.361108 0 - 0.0299513 2026-02-23T15:25:09.651 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:25:09.698 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7903 8 365281 365273 0.361063 0 - 0.0299513 2026-02-23T15:25:09.766 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:25:09.877 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:25:09.876+0000 7f61968bc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:25:09.878 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:25:09.876+0000 7f61968bc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:25:09.989 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:25:09.989 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":311,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:24:20.068033+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:20:38.684224+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:24:15.482052+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:20:01.787973+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:25:10.698 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7904 8 365281 365273 0.361017 0 - 0.0299513 2026-02-23T15:25:10.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:25:10 trial068 ceph-mon[38805]: pgmap v4648: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:25:10.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:25:10 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3705913431' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:25:11.698 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7905 8 365281 365273 0.360971 0 - 0.0299513 2026-02-23T15:25:12.698 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7906 8 365281 365273 0.360926 0 - 0.0299513 2026-02-23T15:25:12.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:25:12 trial068 ceph-mon[38805]: pgmap v4649: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:25:13.027 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:25:13.143 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:25:13.239 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:25:13.237+0000 7f899a712640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:25:13.239 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:25:13.237+0000 7f899a712640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:25:13.350 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:25:13.350 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":311,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:24:20.068033+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:20:38.684224+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:24:15.482052+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:20:01.787973+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:25:13.698 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7907 8 365281 365273 0.36088 0 - 0.0299513 2026-02-23T15:25:13.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:25:13 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/475832740' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:25:14.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:25:13 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:25:13] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:25:14.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:25:13 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:25:13.972Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:25:14.698 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7908 8 365281 365273 0.360834 0 - 0.0299513 2026-02-23T15:25:14.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:25:14 trial068 ceph-mon[38805]: pgmap v4650: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:25:14.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:25:14 trial068 ceph-mon[38805]: 5.5 scrub starts 2026-02-23T15:25:14.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:25:14 trial068 ceph-mon[38805]: 5.5 scrub ok 2026-02-23T15:25:15.698 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7909 8 365281 365273 0.360789 0 - 0.0299513 2026-02-23T15:25:16.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:25:16 trial068 ceph-mon[38805]: pgmap v4651: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:25:16.390 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:25:16.507 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:25:16.605 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:25:16.604+0000 7f43a8530640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:25:16.605 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:25:16.604+0000 7f43a8530640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:25:16.698 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7910 8 365281 365273 0.360743 0 - 0.0299513 2026-02-23T15:25:16.717 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:25:16.717 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":311,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:24:20.068033+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:20:38.684224+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:24:15.482052+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:20:01.787973+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:25:17.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:25:17 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3269258878' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:25:17.698 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7911 8 365281 365273 0.360698 0 - 0.0299513 2026-02-23T15:25:18.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:25:18 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T15:25:18.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:25:18 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T15:25:18.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:25:18 trial068 ceph-mon[38805]: pgmap v4652: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:25:18.698 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7912 8 365281 365273 0.360652 0 - 0.0299513 2026-02-23T15:25:19.699 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7913 8 365281 365273 0.360606 0 - 0.0299513 2026-02-23T15:25:19.754 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:25:19.778 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:25:19 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2026-02-23T15:25:19.870 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:25:19.981 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:25:19.979+0000 7fcaaa569640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:25:19.981 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:25:19.979+0000 7fcaaa569640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:25:20.089 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:25:20.089 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":311,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:24:20.068033+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:20:38.684224+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:24:15.482052+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:20:01.787973+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:25:20.699 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7914 8 365281 365273 0.360561 0 - 0.0299513 2026-02-23T15:25:20.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:25:20 trial068 ceph-mon[38805]: pgmap v4653: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:25:20.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:25:20 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-23T15:25:20.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:25:20 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2026-02-23T15:25:20.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:25:20 trial068 ceph-mon[38805]: pgmap v4654: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:25:20.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:25:20 trial068 ceph-mon[38805]: pgmap v4655: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:25:20.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:25:20 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T15:25:20.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:25:20 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2026-02-23T15:25:20.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:25:20 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2071155248' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:25:21.699 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7915 8 365281 365273 0.360515 0 - 0.0299513 2026-02-23T15:25:21.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:25:21 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T15:25:21.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:25:21 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T15:25:22.699 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7916 8 365281 365273 0.36047 0 - 0.0299513 2026-02-23T15:25:22.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:25:22 trial068 ceph-mon[38805]: pgmap v4656: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:25:23.129 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:25:23.248 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:25:23.346 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:25:23.345+0000 7ff8a04ee640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:25:23.347 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:25:23.345+0000 7ff8a04ee640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:25:23.459 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:25:23.459 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":311,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:24:20.068033+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:20:38.684224+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:24:15.482052+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:20:01.787973+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:25:23.699 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7917 8 365281 365273 0.360424 0 - 0.0299513 2026-02-23T15:25:23.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:25:23 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3318533695' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:25:24.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:25:23 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:25:23] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:25:24.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:25:23 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:25:23.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:25:24.699 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7918 8 365281 365273 0.360379 0 - 0.0299513 2026-02-23T15:25:24.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:25:24 trial068 ceph-mon[38805]: pgmap v4657: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:25:25.699 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7919 8 365281 365273 0.360333 0 - 0.0299513 2026-02-23T15:25:26.497 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:25:26.616 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:25:26.699 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T15:25:26.699229+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T15:25:26.699 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T15:25:26.699 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7920 8 365281 365273 0.360288 0 - 0.0299513 2026-02-23T15:25:26.722 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:25:26.720+0000 7fad0ddf7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:25:26.722 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:25:26.721+0000 7fad0ddf7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:25:26.832 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:25:26.833 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":311,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:24:20.068033+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:20:38.684224+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:24:15.482052+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:20:01.787973+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:25:26.833 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:25:26 trial068 ceph-mon[38805]: pgmap v4658: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:25:27.700 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7921 8 365281 365273 0.360242 0 - 0.0299513 2026-02-23T15:25:28.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:25:27 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3748542383' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:25:28.699 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7922 8 365281 365273 0.360197 0 - 0.0299513 2026-02-23T15:25:29.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:25:28 trial068 ceph-mon[38805]: pgmap v4659: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:25:29.699 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7923 8 365281 365273 0.360151 0 - 0.0299513 2026-02-23T15:25:29.869 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:25:29.989 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:25:30.089 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:25:30.087+0000 7fb23f6da640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:25:30.089 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:25:30.087+0000 7fb23f6da640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:25:30.202 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:25:30.203 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":311,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:24:20.068033+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:20:38.684224+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:24:15.482052+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:20:01.787973+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:25:30.699 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7924 8 365281 365273 0.360106 0 - 0.0299513 2026-02-23T15:25:31.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:25:30 trial068 ceph-mon[38805]: pgmap v4660: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:25:31.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:25:30 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/99460095' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:25:31.700 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7925 8 365281 365273 0.36006 0 - 0.0299513 2026-02-23T15:25:32.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:25:31 trial068 ceph-mon[38805]: 5.a scrub starts 2026-02-23T15:25:32.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:25:31 trial068 ceph-mon[38805]: 5.a scrub ok 2026-02-23T15:25:32.700 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7926 8 365281 365273 0.360015 0 - 0.0299513 2026-02-23T15:25:33.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:25:32 trial068 ceph-mon[38805]: pgmap v4661: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:25:33.240 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:25:33.356 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:25:33.455 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:25:33.453+0000 7ff0d3b0a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:25:33.455 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:25:33.453+0000 7ff0d3b0a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:25:33.564 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:25:33.564 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":311,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:24:20.068033+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:20:38.684224+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:24:15.482052+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:20:01.787973+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:25:33.700 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7927 8 365281 365273 0.35997 0 - 0.0299513 2026-02-23T15:25:33.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:25:33 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2573452783' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:25:34.265 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:25:33 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:25:33.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:25:34.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:25:33 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:25:33] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:25:34.700 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7928 8 365281 365273 0.359924 0 - 0.0299513 2026-02-23T15:25:35.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:25:34 trial068 ceph-mon[38805]: pgmap v4662: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:25:35.700 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7929 8 365281 365273 0.359879 0 - 0.0299513 2026-02-23T15:25:36.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:25:36 trial068 ceph-mon[38805]: pgmap v4663: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:25:36.601 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:25:36.700 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7930 8 365281 365273 0.359833 0 - 0.0299513 2026-02-23T15:25:36.716 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:25:36.814 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:25:36.812+0000 7fce0d7e1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:25:36.814 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:25:36.812+0000 7fce0d7e1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:25:36.923 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:25:36.924 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":312,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:25:36.080493+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:20:38.684224+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:24:15.482052+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:25:16.638959+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:25:37.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:25:37 trial068 ceph-mon[38805]: osdmap e312: 4 total, 3 up, 4 in 2026-02-23T15:25:37.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:25:37 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1130877367' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:25:37.700 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7931 8 365281 365273 0.359788 0 - 0.0299513 2026-02-23T15:25:38.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:25:38 trial068 ceph-mon[38805]: pgmap v4665: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:25:38.700 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7932 8 365281 365273 0.359743 0 - 0.0299513 2026-02-23T15:25:39.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:25:39 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T15:25:39.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:25:39 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T15:25:39.700 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7933 8 365281 365273 0.359697 0 - 0.0299513 2026-02-23T15:25:39.960 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:25:40.079 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:25:40.176 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:25:40.174+0000 7ff02a537640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:25:40.176 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:25:40.175+0000 7ff02a537640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:25:40.287 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:25:40.287 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":313,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:25:40.079604+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:20:38.684224+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:24:15.482052+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:25:16.638959+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:25:40.288 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:25:40 trial068 ceph-mon[38805]: pgmap v4666: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:25:40.288 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:25:40 trial068 ceph-mon[38805]: osdmap e313: 4 total, 3 up, 4 in 2026-02-23T15:25:40.700 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7934 8 365281 365273 0.359652 0 - 0.0299513 2026-02-23T15:25:41.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:25:41 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3476284984' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:25:41.700 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7935 8 365281 365273 0.359607 0 - 0.0299513 2026-02-23T15:25:42.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:25:42 trial068 ceph-mon[38805]: pgmap v4668: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:25:42.700 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7936 8 365281 365273 0.359561 0 - 0.0299513 2026-02-23T15:25:43.325 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:25:43.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:25:43 trial068 ceph-mon[38805]: 5.7 scrub starts 2026-02-23T15:25:43.442 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:25:43.539 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:25:43.537+0000 7f45cf84a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:25:43.539 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:25:43.538+0000 7f45cf84a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:25:43.648 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:25:43.648 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":313,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:25:40.079604+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:20:38.684224+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:24:15.482052+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:25:16.638959+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:25:43.700 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7937 8 365281 365273 0.359516 0 - 0.0299513 2026-02-23T15:25:44.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:25:43 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:25:43] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:25:44.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:25:44 trial068 ceph-mon[38805]: 5.7 scrub ok 2026-02-23T15:25:44.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:25:44 trial068 ceph-mon[38805]: pgmap v4669: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:25:44.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:25:44 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/4160070519' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:25:44.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:25:43 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:25:43.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:25:44.700 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7938 8 365281 365273 0.359471 0 - 0.0299513 2026-02-23T15:25:45.700 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7939 8 365281 365273 0.359425 0 - 0.0299513 2026-02-23T15:25:46.686 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:25:46.700 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T15:25:46.700640+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T15:25:46.701 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T15:25:46.701 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7940 8 365281 365273 0.35938 0 - 0.0299513 2026-02-23T15:25:46.803 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:25:46.902 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:25:46.900+0000 7f1889b19640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:25:46.902 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:25:46.900+0000 7f1889b19640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:25:46.918 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:25:46 trial068 ceph-mon[38805]: pgmap v4670: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:25:47.011 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:25:47.011 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":313,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:25:40.079604+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:20:38.684224+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:24:15.482052+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:25:16.638959+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:25:47.701 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7941 8 365281 365273 0.359335 0 - 0.0299513 2026-02-23T15:25:48.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:25:47 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2112339659' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:25:48.701 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7942 8 365281 365273 0.35929 0 - 0.0299513 2026-02-23T15:25:49.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:25:48 trial068 ceph-mon[38805]: pgmap v4671: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:25:49.701 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7943 8 365281 365273 0.359244 0 - 0.0299513 2026-02-23T15:25:50.048 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:25:50.167 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:25:50.275 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:25:50.273+0000 7f636586f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:25:50.275 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:25:50.273+0000 7f636586f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:25:50.386 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:25:50.386 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":313,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:25:40.079604+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:20:38.684224+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:24:15.482052+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:25:16.638959+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:25:50.701 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7944 8 365281 365273 0.359199 0 - 0.0299513 2026-02-23T15:25:51.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:25:50 trial068 ceph-mon[38805]: pgmap v4672: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:25:51.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:25:50 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2304920601' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:25:51.701 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7945 8 365281 365273 0.359154 0 - 0.0299513 2026-02-23T15:25:52.701 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7946 8 365281 365273 0.359109 0 - 0.0299513 2026-02-23T15:25:53.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:25:52 trial068 ceph-mon[38805]: pgmap v4673: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:25:53.423 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:25:53.539 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:25:53.633 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:25:53.632+0000 7fe4d1358640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:25:53.633 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:25:53.632+0000 7fe4d1358640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:25:53.701 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7947 8 365281 365273 0.359064 0 - 0.0299513 2026-02-23T15:25:53.742 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:25:53.742 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":313,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:25:40.079604+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:20:38.684224+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:24:15.482052+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:25:16.638959+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:25:53.742 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:25:53 trial068 ceph-mon[38805]: 5.8 scrub starts 2026-02-23T15:25:54.093 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:25:53 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:25:53] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:25:54.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:25:53 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:25:53.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:25:54.701 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7948 8 365281 365273 0.359018 0 - 0.0299513 2026-02-23T15:25:55.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:25:54 trial068 ceph-mon[38805]: 5.8 scrub ok 2026-02-23T15:25:55.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:25:54 trial068 ceph-mon[38805]: pgmap v4674: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:25:55.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:25:54 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3433776619' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:25:55.701 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7949 8 365281 365273 0.358973 0 - 0.0299513 2026-02-23T15:25:56.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:25:56 trial068 ceph-mon[38805]: pgmap v4675: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:25:56.701 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7950 8 365281 365273 0.358928 0 - 0.0299513 2026-02-23T15:25:56.779 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:25:56.898 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:25:56.998 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:25:56.996+0000 7ff72a0ae640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:25:56.998 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:25:56.997+0000 7ff72a0ae640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:25:57.107 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:25:57.107 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":313,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:25:40.079604+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:20:38.684224+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:24:15.482052+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:25:16.638959+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:25:57.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:25:57 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3156178563' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:25:57.702 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7951 8 365281 365273 0.358883 0 - 0.0299513 2026-02-23T15:25:58.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:25:58 trial068 ceph-mon[38805]: pgmap v4676: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:25:58.702 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7952 8 365281 365273 0.358838 0 - 0.0299513 2026-02-23T15:25:59.702 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7953 8 365281 365273 0.358793 0 - 0.0299513 2026-02-23T15:26:00.144 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:26:00.264 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:26:00.365 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:26:00.363+0000 7f6e15c53640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:26:00.365 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:26:00.363+0000 7f6e15c53640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:26:00.474 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:26:00.474 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":313,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:25:40.079604+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:20:38.684224+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:24:15.482052+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:25:16.638959+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:26:00.702 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7954 8 365281 365273 0.358748 0 - 0.0299513 2026-02-23T15:26:00.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:26:00 trial068 ceph-mon[38805]: pgmap v4677: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:26:00.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:26:00 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/4089118360' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:26:01.702 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7955 8 365281 365273 0.358703 0 - 0.0299513 2026-02-23T15:26:02.702 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7956 8 365281 365273 0.358657 0 - 0.0299513 2026-02-23T15:26:03.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:26:02 trial068 ceph-mon[38805]: pgmap v4678: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:26:03.514 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:26:03.632 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:26:03.702 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7957 8 365281 365273 0.358612 0 - 0.0299513 2026-02-23T15:26:03.732 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:26:03.730+0000 7f41d8ab4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:26:03.732 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:26:03.731+0000 7f41d8ab4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:26:03.841 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:26:03.841 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":313,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:25:40.079604+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:20:38.684224+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:24:15.482052+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:25:16.638959+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:26:04.093 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:26:03 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:26:03] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:26:04.265 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:26:03 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:26:03.974Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:26:04.702 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7958 8 365281 365273 0.358567 0 - 0.0299513 2026-02-23T15:26:05.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:26:04 trial068 ceph-mon[38805]: pgmap v4679: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:26:05.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:26:04 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1098574439' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:26:05.702 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7959 8 365281 365273 0.358522 0 - 0.0299513 2026-02-23T15:26:06.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:26:06 trial068 ceph-mon[38805]: pgmap v4680: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:26:06.702 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T15:26:06.702217+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T15:26:06.702 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T15:26:06.702 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7960 8 365281 365273 0.358477 0 - 0.0299513 2026-02-23T15:26:06.878 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:26:06.996 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:26:07.093 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:26:07.091+0000 7f1b145d5640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:26:07.093 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:26:07.092+0000 7f1b145d5640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:26:07.202 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:26:07.202 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":313,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:25:40.079604+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:20:38.684224+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:24:15.482052+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:25:16.638959+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:26:07.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:26:07 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1082025983' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:26:07.702 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7961 8 365281 365273 0.358432 0 - 0.0299513 2026-02-23T15:26:08.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:26:08 trial068 ceph-mon[38805]: pgmap v4681: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:26:08.702 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7962 8 365281 365273 0.358387 0 - 0.0299513 2026-02-23T15:26:09.702 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7963 8 365281 365273 0.358342 0 - 0.0299513 2026-02-23T15:26:10.240 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:26:10.357 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:26:10.462 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:26:10.461+0000 7fcb5ac5a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:26:10.463 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:26:10.461+0000 7fcb5ac5a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:26:10.572 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:26:10.572 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":313,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:25:40.079604+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:20:38.684224+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:24:15.482052+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:25:16.638959+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:26:10.702 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7964 8 365281 365273 0.358297 0 - 0.0299513 2026-02-23T15:26:10.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:26:10 trial068 ceph-mon[38805]: pgmap v4682: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:26:10.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:26:10 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/4011952674' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:26:11.702 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7965 8 365281 365273 0.358252 0 - 0.0299513 2026-02-23T15:26:12.703 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7966 8 365281 365273 0.358207 0 - 0.0299513 2026-02-23T15:26:13.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:26:12 trial068 ceph-mon[38805]: pgmap v4683: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:26:13.609 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:26:13.702 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7967 8 365281 365273 0.358162 0 - 0.0299513 2026-02-23T15:26:13.728 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:26:13.829 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:26:13.827+0000 7f2d6c0c2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:26:13.829 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:26:13.827+0000 7f2d6c0c2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:26:13.937 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:26:13.938 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":313,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:25:40.079604+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:20:38.684224+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:24:15.482052+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:25:16.638959+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:26:14.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:26:13 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:26:13] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:26:14.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:26:13 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:26:13.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:26:14.702 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7968 8 365281 365273 0.358117 0 - 0.0299513 2026-02-23T15:26:15.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:26:14 trial068 ceph-mon[38805]: pgmap v4684: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:26:15.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:26:14 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/61061417' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:26:15.702 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7969 8 365281 365273 0.358072 0 - 0.0299513 2026-02-23T15:26:16.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:26:16 trial068 ceph-mon[38805]: pgmap v4685: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:26:16.703 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7970 8 365281 365273 0.358027 0 - 0.0299513 2026-02-23T15:26:16.974 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:26:17.089 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:26:17.188 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:26:17.186+0000 7f6a91bae640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:26:17.188 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:26:17.186+0000 7f6a91bae640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:26:17.298 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:26:17.298 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":313,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:25:40.079604+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:20:38.684224+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:24:15.482052+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:25:16.638959+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:26:17.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:26:17 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T15:26:17.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:26:17 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T15:26:17.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:26:17 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3028291661' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:26:17.703 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7971 8 365281 365273 0.357983 0 - 0.0299513 2026-02-23T15:26:18.703 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7972 8 365281 365273 0.357938 0 - 0.0299513 2026-02-23T15:26:18.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:26:18 trial068 ceph-mon[38805]: pgmap v4686: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:26:19.703 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7973 8 365281 365273 0.357893 0 - 0.0299513 2026-02-23T15:26:20.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:26:19 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2026-02-23T15:26:20.335 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:26:20.452 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:26:20.553 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:26:20.551+0000 7f813334f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:26:20.553 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:26:20.551+0000 7f813334f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:26:20.661 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:26:20.661 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":313,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:25:40.079604+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:20:38.684224+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:24:15.482052+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:25:16.638959+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:26:20.703 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7974 8 365281 365273 0.357848 0 - 0.0299513 2026-02-23T15:26:21.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:26:20 trial068 ceph-mon[38805]: pgmap v4687: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:26:21.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:26:20 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-23T15:26:21.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:26:20 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2026-02-23T15:26:21.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:26:20 trial068 ceph-mon[38805]: pgmap v4688: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:26:21.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:26:20 trial068 ceph-mon[38805]: pgmap v4689: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:26:21.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:26:20 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T15:26:21.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:26:20 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2026-02-23T15:26:21.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:26:20 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/400502469' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:26:21.703 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7975 8 365281 365273 0.357803 0 - 0.0299513 2026-02-23T15:26:22.703 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7976 8 365281 365273 0.357758 0 - 0.0299513 2026-02-23T15:26:23.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:26:22 trial068 ceph-mon[38805]: pgmap v4690: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:26:23.699 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:26:23.703 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7977 8 365281 365273 0.357713 0 - 0.0299513 2026-02-23T15:26:23.816 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:26:23.915 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:26:23.914+0000 7fc949be1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:26:23.916 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:26:23.914+0000 7fc949be1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:26:24.027 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:26:23 trial068 ceph-mon[38805]: 5.e scrub starts 2026-02-23T15:26:24.028 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:26:23 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:26:23] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:26:24.028 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:26:24.028 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":313,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:25:40.079604+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:20:38.684224+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:24:15.482052+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:25:16.638959+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:26:24.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:26:23 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:26:23.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:26:24.703 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7978 8 365281 365273 0.357668 0 - 0.0299513 2026-02-23T15:26:25.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:26:24 trial068 ceph-mon[38805]: 5.e scrub ok 2026-02-23T15:26:25.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:26:24 trial068 ceph-mon[38805]: pgmap v4691: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:26:25.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:26:24 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3050870090' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:26:25.704 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7979 8 365281 365273 0.357624 0 - 0.0299513 2026-02-23T15:26:26.704 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T15:26:26.703776+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T15:26:26.704 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T15:26:26.704 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7980 8 365281 365273 0.357579 0 - 0.0299513 2026-02-23T15:26:27.065 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:26:27.182 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:26:27.202 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:26:26 trial068 ceph-mon[38805]: pgmap v4692: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:26:27.283 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:26:27.282+0000 7f8a270f6640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:26:27.284 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:26:27.282+0000 7f8a270f6640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:26:27.392 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:26:27.392 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":313,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:25:40.079604+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:20:38.684224+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:24:15.482052+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:25:16.638959+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:26:27.704 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7981 8 365281 365273 0.357534 0 - 0.0299513 2026-02-23T15:26:28.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:26:27 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1155165083' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:26:28.704 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7982 8 365281 365273 0.357489 0 - 0.0299513 2026-02-23T15:26:29.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:26:28 trial068 ceph-mon[38805]: pgmap v4693: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:26:29.704 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7983 8 365281 365273 0.357444 0 - 0.0299513 2026-02-23T15:26:30.445 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:26:30.561 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:26:30.664 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:26:30.662+0000 7f0937044640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:26:30.664 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:26:30.662+0000 7f0937044640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:26:30.704 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7984 8 365281 365273 0.3574 0 - 0.0299513 2026-02-23T15:26:30.774 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:26:30.774 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":313,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:25:40.079604+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:20:38.684224+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:24:15.482052+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:25:16.638959+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:26:31.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:26:30 trial068 ceph-mon[38805]: pgmap v4694: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:26:31.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:26:30 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/4189849520' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:26:31.704 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7985 8 365281 365273 0.357355 0 - 0.0299513 2026-02-23T15:26:32.704 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7986 8 365281 365273 0.35731 0 - 0.0299513 2026-02-23T15:26:33.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:26:32 trial068 ceph-mon[38805]: pgmap v4695: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:26:33.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:26:32 trial068 ceph-mon[38805]: 5.5 scrub starts 2026-02-23T15:26:33.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:26:32 trial068 ceph-mon[38805]: 5.5 scrub ok 2026-02-23T15:26:33.704 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7987 8 365281 365273 0.357265 0 - 0.0299513 2026-02-23T15:26:33.811 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:26:33.930 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:26:34.030 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:26:34.028+0000 7fbd06321640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:26:34.030 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:26:34.028+0000 7fbd06321640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:26:34.093 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:26:33 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:26:33] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:26:34.141 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:26:34.141 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":313,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:25:40.079604+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:20:38.684224+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:24:15.482052+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:25:16.638959+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:26:34.265 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:26:33 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:26:33.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:26:34.704 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7988 8 365281 365273 0.357221 0 - 0.0299513 2026-02-23T15:26:35.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:26:34 trial068 ceph-mon[38805]: pgmap v4696: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:26:35.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:26:34 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3822910550' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:26:35.704 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7989 8 365281 365273 0.357176 0 - 0.0299513 2026-02-23T15:26:36.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:26:36 trial068 ceph-mon[38805]: pgmap v4697: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:26:36.704 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7990 8 365281 365273 0.357131 0 - 0.0299513 2026-02-23T15:26:37.179 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:26:37.296 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:26:37.392 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:26:37.391+0000 7fe5c77d9640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:26:37.392 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:26:37.391+0000 7fe5c77d9640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:26:37.501 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:26:37.501 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":313,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:25:40.079604+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:20:38.684224+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:24:15.482052+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:25:16.638959+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:26:37.704 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7991 8 365281 365273 0.357087 0 - 0.0299513 2026-02-23T15:26:37.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:26:37 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/387778500' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:26:38.704 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7992 8 365281 365273 0.357042 0 - 0.0299513 2026-02-23T15:26:38.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:26:38 trial068 ceph-mon[38805]: pgmap v4698: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:26:39.705 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7993 8 365281 365273 0.356997 0 - 0.0299513 2026-02-23T15:26:40.541 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:26:40.658 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:26:40.705 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7994 8 365281 365273 0.356953 0 - 0.0299513 2026-02-23T15:26:40.755 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:26:40.753+0000 7ff5e2667640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:26:40.755 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:26:40.753+0000 7ff5e2667640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:26:40.863 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:26:40.863 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":313,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:25:40.079604+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:20:38.684224+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:24:15.482052+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:25:16.638959+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:26:41.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:26:40 trial068 ceph-mon[38805]: pgmap v4699: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:26:41.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:26:40 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/4288636481' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:26:41.705 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7995 8 365281 365273 0.356908 0 - 0.0299513 2026-02-23T15:26:42.705 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7996 8 365281 365273 0.356863 0 - 0.0299513 2026-02-23T15:26:43.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:26:42 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T15:26:43.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:26:42 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T15:26:43.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:26:42 trial068 ceph-mon[38805]: pgmap v4700: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:26:43.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:26:42 trial068 ceph-mon[38805]: osdmap e314: 4 total, 3 up, 4 in 2026-02-23T15:26:43.705 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7997 8 365281 365273 0.356819 0 - 0.0299513 2026-02-23T15:26:43.901 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:26:44.020 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:26:44.125 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:26:44.123+0000 7f5ed8aba640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:26:44.125 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:26:44.123+0000 7f5ed8aba640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:26:44.237 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:26:44.237 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":314,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:26:41.918848+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:25:20.908986+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:24:15.482052+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:25:16.638959+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:26:44.237 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:26:43 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:26:43] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:26:44.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:26:43 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:26:43.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:26:44.705 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7998 8 365281 365273 0.356774 0 - 0.0299513 2026-02-23T15:26:45.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:26:44 trial068 ceph-mon[38805]: pgmap v4702: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:26:45.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:26:44 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3921256940' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:26:45.705 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 7999 8 365281 365273 0.356729 0 - 0.0299513 2026-02-23T15:26:46.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:26:46 trial068 ceph-mon[38805]: osdmap e315: 4 total, 3 up, 4 in 2026-02-23T15:26:46.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:26:46 trial068 ceph-mon[38805]: pgmap v4704: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:26:46.705 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T15:26:46.705457+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T15:26:46.705 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T15:26:46.706 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8000 8 365281 365273 0.356685 0 - 0.0299513 2026-02-23T15:26:47.274 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:26:47.393 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:26:47.495 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:26:47.493+0000 7fdf40ce6640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:26:47.495 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:26:47.493+0000 7fdf40ce6640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:26:47.604 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:26:47.604 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":315,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:26:45.087567+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:25:20.908986+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:24:15.482052+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:25:16.638959+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:26:47.705 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8001 8 365281 365273 0.35664 0 - 0.0299513 2026-02-23T15:26:48.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:26:47 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/4123773900' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:26:48.705 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8002 8 365281 365273 0.356596 0 - 0.0299513 2026-02-23T15:26:49.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:26:48 trial068 ceph-mon[38805]: pgmap v4705: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:26:49.705 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8003 8 365281 365273 0.356551 0 - 0.0299513 2026-02-23T15:26:50.641 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:26:50.706 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8004 8 365281 365273 0.356507 0 - 0.0299513 2026-02-23T15:26:50.761 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:26:50.862 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:26:50.861+0000 7fbf32d4e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:26:50.863 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:26:50.861+0000 7fbf32d4e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:26:50.971 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:26:50.971 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":315,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:26:45.087567+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:25:20.908986+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:24:15.482052+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:25:16.638959+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:26:51.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:26:50 trial068 ceph-mon[38805]: pgmap v4706: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:26:51.706 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8005 8 365281 365273 0.356462 0 - 0.0299513 2026-02-23T15:26:52.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:26:51 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3673593278' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:26:52.705 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8006 8 365281 365273 0.356418 0 - 0.0299513 2026-02-23T15:26:53.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:26:52 trial068 ceph-mon[38805]: pgmap v4707: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:26:53.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:26:52 trial068 ceph-mon[38805]: 5.a scrub starts 2026-02-23T15:26:53.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:26:52 trial068 ceph-mon[38805]: 5.a scrub ok 2026-02-23T15:26:53.706 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8007 8 365281 365273 0.356373 0 - 0.0299513 2026-02-23T15:26:54.010 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:26:54.127 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:26:54.226 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:26:54.224+0000 7fcb8c6d6640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:26:54.226 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:26:54.225+0000 7fcb8c6d6640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:26:54.227 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:26:53 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:26:53] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:26:54.336 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:26:54.336 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":315,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:26:45.087567+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:25:20.908986+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:24:15.482052+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:25:16.638959+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:26:54.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:26:53 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:26:53.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:26:54.706 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8008 8 365281 365273 0.356329 0 - 0.0299513 2026-02-23T15:26:55.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:26:54 trial068 ceph-mon[38805]: pgmap v4708: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:26:55.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:26:54 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/487906634' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:26:55.706 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8009 8 365281 365273 0.356284 0 - 0.0299513 2026-02-23T15:26:56.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:26:56 trial068 ceph-mon[38805]: pgmap v4709: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:26:56.706 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8010 8 365281 365273 0.35624 0 - 0.0299513 2026-02-23T15:26:57.373 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:26:57.491 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:26:57.592 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:26:57.590+0000 7ff9ee10e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:26:57.592 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:26:57.590+0000 7ff9ee10e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:26:57.704 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:26:57.704 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":315,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:26:45.087567+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:25:20.908986+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:24:15.482052+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:25:16.638959+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:26:57.706 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8011 8 365281 365273 0.356195 0 - 0.0299513 2026-02-23T15:26:58.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:26:57 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1300342352' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:26:58.706 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8012 8 365281 365273 0.356151 0 - 0.0299513 2026-02-23T15:26:59.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:26:58 trial068 ceph-mon[38805]: pgmap v4710: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:26:59.706 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8013 8 365281 365273 0.356106 0 - 0.0299513 2026-02-23T15:27:00.706 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8014 8 365281 365273 0.356062 0 - 0.0299513 2026-02-23T15:27:00.741 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:27:00.857 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:27:00.957 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:27:00.956+0000 7f74cefa1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:27:00.958 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:27:00.956+0000 7f74cefa1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:27:01.065 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:27:01.066 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":315,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:26:45.087567+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:25:20.908986+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:24:15.482052+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:25:16.638959+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:27:01.066 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:27:00 trial068 ceph-mon[38805]: pgmap v4711: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:27:01.706 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8015 8 365281 365273 0.356017 0 - 0.0299513 2026-02-23T15:27:02.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:27:01 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2634429195' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:27:02.706 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8016 8 365281 365273 0.355973 0 - 0.0299513 2026-02-23T15:27:03.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:27:02 trial068 ceph-mon[38805]: pgmap v4712: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:27:03.706 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8017 8 365281 365273 0.355928 0 - 0.0299513 2026-02-23T15:27:04.100 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:27:04.218 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:27:04.240 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:27:03 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:27:03] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:27:04.265 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:27:03 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:27:03.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:27:04.316 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:27:04.314+0000 7f00bc7bb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:27:04.316 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:27:04.314+0000 7f00bc7bb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:27:04.427 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:27:04.428 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":315,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:26:45.087567+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:25:20.908986+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:24:15.482052+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:25:16.638959+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:27:04.706 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8018 8 365281 365273 0.355884 0 - 0.0299513 2026-02-23T15:27:05.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:27:04 trial068 ceph-mon[38805]: pgmap v4713: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:27:05.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:27:04 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1347744351' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:27:05.706 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8019 8 365281 365273 0.35584 0 - 0.0299513 2026-02-23T15:27:06.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:27:05 trial068 ceph-mon[38805]: 5.7 scrub starts 2026-02-23T15:27:06.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:27:05 trial068 ceph-mon[38805]: 5.7 scrub ok 2026-02-23T15:27:06.707 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T15:27:06.706881+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T15:27:06.707 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T15:27:06.707 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8020 8 365281 365273 0.355795 0 - 0.0299513 2026-02-23T15:27:07.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:27:06 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T15:27:07.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:27:06 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T15:27:07.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:27:06 trial068 ceph-mon[38805]: pgmap v4714: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:27:07.469 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:27:07.587 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:27:07.691 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:27:07.690+0000 7f8ff6569640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:27:07.691 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:27:07.690+0000 7f8ff6569640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:27:07.707 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8021 8 365281 365273 0.355751 0 - 0.0299513 2026-02-23T15:27:07.801 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:27:07.801 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":315,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:26:45.087567+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:25:20.908986+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:24:15.482052+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:25:16.638959+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:27:08.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:27:07 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1809901406' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:27:08.707 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8022 8 365281 365273 0.355707 0 - 0.0299513 2026-02-23T15:27:09.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:27:08 trial068 ceph-mon[38805]: pgmap v4715: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:27:09.707 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8023 8 365281 365273 0.355662 0 - 0.0299513 2026-02-23T15:27:10.707 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8024 8 365281 365273 0.355618 0 - 0.0299513 2026-02-23T15:27:10.839 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:27:10.955 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:27:11.056 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:27:11.055+0000 7f98dafda640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:27:11.057 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:27:11.055+0000 7f98dafda640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:27:11.169 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:27:11.169 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":315,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:26:45.087567+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:25:20.908986+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:24:15.482052+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:25:16.638959+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:27:11.169 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:27:10 trial068 ceph-mon[38805]: pgmap v4716: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:27:11.707 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8025 8 365281 365273 0.355574 0 - 0.0299513 2026-02-23T15:27:12.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:27:11 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/68717534' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:27:12.708 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8026 8 365281 365273 0.355529 0 - 0.0299513 2026-02-23T15:27:13.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:27:12 trial068 ceph-mon[38805]: pgmap v4717: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:27:13.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:27:12 trial068 ceph-mon[38805]: 5.8 scrub starts 2026-02-23T15:27:13.708 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8027 8 365281 365273 0.355485 0 - 0.0299513 2026-02-23T15:27:14.205 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:27:14.230 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:27:13 trial068 ceph-mon[38805]: 5.8 scrub ok 2026-02-23T15:27:14.230 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:27:13 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:27:13] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:27:14.321 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:27:14.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:27:13 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:27:13.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:27:14.420 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:27:14.419+0000 7fc3c9a84640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:27:14.420 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:27:14.419+0000 7fc3c9a84640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:27:14.529 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:27:14.529 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":315,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:26:45.087567+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:25:20.908986+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:24:15.482052+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:25:16.638959+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:27:14.708 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8028 8 365281 365273 0.355441 0 - 0.0299513 2026-02-23T15:27:15.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:27:14 trial068 ceph-mon[38805]: pgmap v4718: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:27:15.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:27:14 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1394869424' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:27:15.708 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8029 8 365281 365273 0.355396 0 - 0.0299513 2026-02-23T15:27:16.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:27:16 trial068 ceph-mon[38805]: pgmap v4719: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:27:16.708 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8030 8 365281 365273 0.355352 0 - 0.0299513 2026-02-23T15:27:17.564 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:27:17.683 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:27:17.708 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8031 8 365281 365273 0.355308 0 - 0.0299513 2026-02-23T15:27:17.783 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:27:17.781+0000 7f6883e78640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:27:17.783 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:27:17.781+0000 7f6883e78640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:27:17.892 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:27:17.892 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":315,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:26:45.087567+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:25:20.908986+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:24:15.482052+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:25:16.638959+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:27:18.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:27:17 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/4103474904' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:27:18.708 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8032 8 365281 365273 0.355264 0 - 0.0299513 2026-02-23T15:27:19.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:27:18 trial068 ceph-mon[38805]: pgmap v4720: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:27:19.708 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8033 8 365281 365273 0.35522 0 - 0.0299513 2026-02-23T15:27:20.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:27:19 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2026-02-23T15:27:20.708 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8034 8 365281 365273 0.355175 0 - 0.0299513 2026-02-23T15:27:20.929 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:27:21.045 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:27:21.145 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:27:21.144+0000 7fea6f2f2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:27:21.145 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:27:21.144+0000 7fea6f2f2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:27:21.254 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:27:21.254 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":315,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:26:45.087567+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:25:20.908986+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:24:15.482052+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:25:16.638959+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:27:21.255 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:27:21 trial068 ceph-mon[38805]: pgmap v4721: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:27:21.255 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:27:21 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T15:27:21.255 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:27:21 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T15:27:21.255 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:27:21 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T15:27:21.255 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:27:21 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T15:27:21.255 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:27:21 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-23T15:27:21.255 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:27:21 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2026-02-23T15:27:21.255 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:27:21 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T15:27:21.256 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:27:21 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2026-02-23T15:27:21.708 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8035 8 365281 365273 0.355131 0 - 0.0299513 2026-02-23T15:27:22.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:27:22 trial068 ceph-mon[38805]: pgmap v4722: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:27:22.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:27:22 trial068 ceph-mon[38805]: pgmap v4723: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:27:22.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:27:22 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/67337962' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:27:22.708 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8036 8 365281 365273 0.355087 0 - 0.0299513 2026-02-23T15:27:23.708 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8037 8 365281 365273 0.355043 0 - 0.0299513 2026-02-23T15:27:24.291 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:27:24.316 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:27:24 trial068 ceph-mon[38805]: pgmap v4724: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:27:24.317 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:27:23 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:27:23] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:27:24.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:27:23 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:27:23.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:27:24.410 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:27:24.510 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:27:24.508+0000 7f6bb39cc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:27:24.510 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:27:24.509+0000 7f6bb39cc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:27:24.621 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:27:24.621 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":315,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:26:45.087567+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:25:20.908986+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:24:15.482052+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:25:16.638959+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:27:24.708 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8038 8 365281 365273 0.354999 0 - 0.0299513 2026-02-23T15:27:25.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:27:25 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/289144278' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:27:25.708 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8039 8 365281 365273 0.354954 0 - 0.0299513 2026-02-23T15:27:26.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:27:26 trial068 ceph-mon[38805]: pgmap v4725: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:27:26.709 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T15:27:26.708877+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T15:27:26.709 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T15:27:26.709 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8040 8 365281 365273 0.35491 0 - 0.0299513 2026-02-23T15:27:27.659 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:27:27.709 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8041 8 365281 365273 0.354866 0 - 0.0299513 2026-02-23T15:27:27.778 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:27:27.880 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:27:27.878+0000 7f5199707640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:27:27.880 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:27:27.879+0000 7f5199707640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:27:27.989 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:27:27.989 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":315,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:26:45.087567+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:25:20.908986+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:24:15.482052+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:25:16.638959+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:27:28.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:27:28 trial068 ceph-mon[38805]: pgmap v4726: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:27:28.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:27:28 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2116830171' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:27:28.709 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8042 8 365281 365273 0.354822 0 - 0.0299513 2026-02-23T15:27:29.709 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8043 8 365281 365273 0.354778 0 - 0.0299513 2026-02-23T15:27:30.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:27:30 trial068 ceph-mon[38805]: pgmap v4727: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:27:30.709 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8044 8 365281 365273 0.354734 0 - 0.0299513 2026-02-23T15:27:31.027 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:27:31.145 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:27:31.245 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:27:31.243+0000 7f527270f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:27:31.245 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:27:31.243+0000 7f527270f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:27:31.354 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:27:31.354 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":315,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:26:45.087567+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:25:20.908986+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:24:15.482052+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:25:16.638959+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:27:31.709 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8045 8 365281 365273 0.35469 0 - 0.0299513 2026-02-23T15:27:32.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:27:32 trial068 ceph-mon[38805]: pgmap v4728: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:27:32.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:27:32 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/956014944' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:27:32.709 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8046 8 365281 365273 0.354646 0 - 0.0299513 2026-02-23T15:27:33.709 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8047 8 365281 365273 0.354602 0 - 0.0299513 2026-02-23T15:27:34.265 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:27:33 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:27:33.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:27:34.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:27:34 trial068 ceph-mon[38805]: pgmap v4729: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:27:34.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:27:33 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:27:33] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:27:34.392 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:27:34.508 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:27:34.603 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:27:34.601+0000 7f9024031640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:27:34.603 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:27:34.601+0000 7f9024031640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:27:34.709 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8048 8 365281 365273 0.354557 0 - 0.0299513 2026-02-23T15:27:34.712 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:27:34.712 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":315,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:26:45.087567+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:25:20.908986+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:24:15.482052+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:25:16.638959+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:27:35.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:27:35 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2423193907' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:27:35.709 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8049 8 365281 365273 0.354513 0 - 0.0299513 2026-02-23T15:27:36.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:27:36 trial068 ceph-mon[38805]: pgmap v4730: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:27:36.709 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8050 8 365281 365273 0.354469 0 - 0.0299513 2026-02-23T15:27:37.710 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8051 8 365281 365273 0.354425 0 - 0.0299513 2026-02-23T15:27:37.749 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:27:37.868 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:27:37.965 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:27:37.963+0000 7f5c3e3d2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:27:37.965 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:27:37.964+0000 7f5c3e3d2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:27:38.078 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:27:38.078 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":315,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:26:45.087567+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:25:20.908986+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:24:15.482052+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:25:16.638959+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:27:38.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:27:38 trial068 ceph-mon[38805]: pgmap v4731: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:27:38.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:27:38 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3340377723' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:27:38.709 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8052 8 365281 365273 0.354381 0 - 0.0299513 2026-02-23T15:27:39.709 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8053 8 365281 365273 0.354337 0 - 0.0299513 2026-02-23T15:27:40.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:27:40 trial068 ceph-mon[38805]: pgmap v4732: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:27:40.710 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8054 8 365281 365273 0.354293 0 - 0.0299513 2026-02-23T15:27:41.116 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:27:41.232 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:27:41.331 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:27:41.329+0000 7f9a7a827640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:27:41.331 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:27:41.330+0000 7f9a7a827640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:27:41.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:27:41 trial068 ceph-mon[38805]: pgmap v4733: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:27:41.443 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:27:41.443 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":315,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:26:45.087567+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:25:20.908986+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:24:15.482052+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:25:16.638959+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:27:41.710 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8055 8 365281 365273 0.354249 0 - 0.0299513 2026-02-23T15:27:42.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:27:42 trial068 ceph-mon[38805]: 5.5 scrub starts 2026-02-23T15:27:42.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:27:42 trial068 ceph-mon[38805]: 5.5 scrub ok 2026-02-23T15:27:42.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:27:42 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/85034905' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:27:42.710 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8056 8 365281 365273 0.354205 0 - 0.0299513 2026-02-23T15:27:43.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:27:43 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T15:27:43.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:27:43 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T15:27:43.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:27:43 trial068 ceph-mon[38805]: pgmap v4734: 17 pgs: 1 active+clean+scrubbing, 1 active+recovery_unfound+undersized+degraded, 4 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:27:43.710 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8057 8 365281 365273 0.354161 0 - 0.0299513 2026-02-23T15:27:44.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:27:43 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:27:43] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:27:44.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:27:43 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:27:43.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:27:44.480 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:27:44.597 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:27:44.696 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:27:44.694+0000 7fea0e95a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:27:44.696 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:27:44.694+0000 7fea0e95a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:27:44.710 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8058 8 365281 365273 0.354117 0 - 0.0299513 2026-02-23T15:27:44.806 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:27:44.807 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":315,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:26:45.087567+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:25:20.908986+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:24:15.482052+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:25:16.638959+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:27:45.710 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8059 8 365281 365273 0.354074 0 - 0.0299513 2026-02-23T15:27:45.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:27:45 trial068 ceph-mon[38805]: pgmap v4735: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 1 active+clean+scrubbing, 4 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:27:45.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:27:45 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/901722666' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:27:46.710 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T15:27:46.710193+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T15:27:46.710 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T15:27:46.710 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8060 8 365281 365273 0.35403 0 - 0.0299513 2026-02-23T15:27:47.710 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8061 8 365281 365273 0.353986 0 - 0.0299513 2026-02-23T15:27:47.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:27:47 trial068 ceph-mon[38805]: pgmap v4736: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:27:47.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:27:47 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T15:27:47.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:27:47 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T15:27:47.844 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:27:47.961 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:27:48.066 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:27:48.064+0000 7f0d42a53640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:27:48.066 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:27:48.064+0000 7f0d42a53640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:27:48.174 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:27:48.174 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":315,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:26:45.087567+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:25:20.908986+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:24:15.482052+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:25:16.638959+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:27:48.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:27:48 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/676461161' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:27:48.710 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8062 8 365281 365273 0.353942 0 - 0.0299513 2026-02-23T15:27:49.710 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8063 8 365281 365273 0.353898 0 - 0.0299513 2026-02-23T15:27:49.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:27:49 trial068 ceph-mon[38805]: pgmap v4737: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:27:50.710 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8064 8 365281 365273 0.353854 0 - 0.0299513 2026-02-23T15:27:51.212 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:27:51.328 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:27:51.429 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:27:51.427+0000 7f8583fe0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:27:51.429 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:27:51.428+0000 7f8583fe0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:27:51.539 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:27:51.540 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":315,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:26:45.087567+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:25:20.908986+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:24:15.482052+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:25:16.638959+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:27:51.540 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:27:51 trial068 ceph-mon[38805]: pgmap v4738: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:27:51.710 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8065 8 365281 365273 0.35381 0 - 0.0299513 2026-02-23T15:27:52.711 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8066 8 365281 365273 0.353766 0 - 0.0299513 2026-02-23T15:27:52.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:27:52 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/907378860' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:27:53.711 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8067 8 365281 365273 0.353722 0 - 0.0299513 2026-02-23T15:27:53.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:27:53 trial068 ceph-mon[38805]: pgmap v4739: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:27:54.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:27:53 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:27:53] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:27:54.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:27:53 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:27:53.974Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:27:54.576 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:27:54.602 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:27:54 trial068 ceph-mon[38805]: 5.e scrub starts 2026-02-23T15:27:54.693 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:27:54.711 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8068 8 365281 365273 0.353679 0 - 0.0299513 2026-02-23T15:27:54.791 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:27:54.789+0000 7fab723c0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:27:54.791 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:27:54.790+0000 7fab723c0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:27:54.900 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:27:54.900 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":315,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:26:45.087567+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:25:20.908986+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:24:15.482052+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:25:16.638959+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:27:55.711 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8069 8 365281 365273 0.353635 0 - 0.0299513 2026-02-23T15:27:55.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:27:55 trial068 ceph-mon[38805]: pgmap v4740: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:27:55.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:27:55 trial068 ceph-mon[38805]: 5.e scrub ok 2026-02-23T15:27:55.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:27:55 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/757737397' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:27:56.711 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8070 8 365281 365273 0.353591 0 - 0.0299513 2026-02-23T15:27:57.711 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8071 8 365281 365273 0.353547 0 - 0.0299513 2026-02-23T15:27:57.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:27:57 trial068 ceph-mon[38805]: pgmap v4741: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:27:57.938 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:27:58.056 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:27:58.155 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:27:58.153+0000 7f11501ba640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:27:58.155 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:27:58.154+0000 7f11501ba640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:27:58.263 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:27:58.263 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":315,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:26:45.087567+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:25:20.908986+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:24:15.482052+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:25:16.638959+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:27:58.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:27:58 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1921330339' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:27:58.711 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8072 8 365281 365273 0.353503 0 - 0.0299513 2026-02-23T15:27:59.711 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8073 8 365281 365273 0.353459 0 - 0.0299513 2026-02-23T15:27:59.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:27:59 trial068 ceph-mon[38805]: pgmap v4742: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:28:00.711 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8074 8 365281 365273 0.353416 0 - 0.0299513 2026-02-23T15:28:01.299 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:28:01.416 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:28:01.439 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:28:01 trial068 ceph-mon[38805]: pgmap v4743: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:28:01.517 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:28:01.515+0000 7f420b029640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:28:01.517 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:28:01.515+0000 7f420b029640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:28:01.627 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:28:01.627 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":315,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:26:45.087567+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:25:20.908986+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:24:15.482052+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:25:16.638959+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:28:01.711 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8075 8 365281 365273 0.353372 0 - 0.0299513 2026-02-23T15:28:02.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:28:02 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1126333649' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:28:02.711 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8076 8 365281 365273 0.353328 0 - 0.0299513 2026-02-23T15:28:03.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:28:03 trial068 ceph-mon[38805]: pgmap v4744: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:28:03.711 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8077 8 365281 365273 0.353284 0 - 0.0299513 2026-02-23T15:28:04.265 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:28:03 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:28:03.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:28:04.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:28:03 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:28:03] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:28:04.664 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:28:04.711 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8078 8 365281 365273 0.353241 0 - 0.0299513 2026-02-23T15:28:04.782 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:28:04.880 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:28:04.878+0000 7f5b3e60a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:28:04.880 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:28:04.878+0000 7f5b3e60a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:28:04.989 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:28:04.989 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":315,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:26:45.087567+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:25:20.908986+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:24:15.482052+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:25:16.638959+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:28:05.712 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8079 8 365281 365273 0.353197 0 - 0.0299513 2026-02-23T15:28:05.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:28:05 trial068 ceph-mon[38805]: pgmap v4745: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:28:05.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:28:05 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/4041740497' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:28:06.712 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T15:28:06.711891+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T15:28:06.712 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T15:28:06.712 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8080 8 365281 365273 0.353153 0 - 0.0299513 2026-02-23T15:28:07.712 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8081 8 365281 365273 0.35311 0 - 0.0299513 2026-02-23T15:28:07.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:28:07 trial068 ceph-mon[38805]: pgmap v4746: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:28:08.027 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:28:08.146 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:28:08.247 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:28:08.245+0000 7f1f4f3fb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:28:08.247 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:28:08.245+0000 7f1f4f3fb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:28:08.357 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:28:08.357 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":315,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:26:45.087567+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:25:20.908986+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:24:15.482052+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:25:16.638959+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:28:08.712 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8082 8 365281 365273 0.353066 0 - 0.0299513 2026-02-23T15:28:09.712 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8083 8 365281 365273 0.353022 0 - 0.0299513 2026-02-23T15:28:09.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:28:09 trial068 ceph-mon[38805]: pgmap v4747: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:28:09.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:28:09 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3015423912' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:28:10.712 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8084 8 365281 365273 0.352979 0 - 0.0299513 2026-02-23T15:28:11.395 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:28:11.511 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:28:11.616 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:28:11.614+0000 7fc383c0b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:28:11.616 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:28:11 trial068 ceph-mon[38805]: pgmap v4748: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:28:11.616 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:28:11.614+0000 7fc383c0b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:28:11.712 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8085 8 365281 365273 0.352935 0 - 0.0299513 2026-02-23T15:28:11.726 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:28:11.727 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":315,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:26:45.087567+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:25:20.908986+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:24:15.482052+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:25:16.638959+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:28:12.712 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8086 8 365281 365273 0.352891 0 - 0.0299513 2026-02-23T15:28:12.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:28:12 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1649409848' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:28:13.712 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8087 8 365281 365273 0.352848 0 - 0.0299513 2026-02-23T15:28:13.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:28:13 trial068 ceph-mon[38805]: pgmap v4749: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:28:14.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:28:13 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:28:13] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:28:14.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:28:13 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:28:13.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:28:14.712 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8088 8 365281 365273 0.352804 0 - 0.0299513 2026-02-23T15:28:14.764 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:28:14.884 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:28:14.991 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:28:14.990+0000 7f33d044a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:28:14.992 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:28:14.990+0000 7f33d044a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:28:15.100 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:28:15.101 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":315,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:26:45.087567+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:25:20.908986+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:24:15.482052+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:25:16.638959+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:28:15.712 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8089 8 365281 365273 0.35276 0 - 0.0299513 2026-02-23T15:28:15.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:28:15 trial068 ceph-mon[38805]: pgmap v4750: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:28:15.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:28:15 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2462493146' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:28:16.712 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8090 8 365281 365273 0.352717 0 - 0.0299513 2026-02-23T15:28:17.712 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8091 8 365281 365273 0.352673 0 - 0.0299513 2026-02-23T15:28:17.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:28:17 trial068 ceph-mon[38805]: pgmap v4751: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:28:17.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:28:17 trial068 ceph-mon[38805]: 5.7 scrub starts 2026-02-23T15:28:17.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:28:17 trial068 ceph-mon[38805]: 5.7 scrub ok 2026-02-23T15:28:18.139 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:28:18.257 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:28:18.353 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:28:18.351+0000 7fe103917640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:28:18.353 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:28:18.352+0000 7fe103917640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:28:18.463 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:28:18.464 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":315,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:26:45.087567+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:25:20.908986+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:24:15.482052+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:25:16.638959+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:28:18.713 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8092 8 365281 365273 0.35263 0 - 0.0299513 2026-02-23T15:28:19.713 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8093 8 365281 365273 0.352586 0 - 0.0299513 2026-02-23T15:28:19.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:28:19 trial068 ceph-mon[38805]: pgmap v4752: 17 pgs: 1 active+clean+scrubbing, 1 active+recovery_unfound+undersized+degraded, 4 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:28:19.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:28:19 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3821511497' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:28:20.713 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8094 8 365281 365273 0.352542 0 - 0.0299513 2026-02-23T15:28:21.501 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:28:21.524 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:28:21 trial068 ceph-mon[38805]: pgmap v4753: 17 pgs: 1 active+clean+scrubbing, 1 active+recovery_unfound+undersized+degraded, 4 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:28:21.525 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:28:21 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2026-02-23T15:28:21.525 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:28:21 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T15:28:21.525 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:28:21 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T15:28:21.525 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:28:21 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T15:28:21.525 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:28:21 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T15:28:21.618 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:28:21.713 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8095 8 365281 365273 0.352499 0 - 0.0299513 2026-02-23T15:28:21.724 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:28:21.723+0000 7f483ca8a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:28:21.725 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:28:21.723+0000 7f483ca8a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:28:21.832 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:28:21.833 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":315,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:26:45.087567+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:25:20.908986+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:24:15.482052+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:25:16.638959+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:28:22.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:28:22 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-23T15:28:22.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:28:22 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2026-02-23T15:28:22.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:28:22 trial068 ceph-mon[38805]: pgmap v4754: 17 pgs: 1 active+clean+scrubbing, 1 active+recovery_unfound+undersized+degraded, 4 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:28:22.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:28:22 trial068 ceph-mon[38805]: pgmap v4755: 17 pgs: 1 active+clean+scrubbing, 1 active+recovery_unfound+undersized+degraded, 4 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:28:22.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:28:22 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T15:28:22.595 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:28:22 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2026-02-23T15:28:22.595 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:28:22 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1170940587' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:28:22.713 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8096 8 365281 365273 0.352455 0 - 0.0299513 2026-02-23T15:28:23.713 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8097 8 365281 365273 0.352412 0 - 0.0299513 2026-02-23T15:28:24.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:28:24 trial068 ceph-mon[38805]: pgmap v4756: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:28:24.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:28:23 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:28:23] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:28:24.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:28:23 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:28:23.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:28:24.713 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8098 8 365281 365273 0.352368 0 - 0.0299513 2026-02-23T15:28:24.868 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:28:24.984 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:28:25.086 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:28:25.084+0000 7f0ef583e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:28:25.086 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:28:25.085+0000 7f0ef583e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:28:25.195 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:28:25.196 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":315,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:26:45.087567+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:25:20.908986+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:24:15.482052+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:25:16.638959+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:28:25.713 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8099 8 365281 365273 0.352325 0 - 0.0299513 2026-02-23T15:28:26.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:28:26 trial068 ceph-mon[38805]: pgmap v4757: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:28:26.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:28:26 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2407142424' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:28:26.713 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T15:28:26.713322+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T15:28:26.713 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T15:28:26.713 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8100 8 365281 365273 0.352281 0 - 0.0299513 2026-02-23T15:28:27.713 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8101 8 365281 365273 0.352238 0 - 0.0299513 2026-02-23T15:28:28.250 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:28:28.369 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:28:28.414 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:28:28 trial068 ceph-mon[38805]: pgmap v4758: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:28:28.472 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:28:28.470+0000 7f084520c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:28:28.472 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:28:28.471+0000 7f084520c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:28:28.581 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:28:28.582 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":315,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:26:45.087567+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:25:20.908986+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:24:15.482052+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:25:16.638959+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:28:28.713 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8102 8 365281 365273 0.352194 0 - 0.0299513 2026-02-23T15:28:29.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:28:29 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T15:28:29.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:28:29 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T15:28:29.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:28:29 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3994767857' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:28:29.713 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8103 8 365281 365273 0.352151 0 - 0.0299513 2026-02-23T15:28:30.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:28:30 trial068 ceph-mon[38805]: pgmap v4759: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:28:30.713 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8104 8 365281 365273 0.352107 0 - 0.0299513 2026-02-23T15:28:31.619 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:28:31.713 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8105 8 365281 365273 0.352064 0 - 0.0299513 2026-02-23T15:28:31.738 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:28:31.840 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:28:31.838+0000 7f8f0d3a5640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:28:31.840 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:28:31.838+0000 7f8f0d3a5640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:28:31.951 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:28:31.952 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":315,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:26:45.087567+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:25:20.908986+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:24:15.482052+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:25:16.638959+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:28:32.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:28:32 trial068 ceph-mon[38805]: pgmap v4760: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:28:32.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:28:32 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1902770095' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:28:32.713 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8106 8 365281 365273 0.352021 0 - 0.0299513 2026-02-23T15:28:33.714 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8107 8 365281 365273 0.351977 0 - 0.0299513 2026-02-23T15:28:34.265 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:28:33 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:28:33.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:28:34.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:28:34 trial068 ceph-mon[38805]: pgmap v4761: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:28:34.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:28:33 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:28:33] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:28:34.714 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8108 8 365281 365273 0.351934 0 - 0.0299513 2026-02-23T15:28:34.990 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:28:35.112 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:28:35.216 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:28:35.215+0000 7fe87bba5640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:28:35.217 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:28:35.215+0000 7fe87bba5640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:28:35.329 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:28:35.329 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":315,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:26:45.087567+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:25:20.908986+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:24:15.482052+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:25:16.638959+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:28:35.714 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8109 8 365281 365273 0.35189 0 - 0.0299513 2026-02-23T15:28:36.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:28:36 trial068 ceph-mon[38805]: pgmap v4762: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:28:36.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:28:36 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1762260028' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:28:36.714 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8110 8 365281 365273 0.351847 0 - 0.0299513 2026-02-23T15:28:37.714 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8111 8 365281 365273 0.351804 0 - 0.0299513 2026-02-23T15:28:38.368 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:28:38.486 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:28:38.508 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:28:38 trial068 ceph-mon[38805]: pgmap v4763: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:28:38.508 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:28:38 trial068 ceph-mon[38805]: 5.a scrub starts 2026-02-23T15:28:38.508 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:28:38 trial068 ceph-mon[38805]: 5.a scrub ok 2026-02-23T15:28:38.592 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:28:38.590+0000 7ff53c26b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:28:38.592 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:28:38.590+0000 7ff53c26b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:28:38.701 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:28:38.701 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":315,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:26:45.087567+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:25:20.908986+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:24:15.482052+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:25:16.638959+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:28:38.714 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8112 8 365281 365273 0.35176 0 - 0.0299513 2026-02-23T15:28:39.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:28:39 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3846799731' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:28:39.714 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8113 8 365281 365273 0.351717 0 - 0.0299513 2026-02-23T15:28:40.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:28:40 trial068 ceph-mon[38805]: pgmap v4764: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:28:40.714 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8114 8 365281 365273 0.351673 0 - 0.0299513 2026-02-23T15:28:41.714 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8115 8 365281 365273 0.35163 0 - 0.0299513 2026-02-23T15:28:41.738 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:28:41.856 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:28:41.953 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:28:41.952+0000 7fa8745d7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:28:41.953 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:28:41.952+0000 7fa8745d7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:28:42.065 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:28:42.065 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":315,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:26:45.087567+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:25:20.908986+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:24:15.482052+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:25:16.638959+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:28:42.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:28:42 trial068 ceph-mon[38805]: pgmap v4765: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:28:42.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:28:42 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1296508391' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:28:42.714 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8116 8 365281 365273 0.351587 0 - 0.0299513 2026-02-23T15:28:43.714 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8117 8 365281 365273 0.351543 0 - 0.0299513 2026-02-23T15:28:44.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:28:44 trial068 ceph-mon[38805]: pgmap v4766: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:28:44.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:28:44 trial068 ceph-mon[38805]: 5.8 scrub starts 2026-02-23T15:28:44.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:28:44 trial068 ceph-mon[38805]: 5.8 scrub ok 2026-02-23T15:28:44.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:28:43 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:28:43] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:28:44.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:28:43 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:28:43.974Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:28:44.714 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8118 8 365281 365273 0.3515 0 - 0.0299513 2026-02-23T15:28:45.104 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:28:45.224 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:28:45.328 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:28:45.326+0000 7f103d813640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:28:45.328 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:28:45.326+0000 7f103d813640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:28:45.436 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:28:45.436 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":315,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:26:45.087567+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:25:20.908986+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:24:15.482052+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:25:16.638959+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:28:45.715 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8119 8 365281 365273 0.351457 0 - 0.0299513 2026-02-23T15:28:46.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:28:46 trial068 ceph-mon[38805]: pgmap v4767: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:28:46.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:28:46 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/4108676547' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:28:46.715 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T15:28:46.714791+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T15:28:46.715 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T15:28:46.715 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8120 8 365281 365273 0.351414 0 - 0.0299513 2026-02-23T15:28:47.715 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8121 8 365281 365273 0.35137 0 - 0.0299513 2026-02-23T15:28:48.473 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:28:48.497 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:28:48 trial068 ceph-mon[38805]: pgmap v4768: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:28:48.591 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:28:48.691 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:28:48.689+0000 7f088bd2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:28:48.691 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:28:48.690+0000 7f088bd2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:28:48.715 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8122 8 365281 365273 0.351327 0 - 0.0299513 2026-02-23T15:28:48.803 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:28:48.803 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":315,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:26:45.087567+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:25:20.908986+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:24:15.482052+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:25:16.638959+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:28:49.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:28:49 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3510311057' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:28:49.715 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8123 8 365281 365273 0.351284 0 - 0.0299513 2026-02-23T15:28:50.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:28:50 trial068 ceph-mon[38805]: pgmap v4769: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:28:50.715 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8124 8 365281 365273 0.351241 0 - 0.0299513 2026-02-23T15:28:51.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:28:51 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T15:28:51.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:28:51 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T15:28:51.715 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8125 8 365281 365273 0.351197 0 - 0.0299513 2026-02-23T15:28:51.841 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:28:51.959 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:28:52.062 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:28:52.060+0000 7efd0eac6640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:28:52.062 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:28:52.060+0000 7efd0eac6640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:28:52.172 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:28:52.173 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":315,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:26:45.087567+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:25:20.908986+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:24:15.482052+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:25:16.638959+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:28:52.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:28:52 trial068 ceph-mon[38805]: pgmap v4770: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:28:52.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:28:52 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3395862218' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:28:52.715 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8126 8 365281 365273 0.351154 0 - 0.0299513 2026-02-23T15:28:53.715 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8127 8 365281 365273 0.351111 0 - 0.0299513 2026-02-23T15:28:54.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:28:54 trial068 ceph-mon[38805]: pgmap v4771: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:28:54.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:28:53 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:28:53] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:28:54.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:28:53 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:28:53.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:28:54.715 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8128 8 365281 365273 0.351068 0 - 0.0299513 2026-02-23T15:28:55.213 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:28:55.337 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:28:55.440 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:28:55.439+0000 7fa6263be640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:28:55.440 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:28:55.439+0000 7fa6263be640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:28:55.548 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:28:55.549 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":315,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:26:45.087567+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:25:20.908986+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:24:15.482052+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:25:16.638959+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:28:55.715 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8129 8 365281 365273 0.351025 0 - 0.0299513 2026-02-23T15:28:56.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:28:56 trial068 ceph-mon[38805]: pgmap v4772: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:28:56.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:28:56 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1558845958' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:28:56.715 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8130 8 365281 365273 0.350981 0 - 0.0299513 2026-02-23T15:28:57.715 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8131 8 365281 365273 0.350938 0 - 0.0299513 2026-02-23T15:28:58.587 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:28:58.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:28:58 trial068 ceph-mon[38805]: pgmap v4773: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:28:58.705 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:28:58.715 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8132 8 365281 365273 0.350895 0 - 0.0299513 2026-02-23T15:28:58.808 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:28:58.806+0000 7f8800ff0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:28:58.808 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:28:58.806+0000 7f8800ff0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:28:58.916 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:28:58.916 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":315,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:26:45.087567+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:25:20.908986+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:24:15.482052+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:25:16.638959+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:28:59.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:28:59 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T15:28:59.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:28:59 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T15:28:59.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:28:59 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/806417947' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:28:59.715 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8133 8 365281 365273 0.350852 0 - 0.0299513 2026-02-23T15:29:00.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:29:00 trial068 ceph-mon[38805]: pgmap v4774: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:29:00.715 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8134 8 365281 365273 0.350809 0 - 0.0299513 2026-02-23T15:29:01.716 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8135 8 365281 365273 0.350766 0 - 0.0299513 2026-02-23T15:29:01.954 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:29:02.073 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:29:02.178 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:29:02.176+0000 7fa4f94d1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:29:02.178 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:29:02.177+0000 7fa4f94d1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:29:02.289 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:29:02.289 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":315,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:26:45.087567+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:25:20.908986+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:24:15.482052+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:25:16.638959+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:29:02.289 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:29:02 trial068 ceph-mon[38805]: pgmap v4775: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:29:02.716 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8136 8 365281 365273 0.350723 0 - 0.0299513 2026-02-23T15:29:03.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:29:03 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3024976103' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:29:03.716 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8137 8 365281 365273 0.350679 0 - 0.0299513 2026-02-23T15:29:04.265 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:29:03 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:29:03.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:29:04.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:29:04 trial068 ceph-mon[38805]: pgmap v4776: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:29:04.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:29:03 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:29:03] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:29:04.716 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8138 8 365281 365273 0.350636 0 - 0.0299513 2026-02-23T15:29:05.328 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:29:05.447 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:29:05.544 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:29:05.543+0000 7f755b526640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:29:05.545 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:29:05.543+0000 7f755b526640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:29:05.654 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:29:05.654 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":315,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:26:45.087567+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:25:20.908986+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:24:15.482052+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:25:16.638959+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:29:05.716 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8139 8 365281 365273 0.350593 0 - 0.0299513 2026-02-23T15:29:06.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:29:06 trial068 ceph-mon[38805]: pgmap v4777: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:29:06.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:29:06 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2224061645' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:29:06.716 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T15:29:06.715998+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T15:29:06.716 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T15:29:06.716 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8140 8 365281 365273 0.35055 0 - 0.0299513 2026-02-23T15:29:07.716 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8141 8 365281 365273 0.350507 0 - 0.0299513 2026-02-23T15:29:08.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:29:08 trial068 ceph-mon[38805]: pgmap v4778: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:29:08.691 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:29:08.716 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8142 8 365281 365273 0.350464 0 - 0.0299513 2026-02-23T15:29:08.810 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:29:08.909 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:29:08.908+0000 7fc9ca9fc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:29:08.910 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:29:08.908+0000 7fc9ca9fc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:29:09.018 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:29:09.019 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":315,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:26:45.087567+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:25:20.908986+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:24:15.482052+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:25:16.638959+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:29:09.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:29:09 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1587225333' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:29:09.716 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8143 8 365281 365273 0.350421 0 - 0.0299513 2026-02-23T15:29:10.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:29:10 trial068 ceph-mon[38805]: pgmap v4779: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:29:10.716 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8144 8 365281 365273 0.350378 0 - 0.0299513 2026-02-23T15:29:11.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:29:11 trial068 ceph-mon[38805]: 5.5 scrub starts 2026-02-23T15:29:11.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:29:11 trial068 ceph-mon[38805]: 5.5 scrub ok 2026-02-23T15:29:11.717 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8145 8 365281 365273 0.350335 0 - 0.0299513 2026-02-23T15:29:12.057 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:29:12.178 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:29:12.276 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:29:12.274+0000 7feef8c8e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:29:12.276 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:29:12.275+0000 7feef8c8e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:29:12.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:29:12 trial068 ceph-mon[38805]: pgmap v4780: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:29:12.384 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:29:12.384 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":315,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:26:45.087567+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:25:20.908986+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:24:15.482052+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:25:16.638959+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:29:12.716 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8146 8 365281 365273 0.350292 0 - 0.0299513 2026-02-23T15:29:13.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:29:13 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/888996433' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:29:13.716 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8147 8 365281 365273 0.350249 0 - 0.0299513 2026-02-23T15:29:14.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:29:14 trial068 ceph-mon[38805]: pgmap v4781: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:29:14.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:29:13 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:29:13] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:29:14.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:29:13 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:29:13.974Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:29:14.716 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8148 8 365281 365273 0.350206 0 - 0.0299513 2026-02-23T15:29:15.422 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:29:15.448 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:29:15 trial068 ceph-mon[38805]: 5.e scrub starts 2026-02-23T15:29:15.448 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:29:15 trial068 ceph-mon[38805]: 5.e scrub ok 2026-02-23T15:29:15.542 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:29:15.647 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:29:15.646+0000 7f224d730640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:29:15.648 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:29:15.646+0000 7f224d730640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:29:15.716 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8149 8 365281 365273 0.350163 0 - 0.0299513 2026-02-23T15:29:15.756 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:29:15.756 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":315,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:26:45.087567+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:25:20.908986+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:24:15.482052+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:25:16.638959+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:29:16.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:29:16 trial068 ceph-mon[38805]: pgmap v4782: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:29:16.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:29:16 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1299314394' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:29:16.717 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8150 8 365281 365273 0.35012 0 - 0.0299513 2026-02-23T15:29:17.717 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8151 8 365281 365273 0.350077 0 - 0.0299513 2026-02-23T15:29:18.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:29:18 trial068 ceph-mon[38805]: pgmap v4783: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:29:18.717 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8152 8 365281 365273 0.350034 0 - 0.0299513 2026-02-23T15:29:18.793 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:29:18.912 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:29:19.015 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:29:19.013+0000 7f7358622640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:29:19.015 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:29:19.013+0000 7f7358622640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:29:19.126 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:29:19.126 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":315,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:26:45.087567+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:25:20.908986+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:24:15.482052+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:25:16.638959+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:29:19.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:29:19 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/121103736' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:29:19.717 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8153 8 365281 365273 0.349991 0 - 0.0299513 2026-02-23T15:29:20.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:29:20 trial068 ceph-mon[38805]: pgmap v4784: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:29:20.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:29:20 trial068 ceph-mon[38805]: osdmap e316: 4 total, 3 up, 4 in 2026-02-23T15:29:20.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:29:20 trial068 ceph-mon[38805]: osdmap e317: 4 total, 3 up, 4 in 2026-02-23T15:29:20.717 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8154 8 365281 365273 0.349948 0 - 0.0299513 2026-02-23T15:29:21.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:29:21 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2026-02-23T15:29:21.717 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8155 8 365281 365273 0.349905 0 - 0.0299513 2026-02-23T15:29:22.164 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:29:22.279 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:29:22.377 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:29:22.375+0000 7f7eb12e0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:29:22.377 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:29:22.376+0000 7f7eb12e0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:29:22.486 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:29:22.486 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":317,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:29:20.106230+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:25:20.908986+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:28:28.333996+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:25:16.638959+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:29:22.486 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:29:22 trial068 ceph-mon[38805]: pgmap v4787: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:29:22.487 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:29:22 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-23T15:29:22.487 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:29:22 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2026-02-23T15:29:22.487 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:29:22 trial068 ceph-mon[38805]: pgmap v4788: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:29:22.487 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:29:22 trial068 ceph-mon[38805]: pgmap v4789: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:29:22.487 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:29:22 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T15:29:22.487 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:29:22 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2026-02-23T15:29:22.717 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8156 8 365281 365273 0.349863 0 - 0.0299513 2026-02-23T15:29:23.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:29:23 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/16571049' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:29:23.717 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8157 8 365281 365273 0.34982 0 - 0.0299513 2026-02-23T15:29:24.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:29:24 trial068 ceph-mon[38805]: pgmap v4790: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:29:24.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:29:23 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:29:23] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:29:24.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:29:23 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:29:23.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:29:24.717 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8158 8 365281 365273 0.349777 0 - 0.0299513 2026-02-23T15:29:25.523 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:29:25.642 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:29:25.717 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8159 8 365281 365273 0.349734 0 - 0.0299513 2026-02-23T15:29:25.745 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:29:25.743+0000 7efdfe9ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:29:25.745 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:29:25.743+0000 7efdfe9ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:29:25.856 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:29:25.856 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":317,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:29:20.106230+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:25:20.908986+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:28:28.333996+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:25:16.638959+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:29:26.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:29:26 trial068 ceph-mon[38805]: pgmap v4791: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:29:26.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:29:26 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2429414644' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:29:26.717 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T15:29:26.717428+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T15:29:26.718 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T15:29:26.718 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8160 8 365281 365273 0.349691 0 - 0.0299513 2026-02-23T15:29:27.717 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8161 8 365281 365273 0.349648 0 - 0.0299513 2026-02-23T15:29:28.717 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8162 8 365281 365273 0.349605 0 - 0.0299513 2026-02-23T15:29:28.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:29:28 trial068 ceph-mon[38805]: pgmap v4792: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:29:28.892 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:29:29.009 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:29:29.107 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:29:29.105+0000 7f9df3491640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:29:29.107 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:29:29.105+0000 7f9df3491640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:29:29.217 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:29:29.218 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":317,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:29:20.106230+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:25:20.908986+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:28:28.333996+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:25:16.638959+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:29:29.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:29:29 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2427169716' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:29:29.717 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8163 8 365281 365273 0.349563 0 - 0.0299513 2026-02-23T15:29:30.718 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8164 8 365281 365273 0.34952 0 - 0.0299513 2026-02-23T15:29:30.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:29:30 trial068 ceph-mon[38805]: pgmap v4793: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:29:31.718 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8165 8 365281 365273 0.349477 0 - 0.0299513 2026-02-23T15:29:32.254 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:29:32.372 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:29:32.472 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:29:32.470+0000 7f1458cd2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:29:32.472 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:29:32.470+0000 7f1458cd2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:29:32.581 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:29:32.581 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":317,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:29:20.106230+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:25:20.908986+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:28:28.333996+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:25:16.638959+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:29:32.581 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:29:32 trial068 ceph-mon[38805]: pgmap v4794: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:29:32.718 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8166 8 365281 365273 0.349434 0 - 0.0299513 2026-02-23T15:29:33.718 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8167 8 365281 365273 0.349391 0 - 0.0299513 2026-02-23T15:29:33.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:29:33 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1205676035' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:29:33.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:29:33 trial068 ceph-mon[38805]: 5.7 scrub starts 2026-02-23T15:29:34.264 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:29:33 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:29:33.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:29:34.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:29:33 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:29:33] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:29:34.718 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8168 8 365281 365273 0.349349 0 - 0.0299513 2026-02-23T15:29:34.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:29:34 trial068 ceph-mon[38805]: pgmap v4795: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:29:34.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:29:34 trial068 ceph-mon[38805]: 5.7 scrub ok 2026-02-23T15:29:35.617 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:29:35.718 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8169 8 365281 365273 0.349306 0 - 0.0299513 2026-02-23T15:29:35.736 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:29:35.837 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:29:35.835+0000 7fab4f577640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:29:35.837 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:29:35.836+0000 7fab4f577640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:29:35.948 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:29:35.948 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":317,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:29:20.106230+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:25:20.908986+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:28:28.333996+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:25:16.638959+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:29:36.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:29:36 trial068 ceph-mon[38805]: pgmap v4796: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:29:36.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:29:36 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/975511351' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:29:36.718 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8170 8 365281 365273 0.349263 0 - 0.0299513 2026-02-23T15:29:37.718 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8171 8 365281 365273 0.34922 0 - 0.0299513 2026-02-23T15:29:38.718 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8172 8 365281 365273 0.349178 0 - 0.0299513 2026-02-23T15:29:38.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:29:38 trial068 ceph-mon[38805]: pgmap v4797: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:29:38.988 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:29:39.106 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:29:39.203 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:29:39.202+0000 7fd96954a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:29:39.204 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:29:39.202+0000 7fd96954a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:29:39.313 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:29:39.313 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":317,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:29:20.106230+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:25:20.908986+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:28:28.333996+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:25:16.638959+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:29:39.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:29:39 trial068 ceph-mon[38805]: 5.a scrub starts 2026-02-23T15:29:39.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:29:39 trial068 ceph-mon[38805]: 5.a scrub ok 2026-02-23T15:29:39.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:29:39 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/157998832' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:29:39.718 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8173 8 365281 365273 0.349135 0 - 0.0299513 2026-02-23T15:29:40.718 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8174 8 365281 365273 0.349092 0 - 0.0299513 2026-02-23T15:29:40.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:29:40 trial068 ceph-mon[38805]: pgmap v4798: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:29:41.718 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8175 8 365281 365273 0.349049 0 - 0.0299513 2026-02-23T15:29:42.352 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:29:42.472 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:29:42.572 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:29:42.570+0000 7f9fa38b6640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:29:42.572 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:29:42.570+0000 7f9fa38b6640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:29:42.681 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:29:42.682 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":317,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:29:20.106230+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:25:20.908986+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:28:28.333996+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:25:16.638959+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:29:42.682 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:29:42 trial068 ceph-mon[38805]: pgmap v4799: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:29:42.718 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8176 8 365281 365273 0.349007 0 - 0.0299513 2026-02-23T15:29:43.718 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8177 8 365281 365273 0.348964 0 - 0.0299513 2026-02-23T15:29:43.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:29:43 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/721276147' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:29:44.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:29:43 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:29:43] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:29:44.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:29:43 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:29:43.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:29:44.718 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8178 8 365281 365273 0.348921 0 - 0.0299513 2026-02-23T15:29:44.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:29:44 trial068 ceph-mon[38805]: pgmap v4800: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:29:45.718 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:29:45.719 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8179 8 365281 365273 0.348879 0 - 0.0299513 2026-02-23T15:29:45.836 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:29:45.936 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:29:45.934+0000 7f3be8ed4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:29:45.936 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:29:45.934+0000 7f3be8ed4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:29:46.046 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:29:46.046 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":317,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:29:20.106230+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:25:20.908986+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:28:28.333996+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:25:16.638959+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:29:46.719 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T15:29:46.718697+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T15:29:46.719 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T15:29:46.719 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8180 8 365281 365273 0.348836 0 - 0.0299513 2026-02-23T15:29:46.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:29:46 trial068 ceph-mon[38805]: pgmap v4801: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:29:46.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:29:46 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3756312469' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:29:47.719 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8181 8 365281 365273 0.348793 0 - 0.0299513 2026-02-23T15:29:48.719 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8182 8 365281 365273 0.348751 0 - 0.0299513 2026-02-23T15:29:48.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:29:48 trial068 ceph-mon[38805]: pgmap v4802: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:29:49.085 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:29:49.202 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:29:49.303 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:29:49.301+0000 7fb3e795b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:29:49.303 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:29:49.301+0000 7fb3e795b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:29:49.412 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:29:49.412 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":317,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:29:20.106230+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:25:20.908986+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:28:28.333996+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:25:16.638959+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:29:49.719 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8183 8 365281 365273 0.348708 0 - 0.0299513 2026-02-23T15:29:49.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:29:49 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3684117783' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:29:50.719 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8184 8 365281 365273 0.348666 0 - 0.0299513 2026-02-23T15:29:50.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:29:50 trial068 ceph-mon[38805]: pgmap v4803: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:29:51.719 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8185 8 365281 365273 0.348623 0 - 0.0299513 2026-02-23T15:29:52.449 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:29:52.565 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:29:52.669 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:29:52.667+0000 7fb06a1fc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:29:52.669 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:29:52.667+0000 7fb06a1fc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:29:52.719 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8186 8 365281 365273 0.34858 0 - 0.0299513 2026-02-23T15:29:52.779 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:29:52.779 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":317,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:29:20.106230+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:25:20.908986+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:28:28.333996+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:25:16.638959+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:29:52.780 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:29:52 trial068 ceph-mon[38805]: pgmap v4804: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:29:52.780 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:29:52 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T15:29:52.780 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:29:52 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T15:29:53.719 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8187 8 365281 365273 0.348538 0 - 0.0299513 2026-02-23T15:29:53.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:29:53 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/4289346408' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:29:54.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:29:53 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:29:53] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:29:54.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:29:53 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:29:53.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:29:54.719 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8188 8 365281 365273 0.348495 0 - 0.0299513 2026-02-23T15:29:54.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:29:54 trial068 ceph-mon[38805]: pgmap v4805: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:29:55.719 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8189 8 365281 365273 0.348453 0 - 0.0299513 2026-02-23T15:29:55.817 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:29:55.936 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:29:56.040 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:29:56.039+0000 7f74cefad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:29:56.041 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:29:56.039+0000 7f74cefad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:29:56.148 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:29:56.149 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":317,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:29:20.106230+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:25:20.908986+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:28:28.333996+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:25:16.638959+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:29:56.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:29:56 trial068 ceph-mon[38805]: pgmap v4806: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:29:56.719 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8190 8 365281 365273 0.34841 0 - 0.0299513 2026-02-23T15:29:57.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:29:57 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3281750240' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:29:57.719 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8191 8 365281 365273 0.348368 0 - 0.0299513 2026-02-23T15:29:58.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:29:58 trial068 ceph-mon[38805]: pgmap v4807: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:29:58.719 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8192 8 365281 365273 0.348325 0 - 0.0299513 2026-02-23T15:29:59.185 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:29:59.303 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:29:59.404 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:29:59.403+0000 7fda4347c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:29:59.405 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:29:59.403+0000 7fda4347c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:29:59.515 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:29:59.515 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":317,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:29:20.106230+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:25:20.908986+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:28:28.333996+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:25:16.638959+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:29:59.719 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8193 8 365281 365273 0.348283 0 - 0.0299513 2026-02-23T15:30:00.720 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8194 8 365281 365273 0.34824 0 - 0.0299513 2026-02-23T15:30:00.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:30:00 trial068 ceph-mon[38805]: pgmap v4808: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:30:00.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:30:00 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2072535305' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:30:01.720 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8195 8 365281 365273 0.348198 0 - 0.0299513 2026-02-23T15:30:02.552 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:30:02.671 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:30:02.693 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:30:02 trial068 ceph-mon[38805]: pgmap v4809: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:30:02.720 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8196 8 365281 365273 0.348155 0 - 0.0299513 2026-02-23T15:30:02.775 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:30:02.773+0000 7fe6bca15640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:30:02.775 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:30:02.774+0000 7fe6bca15640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:30:02.883 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:30:02.883 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":317,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:29:20.106230+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:25:20.908986+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:28:28.333996+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:25:16.638959+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:30:03.720 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8197 8 365281 365273 0.348113 0 - 0.0299513 2026-02-23T15:30:03.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:30:03 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/747996861' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:30:04.265 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:30:03 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:30:03.972Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:30:04.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:30:03 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:30:03] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:30:04.720 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8198 8 365281 365273 0.34807 0 - 0.0299513 2026-02-23T15:30:04.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:30:04 trial068 ceph-mon[38805]: pgmap v4810: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:30:05.720 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8199 8 365281 365273 0.348028 0 - 0.0299513 2026-02-23T15:30:05.921 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:30:06.040 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:30:06.144 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:30:06.142+0000 7f79f7fff640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:30:06.144 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:30:06.142+0000 7f79f7fff640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:30:06.252 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:30:06.252 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":317,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:29:20.106230+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:25:20.908986+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:28:28.333996+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:25:16.638959+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:30:06.252 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:30:06 trial068 ceph-mon[38805]: pgmap v4811: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:30:06.720 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T15:30:06.720477+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T15:30:06.721 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T15:30:06.721 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8200 8 365281 365273 0.347985 0 - 0.0299513 2026-02-23T15:30:07.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:30:07 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2347029168' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:30:07.720 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8201 8 365281 365273 0.347943 0 - 0.0299513 2026-02-23T15:30:08.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:30:08 trial068 ceph-mon[38805]: pgmap v4812: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:30:08.720 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8202 8 365281 365273 0.3479 0 - 0.0299513 2026-02-23T15:30:09.289 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:30:09.408 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:30:09.512 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:30:09.510+0000 7f3189d91640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:30:09.512 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:30:09.510+0000 7f3189d91640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:30:09.620 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:30:09.620 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":317,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:29:20.106230+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:25:20.908986+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:28:28.333996+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:25:16.638959+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:30:09.721 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8203 8 365281 365273 0.347858 0 - 0.0299513 2026-02-23T15:30:10.721 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8204 8 365281 365273 0.347816 0 - 0.0299513 2026-02-23T15:30:10.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:30:10 trial068 ceph-mon[38805]: pgmap v4813: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:30:10.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:30:10 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1604454453' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:30:11.721 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8205 8 365281 365273 0.347773 0 - 0.0299513 2026-02-23T15:30:12.658 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:30:12.683 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:30:12 trial068 ceph-mon[38805]: pgmap v4814: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:30:12.721 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8206 8 365281 365273 0.347731 0 - 0.0299513 2026-02-23T15:30:12.776 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:30:12.876 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:30:12.874+0000 7f06cd125640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:30:12.876 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:30:12.875+0000 7f06cd125640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:30:12.984 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:30:12.984 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":317,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:29:20.106230+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:25:20.908986+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:28:28.333996+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:25:16.638959+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:30:13.721 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8207 8 365281 365273 0.347688 0 - 0.0299513 2026-02-23T15:30:13.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:30:13 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T15:30:13.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:30:13 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T15:30:13.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:30:13 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3480550715' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:30:14.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:30:13 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:30:13] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:30:14.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:30:13 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:30:13.974Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:30:14.721 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8208 8 365281 365273 0.347646 0 - 0.0299513 2026-02-23T15:30:14.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:30:14 trial068 ceph-mon[38805]: pgmap v4815: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:30:14.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:30:14 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T15:30:14.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:30:14 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T15:30:15.721 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8209 8 365281 365273 0.347604 0 - 0.0299513 2026-02-23T15:30:16.022 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:30:16.141 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:30:16.245 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:30:16.243+0000 7fc143df8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:30:16.246 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:30:16.243+0000 7fc143df8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:30:16.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:30:16 trial068 ceph-mon[38805]: 5.8 scrub starts 2026-02-23T15:30:16.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:30:16 trial068 ceph-mon[38805]: pgmap v4816: 17 pgs: 1 active+clean+scrubbing, 1 active+recovery_unfound+undersized+degraded, 4 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:30:16.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:30:16 trial068 ceph-mon[38805]: 5.8 scrub ok 2026-02-23T15:30:16.354 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:30:16.354 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":317,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:29:20.106230+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:25:20.908986+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:28:28.333996+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:25:16.638959+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:30:16.721 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8210 8 365281 365273 0.347561 0 - 0.0299513 2026-02-23T15:30:17.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:30:17 trial068 ceph-mon[38805]: 5.5 scrub starts 2026-02-23T15:30:17.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:30:17 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3670689641' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:30:17.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:30:17 trial068 ceph-mon[38805]: 5.5 scrub ok 2026-02-23T15:30:17.721 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8211 8 365281 365273 0.347519 0 - 0.0299513 2026-02-23T15:30:18.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:30:18 trial068 ceph-mon[38805]: pgmap v4817: 17 pgs: 1 active+clean+scrubbing, 1 active+recovery_unfound+undersized+degraded, 4 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:30:18.721 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8212 8 365281 365273 0.347477 0 - 0.0299513 2026-02-23T15:30:19.398 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:30:19.517 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:30:19.627 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:30:19.625+0000 7f5fa9873640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:30:19.627 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:30:19.625+0000 7f5fa9873640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:30:19.721 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8213 8 365281 365273 0.347434 0 - 0.0299513 2026-02-23T15:30:19.737 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:30:19.738 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":317,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:29:20.106230+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:25:20.908986+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:28:28.333996+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:25:16.638959+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:30:20.722 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8214 8 365281 365273 0.347392 0 - 0.0299513 2026-02-23T15:30:20.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:30:20 trial068 ceph-mon[38805]: pgmap v4818: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 1 active+clean+scrubbing, 4 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:30:20.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:30:20 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/46517816' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:30:21.721 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8215 8 365281 365273 0.34735 0 - 0.0299513 2026-02-23T15:30:21.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:30:21 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2026-02-23T15:30:22.722 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8216 8 365281 365273 0.347308 0 - 0.0299513 2026-02-23T15:30:22.780 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:30:22.806 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:30:22 trial068 ceph-mon[38805]: pgmap v4819: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:30:22.806 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:30:22 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-23T15:30:22.806 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:30:22 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2026-02-23T15:30:22.806 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:30:22 trial068 ceph-mon[38805]: pgmap v4820: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:30:22.806 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:30:22 trial068 ceph-mon[38805]: pgmap v4821: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:30:22.806 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:30:22 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T15:30:22.806 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:30:22 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2026-02-23T15:30:22.900 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:30:23.006 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:30:23.005+0000 7fe89ecf6640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:30:23.007 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:30:23.005+0000 7fe89ecf6640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:30:23.118 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:30:23.119 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":317,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:29:20.106230+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:25:20.908986+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:28:28.333996+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:25:16.638959+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:30:23.722 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8217 8 365281 365273 0.347265 0 - 0.0299513 2026-02-23T15:30:23.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:30:23 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1703942349' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:30:24.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:30:23 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:30:23] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:30:24.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:30:23 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:30:23.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:30:24.722 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8218 8 365281 365273 0.347223 0 - 0.0299513 2026-02-23T15:30:24.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:30:24 trial068 ceph-mon[38805]: pgmap v4822: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:30:25.722 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8219 8 365281 365273 0.347181 0 - 0.0299513 2026-02-23T15:30:26.155 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:30:26.271 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:30:26.373 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:30:26.371+0000 7ff2c5810640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:30:26.373 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:30:26.371+0000 7ff2c5810640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:30:26.373 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:30:26 trial068 ceph-mon[38805]: pgmap v4823: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:30:26.486 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:30:26.487 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":317,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:29:20.106230+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:25:20.908986+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:28:28.333996+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:25:16.638959+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:30:26.722 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T15:30:26.722306+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T15:30:26.722 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T15:30:26.723 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8220 8 365281 365273 0.347139 0 - 0.0299513 2026-02-23T15:30:27.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:30:27 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2985480721' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:30:27.722 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8221 8 365281 365273 0.347096 0 - 0.0299513 2026-02-23T15:30:28.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:30:28 trial068 ceph-mon[38805]: pgmap v4824: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:30:28.722 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8222 8 365281 365273 0.347054 0 - 0.0299513 2026-02-23T15:30:29.530 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:30:29.647 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:30:29.722 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8223 8 365281 365273 0.347012 0 - 0.0299513 2026-02-23T15:30:29.752 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:30:29.750+0000 7ff50d7f0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:30:29.752 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:30:29.751+0000 7ff50d7f0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:30:29.861 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:30:29.861 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":317,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:29:20.106230+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:25:20.908986+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:28:28.333996+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:25:16.638959+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:30:30.722 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8224 8 365281 365273 0.34697 0 - 0.0299513 2026-02-23T15:30:31.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:30:30 trial068 ceph-mon[38805]: pgmap v4825: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:30:31.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:30:30 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3559614243' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:30:31.722 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8225 8 365281 365273 0.346927 0 - 0.0299513 2026-02-23T15:30:32.722 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8226 8 365281 365273 0.346885 0 - 0.0299513 2026-02-23T15:30:32.897 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:30:33.015 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:30:33.037 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:30:32 trial068 ceph-mon[38805]: pgmap v4826: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:30:33.119 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:30:33.117+0000 7f9dfc13f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:30:33.119 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:30:33.118+0000 7f9dfc13f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:30:33.228 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:30:33.228 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":317,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:29:20.106230+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:25:20.908986+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:28:28.333996+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:25:16.638959+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:30:33.723 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8227 8 365281 365273 0.346843 0 - 0.0299513 2026-02-23T15:30:33.973 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:30:33 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3691916341' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:30:34.265 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:30:33 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:30:33.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:30:34.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:30:33 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:30:33] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:30:34.723 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8228 8 365281 365273 0.346801 0 - 0.0299513 2026-02-23T15:30:35.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:30:34 trial068 ceph-mon[38805]: pgmap v4827: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:30:35.723 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8229 8 365281 365273 0.346759 0 - 0.0299513 2026-02-23T15:30:36.265 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:30:36.384 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:30:36.407 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:30:36 trial068 ceph-mon[38805]: pgmap v4828: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:30:36.484 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:30:36.483+0000 7fdc7ea31640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:30:36.485 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:30:36.483+0000 7fdc7ea31640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:30:36.595 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:30:36.595 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":317,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:29:20.106230+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:25:20.908986+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:28:28.333996+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:25:16.638959+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:30:36.723 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8230 8 365281 365273 0.346717 0 - 0.0299513 2026-02-23T15:30:37.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:30:37 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2528853564' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:30:37.723 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8231 8 365281 365273 0.346675 0 - 0.0299513 2026-02-23T15:30:38.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:30:38 trial068 ceph-mon[38805]: pgmap v4829: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:30:38.723 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8232 8 365281 365273 0.346632 0 - 0.0299513 2026-02-23T15:30:39.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:30:39 trial068 ceph-mon[38805]: osdmap e318: 4 total, 3 up, 4 in 2026-02-23T15:30:39.633 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:30:39.723 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8233 8 365281 365273 0.34659 0 - 0.0299513 2026-02-23T15:30:39.749 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:30:39.851 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:30:39.849+0000 7faa5ad30640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:30:39.851 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:30:39.850+0000 7faa5ad30640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:30:39.960 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:30:39.960 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":318,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:30:38.119843+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:25:20.908986+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:28:28.333996+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:30:12.470914+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:30:40.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:30:40 trial068 ceph-mon[38805]: 5.e scrub starts 2026-02-23T15:30:40.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:30:40 trial068 ceph-mon[38805]: 5.e scrub ok 2026-02-23T15:30:40.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:30:40 trial068 ceph-mon[38805]: pgmap v4831: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:30:40.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:30:40 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/4148442864' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:30:40.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:30:40 trial068 ceph-mon[38805]: osdmap e319: 4 total, 3 up, 4 in 2026-02-23T15:30:40.723 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8234 8 365281 365273 0.346548 0 - 0.0299513 2026-02-23T15:30:41.723 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8235 8 365281 365273 0.346506 0 - 0.0299513 2026-02-23T15:30:42.724 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8236 8 365281 365273 0.346464 0 - 0.0299513 2026-02-23T15:30:42.998 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:30:43.022 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:30:42 trial068 ceph-mon[38805]: pgmap v4833: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:30:43.115 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:30:43.213 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:30:43.211+0000 7f965c3d7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:30:43.213 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:30:43.211+0000 7f965c3d7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:30:43.323 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:30:43.323 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":319,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:30:40.117422+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:25:20.908986+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:28:28.333996+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:30:12.470914+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:30:43.724 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8237 8 365281 365273 0.346422 0 - 0.0299513 2026-02-23T15:30:43.973 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:30:43 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3927037618' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:30:44.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:30:43 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:30:43] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:30:44.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:30:43 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:30:43.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:30:44.724 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8238 8 365281 365273 0.34638 0 - 0.0299513 2026-02-23T15:30:45.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:30:44 trial068 ceph-mon[38805]: pgmap v4834: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:30:45.724 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8239 8 365281 365273 0.346338 0 - 0.0299513 2026-02-23T15:30:46.361 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:30:46.479 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:30:46.581 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:30:46.579+0000 7f704a800640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:30:46.581 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:30:46.579+0000 7f704a800640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:30:46.691 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:30:46.691 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":319,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:30:40.117422+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:25:20.908986+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:28:28.333996+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:30:12.470914+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:30:46.724 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T15:30:46.724007+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T15:30:46.724 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T15:30:46.724 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8240 8 365281 365273 0.346296 0 - 0.0299513 2026-02-23T15:30:47.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:30:46 trial068 ceph-mon[38805]: pgmap v4835: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:30:47.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:30:46 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1310957575' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:30:47.724 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8241 8 365281 365273 0.346254 0 - 0.0299513 2026-02-23T15:30:48.724 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8242 8 365281 365273 0.346212 0 - 0.0299513 2026-02-23T15:30:49.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:30:48 trial068 ceph-mon[38805]: pgmap v4836: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:30:49.724 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8243 8 365281 365273 0.34617 0 - 0.0299513 2026-02-23T15:30:49.729 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:30:49.846 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:30:49.949 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:30:49.947+0000 7fdd23e26640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:30:49.949 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:30:49.948+0000 7fdd23e26640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:30:50.061 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:30:50.061 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":319,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:30:40.117422+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:25:20.908986+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:28:28.333996+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:30:12.470914+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:30:50.724 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8244 8 365281 365273 0.346128 0 - 0.0299513 2026-02-23T15:30:51.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:30:50 trial068 ceph-mon[38805]: pgmap v4837: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:30:51.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:30:50 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2980016391' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:30:51.724 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8245 8 365281 365273 0.346086 0 - 0.0299513 2026-02-23T15:30:52.724 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8246 8 365281 365273 0.346044 0 - 0.0299513 2026-02-23T15:30:53.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:30:52 trial068 ceph-mon[38805]: pgmap v4838: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:30:53.099 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:30:53.218 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:30:53.316 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:30:53.314+0000 7fe2a2850640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:30:53.316 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:30:53.314+0000 7fe2a2850640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:30:53.424 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:30:53.424 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":319,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:30:40.117422+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:25:20.908986+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:28:28.333996+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:30:12.470914+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:30:53.724 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8247 8 365281 365273 0.346002 0 - 0.0299513 2026-02-23T15:30:53.973 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:30:53 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2272355952' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:30:54.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:30:53 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:30:53] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:30:54.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:30:53 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:30:53.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:30:54.724 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8248 8 365281 365273 0.34596 0 - 0.0299513 2026-02-23T15:30:55.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:30:54 trial068 ceph-mon[38805]: pgmap v4839: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:30:55.724 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8249 8 365281 365273 0.345918 0 - 0.0299513 2026-02-23T15:30:56.461 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:30:56.485 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:30:56 trial068 ceph-mon[38805]: pgmap v4840: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:30:56.578 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:30:56.679 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:30:56.677+0000 7f170eef1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:30:56.679 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:30:56.677+0000 7f170eef1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:30:56.724 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8250 8 365281 365273 0.345876 0 - 0.0299513 2026-02-23T15:30:56.790 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:30:56.790 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":319,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:30:40.117422+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:25:20.908986+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:28:28.333996+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:30:12.470914+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:30:57.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:30:57 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/4116471126' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:30:57.725 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8251 8 365281 365273 0.345834 0 - 0.0299513 2026-02-23T15:30:58.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:30:58 trial068 ceph-mon[38805]: pgmap v4841: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:30:58.725 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8252 8 365281 365273 0.345792 0 - 0.0299513 2026-02-23T15:30:59.725 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8253 8 365281 365273 0.34575 0 - 0.0299513 2026-02-23T15:30:59.828 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:30:59.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:30:59 trial068 ceph-mon[38805]: 5.7 scrub starts 2026-02-23T15:30:59.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:30:59 trial068 ceph-mon[38805]: 5.7 scrub ok 2026-02-23T15:30:59.945 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:31:00.046 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:31:00.044+0000 7ffa5f605640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:31:00.046 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:31:00.044+0000 7ffa5f605640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:31:00.154 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:31:00.155 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":319,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:30:40.117422+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:25:20.908986+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:28:28.333996+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:30:12.470914+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:31:00.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:31:00 trial068 ceph-mon[38805]: pgmap v4842: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:31:00.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:31:00 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2478938465' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:31:00.725 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8254 8 365281 365273 0.345709 0 - 0.0299513 2026-02-23T15:31:01.725 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8255 8 365281 365273 0.345667 0 - 0.0299513 2026-02-23T15:31:02.725 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8256 8 365281 365273 0.345625 0 - 0.0299513 2026-02-23T15:31:03.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:31:02 trial068 ceph-mon[38805]: pgmap v4843: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:31:03.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:31:02 trial068 ceph-mon[38805]: 5.a scrub starts 2026-02-23T15:31:03.193 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:31:03.311 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:31:03.413 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:31:03.411+0000 7fca3e1c9640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:31:03.413 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:31:03.411+0000 7fca3e1c9640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:31:03.525 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:31:03.525 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":319,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:30:40.117422+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:25:20.908986+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:28:28.333996+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:30:12.470914+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:31:03.725 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8257 8 365281 365273 0.345583 0 - 0.0299513 2026-02-23T15:31:03.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:31:03 trial068 ceph-mon[38805]: 5.a scrub ok 2026-02-23T15:31:03.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:31:03 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3009656885' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:31:04.265 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:31:03 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:31:03.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:31:04.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:31:03 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:31:03] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:31:04.725 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8258 8 365281 365273 0.345541 0 - 0.0299513 2026-02-23T15:31:05.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:31:04 trial068 ceph-mon[38805]: pgmap v4844: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:31:05.725 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8259 8 365281 365273 0.345499 0 - 0.0299513 2026-02-23T15:31:06.565 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:31:06.684 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:31:06.725 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T15:31:06.725354+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T15:31:06.725 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T15:31:06.725 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8260 8 365281 365273 0.345457 0 - 0.0299513 2026-02-23T15:31:06.786 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:31:06.784+0000 7f9c4f034640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:31:06.786 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:31:06.785+0000 7f9c4f034640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:31:06.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:31:06 trial068 ceph-mon[38805]: pgmap v4845: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:31:06.896 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:31:06.897 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":319,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:30:40.117422+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:25:20.908986+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:28:28.333996+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:30:12.470914+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:31:07.725 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8261 8 365281 365273 0.345416 0 - 0.0299513 2026-02-23T15:31:08.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:31:07 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2209529336' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:31:08.725 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8262 8 365281 365273 0.345374 0 - 0.0299513 2026-02-23T15:31:09.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:31:08 trial068 ceph-mon[38805]: pgmap v4846: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:31:09.725 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8263 8 365281 365273 0.345332 0 - 0.0299513 2026-02-23T15:31:09.935 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:31:10.054 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:31:10.157 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:31:10.156+0000 7fe50fefb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:31:10.157 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:31:10.156+0000 7fe50fefb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:31:10.266 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:31:10.266 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":319,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:30:40.117422+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:25:20.908986+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:28:28.333996+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:30:12.470914+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:31:10.725 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8264 8 365281 365273 0.34529 0 - 0.0299513 2026-02-23T15:31:11.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:31:10 trial068 ceph-mon[38805]: pgmap v4847: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:31:11.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:31:10 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1895818127' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:31:11.726 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8265 8 365281 365273 0.345248 0 - 0.0299513 2026-02-23T15:31:12.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:31:11 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T15:31:12.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:31:11 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T15:31:12.726 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8266 8 365281 365273 0.345207 0 - 0.0299513 2026-02-23T15:31:13.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:31:12 trial068 ceph-mon[38805]: pgmap v4848: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:31:13.303 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:31:13.423 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:31:13.523 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:31:13.521+0000 7f3cc8dd8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:31:13.523 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:31:13.521+0000 7f3cc8dd8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:31:13.630 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:31:13.630 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":319,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:30:40.117422+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:25:20.908986+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:28:28.333996+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:30:12.470914+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:31:13.726 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8267 8 365281 365273 0.345165 0 - 0.0299513 2026-02-23T15:31:13.974 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:31:13 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3632625458' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:31:14.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:31:13 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:31:13] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:31:14.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:31:13 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:31:13.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:31:14.726 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8268 8 365281 365273 0.345123 0 - 0.0299513 2026-02-23T15:31:15.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:31:14 trial068 ceph-mon[38805]: pgmap v4849: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:31:15.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:31:14 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T15:31:15.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:31:14 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T15:31:15.726 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8269 8 365281 365273 0.345081 0 - 0.0299513 2026-02-23T15:31:16.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:31:16 trial068 ceph-mon[38805]: pgmap v4850: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:31:16.668 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:31:16.726 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8270 8 365281 365273 0.34504 0 - 0.0299513 2026-02-23T15:31:16.786 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:31:16.885 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:31:16.883+0000 7f88f323d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:31:16.885 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:31:16.883+0000 7f88f323d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:31:16.997 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:31:16.997 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":319,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:30:40.117422+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:25:20.908986+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:28:28.333996+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:30:12.470914+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:31:17.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:31:17 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/4245178961' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:31:17.726 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8271 8 365281 365273 0.344998 0 - 0.0299513 2026-02-23T15:31:18.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:31:18 trial068 ceph-mon[38805]: pgmap v4851: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:31:18.726 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8272 8 365281 365273 0.344956 0 - 0.0299513 2026-02-23T15:31:19.726 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8273 8 365281 365273 0.344915 0 - 0.0299513 2026-02-23T15:31:20.040 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:31:20.159 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:31:20.264 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:31:20.263+0000 7ff114be5640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:31:20.264 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:31:20.263+0000 7ff114be5640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:31:20.372 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:31:20.372 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":319,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:30:40.117422+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:25:20.908986+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:28:28.333996+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:30:12.470914+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:31:20.726 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8274 8 365281 365273 0.344873 0 - 0.0299513 2026-02-23T15:31:21.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:31:20 trial068 ceph-mon[38805]: pgmap v4852: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:31:21.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:31:20 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/4294579763' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:31:21.726 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8275 8 365281 365273 0.344831 0 - 0.0299513 2026-02-23T15:31:22.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:31:21 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2026-02-23T15:31:22.726 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8276 8 365281 365273 0.34479 0 - 0.0299513 2026-02-23T15:31:23.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:31:22 trial068 ceph-mon[38805]: pgmap v4853: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:31:23.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:31:22 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-23T15:31:23.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:31:22 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2026-02-23T15:31:23.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:31:22 trial068 ceph-mon[38805]: pgmap v4854: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:31:23.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:31:22 trial068 ceph-mon[38805]: pgmap v4855: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:31:23.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:31:22 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T15:31:23.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:31:22 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2026-02-23T15:31:23.409 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:31:23.525 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:31:23.625 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:31:23.624+0000 7f56339eb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:31:23.626 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:31:23.624+0000 7f56339eb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:31:23.726 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8277 8 365281 365273 0.344748 0 - 0.0299513 2026-02-23T15:31:23.734 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:31:23.734 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":319,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:30:40.117422+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:25:20.908986+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:28:28.333996+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:30:12.470914+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:31:24.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:31:23 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3918353245' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:31:24.094 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:31:23 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:31:23] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:31:24.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:31:23 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:31:23.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:31:24.726 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8278 8 365281 365273 0.344706 0 - 0.0299513 2026-02-23T15:31:25.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:31:24 trial068 ceph-mon[38805]: pgmap v4856: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:31:25.726 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8279 8 365281 365273 0.344665 0 - 0.0299513 2026-02-23T15:31:26.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:31:25 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T15:31:26.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:31:25 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T15:31:26.727 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T15:31:26.726700+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T15:31:26.727 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T15:31:26.727 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8280 8 365281 365273 0.344623 0 - 0.0299513 2026-02-23T15:31:26.773 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:31:26.889 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:31:26.988 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:31:26.986+0000 7fefddc2c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:31:26.988 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:31:26.986+0000 7fefddc2c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:31:27.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:31:26 trial068 ceph-mon[38805]: pgmap v4857: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:31:27.098 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:31:27.099 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":319,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:30:40.117422+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:25:20.908986+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:28:28.333996+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:30:12.470914+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:31:27.727 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8281 8 365281 365273 0.344581 0 - 0.0299513 2026-02-23T15:31:28.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:31:27 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2402245678' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:31:28.727 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8282 8 365281 365273 0.34454 0 - 0.0299513 2026-02-23T15:31:29.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:31:28 trial068 ceph-mon[38805]: pgmap v4858: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:31:29.727 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8283 8 365281 365273 0.344498 0 - 0.0299513 2026-02-23T15:31:30.136 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:31:30.254 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:31:30.357 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:31:30.355+0000 7fa006485640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:31:30.357 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:31:30.355+0000 7fa006485640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:31:30.466 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:31:30.466 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":319,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:30:40.117422+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:25:20.908986+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:28:28.333996+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:30:12.470914+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:31:30.727 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8284 8 365281 365273 0.344457 0 - 0.0299513 2026-02-23T15:31:31.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:31:30 trial068 ceph-mon[38805]: pgmap v4859: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:31:31.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:31:30 trial068 ceph-mon[38805]: 5.5 scrub starts 2026-02-23T15:31:31.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:31:30 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2720547616' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:31:31.727 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8285 8 365281 365273 0.344415 0 - 0.0299513 2026-02-23T15:31:32.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:31:31 trial068 ceph-mon[38805]: 5.5 scrub ok 2026-02-23T15:31:32.727 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8286 8 365281 365273 0.344373 0 - 0.0299513 2026-02-23T15:31:33.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:31:32 trial068 ceph-mon[38805]: pgmap v4860: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:31:33.504 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:31:33.622 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:31:33.726 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:31:33.725+0000 7f380f068640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:31:33.726 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:31:33.725+0000 7f380f068640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:31:33.727 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8287 8 365281 365273 0.344332 0 - 0.0299513 2026-02-23T15:31:33.834 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:31:33.834 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":319,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:30:40.117422+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:25:20.908986+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:28:28.333996+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:30:12.470914+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:31:34.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:31:33 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/935556049' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:31:34.094 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:31:33 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:31:33] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:31:34.265 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:31:33 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:31:33.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:31:34.727 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8288 8 365281 365273 0.34429 0 - 0.0299513 2026-02-23T15:31:35.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:31:34 trial068 ceph-mon[38805]: pgmap v4861: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:31:35.727 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8289 8 365281 365273 0.344249 0 - 0.0299513 2026-02-23T15:31:36.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:31:36 trial068 ceph-mon[38805]: pgmap v4862: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:31:36.727 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8290 8 365281 365273 0.344207 0 - 0.0299513 2026-02-23T15:31:36.870 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:31:36.987 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:31:37.089 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:31:37.087+0000 7fc37ea1f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:31:37.089 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:31:37.088+0000 7fc37ea1f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:31:37.198 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:31:37.198 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":319,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:30:40.117422+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:25:20.908986+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:28:28.333996+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:30:12.470914+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:31:37.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:31:37 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2454251903' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:31:37.727 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8291 8 365281 365273 0.344166 0 - 0.0299513 2026-02-23T15:31:38.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:31:38 trial068 ceph-mon[38805]: pgmap v4863: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:31:38.727 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8292 8 365281 365273 0.344124 0 - 0.0299513 2026-02-23T15:31:39.727 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8293 8 365281 365273 0.344083 0 - 0.0299513 2026-02-23T15:31:40.236 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:31:40.356 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:31:40.456 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:31:40.454+0000 7f04b9ddf640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:31:40.456 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:31:40.454+0000 7f04b9ddf640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:31:40.564 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:31:40.565 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":319,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:30:40.117422+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:25:20.908986+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:28:28.333996+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:30:12.470914+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:31:40.727 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8294 8 365281 365273 0.344041 0 - 0.0299513 2026-02-23T15:31:41.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:31:40 trial068 ceph-mon[38805]: pgmap v4864: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:31:41.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:31:40 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2985384212' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:31:41.727 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8295 8 365281 365273 0.344 0 - 0.0299513 2026-02-23T15:31:42.728 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8296 8 365281 365273 0.343958 0 - 0.0299513 2026-02-23T15:31:43.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:31:42 trial068 ceph-mon[38805]: 5.8 scrub starts 2026-02-23T15:31:43.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:31:42 trial068 ceph-mon[38805]: pgmap v4865: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:31:43.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:31:42 trial068 ceph-mon[38805]: 5.8 scrub ok 2026-02-23T15:31:43.601 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:31:43.720 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:31:43.727 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8297 8 365281 365273 0.343917 0 - 0.0299513 2026-02-23T15:31:43.822 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:31:43.820+0000 7f88f7350640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:31:43.822 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:31:43.821+0000 7f88f7350640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:31:43.931 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:31:43.931 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":320,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:31:42.957786+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:31:24.328052+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:28:28.333996+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:30:12.470914+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:31:44.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:31:43 trial068 ceph-mon[38805]: osdmap e320: 4 total, 3 up, 4 in 2026-02-23T15:31:44.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:31:43 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2834973565' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:31:44.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:31:43 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:31:43] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:31:44.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:31:43 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:31:43.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:31:44.728 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8298 8 365281 365273 0.343875 0 - 0.0299513 2026-02-23T15:31:45.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:31:44 trial068 ceph-mon[38805]: pgmap v4867: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:31:45.728 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8299 8 365281 365273 0.343834 0 - 0.0299513 2026-02-23T15:31:46.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:31:46 trial068 ceph-mon[38805]: osdmap e321: 4 total, 3 up, 4 in 2026-02-23T15:31:46.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:31:46 trial068 ceph-mon[38805]: pgmap v4869: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:31:46.728 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T15:31:46.728010+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T15:31:46.728 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T15:31:46.728 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8300 8 365281 365273 0.343793 0 - 0.0299513 2026-02-23T15:31:46.967 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:31:47.084 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:31:47.188 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:31:47.186+0000 7fd2150e4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:31:47.188 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:31:47.186+0000 7fd2150e4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:31:47.296 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:31:47.296 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":321,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:31:45.125097+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:31:24.328052+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:28:28.333996+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:30:12.470914+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:31:47.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:31:47 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1701122492' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:31:47.728 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8301 8 365281 365273 0.343751 0 - 0.0299513 2026-02-23T15:31:48.728 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8302 8 365281 365273 0.34371 0 - 0.0299513 2026-02-23T15:31:48.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:31:48 trial068 ceph-mon[38805]: pgmap v4870: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:31:49.728 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8303 8 365281 365273 0.343668 0 - 0.0299513 2026-02-23T15:31:50.333 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:31:50.452 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:31:50.554 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:31:50.553+0000 7f7783b69640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:31:50.554 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:31:50.553+0000 7f7783b69640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:31:50.664 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:31:50.664 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":321,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:31:45.125097+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:31:24.328052+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:28:28.333996+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:30:12.470914+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:31:50.728 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8304 8 365281 365273 0.343627 0 - 0.0299513 2026-02-23T15:31:51.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:31:50 trial068 ceph-mon[38805]: pgmap v4871: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:31:51.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:31:50 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3400051447' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:31:51.728 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8305 8 365281 365273 0.343586 0 - 0.0299513 2026-02-23T15:31:52.728 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8306 8 365281 365273 0.343544 0 - 0.0299513 2026-02-23T15:31:53.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:31:52 trial068 ceph-mon[38805]: pgmap v4872: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:31:53.701 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:31:53.728 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8307 8 365281 365273 0.343503 0 - 0.0299513 2026-02-23T15:31:53.817 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:31:53.919 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:31:53.917+0000 7f123ea90640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:31:53.919 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:31:53.917+0000 7f123ea90640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:31:54.026 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:31:54.027 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":321,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:31:45.125097+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:31:24.328052+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:28:28.333996+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:30:12.470914+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:31:54.027 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:31:53 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:31:53] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:31:54.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:31:53 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:31:53.972Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:31:54.728 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8308 8 365281 365273 0.343462 0 - 0.0299513 2026-02-23T15:31:55.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:31:54 trial068 ceph-mon[38805]: pgmap v4873: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:31:55.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:31:54 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2713260646' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:31:55.728 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8309 8 365281 365273 0.34342 0 - 0.0299513 2026-02-23T15:31:56.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:31:56 trial068 ceph-mon[38805]: pgmap v4874: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:31:56.728 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8310 8 365281 365273 0.343379 0 - 0.0299513 2026-02-23T15:31:57.064 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:31:57.182 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:31:57.283 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:31:57.281+0000 7fca041af640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:31:57.283 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:31:57.282+0000 7fca041af640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:31:57.394 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:31:57.394 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":321,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:31:45.125097+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:31:24.328052+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:28:28.333996+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:30:12.470914+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:31:57.729 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8311 8 365281 365273 0.343338 0 - 0.0299513 2026-02-23T15:31:57.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:31:57 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1075967277' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:31:58.729 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8312 8 365281 365273 0.343296 0 - 0.0299513 2026-02-23T15:31:58.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:31:58 trial068 ceph-mon[38805]: pgmap v4875: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:31:59.729 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8313 8 365281 365273 0.343255 0 - 0.0299513 2026-02-23T15:32:00.432 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:32:00.553 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:32:00.651 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:32:00.650+0000 7f0dffd04640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:32:00.652 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:32:00.650+0000 7f0dffd04640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:32:00.729 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8314 8 365281 365273 0.343214 0 - 0.0299513 2026-02-23T15:32:00.762 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:32:00.762 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":321,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:31:45.125097+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:31:24.328052+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:28:28.333996+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:30:12.470914+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:32:01.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:32:00 trial068 ceph-mon[38805]: pgmap v4876: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:32:01.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:32:00 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3103912527' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:32:01.729 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8315 8 365281 365273 0.343172 0 - 0.0299513 2026-02-23T15:32:02.729 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8316 8 365281 365273 0.343131 0 - 0.0299513 2026-02-23T15:32:03.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:32:02 trial068 ceph-mon[38805]: pgmap v4877: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:32:03.729 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8317 8 365281 365273 0.34309 0 - 0.0299513 2026-02-23T15:32:03.800 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:32:03.916 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:32:04.015 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:32:04.013+0000 7f13d4e57640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:32:04.015 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:32:04.013+0000 7f13d4e57640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:32:04.093 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:32:03 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:32:03] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:32:04.125 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:32:04.125 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":321,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:31:45.125097+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:31:24.328052+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:28:28.333996+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:30:12.470914+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:32:04.265 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:32:03 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:32:03.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:32:04.729 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8318 8 365281 365273 0.343049 0 - 0.0299513 2026-02-23T15:32:05.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:32:04 trial068 ceph-mon[38805]: pgmap v4878: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:32:05.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:32:04 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3383180791' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:32:05.729 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8319 8 365281 365273 0.343007 0 - 0.0299513 2026-02-23T15:32:06.729 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T15:32:06.729413+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T15:32:06.729 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T15:32:06.730 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8320 8 365281 365273 0.342966 0 - 0.0299513 2026-02-23T15:32:07.178 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:32:07.296 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:32:07.319 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:32:06 trial068 ceph-mon[38805]: pgmap v4879: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:32:07.395 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:32:07.393+0000 7f9547964640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:32:07.395 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:32:07.394+0000 7f9547964640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:32:07.504 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:32:07.505 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":321,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:31:45.125097+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:31:24.328052+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:28:28.333996+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:30:12.470914+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:32:07.729 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8321 8 365281 365273 0.342925 0 - 0.0299513 2026-02-23T15:32:08.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:32:07 trial068 ceph-mon[38805]: 5.e scrub starts 2026-02-23T15:32:08.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:32:07 trial068 ceph-mon[38805]: 5.e scrub ok 2026-02-23T15:32:08.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:32:07 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/272366499' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:32:08.729 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8322 8 365281 365273 0.342884 0 - 0.0299513 2026-02-23T15:32:09.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:32:08 trial068 ceph-mon[38805]: pgmap v4880: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:32:09.729 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8323 8 365281 365273 0.342843 0 - 0.0299513 2026-02-23T15:32:10.542 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:32:10.659 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:32:10.730 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8324 8 365281 365273 0.342801 0 - 0.0299513 2026-02-23T15:32:10.760 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:32:10.758+0000 7fd42cc71640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:32:10.760 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:32:10.759+0000 7fd42cc71640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:32:10.870 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:32:10.871 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":321,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:31:45.125097+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:31:24.328052+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:28:28.333996+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:30:12.470914+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:32:11.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:32:10 trial068 ceph-mon[38805]: pgmap v4881: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:32:11.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:32:10 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/4239109231' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:32:11.730 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8325 8 365281 365273 0.34276 0 - 0.0299513 2026-02-23T15:32:12.730 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8326 8 365281 365273 0.342719 0 - 0.0299513 2026-02-23T15:32:13.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:32:12 trial068 ceph-mon[38805]: pgmap v4882: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:32:13.730 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8327 8 365281 365273 0.342678 0 - 0.0299513 2026-02-23T15:32:13.908 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:32:14.028 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:32:14.127 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:32:14.125+0000 7ff3372d7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:32:14.127 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:32:14.125+0000 7ff3372d7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:32:14.237 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:32:14.237 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":321,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:31:45.125097+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:31:24.328052+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:28:28.333996+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:30:12.470914+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:32:14.238 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:32:13 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:32:13] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:32:14.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:32:13 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:32:13.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:32:14.730 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8328 8 365281 365273 0.342637 0 - 0.0299513 2026-02-23T15:32:15.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:32:14 trial068 ceph-mon[38805]: pgmap v4883: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:32:15.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:32:14 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1392733279' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:32:15.730 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8329 8 365281 365273 0.342596 0 - 0.0299513 2026-02-23T15:32:16.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:32:16 trial068 ceph-mon[38805]: pgmap v4884: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:32:16.730 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8330 8 365281 365273 0.342554 0 - 0.0299513 2026-02-23T15:32:17.275 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:32:17.392 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:32:17.489 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:32:17.488+0000 7f10dfc38640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:32:17.489 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:32:17.488+0000 7f10dfc38640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:32:17.599 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:32:17.599 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":321,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:31:45.125097+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:31:24.328052+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:28:28.333996+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:30:12.470914+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:32:17.730 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8331 8 365281 365273 0.342513 0 - 0.0299513 2026-02-23T15:32:18.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:32:17 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/27890761' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:32:18.730 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8332 8 365281 365273 0.342472 0 - 0.0299513 2026-02-23T15:32:19.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:32:18 trial068 ceph-mon[38805]: 5.a scrub starts 2026-02-23T15:32:19.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:32:18 trial068 ceph-mon[38805]: pgmap v4885: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:32:19.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:32:18 trial068 ceph-mon[38805]: 5.a scrub ok 2026-02-23T15:32:19.730 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8333 8 365281 365273 0.342431 0 - 0.0299513 2026-02-23T15:32:20.637 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:32:20.730 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8334 8 365281 365273 0.34239 0 - 0.0299513 2026-02-23T15:32:20.754 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:32:20.855 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:32:20.853+0000 7f0365594640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:32:20.855 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:32:20.854+0000 7f0365594640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:32:20.963 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:32:20.963 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":321,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:31:45.125097+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:31:24.328052+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:28:28.333996+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:30:12.470914+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:32:21.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:32:20 trial068 ceph-mon[38805]: pgmap v4886: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:32:21.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:32:20 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1314260358' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:32:21.730 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8335 8 365281 365273 0.342349 0 - 0.0299513 2026-02-23T15:32:22.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:32:21 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2026-02-23T15:32:22.730 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8336 8 365281 365273 0.342308 0 - 0.0299513 2026-02-23T15:32:23.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:32:22 trial068 ceph-mon[38805]: pgmap v4887: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:32:23.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:32:22 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-23T15:32:23.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:32:22 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2026-02-23T15:32:23.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:32:22 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T15:32:23.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:32:22 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2026-02-23T15:32:23.731 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8337 8 365281 365273 0.342267 0 - 0.0299513 2026-02-23T15:32:24.001 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:32:24.118 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:32:24.216 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:32:24.214+0000 7f615eebe640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:32:24.216 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:32:24.214+0000 7f615eebe640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:32:24.232 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:32:23 trial068 ceph-mon[38805]: pgmap v4888: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:32:24.232 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:32:23 trial068 ceph-mon[38805]: pgmap v4889: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:32:24.233 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:32:23 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:32:23] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:32:24.327 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:32:24.327 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":321,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:31:45.125097+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:31:24.328052+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:28:28.333996+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:30:12.470914+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:32:24.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:32:23 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:32:23.974Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:32:24.731 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8338 8 365281 365273 0.342226 0 - 0.0299513 2026-02-23T15:32:25.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:32:24 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/547855383' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:32:25.731 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8339 8 365281 365273 0.342185 0 - 0.0299513 2026-02-23T15:32:26.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:32:25 trial068 ceph-mon[38805]: pgmap v4890: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:32:26.731 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T15:32:26.730962+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T15:32:26.731 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T15:32:26.731 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8340 8 365281 365273 0.342144 0 - 0.0299513 2026-02-23T15:32:27.364 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:32:27.483 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:32:27.586 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:32:27.584+0000 7fccd2472640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:32:27.586 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:32:27.584+0000 7fccd2472640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:32:27.695 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:32:27.695 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":321,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:31:45.125097+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:31:24.328052+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:28:28.333996+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:30:12.470914+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:32:27.731 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8341 8 365281 365273 0.342103 0 - 0.0299513 2026-02-23T15:32:28.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:32:27 trial068 ceph-mon[38805]: pgmap v4891: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:32:28.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:32:27 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2650574969' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:32:28.731 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8342 8 365281 365273 0.342062 0 - 0.0299513 2026-02-23T15:32:29.731 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8343 8 365281 365273 0.342021 0 - 0.0299513 2026-02-23T15:32:30.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:32:29 trial068 ceph-mon[38805]: pgmap v4892: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:32:30.731 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8344 8 365281 365273 0.34198 0 - 0.0299513 2026-02-23T15:32:30.733 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:32:30.852 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:32:30.955 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:32:30.953+0000 7f68cb4da640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:32:30.955 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:32:30.954+0000 7f68cb4da640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:32:31.065 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:32:31.065 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":321,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:31:45.125097+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:31:24.328052+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:28:28.333996+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:30:12.470914+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:32:31.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:32:31 trial068 ceph-mon[38805]: pgmap v4893: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:32:31.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:32:31 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2253206944' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:32:31.731 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8345 8 365281 365273 0.341939 0 - 0.0299513 2026-02-23T15:32:32.731 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8346 8 365281 365273 0.341898 0 - 0.0299513 2026-02-23T15:32:33.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:32:33 trial068 ceph-mon[38805]: pgmap v4894: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:32:33.731 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8347 8 365281 365273 0.341857 0 - 0.0299513 2026-02-23T15:32:34.102 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:32:34.224 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:32:34.247 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:32:33 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:32:33] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:32:34.265 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:32:33 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:32:33.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:32:34.326 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:32:34.324+0000 7f7c5356e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:32:34.326 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:32:34.324+0000 7f7c5356e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:32:34.434 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:32:34.435 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":321,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:31:45.125097+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:31:24.328052+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:28:28.333996+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:30:12.470914+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:32:34.731 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8348 8 365281 365273 0.341816 0 - 0.0299513 2026-02-23T15:32:35.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:32:35 trial068 ceph-mon[38805]: pgmap v4895: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:32:35.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:32:35 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2191219683' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:32:35.731 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8349 8 365281 365273 0.341775 0 - 0.0299513 2026-02-23T15:32:36.731 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8350 8 365281 365273 0.341734 0 - 0.0299513 2026-02-23T15:32:37.472 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:32:37.498 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:32:37 trial068 ceph-mon[38805]: pgmap v4896: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:32:37.498 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:32:37 trial068 ceph-mon[38805]: 5.5 scrub starts 2026-02-23T15:32:37.498 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:32:37 trial068 ceph-mon[38805]: 5.5 scrub ok 2026-02-23T15:32:37.591 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:32:37.691 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:32:37.689+0000 7fb13c4a5640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:32:37.691 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:32:37.689+0000 7fb13c4a5640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:32:37.731 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8351 8 365281 365273 0.341693 0 - 0.0299513 2026-02-23T15:32:37.801 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:32:37.802 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":321,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:31:45.125097+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:31:24.328052+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:28:28.333996+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:30:12.470914+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:32:38.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:32:38 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T15:32:38.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:32:38 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T15:32:38.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:32:38 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/842642129' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:32:38.732 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8352 8 365281 365273 0.341652 0 - 0.0299513 2026-02-23T15:32:39.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:32:39 trial068 ceph-mon[38805]: pgmap v4897: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:32:39.732 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8353 8 365281 365273 0.341611 0 - 0.0299513 2026-02-23T15:32:40.732 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8354 8 365281 365273 0.34157 0 - 0.0299513 2026-02-23T15:32:40.841 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:32:40.958 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:32:41.055 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:32:41.054+0000 7efd803db640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:32:41.055 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:32:41.054+0000 7efd803db640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:32:41.163 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:32:41.164 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":321,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:31:45.125097+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:31:24.328052+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:28:28.333996+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:30:12.470914+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:32:41.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:32:41 trial068 ceph-mon[38805]: 5.7 scrub starts 2026-02-23T15:32:41.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:32:41 trial068 ceph-mon[38805]: 5.7 scrub ok 2026-02-23T15:32:41.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:32:41 trial068 ceph-mon[38805]: pgmap v4898: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:32:41.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:32:41 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1903840728' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:32:41.732 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8355 8 365281 365273 0.341529 0 - 0.0299513 2026-02-23T15:32:42.732 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8356 8 365281 365273 0.341489 0 - 0.0299513 2026-02-23T15:32:43.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:32:43 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T15:32:43.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:32:43 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T15:32:43.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:32:43 trial068 ceph-mon[38805]: pgmap v4899: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:32:43.732 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8357 8 365281 365273 0.341448 0 - 0.0299513 2026-02-23T15:32:44.202 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:32:44.227 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:32:43 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:32:43] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:32:44.319 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:32:44.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:32:43 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:32:43.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:32:44.418 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:32:44.417+0000 7fa7d2c3d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:32:44.419 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:32:44.417+0000 7fa7d2c3d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:32:44.531 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:32:44.532 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":321,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:31:45.125097+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:31:24.328052+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:28:28.333996+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:30:12.470914+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:32:44.732 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8358 8 365281 365273 0.341407 0 - 0.0299513 2026-02-23T15:32:45.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:32:45 trial068 ceph-mon[38805]: pgmap v4900: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:32:45.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:32:45 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/4226158051' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:32:45.732 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8359 8 365281 365273 0.341366 0 - 0.0299513 2026-02-23T15:32:46.732 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T15:32:46.732488+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T15:32:46.732 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T15:32:46.733 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8360 8 365281 365273 0.341325 0 - 0.0299513 2026-02-23T15:32:47.570 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:32:47.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:32:47 trial068 ceph-mon[38805]: pgmap v4901: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:32:47.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:32:47 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T15:32:47.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:32:47 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T15:32:47.689 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:32:47.732 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8361 8 365281 365273 0.341284 0 - 0.0299513 2026-02-23T15:32:47.786 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:32:47.784+0000 7f39065ba640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:32:47.786 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:32:47.785+0000 7f39065ba640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:32:47.896 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:32:47.896 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":321,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:31:45.125097+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:31:24.328052+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:28:28.333996+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:30:12.470914+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:32:48.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:32:48 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/4087780678' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:32:48.732 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8362 8 365281 365273 0.341244 0 - 0.0299513 2026-02-23T15:32:49.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:32:49 trial068 ceph-mon[38805]: pgmap v4902: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:32:49.732 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8363 8 365281 365273 0.341203 0 - 0.0299513 2026-02-23T15:32:50.733 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8364 8 365281 365273 0.341162 0 - 0.0299513 2026-02-23T15:32:50.933 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:32:51.050 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:32:51.152 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:32:51.151+0000 7fead5629640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:32:51.153 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:32:51.151+0000 7fead5629640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:32:51.263 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:32:51.263 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":321,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:31:45.125097+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:31:24.328052+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:28:28.333996+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:30:12.470914+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:32:51.263 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:32:51 trial068 ceph-mon[38805]: pgmap v4903: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:32:51.264 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:32:51 trial068 ceph-mon[38805]: 5.8 scrub starts 2026-02-23T15:32:51.264 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:32:51 trial068 ceph-mon[38805]: 5.8 scrub ok 2026-02-23T15:32:51.733 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8365 8 365281 365273 0.341121 0 - 0.0299513 2026-02-23T15:32:52.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:32:52 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/4007717643' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:32:52.733 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8366 8 365281 365273 0.34108 0 - 0.0299513 2026-02-23T15:32:53.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:32:53 trial068 ceph-mon[38805]: pgmap v4904: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:32:53.733 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8367 8 365281 365273 0.34104 0 - 0.0299513 2026-02-23T15:32:54.299 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:32:54.322 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:32:53 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:32:53] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:32:54.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:32:53 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:32:53.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:32:54.417 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:32:54.517 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:32:54.516+0000 7f04d094a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:32:54.518 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:32:54.516+0000 7f04d094a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:32:54.626 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:32:54.626 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":321,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:31:45.125097+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:31:24.328052+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:28:28.333996+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:30:12.470914+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:32:54.733 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8368 8 365281 365273 0.340999 0 - 0.0299513 2026-02-23T15:32:55.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:32:55 trial068 ceph-mon[38805]: pgmap v4905: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:32:55.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:32:55 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2967171618' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:32:55.733 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8369 8 365281 365273 0.340958 0 - 0.0299513 2026-02-23T15:32:56.733 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8370 8 365281 365273 0.340917 0 - 0.0299513 2026-02-23T15:32:57.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:32:57 trial068 ceph-mon[38805]: pgmap v4906: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:32:57.664 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:32:57.733 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8371 8 365281 365273 0.340877 0 - 0.0299513 2026-02-23T15:32:57.783 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:32:57.879 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:32:57.878+0000 7f245a8fe640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:32:57.879 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:32:57.878+0000 7f245a8fe640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:32:57.992 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:32:57.993 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":321,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:31:45.125097+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:31:24.328052+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:28:28.333996+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:30:12.470914+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:32:58.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:32:58 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3834053126' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:32:58.733 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8372 8 365281 365273 0.340836 0 - 0.0299513 2026-02-23T15:32:59.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:32:59 trial068 ceph-mon[38805]: pgmap v4907: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:32:59.733 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8373 8 365281 365273 0.340795 0 - 0.0299513 2026-02-23T15:33:00.733 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8374 8 365281 365273 0.340755 0 - 0.0299513 2026-02-23T15:33:01.029 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:33:01.147 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:33:01.247 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:33:01.246+0000 7f8d2aaf9640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:33:01.247 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:33:01.246+0000 7f8d2aaf9640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:33:01.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:33:01 trial068 ceph-mon[38805]: pgmap v4908: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:33:01.358 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:33:01.358 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":321,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:31:45.125097+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:31:24.328052+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:28:28.333996+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:30:12.470914+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:33:01.733 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8375 8 365281 365273 0.340714 0 - 0.0299513 2026-02-23T15:33:02.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:33:02 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1542572129' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:33:02.733 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8376 8 365281 365273 0.340673 0 - 0.0299513 2026-02-23T15:33:03.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:33:03 trial068 ceph-mon[38805]: pgmap v4909: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:33:03.733 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8377 8 365281 365273 0.340633 0 - 0.0299513 2026-02-23T15:33:04.265 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:33:03 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:33:03.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:33:04.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:33:03 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:33:03] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:33:04.397 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:33:04.515 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:33:04.627 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:33:04.625+0000 7f517f577640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:33:04.627 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:33:04.625+0000 7f517f577640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:33:04.733 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8378 8 365281 365273 0.340592 0 - 0.0299513 2026-02-23T15:33:04.735 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:33:04.735 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":321,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:31:45.125097+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:31:24.328052+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:28:28.333996+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:30:12.470914+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:33:05.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:33:05 trial068 ceph-mon[38805]: pgmap v4910: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:33:05.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:33:05 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3670949119' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:33:05.734 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8379 8 365281 365273 0.340551 0 - 0.0299513 2026-02-23T15:33:06.734 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T15:33:06.733988+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T15:33:06.734 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T15:33:06.734 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8380 8 365281 365273 0.340511 0 - 0.0299513 2026-02-23T15:33:07.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:33:07 trial068 ceph-mon[38805]: pgmap v4911: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:33:07.734 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8381 8 365281 365273 0.34047 0 - 0.0299513 2026-02-23T15:33:07.772 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:33:07.890 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:33:07.991 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:33:07.989+0000 7f5e7d3fe640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:33:07.991 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:33:07.990+0000 7f5e7d3fe640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:33:08.102 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:33:08.102 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":321,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:31:45.125097+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:31:24.328052+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:28:28.333996+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:30:12.470914+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:33:08.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:33:08 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3224430819' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:33:08.734 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8382 8 365281 365273 0.340429 0 - 0.0299513 2026-02-23T15:33:09.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:33:09 trial068 ceph-mon[38805]: pgmap v4912: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:33:09.734 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8383 8 365281 365273 0.340389 0 - 0.0299513 2026-02-23T15:33:10.734 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8384 8 365281 365273 0.340348 0 - 0.0299513 2026-02-23T15:33:11.140 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:33:11.258 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:33:11.360 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:33:11.359+0000 7f671b67f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:33:11.360 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:33:11.359+0000 7f671b67f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:33:11.469 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:33:11.469 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":321,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:31:45.125097+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:31:24.328052+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:28:28.333996+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:30:12.470914+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:33:11.470 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:33:11 trial068 ceph-mon[38805]: pgmap v4913: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:33:11.734 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8385 8 365281 365273 0.340308 0 - 0.0299513 2026-02-23T15:33:12.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:33:12 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/996783752' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:33:12.734 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8386 8 365281 365273 0.340267 0 - 0.0299513 2026-02-23T15:33:13.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:33:13 trial068 ceph-mon[38805]: pgmap v4914: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:33:13.734 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8387 8 365281 365273 0.340226 0 - 0.0299513 2026-02-23T15:33:14.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:33:13 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:33:13] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:33:14.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:33:13 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:33:13.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:33:14.506 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:33:14.623 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:33:14.723 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:33:14.721+0000 7f915a2ae640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:33:14.723 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:33:14.722+0000 7f915a2ae640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:33:14.734 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8388 8 365281 365273 0.340186 0 - 0.0299513 2026-02-23T15:33:14.832 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:33:14.832 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":321,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:31:45.125097+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:31:24.328052+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:28:28.333996+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:30:12.470914+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:33:15.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:33:15 trial068 ceph-mon[38805]: pgmap v4915: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:33:15.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:33:15 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/419621692' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:33:15.734 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8389 8 365281 365273 0.340145 0 - 0.0299513 2026-02-23T15:33:16.734 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8390 8 365281 365273 0.340105 0 - 0.0299513 2026-02-23T15:33:17.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:33:17 trial068 ceph-mon[38805]: pgmap v4916: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:33:17.734 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8391 8 365281 365273 0.340064 0 - 0.0299513 2026-02-23T15:33:17.869 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:33:17.987 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:33:18.091 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:33:18.089+0000 7fa0f1620640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:33:18.091 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:33:18.090+0000 7fa0f1620640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:33:18.199 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:33:18.200 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":321,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:31:45.125097+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:31:24.328052+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:28:28.333996+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:30:12.470914+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:33:18.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:33:18 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3403889258' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:33:18.735 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8392 8 365281 365273 0.340024 0 - 0.0299513 2026-02-23T15:33:19.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:33:19 trial068 ceph-mon[38805]: pgmap v4917: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:33:19.735 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8393 8 365281 365273 0.339983 0 - 0.0299513 2026-02-23T15:33:20.735 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8394 8 365281 365273 0.339943 0 - 0.0299513 2026-02-23T15:33:21.237 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:33:21.356 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:33:21.397 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:33:21 trial068 ceph-mon[38805]: pgmap v4918: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:33:21.455 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:33:21.453+0000 7f36e2029640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:33:21.455 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:33:21.453+0000 7f36e2029640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:33:21.564 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:33:21.565 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":321,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:31:45.125097+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:31:24.328052+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:28:28.333996+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:30:12.470914+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:33:21.735 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8395 8 365281 365273 0.339902 0 - 0.0299513 2026-02-23T15:33:22.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:33:22 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1710439130' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:33:22.735 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8396 8 365281 365273 0.339862 0 - 0.0299513 2026-02-23T15:33:23.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:33:23 trial068 ceph-mon[38805]: pgmap v4919: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:33:23.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:33:23 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2026-02-23T15:33:23.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:33:23 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-23T15:33:23.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:33:23 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2026-02-23T15:33:23.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:33:23 trial068 ceph-mon[38805]: pgmap v4920: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:33:23.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:33:23 trial068 ceph-mon[38805]: pgmap v4921: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:33:23.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:33:23 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T15:33:23.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:33:23 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2026-02-23T15:33:23.735 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8397 8 365281 365273 0.339821 0 - 0.0299513 2026-02-23T15:33:24.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:33:23 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:33:23] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:33:24.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:33:23 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:33:23.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:33:24.603 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:33:24.720 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:33:24.735 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8398 8 365281 365273 0.339781 0 - 0.0299513 2026-02-23T15:33:24.821 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:33:24.819+0000 7ff4fccdb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:33:24.821 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:33:24.820+0000 7ff4fccdb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:33:24.931 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:33:24.931 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":321,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:31:45.125097+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:31:24.328052+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:28:28.333996+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:30:12.470914+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:33:25.735 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8399 8 365281 365273 0.33974 0 - 0.0299513 2026-02-23T15:33:25.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:33:25 trial068 ceph-mon[38805]: pgmap v4922: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:33:25.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:33:25 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2486792055' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:33:26.735 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T15:33:26.735360+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T15:33:26.735 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T15:33:26.735 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8400 8 365281 365273 0.3397 0 - 0.0299513 2026-02-23T15:33:27.735 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8401 8 365281 365273 0.339659 0 - 0.0299513 2026-02-23T15:33:27.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:33:27 trial068 ceph-mon[38805]: pgmap v4923: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:33:27.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:33:27 trial068 ceph-mon[38805]: 5.e scrub starts 2026-02-23T15:33:27.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:33:27 trial068 ceph-mon[38805]: 5.e scrub ok 2026-02-23T15:33:27.969 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:33:28.090 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:33:28.196 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:33:28.194+0000 7fa1e7c65640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:33:28.196 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:33:28.195+0000 7fa1e7c65640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:33:28.304 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:33:28.304 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":321,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:31:45.125097+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:31:24.328052+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:28:28.333996+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:30:12.470914+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:33:28.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:33:28 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2094475422' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:33:28.735 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8402 8 365281 365273 0.339619 0 - 0.0299513 2026-02-23T15:33:29.735 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8403 8 365281 365273 0.339579 0 - 0.0299513 2026-02-23T15:33:29.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:33:29 trial068 ceph-mon[38805]: pgmap v4924: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:33:30.735 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8404 8 365281 365273 0.339538 0 - 0.0299513 2026-02-23T15:33:31.342 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:33:31.459 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:33:31.556 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:33:31.554+0000 7f22856c2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:33:31.556 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:33:31.554+0000 7f22856c2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:33:31.664 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:33:31.664 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":321,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:31:45.125097+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:31:24.328052+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:28:28.333996+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:30:12.470914+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:33:31.664 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:33:31 trial068 ceph-mon[38805]: pgmap v4925: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:33:31.735 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8405 8 365281 365273 0.339498 0 - 0.0299513 2026-02-23T15:33:32.736 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8406 8 365281 365273 0.339457 0 - 0.0299513 2026-02-23T15:33:32.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:33:32 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2152222461' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:33:33.736 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8407 8 365281 365273 0.339417 0 - 0.0299513 2026-02-23T15:33:33.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:33:33 trial068 ceph-mon[38805]: pgmap v4926: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:33:34.265 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:33:33 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:33:33.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:33:34.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:33:33 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:33:33] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:33:34.702 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:33:34.736 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8408 8 365281 365273 0.339377 0 - 0.0299513 2026-02-23T15:33:34.819 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:33:34.924 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:33:34.922+0000 7f5b35dc0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:33:34.924 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:33:34.922+0000 7f5b35dc0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:33:35.034 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:33:35.034 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":321,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:31:45.125097+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:31:24.328052+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:28:28.333996+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:30:12.470914+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:33:35.736 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8409 8 365281 365273 0.339336 0 - 0.0299513 2026-02-23T15:33:35.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:33:35 trial068 ceph-mon[38805]: pgmap v4927: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:33:35.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:33:35 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3773269967' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:33:36.736 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8410 8 365281 365273 0.339296 0 - 0.0299513 2026-02-23T15:33:37.736 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8411 8 365281 365273 0.339256 0 - 0.0299513 2026-02-23T15:33:37.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:33:37 trial068 ceph-mon[38805]: pgmap v4928: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:33:38.072 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:33:38.188 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:33:38.288 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:33:38.286+0000 7f3ceef48640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:33:38.288 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:33:38.286+0000 7f3ceef48640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:33:38.398 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:33:38.398 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":321,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:31:45.125097+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:31:24.328052+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:28:28.333996+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:30:12.470914+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:33:38.736 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8412 8 365281 365273 0.339215 0 - 0.0299513 2026-02-23T15:33:38.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:33:38 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1654319878' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:33:39.736 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8413 8 365281 365273 0.339175 0 - 0.0299513 2026-02-23T15:33:39.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:33:39 trial068 ceph-mon[38805]: pgmap v4929: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:33:40.736 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8414 8 365281 365273 0.339135 0 - 0.0299513 2026-02-23T15:33:41.434 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:33:41.553 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:33:41.650 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:33:41.649+0000 7f4496942640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:33:41.651 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:33:41.649+0000 7f4496942640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:33:41.736 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8415 8 365281 365273 0.339094 0 - 0.0299513 2026-02-23T15:33:41.763 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:33:41.763 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":321,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:31:45.125097+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:31:24.328052+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:28:28.333996+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:30:12.470914+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:33:41.764 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:33:41 trial068 ceph-mon[38805]: pgmap v4930: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:33:42.736 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8416 8 365281 365273 0.339054 0 - 0.0299513 2026-02-23T15:33:42.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:33:42 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2138518280' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:33:42.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:33:42 trial068 ceph-mon[38805]: 5.a scrub starts 2026-02-23T15:33:42.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:33:42 trial068 ceph-mon[38805]: 5.a scrub ok 2026-02-23T15:33:43.736 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8417 8 365281 365273 0.339014 0 - 0.0299513 2026-02-23T15:33:43.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:33:43 trial068 ceph-mon[38805]: pgmap v4931: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:33:44.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:33:43 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:33:43] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:33:44.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:33:43 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:33:43.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:33:44.736 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8418 8 365281 365273 0.338973 0 - 0.0299513 2026-02-23T15:33:44.803 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:33:44.919 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:33:45.018 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:33:45.017+0000 7fda0b284640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:33:45.018 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:33:45.017+0000 7fda0b284640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:33:45.128 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:33:45.128 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":321,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:31:45.125097+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:31:24.328052+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:28:28.333996+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:30:12.470914+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:33:45.737 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8419 8 365281 365273 0.338933 0 - 0.0299513 2026-02-23T15:33:45.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:33:45 trial068 ceph-mon[38805]: pgmap v4932: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:33:45.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:33:45 trial068 ceph-mon[38805]: 5.7 scrub starts 2026-02-23T15:33:45.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:33:45 trial068 ceph-mon[38805]: 5.7 scrub ok 2026-02-23T15:33:45.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:33:45 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2767792217' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:33:46.736 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T15:33:46.736684+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T15:33:46.737 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T15:33:46.737 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8420 8 365281 365273 0.338893 0 - 0.0299513 2026-02-23T15:33:47.737 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8421 8 365281 365273 0.338853 0 - 0.0299513 2026-02-23T15:33:47.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:33:47 trial068 ceph-mon[38805]: pgmap v4933: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:33:48.164 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:33:48.281 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:33:48.383 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:33:48.381+0000 7fa255363640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:33:48.383 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:33:48.381+0000 7fa255363640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:33:48.491 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:33:48.491 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":321,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:31:45.125097+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:31:24.328052+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:28:28.333996+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:30:12.470914+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:33:48.737 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8422 8 365281 365273 0.338812 0 - 0.0299513 2026-02-23T15:33:48.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:33:48 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3334898117' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:33:49.737 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8423 8 365281 365273 0.338772 0 - 0.0299513 2026-02-23T15:33:49.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:33:49 trial068 ceph-mon[38805]: pgmap v4934: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:33:50.737 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8424 8 365281 365273 0.338732 0 - 0.0299513 2026-02-23T15:33:51.529 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:33:51.555 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:33:51 trial068 ceph-mon[38805]: pgmap v4935: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:33:51.648 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:33:51.737 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8425 8 365281 365273 0.338692 0 - 0.0299513 2026-02-23T15:33:51.752 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:33:51.750+0000 7feb80032640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:33:51.752 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:33:51.750+0000 7feb80032640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:33:51.863 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:33:51.863 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":321,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:31:45.125097+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:31:24.328052+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:28:28.333996+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:30:12.470914+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:33:52.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:33:52 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2271509110' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:33:52.737 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8426 8 365281 365273 0.338652 0 - 0.0299513 2026-02-23T15:33:53.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:33:53 trial068 ceph-mon[38805]: pgmap v4936: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:33:53.737 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8427 8 365281 365273 0.338611 0 - 0.0299513 2026-02-23T15:33:54.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:33:53 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:33:53] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:33:54.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:33:53 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:33:53.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:33:54.737 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8428 8 365281 365273 0.338571 0 - 0.0299513 2026-02-23T15:33:54.900 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:33:55.015 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:33:55.116 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:33:55.115+0000 7fe614d4f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:33:55.117 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:33:55.115+0000 7fe614d4f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:33:55.226 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:33:55.226 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":321,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:31:45.125097+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:31:24.328052+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:28:28.333996+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:30:12.470914+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:33:55.737 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8429 8 365281 365273 0.338531 0 - 0.0299513 2026-02-23T15:33:55.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:33:55 trial068 ceph-mon[38805]: pgmap v4937: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:33:55.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:33:55 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T15:33:55.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:33:55 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T15:33:55.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:33:55 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/4247288152' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:33:56.737 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8430 8 365281 365273 0.338491 0 - 0.0299513 2026-02-23T15:33:56.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:33:56 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T15:33:56.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:33:56 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T15:33:57.737 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8431 8 365281 365273 0.338451 0 - 0.0299513 2026-02-23T15:33:57.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:33:57 trial068 ceph-mon[38805]: pgmap v4938: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:33:58.264 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:33:58.383 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:33:58.481 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:33:58.479+0000 7f3f8a0db640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:33:58.481 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:33:58.480+0000 7f3f8a0db640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:33:58.595 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:33:58.595 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":321,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:31:45.125097+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:31:24.328052+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:28:28.333996+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:30:12.470914+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:33:58.737 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8432 8 365281 365273 0.338411 0 - 0.0299513 2026-02-23T15:33:59.737 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8433 8 365281 365273 0.338371 0 - 0.0299513 2026-02-23T15:33:59.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:33:59 trial068 ceph-mon[38805]: pgmap v4939: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:33:59.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:33:59 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3296921132' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:34:00.737 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8434 8 365281 365273 0.33833 0 - 0.0299513 2026-02-23T15:34:00.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:34:00 trial068 ceph-mon[38805]: 5.5 scrub starts 2026-02-23T15:34:00.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:34:00 trial068 ceph-mon[38805]: 5.5 scrub ok 2026-02-23T15:34:01.635 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:34:01.737 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8435 8 365281 365273 0.33829 0 - 0.0299513 2026-02-23T15:34:01.753 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:34:01.775 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:34:01 trial068 ceph-mon[38805]: pgmap v4940: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:34:01.854 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:34:01.852+0000 7fed2788e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:34:01.854 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:34:01.853+0000 7fed2788e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:34:01.963 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:34:01.963 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":321,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:31:45.125097+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:31:24.328052+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:28:28.333996+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:30:12.470914+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:34:02.738 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8436 8 365281 365273 0.33825 0 - 0.0299513 2026-02-23T15:34:02.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:34:02 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2454688116' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:34:03.738 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8437 8 365281 365273 0.33821 0 - 0.0299513 2026-02-23T15:34:03.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:34:03 trial068 ceph-mon[38805]: pgmap v4941: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:34:04.265 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:34:03 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:34:03.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:34:04.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:34:03 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:34:03] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:34:04.738 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8438 8 365281 365273 0.33817 0 - 0.0299513 2026-02-23T15:34:05.000 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:34:05.118 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:34:05.217 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:34:05.216+0000 7fd250876640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:34:05.218 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:34:05.216+0000 7fd250876640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:34:05.326 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:34:05.326 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":321,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:31:45.125097+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:31:24.328052+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:28:28.333996+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:30:12.470914+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:34:05.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:34:05 trial068 ceph-mon[38805]: pgmap v4942: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:34:05.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:34:05 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/989837114' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:34:05.738 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8439 8 365281 365273 0.33813 0 - 0.0299513 2026-02-23T15:34:06.738 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T15:34:06.738019+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T15:34:06.738 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T15:34:06.738 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8440 8 365281 365273 0.33809 0 - 0.0299513 2026-02-23T15:34:07.738 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8441 8 365281 365273 0.33805 0 - 0.0299513 2026-02-23T15:34:07.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:34:07 trial068 ceph-mon[38805]: pgmap v4943: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:34:08.364 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:34:08.483 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:34:08.579 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:34:08.578+0000 7fb7b5742640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:34:08.580 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:34:08.578+0000 7fb7b5742640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:34:08.686 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:34:08.687 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":321,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:31:45.125097+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:31:24.328052+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:28:28.333996+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:30:12.470914+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:34:08.687 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:34:08 trial068 ceph-mon[38805]: 5.8 scrub starts 2026-02-23T15:34:08.687 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:34:08 trial068 ceph-mon[38805]: 5.8 scrub ok 2026-02-23T15:34:08.738 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8442 8 365281 365273 0.33801 0 - 0.0299513 2026-02-23T15:34:09.738 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8443 8 365281 365273 0.33797 0 - 0.0299513 2026-02-23T15:34:09.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:34:09 trial068 ceph-mon[38805]: pgmap v4944: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:34:09.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:34:09 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3323178498' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:34:10.738 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8444 8 365281 365273 0.33793 0 - 0.0299513 2026-02-23T15:34:11.724 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:34:11.738 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8445 8 365281 365273 0.33789 0 - 0.0299513 2026-02-23T15:34:11.748 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:34:11 trial068 ceph-mon[38805]: pgmap v4945: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:34:11.839 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:34:11.940 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:34:11.939+0000 7f48fddbd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:34:11.941 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:34:11.939+0000 7f48fddbd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:34:12.051 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:34:12.052 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":321,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:31:45.125097+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:31:24.328052+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:28:28.333996+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:30:12.470914+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:34:12.739 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8446 8 365281 365273 0.33785 0 - 0.0299513 2026-02-23T15:34:12.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:34:12 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2290840714' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:34:13.739 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8447 8 365281 365273 0.33781 0 - 0.0299513 2026-02-23T15:34:13.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:34:13 trial068 ceph-mon[38805]: pgmap v4946: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:34:14.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:34:13 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:34:13] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:34:14.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:34:13 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:34:13.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:34:14.739 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8448 8 365281 365273 0.33777 0 - 0.0299513 2026-02-23T15:34:15.088 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:34:15.208 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:34:15.306 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:34:15.304+0000 7f02ad212640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:34:15.306 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:34:15.304+0000 7f02ad212640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:34:15.415 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:34:15.415 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":321,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:31:45.125097+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:31:24.328052+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:28:28.333996+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:30:12.470914+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:34:15.739 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8449 8 365281 365273 0.33773 0 - 0.0299513 2026-02-23T15:34:15.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:34:15 trial068 ceph-mon[38805]: pgmap v4947: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:34:15.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:34:15 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2370504137' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:34:16.739 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8450 8 365281 365273 0.33769 0 - 0.0299513 2026-02-23T15:34:16.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:34:16 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T15:34:16.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:34:16 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T15:34:17.739 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8451 8 365281 365273 0.33765 0 - 0.0299513 2026-02-23T15:34:17.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:34:17 trial068 ceph-mon[38805]: pgmap v4948: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:34:18.453 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:34:18.571 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:34:18.669 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:34:18.667+0000 7f95a9a90640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:34:18.669 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:34:18.667+0000 7f95a9a90640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:34:18.739 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8452 8 365281 365273 0.33761 0 - 0.0299513 2026-02-23T15:34:18.779 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:34:18.779 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":321,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:31:45.125097+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:31:24.328052+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:28:28.333996+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:30:12.470914+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:34:19.739 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8453 8 365281 365273 0.33757 0 - 0.0299513 2026-02-23T15:34:19.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:34:19 trial068 ceph-mon[38805]: pgmap v4949: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:34:19.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:34:19 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2939164989' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:34:20.739 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8454 8 365281 365273 0.33753 0 - 0.0299513 2026-02-23T15:34:21.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:34:21 trial068 ceph-mon[38805]: pgmap v4950: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:34:21.739 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8455 8 365281 365273 0.33749 0 - 0.0299513 2026-02-23T15:34:21.816 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:34:21.935 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:34:22.037 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:34:22.035+0000 7fd5e4a2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:34:22.037 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:34:22.035+0000 7fd5e4a2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:34:22.146 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:34:22.147 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":322,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:34:21.145205+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:31:24.328052+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:33:55.531945+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:30:12.470914+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:34:22.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:34:22 trial068 ceph-mon[38805]: osdmap e322: 4 total, 3 up, 4 in 2026-02-23T15:34:22.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:34:22 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1257873188' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:34:22.739 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8456 8 365281 365273 0.33745 0 - 0.0299513 2026-02-23T15:34:23.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:34:23 trial068 ceph-mon[38805]: pgmap v4952: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:34:23.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:34:23 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2026-02-23T15:34:23.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:34:23 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-23T15:34:23.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:34:23 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2026-02-23T15:34:23.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:34:23 trial068 ceph-mon[38805]: pgmap v4953: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:34:23.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:34:23 trial068 ceph-mon[38805]: pgmap v4954: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:34:23.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:34:23 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T15:34:23.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:34:23 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2026-02-23T15:34:23.739 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8457 8 365281 365273 0.33741 0 - 0.0299513 2026-02-23T15:34:24.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:34:23 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:34:23] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:34:24.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:34:23 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:34:23.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:34:24.739 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8458 8 365281 365273 0.33737 0 - 0.0299513 2026-02-23T15:34:25.186 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:34:25.301 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:34:25.400 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:34:25.399+0000 7f01175fa640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:34:25.401 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:34:25.399+0000 7f01175fa640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:34:25.513 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:34:25.513 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":323,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:34:25.145530+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:31:24.328052+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:33:55.531945+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:30:12.470914+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:34:25.739 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8459 8 365281 365273 0.337331 0 - 0.0299513 2026-02-23T15:34:25.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:34:25 trial068 ceph-mon[38805]: pgmap v4955: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:34:25.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:34:25 trial068 ceph-mon[38805]: osdmap e323: 4 total, 3 up, 4 in 2026-02-23T15:34:25.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:34:25 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/41786019' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:34:26.739 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T15:34:26.739639+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T15:34:26.740 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T15:34:26.740 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8460 8 365281 365273 0.337291 0 - 0.0299513 2026-02-23T15:34:27.740 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8461 8 365281 365273 0.337251 0 - 0.0299513 2026-02-23T15:34:28.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:34:27 trial068 ceph-mon[38805]: pgmap v4957: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:34:28.551 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:34:28.672 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:34:28.739 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8462 8 365281 365273 0.337211 0 - 0.0299513 2026-02-23T15:34:28.781 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:34:28.779+0000 7f406a79f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:34:28.781 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:34:28.779+0000 7f406a79f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:34:28.890 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:34:28.891 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":323,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:34:25.145530+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:31:24.328052+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:33:55.531945+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:30:12.470914+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:34:29.740 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8463 8 365281 365273 0.337171 0 - 0.0299513 2026-02-23T15:34:30.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:34:29 trial068 ceph-mon[38805]: pgmap v4958: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:34:30.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:34:29 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2882020356' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:34:30.740 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8464 8 365281 365273 0.337131 0 - 0.0299513 2026-02-23T15:34:31.740 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8465 8 365281 365273 0.337091 0 - 0.0299513 2026-02-23T15:34:31.928 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:34:32.045 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:34:32.067 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:34:31 trial068 ceph-mon[38805]: pgmap v4959: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:34:32.149 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:34:32.147+0000 7f2f7da34640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:34:32.149 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:34:32.147+0000 7f2f7da34640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:34:32.258 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:34:32.258 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":323,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:34:25.145530+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:31:24.328052+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:33:55.531945+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:30:12.470914+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:34:32.740 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8466 8 365281 365273 0.337052 0 - 0.0299513 2026-02-23T15:34:33.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:34:32 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2087297550' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:34:33.740 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8467 8 365281 365273 0.337012 0 - 0.0299513 2026-02-23T15:34:34.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:34:33 trial068 ceph-mon[38805]: pgmap v4960: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:34:34.094 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:34:33 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:34:33] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:34:34.265 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:34:33 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:34:33.974Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:34:34.740 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8468 8 365281 365273 0.336972 0 - 0.0299513 2026-02-23T15:34:35.297 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:34:35.417 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:34:35.520 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:34:35.518+0000 7f4d44927640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:34:35.520 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:34:35.518+0000 7f4d44927640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:34:35.628 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:34:35.628 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":323,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:34:25.145530+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:31:24.328052+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:33:55.531945+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:30:12.470914+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:34:35.740 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8469 8 365281 365273 0.336932 0 - 0.0299513 2026-02-23T15:34:36.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:34:35 trial068 ceph-mon[38805]: pgmap v4961: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:34:36.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:34:35 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2782652561' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:34:36.740 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8470 8 365281 365273 0.336892 0 - 0.0299513 2026-02-23T15:34:37.740 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8471 8 365281 365273 0.336853 0 - 0.0299513 2026-02-23T15:34:38.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:34:37 trial068 ceph-mon[38805]: pgmap v4962: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:34:38.666 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:34:38.740 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8472 8 365281 365273 0.336813 0 - 0.0299513 2026-02-23T15:34:38.782 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:34:38.879 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:34:38.878+0000 7fcd3113f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:34:38.880 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:34:38.878+0000 7fcd3113f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:34:38.988 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:34:38.988 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":323,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:34:25.145530+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:31:24.328052+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:33:55.531945+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:30:12.470914+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:34:39.740 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8473 8 365281 365273 0.336773 0 - 0.0299513 2026-02-23T15:34:40.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:34:39 trial068 ceph-mon[38805]: pgmap v4963: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:34:40.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:34:39 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2369812641' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:34:40.740 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8474 8 365281 365273 0.336733 0 - 0.0299513 2026-02-23T15:34:41.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:34:41 trial068 ceph-mon[38805]: pgmap v4964: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:34:41.740 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8475 8 365281 365273 0.336694 0 - 0.0299513 2026-02-23T15:34:42.042 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:34:42.160 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:34:42.258 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:34:42.256+0000 7f788cf23640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:34:42.258 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:34:42.257+0000 7f788cf23640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:34:42.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:34:42 trial068 ceph-mon[38805]: 5.e scrub starts 2026-02-23T15:34:42.368 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:34:42.368 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":323,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:34:25.145530+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:31:24.328052+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:33:55.531945+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:30:12.470914+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:34:42.741 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8476 8 365281 365273 0.336654 0 - 0.0299513 2026-02-23T15:34:43.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:34:43 trial068 ceph-mon[38805]: 5.e scrub ok 2026-02-23T15:34:43.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:34:43 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3333130589' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:34:43.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:34:43 trial068 ceph-mon[38805]: pgmap v4965: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:34:43.741 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8477 8 365281 365273 0.336614 0 - 0.0299513 2026-02-23T15:34:44.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:34:43 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:34:43] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:34:44.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:34:43 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:34:43.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:34:44.741 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8478 8 365281 365273 0.336575 0 - 0.0299513 2026-02-23T15:34:45.405 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:34:45.523 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:34:45.627 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:34:45.626+0000 7f5997897640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:34:45.627 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:34:45.626+0000 7f5997897640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:34:45.738 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:34:45.738 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":323,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:34:25.145530+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:31:24.328052+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:33:55.531945+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:30:12.470914+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:34:45.741 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8479 8 365281 365273 0.336535 0 - 0.0299513 2026-02-23T15:34:46.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:34:45 trial068 ceph-mon[38805]: pgmap v4966: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:34:46.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:34:45 trial068 ceph-mon[38805]: 5.a scrub starts 2026-02-23T15:34:46.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:34:45 trial068 ceph-mon[38805]: 5.a scrub ok 2026-02-23T15:34:46.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:34:45 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2320451313' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:34:46.741 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T15:34:46.741057+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T15:34:46.741 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T15:34:46.741 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8480 8 365281 365273 0.336495 0 - 0.0299513 2026-02-23T15:34:47.741 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8481 8 365281 365273 0.336455 0 - 0.0299513 2026-02-23T15:34:48.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:34:47 trial068 ceph-mon[38805]: pgmap v4967: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:34:48.741 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8482 8 365281 365273 0.336416 0 - 0.0299513 2026-02-23T15:34:48.775 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:34:48.893 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:34:48.990 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:34:48.988+0000 7fcc66852640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:34:48.990 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:34:48.988+0000 7fcc66852640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:34:49.099 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:34:49.099 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":323,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:34:25.145530+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:31:24.328052+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:33:55.531945+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:30:12.470914+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:34:49.741 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8483 8 365281 365273 0.336376 0 - 0.0299513 2026-02-23T15:34:50.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:34:49 trial068 ceph-mon[38805]: pgmap v4968: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:34:50.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:34:49 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1993320342' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:34:50.741 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8484 8 365281 365273 0.336337 0 - 0.0299513 2026-02-23T15:34:51.741 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8485 8 365281 365273 0.336297 0 - 0.0299513 2026-02-23T15:34:52.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:34:51 trial068 ceph-mon[38805]: pgmap v4969: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:34:52.137 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:34:52.253 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:34:52.357 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:34:52.355+0000 7f262c9a9640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:34:52.357 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:34:52.355+0000 7f262c9a9640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:34:52.465 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:34:52.465 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":323,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:34:25.145530+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:31:24.328052+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:33:55.531945+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:30:12.470914+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:34:52.741 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8486 8 365281 365273 0.336257 0 - 0.0299513 2026-02-23T15:34:53.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:34:52 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1019554932' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:34:53.741 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8487 8 365281 365273 0.336218 0 - 0.0299513 2026-02-23T15:34:54.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:34:53 trial068 ceph-mon[38805]: pgmap v4970: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:34:54.094 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:34:53 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:34:53] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:34:54.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:34:53 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:34:53.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:34:54.741 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8488 8 365281 365273 0.336178 0 - 0.0299513 2026-02-23T15:34:55.503 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:34:55.621 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:34:55.723 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:34:55.721+0000 7fa7627ae640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:34:55.723 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:34:55.721+0000 7fa7627ae640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:34:55.741 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8489 8 365281 365273 0.336138 0 - 0.0299513 2026-02-23T15:34:55.836 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:34:55.836 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":323,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:34:25.145530+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:31:24.328052+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:33:55.531945+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:30:12.470914+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:34:55.837 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:34:55 trial068 ceph-mon[38805]: pgmap v4971: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:34:56.741 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8490 8 365281 365273 0.336099 0 - 0.0299513 2026-02-23T15:34:57.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:34:56 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2920662500' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:34:57.741 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8491 8 365281 365273 0.336059 0 - 0.0299513 2026-02-23T15:34:58.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:34:57 trial068 ceph-mon[38805]: pgmap v4972: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:34:58.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:34:57 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T15:34:58.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:34:57 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T15:34:58.742 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8492 8 365281 365273 0.33602 0 - 0.0299513 2026-02-23T15:34:58.889 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:34:59.008 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:34:59.112 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:34:59.111+0000 7f5f02acc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:34:59.113 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:34:59.111+0000 7f5f02acc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:34:59.223 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:34:59.223 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":323,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:34:25.145530+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:31:24.328052+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:33:55.531945+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:30:12.470914+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:34:59.742 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8493 8 365281 365273 0.33598 0 - 0.0299513 2026-02-23T15:35:00.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:34:59 trial068 ceph-mon[38805]: pgmap v4973: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:35:00.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:34:59 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/4189533707' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:35:00.742 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8494 8 365281 365273 0.335941 0 - 0.0299513 2026-02-23T15:35:01.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:35:01 trial068 ceph-mon[38805]: pgmap v4974: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:35:01.742 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8495 8 365281 365273 0.335901 0 - 0.0299513 2026-02-23T15:35:02.261 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:35:02.379 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:35:02.484 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:35:02.482+0000 7f737ff1f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:35:02.484 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:35:02.483+0000 7f737ff1f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:35:02.594 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:35:02.595 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":323,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:34:25.145530+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:31:24.328052+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:33:55.531945+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:30:12.470914+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:35:02.742 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8496 8 365281 365273 0.335861 0 - 0.0299513 2026-02-23T15:35:03.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:35:02 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2088670860' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:35:03.742 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8497 8 365281 365273 0.335822 0 - 0.0299513 2026-02-23T15:35:03.973 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:35:03 trial068 ceph-mon[38805]: pgmap v4975: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:35:04.265 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:35:03 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:35:03.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:35:04.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:35:03 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:35:03] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:35:04.742 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8498 8 365281 365273 0.335782 0 - 0.0299513 2026-02-23T15:35:05.631 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:35:05.742 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8499 8 365281 365273 0.335743 0 - 0.0299513 2026-02-23T15:35:05.751 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:35:05.850 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:35:05.849+0000 7fbb336e2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:35:05.851 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:35:05.849+0000 7fbb336e2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:35:05.958 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:35:05.959 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":323,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:34:25.145530+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:31:24.328052+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:33:55.531945+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:30:12.470914+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:35:05.959 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:35:05 trial068 ceph-mon[38805]: pgmap v4976: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:35:06.742 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T15:35:06.742278+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T15:35:06.742 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T15:35:06.742 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8500 8 365281 365273 0.335703 0 - 0.0299513 2026-02-23T15:35:07.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:35:06 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3031042760' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:35:07.742 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8501 8 365281 365273 0.335664 0 - 0.0299513 2026-02-23T15:35:08.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:35:07 trial068 ceph-mon[38805]: pgmap v4977: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:35:08.742 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8502 8 365281 365273 0.335624 0 - 0.0299513 2026-02-23T15:35:08.995 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:35:09.113 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:35:09.219 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:35:09.217+0000 7fcc3a7e7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:35:09.219 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:35:09.218+0000 7fcc3a7e7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:35:09.329 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:35:09.329 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":323,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:34:25.145530+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:31:24.328052+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:33:55.531945+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:30:12.470914+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:35:09.742 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8503 8 365281 365273 0.335585 0 - 0.0299513 2026-02-23T15:35:10.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:35:09 trial068 ceph-mon[38805]: pgmap v4978: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:35:10.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:35:09 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1918399438' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:35:10.742 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8504 8 365281 365273 0.335546 0 - 0.0299513 2026-02-23T15:35:11.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:35:10 trial068 ceph-mon[38805]: 5.7 scrub starts 2026-02-23T15:35:11.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:35:10 trial068 ceph-mon[38805]: 5.7 scrub ok 2026-02-23T15:35:11.742 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8505 8 365281 365273 0.335506 0 - 0.0299513 2026-02-23T15:35:12.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:35:11 trial068 ceph-mon[38805]: pgmap v4979: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:35:12.366 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:35:12.482 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:35:12.576 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:35:12.574+0000 7f6939747640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:35:12.576 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:35:12.575+0000 7f6939747640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:35:12.685 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:35:12.685 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":323,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:34:25.145530+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:31:24.328052+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:33:55.531945+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:30:12.470914+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:35:12.742 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8506 8 365281 365273 0.335467 0 - 0.0299513 2026-02-23T15:35:13.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:35:12 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/4083324185' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:35:13.743 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8507 8 365281 365273 0.335427 0 - 0.0299513 2026-02-23T15:35:14.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:35:13 trial068 ceph-mon[38805]: pgmap v4980: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:35:14.094 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:35:13 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:35:13] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:35:14.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:35:13 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:35:13.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:35:14.743 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8508 8 365281 365273 0.335388 0 - 0.0299513 2026-02-23T15:35:15.722 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:35:15.743 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8509 8 365281 365273 0.335348 0 - 0.0299513 2026-02-23T15:35:15.841 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:35:15.940 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:35:15.938+0000 7fa9486a9640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:35:15.940 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:35:15.939+0000 7fa9486a9640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:35:16.050 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:35:16.050 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":323,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:34:25.145530+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:31:24.328052+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:33:55.531945+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:30:12.470914+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:35:16.050 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:35:15 trial068 ceph-mon[38805]: pgmap v4981: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:35:16.743 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8510 8 365281 365273 0.335309 0 - 0.0299513 2026-02-23T15:35:17.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:35:16 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3602377201' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:35:17.743 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8511 8 365281 365273 0.33527 0 - 0.0299513 2026-02-23T15:35:18.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:35:17 trial068 ceph-mon[38805]: 5.8 scrub starts 2026-02-23T15:35:18.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:35:17 trial068 ceph-mon[38805]: pgmap v4982: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:35:18.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:35:17 trial068 ceph-mon[38805]: 5.8 scrub ok 2026-02-23T15:35:18.743 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8512 8 365281 365273 0.33523 0 - 0.0299513 2026-02-23T15:35:19.087 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:35:19.205 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:35:19.299 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:35:19.297+0000 7f3a42f9f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:35:19.299 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:35:19.298+0000 7f3a42f9f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:35:19.410 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:35:19.411 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":323,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:34:25.145530+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:31:24.328052+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:33:55.531945+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:30:12.470914+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:35:19.743 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8513 8 365281 365273 0.335191 0 - 0.0299513 2026-02-23T15:35:20.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:35:19 trial068 ceph-mon[38805]: 5.5 scrub starts 2026-02-23T15:35:20.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:35:19 trial068 ceph-mon[38805]: pgmap v4983: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:35:20.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:35:19 trial068 ceph-mon[38805]: 5.5 scrub ok 2026-02-23T15:35:20.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:35:19 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1980010629' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:35:20.743 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8514 8 365281 365273 0.335151 0 - 0.0299513 2026-02-23T15:35:21.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:35:20 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T15:35:21.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:35:20 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T15:35:21.743 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8515 8 365281 365273 0.335112 0 - 0.0299513 2026-02-23T15:35:22.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:35:21 trial068 ceph-mon[38805]: pgmap v4984: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:35:22.447 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:35:22.566 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:35:22.671 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:35:22.669+0000 7fbdb5d9a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:35:22.671 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:35:22.669+0000 7fbdb5d9a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:35:22.743 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8516 8 365281 365273 0.335073 0 - 0.0299513 2026-02-23T15:35:22.781 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:35:22.781 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":323,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:34:25.145530+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:31:24.328052+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:33:55.531945+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:30:12.470914+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:35:23.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:35:22 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2026-02-23T15:35:23.743 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8517 8 365281 365273 0.335033 0 - 0.0299513 2026-02-23T15:35:24.093 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:35:23 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:35:23] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:35:24.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:35:23 trial068 ceph-mon[38805]: pgmap v4985: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:35:24.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:35:23 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3967988494' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:35:24.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:35:23 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-23T15:35:24.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:35:23 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2026-02-23T15:35:24.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:35:23 trial068 ceph-mon[38805]: pgmap v4986: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:35:24.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:35:23 trial068 ceph-mon[38805]: pgmap v4987: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:35:24.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:35:23 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T15:35:24.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:35:23 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2026-02-23T15:35:24.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:35:23 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:35:23.974Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:35:24.743 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8518 8 365281 365273 0.334994 0 - 0.0299513 2026-02-23T15:35:25.743 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8519 8 365281 365273 0.334955 0 - 0.0299513 2026-02-23T15:35:25.819 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:35:25.937 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:35:26.041 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:35:26.040+0000 7feb957aa640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:35:26.042 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:35:26.040+0000 7feb957aa640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:35:26.149 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:35:26.149 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":323,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:34:25.145530+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:31:24.328052+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:33:55.531945+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:30:12.470914+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:35:26.150 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:35:26 trial068 ceph-mon[38805]: pgmap v4988: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:35:26.743 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T15:35:26.743611+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T15:35:26.744 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T15:35:26.744 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8520 8 365281 365273 0.334915 0 - 0.0299513 2026-02-23T15:35:27.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:35:27 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1611754985' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:35:27.743 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8521 8 365281 365273 0.334876 0 - 0.0299513 2026-02-23T15:35:28.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:35:28 trial068 ceph-mon[38805]: pgmap v4989: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:35:28.743 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8522 8 365281 365273 0.334837 0 - 0.0299513 2026-02-23T15:35:29.201 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:35:29.318 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:35:29.416 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:35:29.415+0000 7f6f16baa640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:35:29.416 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:35:29.415+0000 7f6f16baa640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:35:29.527 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:35:29.527 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":323,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:34:25.145530+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:31:24.328052+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:33:55.531945+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:30:12.470914+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:35:29.744 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8523 8 365281 365273 0.334797 0 - 0.0299513 2026-02-23T15:35:30.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:35:30 trial068 ceph-mon[38805]: pgmap v4990: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:35:30.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:35:30 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3531823001' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:35:30.744 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8524 8 365281 365273 0.334758 0 - 0.0299513 2026-02-23T15:35:31.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:35:31 trial068 ceph-mon[38805]: pgmap v4991: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:35:31.744 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8525 8 365281 365273 0.334719 0 - 0.0299513 2026-02-23T15:35:32.564 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:35:32.683 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:35:32.744 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8526 8 365281 365273 0.33468 0 - 0.0299513 2026-02-23T15:35:32.781 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:35:32.779+0000 7f30435c8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:35:32.781 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:35:32.779+0000 7f30435c8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:35:32.890 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:35:32.890 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":323,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:34:25.145530+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:31:24.328052+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:33:55.531945+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:30:12.470914+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:35:33.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:35:32 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3001982443' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:35:33.744 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8527 8 365281 365273 0.33464 0 - 0.0299513 2026-02-23T15:35:34.265 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:35:33 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:35:33.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:35:34.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:35:33 trial068 ceph-mon[38805]: pgmap v4992: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:35:34.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:35:33 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:35:33] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:35:34.744 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8528 8 365281 365273 0.334601 0 - 0.0299513 2026-02-23T15:35:35.744 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8529 8 365281 365273 0.334562 0 - 0.0299513 2026-02-23T15:35:35.926 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:35:36.046 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:35:36.151 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:35:36.149+0000 7f1344e1a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:35:36.151 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:35:36.150+0000 7f1344e1a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:35:36.258 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:35:36.259 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":323,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:34:25.145530+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:31:24.328052+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:33:55.531945+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:30:12.470914+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:35:36.259 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:35:36 trial068 ceph-mon[38805]: pgmap v4993: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:35:36.744 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8530 8 365281 365273 0.334523 0 - 0.0299513 2026-02-23T15:35:37.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:35:37 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2386365323' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:35:37.744 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8531 8 365281 365273 0.334484 0 - 0.0299513 2026-02-23T15:35:38.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:35:38 trial068 ceph-mon[38805]: pgmap v4994: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:35:38.744 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8532 8 365281 365273 0.334444 0 - 0.0299513 2026-02-23T15:35:39.296 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:35:39.415 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:35:39.519 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:35:39.517+0000 7f0bfa2c4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:35:39.519 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:35:39.518+0000 7f0bfa2c4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:35:39.628 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:35:39.628 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":324,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:35:39.006686+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:31:24.328052+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:33:55.531945+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:34:56.867874+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:35:39.744 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8533 8 365281 365273 0.334405 0 - 0.0299513 2026-02-23T15:35:40.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:35:40 trial068 ceph-mon[38805]: pgmap v4995: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:35:40.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:35:40 trial068 ceph-mon[38805]: osdmap e324: 4 total, 3 up, 4 in 2026-02-23T15:35:40.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:35:40 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1224611392' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:35:40.744 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8534 8 365281 365273 0.334366 0 - 0.0299513 2026-02-23T15:35:41.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:35:41 trial068 ceph-mon[38805]: osdmap e325: 4 total, 3 up, 4 in 2026-02-23T15:35:41.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:35:41 trial068 ceph-mon[38805]: pgmap v4998: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:35:41.745 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8535 8 365281 365273 0.334327 0 - 0.0299513 2026-02-23T15:35:42.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:35:42 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T15:35:42.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:35:42 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T15:35:42.665 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:35:42.745 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8536 8 365281 365273 0.334288 0 - 0.0299513 2026-02-23T15:35:42.782 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:35:42.879 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:35:42.877+0000 7f10e5bce640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:35:42.879 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:35:42.877+0000 7f10e5bce640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:35:42.989 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:35:42.989 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":325,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:35:40.154151+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:31:24.328052+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:33:55.531945+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:34:56.867874+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:35:43.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:35:43 trial068 ceph-mon[38805]: pgmap v4999: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:35:43.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:35:43 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3730567943' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:35:43.745 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8537 8 365281 365273 0.334248 0 - 0.0299513 2026-02-23T15:35:44.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:35:43 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:35:43] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:35:44.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:35:43 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:35:43.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:35:44.745 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8538 8 365281 365273 0.334209 0 - 0.0299513 2026-02-23T15:35:45.745 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8539 8 365281 365273 0.33417 0 - 0.0299513 2026-02-23T15:35:46.043 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:35:46.162 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:35:46.260 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:35:46.259+0000 7f284e327640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:35:46.261 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:35:46 trial068 ceph-mon[38805]: pgmap v5000: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:35:46.261 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:35:46.259+0000 7f284e327640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:35:46.371 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:35:46.371 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":325,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:35:40.154151+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:31:24.328052+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:33:55.531945+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:34:56.867874+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:35:46.745 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T15:35:46.745197+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T15:35:46.745 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T15:35:46.745 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8540 8 365281 365273 0.334131 0 - 0.0299513 2026-02-23T15:35:47.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:35:47 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3460009102' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:35:47.745 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8541 8 365281 365273 0.334092 0 - 0.0299513 2026-02-23T15:35:48.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:35:48 trial068 ceph-mon[38805]: pgmap v5001: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:35:48.745 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8542 8 365281 365273 0.334053 0 - 0.0299513 2026-02-23T15:35:49.408 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:35:49.526 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:35:49.626 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:35:49.624+0000 7fd1ba833640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:35:49.626 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:35:49.625+0000 7fd1ba833640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:35:49.742 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:35:49.742 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":325,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:35:40.154151+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:31:24.328052+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:33:55.531945+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:34:56.867874+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:35:49.745 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8543 8 365281 365273 0.334014 0 - 0.0299513 2026-02-23T15:35:50.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:35:50 trial068 ceph-mon[38805]: pgmap v5002: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:35:50.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:35:50 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3537974265' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:35:50.745 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8544 8 365281 365273 0.333975 0 - 0.0299513 2026-02-23T15:35:51.745 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8545 8 365281 365273 0.333936 0 - 0.0299513 2026-02-23T15:35:52.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:35:52 trial068 ceph-mon[38805]: pgmap v5003: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:35:52.745 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8546 8 365281 365273 0.333896 0 - 0.0299513 2026-02-23T15:35:52.779 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:35:52.895 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:35:52.996 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:35:52.995+0000 7f3377837640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:35:52.997 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:35:52.995+0000 7f3377837640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:35:53.108 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:35:53.108 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":325,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:35:40.154151+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:31:24.328052+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:33:55.531945+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:34:56.867874+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:35:53.746 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8547 8 365281 365273 0.333857 0 - 0.0299513 2026-02-23T15:35:54.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:35:54 trial068 ceph-mon[38805]: pgmap v5004: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:35:54.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:35:54 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/161438706' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:35:54.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:35:53 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:35:53] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:35:54.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:35:53 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:35:53.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:35:54.746 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8548 8 365281 365273 0.333818 0 - 0.0299513 2026-02-23T15:35:55.746 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8549 8 365281 365273 0.333779 0 - 0.0299513 2026-02-23T15:35:56.146 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:35:56.265 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:35:56.287 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:35:56 trial068 ceph-mon[38805]: pgmap v5005: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:35:56.361 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:35:56.360+0000 7f38efd2f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:35:56.361 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:35:56.360+0000 7f38efd2f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:35:56.470 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:35:56.470 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":325,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:35:40.154151+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:31:24.328052+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:33:55.531945+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:34:56.867874+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:35:56.746 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8550 8 365281 365273 0.33374 0 - 0.0299513 2026-02-23T15:35:57.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:35:57 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1132498794' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:35:57.746 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8551 8 365281 365273 0.333701 0 - 0.0299513 2026-02-23T15:35:58.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:35:58 trial068 ceph-mon[38805]: pgmap v5006: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:35:58.746 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8552 8 365281 365273 0.333662 0 - 0.0299513 2026-02-23T15:35:59.507 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:35:59.625 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:35:59.724 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:35:59.722+0000 7fb9f3e66640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:35:59.724 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:35:59.722+0000 7fb9f3e66640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:35:59.746 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8553 8 365281 365273 0.333623 0 - 0.0299513 2026-02-23T15:35:59.834 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:35:59.835 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":325,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:35:40.154151+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:31:24.328052+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:33:55.531945+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:34:56.867874+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:36:00.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:36:00 trial068 ceph-mon[38805]: pgmap v5007: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:36:00.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:36:00 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/421404811' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:36:00.746 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8554 8 365281 365273 0.333584 0 - 0.0299513 2026-02-23T15:36:01.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:36:01 trial068 ceph-mon[38805]: pgmap v5008: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:36:01.746 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8555 8 365281 365273 0.333545 0 - 0.0299513 2026-02-23T15:36:02.746 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8556 8 365281 365273 0.333506 0 - 0.0299513 2026-02-23T15:36:02.872 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:36:02.989 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:36:03.092 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:36:03.090+0000 7fbafec8e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:36:03.092 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:36:03.090+0000 7fbafec8e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:36:03.201 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:36:03.202 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":325,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:35:40.154151+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:31:24.328052+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:33:55.531945+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:34:56.867874+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:36:03.746 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8557 8 365281 365273 0.333467 0 - 0.0299513 2026-02-23T15:36:04.264 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:36:03 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:36:03.974Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:36:04.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:36:04 trial068 ceph-mon[38805]: pgmap v5009: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:36:04.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:36:04 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/676451450' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:36:04.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:36:03 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:36:03] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:36:04.746 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8558 8 365281 365273 0.333428 0 - 0.0299513 2026-02-23T15:36:05.746 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8559 8 365281 365273 0.333389 0 - 0.0299513 2026-02-23T15:36:06.239 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:36:06.263 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:36:06 trial068 ceph-mon[38805]: 5.e scrub starts 2026-02-23T15:36:06.263 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:36:06 trial068 ceph-mon[38805]: 5.e scrub ok 2026-02-23T15:36:06.263 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:36:06 trial068 ceph-mon[38805]: pgmap v5010: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:36:06.355 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:36:06.462 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:36:06.460+0000 7fcab349d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:36:06.462 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:36:06.461+0000 7fcab349d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:36:06.571 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:36:06.571 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":325,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:35:40.154151+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:31:24.328052+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:33:55.531945+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:34:56.867874+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:36:06.747 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T15:36:06.746713+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T15:36:06.747 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T15:36:06.747 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8560 8 365281 365273 0.33335 0 - 0.0299513 2026-02-23T15:36:07.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:36:07 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2747462099' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:36:07.747 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8561 8 365281 365273 0.333311 0 - 0.0299513 2026-02-23T15:36:08.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:36:08 trial068 ceph-mon[38805]: pgmap v5011: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:36:08.747 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8562 8 365281 365273 0.333272 0 - 0.0299513 2026-02-23T15:36:09.609 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:36:09.726 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:36:09.747 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8563 8 365281 365273 0.333234 0 - 0.0299513 2026-02-23T15:36:09.825 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:36:09.823+0000 7f6924e52640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:36:09.825 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:36:09.823+0000 7f6924e52640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:36:09.933 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:36:09.934 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":325,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:35:40.154151+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:31:24.328052+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:33:55.531945+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:34:56.867874+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:36:10.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:36:10 trial068 ceph-mon[38805]: pgmap v5012: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:36:10.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:36:10 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2606549373' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:36:10.747 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8564 8 365281 365273 0.333195 0 - 0.0299513 2026-02-23T15:36:11.747 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8565 8 365281 365273 0.333156 0 - 0.0299513 2026-02-23T15:36:12.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:36:12 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T15:36:12.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:36:12 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T15:36:12.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:36:12 trial068 ceph-mon[38805]: pgmap v5013: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:36:12.747 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8566 8 365281 365273 0.333117 0 - 0.0299513 2026-02-23T15:36:12.970 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:36:13.090 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:36:13.193 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:36:13.191+0000 7ff5a18bd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:36:13.193 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:36:13.192+0000 7ff5a18bd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:36:13.304 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:36:13.305 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":325,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:35:40.154151+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:31:24.328052+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:33:55.531945+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:34:56.867874+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:36:13.747 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8567 8 365281 365273 0.333078 0 - 0.0299513 2026-02-23T15:36:14.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:36:14 trial068 ceph-mon[38805]: pgmap v5014: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:36:14.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:36:14 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1935738589' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:36:14.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:36:13 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:36:13] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:36:14.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:36:13 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:36:13.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:36:14.747 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8568 8 365281 365273 0.333039 0 - 0.0299513 2026-02-23T15:36:15.747 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8569 8 365281 365273 0.333 0 - 0.0299513 2026-02-23T15:36:16.342 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:36:16.346 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:36:16 trial068 ceph-mon[38805]: pgmap v5015: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:36:16.459 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:36:16.555 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:36:16.553+0000 7f7f36898640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:36:16.555 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:36:16.553+0000 7f7f36898640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:36:16.664 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:36:16.665 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":325,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:35:40.154151+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:31:24.328052+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:33:55.531945+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:34:56.867874+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:36:16.747 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8570 8 365281 365273 0.332961 0 - 0.0299513 2026-02-23T15:36:17.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:36:17 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/641700346' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:36:17.747 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8571 8 365281 365273 0.332923 0 - 0.0299513 2026-02-23T15:36:18.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:36:18 trial068 ceph-mon[38805]: 5.a scrub starts 2026-02-23T15:36:18.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:36:18 trial068 ceph-mon[38805]: pgmap v5016: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:36:18.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:36:18 trial068 ceph-mon[38805]: 5.a scrub ok 2026-02-23T15:36:18.747 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8572 8 365281 365273 0.332884 0 - 0.0299513 2026-02-23T15:36:19.702 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:36:19.747 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8573 8 365281 365273 0.332845 0 - 0.0299513 2026-02-23T15:36:19.820 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:36:19.921 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:36:19.919+0000 7fd3b01ef640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:36:19.921 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:36:19.920+0000 7fd3b01ef640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:36:20.030 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:36:20.031 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":325,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:35:40.154151+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:31:24.328052+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:33:55.531945+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:34:56.867874+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:36:20.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:36:20 trial068 ceph-mon[38805]: pgmap v5017: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:36:20.747 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8574 8 365281 365273 0.332806 0 - 0.0299513 2026-02-23T15:36:21.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:36:21 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/995299419' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:36:21.748 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8575 8 365281 365273 0.332767 0 - 0.0299513 2026-02-23T15:36:22.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:36:22 trial068 ceph-mon[38805]: pgmap v5018: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:36:22.748 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8576 8 365281 365273 0.332728 0 - 0.0299513 2026-02-23T15:36:23.069 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:36:23.184 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:36:23.286 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:36:23.284+0000 7f1978c42640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:36:23.286 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:36:23.285+0000 7f1978c42640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:36:23.286 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:36:23 trial068 ceph-mon[38805]: 5.7 scrub starts 2026-02-23T15:36:23.287 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:36:23 trial068 ceph-mon[38805]: 5.7 scrub ok 2026-02-23T15:36:23.287 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:36:23 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2026-02-23T15:36:23.394 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:36:23.394 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":325,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:35:40.154151+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:31:24.328052+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:33:55.531945+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:34:56.867874+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:36:23.748 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8577 8 365281 365273 0.33269 0 - 0.0299513 2026-02-23T15:36:24.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:36:23 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:36:23] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:36:24.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:36:24 trial068 ceph-mon[38805]: pgmap v5019: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:36:24.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:36:24 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-23T15:36:24.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:36:24 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2026-02-23T15:36:24.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:36:24 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T15:36:24.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:36:24 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2026-02-23T15:36:24.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:36:24 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1752996388' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:36:24.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:36:23 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:36:23.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:36:24.748 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8578 8 365281 365273 0.332651 0 - 0.0299513 2026-02-23T15:36:25.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:36:25 trial068 ceph-mon[38805]: pgmap v5020: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:36:25.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:36:25 trial068 ceph-mon[38805]: pgmap v5021: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:36:25.748 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8579 8 365281 365273 0.332612 0 - 0.0299513 2026-02-23T15:36:26.432 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:36:26.548 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:36:26.644 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:36:26.643+0000 7fb5dfb9a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:36:26.645 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:36:26.643+0000 7fb5dfb9a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:36:26.748 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T15:36:26.748061+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T15:36:26.748 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T15:36:26.748 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8580 8 365281 365273 0.332573 0 - 0.0299513 2026-02-23T15:36:26.754 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:36:26.755 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":325,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:35:40.154151+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:31:24.328052+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:33:55.531945+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:34:56.867874+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:36:27.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:36:27 trial068 ceph-mon[38805]: pgmap v5022: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:36:27.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:36:27 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1401279507' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:36:27.748 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8581 8 365281 365273 0.332535 0 - 0.0299513 2026-02-23T15:36:28.748 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8582 8 365281 365273 0.332496 0 - 0.0299513 2026-02-23T15:36:29.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:36:29 trial068 ceph-mon[38805]: pgmap v5023: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:36:29.748 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8583 8 365281 365273 0.332457 0 - 0.0299513 2026-02-23T15:36:29.791 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:36:29.906 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:36:30.008 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:36:30.006+0000 7fd436f1c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:36:30.008 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:36:30.007+0000 7fd436f1c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:36:30.119 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:36:30.119 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":325,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:35:40.154151+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:31:24.328052+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:33:55.531945+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:34:56.867874+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:36:30.748 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8584 8 365281 365273 0.332418 0 - 0.0299513 2026-02-23T15:36:31.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:36:31 trial068 ceph-mon[38805]: pgmap v5024: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:36:31.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:36:31 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3826479664' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:36:31.748 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8585 8 365281 365273 0.33238 0 - 0.0299513 2026-02-23T15:36:32.748 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8586 8 365281 365273 0.332341 0 - 0.0299513 2026-02-23T15:36:33.157 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:36:33.277 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:36:33.299 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:36:33 trial068 ceph-mon[38805]: pgmap v5025: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:36:33.380 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:36:33.379+0000 7f5948915640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:36:33.381 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:36:33.379+0000 7f5948915640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:36:33.488 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:36:33.489 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":325,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:35:40.154151+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:31:24.328052+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:33:55.531945+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:34:56.867874+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:36:33.748 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8587 8 365281 365273 0.332302 0 - 0.0299513 2026-02-23T15:36:34.265 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:36:33 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:36:33.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:36:34.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:36:34 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1554685141' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:36:34.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:36:33 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:36:33] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:36:34.748 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8588 8 365281 365273 0.332264 0 - 0.0299513 2026-02-23T15:36:35.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:36:35 trial068 ceph-mon[38805]: pgmap v5026: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:36:35.748 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8589 8 365281 365273 0.332225 0 - 0.0299513 2026-02-23T15:36:36.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:36:36 trial068 ceph-mon[38805]: 5.5 scrub starts 2026-02-23T15:36:36.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:36:36 trial068 ceph-mon[38805]: 5.5 scrub ok 2026-02-23T15:36:36.528 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:36:36.647 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:36:36.749 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8590 8 365281 365273 0.332186 0 - 0.0299513 2026-02-23T15:36:36.750 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:36:36.748+0000 7f2edd9c4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:36:36.750 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:36:36.749+0000 7f2edd9c4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:36:36.860 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:36:36.860 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":325,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:35:40.154151+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:31:24.328052+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:33:55.531945+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:34:56.867874+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:36:37.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:36:37 trial068 ceph-mon[38805]: pgmap v5027: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:36:37.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:36:37 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2494648135' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:36:37.749 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8591 8 365281 365273 0.332147 0 - 0.0299513 2026-02-23T15:36:38.749 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8592 8 365281 365273 0.332109 0 - 0.0299513 2026-02-23T15:36:39.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:36:39 trial068 ceph-mon[38805]: pgmap v5028: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:36:39.749 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8593 8 365281 365273 0.33207 0 - 0.0299513 2026-02-23T15:36:39.900 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:36:40.017 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:36:40.119 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:36:40.117+0000 7faed42b9640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:36:40.119 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:36:40.117+0000 7faed42b9640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:36:40.228 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:36:40.228 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":325,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:35:40.154151+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:31:24.328052+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:33:55.531945+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:34:56.867874+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:36:40.749 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8594 8 365281 365273 0.332032 0 - 0.0299513 2026-02-23T15:36:41.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:36:41 trial068 ceph-mon[38805]: pgmap v5029: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:36:41.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:36:41 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3171285925' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:36:41.749 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8595 8 365281 365273 0.331993 0 - 0.0299513 2026-02-23T15:36:42.749 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8596 8 365281 365273 0.331954 0 - 0.0299513 2026-02-23T15:36:43.267 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:36:43.290 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:36:43 trial068 ceph-mon[38805]: pgmap v5030: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:36:43.291 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:36:43 trial068 ceph-mon[38805]: 5.8 scrub starts 2026-02-23T15:36:43.291 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:36:43 trial068 ceph-mon[38805]: 5.8 scrub ok 2026-02-23T15:36:43.382 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:36:43.490 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:36:43.489+0000 7f562f567640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:36:43.490 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:36:43.489+0000 7f562f567640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:36:43.599 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:36:43.600 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":325,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:35:40.154151+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:31:24.328052+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:33:55.531945+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:34:56.867874+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:36:43.749 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8597 8 365281 365273 0.331916 0 - 0.0299513 2026-02-23T15:36:44.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:36:44 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1130472782' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:36:44.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:36:43 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:36:43] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:36:44.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:36:43 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:36:43.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:36:44.749 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8598 8 365281 365273 0.331877 0 - 0.0299513 2026-02-23T15:36:45.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:36:45 trial068 ceph-mon[38805]: pgmap v5031: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:36:45.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:36:45 trial068 ceph-mon[38805]: osdmap e326: 4 total, 3 up, 4 in 2026-02-23T15:36:45.749 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8599 8 365281 365273 0.331838 0 - 0.0299513 2026-02-23T15:36:46.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:36:46 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T15:36:46.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:36:46 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T15:36:46.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:36:46 trial068 ceph-mon[38805]: osdmap e327: 4 total, 3 up, 4 in 2026-02-23T15:36:46.637 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:36:46.749 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T15:36:46.749454+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T15:36:46.750 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T15:36:46.750 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8600 8 365281 365273 0.3318 0 - 0.0299513 2026-02-23T15:36:46.755 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:36:46.853 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:36:46.852+0000 7f4224e27640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:36:46.853 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:36:46.852+0000 7f4224e27640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:36:46.964 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:36:46.964 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":327,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:36:45.161481+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:35:41.278300+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:33:55.531945+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:34:56.867874+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:36:47.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:36:47 trial068 ceph-mon[38805]: pgmap v5034: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:36:47.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:36:47 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/213258651' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:36:47.749 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8601 8 365281 365273 0.331761 0 - 0.0299513 2026-02-23T15:36:48.749 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8602 8 365281 365273 0.331723 0 - 0.0299513 2026-02-23T15:36:49.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:36:49 trial068 ceph-mon[38805]: pgmap v5035: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:36:49.749 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8603 8 365281 365273 0.331684 0 - 0.0299513 2026-02-23T15:36:50.001 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:36:50.117 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:36:50.217 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:36:50.216+0000 7ff2d0e12640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:36:50.218 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:36:50.216+0000 7ff2d0e12640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:36:50.327 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:36:50.327 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":327,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:36:45.161481+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:35:41.278300+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:33:55.531945+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:34:56.867874+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:36:50.750 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8604 8 365281 365273 0.331646 0 - 0.0299513 2026-02-23T15:36:51.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:36:51 trial068 ceph-mon[38805]: pgmap v5036: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:36:51.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:36:51 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/4199544866' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:36:51.750 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8605 8 365281 365273 0.331607 0 - 0.0299513 2026-02-23T15:36:52.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:36:52 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T15:36:52.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:36:52 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T15:36:52.750 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8606 8 365281 365273 0.331569 0 - 0.0299513 2026-02-23T15:36:53.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:36:53 trial068 ceph-mon[38805]: pgmap v5037: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:36:53.366 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:36:53.483 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:36:53.584 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:36:53.583+0000 7f34c84df640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:36:53.585 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:36:53.583+0000 7f34c84df640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:36:53.694 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:36:53.694 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":327,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:36:45.161481+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:35:41.278300+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:33:55.531945+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:34:56.867874+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:36:53.750 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8607 8 365281 365273 0.33153 0 - 0.0299513 2026-02-23T15:36:54.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:36:54 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/998024621' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:36:54.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:36:53 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:36:53] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:36:54.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:36:53 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:36:53.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:36:54.750 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8608 8 365281 365273 0.331492 0 - 0.0299513 2026-02-23T15:36:55.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:36:55 trial068 ceph-mon[38805]: pgmap v5038: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:36:55.750 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8609 8 365281 365273 0.331453 0 - 0.0299513 2026-02-23T15:36:56.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:36:56 trial068 ceph-mon[38805]: pgmap v5039: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:36:56.730 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:36:56.750 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8610 8 365281 365273 0.331415 0 - 0.0299513 2026-02-23T15:36:56.848 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:36:56.946 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:36:56.944+0000 7f60a8efa640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:36:56.946 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:36:56.944+0000 7f60a8efa640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:36:57.057 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:36:57.057 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":327,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:36:45.161481+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:35:41.278300+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:33:55.531945+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:34:56.867874+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:36:57.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:36:57 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1316178428' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:36:57.750 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8611 8 365281 365273 0.331376 0 - 0.0299513 2026-02-23T15:36:58.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:36:58 trial068 ceph-mon[38805]: pgmap v5040: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:36:58.750 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8612 8 365281 365273 0.331338 0 - 0.0299513 2026-02-23T15:36:59.750 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8613 8 365281 365273 0.331299 0 - 0.0299513 2026-02-23T15:37:00.095 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:37:00.214 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:37:00.314 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:37:00.312+0000 7ff5f5be2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:37:00.314 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:37:00.313+0000 7ff5f5be2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:37:00.424 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:37:00.424 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":327,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:36:45.161481+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:35:41.278300+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:33:55.531945+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:34:56.867874+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:37:00.424 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:37:00 trial068 ceph-mon[38805]: pgmap v5041: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:37:00.750 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8614 8 365281 365273 0.331261 0 - 0.0299513 2026-02-23T15:37:01.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:37:01 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/408865297' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:37:01.750 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8615 8 365281 365273 0.331222 0 - 0.0299513 2026-02-23T15:37:02.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:37:02 trial068 ceph-mon[38805]: pgmap v5042: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:37:02.750 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8616 8 365281 365273 0.331184 0 - 0.0299513 2026-02-23T15:37:03.460 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:37:03.578 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:37:03.675 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:37:03.673+0000 7f7a3bede640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:37:03.675 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:37:03.674+0000 7f7a3bede640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:37:03.750 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8617 8 365281 365273 0.331145 0 - 0.0299513 2026-02-23T15:37:03.784 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:37:03.784 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":327,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:36:45.161481+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:35:41.278300+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:33:55.531945+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:34:56.867874+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:37:04.093 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:37:03 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:37:03] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:37:04.264 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:37:03 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:37:03.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:37:04.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:37:04 trial068 ceph-mon[38805]: pgmap v5043: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:37:04.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:37:04 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3768422369' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:37:04.750 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8618 8 365281 365273 0.331107 0 - 0.0299513 2026-02-23T15:37:05.751 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8619 8 365281 365273 0.331068 0 - 0.0299513 2026-02-23T15:37:06.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:37:06 trial068 ceph-mon[38805]: pgmap v5044: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:37:06.751 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T15:37:06.750791+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T15:37:06.751 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T15:37:06.751 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8620 8 365281 365273 0.33103 0 - 0.0299513 2026-02-23T15:37:06.821 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:37:06.940 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:37:07.045 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:37:07.044+0000 7fa2b609d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:37:07.046 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:37:07.044+0000 7fa2b609d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:37:07.155 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:37:07.155 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":327,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:36:45.161481+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:35:41.278300+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:33:55.531945+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:34:56.867874+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:37:07.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:37:07 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2854765035' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:37:07.751 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8621 8 365281 365273 0.330992 0 - 0.0299513 2026-02-23T15:37:08.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:37:08 trial068 ceph-mon[38805]: pgmap v5045: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:37:08.751 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8622 8 365281 365273 0.330953 0 - 0.0299513 2026-02-23T15:37:09.751 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8623 8 365281 365273 0.330915 0 - 0.0299513 2026-02-23T15:37:10.209 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:37:10.329 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:37:10.431 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:37:10.429+0000 7f76070ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:37:10.431 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:37:10.429+0000 7f76070ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:37:10.540 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:37:10.540 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":327,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:36:45.161481+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:35:41.278300+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:33:55.531945+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:34:56.867874+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:37:10.540 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:37:10 trial068 ceph-mon[38805]: pgmap v5046: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:37:10.751 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8624 8 365281 365273 0.330877 0 - 0.0299513 2026-02-23T15:37:11.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:37:11 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3330774088' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:37:11.751 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8625 8 365281 365273 0.330838 0 - 0.0299513 2026-02-23T15:37:12.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:37:12 trial068 ceph-mon[38805]: pgmap v5047: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:37:12.751 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8626 8 365281 365273 0.3308 0 - 0.0299513 2026-02-23T15:37:13.576 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:37:13.695 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:37:13.751 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8627 8 365281 365273 0.330761 0 - 0.0299513 2026-02-23T15:37:13.791 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:37:13.790+0000 7f41a5fab640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:37:13.792 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:37:13.790+0000 7f41a5fab640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:37:13.903 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:37:13.903 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":327,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:36:45.161481+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:35:41.278300+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:33:55.531945+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:34:56.867874+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:37:14.277 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:37:13 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:37:13] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:37:14.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:37:13 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:37:13.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:37:14.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:37:14 trial068 ceph-mon[38805]: pgmap v5048: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:37:14.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:37:14 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/655055884' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:37:14.751 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8628 8 365281 365273 0.330723 0 - 0.0299513 2026-02-23T15:37:15.752 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8629 8 365281 365273 0.330685 0 - 0.0299513 2026-02-23T15:37:16.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:37:16 trial068 ceph-mon[38805]: pgmap v5049: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:37:16.751 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8630 8 365281 365273 0.330646 0 - 0.0299513 2026-02-23T15:37:16.943 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:37:17.060 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:37:17.172 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:37:17.170+0000 7f870147c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:37:17.172 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:37:17.170+0000 7f870147c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:37:17.284 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:37:17.284 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":327,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:36:45.161481+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:35:41.278300+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:33:55.531945+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:34:56.867874+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:37:17.752 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8631 8 365281 365273 0.330608 0 - 0.0299513 2026-02-23T15:37:18.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:37:18 trial068 ceph-mon[38805]: pgmap v5050: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:37:18.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:37:18 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1928920622' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:37:18.752 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8632 8 365281 365273 0.33057 0 - 0.0299513 2026-02-23T15:37:19.752 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8633 8 365281 365273 0.330532 0 - 0.0299513 2026-02-23T15:37:20.323 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:37:20.440 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:37:20.545 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:37:20.544+0000 7fb1e3843640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:37:20.545 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:37:20 trial068 ceph-mon[38805]: pgmap v5051: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:37:20.546 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:37:20.544+0000 7fb1e3843640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:37:20.655 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:37:20.655 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":327,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:36:45.161481+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:35:41.278300+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:33:55.531945+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:34:56.867874+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:37:20.752 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8634 8 365281 365273 0.330493 0 - 0.0299513 2026-02-23T15:37:21.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:37:21 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2880254989' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:37:21.752 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8635 8 365281 365273 0.330455 0 - 0.0299513 2026-02-23T15:37:22.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:37:22 trial068 ceph-mon[38805]: 5.e scrub starts 2026-02-23T15:37:22.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:37:22 trial068 ceph-mon[38805]: 5.e scrub ok 2026-02-23T15:37:22.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:37:22 trial068 ceph-mon[38805]: pgmap v5052: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:37:22.752 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8636 8 365281 365273 0.330417 0 - 0.0299513 2026-02-23T15:37:23.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:37:23 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2026-02-23T15:37:23.692 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:37:23.752 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8637 8 365281 365273 0.330379 0 - 0.0299513 2026-02-23T15:37:23.807 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:37:23.907 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:37:23.906+0000 7f2cf19e0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:37:23.908 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:37:23.906+0000 7f2cf19e0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:37:24.022 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:37:24.023 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":327,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:36:45.161481+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:35:41.278300+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:33:55.531945+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:34:56.867874+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:37:24.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:37:23 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:37:23] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:37:24.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:37:23 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:37:23.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:37:24.752 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8638 8 365281 365273 0.33034 0 - 0.0299513 2026-02-23T15:37:24.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:37:24 trial068 ceph-mon[38805]: pgmap v5053: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:37:24.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:37:24 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T15:37:24.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:37:24 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T15:37:24.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:37:24 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T15:37:24.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:37:24 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T15:37:24.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:37:24 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-23T15:37:24.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:37:24 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2026-02-23T15:37:24.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:37:24 trial068 ceph-mon[38805]: pgmap v5054: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:37:24.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:37:24 trial068 ceph-mon[38805]: pgmap v5055: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:37:24.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:37:24 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T15:37:24.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:37:24 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2026-02-23T15:37:24.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:37:24 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1100002855' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:37:25.752 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8639 8 365281 365273 0.330302 0 - 0.0299513 2026-02-23T15:37:26.494 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:37:26 trial068 ceph-mon[38805]: pgmap v5056: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:37:26.752 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T15:37:26.752439+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T15:37:26.752 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T15:37:26.753 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8640 8 365281 365273 0.330264 0 - 0.0299513 2026-02-23T15:37:27.060 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:37:27.181 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:37:27.279 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:37:27.277+0000 7fd7e6fb0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:37:27.279 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:37:27.278+0000 7fd7e6fb0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:37:27.388 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:37:27.388 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":327,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:36:45.161481+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:35:41.278300+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:33:55.531945+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:34:56.867874+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:37:27.752 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8641 8 365281 365273 0.330226 0 - 0.0299513 2026-02-23T15:37:27.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:37:27 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T15:37:27.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:37:27 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T15:37:27.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:37:27 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2904213032' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:37:28.752 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8642 8 365281 365273 0.330187 0 - 0.0299513 2026-02-23T15:37:28.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:37:28 trial068 ceph-mon[38805]: pgmap v5057: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:37:29.753 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8643 8 365281 365273 0.330149 0 - 0.0299513 2026-02-23T15:37:30.425 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:37:30.544 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:37:30.647 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:37:30.645+0000 7f9282ffe640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:37:30.647 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:37:30.645+0000 7f9282ffe640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:37:30.753 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8644 8 365281 365273 0.330111 0 - 0.0299513 2026-02-23T15:37:30.759 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:37:30.759 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":327,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:36:45.161481+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:35:41.278300+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:33:55.531945+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:34:56.867874+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:37:30.759 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:37:30 trial068 ceph-mon[38805]: pgmap v5058: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:37:31.753 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8645 8 365281 365273 0.330073 0 - 0.0299513 2026-02-23T15:37:32.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:37:31 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3755110480' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:37:32.753 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8646 8 365281 365273 0.330035 0 - 0.0299513 2026-02-23T15:37:33.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:37:32 trial068 ceph-mon[38805]: pgmap v5059: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:37:33.753 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8647 8 365281 365273 0.329996 0 - 0.0299513 2026-02-23T15:37:33.797 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:37:33.916 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:37:34.019 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:37:34.017+0000 7f18ce038640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:37:34.019 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:37:34.018+0000 7f18ce038640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:37:34.093 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:37:33 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:37:33] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:37:34.129 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:37:34.129 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":327,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:36:45.161481+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:35:41.278300+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:33:55.531945+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:34:56.867874+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:37:34.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:37:33 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:37:33.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:37:34.753 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8648 8 365281 365273 0.329958 0 - 0.0299513 2026-02-23T15:37:35.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:37:34 trial068 ceph-mon[38805]: pgmap v5060: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:37:35.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:37:34 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1737472972' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:37:35.753 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8649 8 365281 365273 0.32992 0 - 0.0299513 2026-02-23T15:37:36.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:37:36 trial068 ceph-mon[38805]: pgmap v5061: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:37:36.753 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8650 8 365281 365273 0.329882 0 - 0.0299513 2026-02-23T15:37:37.167 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:37:37.287 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:37:37.385 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:37:37.383+0000 7fec8d014640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:37:37.385 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:37:37.384+0000 7fec8d014640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:37:37.495 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:37:37.496 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":327,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:36:45.161481+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:35:41.278300+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:33:55.531945+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:34:56.867874+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:37:37.753 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8651 8 365281 365273 0.329844 0 - 0.0299513 2026-02-23T15:37:37.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:37:37 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1383806012' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:37:38.753 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8652 8 365281 365273 0.329806 0 - 0.0299513 2026-02-23T15:37:38.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:37:38 trial068 ceph-mon[38805]: pgmap v5062: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:37:39.753 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8653 8 365281 365273 0.329768 0 - 0.0299513 2026-02-23T15:37:40.534 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:37:40.653 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:37:40.750 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:37:40.748+0000 7ff630ef4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:37:40.750 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:37:40.748+0000 7ff630ef4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:37:40.753 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8654 8 365281 365273 0.32973 0 - 0.0299513 2026-02-23T15:37:40.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:37:40 trial068 ceph-mon[38805]: pgmap v5063: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:37:40.858 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:37:40.859 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":327,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:36:45.161481+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:35:41.278300+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:33:55.531945+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:34:56.867874+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:37:41.754 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8655 8 365281 365273 0.329691 0 - 0.0299513 2026-02-23T15:37:42.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:37:41 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3037844822' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:37:42.753 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8656 8 365281 365273 0.329653 0 - 0.0299513 2026-02-23T15:37:43.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:37:42 trial068 ceph-mon[38805]: pgmap v5064: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:37:43.754 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8657 8 365281 365273 0.329615 0 - 0.0299513 2026-02-23T15:37:43.896 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:37:44.014 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:37:44.114 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:37:44.113+0000 7f43249e0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:37:44.114 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:37:44.113+0000 7f43249e0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:37:44.223 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:37:44.223 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":327,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:36:45.161481+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:35:41.278300+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:33:55.531945+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:34:56.867874+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:37:44.223 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:37:43 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:37:43] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:37:44.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:37:43 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:37:43.972Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:37:44.754 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8658 8 365281 365273 0.329577 0 - 0.0299513 2026-02-23T15:37:45.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:37:44 trial068 ceph-mon[38805]: pgmap v5065: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:37:45.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:37:44 trial068 ceph-mon[38805]: 5.a scrub starts 2026-02-23T15:37:45.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:37:44 trial068 ceph-mon[38805]: 5.a scrub ok 2026-02-23T15:37:45.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:37:44 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3534837026' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:37:45.754 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8659 8 365281 365273 0.329539 0 - 0.0299513 2026-02-23T15:37:46.754 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T15:37:46.754039+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T15:37:46.754 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T15:37:46.754 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8660 8 365281 365273 0.329501 0 - 0.0299513 2026-02-23T15:37:47.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:37:46 trial068 ceph-mon[38805]: pgmap v5066: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:37:47.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:37:46 trial068 ceph-mon[38805]: 5.5 scrub starts 2026-02-23T15:37:47.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:37:46 trial068 ceph-mon[38805]: 5.5 scrub ok 2026-02-23T15:37:47.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:37:46 trial068 ceph-mon[38805]: 5.7 scrub starts 2026-02-23T15:37:47.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:37:46 trial068 ceph-mon[38805]: 5.7 scrub ok 2026-02-23T15:37:47.260 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:37:47.379 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:37:47.482 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:37:47.480+0000 7fa2d1bb8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:37:47.482 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:37:47.480+0000 7fa2d1bb8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:37:47.590 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:37:47.590 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":327,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:36:45.161481+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:35:41.278300+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:33:55.531945+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:34:56.867874+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:37:47.754 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8661 8 365281 365273 0.329463 0 - 0.0299513 2026-02-23T15:37:47.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:37:47 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2870736972' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:37:48.754 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8662 8 365281 365273 0.329425 0 - 0.0299513 2026-02-23T15:37:49.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:37:48 trial068 ceph-mon[38805]: pgmap v5067: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:37:49.754 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8663 8 365281 365273 0.329387 0 - 0.0299513 2026-02-23T15:37:50.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:37:49 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T15:37:50.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:37:49 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T15:37:50.628 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:37:50.744 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:37:50.754 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8664 8 365281 365273 0.329349 0 - 0.0299513 2026-02-23T15:37:50.843 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:37:50.841+0000 7f0845b29640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:37:50.843 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:37:50.842+0000 7f0845b29640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:37:50.952 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:37:50.953 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":327,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:36:45.161481+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:35:41.278300+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:33:55.531945+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:34:56.867874+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:37:50.953 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:37:50 trial068 ceph-mon[38805]: pgmap v5068: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:37:51.754 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8665 8 365281 365273 0.329311 0 - 0.0299513 2026-02-23T15:37:52.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:37:51 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1561479983' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:37:52.754 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8666 8 365281 365273 0.329273 0 - 0.0299513 2026-02-23T15:37:53.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:37:52 trial068 ceph-mon[38805]: pgmap v5069: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:37:53.754 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8667 8 365281 365273 0.329235 0 - 0.0299513 2026-02-23T15:37:53.989 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:37:54.104 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:37:54.211 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:37:54.209+0000 7fdc54bd2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:37:54.211 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:37:54.210+0000 7fdc54bd2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:37:54.319 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:37:54.320 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":327,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:36:45.161481+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:35:41.278300+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:33:55.531945+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:34:56.867874+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:37:54.320 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:37:53 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:37:53] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:37:54.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:37:53 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:37:53.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:37:54.755 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8668 8 365281 365273 0.329197 0 - 0.0299513 2026-02-23T15:37:55.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:37:54 trial068 ceph-mon[38805]: pgmap v5070: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:37:55.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:37:54 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1744921575' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:37:55.755 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8669 8 365281 365273 0.329159 0 - 0.0299513 2026-02-23T15:37:56.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:37:56 trial068 ceph-mon[38805]: pgmap v5071: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:37:56.755 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8670 8 365281 365273 0.329121 0 - 0.0299513 2026-02-23T15:37:57.356 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:37:57.474 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:37:57.577 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:37:57.575+0000 7f64dadab640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:37:57.577 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:37:57.575+0000 7f64dadab640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:37:57.686 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:37:57.687 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":327,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:36:45.161481+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:35:41.278300+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:33:55.531945+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:34:56.867874+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:37:57.755 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8671 8 365281 365273 0.329083 0 - 0.0299513 2026-02-23T15:37:58.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:37:57 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/913360792' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:37:58.755 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8672 8 365281 365273 0.329045 0 - 0.0299513 2026-02-23T15:37:59.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:37:58 trial068 ceph-mon[38805]: pgmap v5072: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:37:59.755 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8673 8 365281 365273 0.329007 0 - 0.0299513 2026-02-23T15:38:00.723 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:38:00.755 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8674 8 365281 365273 0.328969 0 - 0.0299513 2026-02-23T15:38:00.842 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:38:00.945 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:38:00.943+0000 7f511320d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:38:00.945 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:38:00.943+0000 7f511320d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:38:01.055 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:38:01.055 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":327,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:36:45.161481+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:35:41.278300+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:33:55.531945+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:34:56.867874+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:38:01.055 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:38:00 trial068 ceph-mon[38805]: pgmap v5073: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:38:01.755 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8675 8 365281 365273 0.328931 0 - 0.0299513 2026-02-23T15:38:02.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:38:01 trial068 ceph-mon[38805]: 5.8 scrub starts 2026-02-23T15:38:02.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:38:01 trial068 ceph-mon[38805]: 5.8 scrub ok 2026-02-23T15:38:02.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:38:01 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2843982005' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:38:02.755 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8676 8 365281 365273 0.328893 0 - 0.0299513 2026-02-23T15:38:03.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:38:02 trial068 ceph-mon[38805]: pgmap v5074: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:38:03.755 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8677 8 365281 365273 0.328855 0 - 0.0299513 2026-02-23T15:38:04.093 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:38:04.210 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:38:04.231 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:38:03 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:38:03] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:38:04.265 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:38:03 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:38:03.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:38:04.303 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:38:04.302+0000 7fc52135f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:38:04.304 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:38:04.302+0000 7fc52135f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:38:04.414 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:38:04.414 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":327,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:36:45.161481+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:35:41.278300+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:33:55.531945+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:34:56.867874+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:38:04.755 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8678 8 365281 365273 0.328818 0 - 0.0299513 2026-02-23T15:38:05.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:38:04 trial068 ceph-mon[38805]: pgmap v5075: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:38:05.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:38:04 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3417732054' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:38:05.755 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8679 8 365281 365273 0.32878 0 - 0.0299513 2026-02-23T15:38:06.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:38:06 trial068 ceph-mon[38805]: pgmap v5076: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:38:06.755 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T15:38:06.755538+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T15:38:06.755 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T15:38:06.756 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8680 8 365281 365273 0.328742 0 - 0.0299513 2026-02-23T15:38:07.450 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:38:07.567 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:38:07.665 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:38:07.663+0000 7fe10ae94640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:38:07.665 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:38:07.663+0000 7fe10ae94640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:38:07.755 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8681 8 365281 365273 0.328704 0 - 0.0299513 2026-02-23T15:38:07.774 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:38:07.775 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":327,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:36:45.161481+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:35:41.278300+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:33:55.531945+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:34:56.867874+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:38:08.756 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8682 8 365281 365273 0.328666 0 - 0.0299513 2026-02-23T15:38:09.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:38:08 trial068 ceph-mon[38805]: pgmap v5077: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:38:09.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:38:08 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1650847598' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:38:09.756 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8683 8 365281 365273 0.328628 0 - 0.0299513 2026-02-23T15:38:10.756 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8684 8 365281 365273 0.32859 0 - 0.0299513 2026-02-23T15:38:10.813 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:38:10.929 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:38:10.973 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:38:10 trial068 ceph-mon[38805]: pgmap v5078: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:38:11.029 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:38:11.027+0000 7f3f98492640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:38:11.029 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:38:11.028+0000 7f3f98492640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:38:11.139 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:38:11.139 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":327,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:36:45.161481+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:35:41.278300+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:33:55.531945+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:34:56.867874+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:38:11.756 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8685 8 365281 365273 0.328553 0 - 0.0299513 2026-02-23T15:38:12.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:38:11 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/577993374' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:38:12.756 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8686 8 365281 365273 0.328515 0 - 0.0299513 2026-02-23T15:38:13.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:38:12 trial068 ceph-mon[38805]: pgmap v5079: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:38:13.756 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8687 8 365281 365273 0.328477 0 - 0.0299513 2026-02-23T15:38:13.974 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:38:13 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T15:38:13.974 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:38:13 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T15:38:14.176 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:38:14.294 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:38:14.316 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:38:13 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:38:13] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:38:14.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:38:13 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:38:13.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:38:14.393 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:38:14.392+0000 7ffa5bc6c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:38:14.394 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:38:14.392+0000 7ffa5bc6c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:38:14.502 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:38:14.503 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":327,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:36:45.161481+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:35:41.278300+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:33:55.531945+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:34:56.867874+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:38:14.756 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8688 8 365281 365273 0.328439 0 - 0.0299513 2026-02-23T15:38:15.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:38:14 trial068 ceph-mon[38805]: pgmap v5080: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:38:15.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:38:14 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1457286949' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:38:15.756 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8689 8 365281 365273 0.328401 0 - 0.0299513 2026-02-23T15:38:16.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:38:16 trial068 ceph-mon[38805]: pgmap v5081: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:38:16.756 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8690 8 365281 365273 0.328364 0 - 0.0299513 2026-02-23T15:38:17.540 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:38:17.660 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:38:17.757 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8691 8 365281 365273 0.328326 0 - 0.0299513 2026-02-23T15:38:17.760 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:38:17.758+0000 7f5ea1be0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:38:17.760 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:38:17.758+0000 7f5ea1be0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:38:17.869 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:38:17.870 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":327,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:36:45.161481+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:35:41.278300+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:33:55.531945+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:34:56.867874+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:38:18.757 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8692 8 365281 365273 0.328288 0 - 0.0299513 2026-02-23T15:38:19.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:38:18 trial068 ceph-mon[38805]: pgmap v5082: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:38:19.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:38:18 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1034428863' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:38:19.757 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8693 8 365281 365273 0.32825 0 - 0.0299513 2026-02-23T15:38:20.757 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8694 8 365281 365273 0.328212 0 - 0.0299513 2026-02-23T15:38:20.907 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:38:21.022 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:38:21.045 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:38:20 trial068 ceph-mon[38805]: pgmap v5083: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:38:21.127 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:38:21.126+0000 7fdb45457640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:38:21.128 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:38:21.126+0000 7fdb45457640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:38:21.237 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:38:21.237 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":327,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:36:45.161481+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:35:41.278300+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:33:55.531945+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:34:56.867874+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:38:21.757 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8695 8 365281 365273 0.328175 0 - 0.0299513 2026-02-23T15:38:22.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:38:21 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3497041436' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:38:22.757 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8696 8 365281 365273 0.328137 0 - 0.0299513 2026-02-23T15:38:23.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:38:22 trial068 ceph-mon[38805]: pgmap v5084: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:38:23.757 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8697 8 365281 365273 0.328099 0 - 0.0299513 2026-02-23T15:38:23.973 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:38:23 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2026-02-23T15:38:24.187 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:38:23 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:38:23.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:38:24.274 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:38:24.297 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:38:23 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:38:23] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:38:24.391 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:38:24.502 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:38:24.500+0000 7f2b6a3b9640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:38:24.502 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:38:24.500+0000 7f2b6a3b9640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:38:24.615 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:38:24.615 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":327,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:36:45.161481+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:35:41.278300+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:33:55.531945+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:34:56.867874+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:38:24.757 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8698 8 365281 365273 0.328062 0 - 0.0299513 2026-02-23T15:38:25.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:38:24 trial068 ceph-mon[38805]: pgmap v5085: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:38:25.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:38:24 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T15:38:25.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:38:24 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T15:38:25.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:38:24 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T15:38:25.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:38:24 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T15:38:25.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:38:24 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T15:38:25.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:38:24 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T15:38:25.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:38:24 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T15:38:25.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:38:24 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T15:38:25.095 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:38:24 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/113645068' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:38:25.757 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8699 8 365281 365273 0.328024 0 - 0.0299513 2026-02-23T15:38:26.748 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:38:26 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T15:38:26.748 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:38:26 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T15:38:26.749 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:38:26 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T15:38:26.749 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:38:26 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T15:38:26.749 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:38:26 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-23T15:38:26.749 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:38:26 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2026-02-23T15:38:26.749 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:38:26 trial068 ceph-mon[38805]: pgmap v5086: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:38:26.749 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:38:26 trial068 ceph-mon[38805]: pgmap v5087: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:38:26.749 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:38:26 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T15:38:26.749 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:38:26 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2026-02-23T15:38:26.757 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T15:38:26.757256+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T15:38:26.757 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T15:38:26.757 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8700 8 365281 365273 0.327986 0 - 0.0299513 2026-02-23T15:38:27.655 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:38:27.757 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8701 8 365281 365273 0.327948 0 - 0.0299513 2026-02-23T15:38:27.773 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:38:27.872 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:38:27.870+0000 7f8df950f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:38:27.872 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:38:27.870+0000 7f8df950f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:38:27.980 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:38:27.981 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":327,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:36:45.161481+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:35:41.278300+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:33:55.531945+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:34:56.867874+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:38:28.757 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8702 8 365281 365273 0.327911 0 - 0.0299513 2026-02-23T15:38:28.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:38:28 trial068 ceph-mon[38805]: pgmap v5088: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:38:28.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:38:28 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/4263286622' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:38:29.757 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8703 8 365281 365273 0.327873 0 - 0.0299513 2026-02-23T15:38:30.757 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8704 8 365281 365273 0.327835 0 - 0.0299513 2026-02-23T15:38:30.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:38:30 trial068 ceph-mon[38805]: pgmap v5089: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:38:31.019 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:38:31.139 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:38:31.240 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:38:31.238+0000 7f7df765e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:38:31.240 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:38:31.239+0000 7f7df765e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:38:31.350 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:38:31.350 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":327,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:36:45.161481+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:35:41.278300+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:33:55.531945+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:34:56.867874+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:38:31.758 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8705 8 365281 365273 0.327798 0 - 0.0299513 2026-02-23T15:38:31.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:38:31 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2156275897' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:38:32.758 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8706 8 365281 365273 0.32776 0 - 0.0299513 2026-02-23T15:38:32.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:38:32 trial068 ceph-mon[38805]: pgmap v5090: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:38:33.758 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8707 8 365281 365273 0.327722 0 - 0.0299513 2026-02-23T15:38:34.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:38:33 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:38:33] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:38:34.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:38:33 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:38:33.974Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:38:34.388 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:38:34.506 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:38:34.612 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:38:34.611+0000 7f1927deb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:38:34.613 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:38:34.611+0000 7f1927deb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:38:34.724 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:38:34.725 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":327,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:36:45.161481+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:35:41.278300+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:33:55.531945+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:34:56.867874+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:38:34.725 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:38:34 trial068 ceph-mon[38805]: pgmap v5091: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:38:34.758 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8708 8 365281 365273 0.327685 0 - 0.0299513 2026-02-23T15:38:35.758 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8709 8 365281 365273 0.327647 0 - 0.0299513 2026-02-23T15:38:35.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:38:35 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1816873771' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:38:36.758 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8710 8 365281 365273 0.32761 0 - 0.0299513 2026-02-23T15:38:36.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:38:36 trial068 ceph-mon[38805]: pgmap v5092: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:38:37.758 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8711 8 365281 365273 0.327572 0 - 0.0299513 2026-02-23T15:38:37.764 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:38:37.884 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:38:37.980 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:38:37.978+0000 7facd8e37640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:38:37.980 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:38:37.979+0000 7facd8e37640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:38:38.090 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:38:38.090 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":327,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:36:45.161481+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:35:41.278300+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:33:55.531945+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:34:56.867874+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:38:38.758 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8712 8 365281 365273 0.327534 0 - 0.0299513 2026-02-23T15:38:38.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:38:38 trial068 ceph-mon[38805]: pgmap v5093: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:38:38.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:38:38 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2310677331' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:38:39.758 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8713 8 365281 365273 0.327497 0 - 0.0299513 2026-02-23T15:38:40.758 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8714 8 365281 365273 0.327459 0 - 0.0299513 2026-02-23T15:38:40.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:38:40 trial068 ceph-mon[38805]: pgmap v5094: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:38:41.128 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:38:41.246 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:38:41.344 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:38:41.342+0000 7f8f96996640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:38:41.344 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:38:41.343+0000 7f8f96996640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:38:41.455 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:38:41.456 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":327,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:36:45.161481+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:35:41.278300+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:33:55.531945+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:34:56.867874+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:38:41.758 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8715 8 365281 365273 0.327422 0 - 0.0299513 2026-02-23T15:38:41.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:38:41 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2175568957' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:38:42.758 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8716 8 365281 365273 0.327384 0 - 0.0299513 2026-02-23T15:38:42.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:38:42 trial068 ceph-mon[38805]: pgmap v5095: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:38:43.758 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8717 8 365281 365273 0.327346 0 - 0.0299513 2026-02-23T15:38:44.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:38:43 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:38:43] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:38:44.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:38:43 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:38:43.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:38:44.491 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:38:44.609 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:38:44.711 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:38:44.709+0000 7f966ff22640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:38:44.711 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:38:44.710+0000 7f966ff22640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:38:44.758 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8718 8 365281 365273 0.327309 0 - 0.0299513 2026-02-23T15:38:44.820 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:38:44.820 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":327,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:36:45.161481+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:35:41.278300+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:33:55.531945+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:34:56.867874+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:38:44.820 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:38:44 trial068 ceph-mon[38805]: pgmap v5096: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:38:45.758 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8719 8 365281 365273 0.327271 0 - 0.0299513 2026-02-23T15:38:45.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:38:45 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1561431114' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:38:46.758 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T15:38:46.758697+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T15:38:46.759 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T15:38:46.759 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8720 8 365281 365273 0.327234 0 - 0.0299513 2026-02-23T15:38:46.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:38:46 trial068 ceph-mon[38805]: pgmap v5097: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:38:47.759 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8721 8 365281 365273 0.327196 0 - 0.0299513 2026-02-23T15:38:47.857 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:38:47.976 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:38:48.076 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:38:48.074+0000 7fa5585a1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:38:48.076 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:38:48.075+0000 7fa5585a1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:38:48.185 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:38:48.186 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":327,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:36:45.161481+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:35:41.278300+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:33:55.531945+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:34:56.867874+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:38:48.759 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8722 8 365281 365273 0.327159 0 - 0.0299513 2026-02-23T15:38:48.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:38:48 trial068 ceph-mon[38805]: pgmap v5098: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:38:48.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:38:48 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3131981164' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:38:49.759 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8723 8 365281 365273 0.327121 0 - 0.0299513 2026-02-23T15:38:49.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:38:49 trial068 ceph-mon[38805]: 5.e scrub starts 2026-02-23T15:38:49.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:38:49 trial068 ceph-mon[38805]: 5.e scrub ok 2026-02-23T15:38:50.759 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8724 8 365281 365273 0.327084 0 - 0.0299513 2026-02-23T15:38:50.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:38:50 trial068 ceph-mon[38805]: pgmap v5099: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:38:51.223 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:38:51.338 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:38:51.438 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:38:51.437+0000 7f0672446640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:38:51.438 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:38:51.437+0000 7f0672446640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:38:51.549 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:38:51.549 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":327,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:36:45.161481+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:35:41.278300+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:33:55.531945+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:34:56.867874+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:38:51.759 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8725 8 365281 365273 0.327046 0 - 0.0299513 2026-02-23T15:38:52.759 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8726 8 365281 365273 0.327009 0 - 0.0299513 2026-02-23T15:38:52.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:38:52 trial068 ceph-mon[38805]: pgmap v5100: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:38:52.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:38:52 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3544861748' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:38:53.759 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8727 8 365281 365273 0.326971 0 - 0.0299513 2026-02-23T15:38:54.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:38:53 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:38:53] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:38:54.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:38:53 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:38:53.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:38:54.587 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:38:54.705 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:38:54.759 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8728 8 365281 365273 0.326934 0 - 0.0299513 2026-02-23T15:38:54.812 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:38:54.810+0000 7f99067c6640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:38:54.812 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:38:54.811+0000 7f99067c6640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:38:54.812 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:38:54 trial068 ceph-mon[38805]: pgmap v5101: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:38:54.920 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:38:54.920 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":327,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:36:45.161481+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:35:41.278300+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:33:55.531945+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:34:56.867874+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:38:55.759 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8729 8 365281 365273 0.326896 0 - 0.0299513 2026-02-23T15:38:55.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:38:55 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2905773042' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:38:56.759 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8730 8 365281 365273 0.326859 0 - 0.0299513 2026-02-23T15:38:56.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:38:56 trial068 ceph-mon[38805]: pgmap v5102: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:38:56.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:38:56 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T15:38:56.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:38:56 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T15:38:57.760 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8731 8 365281 365273 0.326822 0 - 0.0299513 2026-02-23T15:38:57.956 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:38:58.075 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:38:58.179 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:38:58.177+0000 7fcefcb9a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:38:58.179 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:38:58.178+0000 7fcefcb9a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:38:58.290 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:38:58.290 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":327,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:36:45.161481+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:35:41.278300+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:33:55.531945+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:34:56.867874+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:38:58.760 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8732 8 365281 365273 0.326784 0 - 0.0299513 2026-02-23T15:38:58.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:38:58 trial068 ceph-mon[38805]: pgmap v5103: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:38:58.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:38:58 trial068 ceph-mon[38805]: 5.a scrub starts 2026-02-23T15:38:58.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:38:58 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3323902214' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:38:59.760 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8733 8 365281 365273 0.326747 0 - 0.0299513 2026-02-23T15:38:59.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:38:59 trial068 ceph-mon[38805]: 5.a scrub ok 2026-02-23T15:39:00.760 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8734 8 365281 365273 0.326709 0 - 0.0299513 2026-02-23T15:39:00.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:39:00 trial068 ceph-mon[38805]: pgmap v5104: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:39:01.328 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:39:01.445 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:39:01.544 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:39:01.542+0000 7f6920666640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:39:01.544 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:39:01.542+0000 7f6920666640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:39:01.654 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:39:01.654 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":327,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:36:45.161481+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:35:41.278300+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:33:55.531945+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:34:56.867874+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:39:01.760 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8735 8 365281 365273 0.326672 0 - 0.0299513 2026-02-23T15:39:02.760 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8736 8 365281 365273 0.326634 0 - 0.0299513 2026-02-23T15:39:02.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:39:02 trial068 ceph-mon[38805]: pgmap v5105: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:39:02.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:39:02 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2625882565' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:39:03.760 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8737 8 365281 365273 0.326597 0 - 0.0299513 2026-02-23T15:39:04.264 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:39:03 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:39:03.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:39:04.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:39:03 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:39:03] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:39:04.690 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:39:04.760 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8738 8 365281 365273 0.32656 0 - 0.0299513 2026-02-23T15:39:04.809 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:39:04.831 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:39:04 trial068 ceph-mon[38805]: pgmap v5106: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:39:04.907 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:39:04.905+0000 7f4ed3b3c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:39:04.907 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:39:04.906+0000 7f4ed3b3c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:39:05.016 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:39:05.016 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":327,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:36:45.161481+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:35:41.278300+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:33:55.531945+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:34:56.867874+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:39:05.760 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8739 8 365281 365273 0.326522 0 - 0.0299513 2026-02-23T15:39:05.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:39:05 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3889572208' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:39:06.760 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T15:39:06.760451+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T15:39:06.761 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T15:39:06.761 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8740 8 365281 365273 0.326485 0 - 0.0299513 2026-02-23T15:39:06.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:39:06 trial068 ceph-mon[38805]: pgmap v5107: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:39:07.760 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8741 8 365281 365273 0.326448 0 - 0.0299513 2026-02-23T15:39:08.054 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:39:08.173 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:39:08.277 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:39:08.275+0000 7f0a92df8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:39:08.277 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:39:08.275+0000 7f0a92df8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:39:08.387 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:39:08.388 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":327,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:36:45.161481+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:35:41.278300+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:33:55.531945+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:34:56.867874+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:39:08.761 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8742 8 365281 365273 0.32641 0 - 0.0299513 2026-02-23T15:39:08.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:39:08 trial068 ceph-mon[38805]: pgmap v5108: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:39:08.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:39:08 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/341452651' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:39:09.761 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8743 8 365281 365273 0.326373 0 - 0.0299513 2026-02-23T15:39:10.761 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8744 8 365281 365273 0.326336 0 - 0.0299513 2026-02-23T15:39:10.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:39:10 trial068 ceph-mon[38805]: pgmap v5109: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:39:11.425 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:39:11.541 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:39:11.652 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:39:11.651+0000 7f32d1844640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:39:11.653 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:39:11.651+0000 7f32d1844640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:39:11.761 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8745 8 365281 365273 0.326298 0 - 0.0299513 2026-02-23T15:39:11.764 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:39:11.764 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":327,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:36:45.161481+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:35:41.278300+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:33:55.531945+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:34:56.867874+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:39:12.761 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8746 8 365281 365273 0.326261 0 - 0.0299513 2026-02-23T15:39:12.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:39:12 trial068 ceph-mon[38805]: pgmap v5110: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:39:12.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:39:12 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1971770188' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:39:13.761 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8747 8 365281 365273 0.326224 0 - 0.0299513 2026-02-23T15:39:14.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:39:13 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:39:13] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:39:14.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:39:13 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:39:13.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:39:14.761 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8748 8 365281 365273 0.326186 0 - 0.0299513 2026-02-23T15:39:14.801 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:39:14.826 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:39:14 trial068 ceph-mon[38805]: pgmap v5111: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:39:14.920 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:39:15.016 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:39:15.014+0000 7fe0f5e88640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:39:15.016 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:39:15.015+0000 7fe0f5e88640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:39:15.125 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:39:15.125 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":327,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:36:45.161481+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:35:41.278300+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:33:55.531945+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:34:56.867874+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:39:15.761 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8749 8 365281 365273 0.326149 0 - 0.0299513 2026-02-23T15:39:15.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:39:15 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T15:39:15.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:39:15 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T15:39:15.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:39:15 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2289711013' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:39:16.761 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8750 8 365281 365273 0.326112 0 - 0.0299513 2026-02-23T15:39:16.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:39:16 trial068 ceph-mon[38805]: pgmap v5112: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:39:17.761 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8751 8 365281 365273 0.326075 0 - 0.0299513 2026-02-23T15:39:17.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:39:17 trial068 ceph-mon[38805]: 5.5 scrub starts 2026-02-23T15:39:17.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:39:17 trial068 ceph-mon[38805]: 5.5 scrub ok 2026-02-23T15:39:18.163 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:39:18.279 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:39:18.379 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:39:18.377+0000 7f71f07f8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:39:18.379 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:39:18.377+0000 7f71f07f8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:39:18.488 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:39:18.488 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":327,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:36:45.161481+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:35:41.278300+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:33:55.531945+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:34:56.867874+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:39:18.761 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8752 8 365281 365273 0.326037 0 - 0.0299513 2026-02-23T15:39:18.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:39:18 trial068 ceph-mon[38805]: pgmap v5113: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:39:18.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:39:18 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1699857289' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:39:19.761 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8753 8 365281 365273 0.326 0 - 0.0299513 2026-02-23T15:39:20.762 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8754 8 365281 365273 0.325963 0 - 0.0299513 2026-02-23T15:39:20.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:39:20 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T15:39:20.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:39:20 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T15:39:20.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:39:20 trial068 ceph-mon[38805]: pgmap v5114: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:39:21.524 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:39:21.642 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:39:21.741 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:39:21.739+0000 7f273402d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:39:21.741 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:39:21.739+0000 7f273402d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:39:21.761 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8755 8 365281 365273 0.325926 0 - 0.0299513 2026-02-23T15:39:21.852 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:39:21.852 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":327,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:36:45.161481+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:35:41.278300+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:33:55.531945+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:34:56.867874+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:39:22.762 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8756 8 365281 365273 0.325888 0 - 0.0299513 2026-02-23T15:39:22.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:39:22 trial068 ceph-mon[38805]: pgmap v5115: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:39:22.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:39:22 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2139394959' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:39:23.762 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8757 8 365281 365273 0.325851 0 - 0.0299513 2026-02-23T15:39:23.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:39:23 trial068 ceph-mon[38805]: osdmap e328: 4 total, 3 up, 4 in 2026-02-23T15:39:24.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:39:23 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:39:23] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:39:24.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:39:23 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:39:23.974Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:39:24.762 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8758 8 365281 365273 0.325814 0 - 0.0299513 2026-02-23T15:39:24.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:39:24 trial068 ceph-mon[38805]: pgmap v5117: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:39:24.889 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:39:25.004 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:39:25.103 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:39:25.102+0000 7f4b6cda1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:39:25.103 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:39:25.102+0000 7f4b6cda1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:39:25.213 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:39:25.214 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":329,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:39:25.180542+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:35:41.278300+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:39:14.873799+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:34:56.867874+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:39:25.762 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8759 8 365281 365273 0.325777 0 - 0.0299513 2026-02-23T15:39:26.495 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:39:26 trial068 ceph-mon[38805]: 5.7 scrub starts 2026-02-23T15:39:26.495 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:39:26 trial068 ceph-mon[38805]: 5.7 scrub ok 2026-02-23T15:39:26.495 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:39:26 trial068 ceph-mon[38805]: osdmap e329: 4 total, 3 up, 4 in 2026-02-23T15:39:26.496 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:39:26 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3236380550' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:39:26.496 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:39:26 trial068 ceph-mon[38805]: pgmap v5119: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:39:26.496 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:39:26 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2026-02-23T15:39:26.496 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:39:26 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-23T15:39:26.496 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:39:26 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2026-02-23T15:39:26.496 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:39:26 trial068 ceph-mon[38805]: pgmap v5120: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:39:26.496 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:39:26 trial068 ceph-mon[38805]: pgmap v5121: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:39:26.496 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:39:26 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T15:39:26.496 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:39:26 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2026-02-23T15:39:26.762 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T15:39:26.762033+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T15:39:26.762 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T15:39:26.762 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8760 8 365281 365273 0.32574 0 - 0.0299513 2026-02-23T15:39:27.762 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8761 8 365281 365273 0.325702 0 - 0.0299513 2026-02-23T15:39:28.252 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:39:28.368 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:39:28.467 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:39:28.465+0000 7f80f356e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:39:28.468 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:39:28.466+0000 7f80f356e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:39:28.578 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:39:28.578 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":329,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:39:25.180542+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:35:41.278300+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:39:14.873799+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:34:56.867874+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:39:28.762 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8762 8 365281 365273 0.325665 0 - 0.0299513 2026-02-23T15:39:28.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:39:28 trial068 ceph-mon[38805]: pgmap v5122: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:39:28.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:39:28 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/669977348' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:39:29.762 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8763 8 365281 365273 0.325628 0 - 0.0299513 2026-02-23T15:39:30.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:39:29 trial068 ceph-mon[38805]: 5.8 scrub starts 2026-02-23T15:39:30.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:39:29 trial068 ceph-mon[38805]: 5.8 scrub ok 2026-02-23T15:39:30.762 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8764 8 365281 365273 0.325591 0 - 0.0299513 2026-02-23T15:39:31.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:39:30 trial068 ceph-mon[38805]: pgmap v5123: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:39:31.617 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:39:31.735 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:39:31.762 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8765 8 365281 365273 0.325554 0 - 0.0299513 2026-02-23T15:39:31.840 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:39:31.839+0000 7f130ba23640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:39:31.841 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:39:31.839+0000 7f130ba23640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:39:31.950 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:39:31.951 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":329,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:39:25.180542+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:35:41.278300+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:39:14.873799+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:34:56.867874+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:39:32.762 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8766 8 365281 365273 0.325517 0 - 0.0299513 2026-02-23T15:39:33.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:39:32 trial068 ceph-mon[38805]: pgmap v5124: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:39:33.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:39:32 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3336378323' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:39:33.762 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8767 8 365281 365273 0.32548 0 - 0.0299513 2026-02-23T15:39:34.265 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:39:33 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:39:33.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:39:34.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:39:33 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:39:33] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:39:34.762 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8768 8 365281 365273 0.325442 0 - 0.0299513 2026-02-23T15:39:34.988 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:39:35.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:39:34 trial068 ceph-mon[38805]: pgmap v5125: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:39:35.107 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:39:35.208 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:39:35.206+0000 7f148a180640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:39:35.208 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:39:35.206+0000 7f148a180640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:39:35.318 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:39:35.318 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":329,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:39:25.180542+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:35:41.278300+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:39:14.873799+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:34:56.867874+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:39:35.763 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8769 8 365281 365273 0.325405 0 - 0.0299513 2026-02-23T15:39:36.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:39:36 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2265722424' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:39:36.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:39:36 trial068 ceph-mon[38805]: pgmap v5126: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:39:36.763 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8770 8 365281 365273 0.325368 0 - 0.0299513 2026-02-23T15:39:37.763 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8771 8 365281 365273 0.325331 0 - 0.0299513 2026-02-23T15:39:38.357 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:39:38.483 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:39:38.584 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:39:38.583+0000 7fd1f4171640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:39:38.584 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:39:38.583+0000 7fd1f4171640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:39:38.698 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:39:38.698 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":329,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:39:25.180542+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:35:41.278300+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:39:14.873799+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:34:56.867874+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:39:38.763 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8772 8 365281 365273 0.325294 0 - 0.0299513 2026-02-23T15:39:39.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:39:38 trial068 ceph-mon[38805]: pgmap v5127: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:39:39.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:39:38 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1694336218' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:39:39.763 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8773 8 365281 365273 0.325257 0 - 0.0299513 2026-02-23T15:39:40.763 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8774 8 365281 365273 0.32522 0 - 0.0299513 2026-02-23T15:39:41.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:39:40 trial068 ceph-mon[38805]: pgmap v5128: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:39:41.735 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:39:41.763 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8775 8 365281 365273 0.325183 0 - 0.0299513 2026-02-23T15:39:41.853 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:39:41.961 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:39:41.960+0000 7f1b7fe86640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:39:41.962 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:39:41.960+0000 7f1b7fe86640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:39:42.073 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:39:42.074 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":329,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:39:25.180542+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:35:41.278300+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:39:14.873799+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:34:56.867874+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:39:42.763 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8776 8 365281 365273 0.325146 0 - 0.0299513 2026-02-23T15:39:43.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:39:42 trial068 ceph-mon[38805]: pgmap v5129: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:39:43.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:39:42 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3562708302' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:39:43.763 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8777 8 365281 365273 0.325109 0 - 0.0299513 2026-02-23T15:39:44.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:39:43 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:39:43] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:39:44.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:39:43 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:39:43.972Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:39:44.763 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8778 8 365281 365273 0.325072 0 - 0.0299513 2026-02-23T15:39:45.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:39:44 trial068 ceph-mon[38805]: pgmap v5130: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:39:45.127 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:39:45.247 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:39:45.346 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:39:45.345+0000 7ffb43bd7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:39:45.347 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:39:45.345+0000 7ffb43bd7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:39:45.458 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:39:45.458 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":329,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:39:25.180542+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:35:41.278300+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:39:14.873799+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:34:56.867874+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:39:45.763 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8779 8 365281 365273 0.325035 0 - 0.0299513 2026-02-23T15:39:46.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:39:45 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2640009863' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:39:46.763 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T15:39:46.763433+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T15:39:46.763 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T15:39:46.764 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8780 8 365281 365273 0.324998 0 - 0.0299513 2026-02-23T15:39:47.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:39:46 trial068 ceph-mon[38805]: pgmap v5131: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:39:47.763 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8781 8 365281 365273 0.324961 0 - 0.0299513 2026-02-23T15:39:48.496 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:39:48.616 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:39:48.715 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:39:48.714+0000 7f206bbb7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:39:48.716 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:39:48.714+0000 7f206bbb7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:39:48.763 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8782 8 365281 365273 0.324924 0 - 0.0299513 2026-02-23T15:39:48.824 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:39:48.824 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":329,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:39:25.180542+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:35:41.278300+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:39:14.873799+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:34:56.867874+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:39:49.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:39:48 trial068 ceph-mon[38805]: pgmap v5132: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:39:49.764 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8783 8 365281 365273 0.324887 0 - 0.0299513 2026-02-23T15:39:50.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:39:49 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2624075985' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:39:50.764 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8784 8 365281 365273 0.32485 0 - 0.0299513 2026-02-23T15:39:51.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:39:50 trial068 ceph-mon[38805]: pgmap v5133: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:39:51.764 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8785 8 365281 365273 0.324813 0 - 0.0299513 2026-02-23T15:39:51.861 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:39:51.981 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:39:52.085 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:39:52.084+0000 7fe44c1a8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:39:52.086 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:39:52.084+0000 7fe44c1a8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:39:52.199 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:39:52.199 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":329,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:39:25.180542+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:35:41.278300+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:39:14.873799+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:34:56.867874+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:39:52.764 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8786 8 365281 365273 0.324776 0 - 0.0299513 2026-02-23T15:39:53.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:39:52 trial068 ceph-mon[38805]: pgmap v5134: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:39:53.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:39:52 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/438682865' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:39:53.764 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8787 8 365281 365273 0.324739 0 - 0.0299513 2026-02-23T15:39:54.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:39:53 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:39:53] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:39:54.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:39:53 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:39:53.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:39:54.764 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8788 8 365281 365273 0.324702 0 - 0.0299513 2026-02-23T15:39:55.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:39:54 trial068 ceph-mon[38805]: pgmap v5135: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:39:55.239 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:39:55.356 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:39:55.464 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:39:55.462+0000 7f240664b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:39:55.464 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:39:55.462+0000 7f240664b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:39:55.575 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:39:55.575 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":329,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:39:25.180542+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:35:41.278300+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:39:14.873799+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:34:56.867874+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:39:55.764 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8789 8 365281 365273 0.324665 0 - 0.0299513 2026-02-23T15:39:56.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:39:56 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/621948162' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:39:56.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:39:56 trial068 ceph-mon[38805]: pgmap v5136: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:39:56.764 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8790 8 365281 365273 0.324628 0 - 0.0299513 2026-02-23T15:39:57.764 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8791 8 365281 365273 0.324591 0 - 0.0299513 2026-02-23T15:39:58.612 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:39:58.730 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:39:58.764 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8792 8 365281 365273 0.324554 0 - 0.0299513 2026-02-23T15:39:58.829 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:39:58.827+0000 7f8d54e9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:39:58.829 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:39:58.828+0000 7f8d54e9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:39:58.939 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:39:58.939 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":329,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:39:25.180542+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:35:41.278300+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:39:14.873799+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:34:56.867874+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:39:58.939 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:39:58 trial068 ceph-mon[38805]: pgmap v5137: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:39:59.764 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8793 8 365281 365273 0.324517 0 - 0.0299513 2026-02-23T15:40:00.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:39:59 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2970330395' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:40:00.764 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8794 8 365281 365273 0.32448 0 - 0.0299513 2026-02-23T15:40:01.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:40:00 trial068 ceph-mon[38805]: pgmap v5138: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:40:01.764 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8795 8 365281 365273 0.324443 0 - 0.0299513 2026-02-23T15:40:01.977 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:40:02.095 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:40:02.196 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:40:02.194+0000 7f3ea734a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:40:02.196 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:40:02.195+0000 7f3ea734a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:40:02.305 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:40:02.305 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":329,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:39:25.180542+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:35:41.278300+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:39:14.873799+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:34:56.867874+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:40:02.764 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8796 8 365281 365273 0.324406 0 - 0.0299513 2026-02-23T15:40:03.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:40:02 trial068 ceph-mon[38805]: pgmap v5139: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:40:03.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:40:02 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/277810355' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:40:03.765 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8797 8 365281 365273 0.32437 0 - 0.0299513 2026-02-23T15:40:04.265 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:40:03 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:40:03.974Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:40:04.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:40:03 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:40:03] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:40:04.765 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8798 8 365281 365273 0.324333 0 - 0.0299513 2026-02-23T15:40:05.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:40:04 trial068 ceph-mon[38805]: pgmap v5140: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:40:05.341 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:40:05.460 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:40:05.558 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:40:05.557+0000 7fa362c0a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:40:05.559 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:40:05.557+0000 7fa362c0a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:40:05.668 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:40:05.668 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":329,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:39:25.180542+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:35:41.278300+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:39:14.873799+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:34:56.867874+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:40:05.765 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8799 8 365281 365273 0.324296 0 - 0.0299513 2026-02-23T15:40:06.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:40:06 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2643973469' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:40:06.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:40:06 trial068 ceph-mon[38805]: pgmap v5141: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:40:06.765 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T15:40:06.764775+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T15:40:06.765 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T15:40:06.765 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8800 8 365281 365273 0.324259 0 - 0.0299513 2026-02-23T15:40:07.765 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8801 8 365281 365273 0.324222 0 - 0.0299513 2026-02-23T15:40:08.706 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:40:08.765 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8802 8 365281 365273 0.324185 0 - 0.0299513 2026-02-23T15:40:08.826 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:40:08.926 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:40:08.924+0000 7fb903cdb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:40:08.926 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:40:08.925+0000 7fb903cdb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:40:09.035 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:40:09.036 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":329,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:39:25.180542+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:35:41.278300+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:39:14.873799+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:34:56.867874+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:40:09.036 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:40:08 trial068 ceph-mon[38805]: pgmap v5142: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:40:09.765 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8803 8 365281 365273 0.324148 0 - 0.0299513 2026-02-23T15:40:10.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:40:09 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/37899551' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:40:10.765 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8804 8 365281 365273 0.324112 0 - 0.0299513 2026-02-23T15:40:11.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:40:10 trial068 ceph-mon[38805]: pgmap v5143: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:40:11.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:40:10 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T15:40:11.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:40:10 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T15:40:11.765 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8805 8 365281 365273 0.324075 0 - 0.0299513 2026-02-23T15:40:12.073 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:40:12.191 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:40:12.297 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:40:12.295+0000 7f85ea7d5640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:40:12.297 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:40:12.295+0000 7f85ea7d5640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:40:12.405 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:40:12.406 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":329,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:39:25.180542+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:35:41.278300+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:39:14.873799+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:34:56.867874+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:40:12.765 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8806 8 365281 365273 0.324038 0 - 0.0299513 2026-02-23T15:40:13.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:40:12 trial068 ceph-mon[38805]: pgmap v5144: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:40:13.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:40:12 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1628511252' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:40:13.765 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8807 8 365281 365273 0.324001 0 - 0.0299513 2026-02-23T15:40:14.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:40:13 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:40:13] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:40:14.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:40:13 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:40:13.972Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:40:14.765 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8808 8 365281 365273 0.323964 0 - 0.0299513 2026-02-23T15:40:15.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:40:14 trial068 ceph-mon[38805]: pgmap v5145: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:40:15.444 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:40:15.563 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:40:15.665 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:40:15.663+0000 7f1a4611b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:40:15.666 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:40:15.664+0000 7f1a4611b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:40:15.765 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8809 8 365281 365273 0.323928 0 - 0.0299513 2026-02-23T15:40:15.776 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:40:15.776 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":329,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:39:25.180542+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:35:41.278300+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:39:14.873799+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:34:56.867874+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:40:16.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:40:16 trial068 ceph-mon[38805]: pgmap v5146: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:40:16.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:40:16 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2015939433' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:40:16.765 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8810 8 365281 365273 0.323891 0 - 0.0299513 2026-02-23T15:40:17.766 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8811 8 365281 365273 0.323854 0 - 0.0299513 2026-02-23T15:40:18.766 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8812 8 365281 365273 0.323817 0 - 0.0299513 2026-02-23T15:40:18.813 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:40:18.931 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:40:19.036 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:40:19.035+0000 7fc3d118f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:40:19.036 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:40:19.035+0000 7fc3d118f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:40:19.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:40:18 trial068 ceph-mon[38805]: pgmap v5147: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:40:19.146 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:40:19.146 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":329,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:39:25.180542+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:35:41.278300+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:39:14.873799+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:34:56.867874+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:40:19.766 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8813 8 365281 365273 0.323781 0 - 0.0299513 2026-02-23T15:40:20.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:40:19 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/380011039' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:40:20.766 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8814 8 365281 365273 0.323744 0 - 0.0299513 2026-02-23T15:40:21.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:40:20 trial068 ceph-mon[38805]: pgmap v5148: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:40:21.766 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8815 8 365281 365273 0.323707 0 - 0.0299513 2026-02-23T15:40:22.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:40:21 trial068 ceph-mon[38805]: 5.a scrub starts 2026-02-23T15:40:22.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:40:21 trial068 ceph-mon[38805]: 5.a scrub ok 2026-02-23T15:40:22.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:40:21 trial068 ceph-mon[38805]: 5.e scrub starts 2026-02-23T15:40:22.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:40:21 trial068 ceph-mon[38805]: 5.e scrub ok 2026-02-23T15:40:22.183 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:40:22.298 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:40:22.399 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:40:22.397+0000 7f926608e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:40:22.399 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:40:22.398+0000 7f926608e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:40:22.510 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:40:22.511 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":329,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:39:25.180542+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:35:41.278300+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:39:14.873799+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:34:56.867874+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:40:22.766 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8816 8 365281 365273 0.32367 0 - 0.0299513 2026-02-23T15:40:23.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:40:22 trial068 ceph-mon[38805]: pgmap v5149: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:40:23.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:40:22 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2856337183' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:40:23.766 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8817 8 365281 365273 0.323634 0 - 0.0299513 2026-02-23T15:40:24.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:40:23 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:40:23] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:40:24.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:40:23 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:40:23.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:40:24.766 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8818 8 365281 365273 0.323597 0 - 0.0299513 2026-02-23T15:40:25.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:40:24 trial068 ceph-mon[38805]: pgmap v5150: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:40:25.550 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:40:25.667 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:40:25.766 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8819 8 365281 365273 0.32356 0 - 0.0299513 2026-02-23T15:40:25.767 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:40:25.765+0000 7f2699155640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:40:25.767 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:40:25.766+0000 7f2699155640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:40:25.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:40:25 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2026-02-23T15:40:25.879 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:40:25.879 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":329,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:39:25.180542+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:35:41.278300+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:39:14.873799+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:34:56.867874+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:40:26.766 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T15:40:26.766211+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T15:40:26.766 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T15:40:26.767 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8820 8 365281 365273 0.323524 0 - 0.0299513 2026-02-23T15:40:27.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:40:26 trial068 ceph-mon[38805]: pgmap v5151: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:40:27.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:40:26 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1198392070' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:40:27.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:40:26 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-23T15:40:27.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:40:26 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2026-02-23T15:40:27.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:40:26 trial068 ceph-mon[38805]: pgmap v5152: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:40:27.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:40:26 trial068 ceph-mon[38805]: pgmap v5153: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:40:27.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:40:26 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T15:40:27.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:40:26 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2026-02-23T15:40:27.766 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8821 8 365281 365273 0.323487 0 - 0.0299513 2026-02-23T15:40:28.766 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8822 8 365281 365273 0.32345 0 - 0.0299513 2026-02-23T15:40:28.919 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:40:29.037 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:40:29.139 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:40:29.137+0000 7fa65aa9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:40:29.139 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:40:29.138+0000 7fa65aa9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:40:29.251 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:40:29.251 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":329,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:39:25.180542+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:35:41.278300+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:39:14.873799+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:34:56.867874+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:40:29.252 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:40:29 trial068 ceph-mon[38805]: pgmap v5154: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:40:29.766 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8823 8 365281 365273 0.323414 0 - 0.0299513 2026-02-23T15:40:30.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:40:30 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2574523137' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:40:30.766 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8824 8 365281 365273 0.323377 0 - 0.0299513 2026-02-23T15:40:31.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:40:31 trial068 ceph-mon[38805]: pgmap v5155: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:40:31.766 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8825 8 365281 365273 0.32334 0 - 0.0299513 2026-02-23T15:40:32.290 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:40:32.314 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:40:32 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T15:40:32.314 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:40:32 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T15:40:32.410 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:40:32.512 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:40:32.510+0000 7fe92271b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:40:32.512 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:40:32.510+0000 7fe92271b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:40:32.623 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:40:32.623 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":329,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:39:25.180542+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:35:41.278300+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:39:14.873799+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:34:56.867874+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:40:32.767 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8826 8 365281 365273 0.323304 0 - 0.0299513 2026-02-23T15:40:33.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:40:33 trial068 ceph-mon[38805]: pgmap v5156: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:40:33.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:40:33 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3910975792' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:40:33.767 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8827 8 365281 365273 0.323267 0 - 0.0299513 2026-02-23T15:40:34.265 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:40:33 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:40:33.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:40:34.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:40:33 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:40:33] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:40:34.767 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8828 8 365281 365273 0.323231 0 - 0.0299513 2026-02-23T15:40:35.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:40:35 trial068 ceph-mon[38805]: pgmap v5157: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:40:35.662 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:40:35.767 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8829 8 365281 365273 0.323194 0 - 0.0299513 2026-02-23T15:40:35.780 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:40:35.878 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:40:35.876+0000 7f9c4d42f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:40:35.878 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:40:35.876+0000 7f9c4d42f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:40:35.987 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:40:35.987 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":329,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:39:25.180542+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:35:41.278300+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:39:14.873799+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:34:56.867874+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:40:36.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:40:36 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3428242089' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:40:36.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:40:36 trial068 ceph-mon[38805]: pgmap v5158: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:40:36.767 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8830 8 365281 365273 0.323157 0 - 0.0299513 2026-02-23T15:40:37.767 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8831 8 365281 365273 0.323121 0 - 0.0299513 2026-02-23T15:40:38.767 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8832 8 365281 365273 0.323084 0 - 0.0299513 2026-02-23T15:40:39.025 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:40:39.142 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:40:39.244 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:40:39.242+0000 7fa1ae5d8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:40:39.244 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:40:39.242+0000 7fa1ae5d8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:40:39.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:40:39 trial068 ceph-mon[38805]: pgmap v5159: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:40:39.352 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:40:39.352 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":329,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:39:25.180542+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:35:41.278300+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:39:14.873799+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:34:56.867874+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:40:39.767 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8833 8 365281 365273 0.323048 0 - 0.0299513 2026-02-23T15:40:40.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:40:40 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1681719697' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:40:40.767 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8834 8 365281 365273 0.323011 0 - 0.0299513 2026-02-23T15:40:41.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:40:41 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T15:40:41.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:40:41 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T15:40:41.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:40:41 trial068 ceph-mon[38805]: pgmap v5160: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:40:41.767 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8835 8 365281 365273 0.322974 0 - 0.0299513 2026-02-23T15:40:42.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:40:42 trial068 ceph-mon[38805]: osdmap e330: 4 total, 3 up, 4 in 2026-02-23T15:40:42.390 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:40:42.508 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:40:42.607 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:40:42.606+0000 7fca68335640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:40:42.608 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:40:42.606+0000 7fca68335640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:40:42.716 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:40:42.717 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":330,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:40:41.055704+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:35:41.278300+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:39:14.873799+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:40:09.841031+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:40:42.767 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8836 8 365281 365273 0.322938 0 - 0.0299513 2026-02-23T15:40:43.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:40:43 trial068 ceph-mon[38805]: pgmap v5162: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:40:43.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:40:43 trial068 ceph-mon[38805]: 5.5 scrub starts 2026-02-23T15:40:43.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:40:43 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/271649821' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:40:43.767 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8837 8 365281 365273 0.322901 0 - 0.0299513 2026-02-23T15:40:44.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:40:44 trial068 ceph-mon[38805]: 5.5 scrub ok 2026-02-23T15:40:44.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:40:43 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:40:43] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:40:44.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:40:43 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:40:43.974Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:40:44.767 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8838 8 365281 365273 0.322865 0 - 0.0299513 2026-02-23T15:40:45.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:40:45 trial068 ceph-mon[38805]: pgmap v5163: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:40:45.755 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:40:45.767 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8839 8 365281 365273 0.322828 0 - 0.0299513 2026-02-23T15:40:45.874 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:40:45.982 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:40:45.980+0000 7fdee0083640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:40:45.982 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:40:45.981+0000 7fdee0083640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:40:46.090 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:40:46.090 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":331,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:40:45.189650+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:35:41.278300+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:39:14.873799+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:40:09.841031+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:40:46.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:40:46 trial068 ceph-mon[38805]: osdmap e331: 4 total, 3 up, 4 in 2026-02-23T15:40:46.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:40:46 trial068 ceph-mon[38805]: pgmap v5165: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:40:46.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:40:46 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2874522381' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:40:46.767 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T15:40:46.767645+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T15:40:46.767 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T15:40:46.767 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8840 8 365281 365273 0.322792 0 - 0.0299513 2026-02-23T15:40:47.767 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8841 8 365281 365273 0.322755 0 - 0.0299513 2026-02-23T15:40:48.767 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8842 8 365281 365273 0.322719 0 - 0.0299513 2026-02-23T15:40:49.126 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:40:49.242 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:40:49.341 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:40:49.340+0000 7fa1b5ee2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:40:49.342 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:40:49 trial068 ceph-mon[38805]: pgmap v5166: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:40:49.342 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:40:49.340+0000 7fa1b5ee2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:40:49.451 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:40:49.451 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":331,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:40:45.189650+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:35:41.278300+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:39:14.873799+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:40:09.841031+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:40:49.767 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8843 8 365281 365273 0.322682 0 - 0.0299513 2026-02-23T15:40:50.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:40:50 trial068 ceph-mon[38805]: 5.7 scrub starts 2026-02-23T15:40:50.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:40:50 trial068 ceph-mon[38805]: 5.7 scrub ok 2026-02-23T15:40:50.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:40:50 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/671354720' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:40:50.767 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8844 8 365281 365273 0.322646 0 - 0.0299513 2026-02-23T15:40:51.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:40:51 trial068 ceph-mon[38805]: pgmap v5167: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:40:51.768 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8845 8 365281 365273 0.322609 0 - 0.0299513 2026-02-23T15:40:52.487 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:40:52.603 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:40:52.709 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:40:52.707+0000 7f00ab0a8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:40:52.709 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:40:52.708+0000 7f00ab0a8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:40:52.768 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8846 8 365281 365273 0.322573 0 - 0.0299513 2026-02-23T15:40:52.818 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:40:52.818 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":331,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:40:45.189650+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:35:41.278300+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:39:14.873799+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:40:09.841031+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:40:53.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:40:53 trial068 ceph-mon[38805]: pgmap v5168: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:40:53.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:40:53 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/4206414562' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:40:53.768 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8847 8 365281 365273 0.322536 0 - 0.0299513 2026-02-23T15:40:54.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:40:53 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:40:53] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:40:54.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:40:53 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:40:53.974Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:40:54.768 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8848 8 365281 365273 0.3225 0 - 0.0299513 2026-02-23T15:40:55.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:40:55 trial068 ceph-mon[38805]: pgmap v5169: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:40:55.768 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8849 8 365281 365273 0.322463 0 - 0.0299513 2026-02-23T15:40:55.854 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:40:55.971 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:40:56.072 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:40:56.071+0000 7f9f198cf640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:40:56.073 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:40:56.071+0000 7f9f198cf640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:40:56.182 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:40:56.183 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":331,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:40:45.189650+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:35:41.278300+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:39:14.873799+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:40:09.841031+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:40:56.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:40:56 trial068 ceph-mon[38805]: pgmap v5170: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:40:56.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:40:56 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2880696766' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:40:56.768 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8850 8 365281 365273 0.322427 0 - 0.0299513 2026-02-23T15:40:57.768 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8851 8 365281 365273 0.322391 0 - 0.0299513 2026-02-23T15:40:58.768 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8852 8 365281 365273 0.322354 0 - 0.0299513 2026-02-23T15:40:59.220 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:40:59.338 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:40:59.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:40:59 trial068 ceph-mon[38805]: pgmap v5171: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:40:59.431 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:40:59.430+0000 7fc3a2dfc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:40:59.432 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:40:59.430+0000 7fc3a2dfc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:40:59.541 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:40:59.541 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":331,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:40:45.189650+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:35:41.278300+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:39:14.873799+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:40:09.841031+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:40:59.768 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8853 8 365281 365273 0.322318 0 - 0.0299513 2026-02-23T15:41:00.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:41:00 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1004442515' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:41:00.768 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8854 8 365281 365273 0.322281 0 - 0.0299513 2026-02-23T15:41:01.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:41:01 trial068 ceph-mon[38805]: pgmap v5172: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:41:01.768 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8855 8 365281 365273 0.322245 0 - 0.0299513 2026-02-23T15:41:02.577 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:41:02.693 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:41:02.769 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8856 8 365281 365273 0.322209 0 - 0.0299513 2026-02-23T15:41:02.796 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:41:02.794+0000 7f138a3c7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:41:02.796 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:41:02.794+0000 7f138a3c7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:41:02.906 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:41:02.906 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":331,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:40:45.189650+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:35:41.278300+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:39:14.873799+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:40:09.841031+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:41:03.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:41:03 trial068 ceph-mon[38805]: pgmap v5173: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:41:03.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:41:03 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3081590845' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:41:03.769 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8857 8 365281 365273 0.322172 0 - 0.0299513 2026-02-23T15:41:04.265 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:41:03 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:41:03.974Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:41:04.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:41:03 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:41:03] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:41:04.769 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8858 8 365281 365273 0.322136 0 - 0.0299513 2026-02-23T15:41:05.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:41:05 trial068 ceph-mon[38805]: pgmap v5174: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:41:05.769 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8859 8 365281 365273 0.322099 0 - 0.0299513 2026-02-23T15:41:05.945 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:41:06.063 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:41:06.161 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:41:06.159+0000 7f087052d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:41:06.161 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:41:06.160+0000 7f087052d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:41:06.270 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:41:06.271 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":331,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:40:45.189650+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:35:41.278300+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:39:14.873799+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:40:09.841031+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:41:06.271 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:41:06 trial068 ceph-mon[38805]: 5.8 scrub starts 2026-02-23T15:41:06.271 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:41:06 trial068 ceph-mon[38805]: 5.8 scrub ok 2026-02-23T15:41:06.769 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T15:41:06.768914+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T15:41:06.769 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T15:41:06.769 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8860 8 365281 365273 0.322063 0 - 0.0299513 2026-02-23T15:41:07.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:41:07 trial068 ceph-mon[38805]: pgmap v5175: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:41:07.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:41:07 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3583027506' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:41:07.769 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8861 8 365281 365273 0.322027 0 - 0.0299513 2026-02-23T15:41:08.769 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8862 8 365281 365273 0.32199 0 - 0.0299513 2026-02-23T15:41:09.307 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:41:09.332 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:41:09 trial068 ceph-mon[38805]: pgmap v5176: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:41:09.425 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:41:09.524 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:41:09.522+0000 7fde532a9640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:41:09.524 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:41:09.522+0000 7fde532a9640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:41:09.633 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:41:09.633 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":331,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:40:45.189650+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:35:41.278300+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:39:14.873799+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:40:09.841031+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:41:09.769 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8863 8 365281 365273 0.321954 0 - 0.0299513 2026-02-23T15:41:10.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:41:10 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/753990176' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:41:10.769 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8864 8 365281 365273 0.321918 0 - 0.0299513 2026-02-23T15:41:11.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:41:11 trial068 ceph-mon[38805]: pgmap v5177: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:41:11.769 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8865 8 365281 365273 0.321881 0 - 0.0299513 2026-02-23T15:41:12.671 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:41:12.769 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8866 8 365281 365273 0.321845 0 - 0.0299513 2026-02-23T15:41:12.791 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:41:12.898 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:41:12.896+0000 7fe110aa5640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:41:12.898 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:41:12.896+0000 7fe110aa5640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:41:13.009 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:41:13.009 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":331,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:40:45.189650+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:35:41.278300+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:39:14.873799+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:40:09.841031+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:41:13.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:41:13 trial068 ceph-mon[38805]: pgmap v5178: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:41:13.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:41:13 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2574129746' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:41:13.769 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8867 8 365281 365273 0.321809 0 - 0.0299513 2026-02-23T15:41:14.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:41:13 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:41:13] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:41:14.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:41:13 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:41:13.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:41:14.769 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8868 8 365281 365273 0.321773 0 - 0.0299513 2026-02-23T15:41:15.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:41:15 trial068 ceph-mon[38805]: pgmap v5179: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:41:15.769 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8869 8 365281 365273 0.321736 0 - 0.0299513 2026-02-23T15:41:16.047 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:41:16.165 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:41:16.265 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:41:16.264+0000 7fc0c840e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:41:16.266 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:41:16.264+0000 7fc0c840e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:41:16.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:41:16 trial068 ceph-mon[38805]: pgmap v5180: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:41:16.377 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:41:16.377 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":331,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:40:45.189650+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:35:41.278300+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:39:14.873799+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:40:09.841031+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:41:16.770 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8870 8 365281 365273 0.3217 0 - 0.0299513 2026-02-23T15:41:17.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:41:17 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T15:41:17.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:41:17 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T15:41:17.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:41:17 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1033536700' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:41:17.770 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8871 8 365281 365273 0.321664 0 - 0.0299513 2026-02-23T15:41:18.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:41:18 trial068 ceph-mon[38805]: pgmap v5181: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:41:18.770 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8872 8 365281 365273 0.321628 0 - 0.0299513 2026-02-23T15:41:19.429 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:41:19.547 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:41:19.647 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:41:19.645+0000 7f66d071b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:41:19.647 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:41:19.645+0000 7f66d071b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:41:19.759 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:41:19.759 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":331,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:40:45.189650+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:35:41.278300+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:39:14.873799+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:40:09.841031+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:41:19.770 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8873 8 365281 365273 0.321591 0 - 0.0299513 2026-02-23T15:41:20.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:41:19 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1165788831' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:41:20.770 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8874 8 365281 365273 0.321555 0 - 0.0299513 2026-02-23T15:41:21.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:41:20 trial068 ceph-mon[38805]: pgmap v5182: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:41:21.770 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8875 8 365281 365273 0.321519 0 - 0.0299513 2026-02-23T15:41:22.770 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8876 8 365281 365273 0.321483 0 - 0.0299513 2026-02-23T15:41:22.795 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:41:22.912 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:41:23.019 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:41:23.017+0000 7fc1cd0e0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:41:23.019 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:41:23.017+0000 7fc1cd0e0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:41:23.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:41:23 trial068 ceph-mon[38805]: pgmap v5183: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:41:23.128 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:41:23.128 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":331,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:40:45.189650+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:35:41.278300+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:39:14.873799+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:40:09.841031+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:41:23.770 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8877 8 365281 365273 0.321446 0 - 0.0299513 2026-02-23T15:41:24.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:41:24 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2424288508' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:41:24.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:41:23 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:41:23] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:41:24.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:41:23 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:41:23.974Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:41:24.770 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8878 8 365281 365273 0.32141 0 - 0.0299513 2026-02-23T15:41:25.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:41:25 trial068 ceph-mon[38805]: pgmap v5184: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:41:25.770 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8879 8 365281 365273 0.321374 0 - 0.0299513 2026-02-23T15:41:26.168 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:41:26.288 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:41:26.341 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:41:26 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2026-02-23T15:41:26.399 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:41:26.398+0000 7f0a44557640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:41:26.399 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:41:26.398+0000 7f0a44557640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:41:26.510 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:41:26.510 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":331,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:40:45.189650+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:35:41.278300+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:39:14.873799+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:40:09.841031+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:41:26.770 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T15:41:26.770295+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T15:41:26.771 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T15:41:26.771 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8880 8 365281 365273 0.321338 0 - 0.0299513 2026-02-23T15:41:27.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:41:27 trial068 ceph-mon[38805]: pgmap v5185: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:41:27.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:41:27 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-23T15:41:27.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:41:27 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2026-02-23T15:41:27.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:41:27 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T15:41:27.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:41:27 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2026-02-23T15:41:27.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:41:27 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1478013654' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:41:27.770 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8881 8 365281 365273 0.321302 0 - 0.0299513 2026-02-23T15:41:28.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:41:28 trial068 ceph-mon[38805]: pgmap v5186: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:41:28.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:41:28 trial068 ceph-mon[38805]: pgmap v5187: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:41:28.770 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8882 8 365281 365273 0.321265 0 - 0.0299513 2026-02-23T15:41:29.548 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:41:29.664 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:41:29.764 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:41:29.762+0000 7f0afd82a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:41:29.764 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:41:29.763+0000 7f0afd82a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:41:29.770 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8883 8 365281 365273 0.321229 0 - 0.0299513 2026-02-23T15:41:29.873 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:41:29.873 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":331,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:40:45.189650+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:35:41.278300+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:39:14.873799+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:40:09.841031+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:41:30.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:41:30 trial068 ceph-mon[38805]: pgmap v5188: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:41:30.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:41:30 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2866536995' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:41:30.771 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8884 8 365281 365273 0.321193 0 - 0.0299513 2026-02-23T15:41:31.771 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8885 8 365281 365273 0.321157 0 - 0.0299513 2026-02-23T15:41:32.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:41:32 trial068 ceph-mon[38805]: pgmap v5189: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:41:32.771 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8886 8 365281 365273 0.321121 0 - 0.0299513 2026-02-23T15:41:32.911 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:41:33.029 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:41:33.133 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:41:33.131+0000 7fa172071640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:41:33.133 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:41:33.132+0000 7fa172071640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:41:33.245 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:41:33.246 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":331,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:40:45.189650+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:35:41.278300+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:39:14.873799+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:40:09.841031+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:41:33.771 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8887 8 365281 365273 0.321085 0 - 0.0299513 2026-02-23T15:41:34.264 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:41:33 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:41:33.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:41:34.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:41:34 trial068 ceph-mon[38805]: pgmap v5190: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:41:34.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:41:34 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/829794548' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:41:34.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:41:33 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:41:33] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:41:34.771 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8888 8 365281 365273 0.321049 0 - 0.0299513 2026-02-23T15:41:35.771 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8889 8 365281 365273 0.321012 0 - 0.0299513 2026-02-23T15:41:36.284 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:41:36.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:41:36 trial068 ceph-mon[38805]: pgmap v5191: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:41:36.403 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:41:36.507 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:41:36.505+0000 7fb4b15cc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:41:36.507 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:41:36.506+0000 7fb4b15cc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:41:36.617 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:41:36.617 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":331,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:40:45.189650+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:35:41.278300+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:39:14.873799+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:40:09.841031+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:41:36.771 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8890 8 365281 365273 0.320976 0 - 0.0299513 2026-02-23T15:41:37.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:41:37 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2423871780' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:41:37.771 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8891 8 365281 365273 0.32094 0 - 0.0299513 2026-02-23T15:41:38.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:41:38 trial068 ceph-mon[38805]: pgmap v5192: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:41:38.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:41:38 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T15:41:38.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:41:38 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T15:41:38.771 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8892 8 365281 365273 0.320904 0 - 0.0299513 2026-02-23T15:41:39.655 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:41:39.771 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8893 8 365281 365273 0.320868 0 - 0.0299513 2026-02-23T15:41:39.772 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:41:39.870 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:41:39.868+0000 7f72bfb46640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:41:39.870 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:41:39.869+0000 7f72bfb46640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:41:39.980 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:41:39.980 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":331,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:40:45.189650+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:35:41.278300+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:39:14.873799+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:40:09.841031+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:41:40.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:41:40 trial068 ceph-mon[38805]: pgmap v5193: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:41:40.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:41:40 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2792573961' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:41:40.771 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8894 8 365281 365273 0.320832 0 - 0.0299513 2026-02-23T15:41:41.771 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8895 8 365281 365273 0.320796 0 - 0.0299513 2026-02-23T15:41:42.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:41:42 trial068 ceph-mon[38805]: pgmap v5194: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:41:42.772 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8896 8 365281 365273 0.32076 0 - 0.0299513 2026-02-23T15:41:43.017 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:41:43.135 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:41:43.244 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:41:43.242+0000 7fbdb6dfe640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:41:43.244 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:41:43.242+0000 7fbdb6dfe640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:41:43.353 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:41:43.353 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":331,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:40:45.189650+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:35:41.278300+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:39:14.873799+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:40:09.841031+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:41:43.771 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8897 8 365281 365273 0.320724 0 - 0.0299513 2026-02-23T15:41:44.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:41:43 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:41:43] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:41:44.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:41:44 trial068 ceph-mon[38805]: pgmap v5195: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:41:44.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:41:44 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T15:41:44.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:41:44 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T15:41:44.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:41:44 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2018299745' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:41:44.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:41:43 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:41:43.974Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:41:44.771 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8898 8 365281 365273 0.320688 0 - 0.0299513 2026-02-23T15:41:45.772 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8899 8 365281 365273 0.320652 0 - 0.0299513 2026-02-23T15:41:46.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:41:46 trial068 ceph-mon[38805]: pgmap v5196: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:41:46.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:41:46 trial068 ceph-mon[38805]: 5.a scrub starts 2026-02-23T15:41:46.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:41:46 trial068 ceph-mon[38805]: osdmap e332: 4 total, 3 up, 4 in 2026-02-23T15:41:46.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:41:46 trial068 ceph-mon[38805]: osdmap e333: 4 total, 3 up, 4 in 2026-02-23T15:41:46.390 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:41:46.510 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:41:46.614 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:41:46.612+0000 7faf9668b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:41:46.614 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:41:46.612+0000 7faf9668b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:41:46.723 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:41:46.724 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":333,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:41:45.196353+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:41:36.954923+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:39:14.873799+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:40:09.841031+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:41:46.772 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T15:41:46.771736+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T15:41:46.772 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T15:41:46.772 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8900 8 365281 365273 0.320616 0 - 0.0299513 2026-02-23T15:41:47.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:41:47 trial068 ceph-mon[38805]: 5.a scrub ok 2026-02-23T15:41:47.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:41:47 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2755004376' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:41:47.772 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8901 8 365281 365273 0.32058 0 - 0.0299513 2026-02-23T15:41:48.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:41:48 trial068 ceph-mon[38805]: pgmap v5199: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:41:48.772 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8902 8 365281 365273 0.320544 0 - 0.0299513 2026-02-23T15:41:49.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:41:49 trial068 ceph-mon[38805]: 5.e scrub starts 2026-02-23T15:41:49.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:41:49 trial068 ceph-mon[38805]: 5.e scrub ok 2026-02-23T15:41:49.762 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:41:49.772 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8903 8 365281 365273 0.320508 0 - 0.0299513 2026-02-23T15:41:49.880 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:41:49.982 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:41:49.981+0000 7f8161c69640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:41:49.982 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:41:49.981+0000 7f8161c69640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:41:50.091 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:41:50.091 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":333,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:41:45.196353+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:41:36.954923+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:39:14.873799+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:40:09.841031+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:41:50.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:41:50 trial068 ceph-mon[38805]: pgmap v5200: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:41:50.772 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8904 8 365281 365273 0.320472 0 - 0.0299513 2026-02-23T15:41:51.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:41:51 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2178643932' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:41:51.772 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8905 8 365281 365273 0.320436 0 - 0.0299513 2026-02-23T15:41:52.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:41:52 trial068 ceph-mon[38805]: pgmap v5201: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:41:52.772 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8906 8 365281 365273 0.3204 0 - 0.0299513 2026-02-23T15:41:53.128 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:41:53.245 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:41:53.348 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:41:53.346+0000 7f890655f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:41:53.348 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:41:53.347+0000 7f890655f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:41:53.459 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:41:53.459 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":333,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:41:45.196353+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:41:36.954923+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:39:14.873799+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:40:09.841031+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:41:53.772 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8907 8 365281 365273 0.320364 0 - 0.0299513 2026-02-23T15:41:54.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:41:54 trial068 ceph-mon[38805]: pgmap v5202: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:41:54.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:41:54 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2770298969' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:41:54.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:41:53 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:41:53] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:41:54.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:41:53 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:41:53.974Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:41:54.772 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8908 8 365281 365273 0.320328 0 - 0.0299513 2026-02-23T15:41:55.772 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8909 8 365281 365273 0.320292 0 - 0.0299513 2026-02-23T15:41:56.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:41:56 trial068 ceph-mon[38805]: pgmap v5203: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:41:56.495 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:41:56.613 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:41:56.714 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:41:56.712+0000 7f16494e1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:41:56.714 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:41:56.712+0000 7f16494e1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:41:56.772 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8910 8 365281 365273 0.320256 0 - 0.0299513 2026-02-23T15:41:56.823 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:41:56.823 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":333,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:41:45.196353+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:41:36.954923+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:39:14.873799+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:40:09.841031+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:41:57.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:41:57 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/645419052' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:41:57.773 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8911 8 365281 365273 0.32022 0 - 0.0299513 2026-02-23T15:41:58.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:41:58 trial068 ceph-mon[38805]: pgmap v5204: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:41:58.773 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8912 8 365281 365273 0.320184 0 - 0.0299513 2026-02-23T15:41:59.773 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8913 8 365281 365273 0.320148 0 - 0.0299513 2026-02-23T15:41:59.862 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:41:59.978 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:42:00.080 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:42:00.078+0000 7fedfe693640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:42:00.080 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:42:00.078+0000 7fedfe693640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:42:00.189 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:42:00.190 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":333,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:41:45.196353+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:41:36.954923+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:39:14.873799+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:40:09.841031+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:42:00.190 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:42:00 trial068 ceph-mon[38805]: pgmap v5205: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:42:00.190 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:42:00 trial068 ceph-mon[38805]: 5.5 scrub starts 2026-02-23T15:42:00.190 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:42:00 trial068 ceph-mon[38805]: 5.5 scrub ok 2026-02-23T15:42:00.773 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8914 8 365281 365273 0.320112 0 - 0.0299513 2026-02-23T15:42:01.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:42:01 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/69601423' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:42:01.773 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8915 8 365281 365273 0.320076 0 - 0.0299513 2026-02-23T15:42:02.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:42:02 trial068 ceph-mon[38805]: pgmap v5206: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:42:02.773 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8916 8 365281 365273 0.32004 0 - 0.0299513 2026-02-23T15:42:03.228 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:42:03.347 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:42:03.446 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:42:03.444+0000 7f662a036640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:42:03.446 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:42:03.444+0000 7f662a036640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:42:03.555 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:42:03.555 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":333,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:41:45.196353+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:41:36.954923+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:39:14.873799+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:40:09.841031+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:42:03.773 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8917 8 365281 365273 0.320004 0 - 0.0299513 2026-02-23T15:42:04.097 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:42:03 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:42:03] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:42:04.265 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:42:03 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:42:03.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:42:04.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:42:04 trial068 ceph-mon[38805]: pgmap v5207: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:42:04.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:42:04 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/239025472' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:42:04.773 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8918 8 365281 365273 0.319969 0 - 0.0299513 2026-02-23T15:42:05.773 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8919 8 365281 365273 0.319933 0 - 0.0299513 2026-02-23T15:42:06.593 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:42:06.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:42:06 trial068 ceph-mon[38805]: pgmap v5208: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:42:06.709 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:42:06.773 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T15:42:06.773298+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T15:42:06.773 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T15:42:06.774 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8920 8 365281 365273 0.319897 0 - 0.0299513 2026-02-23T15:42:06.810 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:42:06.809+0000 7fdf816c7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:42:06.811 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:42:06.809+0000 7fdf816c7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:42:06.919 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:42:06.919 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":333,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:41:45.196353+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:41:36.954923+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:39:14.873799+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:40:09.841031+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:42:07.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:42:07 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3097562590' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:42:07.773 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8921 8 365281 365273 0.319861 0 - 0.0299513 2026-02-23T15:42:08.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:42:08 trial068 ceph-mon[38805]: pgmap v5209: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:42:08.773 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8922 8 365281 365273 0.319825 0 - 0.0299513 2026-02-23T15:42:09.774 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8923 8 365281 365273 0.319789 0 - 0.0299513 2026-02-23T15:42:09.959 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:42:10.075 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:42:10.178 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:42:10.177+0000 7f269acff640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:42:10.179 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:42:10.177+0000 7f269acff640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:42:10.288 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:42:10.288 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":333,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:41:45.196353+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:41:36.954923+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:39:14.873799+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:40:09.841031+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:42:10.288 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:42:10 trial068 ceph-mon[38805]: pgmap v5210: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:42:10.289 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:42:10 trial068 ceph-mon[38805]: 5.7 scrub starts 2026-02-23T15:42:10.289 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:42:10 trial068 ceph-mon[38805]: 5.7 scrub ok 2026-02-23T15:42:10.774 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8924 8 365281 365273 0.319753 0 - 0.0299513 2026-02-23T15:42:11.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:42:11 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/19307360' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:42:11.774 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8925 8 365281 365273 0.319718 0 - 0.0299513 2026-02-23T15:42:12.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:42:12 trial068 ceph-mon[38805]: pgmap v5211: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:42:12.774 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8926 8 365281 365273 0.319682 0 - 0.0299513 2026-02-23T15:42:13.325 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:42:13.445 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:42:13.550 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:42:13.548+0000 7f54d407c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:42:13.550 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:42:13.548+0000 7f54d407c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:42:13.657 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:42:13.657 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":333,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:41:45.196353+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:41:36.954923+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:39:14.873799+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:40:09.841031+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:42:13.774 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8927 8 365281 365273 0.319646 0 - 0.0299513 2026-02-23T15:42:14.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:42:14 trial068 ceph-mon[38805]: pgmap v5212: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:42:14.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:42:14 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/700108298' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:42:14.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:42:13 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:42:13] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:42:14.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:42:13 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:42:13.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:42:14.774 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8928 8 365281 365273 0.31961 0 - 0.0299513 2026-02-23T15:42:15.774 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8929 8 365281 365273 0.319574 0 - 0.0299513 2026-02-23T15:42:16.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:42:16 trial068 ceph-mon[38805]: pgmap v5213: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:42:16.695 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:42:16.774 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8930 8 365281 365273 0.319539 0 - 0.0299513 2026-02-23T15:42:16.812 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:42:16.910 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:42:16.909+0000 7f9423a41640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:42:16.910 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:42:16.909+0000 7f9423a41640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:42:17.019 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:42:17.019 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":333,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:41:45.196353+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:41:36.954923+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:39:14.873799+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:40:09.841031+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:42:17.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:42:17 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1579175505' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:42:17.774 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8931 8 365281 365273 0.319503 0 - 0.0299513 2026-02-23T15:42:18.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:42:18 trial068 ceph-mon[38805]: pgmap v5214: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:42:18.774 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8932 8 365281 365273 0.319467 0 - 0.0299513 2026-02-23T15:42:19.774 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8933 8 365281 365273 0.319431 0 - 0.0299513 2026-02-23T15:42:20.057 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:42:20.176 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:42:20.280 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:42:20.279+0000 7ff1ef03a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:42:20.281 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:42:20.279+0000 7ff1ef03a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:42:20.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:42:20 trial068 ceph-mon[38805]: pgmap v5215: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:42:20.391 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:42:20.391 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":333,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:41:45.196353+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:41:36.954923+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:39:14.873799+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:40:09.841031+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:42:20.774 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8934 8 365281 365273 0.319395 0 - 0.0299513 2026-02-23T15:42:21.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:42:21 trial068 ceph-mon[38805]: pgmap v5216: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:42:21.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:42:21 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/4100505751' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:42:21.774 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8935 8 365281 365273 0.31936 0 - 0.0299513 2026-02-23T15:42:22.775 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8936 8 365281 365273 0.319324 0 - 0.0299513 2026-02-23T15:42:23.429 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:42:23.546 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:42:23.591 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:42:23 trial068 ceph-mon[38805]: pgmap v5217: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:42:23.648 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:42:23.646+0000 7fe5dd9af640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:42:23.648 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:42:23.646+0000 7fe5dd9af640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:42:23.757 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:42:23.757 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":333,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:41:45.196353+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:41:36.954923+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:39:14.873799+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:40:09.841031+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:42:23.775 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8937 8 365281 365273 0.319288 0 - 0.0299513 2026-02-23T15:42:24.327 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:42:23 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:42:23] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:42:24.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:42:23 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:42:23.974Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:42:24.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:42:24 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2360785965' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:42:24.775 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8938 8 365281 365273 0.319253 0 - 0.0299513 2026-02-23T15:42:25.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:42:25 trial068 ceph-mon[38805]: pgmap v5218: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:42:25.775 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8939 8 365281 365273 0.319217 0 - 0.0299513 2026-02-23T15:42:26.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:42:26 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2026-02-23T15:42:26.775 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T15:42:26.774870+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T15:42:26.775 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T15:42:26.775 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8940 8 365281 365273 0.319181 0 - 0.0299513 2026-02-23T15:42:26.794 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:42:26.911 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:42:27.011 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:42:27.009+0000 7f2bb1caf640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:42:27.011 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:42:27.010+0000 7f2bb1caf640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:42:27.125 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:42:27.125 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":333,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:41:45.196353+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:41:36.954923+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:39:14.873799+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:40:09.841031+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:42:27.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:42:27 trial068 ceph-mon[38805]: pgmap v5219: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:42:27.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:42:27 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-23T15:42:27.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:42:27 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2026-02-23T15:42:27.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:42:27 trial068 ceph-mon[38805]: pgmap v5220: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:42:27.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:42:27 trial068 ceph-mon[38805]: pgmap v5221: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:42:27.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:42:27 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T15:42:27.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:42:27 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2026-02-23T15:42:27.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:42:27 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/473077048' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:42:27.775 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8941 8 365281 365273 0.319145 0 - 0.0299513 2026-02-23T15:42:28.775 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8942 8 365281 365273 0.31911 0 - 0.0299513 2026-02-23T15:42:29.775 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8943 8 365281 365273 0.319074 0 - 0.0299513 2026-02-23T15:42:29.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:42:29 trial068 ceph-mon[38805]: pgmap v5222: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:42:30.163 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:42:30.284 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:42:30.385 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:42:30.383+0000 7f78a4a9c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:42:30.385 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:42:30.383+0000 7f78a4a9c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:42:30.495 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:42:30.495 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":333,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:41:45.196353+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:41:36.954923+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:39:14.873799+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:40:09.841031+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:42:30.775 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8944 8 365281 365273 0.319038 0 - 0.0299513 2026-02-23T15:42:30.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:42:30 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/754240088' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:42:31.775 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8945 8 365281 365273 0.319003 0 - 0.0299513 2026-02-23T15:42:31.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:42:31 trial068 ceph-mon[38805]: pgmap v5223: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:42:32.775 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8946 8 365281 365273 0.318967 0 - 0.0299513 2026-02-23T15:42:32.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:42:32 trial068 ceph-mon[38805]: 5.8 scrub starts 2026-02-23T15:42:33.532 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:42:33.652 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:42:33.754 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:42:33.752+0000 7fa91189b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:42:33.754 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:42:33.753+0000 7fa91189b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:42:33.775 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8947 8 365281 365273 0.318931 0 - 0.0299513 2026-02-23T15:42:33.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:42:33 trial068 ceph-mon[38805]: 5.8 scrub ok 2026-02-23T15:42:33.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:42:33 trial068 ceph-mon[38805]: pgmap v5224: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:42:33.868 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:42:33.868 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":333,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:41:45.196353+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:41:36.954923+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:39:14.873799+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:40:09.841031+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:42:34.265 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:42:33 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:42:33.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:42:34.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:42:33 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:42:33] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:42:34.776 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8948 8 365281 365273 0.318896 0 - 0.0299513 2026-02-23T15:42:34.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:42:34 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1051235034' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:42:35.776 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8949 8 365281 365273 0.31886 0 - 0.0299513 2026-02-23T15:42:35.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:42:35 trial068 ceph-mon[38805]: pgmap v5225: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:42:36.776 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8950 8 365281 365273 0.318824 0 - 0.0299513 2026-02-23T15:42:36.906 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:42:37.027 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:42:37.129 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:42:37.127+0000 7f9e0ea0b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:42:37.129 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:42:37.127+0000 7f9e0ea0b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:42:37.240 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:42:37.240 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":333,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:41:45.196353+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:41:36.954923+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:39:14.873799+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:40:09.841031+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:42:37.776 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8951 8 365281 365273 0.318789 0 - 0.0299513 2026-02-23T15:42:37.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:42:37 trial068 ceph-mon[38805]: pgmap v5226: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:42:37.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:42:37 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2333744017' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:42:38.776 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8952 8 365281 365273 0.318753 0 - 0.0299513 2026-02-23T15:42:39.776 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8953 8 365281 365273 0.318718 0 - 0.0299513 2026-02-23T15:42:39.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:42:39 trial068 ceph-mon[38805]: pgmap v5227: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:42:40.277 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:42:40.393 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:42:40.491 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:42:40.489+0000 7f5546133640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:42:40.491 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:42:40.490+0000 7f5546133640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:42:40.601 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:42:40.601 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":333,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:41:45.196353+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:41:36.954923+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:39:14.873799+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:40:09.841031+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:42:40.776 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8954 8 365281 365273 0.318682 0 - 0.0299513 2026-02-23T15:42:41.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:42:40 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T15:42:41.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:42:40 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T15:42:41.776 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8955 8 365281 365273 0.318646 0 - 0.0299513 2026-02-23T15:42:42.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:42:41 trial068 ceph-mon[38805]: pgmap v5228: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:42:42.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:42:41 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2493182184' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:42:42.776 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8956 8 365281 365273 0.318611 0 - 0.0299513 2026-02-23T15:42:43.639 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:42:43.756 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:42:43.776 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8957 8 365281 365273 0.318575 0 - 0.0299513 2026-02-23T15:42:43.859 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:42:43.857+0000 7f4b93f54640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:42:43.859 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:42:43.857+0000 7f4b93f54640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:42:43.859 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:42:43 trial068 ceph-mon[38805]: pgmap v5229: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:42:43.967 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:42:43.967 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":333,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:41:45.196353+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:41:36.954923+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:39:14.873799+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:40:09.841031+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:42:44.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:42:43 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:42:43] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:42:44.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:42:43 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:42:43.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:42:44.776 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8958 8 365281 365273 0.31854 0 - 0.0299513 2026-02-23T15:42:45.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:42:44 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1130465147' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:42:45.776 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8959 8 365281 365273 0.318504 0 - 0.0299513 2026-02-23T15:42:46.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:42:45 trial068 ceph-mon[38805]: pgmap v5230: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:42:46.776 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T15:42:46.776508+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T15:42:46.777 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T15:42:46.777 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8960 8 365281 365273 0.318469 0 - 0.0299513 2026-02-23T15:42:47.007 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:42:47.124 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:42:47.233 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:42:47.231+0000 7f3733806640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:42:47.233 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:42:47.231+0000 7f3733806640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:42:47.343 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:42:47.343 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":333,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:41:45.196353+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:41:36.954923+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:39:14.873799+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:40:09.841031+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:42:47.777 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8961 8 365281 365273 0.318433 0 - 0.0299513 2026-02-23T15:42:48.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:42:47 trial068 ceph-mon[38805]: pgmap v5231: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:42:48.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:42:47 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/365968719' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:42:48.777 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8962 8 365281 365273 0.318398 0 - 0.0299513 2026-02-23T15:42:49.777 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8963 8 365281 365273 0.318362 0 - 0.0299513 2026-02-23T15:42:50.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:42:49 trial068 ceph-mon[38805]: pgmap v5232: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:42:50.381 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:42:50.499 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:42:50.600 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:42:50.598+0000 7f3520213640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:42:50.600 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:42:50.598+0000 7f3520213640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:42:50.709 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:42:50.709 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":333,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:41:45.196353+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:41:36.954923+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:39:14.873799+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:40:09.841031+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:42:50.777 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8964 8 365281 365273 0.318327 0 - 0.0299513 2026-02-23T15:42:51.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:42:51 trial068 ceph-mon[38805]: pgmap v5233: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:42:51.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:42:51 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2806152603' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:42:51.777 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8965 8 365281 365273 0.318291 0 - 0.0299513 2026-02-23T15:42:52.777 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8966 8 365281 365273 0.318256 0 - 0.0299513 2026-02-23T15:42:53.747 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:42:53.777 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8967 8 365281 365273 0.31822 0 - 0.0299513 2026-02-23T15:42:53.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:42:53 trial068 ceph-mon[38805]: pgmap v5234: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:42:53.864 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:42:53.965 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:42:53.963+0000 7f2dbe4ee640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:42:53.965 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:42:53.963+0000 7f2dbe4ee640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:42:54.079 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:42:54.080 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":333,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:41:45.196353+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:41:36.954923+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:39:14.873799+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:40:09.841031+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:42:54.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:42:53 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:42:53] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:42:54.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:42:53 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:42:53.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:42:54.777 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8968 8 365281 365273 0.318185 0 - 0.0299513 2026-02-23T15:42:54.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:42:54 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T15:42:54.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:42:54 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T15:42:54.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:42:54 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1843980156' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:42:55.777 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8969 8 365281 365273 0.318149 0 - 0.0299513 2026-02-23T15:42:55.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:42:55 trial068 ceph-mon[38805]: pgmap v5235: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:42:56.777 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8970 8 365281 365273 0.318114 0 - 0.0299513 2026-02-23T15:42:57.118 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:42:57.238 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:42:57.342 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:42:57.340+0000 7f4c53141640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:42:57.342 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:42:57.341+0000 7f4c53141640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:42:57.454 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:42:57.454 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":333,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:41:45.196353+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:41:36.954923+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:39:14.873799+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:40:09.841031+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:42:57.777 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8971 8 365281 365273 0.318078 0 - 0.0299513 2026-02-23T15:42:57.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:42:57 trial068 ceph-mon[38805]: pgmap v5236: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:42:57.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:42:57 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T15:42:57.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:42:57 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T15:42:57.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:42:57 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1060158533' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:42:58.777 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8972 8 365281 365273 0.318043 0 - 0.0299513 2026-02-23T15:42:59.778 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8973 8 365281 365273 0.318007 0 - 0.0299513 2026-02-23T15:42:59.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:42:59 trial068 ceph-mon[38805]: pgmap v5237: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:43:00.492 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:43:00.612 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:43:00.708 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:43:00.707+0000 7f4c36e30640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:43:00.709 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:43:00.707+0000 7f4c36e30640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:43:00.777 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8974 8 365281 365273 0.317972 0 - 0.0299513 2026-02-23T15:43:00.818 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:43:00.819 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":333,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:41:45.196353+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:41:36.954923+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:39:14.873799+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:40:09.841031+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:43:00.819 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:43:00 trial068 ceph-mon[38805]: 5.a scrub starts 2026-02-23T15:43:00.819 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:43:00 trial068 ceph-mon[38805]: 5.a scrub ok 2026-02-23T15:43:01.778 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8975 8 365281 365273 0.317936 0 - 0.0299513 2026-02-23T15:43:01.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:43:01 trial068 ceph-mon[38805]: pgmap v5238: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:43:01.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:43:01 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/563130975' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:43:02.778 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8976 8 365281 365273 0.317901 0 - 0.0299513 2026-02-23T15:43:02.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:43:02 trial068 ceph-mon[38805]: 5.5 scrub starts 2026-02-23T15:43:02.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:43:02 trial068 ceph-mon[38805]: 5.5 scrub ok 2026-02-23T15:43:03.778 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8977 8 365281 365273 0.317866 0 - 0.0299513 2026-02-23T15:43:03.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:43:03 trial068 ceph-mon[38805]: pgmap v5239: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:43:03.855 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:43:03.974 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:43:04.076 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:43:04.075+0000 7f9aff23c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:43:04.077 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:43:04.075+0000 7f9aff23c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:43:04.188 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:43:04.188 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":333,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:41:45.196353+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:41:36.954923+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:39:14.873799+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:40:09.841031+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:43:04.188 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:43:03 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:43:03] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:43:04.265 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:43:03 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:43:03.972Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:43:04.778 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8978 8 365281 365273 0.31783 0 - 0.0299513 2026-02-23T15:43:05.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:43:04 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/4048407965' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:43:05.778 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8979 8 365281 365273 0.317795 0 - 0.0299513 2026-02-23T15:43:06.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:43:05 trial068 ceph-mon[38805]: pgmap v5240: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:43:06.778 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T15:43:06.777981+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T15:43:06.778 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T15:43:06.778 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8980 8 365281 365273 0.317759 0 - 0.0299513 2026-02-23T15:43:07.225 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:43:07.341 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:43:07.445 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:43:07.443+0000 7fb989ad0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:43:07.445 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:43:07.443+0000 7fb989ad0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:43:07.554 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:43:07.554 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":333,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:41:45.196353+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:41:36.954923+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:39:14.873799+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:40:09.841031+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:43:07.778 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8981 8 365281 365273 0.317724 0 - 0.0299513 2026-02-23T15:43:07.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:43:07 trial068 ceph-mon[38805]: pgmap v5241: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:43:07.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:43:07 trial068 ceph-mon[38805]: 5.e scrub starts 2026-02-23T15:43:07.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:43:07 trial068 ceph-mon[38805]: 5.e scrub ok 2026-02-23T15:43:07.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:43:07 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/427889469' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:43:08.778 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8982 8 365281 365273 0.317689 0 - 0.0299513 2026-02-23T15:43:09.778 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8983 8 365281 365273 0.317653 0 - 0.0299513 2026-02-23T15:43:10.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:43:09 trial068 ceph-mon[38805]: pgmap v5242: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:43:10.593 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:43:10.710 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:43:10.778 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8984 8 365281 365273 0.317618 0 - 0.0299513 2026-02-23T15:43:10.814 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:43:10.813+0000 7f178ea6a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:43:10.815 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:43:10.813+0000 7f178ea6a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:43:10.925 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:43:10.925 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":333,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:41:45.196353+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:41:36.954923+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:39:14.873799+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:40:09.841031+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:43:11.778 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8985 8 365281 365273 0.317583 0 - 0.0299513 2026-02-23T15:43:12.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:43:11 trial068 ceph-mon[38805]: pgmap v5243: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:43:12.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:43:11 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3466008806' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:43:12.778 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8986 8 365281 365273 0.317547 0 - 0.0299513 2026-02-23T15:43:13.778 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8987 8 365281 365273 0.317512 0 - 0.0299513 2026-02-23T15:43:13.961 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:43:13.973 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:43:13 trial068 ceph-mon[38805]: pgmap v5244: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:43:14.080 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:43:14.189 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:43:14.187+0000 7fe693027640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:43:14.189 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:43:14.188+0000 7fe693027640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:43:14.299 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:43:14.300 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":333,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:41:45.196353+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:41:36.954923+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:39:14.873799+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:40:09.841031+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:43:14.300 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:43:13 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:43:13] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:43:14.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:43:13 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:43:13.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:43:14.778 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8988 8 365281 365273 0.317477 0 - 0.0299513 2026-02-23T15:43:15.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:43:14 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2870072067' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:43:15.778 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8989 8 365281 365273 0.317441 0 - 0.0299513 2026-02-23T15:43:16.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:43:15 trial068 ceph-mon[38805]: pgmap v5245: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:43:16.779 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8990 8 365281 365273 0.317406 0 - 0.0299513 2026-02-23T15:43:17.337 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:43:17.456 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:43:17.557 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:43:17.555+0000 7fdafcadd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:43:17.557 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:43:17.555+0000 7fdafcadd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:43:17.666 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:43:17.667 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":333,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:41:45.196353+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:41:36.954923+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:39:14.873799+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:40:09.841031+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:43:17.667 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:43:17 trial068 ceph-mon[38805]: pgmap v5246: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:43:17.779 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8991 8 365281 365273 0.317371 0 - 0.0299513 2026-02-23T15:43:18.779 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8992 8 365281 365273 0.317335 0 - 0.0299513 2026-02-23T15:43:19.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:43:18 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2364094118' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:43:19.779 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8993 8 365281 365273 0.3173 0 - 0.0299513 2026-02-23T15:43:20.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:43:19 trial068 ceph-mon[38805]: pgmap v5247: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:43:20.704 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:43:20.779 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8994 8 365281 365273 0.317265 0 - 0.0299513 2026-02-23T15:43:20.822 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:43:20.926 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:43:20.924+0000 7f4e21c0a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:43:20.926 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:43:20.925+0000 7f4e21c0a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:43:21.036 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:43:21.036 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":333,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:41:45.196353+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:41:36.954923+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:39:14.873799+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:40:09.841031+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:43:21.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:43:21 trial068 ceph-mon[38805]: pgmap v5248: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:43:21.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:43:21 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1336117418' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:43:21.779 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8995 8 365281 365273 0.317229 0 - 0.0299513 2026-02-23T15:43:22.779 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8996 8 365281 365273 0.317194 0 - 0.0299513 2026-02-23T15:43:23.779 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8997 8 365281 365273 0.317159 0 - 0.0299513 2026-02-23T15:43:23.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:43:23 trial068 ceph-mon[38805]: pgmap v5249: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:43:24.073 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:43:24.194 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:43:24.238 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:43:23 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:43:23] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:43:24.295 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:43:24.293+0000 7febc7c9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:43:24.295 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:43:24.293+0000 7febc7c9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:43:24.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:43:23 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:43:23.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:43:24.404 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:43:24.405 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":333,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:41:45.196353+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:41:36.954923+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:39:14.873799+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:40:09.841031+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:43:24.779 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8998 8 365281 365273 0.317124 0 - 0.0299513 2026-02-23T15:43:24.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:43:24 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1488240118' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:43:25.779 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 8999 8 365281 365273 0.317088 0 - 0.0299513 2026-02-23T15:43:25.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:43:25 trial068 ceph-mon[38805]: pgmap v5250: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:43:26.779 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T15:43:26.779428+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T15:43:26.779 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T15:43:26.780 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9000 8 365281 365273 0.317053 0 - 0.0299513 2026-02-23T15:43:26.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:43:26 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2026-02-23T15:43:27.442 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:43:27.559 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:43:27.660 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:43:27.658+0000 7f268da5f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:43:27.660 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:43:27.659+0000 7f268da5f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:43:27.744 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:43:27 trial068 ceph-mon[38805]: pgmap v5251: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:43:27.745 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:43:27 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-23T15:43:27.745 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:43:27 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2026-02-23T15:43:27.745 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:43:27 trial068 ceph-mon[38805]: pgmap v5252: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:43:27.745 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:43:27 trial068 ceph-mon[38805]: pgmap v5253: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:43:27.745 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:43:27 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T15:43:27.745 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:43:27 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2026-02-23T15:43:27.770 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:43:27.770 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":333,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:41:45.196353+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:41:36.954923+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:39:14.873799+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:40:09.841031+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:43:27.779 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9001 8 365281 365273 0.317018 0 - 0.0299513 2026-02-23T15:43:28.780 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9002 8 365281 365273 0.316983 0 - 0.0299513 2026-02-23T15:43:29.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:43:28 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3292884773' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:43:29.780 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9003 8 365281 365273 0.316948 0 - 0.0299513 2026-02-23T15:43:30.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:43:29 trial068 ceph-mon[38805]: pgmap v5254: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:43:30.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:43:29 trial068 ceph-mon[38805]: 5.7 scrub starts 2026-02-23T15:43:30.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:43:29 trial068 ceph-mon[38805]: 5.7 scrub ok 2026-02-23T15:43:30.780 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9004 8 365281 365273 0.316912 0 - 0.0299513 2026-02-23T15:43:30.807 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:43:30.925 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:43:31.021 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:43:31.019+0000 7f8ca4ea7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:43:31.021 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:43:31.020+0000 7f8ca4ea7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:43:31.131 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:43:31.131 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":333,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:41:45.196353+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:41:36.954923+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:39:14.873799+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:40:09.841031+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:43:31.780 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9005 8 365281 365273 0.316877 0 - 0.0299513 2026-02-23T15:43:32.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:43:31 trial068 ceph-mon[38805]: pgmap v5255: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:43:32.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:43:31 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/372827837' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:43:32.780 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9006 8 365281 365273 0.316842 0 - 0.0299513 2026-02-23T15:43:33.780 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9007 8 365281 365273 0.316807 0 - 0.0299513 2026-02-23T15:43:34.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:43:33 trial068 ceph-mon[38805]: pgmap v5256: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:43:34.094 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:43:33 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:43:33] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:43:34.168 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:43:34.265 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:43:33 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:43:33.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:43:34.286 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:43:34.386 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:43:34.384+0000 7f62430d9640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:43:34.386 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:43:34.385+0000 7f62430d9640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:43:34.499 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:43:34.499 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":333,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:41:45.196353+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:41:36.954923+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:39:14.873799+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:40:09.841031+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:43:34.780 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9008 8 365281 365273 0.316772 0 - 0.0299513 2026-02-23T15:43:35.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:43:34 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3677147877' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:43:35.780 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9009 8 365281 365273 0.316737 0 - 0.0299513 2026-02-23T15:43:36.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:43:35 trial068 ceph-mon[38805]: pgmap v5257: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:43:36.780 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9010 8 365281 365273 0.316701 0 - 0.0299513 2026-02-23T15:43:37.537 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:43:37.656 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:43:37.758 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:43:37.756+0000 7fb84a0fd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:43:37.758 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:43:37.757+0000 7fb84a0fd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:43:37.780 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9011 8 365281 365273 0.316666 0 - 0.0299513 2026-02-23T15:43:37.871 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:43:37.871 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":333,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:41:45.196353+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:41:36.954923+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:39:14.873799+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:40:09.841031+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:43:38.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:43:37 trial068 ceph-mon[38805]: pgmap v5258: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:43:38.780 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9012 8 365281 365273 0.316631 0 - 0.0299513 2026-02-23T15:43:39.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:43:38 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2455112162' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:43:39.780 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9013 8 365281 365273 0.316596 0 - 0.0299513 2026-02-23T15:43:40.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:43:39 trial068 ceph-mon[38805]: pgmap v5259: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:43:40.781 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9014 8 365281 365273 0.316561 0 - 0.0299513 2026-02-23T15:43:40.909 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:43:41.028 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:43:41.122 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:43:41.120+0000 7f05c7f64640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:43:41.122 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:43:41.120+0000 7f05c7f64640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:43:41.231 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:43:41.232 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":333,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:41:45.196353+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:41:36.954923+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:39:14.873799+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:40:09.841031+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:43:41.781 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9015 8 365281 365273 0.316526 0 - 0.0299513 2026-02-23T15:43:42.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:43:41 trial068 ceph-mon[38805]: pgmap v5260: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:43:42.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:43:41 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/460626069' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:43:42.781 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9016 8 365281 365273 0.316491 0 - 0.0299513 2026-02-23T15:43:43.781 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9017 8 365281 365273 0.316455 0 - 0.0299513 2026-02-23T15:43:44.292 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:43:44.317 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:43:43 trial068 ceph-mon[38805]: pgmap v5261: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:43:44.317 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:43:43 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:43:43] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:43:44.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:43:43 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:43:43.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:43:44.411 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:43:44.514 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:43:44.513+0000 7f69b473c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:43:44.515 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:43:44.513+0000 7f69b473c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:43:44.627 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:43:44.628 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":333,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:41:45.196353+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:41:36.954923+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:39:14.873799+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:40:09.841031+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:43:44.781 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9018 8 365281 365273 0.31642 0 - 0.0299513 2026-02-23T15:43:45.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:43:44 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T15:43:45.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:43:44 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T15:43:45.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:43:44 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/4224668831' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:43:45.781 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9019 8 365281 365273 0.316385 0 - 0.0299513 2026-02-23T15:43:46.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:43:45 trial068 ceph-mon[38805]: pgmap v5262: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:43:46.781 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T15:43:46.781011+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T15:43:46.781 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T15:43:46.781 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9020 8 365281 365273 0.31635 0 - 0.0299513 2026-02-23T15:43:47.665 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:43:47.781 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9021 8 365281 365273 0.316315 0 - 0.0299513 2026-02-23T15:43:47.783 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:43:47.885 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:43:47.883+0000 7f5623d2f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:43:47.885 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:43:47.883+0000 7f5623d2f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:43:47.994 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:43:47.995 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":333,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:41:45.196353+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:41:36.954923+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:39:14.873799+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:40:09.841031+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:43:47.995 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:43:47 trial068 ceph-mon[38805]: pgmap v5263: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:43:48.781 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9022 8 365281 365273 0.31628 0 - 0.0299513 2026-02-23T15:43:49.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:43:48 trial068 ceph-mon[38805]: 5.8 scrub starts 2026-02-23T15:43:49.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:43:48 trial068 ceph-mon[38805]: 5.8 scrub ok 2026-02-23T15:43:49.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:43:48 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1156981075' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:43:49.781 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9023 8 365281 365273 0.316245 0 - 0.0299513 2026-02-23T15:43:50.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:43:49 trial068 ceph-mon[38805]: pgmap v5264: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:43:50.781 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9024 8 365281 365273 0.31621 0 - 0.0299513 2026-02-23T15:43:51.031 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:43:51.149 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:43:51.252 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:43:51.251+0000 7f5d3a364640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:43:51.253 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:43:51.251+0000 7f5d3a364640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:43:51.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:43:51 trial068 ceph-mon[38805]: pgmap v5265: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:43:51.362 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:43:51.362 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":333,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:41:45.196353+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:41:36.954923+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:39:14.873799+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:40:09.841031+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:43:51.782 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9025 8 365281 365273 0.316175 0 - 0.0299513 2026-02-23T15:43:52.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:43:52 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2290137518' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:43:52.782 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9026 8 365281 365273 0.31614 0 - 0.0299513 2026-02-23T15:43:53.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:43:53 trial068 ceph-mon[38805]: pgmap v5266: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:43:53.782 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9027 8 365281 365273 0.316105 0 - 0.0299513 2026-02-23T15:43:54.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:43:53 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:43:53] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:43:54.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:43:53 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:43:53.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:43:54.399 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:43:54.514 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:43:54.612 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:43:54.610+0000 7f5d2b6b8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:43:54.612 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:43:54.611+0000 7f5d2b6b8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:43:54.722 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:43:54.722 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":333,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:41:45.196353+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:41:36.954923+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:39:14.873799+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:40:09.841031+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:43:54.782 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9028 8 365281 365273 0.31607 0 - 0.0299513 2026-02-23T15:43:55.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:43:54 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/16062440' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:43:55.782 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9029 8 365281 365273 0.316035 0 - 0.0299513 2026-02-23T15:43:56.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:43:55 trial068 ceph-mon[38805]: pgmap v5267: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:43:56.782 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9030 8 365281 365273 0.316 0 - 0.0299513 2026-02-23T15:43:57.760 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:43:57.782 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9031 8 365281 365273 0.315965 0 - 0.0299513 2026-02-23T15:43:57.878 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:43:57.980 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:43:57.978+0000 7fc53bad7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:43:57.980 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:43:57.978+0000 7fc53bad7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:43:58.089 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:43:58.089 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":333,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:41:45.196353+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:41:36.954923+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:39:14.873799+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:40:09.841031+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:43:58.090 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:43:57 trial068 ceph-mon[38805]: pgmap v5268: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:43:58.782 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9032 8 365281 365273 0.31593 0 - 0.0299513 2026-02-23T15:43:59.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:43:58 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2849850837' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:43:59.782 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9033 8 365281 365273 0.315895 0 - 0.0299513 2026-02-23T15:44:00.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:43:59 trial068 ceph-mon[38805]: pgmap v5269: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:44:00.782 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9034 8 365281 365273 0.31586 0 - 0.0299513 2026-02-23T15:44:01.127 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:44:01.246 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:44:01.349 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:44:01.347+0000 7f034c3fa640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:44:01.349 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:44:01.348+0000 7f034c3fa640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:44:01.459 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:44:01.459 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":333,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:41:45.196353+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:41:36.954923+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:39:14.873799+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:40:09.841031+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:44:01.459 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:44:01 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T15:44:01.460 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:44:01 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T15:44:01.460 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:44:01 trial068 ceph-mon[38805]: pgmap v5270: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:44:01.782 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9035 8 365281 365273 0.315825 0 - 0.0299513 2026-02-23T15:44:02.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:44:02 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3636718684' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:44:02.782 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9036 8 365281 365273 0.31579 0 - 0.0299513 2026-02-23T15:44:03.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:44:03 trial068 ceph-mon[38805]: pgmap v5271: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:44:03.782 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9037 8 365281 365273 0.315755 0 - 0.0299513 2026-02-23T15:44:04.264 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:44:03 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:44:03.974Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:44:04.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:44:03 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:44:03] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:44:04.496 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:44:04.614 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:44:04.712 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:44:04.710+0000 7fc0c68ef640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:44:04.712 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:44:04.710+0000 7fc0c68ef640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:44:04.782 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9038 8 365281 365273 0.31572 0 - 0.0299513 2026-02-23T15:44:04.820 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:44:04.820 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":333,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:41:45.196353+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:41:36.954923+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:39:14.873799+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:40:09.841031+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:44:05.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:44:04 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2528987819' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:44:05.782 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9039 8 365281 365273 0.315685 0 - 0.0299513 2026-02-23T15:44:06.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:44:05 trial068 ceph-mon[38805]: pgmap v5272: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:44:06.782 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T15:44:06.782501+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T15:44:06.783 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T15:44:06.783 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9040 8 365281 365273 0.31565 0 - 0.0299513 2026-02-23T15:44:07.783 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9041 8 365281 365273 0.315615 0 - 0.0299513 2026-02-23T15:44:07.857 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:44:07.973 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:44:08.073 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:44:08.071+0000 7fc20c0f5640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:44:08.073 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:44:08.071+0000 7fc20c0f5640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:44:08.182 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:44:08.182 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":333,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:41:45.196353+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:41:36.954923+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:39:14.873799+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:40:09.841031+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:44:08.182 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:44:07 trial068 ceph-mon[38805]: pgmap v5273: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:44:08.783 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9042 8 365281 365273 0.315581 0 - 0.0299513 2026-02-23T15:44:09.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:44:08 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2624549601' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:44:09.783 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9043 8 365281 365273 0.315546 0 - 0.0299513 2026-02-23T15:44:10.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:44:09 trial068 ceph-mon[38805]: pgmap v5274: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:44:10.783 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9044 8 365281 365273 0.315511 0 - 0.0299513 2026-02-23T15:44:11.218 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:44:11.338 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:44:11.438 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:44:11.436+0000 7f91a4a90640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:44:11.438 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:44:11.437+0000 7f91a4a90640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:44:11.548 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:44:11.549 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":333,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:41:45.196353+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:41:36.954923+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:39:14.873799+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:40:09.841031+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:44:11.783 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9045 8 365281 365273 0.315476 0 - 0.0299513 2026-02-23T15:44:12.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:44:11 trial068 ceph-mon[38805]: pgmap v5275: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:44:12.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:44:11 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1755719455' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:44:12.783 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9046 8 365281 365273 0.315441 0 - 0.0299513 2026-02-23T15:44:13.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:44:12 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T15:44:13.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:44:12 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T15:44:13.783 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9047 8 365281 365273 0.315406 0 - 0.0299513 2026-02-23T15:44:14.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:44:13 trial068 ceph-mon[38805]: pgmap v5276: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:44:14.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:44:13 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:44:13] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:44:14.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:44:13 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:44:13.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:44:14.585 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:44:14.703 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:44:14.783 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9048 8 365281 365273 0.315371 0 - 0.0299513 2026-02-23T15:44:14.803 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:44:14.801+0000 7f45a7365640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:44:14.803 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:44:14.801+0000 7f45a7365640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:44:14.911 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:44:14.911 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":333,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:41:45.196353+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:41:36.954923+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:39:14.873799+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:40:09.841031+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:44:15.783 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9049 8 365281 365273 0.315336 0 - 0.0299513 2026-02-23T15:44:16.252 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:44:15 trial068 ceph-mon[38805]: pgmap v5277: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:44:16.252 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:44:15 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3578671989' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:44:16.783 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9050 8 365281 365273 0.315302 0 - 0.0299513 2026-02-23T15:44:17.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:44:16 trial068 ceph-mon[38805]: 5.e scrub starts 2026-02-23T15:44:17.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:44:16 trial068 ceph-mon[38805]: 5.e scrub ok 2026-02-23T15:44:17.783 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9051 8 365281 365273 0.315267 0 - 0.0299513 2026-02-23T15:44:17.949 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:44:18.071 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:44:18.177 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:44:17 trial068 ceph-mon[38805]: pgmap v5278: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:44:18.177 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:44:18.175+0000 7f1f71014640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:44:18.177 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:44:18.175+0000 7f1f71014640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:44:18.286 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:44:18.286 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":333,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:41:45.196353+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:41:36.954923+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:39:14.873799+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:40:09.841031+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:44:18.784 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9052 8 365281 365273 0.315232 0 - 0.0299513 2026-02-23T15:44:19.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:44:18 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1865940963' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:44:19.784 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9053 8 365281 365273 0.315197 0 - 0.0299513 2026-02-23T15:44:20.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:44:19 trial068 ceph-mon[38805]: pgmap v5279: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:44:20.784 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9054 8 365281 365273 0.315162 0 - 0.0299513 2026-02-23T15:44:21.325 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:44:21.443 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:44:21.489 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:44:21 trial068 ceph-mon[38805]: 5.a scrub starts 2026-02-23T15:44:21.489 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:44:21 trial068 ceph-mon[38805]: 5.a scrub ok 2026-02-23T15:44:21.490 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:44:21 trial068 ceph-mon[38805]: pgmap v5280: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:44:21.547 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:44:21.545+0000 7f3a413a9640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:44:21.547 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:44:21.546+0000 7f3a413a9640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:44:21.659 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:44:21.659 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":333,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:41:45.196353+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:41:36.954923+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:39:14.873799+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:40:09.841031+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:44:21.784 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9055 8 365281 365273 0.315127 0 - 0.0299513 2026-02-23T15:44:22.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:44:22 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2987953796' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:44:22.784 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9056 8 365281 365273 0.315093 0 - 0.0299513 2026-02-23T15:44:23.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:44:23 trial068 ceph-mon[38805]: 5.5 scrub starts 2026-02-23T15:44:23.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:44:23 trial068 ceph-mon[38805]: 5.5 scrub ok 2026-02-23T15:44:23.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:44:23 trial068 ceph-mon[38805]: pgmap v5281: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:44:23.784 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9057 8 365281 365273 0.315058 0 - 0.0299513 2026-02-23T15:44:24.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:44:23 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:44:23] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:44:24.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:44:23 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:44:23.974Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:44:24.695 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:44:24.784 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9058 8 365281 365273 0.315023 0 - 0.0299513 2026-02-23T15:44:24.812 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:44:24.912 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:44:24.911+0000 7f067c92e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:44:24.913 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:44:24.911+0000 7f067c92e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:44:25.021 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:44:25.022 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":334,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:44:24.222890+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:41:36.954923+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:44:11.881302+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:40:09.841031+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:44:25.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:44:25 trial068 ceph-mon[38805]: osdmap e334: 4 total, 3 up, 4 in 2026-02-23T15:44:25.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:44:25 trial068 ceph-mon[38805]: pgmap v5283: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:44:25.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:44:25 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/4116710527' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:44:25.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:44:25 trial068 ceph-mon[38805]: osdmap e335: 4 total, 3 up, 4 in 2026-02-23T15:44:25.784 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9059 8 365281 365273 0.314988 0 - 0.0299513 2026-02-23T15:44:26.784 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T15:44:26.784087+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T15:44:26.784 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T15:44:26.784 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9060 8 365281 365273 0.314954 0 - 0.0299513 2026-02-23T15:44:27.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:44:26 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2026-02-23T15:44:27.784 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9061 8 365281 365273 0.314919 0 - 0.0299513 2026-02-23T15:44:28.059 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:44:28.176 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:44:28.200 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:44:27 trial068 ceph-mon[38805]: pgmap v5285: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:44:28.200 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:44:27 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-23T15:44:28.200 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:44:27 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2026-02-23T15:44:28.201 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:44:27 trial068 ceph-mon[38805]: pgmap v5286: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:44:28.201 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:44:27 trial068 ceph-mon[38805]: pgmap v5287: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:44:28.201 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:44:27 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T15:44:28.201 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:44:27 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2026-02-23T15:44:28.288 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:44:28.286+0000 7f4ec39d9640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:44:28.288 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:44:28.286+0000 7f4ec39d9640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:44:28.398 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:44:28.398 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":335,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:44:25.215114+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:41:36.954923+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:44:11.881302+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:40:09.841031+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:44:28.784 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9062 8 365281 365273 0.314884 0 - 0.0299513 2026-02-23T15:44:29.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:44:28 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1214901576' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:44:29.784 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9063 8 365281 365273 0.314849 0 - 0.0299513 2026-02-23T15:44:30.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:44:29 trial068 ceph-mon[38805]: pgmap v5288: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:44:30.784 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9064 8 365281 365273 0.314815 0 - 0.0299513 2026-02-23T15:44:31.436 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:44:31.558 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:44:31.658 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:44:31.657+0000 7f76bb713640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:44:31.659 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:44:31.657+0000 7f76bb713640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:44:31.768 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:44:31.768 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":335,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:44:25.215114+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:41:36.954923+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:44:11.881302+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:40:09.841031+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:44:31.784 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9065 8 365281 365273 0.31478 0 - 0.0299513 2026-02-23T15:44:32.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:44:32 trial068 ceph-mon[38805]: pgmap v5289: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:44:32.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:44:32 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2240414280' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:44:32.784 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9066 8 365281 365273 0.314745 0 - 0.0299513 2026-02-23T15:44:33.785 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9067 8 365281 365273 0.31471 0 - 0.0299513 2026-02-23T15:44:34.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:44:34 trial068 ceph-mon[38805]: pgmap v5290: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:44:34.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:44:33 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:44:33] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:44:34.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:44:33 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:44:33.972Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:44:34.785 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9068 8 365281 365273 0.314676 0 - 0.0299513 2026-02-23T15:44:34.806 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:44:34.924 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:44:35.025 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:44:35.024+0000 7f38d57b7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:44:35.026 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:44:35.024+0000 7f38d57b7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:44:35.135 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:44:35.135 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":335,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:44:25.215114+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:41:36.954923+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:44:11.881302+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:40:09.841031+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:44:35.785 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9069 8 365281 365273 0.314641 0 - 0.0299513 2026-02-23T15:44:36.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:44:36 trial068 ceph-mon[38805]: pgmap v5291: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:44:36.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:44:36 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3272023645' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:44:36.785 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9070 8 365281 365273 0.314606 0 - 0.0299513 2026-02-23T15:44:37.785 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9071 8 365281 365273 0.314572 0 - 0.0299513 2026-02-23T15:44:38.174 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:44:38.292 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:44:38.395 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:44:38.393+0000 7f3ca9872640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:44:38.395 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:44:38.393+0000 7f3ca9872640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:44:38.396 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:44:38 trial068 ceph-mon[38805]: pgmap v5292: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:44:38.507 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:44:38.508 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":335,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:44:25.215114+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:41:36.954923+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:44:11.881302+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:40:09.841031+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:44:38.785 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9072 8 365281 365273 0.314537 0 - 0.0299513 2026-02-23T15:44:39.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:44:39 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2266710335' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:44:39.785 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9073 8 365281 365273 0.314502 0 - 0.0299513 2026-02-23T15:44:40.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:44:40 trial068 ceph-mon[38805]: pgmap v5293: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:44:40.785 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9074 8 365281 365273 0.314468 0 - 0.0299513 2026-02-23T15:44:41.545 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:44:41.571 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:44:41 trial068 ceph-mon[38805]: pgmap v5294: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:44:41.665 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:44:41.761 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:44:41.759+0000 7f0243f56640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:44:41.761 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:44:41.759+0000 7f0243f56640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:44:41.785 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9075 8 365281 365273 0.314433 0 - 0.0299513 2026-02-23T15:44:41.869 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:44:41.869 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":335,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:44:25.215114+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:41:36.954923+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:44:11.881302+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:40:09.841031+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:44:42.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:44:42 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/111447658' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:44:42.785 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9076 8 365281 365273 0.314398 0 - 0.0299513 2026-02-23T15:44:43.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:44:43 trial068 ceph-mon[38805]: pgmap v5295: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:44:43.785 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9077 8 365281 365273 0.314364 0 - 0.0299513 2026-02-23T15:44:44.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:44:43 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:44:43] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:44:44.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:44:43 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:44:43.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:44:44.785 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9078 8 365281 365273 0.314329 0 - 0.0299513 2026-02-23T15:44:44.907 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:44:45.028 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:44:45.126 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:44:45.124+0000 7fb4f2152640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:44:45.126 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:44:45.124+0000 7fb4f2152640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:44:45.235 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:44:45.235 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":335,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:44:25.215114+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:41:36.954923+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:44:11.881302+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:40:09.841031+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:44:45.785 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9079 8 365281 365273 0.314294 0 - 0.0299513 2026-02-23T15:44:46.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:44:46 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T15:44:46.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:44:46 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T15:44:46.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:44:46 trial068 ceph-mon[38805]: pgmap v5296: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:44:46.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:44:46 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1486242314' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:44:46.785 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T15:44:46.785371+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T15:44:46.786 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T15:44:46.786 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9080 8 365281 365273 0.31426 0 - 0.0299513 2026-02-23T15:44:47.785 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9081 8 365281 365273 0.314225 0 - 0.0299513 2026-02-23T15:44:48.272 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:44:48.391 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:44:48.413 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:44:48 trial068 ceph-mon[38805]: pgmap v5297: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:44:48.488 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:44:48.486+0000 7fa50c395640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:44:48.488 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:44:48.486+0000 7fa50c395640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:44:48.597 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:44:48.598 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":335,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:44:25.215114+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:41:36.954923+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:44:11.881302+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:40:09.841031+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:44:48.785 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9082 8 365281 365273 0.314191 0 - 0.0299513 2026-02-23T15:44:49.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:44:49 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/561506908' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:44:49.785 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9083 8 365281 365273 0.314156 0 - 0.0299513 2026-02-23T15:44:50.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:44:50 trial068 ceph-mon[38805]: pgmap v5298: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:44:50.785 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9084 8 365281 365273 0.314121 0 - 0.0299513 2026-02-23T15:44:51.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:44:51 trial068 ceph-mon[38805]: pgmap v5299: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:44:51.635 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:44:51.753 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:44:51.785 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9085 8 365281 365273 0.314087 0 - 0.0299513 2026-02-23T15:44:51.860 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:44:51.858+0000 7f85c12bc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:44:51.860 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:44:51.859+0000 7f85c12bc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:44:51.969 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:44:51.969 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":335,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:44:25.215114+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:41:36.954923+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:44:11.881302+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:40:09.841031+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:44:52.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:44:52 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/595083376' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:44:52.785 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9086 8 365281 365273 0.314052 0 - 0.0299513 2026-02-23T15:44:53.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:44:53 trial068 ceph-mon[38805]: pgmap v5300: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:44:53.786 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9087 8 365281 365273 0.314018 0 - 0.0299513 2026-02-23T15:44:54.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:44:53 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:44:53] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:44:54.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:44:53 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:44:53.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:44:54.786 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9088 8 365281 365273 0.313983 0 - 0.0299513 2026-02-23T15:44:55.006 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:44:55.124 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:44:55.228 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:44:55.226+0000 7f76db840640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:44:55.228 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:44:55.226+0000 7f76db840640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:44:55.337 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:44:55.337 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":335,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:44:25.215114+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:41:36.954923+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:44:11.881302+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:40:09.841031+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:44:55.786 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9089 8 365281 365273 0.313949 0 - 0.0299513 2026-02-23T15:44:56.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:44:56 trial068 ceph-mon[38805]: pgmap v5301: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:44:56.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:44:56 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/830504363' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:44:56.786 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9090 8 365281 365273 0.313914 0 - 0.0299513 2026-02-23T15:44:57.786 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9091 8 365281 365273 0.31388 0 - 0.0299513 2026-02-23T15:44:58.375 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:44:58.395 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:44:58 trial068 ceph-mon[38805]: pgmap v5302: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:44:58.493 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:44:58.592 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:44:58.590+0000 7f59a1e6c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:44:58.592 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:44:58.590+0000 7f59a1e6c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:44:58.702 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:44:58.702 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":335,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:44:25.215114+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:41:36.954923+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:44:11.881302+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:40:09.841031+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:44:58.786 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9092 8 365281 365273 0.313845 0 - 0.0299513 2026-02-23T15:44:59.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:44:59 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/875026171' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:44:59.786 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9093 8 365281 365273 0.313811 0 - 0.0299513 2026-02-23T15:45:00.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:45:00 trial068 ceph-mon[38805]: pgmap v5303: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:45:00.786 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9094 8 365281 365273 0.313776 0 - 0.0299513 2026-02-23T15:45:01.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:45:01 trial068 ceph-mon[38805]: pgmap v5304: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:45:01.740 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:45:01.786 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9095 8 365281 365273 0.313742 0 - 0.0299513 2026-02-23T15:45:01.858 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:45:01.960 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:45:01.958+0000 7f900f8c5640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:45:01.960 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:45:01.958+0000 7f900f8c5640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:45:02.072 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:45:02.072 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":335,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:44:25.215114+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:41:36.954923+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:44:11.881302+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:40:09.841031+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:45:02.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:45:02 trial068 ceph-mon[38805]: 5.7 scrub starts 2026-02-23T15:45:02.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:45:02 trial068 ceph-mon[38805]: 5.7 scrub ok 2026-02-23T15:45:02.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:45:02 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2661175064' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:45:02.786 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9096 8 365281 365273 0.313707 0 - 0.0299513 2026-02-23T15:45:03.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:45:03 trial068 ceph-mon[38805]: pgmap v5305: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:45:03.786 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9097 8 365281 365273 0.313673 0 - 0.0299513 2026-02-23T15:45:04.224 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:45:03 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:45:03] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:45:04.265 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:45:03 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:45:03.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:45:04.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:45:04 trial068 ceph-mon[38805]: 5.8 scrub starts 2026-02-23T15:45:04.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:45:04 trial068 ceph-mon[38805]: 5.8 scrub ok 2026-02-23T15:45:04.787 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9098 8 365281 365273 0.313638 0 - 0.0299513 2026-02-23T15:45:05.109 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:45:05.228 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:45:05.329 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:45:05.327+0000 7f4575968640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:45:05.329 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:45:05.327+0000 7f4575968640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:45:05.440 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:45:05.440 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":335,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:44:25.215114+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:41:36.954923+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:44:11.881302+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:40:09.841031+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:45:05.441 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:45:05 trial068 ceph-mon[38805]: pgmap v5306: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:45:05.787 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9099 8 365281 365273 0.313604 0 - 0.0299513 2026-02-23T15:45:06.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:45:06 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3469517940' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:45:06.787 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T15:45:06.786680+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T15:45:06.787 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T15:45:06.787 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9100 8 365281 365273 0.313569 0 - 0.0299513 2026-02-23T15:45:07.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:45:07 trial068 ceph-mon[38805]: pgmap v5307: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:45:07.787 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9101 8 365281 365273 0.313535 0 - 0.0299513 2026-02-23T15:45:08.477 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:45:08.596 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:45:08.696 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:45:08.694+0000 7fa956471640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:45:08.696 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:45:08.694+0000 7fa956471640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:45:08.787 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9102 8 365281 365273 0.3135 0 - 0.0299513 2026-02-23T15:45:08.807 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:45:08.807 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":335,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:44:25.215114+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:41:36.954923+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:44:11.881302+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:40:09.841031+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:45:09.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:45:08 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1538023947' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:45:09.787 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9103 8 365281 365273 0.313466 0 - 0.0299513 2026-02-23T15:45:10.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:45:09 trial068 ceph-mon[38805]: pgmap v5308: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:45:10.787 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9104 8 365281 365273 0.313431 0 - 0.0299513 2026-02-23T15:45:11.787 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9105 8 365281 365273 0.313397 0 - 0.0299513 2026-02-23T15:45:11.844 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:45:11.962 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:45:12.058 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:45:12.056+0000 7f216336f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:45:12.058 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:45:12.056+0000 7f216336f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:45:12.171 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:45:12.171 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":335,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:44:25.215114+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:41:36.954923+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:44:11.881302+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:40:09.841031+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:45:12.172 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:45:12 trial068 ceph-mon[38805]: pgmap v5309: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:45:12.787 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9106 8 365281 365273 0.313363 0 - 0.0299513 2026-02-23T15:45:13.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:45:13 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1006436069' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:45:13.787 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9107 8 365281 365273 0.313328 0 - 0.0299513 2026-02-23T15:45:14.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:45:14 trial068 ceph-mon[38805]: pgmap v5310: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:45:14.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:45:13 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:45:13] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:45:14.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:45:13 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:45:13.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:45:14.787 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9108 8 365281 365273 0.313294 0 - 0.0299513 2026-02-23T15:45:15.210 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:45:15.328 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:45:15.425 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:45:15.423+0000 7f0eef941640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:45:15.425 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:45:15.423+0000 7f0eef941640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:45:15.536 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:45:15.536 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":335,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:44:25.215114+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:41:36.954923+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:44:11.881302+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:40:09.841031+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:45:15.787 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9109 8 365281 365273 0.313259 0 - 0.0299513 2026-02-23T15:45:16.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:45:16 trial068 ceph-mon[38805]: pgmap v5311: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:45:16.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:45:16 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3450707022' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:45:16.787 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9110 8 365281 365273 0.313225 0 - 0.0299513 2026-02-23T15:45:17.787 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9111 8 365281 365273 0.313191 0 - 0.0299513 2026-02-23T15:45:18.576 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:45:18.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:45:18 trial068 ceph-mon[38805]: pgmap v5312: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:45:18.693 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:45:18.787 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9112 8 365281 365273 0.313156 0 - 0.0299513 2026-02-23T15:45:18.790 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:45:18.789+0000 7f6156546640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:45:18.790 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:45:18.789+0000 7f6156546640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:45:18.901 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:45:18.901 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":335,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:44:25.215114+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:41:36.954923+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:44:11.881302+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:40:09.841031+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:45:19.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:45:19 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3163228041' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:45:19.787 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9113 8 365281 365273 0.313122 0 - 0.0299513 2026-02-23T15:45:20.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:45:20 trial068 ceph-mon[38805]: pgmap v5313: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:45:20.787 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9114 8 365281 365273 0.313087 0 - 0.0299513 2026-02-23T15:45:21.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:45:21 trial068 ceph-mon[38805]: pgmap v5314: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:45:21.787 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9115 8 365281 365273 0.313053 0 - 0.0299513 2026-02-23T15:45:21.938 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:45:22.054 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:45:22.153 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:45:22.151+0000 7f0ff09dc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:45:22.153 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:45:22.151+0000 7f0ff09dc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:45:22.261 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:45:22.261 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":335,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:44:25.215114+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:41:36.954923+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:44:11.881302+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:40:09.841031+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:45:22.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:45:22 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3279687843' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:45:22.787 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9116 8 365281 365273 0.313019 0 - 0.0299513 2026-02-23T15:45:23.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:45:23 trial068 ceph-mon[38805]: pgmap v5315: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:45:23.788 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9117 8 365281 365273 0.312984 0 - 0.0299513 2026-02-23T15:45:24.315 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:45:23 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:45:23] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:45:24.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:45:23 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:45:23.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:45:24.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:45:24 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T15:45:24.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:45:24 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T15:45:24.788 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9118 8 365281 365273 0.31295 0 - 0.0299513 2026-02-23T15:45:25.298 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:45:25.418 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:45:25.516 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:45:25.514+0000 7ff1cef04640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:45:25.516 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:45:25.515+0000 7ff1cef04640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:45:25.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:45:25 trial068 ceph-mon[38805]: pgmap v5316: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:45:25.625 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:45:25.625 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":335,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:44:25.215114+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:41:36.954923+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:44:11.881302+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:40:09.841031+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:45:25.788 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9119 8 365281 365273 0.312916 0 - 0.0299513 2026-02-23T15:45:26.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:45:26 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2431868823' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:45:26.788 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T15:45:26.788002+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T15:45:26.788 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T15:45:26.788 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9120 8 365281 365273 0.312882 0 - 0.0299513 2026-02-23T15:45:27.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:45:27 trial068 ceph-mon[38805]: pgmap v5317: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:45:27.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:45:27 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2026-02-23T15:45:27.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:45:27 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-23T15:45:27.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:45:27 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2026-02-23T15:45:27.788 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9121 8 365281 365273 0.312847 0 - 0.0299513 2026-02-23T15:45:28.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:45:28 trial068 ceph-mon[38805]: pgmap v5318: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:45:28.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:45:28 trial068 ceph-mon[38805]: pgmap v5319: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:45:28.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:45:28 trial068 ceph-mon[38805]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s)) 2026-02-23T15:45:28.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:45:28 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T15:45:28.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:45:28 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2026-02-23T15:45:28.662 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:45:28.777 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:45:28.788 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9122 8 365281 365273 0.312813 0 - 0.0299513 2026-02-23T15:45:28.877 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:45:28.876+0000 7fc511169640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:45:28.877 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:45:28.876+0000 7fc511169640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:45:28.987 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:45:28.987 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":335,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:44:25.215114+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:41:36.954923+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:44:11.881302+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:40:09.841031+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:45:29.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:45:29 trial068 ceph-mon[38805]: 5.e scrub starts 2026-02-23T15:45:29.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:45:29 trial068 ceph-mon[38805]: 5.e scrub ok 2026-02-23T15:45:29.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:45:29 trial068 ceph-mon[38805]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2026-02-23T15:45:29.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:45:29 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1440409882' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:45:29.788 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9123 8 365281 365273 0.312779 0 - 0.0299513 2026-02-23T15:45:30.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:45:30 trial068 ceph-mon[38805]: pgmap v5320: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:45:30.788 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9124 8 365281 365273 0.312744 0 - 0.0299513 2026-02-23T15:45:31.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:45:31 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T15:45:31.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:45:31 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T15:45:31.788 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9125 8 365281 365273 0.31271 0 - 0.0299513 2026-02-23T15:45:32.042 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:45:32.161 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:45:32.265 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:45:32.263+0000 7ff91c3d4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:45:32.265 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:45:32.264+0000 7ff91c3d4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:45:32.375 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:45:32.375 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":335,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:44:25.215114+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:41:36.954923+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:44:11.881302+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:40:09.841031+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:45:32.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:45:32 trial068 ceph-mon[38805]: pgmap v5321: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:45:32.788 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9126 8 365281 365273 0.312676 0 - 0.0299513 2026-02-23T15:45:33.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:45:33 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1319653164' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:45:33.788 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9127 8 365281 365273 0.312642 0 - 0.0299513 2026-02-23T15:45:34.265 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:45:33 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:45:33.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:45:34.330 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:45:33 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:45:33] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:45:34.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:45:34 trial068 ceph-mon[38805]: pgmap v5322: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:45:34.788 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9128 8 365281 365273 0.312607 0 - 0.0299513 2026-02-23T15:45:35.413 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:45:35.533 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:45:35.634 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:45:35.633+0000 7f8e5a93f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:45:35.635 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:45:35.633+0000 7f8e5a93f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:45:35.743 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:45:35.743 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":335,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:44:25.215114+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:41:36.954923+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:44:11.881302+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:40:09.841031+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:45:35.789 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9129 8 365281 365273 0.312573 0 - 0.0299513 2026-02-23T15:45:36.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:45:36 trial068 ceph-mon[38805]: pgmap v5323: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:45:36.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:45:36 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2297804595' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:45:36.789 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9130 8 365281 365273 0.312539 0 - 0.0299513 2026-02-23T15:45:37.789 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9131 8 365281 365273 0.312505 0 - 0.0299513 2026-02-23T15:45:38.782 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:45:38.789 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9132 8 365281 365273 0.31247 0 - 0.0299513 2026-02-23T15:45:38.806 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:45:38 trial068 ceph-mon[38805]: pgmap v5324: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:45:38.807 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:45:38 trial068 ceph-mon[38805]: 5.5 scrub starts 2026-02-23T15:45:38.807 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:45:38 trial068 ceph-mon[38805]: 5.5 scrub ok 2026-02-23T15:45:38.899 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:45:39.004 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:45:39.002+0000 7fb643955640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:45:39.004 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:45:39.002+0000 7fb643955640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:45:39.113 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:45:39.113 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":335,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:44:25.215114+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:41:36.954923+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:44:11.881302+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:40:09.841031+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:45:39.789 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9133 8 365281 365273 0.312436 0 - 0.0299513 2026-02-23T15:45:39.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:45:39 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/847930657' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:45:40.789 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9134 8 365281 365273 0.312402 0 - 0.0299513 2026-02-23T15:45:40.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:45:40 trial068 ceph-mon[38805]: pgmap v5325: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:45:41.789 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9135 8 365281 365273 0.312368 0 - 0.0299513 2026-02-23T15:45:42.150 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:45:42.267 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:45:42.370 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:45:42.369+0000 7fd2054d8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:45:42.371 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:45:42.369+0000 7fd2054d8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:45:42.479 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:45:42.479 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":336,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:45:42.374015+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:41:36.954923+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:44:11.881302+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:44:45.023280+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:45:42.479 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:45:42 trial068 ceph-mon[38805]: pgmap v5326: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:45:42.789 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9136 8 365281 365273 0.312334 0 - 0.0299513 2026-02-23T15:45:43.789 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9137 8 365281 365273 0.312299 0 - 0.0299513 2026-02-23T15:45:43.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:45:43 trial068 ceph-mon[38805]: osdmap e336: 4 total, 3 up, 4 in 2026-02-23T15:45:43.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:45:43 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3796521385' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:45:44.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:45:43 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:45:43] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:45:44.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:45:43 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:45:43.974Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:45:44.789 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9138 8 365281 365273 0.312265 0 - 0.0299513 2026-02-23T15:45:44.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:45:44 trial068 ceph-mon[38805]: pgmap v5328: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:45:45.518 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:45:45.638 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:45:45.738 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:45:45.736+0000 7fe377f23640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:45:45.738 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:45:45.736+0000 7fe377f23640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:45:45.789 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9139 8 365281 365273 0.312231 0 - 0.0299513 2026-02-23T15:45:45.847 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:45:45.847 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":337,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:45:45.223960+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:41:36.954923+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:44:11.881302+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:44:45.023280+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:45:46.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:45:46 trial068 ceph-mon[38805]: osdmap e337: 4 total, 3 up, 4 in 2026-02-23T15:45:46.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:45:46 trial068 ceph-mon[38805]: pgmap v5330: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:45:46.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:45:46 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2661648449' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:45:46.789 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T15:45:46.789357+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T15:45:46.790 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T15:45:46.790 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9140 8 365281 365273 0.312197 0 - 0.0299513 2026-02-23T15:45:47.789 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9141 8 365281 365273 0.312163 0 - 0.0299513 2026-02-23T15:45:48.789 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9142 8 365281 365273 0.312129 0 - 0.0299513 2026-02-23T15:45:48.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:45:48 trial068 ceph-mon[38805]: pgmap v5331: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:45:48.886 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:45:49.004 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:45:49.105 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:45:49.103+0000 7f93abb5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:45:49.105 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:45:49.104+0000 7f93abb5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:45:49.214 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:45:49.214 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":337,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:45:45.223960+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:41:36.954923+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:44:11.881302+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:44:45.023280+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:45:49.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:45:49 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3034748160' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:45:49.790 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9143 8 365281 365273 0.312094 0 - 0.0299513 2026-02-23T15:45:50.790 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9144 8 365281 365273 0.31206 0 - 0.0299513 2026-02-23T15:45:50.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:45:50 trial068 ceph-mon[38805]: pgmap v5332: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:45:51.790 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9145 8 365281 365273 0.312026 0 - 0.0299513 2026-02-23T15:45:52.253 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:45:52.371 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:45:52.472 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:45:52.470+0000 7f3bc8720640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:45:52.472 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:45:52.470+0000 7f3bc8720640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:45:52.582 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:45:52.582 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":337,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:45:45.223960+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:41:36.954923+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:44:11.881302+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:44:45.023280+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:45:52.582 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:45:52 trial068 ceph-mon[38805]: pgmap v5333: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:45:52.790 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9146 8 365281 365273 0.311992 0 - 0.0299513 2026-02-23T15:45:53.790 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9147 8 365281 365273 0.311958 0 - 0.0299513 2026-02-23T15:45:53.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:45:53 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1579824966' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:45:54.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:45:53 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:45:53] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:45:54.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:45:53 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:45:53.974Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:45:54.790 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9148 8 365281 365273 0.311924 0 - 0.0299513 2026-02-23T15:45:54.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:45:54 trial068 ceph-mon[38805]: pgmap v5334: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:45:55.620 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:45:55.738 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:45:55.790 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9149 8 365281 365273 0.31189 0 - 0.0299513 2026-02-23T15:45:55.839 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:45:55.837+0000 7fd0d8ae7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:45:55.839 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:45:55.837+0000 7fd0d8ae7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:45:55.947 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:45:55.947 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":337,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:45:45.223960+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:41:36.954923+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:44:11.881302+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:44:45.023280+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:45:56.790 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9150 8 365281 365273 0.311856 0 - 0.0299513 2026-02-23T15:45:56.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:45:56 trial068 ceph-mon[38805]: pgmap v5335: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:45:56.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:45:56 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2948966096' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:45:56.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:45:56 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T15:45:56.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:45:56 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T15:45:57.790 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9151 8 365281 365273 0.311822 0 - 0.0299513 2026-02-23T15:45:58.645 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:45:58 trial068 ceph-mon[38805]: pgmap v5336: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:45:58.790 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9152 8 365281 365273 0.311788 0 - 0.0299513 2026-02-23T15:45:58.983 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:45:59.102 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:45:59.201 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:45:59.199+0000 7f7b43ae0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:45:59.201 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:45:59.199+0000 7f7b43ae0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:45:59.309 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:45:59.309 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":337,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:45:45.223960+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:41:36.954923+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:44:11.881302+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:44:45.023280+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:45:59.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:45:59 trial068 ceph-mon[38805]: 5.a scrub starts 2026-02-23T15:45:59.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:45:59 trial068 ceph-mon[38805]: 5.a scrub ok 2026-02-23T15:45:59.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:45:59 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2271226340' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:45:59.790 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9153 8 365281 365273 0.311753 0 - 0.0299513 2026-02-23T15:46:00.790 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9154 8 365281 365273 0.311719 0 - 0.0299513 2026-02-23T15:46:00.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:46:00 trial068 ceph-mon[38805]: pgmap v5337: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:46:01.790 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9155 8 365281 365273 0.311685 0 - 0.0299513 2026-02-23T15:46:02.346 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:46:02.464 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:46:02.564 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:46:02.562+0000 7fab4c691640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:46:02.565 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:46:02.563+0000 7fab4c691640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:46:02.673 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:46:02.673 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":337,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:45:45.223960+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:41:36.954923+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:44:11.881302+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:44:45.023280+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:46:02.673 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:46:02 trial068 ceph-mon[38805]: pgmap v5338: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:46:02.791 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9156 8 365281 365273 0.311651 0 - 0.0299513 2026-02-23T15:46:03.791 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9157 8 365281 365273 0.311617 0 - 0.0299513 2026-02-23T15:46:03.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:46:03 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2080517603' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:46:04.265 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:46:03 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:46:03.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:46:04.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:46:03 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:46:03] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:46:04.791 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9158 8 365281 365273 0.311583 0 - 0.0299513 2026-02-23T15:46:04.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:46:04 trial068 ceph-mon[38805]: pgmap v5339: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:46:04.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:46:04 trial068 ceph-mon[38805]: 5.7 scrub starts 2026-02-23T15:46:04.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:46:04 trial068 ceph-mon[38805]: 5.7 scrub ok 2026-02-23T15:46:05.711 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:46:05.791 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9159 8 365281 365273 0.311549 0 - 0.0299513 2026-02-23T15:46:05.829 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:46:05.930 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:46:05.928+0000 7f33d060b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:46:05.930 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:46:05.928+0000 7f33d060b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:46:06.040 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:46:06.041 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":337,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:45:45.223960+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:41:36.954923+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:44:11.881302+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:44:45.023280+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:46:06.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:46:06 trial068 ceph-mon[38805]: pgmap v5340: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:46:06.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:46:06 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3561293508' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:46:06.791 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T15:46:06.790824+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T15:46:06.791 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T15:46:06.791 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9160 8 365281 365273 0.311515 0 - 0.0299513 2026-02-23T15:46:07.791 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9161 8 365281 365273 0.311481 0 - 0.0299513 2026-02-23T15:46:08.791 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9162 8 365281 365273 0.311447 0 - 0.0299513 2026-02-23T15:46:08.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:46:08 trial068 ceph-mon[38805]: pgmap v5341: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:46:09.080 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:46:09.198 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:46:09.298 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:46:09.296+0000 7fb607387640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:46:09.298 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:46:09.296+0000 7fb607387640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:46:09.407 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:46:09.408 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":337,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:45:45.223960+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:41:36.954923+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:44:11.881302+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:44:45.023280+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:46:09.791 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9163 8 365281 365273 0.311413 0 - 0.0299513 2026-02-23T15:46:10.791 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9164 8 365281 365273 0.311379 0 - 0.0299513 2026-02-23T15:46:10.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:46:10 trial068 ceph-mon[38805]: pgmap v5342: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:46:10.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:46:10 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3425676589' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:46:11.791 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9165 8 365281 365273 0.311345 0 - 0.0299513 2026-02-23T15:46:12.445 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:46:12.564 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:46:12.661 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:46:12.660+0000 7f21c5000640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:46:12.662 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:46:12 trial068 ceph-mon[38805]: pgmap v5343: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:46:12.662 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:46:12.660+0000 7f21c5000640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:46:12.771 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:46:12.771 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":337,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:45:45.223960+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:41:36.954923+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:44:11.881302+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:44:45.023280+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:46:12.791 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9166 8 365281 365273 0.311311 0 - 0.0299513 2026-02-23T15:46:13.791 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9167 8 365281 365273 0.311277 0 - 0.0299513 2026-02-23T15:46:13.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:46:13 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/4033645821' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:46:13.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:46:13 trial068 ceph-mon[38805]: 5.8 scrub starts 2026-02-23T15:46:13.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:46:13 trial068 ceph-mon[38805]: 5.8 scrub ok 2026-02-23T15:46:14.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:46:13 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:46:13] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:46:14.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:46:13 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:46:13.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:46:14.791 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9168 8 365281 365273 0.311243 0 - 0.0299513 2026-02-23T15:46:14.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:46:14 trial068 ceph-mon[38805]: pgmap v5344: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:46:15.791 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9169 8 365281 365273 0.311209 0 - 0.0299513 2026-02-23T15:46:15.807 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:46:15.927 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:46:16.027 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:46:16.025+0000 7f2144818640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:46:16.027 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:46:16.025+0000 7f2144818640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:46:16.140 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:46:16.141 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":337,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:45:45.223960+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:41:36.954923+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:44:11.881302+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:44:45.023280+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:46:16.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:46:16 trial068 ceph-mon[38805]: pgmap v5345: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:46:16.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:46:16 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/4259331819' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:46:16.791 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9170 8 365281 365273 0.311176 0 - 0.0299513 2026-02-23T15:46:17.792 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9171 8 365281 365273 0.311142 0 - 0.0299513 2026-02-23T15:46:18.792 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9172 8 365281 365273 0.311108 0 - 0.0299513 2026-02-23T15:46:18.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:46:18 trial068 ceph-mon[38805]: pgmap v5346: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:46:19.178 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:46:19.295 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:46:19.393 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:46:19.391+0000 7fd07ddce640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:46:19.393 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:46:19.392+0000 7fd07ddce640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:46:19.502 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:46:19.502 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":337,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:45:45.223960+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:41:36.954923+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:44:11.881302+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:44:45.023280+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:46:19.792 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9173 8 365281 365273 0.311074 0 - 0.0299513 2026-02-23T15:46:20.792 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9174 8 365281 365273 0.31104 0 - 0.0299513 2026-02-23T15:46:20.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:46:20 trial068 ceph-mon[38805]: pgmap v5347: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:46:20.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:46:20 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3110754760' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:46:21.792 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9175 8 365281 365273 0.311006 0 - 0.0299513 2026-02-23T15:46:22.540 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:46:22.660 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:46:22.684 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:46:22 trial068 ceph-mon[38805]: pgmap v5348: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:46:22.759 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:46:22.758+0000 7f52697ae640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:46:22.760 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:46:22.758+0000 7f52697ae640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:46:22.792 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9176 8 365281 365273 0.310972 0 - 0.0299513 2026-02-23T15:46:22.868 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:46:22.868 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":337,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:45:45.223960+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:41:36.954923+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:44:11.881302+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:44:45.023280+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:46:23.792 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9177 8 365281 365273 0.310938 0 - 0.0299513 2026-02-23T15:46:23.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:46:23 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/503099930' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:46:24.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:46:23 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:46:23] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:46:24.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:46:23 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:46:23.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:46:24.792 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9178 8 365281 365273 0.310904 0 - 0.0299513 2026-02-23T15:46:24.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:46:24 trial068 ceph-mon[38805]: pgmap v5349: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:46:25.792 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9179 8 365281 365273 0.31087 0 - 0.0299513 2026-02-23T15:46:25.907 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:46:26.025 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:46:26.126 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:46:26.124+0000 7f08a2a7e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:46:26.126 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:46:26.124+0000 7f08a2a7e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:46:26.237 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:46:26.238 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":337,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:45:45.223960+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:41:36.954923+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:44:11.881302+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:44:45.023280+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:46:26.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:46:26 trial068 ceph-mon[38805]: pgmap v5350: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:46:26.792 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T15:46:26.792279+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T15:46:26.792 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T15:46:26.793 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9180 8 365281 365273 0.310837 0 - 0.0299513 2026-02-23T15:46:27.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:46:27 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/4209682012' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:46:27.792 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9181 8 365281 365273 0.310803 0 - 0.0299513 2026-02-23T15:46:28.495 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:46:28 trial068 ceph-mon[38805]: pgmap v5351: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:46:28.495 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:46:28 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2026-02-23T15:46:28.495 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:46:28 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-23T15:46:28.495 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:46:28 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2026-02-23T15:46:28.495 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:46:28 trial068 ceph-mon[38805]: pgmap v5352: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:46:28.495 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:46:28 trial068 ceph-mon[38805]: pgmap v5353: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:46:28.495 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:46:28 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T15:46:28.495 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:46:28 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2026-02-23T15:46:28.792 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9182 8 365281 365273 0.310769 0 - 0.0299513 2026-02-23T15:46:29.277 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:46:29.394 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:46:29.497 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:46:29.495+0000 7fd5cfc0b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:46:29.497 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:46:29.495+0000 7fd5cfc0b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:46:29.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:46:29 trial068 ceph-mon[38805]: 5.e scrub starts 2026-02-23T15:46:29.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:46:29 trial068 ceph-mon[38805]: 5.e scrub ok 2026-02-23T15:46:29.607 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:46:29.607 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":337,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:45:45.223960+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:41:36.954923+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:44:11.881302+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:44:45.023280+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:46:29.793 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9183 8 365281 365273 0.310735 0 - 0.0299513 2026-02-23T15:46:30.793 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9184 8 365281 365273 0.310701 0 - 0.0299513 2026-02-23T15:46:30.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:46:30 trial068 ceph-mon[38805]: pgmap v5354: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:46:30.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:46:30 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1460989346' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:46:31.793 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9185 8 365281 365273 0.310667 0 - 0.0299513 2026-02-23T15:46:32.644 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:46:32.760 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:46:32.793 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9186 8 365281 365273 0.310634 0 - 0.0299513 2026-02-23T15:46:32.857 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:46:32.855+0000 7ff6a8215640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:46:32.858 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:46:32.856+0000 7ff6a8215640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:46:32.967 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:46:32.967 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":337,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:45:45.223960+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:41:36.954923+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:44:11.881302+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:44:45.023280+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:46:32.967 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:46:32 trial068 ceph-mon[38805]: pgmap v5355: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:46:33.793 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9187 8 365281 365273 0.3106 0 - 0.0299513 2026-02-23T15:46:33.973 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:46:33 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3737743063' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:46:34.265 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:46:33 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:46:33.972Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:46:34.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:46:33 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:46:33] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:46:34.793 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9188 8 365281 365273 0.310566 0 - 0.0299513 2026-02-23T15:46:35.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:46:34 trial068 ceph-mon[38805]: pgmap v5356: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:46:35.793 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9189 8 365281 365273 0.310532 0 - 0.0299513 2026-02-23T15:46:36.004 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:46:36.122 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:46:36.225 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:46:36.223+0000 7f1e411ef640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:46:36.225 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:46:36.223+0000 7f1e411ef640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:46:36.334 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:46:36.334 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":337,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:45:45.223960+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:41:36.954923+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:44:11.881302+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:44:45.023280+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:46:36.793 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9190 8 365281 365273 0.310498 0 - 0.0299513 2026-02-23T15:46:37.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:46:36 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T15:46:37.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:46:36 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T15:46:37.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:46:36 trial068 ceph-mon[38805]: pgmap v5357: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:46:37.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:46:36 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2890987966' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:46:37.793 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9191 8 365281 365273 0.310465 0 - 0.0299513 2026-02-23T15:46:38.793 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9192 8 365281 365273 0.310431 0 - 0.0299513 2026-02-23T15:46:39.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:46:38 trial068 ceph-mon[38805]: pgmap v5358: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:46:39.371 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:46:39.486 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:46:39.584 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:46:39.582+0000 7fd30bcc5640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:46:39.584 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:46:39.583+0000 7fd30bcc5640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:46:39.692 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:46:39.692 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":337,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:45:45.223960+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:41:36.954923+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:44:11.881302+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:44:45.023280+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:46:39.793 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9193 8 365281 365273 0.310397 0 - 0.0299513 2026-02-23T15:46:40.793 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9194 8 365281 365273 0.310363 0 - 0.0299513 2026-02-23T15:46:41.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:46:40 trial068 ceph-mon[38805]: pgmap v5359: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:46:41.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:46:40 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3819396486' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:46:41.793 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9195 8 365281 365273 0.310329 0 - 0.0299513 2026-02-23T15:46:42.730 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:46:42.794 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9196 8 365281 365273 0.310296 0 - 0.0299513 2026-02-23T15:46:42.848 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:46:42.894 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:46:42 trial068 ceph-mon[38805]: pgmap v5360: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:46:42.953 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:46:42.952+0000 7f395b3f6640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:46:42.954 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:46:42.952+0000 7f395b3f6640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:46:43.065 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:46:43.066 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":337,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:45:45.223960+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:41:36.954923+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:44:11.881302+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:44:45.023280+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:46:43.794 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9197 8 365281 365273 0.310262 0 - 0.0299513 2026-02-23T15:46:43.972 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:46:43 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1182049945' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:46:44.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:46:43 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:46:43] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:46:44.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:46:43 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:46:43.972Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:46:44.794 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9198 8 365281 365273 0.310228 0 - 0.0299513 2026-02-23T15:46:45.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:46:44 trial068 ceph-mon[38805]: pgmap v5361: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:46:45.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:46:44 trial068 ceph-mon[38805]: 5.5 scrub starts 2026-02-23T15:46:45.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:46:44 trial068 ceph-mon[38805]: 5.5 scrub ok 2026-02-23T15:46:45.794 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9199 8 365281 365273 0.310195 0 - 0.0299513 2026-02-23T15:46:46.104 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:46:46.224 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:46:46.324 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:46:46.322+0000 7fde0cca5640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:46:46.324 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:46:46.322+0000 7fde0cca5640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:46:46.432 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:46:46.432 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":337,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:45:45.223960+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:41:36.954923+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:44:11.881302+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:44:45.023280+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:46:46.794 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T15:46:46.793845+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T15:46:46.794 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T15:46:46.794 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9200 8 365281 365273 0.310161 0 - 0.0299513 2026-02-23T15:46:46.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:46:46 trial068 ceph-mon[38805]: pgmap v5362: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:46:46.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:46:46 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2497469731' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:46:47.794 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9201 8 365281 365273 0.310127 0 - 0.0299513 2026-02-23T15:46:48.794 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9202 8 365281 365273 0.310093 0 - 0.0299513 2026-02-23T15:46:49.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:46:48 trial068 ceph-mon[38805]: pgmap v5363: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:46:49.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:46:48 trial068 ceph-mon[38805]: osdmap e338: 4 total, 3 up, 4 in 2026-02-23T15:46:49.470 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:46:49.589 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:46:49.685 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:46:49.683+0000 7f3f85882640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:46:49.685 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:46:49.683+0000 7f3f85882640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:46:49.794 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9203 8 365281 365273 0.31006 0 - 0.0299513 2026-02-23T15:46:49.796 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:46:49.797 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":338,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:46:47.644758+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:46:35.370242+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:44:11.881302+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:44:45.023280+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:46:50.794 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9204 8 365281 365273 0.310026 0 - 0.0299513 2026-02-23T15:46:51.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:46:50 trial068 ceph-mon[38805]: pgmap v5365: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:46:51.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:46:50 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2980851030' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:46:51.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:46:50 trial068 ceph-mon[38805]: osdmap e339: 4 total, 3 up, 4 in 2026-02-23T15:46:51.794 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9205 8 365281 365273 0.309992 0 - 0.0299513 2026-02-23T15:46:52.794 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9206 8 365281 365273 0.309959 0 - 0.0299513 2026-02-23T15:46:52.834 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:46:52.952 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:46:52.973 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:46:52 trial068 ceph-mon[38805]: pgmap v5367: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:46:53.052 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:46:53.050+0000 7f26fafa7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:46:53.052 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:46:53.051+0000 7f26fafa7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:46:53.161 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:46:53.161 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":339,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:46:50.233656+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:46:35.370242+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:44:11.881302+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:44:45.023280+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:46:53.794 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9207 8 365281 365273 0.309925 0 - 0.0299513 2026-02-23T15:46:53.974 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:46:53 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1564842976' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:46:54.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:46:53 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:46:53] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:46:54.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:46:53 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:46:53.974Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:46:54.794 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9208 8 365281 365273 0.309891 0 - 0.0299513 2026-02-23T15:46:55.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:46:54 trial068 ceph-mon[38805]: pgmap v5368: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:46:55.794 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9209 8 365281 365273 0.309858 0 - 0.0299513 2026-02-23T15:46:56.198 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:46:56.316 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:46:56.419 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:46:56.417+0000 7f2b0ff8e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:46:56.419 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:46:56.418+0000 7f2b0ff8e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:46:56.531 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:46:56.531 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":339,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:46:50.233656+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:46:35.370242+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:44:11.881302+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:44:45.023280+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:46:56.531 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:46:56 trial068 ceph-mon[38805]: pgmap v5369: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:46:56.531 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:46:56 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T15:46:56.532 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:46:56 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T15:46:56.795 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9210 8 365281 365273 0.309824 0 - 0.0299513 2026-02-23T15:46:57.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:46:57 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2363403445' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:46:57.795 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9211 8 365281 365273 0.30979 0 - 0.0299513 2026-02-23T15:46:58.494 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:46:58 trial068 ceph-mon[38805]: pgmap v5370: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:46:58.795 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9212 8 365281 365273 0.309757 0 - 0.0299513 2026-02-23T15:46:59.569 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:46:59.690 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:46:59.791 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:46:59.789+0000 7fc0cddcb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:46:59.791 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:46:59.789+0000 7fc0cddcb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:46:59.795 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9213 8 365281 365273 0.309723 0 - 0.0299513 2026-02-23T15:46:59.900 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:46:59.900 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":339,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:46:50.233656+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:46:35.370242+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:44:11.881302+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:44:45.023280+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:47:00.795 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9214 8 365281 365273 0.30969 0 - 0.0299513 2026-02-23T15:47:01.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:47:00 trial068 ceph-mon[38805]: pgmap v5371: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:47:01.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:47:00 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2684590061' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:47:01.795 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9215 8 365281 365273 0.309656 0 - 0.0299513 2026-02-23T15:47:02.795 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9216 8 365281 365273 0.309622 0 - 0.0299513 2026-02-23T15:47:02.937 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:47:02.961 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:47:02 trial068 ceph-mon[38805]: pgmap v5372: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:47:03.056 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:47:03.154 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:47:03.152+0000 7ff132af2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:47:03.154 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:47:03.152+0000 7ff132af2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:47:03.263 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:47:03.263 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":339,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:46:50.233656+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:46:35.370242+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:44:11.881302+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:44:45.023280+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:47:03.795 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9217 8 365281 365273 0.309589 0 - 0.0299513 2026-02-23T15:47:03.974 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:47:03 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3383296069' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:47:04.264 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:47:03 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:47:03.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:47:04.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:47:03 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:47:03] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:47:04.795 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9218 8 365281 365273 0.309555 0 - 0.0299513 2026-02-23T15:47:05.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:47:04 trial068 ceph-mon[38805]: pgmap v5373: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:47:05.795 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9219 8 365281 365273 0.309522 0 - 0.0299513 2026-02-23T15:47:06.300 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:47:06.418 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:47:06.520 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:47:06.518+0000 7f5c3dbc7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:47:06.520 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:47:06.518+0000 7f5c3dbc7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:47:06.632 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:47:06.632 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":339,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:46:50.233656+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:46:35.370242+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:44:11.881302+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:44:45.023280+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:47:06.795 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T15:47:06.795315+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T15:47:06.796 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T15:47:06.796 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9220 8 365281 365273 0.309488 0 - 0.0299513 2026-02-23T15:47:07.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:47:06 trial068 ceph-mon[38805]: pgmap v5374: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:47:07.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:47:06 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/471816160' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:47:07.795 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9221 8 365281 365273 0.309454 0 - 0.0299513 2026-02-23T15:47:08.795 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9222 8 365281 365273 0.309421 0 - 0.0299513 2026-02-23T15:47:09.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:47:08 trial068 ceph-mon[38805]: pgmap v5375: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:47:09.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:47:08 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T15:47:09.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:47:08 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T15:47:09.691 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:47:09.796 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9223 8 365281 365273 0.309387 0 - 0.0299513 2026-02-23T15:47:09.809 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:47:09.907 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:47:09.905+0000 7facbeb5f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:47:09.907 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:47:09.905+0000 7facbeb5f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:47:10.015 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:47:10.015 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":339,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:46:50.233656+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:46:35.370242+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:44:11.881302+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:44:45.023280+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:47:10.796 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9224 8 365281 365273 0.309354 0 - 0.0299513 2026-02-23T15:47:11.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:47:10 trial068 ceph-mon[38805]: pgmap v5376: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:47:11.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:47:10 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2723985834' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:47:11.796 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9225 8 365281 365273 0.30932 0 - 0.0299513 2026-02-23T15:47:12.796 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9226 8 365281 365273 0.309287 0 - 0.0299513 2026-02-23T15:47:13.052 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:47:13.077 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:47:12 trial068 ceph-mon[38805]: pgmap v5377: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:47:13.171 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:47:13.269 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:47:13.267+0000 7f5f463d1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:47:13.269 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:47:13.267+0000 7f5f463d1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:47:13.378 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:47:13.378 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":339,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:46:50.233656+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:46:35.370242+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:44:11.881302+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:44:45.023280+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:47:13.796 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9227 8 365281 365273 0.309253 0 - 0.0299513 2026-02-23T15:47:13.973 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:47:13 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/4175170183' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:47:14.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:47:13 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:47:13] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:47:14.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:47:13 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:47:13.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:47:14.796 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9228 8 365281 365273 0.30922 0 - 0.0299513 2026-02-23T15:47:15.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:47:14 trial068 ceph-mon[38805]: pgmap v5378: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:47:15.796 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9229 8 365281 365273 0.309186 0 - 0.0299513 2026-02-23T15:47:16.432 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:47:16.551 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:47:16.573 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:47:16 trial068 ceph-mon[38805]: pgmap v5379: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:47:16.649 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:47:16.647+0000 7f6b4d01a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:47:16.649 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:47:16.647+0000 7f6b4d01a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:47:16.757 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:47:16.758 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":339,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:46:50.233656+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:46:35.370242+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:44:11.881302+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:44:45.023280+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:47:16.796 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9230 8 365281 365273 0.309153 0 - 0.0299513 2026-02-23T15:47:17.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:47:17 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3380245740' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:47:17.796 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9231 8 365281 365273 0.309119 0 - 0.0299513 2026-02-23T15:47:18.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:47:18 trial068 ceph-mon[38805]: pgmap v5380: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:47:18.797 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9232 8 365281 365273 0.309086 0 - 0.0299513 2026-02-23T15:47:19.794 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:47:19.798 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9233 8 365281 365273 0.309052 0 - 0.0299513 2026-02-23T15:47:19.911 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:47:20.007 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:47:20.005+0000 7f4a71bc5640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:47:20.007 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:47:20.006+0000 7f4a71bc5640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:47:20.116 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:47:20.116 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":339,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:46:50.233656+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:46:35.370242+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:44:11.881302+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:44:45.023280+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:47:20.797 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9234 8 365281 365273 0.309019 0 - 0.0299513 2026-02-23T15:47:21.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:47:20 trial068 ceph-mon[38805]: pgmap v5381: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:47:21.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:47:20 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/242577364' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:47:21.797 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9235 8 365281 365273 0.308985 0 - 0.0299513 2026-02-23T15:47:22.797 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9236 8 365281 365273 0.308952 0 - 0.0299513 2026-02-23T15:47:23.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:47:22 trial068 ceph-mon[38805]: pgmap v5382: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:47:23.154 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:47:23.275 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:47:23.374 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:47:23.372+0000 7fc363b94640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:47:23.374 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:47:23.372+0000 7fc363b94640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:47:23.485 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:47:23.485 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":339,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:46:50.233656+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:46:35.370242+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:44:11.881302+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:44:45.023280+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:47:23.797 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9237 8 365281 365273 0.308918 0 - 0.0299513 2026-02-23T15:47:23.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:47:23 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/4180908547' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:47:24.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:47:23 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:47:23] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:47:24.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:47:23 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:47:23.972Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:47:24.797 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9238 8 365281 365273 0.308885 0 - 0.0299513 2026-02-23T15:47:25.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:47:24 trial068 ceph-mon[38805]: pgmap v5383: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:47:25.797 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9239 8 365281 365273 0.308852 0 - 0.0299513 2026-02-23T15:47:26.527 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:47:26.646 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:47:26.744 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:47:26.742+0000 7f1a394f1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:47:26.744 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:47:26.742+0000 7f1a394f1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:47:26.797 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T15:47:26.797156+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T15:47:26.797 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T15:47:26.797 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9240 8 365281 365273 0.308818 0 - 0.0299513 2026-02-23T15:47:26.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:47:26 trial068 ceph-mon[38805]: pgmap v5384: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:47:26.853 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:47:26.854 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":339,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:46:50.233656+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:46:35.370242+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:44:11.881302+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:44:45.023280+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:47:27.797 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9241 8 365281 365273 0.308785 0 - 0.0299513 2026-02-23T15:47:28.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:47:27 trial068 ceph-mon[38805]: 5.7 scrub starts 2026-02-23T15:47:28.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:47:27 trial068 ceph-mon[38805]: 5.7 scrub ok 2026-02-23T15:47:28.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:47:27 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/4060814893' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:47:28.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:47:27 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2026-02-23T15:47:28.797 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9242 8 365281 365273 0.308751 0 - 0.0299513 2026-02-23T15:47:29.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:47:28 trial068 ceph-mon[38805]: pgmap v5385: 17 pgs: 1 active+clean+scrubbing, 1 active+recovery_unfound+undersized+degraded, 4 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:47:29.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:47:28 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T15:47:29.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:47:28 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T15:47:29.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:47:28 trial068 ceph-mon[38805]: 5.a scrub starts 2026-02-23T15:47:29.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:47:28 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T15:47:29.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:47:28 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T15:47:29.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:47:28 trial068 ceph-mon[38805]: 5.a scrub ok 2026-02-23T15:47:29.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:47:28 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-23T15:47:29.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:47:28 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2026-02-23T15:47:29.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:47:28 trial068 ceph-mon[38805]: pgmap v5386: 17 pgs: 1 active+clean+scrubbing, 1 active+recovery_unfound+undersized+degraded, 4 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:47:29.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:47:28 trial068 ceph-mon[38805]: pgmap v5387: 17 pgs: 1 active+clean+scrubbing, 1 active+recovery_unfound+undersized+degraded, 4 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:47:29.095 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:47:28 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T15:47:29.095 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:47:28 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2026-02-23T15:47:29.797 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9243 8 365281 365273 0.308718 0 - 0.0299513 2026-02-23T15:47:29.891 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:47:30.007 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:47:30.108 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:47:30.106+0000 7fc1d0418640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:47:30.108 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:47:30.107+0000 7fc1d0418640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:47:30.219 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:47:30.220 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":339,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:46:50.233656+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:46:35.370242+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:44:11.881302+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:44:45.023280+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:47:30.798 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9244 8 365281 365273 0.308684 0 - 0.0299513 2026-02-23T15:47:31.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:47:31 trial068 ceph-mon[38805]: 5.8 scrub starts 2026-02-23T15:47:31.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:47:31 trial068 ceph-mon[38805]: 5.8 scrub ok 2026-02-23T15:47:31.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:47:31 trial068 ceph-mon[38805]: pgmap v5388: 17 pgs: 1 active+clean+scrubbing, 1 active+recovery_unfound+undersized+degraded, 4 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:47:31.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:47:31 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2951980861' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:47:31.798 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9245 8 365281 365273 0.308651 0 - 0.0299513 2026-02-23T15:47:32.798 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9246 8 365281 365273 0.308618 0 - 0.0299513 2026-02-23T15:47:33.258 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:47:33.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:47:33 trial068 ceph-mon[38805]: pgmap v5389: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:47:33.376 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:47:33.476 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:47:33.474+0000 7f59b7fb0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:47:33.476 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:47:33.474+0000 7f59b7fb0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:47:33.584 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:47:33.585 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":339,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:46:50.233656+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:46:35.370242+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:44:11.881302+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:44:45.023280+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:47:33.798 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9247 8 365281 365273 0.308584 0 - 0.0299513 2026-02-23T15:47:34.265 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:47:33 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:47:33.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:47:34.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:47:34 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1778365332' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:47:34.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:47:33 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:47:33] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:47:34.798 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9248 8 365281 365273 0.308551 0 - 0.0299513 2026-02-23T15:47:35.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:47:35 trial068 ceph-mon[38805]: pgmap v5390: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:47:35.798 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9249 8 365281 365273 0.308518 0 - 0.0299513 2026-02-23T15:47:36.622 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:47:36.741 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:47:36.798 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9250 8 365281 365273 0.308484 0 - 0.0299513 2026-02-23T15:47:36.835 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:47:36.833+0000 7f60dcb8f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:47:36.835 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:47:36.833+0000 7f60dcb8f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:47:36.945 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:47:36.946 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":339,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:46:50.233656+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:46:35.370242+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:44:11.881302+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:44:45.023280+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:47:37.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:47:37 trial068 ceph-mon[38805]: pgmap v5391: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:47:37.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:47:37 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1261413020' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:47:37.798 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9251 8 365281 365273 0.308451 0 - 0.0299513 2026-02-23T15:47:38.798 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9252 8 365281 365273 0.308418 0 - 0.0299513 2026-02-23T15:47:39.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:47:39 trial068 ceph-mon[38805]: pgmap v5392: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:47:39.798 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9253 8 365281 365273 0.308384 0 - 0.0299513 2026-02-23T15:47:39.982 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:47:40.100 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:47:40.197 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:47:40.195+0000 7f9f3e50a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:47:40.197 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:47:40.195+0000 7f9f3e50a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:47:40.307 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:47:40.307 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":339,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:46:50.233656+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:46:35.370242+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:44:11.881302+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:44:45.023280+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:47:40.798 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9254 8 365281 365273 0.308351 0 - 0.0299513 2026-02-23T15:47:41.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:47:41 trial068 ceph-mon[38805]: pgmap v5393: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:47:41.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:47:41 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3284683283' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:47:41.798 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9255 8 365281 365273 0.308318 0 - 0.0299513 2026-02-23T15:47:42.798 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9256 8 365281 365273 0.308284 0 - 0.0299513 2026-02-23T15:47:43.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:47:43 trial068 ceph-mon[38805]: pgmap v5394: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:47:43.346 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:47:43.463 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:47:43.568 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:47:43.566+0000 7fd729123640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:47:43.568 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:47:43.566+0000 7fd729123640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:47:43.676 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:47:43.677 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":339,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:46:50.233656+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:46:35.370242+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:44:11.881302+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:44:45.023280+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:47:43.798 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9257 8 365281 365273 0.308251 0 - 0.0299513 2026-02-23T15:47:44.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:47:44 trial068 ceph-mon[38805]: 5.e scrub starts 2026-02-23T15:47:44.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:47:44 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2145969507' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:47:44.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:47:43 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:47:43] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:47:44.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:47:43 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:47:43.974Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:47:44.799 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9258 8 365281 365273 0.308218 0 - 0.0299513 2026-02-23T15:47:45.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:47:45 trial068 ceph-mon[38805]: 5.e scrub ok 2026-02-23T15:47:45.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:47:45 trial068 ceph-mon[38805]: pgmap v5395: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:47:45.799 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9259 8 365281 365273 0.308184 0 - 0.0299513 2026-02-23T15:47:46.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:47:46 trial068 ceph-mon[38805]: pgmap v5396: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:47:46.713 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:47:46.799 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T15:47:46.798654+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T15:47:46.799 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T15:47:46.799 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9260 8 365281 365273 0.308151 0 - 0.0299513 2026-02-23T15:47:46.833 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:47:46.935 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:47:46.934+0000 7fcde054d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:47:46.936 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:47:46.934+0000 7fcde054d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:47:47.044 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:47:47.044 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":339,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:46:50.233656+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:46:35.370242+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:44:11.881302+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:44:45.023280+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:47:47.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:47:47 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1393286100' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:47:47.799 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9261 8 365281 365273 0.308118 0 - 0.0299513 2026-02-23T15:47:48.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:47:48 trial068 ceph-mon[38805]: pgmap v5397: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:47:48.799 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9262 8 365281 365273 0.308085 0 - 0.0299513 2026-02-23T15:47:49.799 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9263 8 365281 365273 0.308051 0 - 0.0299513 2026-02-23T15:47:50.081 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:47:50.202 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:47:50.311 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:47:50.309+0000 7fb531c4e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:47:50.311 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:47:50.309+0000 7fb531c4e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:47:50.420 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:47:50.420 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":339,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:46:50.233656+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:46:35.370242+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:44:11.881302+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:44:45.023280+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:47:50.799 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9264 8 365281 365273 0.308018 0 - 0.0299513 2026-02-23T15:47:51.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:47:51 trial068 ceph-mon[38805]: pgmap v5398: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:47:51.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:47:51 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1115318305' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:47:51.799 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9265 8 365281 365273 0.307985 0 - 0.0299513 2026-02-23T15:47:52.799 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9266 8 365281 365273 0.307952 0 - 0.0299513 2026-02-23T15:47:53.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:47:53 trial068 ceph-mon[38805]: pgmap v5399: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:47:53.457 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:47:53.576 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:47:53.676 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:47:53.674+0000 7f5c6b1e7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:47:53.677 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:47:53.675+0000 7f5c6b1e7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:47:53.785 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:47:53.785 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":339,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:46:50.233656+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:46:35.370242+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:44:11.881302+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:44:45.023280+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:47:53.800 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9267 8 365281 365273 0.307918 0 - 0.0299513 2026-02-23T15:47:54.062 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:47:53 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:47:53] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:47:54.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:47:54 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2999300966' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:47:54.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:47:53 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:47:53.974Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:47:54.800 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9268 8 365281 365273 0.307885 0 - 0.0299513 2026-02-23T15:47:55.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:47:55 trial068 ceph-mon[38805]: pgmap v5400: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:47:55.800 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9269 8 365281 365273 0.307852 0 - 0.0299513 2026-02-23T15:47:56.800 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9270 8 365281 365273 0.307819 0 - 0.0299513 2026-02-23T15:47:56.823 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:47:56.940 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:47:57.038 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:47:57.036+0000 7f9488b14640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:47:57.039 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:47:57.037+0000 7f9488b14640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:47:57.149 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:47:57.149 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":339,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:46:50.233656+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:46:35.370242+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:44:11.881302+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:44:45.023280+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:47:57.149 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:47:57 trial068 ceph-mon[38805]: pgmap v5401: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:47:57.800 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9271 8 365281 365273 0.307785 0 - 0.0299513 2026-02-23T15:47:58.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:47:58 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3307257862' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:47:58.800 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9272 8 365281 365273 0.307752 0 - 0.0299513 2026-02-23T15:47:59.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:47:59 trial068 ceph-mon[38805]: pgmap v5402: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:47:59.800 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9273 8 365281 365273 0.307719 0 - 0.0299513 2026-02-23T15:48:00.187 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:48:00.306 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:48:00.409 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:48:00.408+0000 7fea53716640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:48:00.410 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:48:00.408+0000 7fea53716640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:48:00.519 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:48:00.519 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":339,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:46:50.233656+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:46:35.370242+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:44:11.881302+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:44:45.023280+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:48:00.800 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9274 8 365281 365273 0.307686 0 - 0.0299513 2026-02-23T15:48:01.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:48:01 trial068 ceph-mon[38805]: pgmap v5403: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:48:01.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:48:01 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3644660826' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:48:01.800 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9275 8 365281 365273 0.307653 0 - 0.0299513 2026-02-23T15:48:02.800 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9276 8 365281 365273 0.30762 0 - 0.0299513 2026-02-23T15:48:03.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:48:03 trial068 ceph-mon[38805]: pgmap v5404: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:48:03.558 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:48:03.677 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:48:03.776 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:48:03.774+0000 7f8ac49d5640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:48:03.776 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:48:03.774+0000 7f8ac49d5640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:48:03.800 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9277 8 365281 365273 0.307586 0 - 0.0299513 2026-02-23T15:48:03.885 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:48:03.886 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":339,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:46:50.233656+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:46:35.370242+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:44:11.881302+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:44:45.023280+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:48:04.265 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:48:03 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:48:03.972Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:48:04.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:48:04 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T15:48:04.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:48:04 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T15:48:04.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:48:04 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/351989875' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:48:04.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:48:03 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:48:03] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:48:04.800 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9278 8 365281 365273 0.307553 0 - 0.0299513 2026-02-23T15:48:05.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:48:05 trial068 ceph-mon[38805]: pgmap v5405: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:48:05.800 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9279 8 365281 365273 0.30752 0 - 0.0299513 2026-02-23T15:48:06.801 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T15:48:06.800471+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T15:48:06.801 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T15:48:06.801 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9280 8 365281 365273 0.307487 0 - 0.0299513 2026-02-23T15:48:06.922 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:48:07.040 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:48:07.140 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:48:07.138+0000 7f82a754e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:48:07.140 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:48:07.138+0000 7f82a754e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:48:07.249 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:48:07.250 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":339,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:46:50.233656+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:46:35.370242+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:44:11.881302+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:44:45.023280+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:48:07.250 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:48:07 trial068 ceph-mon[38805]: pgmap v5406: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:48:07.801 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9281 8 365281 365273 0.307454 0 - 0.0299513 2026-02-23T15:48:08.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:48:08 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/770535074' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:48:08.801 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9282 8 365281 365273 0.307421 0 - 0.0299513 2026-02-23T15:48:09.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:48:09 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T15:48:09.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:48:09 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T15:48:09.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:48:09 trial068 ceph-mon[38805]: pgmap v5407: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:48:09.801 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9283 8 365281 365273 0.307388 0 - 0.0299513 2026-02-23T15:48:10.286 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:48:10.405 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:48:10.516 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:48:10.515+0000 7f8f4661d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:48:10.517 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:48:10.515+0000 7f8f4661d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:48:10.625 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:48:10.625 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":339,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:46:50.233656+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:46:35.370242+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:44:11.881302+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:44:45.023280+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:48:10.801 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9284 8 365281 365273 0.307355 0 - 0.0299513 2026-02-23T15:48:11.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:48:11 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T15:48:11.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:48:11 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T15:48:11.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:48:11 trial068 ceph-mon[38805]: pgmap v5408: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:48:11.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:48:11 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3046080310' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:48:11.801 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9285 8 365281 365273 0.307321 0 - 0.0299513 2026-02-23T15:48:12.801 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9286 8 365281 365273 0.307288 0 - 0.0299513 2026-02-23T15:48:13.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:48:13 trial068 ceph-mon[38805]: pgmap v5409: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:48:13.663 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:48:13.781 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:48:13.801 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9287 8 365281 365273 0.307255 0 - 0.0299513 2026-02-23T15:48:13.878 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:48:13.876+0000 7f4094c87640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:48:13.879 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:48:13.877+0000 7f4094c87640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:48:13.988 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:48:13.988 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":339,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:46:50.233656+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:46:35.370242+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:44:11.881302+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:44:45.023280+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:48:14.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:48:14 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2745933316' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:48:14.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:48:13 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:48:13] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:48:14.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:48:13 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:48:13.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:48:14.801 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9288 8 365281 365273 0.307222 0 - 0.0299513 2026-02-23T15:48:15.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:48:15 trial068 ceph-mon[38805]: pgmap v5410: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:48:15.801 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9289 8 365281 365273 0.307189 0 - 0.0299513 2026-02-23T15:48:16.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:48:16 trial068 ceph-mon[38805]: 5.5 scrub starts 2026-02-23T15:48:16.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:48:16 trial068 ceph-mon[38805]: 5.5 scrub ok 2026-02-23T15:48:16.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:48:16 trial068 ceph-mon[38805]: pgmap v5411: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:48:16.801 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9290 8 365281 365273 0.307156 0 - 0.0299513 2026-02-23T15:48:17.026 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:48:17.142 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:48:17.245 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:48:17.243+0000 7f92f7a3d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:48:17.245 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:48:17.243+0000 7f92f7a3d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:48:17.354 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:48:17.354 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":339,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:46:50.233656+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:46:35.370242+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:44:11.881302+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:44:45.023280+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:48:17.801 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9291 8 365281 365273 0.307123 0 - 0.0299513 2026-02-23T15:48:17.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:48:17 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3360453571' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:48:18.801 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9292 8 365281 365273 0.30709 0 - 0.0299513 2026-02-23T15:48:18.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:48:18 trial068 ceph-mon[38805]: pgmap v5412: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:48:19.801 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9293 8 365281 365273 0.307057 0 - 0.0299513 2026-02-23T15:48:20.392 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:48:20.509 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:48:20.608 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:48:20.606+0000 7f4dfd14d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:48:20.608 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:48:20.606+0000 7f4dfd14d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:48:20.718 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:48:20.718 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":339,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:46:50.233656+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:46:35.370242+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:44:11.881302+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:44:45.023280+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:48:20.801 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9294 8 365281 365273 0.307024 0 - 0.0299513 2026-02-23T15:48:21.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:48:21 trial068 ceph-mon[38805]: pgmap v5413: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:48:21.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:48:21 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3964383568' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:48:21.801 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9295 8 365281 365273 0.306991 0 - 0.0299513 2026-02-23T15:48:22.802 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9296 8 365281 365273 0.306958 0 - 0.0299513 2026-02-23T15:48:23.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:48:23 trial068 ceph-mon[38805]: pgmap v5414: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:48:23.754 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:48:23.801 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9297 8 365281 365273 0.306925 0 - 0.0299513 2026-02-23T15:48:23.870 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:48:23.972 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:48:23.970+0000 7f6357767640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:48:23.972 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:48:23.970+0000 7f6357767640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:48:24.080 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:48:24.081 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":339,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:46:50.233656+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:46:35.370242+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:44:11.881302+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:44:45.023280+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:48:24.081 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:48:23 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:48:23] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:48:24.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:48:23 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:48:23.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:48:24.802 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9298 8 365281 365273 0.306892 0 - 0.0299513 2026-02-23T15:48:25.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:48:25 trial068 ceph-mon[38805]: pgmap v5415: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:48:25.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:48:25 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/4000668528' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:48:25.802 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9299 8 365281 365273 0.306859 0 - 0.0299513 2026-02-23T15:48:26.802 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T15:48:26.801792+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T15:48:26.802 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T15:48:26.802 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9300 8 365281 365273 0.306826 0 - 0.0299513 2026-02-23T15:48:27.119 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:48:27.238 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:48:27.338 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:48:27.336+0000 7f0af3a47640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:48:27.338 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:48:27.336+0000 7f0af3a47640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:48:27.338 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:48:27 trial068 ceph-mon[38805]: pgmap v5416: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:48:27.447 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:48:27.447 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":339,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:46:50.233656+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:46:35.370242+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:44:11.881302+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:44:45.023280+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:48:27.802 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9301 8 365281 365273 0.306793 0 - 0.0299513 2026-02-23T15:48:28.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:48:28 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1125918404' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:48:28.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:48:28 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2026-02-23T15:48:28.802 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9302 8 365281 365273 0.30676 0 - 0.0299513 2026-02-23T15:48:29.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:48:29 trial068 ceph-mon[38805]: pgmap v5417: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:48:29.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:48:29 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T15:48:29.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:48:29 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T15:48:29.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:48:29 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T15:48:29.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:48:29 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T15:48:29.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:48:29 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-23T15:48:29.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:48:29 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2026-02-23T15:48:29.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:48:29 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T15:48:29.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:48:29 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2026-02-23T15:48:29.802 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9303 8 365281 365273 0.306727 0 - 0.0299513 2026-02-23T15:48:30.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:48:30 trial068 ceph-mon[38805]: pgmap v5418: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:48:30.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:48:30 trial068 ceph-mon[38805]: pgmap v5419: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:48:30.502 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:48:30.620 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:48:30.722 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:48:30.720+0000 7f99b7df2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:48:30.722 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:48:30.720+0000 7f99b7df2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:48:30.802 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9304 8 365281 365273 0.306694 0 - 0.0299513 2026-02-23T15:48:30.830 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:48:30.831 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":339,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:46:50.233656+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:46:35.370242+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:44:11.881302+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:44:45.023280+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:48:31.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:48:31 trial068 ceph-mon[38805]: pgmap v5420: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:48:31.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:48:31 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/849454659' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:48:31.802 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9305 8 365281 365273 0.306661 0 - 0.0299513 2026-02-23T15:48:32.802 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9306 8 365281 365273 0.306628 0 - 0.0299513 2026-02-23T15:48:33.802 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9307 8 365281 365273 0.306595 0 - 0.0299513 2026-02-23T15:48:33.870 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:48:33.989 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:48:34.093 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:48:34.091+0000 7fe1c4543640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:48:34.093 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:48:34.092+0000 7fe1c4543640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:48:34.105 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:48:33 trial068 ceph-mon[38805]: pgmap v5421: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:48:34.105 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:48:33 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:48:33] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:48:34.206 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:48:34.206 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":339,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:46:50.233656+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:46:35.370242+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:44:11.881302+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:44:45.023280+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:48:34.260 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:48:33 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:48:33.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:48:34.802 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9308 8 365281 365273 0.306562 0 - 0.0299513 2026-02-23T15:48:35.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:48:34 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3879257289' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:48:35.802 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9309 8 365281 365273 0.306529 0 - 0.0299513 2026-02-23T15:48:36.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:48:35 trial068 ceph-mon[38805]: pgmap v5422: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:48:36.802 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9310 8 365281 365273 0.306496 0 - 0.0299513 2026-02-23T15:48:37.245 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:48:37.363 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:48:37.467 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:48:37.465+0000 7f7b82201640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:48:37.467 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:48:37.465+0000 7f7b82201640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:48:37.581 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:48:37.581 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":339,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:46:50.233656+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:46:35.370242+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:44:11.881302+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:44:45.023280+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:48:37.803 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9311 8 365281 365273 0.306463 0 - 0.0299513 2026-02-23T15:48:38.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:48:37 trial068 ceph-mon[38805]: pgmap v5423: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:48:38.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:48:37 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3091328672' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:48:38.803 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9312 8 365281 365273 0.30643 0 - 0.0299513 2026-02-23T15:48:39.803 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9313 8 365281 365273 0.306397 0 - 0.0299513 2026-02-23T15:48:40.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:48:39 trial068 ceph-mon[38805]: pgmap v5424: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:48:40.621 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:48:40.739 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:48:40.803 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9314 8 365281 365273 0.306365 0 - 0.0299513 2026-02-23T15:48:40.848 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:48:40.847+0000 7fb6ea648640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:48:40.849 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:48:40.847+0000 7fb6ea648640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:48:40.960 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:48:40.960 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":339,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:46:50.233656+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:46:35.370242+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:44:11.881302+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:44:45.023280+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:48:41.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:48:41 trial068 ceph-mon[38805]: pgmap v5425: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:48:41.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:48:41 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1058727852' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:48:41.803 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9315 8 365281 365273 0.306332 0 - 0.0299513 2026-02-23T15:48:42.803 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9316 8 365281 365273 0.306299 0 - 0.0299513 2026-02-23T15:48:42.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:48:42 trial068 ceph-mon[38805]: 5.7 scrub starts 2026-02-23T15:48:42.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:48:42 trial068 ceph-mon[38805]: 5.7 scrub ok 2026-02-23T15:48:43.803 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9317 8 365281 365273 0.306266 0 - 0.0299513 2026-02-23T15:48:43.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:48:43 trial068 ceph-mon[38805]: pgmap v5426: 17 pgs: 1 active+clean+scrubbing, 1 active+recovery_unfound+undersized+degraded, 4 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:48:43.997 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:48:44.115 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:48:44.214 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:48:44.212+0000 7f161391f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:48:44.214 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:48:44.213+0000 7f161391f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:48:44.229 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:48:43 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:48:43] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:48:44.324 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:48:44.325 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":339,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:46:50.233656+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:46:35.370242+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:44:11.881302+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:44:45.023280+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:48:44.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:48:43 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:48:43.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:48:44.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:48:44 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3197635843' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:48:44.803 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9318 8 365281 365273 0.306233 0 - 0.0299513 2026-02-23T15:48:45.803 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9319 8 365281 365273 0.3062 0 - 0.0299513 2026-02-23T15:48:45.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:48:45 trial068 ceph-mon[38805]: pgmap v5427: 17 pgs: 1 active+clean+scrubbing, 1 active+recovery_unfound+undersized+degraded, 4 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:48:46.803 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T15:48:46.803345+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T15:48:46.804 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T15:48:46.804 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9320 8 365281 365273 0.306167 0 - 0.0299513 2026-02-23T15:48:46.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:48:46 trial068 ceph-mon[38805]: 5.a scrub starts 2026-02-23T15:48:46.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:48:46 trial068 ceph-mon[38805]: 5.a scrub ok 2026-02-23T15:48:47.362 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:48:47.481 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:48:47.585 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:48:47.584+0000 7fae623a0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:48:47.586 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:48:47.584+0000 7fae623a0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:48:47.695 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:48:47.696 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":339,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:46:50.233656+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:46:35.370242+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:44:11.881302+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:44:45.023280+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:48:47.696 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:48:47 trial068 ceph-mon[38805]: pgmap v5428: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:48:47.803 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9321 8 365281 365273 0.306134 0 - 0.0299513 2026-02-23T15:48:48.804 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9322 8 365281 365273 0.306102 0 - 0.0299513 2026-02-23T15:48:48.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:48:48 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1677084796' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:48:48.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:48:48 trial068 ceph-mon[38805]: 5.8 scrub starts 2026-02-23T15:48:48.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:48:48 trial068 ceph-mon[38805]: 5.8 scrub ok 2026-02-23T15:48:49.803 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9323 8 365281 365273 0.306069 0 - 0.0299513 2026-02-23T15:48:49.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:48:49 trial068 ceph-mon[38805]: pgmap v5429: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:48:50.751 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:48:50.775 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:48:50 trial068 ceph-mon[38805]: 5.e scrub starts 2026-02-23T15:48:50.803 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9324 8 365281 365273 0.306036 0 - 0.0299513 2026-02-23T15:48:50.870 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:48:50.974 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:48:50.973+0000 7f99f7644640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:48:50.975 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:48:50.973+0000 7f99f7644640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:48:51.083 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:48:51.083 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":339,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:46:50.233656+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:46:35.370242+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:44:11.881302+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:44:45.023280+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:48:51.804 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9325 8 365281 365273 0.306003 0 - 0.0299513 2026-02-23T15:48:51.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:48:51 trial068 ceph-mon[38805]: 5.e scrub ok 2026-02-23T15:48:51.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:48:51 trial068 ceph-mon[38805]: pgmap v5430: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:48:51.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:48:51 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2573285142' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:48:52.804 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9326 8 365281 365273 0.30597 0 - 0.0299513 2026-02-23T15:48:53.804 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9327 8 365281 365273 0.305938 0 - 0.0299513 2026-02-23T15:48:54.119 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:48:54.145 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:48:53 trial068 ceph-mon[38805]: pgmap v5431: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:48:54.145 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:48:53 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:48:53] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:48:54.239 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:48:54.335 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:48:54.333+0000 7fb086ff0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:48:54.335 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:48:54.333+0000 7fb086ff0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:48:54.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:48:53 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:48:53.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:48:54.443 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:48:54.443 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":339,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:46:50.233656+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:46:35.370242+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:44:11.881302+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:44:45.023280+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:48:54.804 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9328 8 365281 365273 0.305905 0 - 0.0299513 2026-02-23T15:48:55.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:48:54 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2340772485' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:48:55.804 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9329 8 365281 365273 0.305872 0 - 0.0299513 2026-02-23T15:48:56.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:48:55 trial068 ceph-mon[38805]: pgmap v5432: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:48:56.804 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9330 8 365281 365273 0.305839 0 - 0.0299513 2026-02-23T15:48:57.479 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:48:57.596 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:48:57.694 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:48:57.692+0000 7f6b7669f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:48:57.694 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:48:57.692+0000 7f6b7669f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:48:57.804 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:48:57.804 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":339,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:46:50.233656+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:46:35.370242+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:44:11.881302+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:44:45.023280+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:48:57.804 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9331 8 365281 365273 0.305806 0 - 0.0299513 2026-02-23T15:48:58.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:48:57 trial068 ceph-mon[38805]: pgmap v5433: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:48:58.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:48:57 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/597799040' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:48:58.804 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9332 8 365281 365273 0.305774 0 - 0.0299513 2026-02-23T15:48:59.805 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9333 8 365281 365273 0.305741 0 - 0.0299513 2026-02-23T15:49:00.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:48:59 trial068 ceph-mon[38805]: pgmap v5434: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:49:00.805 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9334 8 365281 365273 0.305708 0 - 0.0299513 2026-02-23T15:49:00.841 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:49:00.958 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:49:01.062 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:49:01.060+0000 7f1df4ef0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:49:01.062 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:49:01.060+0000 7f1df4ef0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:49:01.171 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:49:01.171 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":339,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:46:50.233656+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:46:35.370242+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:44:11.881302+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:44:45.023280+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:49:01.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:49:01 trial068 ceph-mon[38805]: pgmap v5435: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:49:01.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:49:01 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/852460330' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:49:01.805 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9335 8 365281 365273 0.305675 0 - 0.0299513 2026-02-23T15:49:02.805 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9336 8 365281 365273 0.305643 0 - 0.0299513 2026-02-23T15:49:03.805 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9337 8 365281 365273 0.30561 0 - 0.0299513 2026-02-23T15:49:04.208 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:49:04.233 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:49:03 trial068 ceph-mon[38805]: pgmap v5436: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:49:04.234 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:49:03 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:49:03] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:49:04.265 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:49:03 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:49:03.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:49:04.326 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:49:04.426 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:49:04.424+0000 7f8193a10640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:49:04.426 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:49:04.425+0000 7f8193a10640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:49:04.535 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:49:04.535 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":339,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:46:50.233656+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:46:35.370242+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:44:11.881302+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:44:45.023280+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:49:04.805 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9338 8 365281 365273 0.305577 0 - 0.0299513 2026-02-23T15:49:05.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:49:04 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2884104269' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:49:05.805 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9339 8 365281 365273 0.305544 0 - 0.0299513 2026-02-23T15:49:06.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:49:05 trial068 ceph-mon[38805]: pgmap v5437: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:49:06.805 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T15:49:06.805081+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T15:49:06.805 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T15:49:06.805 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9340 8 365281 365273 0.305512 0 - 0.0299513 2026-02-23T15:49:07.576 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:49:07.693 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:49:07.794 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:49:07.792+0000 7f3bd4462640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:49:07.794 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:49:07.792+0000 7f3bd4462640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:49:07.805 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9341 8 365281 365273 0.305479 0 - 0.0299513 2026-02-23T15:49:07.903 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:49:07.903 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":339,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:46:50.233656+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:46:35.370242+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:44:11.881302+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:44:45.023280+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:49:08.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:49:07 trial068 ceph-mon[38805]: pgmap v5438: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:49:08.805 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9342 8 365281 365273 0.305446 0 - 0.0299513 2026-02-23T15:49:09.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:49:08 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3097596161' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:49:09.805 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9343 8 365281 365273 0.305414 0 - 0.0299513 2026-02-23T15:49:10.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:49:09 trial068 ceph-mon[38805]: pgmap v5439: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:49:10.805 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9344 8 365281 365273 0.305381 0 - 0.0299513 2026-02-23T15:49:10.939 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:49:11.057 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:49:11.157 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:49:11.155+0000 7f9f3445e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:49:11.157 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:49:11.155+0000 7f9f3445e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:49:11.266 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:49:11.266 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":339,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:46:50.233656+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:46:35.370242+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:44:11.881302+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:44:45.023280+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:49:11.805 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9345 8 365281 365273 0.305348 0 - 0.0299513 2026-02-23T15:49:12.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:49:11 trial068 ceph-mon[38805]: pgmap v5440: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:49:12.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:49:11 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3848993291' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:49:12.805 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9346 8 365281 365273 0.305316 0 - 0.0299513 2026-02-23T15:49:13.806 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9347 8 365281 365273 0.305283 0 - 0.0299513 2026-02-23T15:49:14.304 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:49:14.329 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:49:13 trial068 ceph-mon[38805]: pgmap v5441: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:49:14.330 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:49:13 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:49:13] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:49:14.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:49:13 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:49:13.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:49:14.421 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:49:14.519 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:49:14.517+0000 7f21a1327640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:49:14.519 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:49:14.517+0000 7f21a1327640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:49:14.629 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:49:14.629 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":339,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:46:50.233656+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:46:35.370242+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:44:11.881302+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:44:45.023280+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:49:14.806 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9348 8 365281 365273 0.30525 0 - 0.0299513 2026-02-23T15:49:15.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:49:14 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/4178844418' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:49:15.806 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9349 8 365281 365273 0.305218 0 - 0.0299513 2026-02-23T15:49:16.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:49:15 trial068 ceph-mon[38805]: pgmap v5442: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:49:16.806 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9350 8 365281 365273 0.305185 0 - 0.0299513 2026-02-23T15:49:17.666 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:49:17.787 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:49:17.806 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9351 8 365281 365273 0.305152 0 - 0.0299513 2026-02-23T15:49:17.893 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:49:17.891+0000 7fc0c6b27640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:49:17.893 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:49:17.892+0000 7fc0c6b27640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:49:18.002 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:49:18.003 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":339,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:46:50.233656+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:46:35.370242+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:44:11.881302+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:44:45.023280+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:49:18.003 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:49:17 trial068 ceph-mon[38805]: pgmap v5443: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:49:18.806 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9352 8 365281 365273 0.30512 0 - 0.0299513 2026-02-23T15:49:19.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:49:18 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/46343341' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:49:19.806 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9353 8 365281 365273 0.305087 0 - 0.0299513 2026-02-23T15:49:20.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:49:19 trial068 ceph-mon[38805]: pgmap v5444: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:49:20.806 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9354 8 365281 365273 0.305054 0 - 0.0299513 2026-02-23T15:49:21.040 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:49:21.160 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:49:21.182 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:49:20 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T15:49:21.182 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:49:20 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T15:49:21.262 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:49:21.261+0000 7fe57083f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:49:21.263 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:49:21.261+0000 7fe57083f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:49:21.371 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:49:21.372 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":339,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:46:50.233656+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:46:35.370242+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:44:11.881302+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:44:45.023280+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:49:21.806 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9355 8 365281 365273 0.305022 0 - 0.0299513 2026-02-23T15:49:22.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:49:21 trial068 ceph-mon[38805]: pgmap v5445: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:49:22.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:49:21 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/67002960' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:49:22.806 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9356 8 365281 365273 0.304989 0 - 0.0299513 2026-02-23T15:49:23.806 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9357 8 365281 365273 0.304957 0 - 0.0299513 2026-02-23T15:49:24.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:49:23 trial068 ceph-mon[38805]: pgmap v5446: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:49:24.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:49:23 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:49:23] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:49:24.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:49:23 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:49:23.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:49:24.410 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:49:24.530 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:49:24.625 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:49:24.623+0000 7f80e166c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:49:24.625 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:49:24.624+0000 7f80e166c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:49:24.735 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:49:24.735 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":339,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:46:50.233656+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:46:35.370242+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:44:11.881302+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:44:45.023280+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:49:24.806 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9358 8 365281 365273 0.304924 0 - 0.0299513 2026-02-23T15:49:25.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:49:24 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1597715653' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:49:25.806 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9359 8 365281 365273 0.304891 0 - 0.0299513 2026-02-23T15:49:26.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:49:25 trial068 ceph-mon[38805]: pgmap v5447: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:49:26.807 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T15:49:26.806585+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T15:49:26.807 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T15:49:26.807 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9360 8 365281 365273 0.304859 0 - 0.0299513 2026-02-23T15:49:27.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:49:26 trial068 ceph-mon[38805]: osdmap e340: 4 total, 3 up, 4 in 2026-02-23T15:49:27.771 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:49:27.807 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9361 8 365281 365273 0.304826 0 - 0.0299513 2026-02-23T15:49:27.887 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:49:27.985 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:49:27.984+0000 7f1bf8952640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:49:27.985 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:49:27.984+0000 7f1bf8952640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:49:28.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:49:27 trial068 ceph-mon[38805]: pgmap v5449: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:49:28.094 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:49:28.094 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":340,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:49:25.876072+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:46:35.370242+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:48:07.758788+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:44:45.023280+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:49:28.807 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9362 8 365281 365273 0.304794 0 - 0.0299513 2026-02-23T15:49:29.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:49:28 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T15:49:29.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:49:28 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T15:49:29.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:49:28 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T15:49:29.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:49:28 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T15:49:29.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:49:28 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2539365440' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:49:29.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:49:28 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2026-02-23T15:49:29.807 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9363 8 365281 365273 0.304761 0 - 0.0299513 2026-02-23T15:49:30.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:49:29 trial068 ceph-mon[38805]: pgmap v5450: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:49:30.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:49:29 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-23T15:49:30.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:49:29 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2026-02-23T15:49:30.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:49:29 trial068 ceph-mon[38805]: pgmap v5451: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:49:30.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:49:29 trial068 ceph-mon[38805]: pgmap v5452: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:49:30.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:49:29 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T15:49:30.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:49:29 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2026-02-23T15:49:30.807 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9364 8 365281 365273 0.304729 0 - 0.0299513 2026-02-23T15:49:31.132 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:49:31.248 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:49:31.354 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:49:31.352+0000 7f207defa640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:49:31.355 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:49:31.353+0000 7f207defa640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:49:31.463 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:49:31.463 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":341,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:49:30.253754+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:46:35.370242+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:48:07.758788+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:44:45.023280+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:49:31.463 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:49:31 trial068 ceph-mon[38805]: osdmap e341: 4 total, 3 up, 4 in 2026-02-23T15:49:31.464 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:49:31 trial068 ceph-mon[38805]: pgmap v5454: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:49:31.807 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9365 8 365281 365273 0.304696 0 - 0.0299513 2026-02-23T15:49:32.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:49:32 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2125031515' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:49:32.807 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9366 8 365281 365273 0.304664 0 - 0.0299513 2026-02-23T15:49:33.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:49:33 trial068 ceph-mon[38805]: pgmap v5455: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:49:33.807 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9367 8 365281 365273 0.304631 0 - 0.0299513 2026-02-23T15:49:34.265 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:49:33 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:49:33.974Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:49:34.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:49:33 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:49:33] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:49:34.502 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:49:34.621 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:49:34.722 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:49:34.719+0000 7ff7aaf5e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:49:34.722 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:49:34.720+0000 7ff7aaf5e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:49:34.807 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9368 8 365281 365273 0.304599 0 - 0.0299513 2026-02-23T15:49:34.830 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:49:34.830 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":341,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:49:30.253754+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:46:35.370242+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:48:07.758788+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:44:45.023280+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:49:35.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:49:34 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/865335606' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:49:35.807 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9369 8 365281 365273 0.304566 0 - 0.0299513 2026-02-23T15:49:36.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:49:35 trial068 ceph-mon[38805]: pgmap v5456: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:49:36.807 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9370 8 365281 365273 0.304534 0 - 0.0299513 2026-02-23T15:49:37.807 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9371 8 365281 365273 0.304501 0 - 0.0299513 2026-02-23T15:49:37.866 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:49:37.984 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:49:38.087 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:49:38.086+0000 7fd313322640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:49:38.087 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:49:38.086+0000 7fd313322640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:49:38.199 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:49:38.199 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":341,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:49:30.253754+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:46:35.370242+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:48:07.758788+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:44:45.023280+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:49:38.199 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:49:38 trial068 ceph-mon[38805]: pgmap v5457: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:49:38.807 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9372 8 365281 365273 0.304469 0 - 0.0299513 2026-02-23T15:49:39.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:49:39 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2410437481' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:49:39.808 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9373 8 365281 365273 0.304436 0 - 0.0299513 2026-02-23T15:49:40.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:49:40 trial068 ceph-mon[38805]: pgmap v5458: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:49:40.808 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9374 8 365281 365273 0.304404 0 - 0.0299513 2026-02-23T15:49:41.238 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:49:41.356 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:49:41.459 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:49:41.457+0000 7f44a1d20640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:49:41.459 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:49:41.457+0000 7f44a1d20640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:49:41.569 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:49:41.569 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":341,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:49:30.253754+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:46:35.370242+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:48:07.758788+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:44:45.023280+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:49:41.569 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:49:41 trial068 ceph-mon[38805]: pgmap v5459: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:49:41.808 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9375 8 365281 365273 0.304371 0 - 0.0299513 2026-02-23T15:49:42.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:49:42 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/578613007' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:49:42.808 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9376 8 365281 365273 0.304339 0 - 0.0299513 2026-02-23T15:49:43.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:49:43 trial068 ceph-mon[38805]: pgmap v5460: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:49:43.808 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9377 8 365281 365273 0.304306 0 - 0.0299513 2026-02-23T15:49:44.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:49:43 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:49:43] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:49:44.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:49:43 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:49:43.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:49:44.607 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:49:44.724 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:49:44.808 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9378 8 365281 365273 0.304274 0 - 0.0299513 2026-02-23T15:49:44.826 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:49:44.824+0000 7f666bccb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:49:44.826 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:49:44.824+0000 7f666bccb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:49:44.935 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:49:44.935 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":341,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:49:30.253754+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:46:35.370242+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:48:07.758788+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:44:45.023280+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:49:45.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:49:44 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1765684988' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:49:45.808 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9379 8 365281 365273 0.304241 0 - 0.0299513 2026-02-23T15:49:46.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:49:45 trial068 ceph-mon[38805]: pgmap v5461: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:49:46.808 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T15:49:46.808145+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T15:49:46.808 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T15:49:46.808 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9380 8 365281 365273 0.304209 0 - 0.0299513 2026-02-23T15:49:47.808 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9381 8 365281 365273 0.304176 0 - 0.0299513 2026-02-23T15:49:47.972 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:49:48.090 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:49:48.193 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:49:48.191+0000 7f541be54640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:49:48.193 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:49:48.191+0000 7f541be54640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:49:48.302 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:49:48.302 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":341,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:49:30.253754+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:46:35.370242+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:48:07.758788+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:44:45.023280+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:49:48.302 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:49:48 trial068 ceph-mon[38805]: 5.5 scrub starts 2026-02-23T15:49:48.302 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:49:48 trial068 ceph-mon[38805]: 5.5 scrub ok 2026-02-23T15:49:48.302 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:49:48 trial068 ceph-mon[38805]: pgmap v5462: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:49:48.808 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9382 8 365281 365273 0.304144 0 - 0.0299513 2026-02-23T15:49:49.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:49:49 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1991440984' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:49:49.808 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9383 8 365281 365273 0.304112 0 - 0.0299513 2026-02-23T15:49:50.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:49:50 trial068 ceph-mon[38805]: pgmap v5463: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:49:50.808 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9384 8 365281 365273 0.304079 0 - 0.0299513 2026-02-23T15:49:51.339 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:49:51.460 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:49:51.557 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:49:51.555+0000 7fd0d3362640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:49:51.557 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:49:51.555+0000 7fd0d3362640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:49:51.670 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:49:51.670 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":341,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:49:30.253754+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:46:35.370242+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:48:07.758788+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:44:45.023280+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:49:51.809 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9385 8 365281 365273 0.304047 0 - 0.0299513 2026-02-23T15:49:52.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:49:52 trial068 ceph-mon[38805]: pgmap v5464: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:49:52.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:49:52 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1405417392' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:49:52.809 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9386 8 365281 365273 0.304014 0 - 0.0299513 2026-02-23T15:49:53.809 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9387 8 365281 365273 0.303982 0 - 0.0299513 2026-02-23T15:49:54.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:49:54 trial068 ceph-mon[38805]: pgmap v5465: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:49:54.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:49:53 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:49:53] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:49:54.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:49:53 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:49:53.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:49:54.708 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:49:54.809 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9388 8 365281 365273 0.30395 0 - 0.0299513 2026-02-23T15:49:54.824 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:49:54.926 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:49:54.924+0000 7f88768d4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:49:54.926 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:49:54.924+0000 7f88768d4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:49:55.036 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:49:55.036 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":341,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:49:30.253754+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:46:35.370242+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:48:07.758788+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:44:45.023280+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:49:55.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:49:55 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3692768267' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:49:55.809 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9389 8 365281 365273 0.303917 0 - 0.0299513 2026-02-23T15:49:56.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:49:56 trial068 ceph-mon[38805]: pgmap v5466: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:49:56.809 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9390 8 365281 365273 0.303885 0 - 0.0299513 2026-02-23T15:49:57.809 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9391 8 365281 365273 0.303853 0 - 0.0299513 2026-02-23T15:49:58.074 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:49:58.192 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:49:58.288 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:49:58.286+0000 7fc2727ee640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:49:58.288 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:49:58.287+0000 7fc2727ee640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:49:58.395 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:49:58 trial068 ceph-mon[38805]: pgmap v5467: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:49:58.398 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:49:58.398 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":341,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:49:30.253754+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:46:35.370242+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:48:07.758788+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:44:45.023280+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:49:58.809 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9392 8 365281 365273 0.30382 0 - 0.0299513 2026-02-23T15:49:59.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:49:59 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2993525729' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:49:59.809 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9393 8 365281 365273 0.303788 0 - 0.0299513 2026-02-23T15:50:00.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:50:00 trial068 ceph-mon[38805]: pgmap v5468: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:50:00.809 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9394 8 365281 365273 0.303756 0 - 0.0299513 2026-02-23T15:50:01.436 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:50:01.556 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:50:01.579 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:50:01 trial068 ceph-mon[38805]: pgmap v5469: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:50:01.658 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:50:01.656+0000 7f0428238640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:50:01.658 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:50:01.656+0000 7f0428238640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:50:01.778 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:50:01.778 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":341,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:49:30.253754+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:46:35.370242+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:48:07.758788+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:44:45.023280+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:50:01.809 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9395 8 365281 365273 0.303723 0 - 0.0299513 2026-02-23T15:50:02.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:50:02 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3985847252' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:50:02.809 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9396 8 365281 365273 0.303691 0 - 0.0299513 2026-02-23T15:50:03.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:50:03 trial068 ceph-mon[38805]: 5.7 scrub starts 2026-02-23T15:50:03.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:50:03 trial068 ceph-mon[38805]: 5.7 scrub ok 2026-02-23T15:50:03.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:50:03 trial068 ceph-mon[38805]: pgmap v5470: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:50:03.809 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9397 8 365281 365273 0.303659 0 - 0.0299513 2026-02-23T15:50:04.265 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:50:03 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:50:03.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:50:04.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:50:03 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:50:03] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:50:04.809 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9398 8 365281 365273 0.303626 0 - 0.0299513 2026-02-23T15:50:04.816 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:50:04.937 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:50:05.032 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:50:05.030+0000 7ff902338640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:50:05.032 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:50:05.031+0000 7ff902338640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:50:05.141 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:50:05.141 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":341,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:49:30.253754+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:46:35.370242+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:48:07.758788+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:44:45.023280+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:50:05.810 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9399 8 365281 365273 0.303594 0 - 0.0299513 2026-02-23T15:50:06.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:50:06 trial068 ceph-mon[38805]: pgmap v5471: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:50:06.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:50:06 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1830582879' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:50:06.810 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T15:50:06.809606+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T15:50:06.810 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T15:50:06.810 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9400 8 365281 365273 0.303562 0 - 0.0299513 2026-02-23T15:50:07.810 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9401 8 365281 365273 0.303529 0 - 0.0299513 2026-02-23T15:50:08.178 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:50:08.296 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:50:08.392 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:50:08.391+0000 7f4f1ee0a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:50:08.393 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:50:08 trial068 ceph-mon[38805]: pgmap v5472: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:50:08.393 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:50:08.391+0000 7f4f1ee0a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:50:08.501 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:50:08.501 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":341,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:49:30.253754+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:46:35.370242+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:48:07.758788+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:44:45.023280+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:50:08.810 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9402 8 365281 365273 0.303497 0 - 0.0299513 2026-02-23T15:50:09.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:50:09 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3864171716' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:50:09.810 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9403 8 365281 365273 0.303465 0 - 0.0299513 2026-02-23T15:50:10.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:50:10 trial068 ceph-mon[38805]: 5.8 scrub starts 2026-02-23T15:50:10.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:50:10 trial068 ceph-mon[38805]: 5.8 scrub ok 2026-02-23T15:50:10.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:50:10 trial068 ceph-mon[38805]: pgmap v5473: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:50:10.810 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9404 8 365281 365273 0.303433 0 - 0.0299513 2026-02-23T15:50:11.538 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:50:11.656 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:50:11.758 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:50:11.756+0000 7f07f0c21640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:50:11.758 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:50:11.757+0000 7f07f0c21640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:50:11.810 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9405 8 365281 365273 0.3034 0 - 0.0299513 2026-02-23T15:50:11.867 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:50:11.868 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":341,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:49:30.253754+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:46:35.370242+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:48:07.758788+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:44:45.023280+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:50:12.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:50:12 trial068 ceph-mon[38805]: pgmap v5474: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:50:12.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:50:12 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/4269071802' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:50:12.810 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9406 8 365281 365273 0.303368 0 - 0.0299513 2026-02-23T15:50:13.810 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9407 8 365281 365273 0.303336 0 - 0.0299513 2026-02-23T15:50:14.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:50:14 trial068 ceph-mon[38805]: pgmap v5475: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:50:14.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:50:13 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:50:13] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:50:14.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:50:13 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:50:13.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:50:14.810 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9408 8 365281 365273 0.303303 0 - 0.0299513 2026-02-23T15:50:14.905 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:50:15.022 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:50:15.124 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:50:15.123+0000 7f0c6e996640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:50:15.124 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:50:15.123+0000 7f0c6e996640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:50:15.233 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:50:15.233 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":341,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:49:30.253754+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:46:35.370242+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:48:07.758788+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:44:45.023280+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:50:15.810 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9409 8 365281 365273 0.303271 0 - 0.0299513 2026-02-23T15:50:16.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:50:16 trial068 ceph-mon[38805]: 5.a scrub starts 2026-02-23T15:50:16.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:50:16 trial068 ceph-mon[38805]: 5.a scrub ok 2026-02-23T15:50:16.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:50:16 trial068 ceph-mon[38805]: pgmap v5476: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:50:16.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:50:16 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1377935235' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:50:16.810 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9410 8 365281 365273 0.303239 0 - 0.0299513 2026-02-23T15:50:17.810 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9411 8 365281 365273 0.303207 0 - 0.0299513 2026-02-23T15:50:18.289 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:50:18.406 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:50:18.428 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:50:18 trial068 ceph-mon[38805]: pgmap v5477: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:50:18.506 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:50:18.504+0000 7fe52d858640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:50:18.506 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:50:18.505+0000 7fe52d858640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:50:18.616 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:50:18.616 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":341,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:49:30.253754+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:46:35.370242+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:48:07.758788+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:44:45.023280+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:50:18.810 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9412 8 365281 365273 0.303175 0 - 0.0299513 2026-02-23T15:50:19.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:50:19 trial068 ceph-mon[38805]: 5.e scrub starts 2026-02-23T15:50:19.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:50:19 trial068 ceph-mon[38805]: 5.e scrub ok 2026-02-23T15:50:19.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:50:19 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/846082361' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:50:19.810 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9413 8 365281 365273 0.303142 0 - 0.0299513 2026-02-23T15:50:20.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:50:20 trial068 ceph-mon[38805]: pgmap v5478: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:50:20.811 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9414 8 365281 365273 0.30311 0 - 0.0299513 2026-02-23T15:50:21.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:50:21 trial068 ceph-mon[38805]: pgmap v5479: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:50:21.653 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:50:21.770 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:50:21.811 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9415 8 365281 365273 0.303078 0 - 0.0299513 2026-02-23T15:50:21.870 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:50:21.868+0000 7f916304f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:50:21.870 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:50:21.868+0000 7f916304f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:50:21.981 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:50:21.982 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":341,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:49:30.253754+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:46:35.370242+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:48:07.758788+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:44:45.023280+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:50:22.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:50:22 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/523038984' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:50:22.811 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9416 8 365281 365273 0.303046 0 - 0.0299513 2026-02-23T15:50:23.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:50:23 trial068 ceph-mon[38805]: pgmap v5480: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:50:23.811 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9417 8 365281 365273 0.303014 0 - 0.0299513 2026-02-23T15:50:24.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:50:23 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:50:23] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:50:24.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:50:23 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:50:23.972Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:50:24.811 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9418 8 365281 365273 0.302981 0 - 0.0299513 2026-02-23T15:50:25.018 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:50:25.135 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:50:25.238 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:50:25.236+0000 7f88ef739640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:50:25.238 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:50:25.236+0000 7f88ef739640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:50:25.348 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:50:25.348 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":341,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:49:30.253754+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:46:35.370242+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:48:07.758788+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:44:45.023280+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:50:25.811 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9419 8 365281 365273 0.302949 0 - 0.0299513 2026-02-23T15:50:26.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:50:26 trial068 ceph-mon[38805]: pgmap v5481: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:50:26.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:50:26 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2152959403' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:50:26.811 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T15:50:26.810983+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T15:50:26.811 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T15:50:26.811 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9420 8 365281 365273 0.302917 0 - 0.0299513 2026-02-23T15:50:27.811 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9421 8 365281 365273 0.302885 0 - 0.0299513 2026-02-23T15:50:28.384 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:50:28.395 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:50:28 trial068 ceph-mon[38805]: pgmap v5482: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:50:28.501 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:50:28.600 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:50:28.598+0000 7f3283dfb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:50:28.600 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:50:28.598+0000 7f3283dfb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:50:28.708 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:50:28.709 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":341,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:49:30.253754+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:46:35.370242+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:48:07.758788+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:44:45.023280+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:50:28.811 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9422 8 365281 365273 0.302853 0 - 0.0299513 2026-02-23T15:50:29.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:50:29 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3340991390' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:50:29.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:50:29 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2026-02-23T15:50:29.811 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9423 8 365281 365273 0.302821 0 - 0.0299513 2026-02-23T15:50:30.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:50:30 trial068 ceph-mon[38805]: pgmap v5483: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:50:30.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:50:30 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-23T15:50:30.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:50:30 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2026-02-23T15:50:30.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:50:30 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T15:50:30.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:50:30 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2026-02-23T15:50:30.811 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9424 8 365281 365273 0.302789 0 - 0.0299513 2026-02-23T15:50:31.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:50:31 trial068 ceph-mon[38805]: pgmap v5484: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:50:31.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:50:31 trial068 ceph-mon[38805]: pgmap v5485: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:50:31.745 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:50:31.811 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9425 8 365281 365273 0.302756 0 - 0.0299513 2026-02-23T15:50:31.861 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:50:31.962 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:50:31.961+0000 7f9916e71640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:50:31.963 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:50:31.961+0000 7f9916e71640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:50:32.076 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:50:32.076 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":341,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:49:30.253754+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:46:35.370242+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:48:07.758788+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:44:45.023280+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:50:32.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:50:32 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1389722998' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:50:32.811 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9426 8 365281 365273 0.302724 0 - 0.0299513 2026-02-23T15:50:33.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:50:33 trial068 ceph-mon[38805]: pgmap v5486: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:50:33.811 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9427 8 365281 365273 0.302692 0 - 0.0299513 2026-02-23T15:50:34.265 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:50:33 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:50:33.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:50:34.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:50:34 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T15:50:34.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:50:34 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T15:50:34.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:50:33 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:50:33] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:50:34.811 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9428 8 365281 365273 0.30266 0 - 0.0299513 2026-02-23T15:50:35.114 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:50:35.235 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:50:35.338 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:50:35.336+0000 7f5d86b1c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:50:35.338 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:50:35.337+0000 7f5d86b1c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:50:35.447 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:50:35.447 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":341,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:49:30.253754+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:46:35.370242+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:48:07.758788+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:44:45.023280+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:50:35.448 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:50:35 trial068 ceph-mon[38805]: pgmap v5487: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:50:35.812 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9429 8 365281 365273 0.302628 0 - 0.0299513 2026-02-23T15:50:36.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:50:36 trial068 ceph-mon[38805]: pgmap v5488: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:50:36.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:50:36 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/970986803' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:50:36.812 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9430 8 365281 365273 0.302596 0 - 0.0299513 2026-02-23T15:50:37.812 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9431 8 365281 365273 0.302564 0 - 0.0299513 2026-02-23T15:50:38.486 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:50:38.602 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:50:38.642 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:50:38 trial068 ceph-mon[38805]: pgmap v5489: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:50:38.699 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:50:38.698+0000 7f0e0da58640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:50:38.700 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:50:38.698+0000 7f0e0da58640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:50:38.808 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:50:38.808 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":341,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:49:30.253754+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:46:35.370242+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:48:07.758788+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:44:45.023280+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:50:38.812 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9432 8 365281 365273 0.302532 0 - 0.0299513 2026-02-23T15:50:39.812 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9433 8 365281 365273 0.3025 0 - 0.0299513 2026-02-23T15:50:39.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:50:39 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/4084253436' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:50:39.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:50:39 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T15:50:39.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:50:39 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T15:50:40.812 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9434 8 365281 365273 0.302468 0 - 0.0299513 2026-02-23T15:50:40.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:50:40 trial068 ceph-mon[38805]: pgmap v5490: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:50:41.812 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9435 8 365281 365273 0.302436 0 - 0.0299513 2026-02-23T15:50:41.846 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:50:41.964 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:50:42.061 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:50:42.060+0000 7fa1a2d7a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:50:42.062 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:50:42.060+0000 7fa1a2d7a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:50:42.173 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:50:42.173 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":341,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:49:30.253754+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:46:35.370242+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:48:07.758788+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:44:45.023280+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:50:42.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:50:42 trial068 ceph-mon[38805]: pgmap v5491: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:50:42.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:50:42 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1068300667' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:50:42.812 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9436 8 365281 365273 0.302403 0 - 0.0299513 2026-02-23T15:50:43.812 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9437 8 365281 365273 0.302371 0 - 0.0299513 2026-02-23T15:50:43.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:50:43 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T15:50:43.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:50:43 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T15:50:44.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:50:43 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:50:43] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:50:44.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:50:43 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:50:43.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:50:44.812 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9438 8 365281 365273 0.302339 0 - 0.0299513 2026-02-23T15:50:44.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:50:44 trial068 ceph-mon[38805]: pgmap v5492: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:50:45.210 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:50:45.328 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:50:45.427 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:50:45.425+0000 7fd572f85640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:50:45.427 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:50:45.425+0000 7fd572f85640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:50:45.535 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:50:45.536 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":343,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:50:45.262972+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:46:35.370242+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:48:07.758788+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:50:33.211242+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:50:45.536 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:50:45 trial068 ceph-mon[38805]: osdmap e342: 4 total, 3 up, 4 in 2026-02-23T15:50:45.536 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:50:45 trial068 ceph-mon[38805]: osdmap e343: 4 total, 3 up, 4 in 2026-02-23T15:50:45.812 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9439 8 365281 365273 0.302307 0 - 0.0299513 2026-02-23T15:50:46.813 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T15:50:46.812455+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T15:50:46.813 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T15:50:46.813 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9440 8 365281 365273 0.302275 0 - 0.0299513 2026-02-23T15:50:46.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:50:46 trial068 ceph-mon[38805]: pgmap v5495: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:50:46.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:50:46 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/414757462' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:50:47.813 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9441 8 365281 365273 0.302243 0 - 0.0299513 2026-02-23T15:50:48.573 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:50:48.691 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:50:48.713 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:50:48 trial068 ceph-mon[38805]: pgmap v5496: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:50:48.790 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:50:48.788+0000 7f4c62a16640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:50:48.790 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:50:48.788+0000 7f4c62a16640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:50:48.813 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9442 8 365281 365273 0.302211 0 - 0.0299513 2026-02-23T15:50:48.900 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:50:48.900 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":343,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:50:45.262972+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:46:35.370242+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:48:07.758788+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:50:33.211242+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:50:49.813 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9443 8 365281 365273 0.302179 0 - 0.0299513 2026-02-23T15:50:49.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:50:49 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/259731467' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:50:50.813 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9444 8 365281 365273 0.302147 0 - 0.0299513 2026-02-23T15:50:50.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:50:50 trial068 ceph-mon[38805]: pgmap v5497: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:50:51.813 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9445 8 365281 365273 0.302115 0 - 0.0299513 2026-02-23T15:50:51.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:50:51 trial068 ceph-mon[38805]: 5.5 scrub starts 2026-02-23T15:50:51.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:50:51 trial068 ceph-mon[38805]: 5.5 scrub ok 2026-02-23T15:50:51.937 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:50:52.055 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:50:52.153 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:50:52.151+0000 7ff9663ce640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:50:52.153 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:50:52.151+0000 7ff9663ce640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:50:52.265 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:50:52.265 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":343,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:50:45.262972+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:46:35.370242+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:48:07.758788+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:50:33.211242+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:50:52.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:50:52 trial068 ceph-mon[38805]: pgmap v5498: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:50:52.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:50:52 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1732423567' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:50:52.813 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9446 8 365281 365273 0.302083 0 - 0.0299513 2026-02-23T15:50:53.813 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9447 8 365281 365273 0.302051 0 - 0.0299513 2026-02-23T15:50:54.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:50:53 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:50:53] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:50:54.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:50:53 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:50:53.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:50:54.813 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9448 8 365281 365273 0.302019 0 - 0.0299513 2026-02-23T15:50:54.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:50:54 trial068 ceph-mon[38805]: pgmap v5499: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:50:55.303 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:50:55.421 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:50:55.517 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:50:55.515+0000 7fe716973640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:50:55.517 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:50:55.515+0000 7fe716973640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:50:55.627 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:50:55.627 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":343,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:50:45.262972+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:46:35.370242+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:48:07.758788+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:50:33.211242+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:50:55.813 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9449 8 365281 365273 0.301987 0 - 0.0299513 2026-02-23T15:50:56.813 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9450 8 365281 365273 0.301955 0 - 0.0299513 2026-02-23T15:50:56.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:50:56 trial068 ceph-mon[38805]: pgmap v5500: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:50:56.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:50:56 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2809382059' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:50:57.813 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9451 8 365281 365273 0.301924 0 - 0.0299513 2026-02-23T15:50:58.645 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:50:58 trial068 ceph-mon[38805]: pgmap v5501: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:50:58.664 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:50:58.779 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:50:58.813 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9452 8 365281 365273 0.301892 0 - 0.0299513 2026-02-23T15:50:58.885 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:50:58.883+0000 7f858e933640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:50:58.885 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:50:58.883+0000 7f858e933640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:50:58.994 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:50:58.995 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":343,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:50:45.262972+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:46:35.370242+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:48:07.758788+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:50:33.211242+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:50:59.814 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9453 8 365281 365273 0.30186 0 - 0.0299513 2026-02-23T15:50:59.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:50:59 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2675092731' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:51:00.814 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9454 8 365281 365273 0.301828 0 - 0.0299513 2026-02-23T15:51:00.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:51:00 trial068 ceph-mon[38805]: pgmap v5502: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:51:01.814 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9455 8 365281 365273 0.301796 0 - 0.0299513 2026-02-23T15:51:02.033 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:51:02.151 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:51:02.249 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:51:02.247+0000 7f3ef8eea640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:51:02.249 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:51:02.247+0000 7f3ef8eea640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:51:02.357 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:51:02.357 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":343,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:50:45.262972+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:46:35.370242+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:48:07.758788+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:50:33.211242+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:51:02.814 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9456 8 365281 365273 0.301764 0 - 0.0299513 2026-02-23T15:51:02.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:51:02 trial068 ceph-mon[38805]: pgmap v5503: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:51:02.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:51:02 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/679389591' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:51:03.814 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9457 8 365281 365273 0.301732 0 - 0.0299513 2026-02-23T15:51:04.263 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:51:03 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:51:03.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:51:04.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:51:03 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:51:03] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:51:04.814 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9458 8 365281 365273 0.3017 0 - 0.0299513 2026-02-23T15:51:04.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:51:04 trial068 ceph-mon[38805]: pgmap v5504: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:51:05.393 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:51:05.513 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:51:05.622 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:51:05.621+0000 7f28dc37b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:51:05.623 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:51:05.621+0000 7f28dc37b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:51:05.734 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:51:05.734 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":343,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:50:45.262972+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:46:35.370242+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:48:07.758788+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:50:33.211242+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:51:05.814 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9459 8 365281 365273 0.301668 0 - 0.0299513 2026-02-23T15:51:06.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:51:06 trial068 ceph-mon[38805]: pgmap v5505: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:51:06.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:51:06 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1191588105' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:51:06.814 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T15:51:06.813929+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T15:51:06.814 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T15:51:06.814 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9460 8 365281 365273 0.301636 0 - 0.0299513 2026-02-23T15:51:07.814 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9461 8 365281 365273 0.301604 0 - 0.0299513 2026-02-23T15:51:08.771 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:51:08.796 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:51:08 trial068 ceph-mon[38805]: pgmap v5506: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:51:08.814 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9462 8 365281 365273 0.301573 0 - 0.0299513 2026-02-23T15:51:08.889 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:51:08.991 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:51:08.989+0000 7f8eb2dfd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:51:08.991 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:51:08.989+0000 7f8eb2dfd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:51:09.106 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:51:09.107 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":343,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:50:45.262972+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:46:35.370242+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:48:07.758788+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:50:33.211242+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:51:09.814 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9463 8 365281 365273 0.301541 0 - 0.0299513 2026-02-23T15:51:09.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:51:09 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/296942671' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:51:10.814 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9464 8 365281 365273 0.301509 0 - 0.0299513 2026-02-23T15:51:10.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:51:10 trial068 ceph-mon[38805]: pgmap v5507: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:51:11.814 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9465 8 365281 365273 0.301477 0 - 0.0299513 2026-02-23T15:51:12.144 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:51:12.261 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:51:12.366 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:51:12.364+0000 7f17dde16640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:51:12.366 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:51:12.364+0000 7f17dde16640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:51:12.474 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:51:12.475 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":343,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:50:45.262972+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:46:35.370242+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:48:07.758788+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:50:33.211242+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:51:12.475 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:51:12 trial068 ceph-mon[38805]: pgmap v5508: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:51:12.814 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9466 8 365281 365273 0.301445 0 - 0.0299513 2026-02-23T15:51:13.814 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9467 8 365281 365273 0.301413 0 - 0.0299513 2026-02-23T15:51:13.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:51:13 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2635738021' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:51:14.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:51:13 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:51:13] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:51:14.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:51:13 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:51:13.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:51:14.814 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9468 8 365281 365273 0.301381 0 - 0.0299513 2026-02-23T15:51:14.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:51:14 trial068 ceph-mon[38805]: pgmap v5509: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:51:15.513 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:51:15.631 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:51:15.732 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:51:15.730+0000 7fcdced5c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:51:15.732 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:51:15.730+0000 7fcdced5c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:51:15.814 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9469 8 365281 365273 0.30135 0 - 0.0299513 2026-02-23T15:51:15.841 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:51:15.842 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":343,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:50:45.262972+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:46:35.370242+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:48:07.758788+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:50:33.211242+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:51:16.815 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9470 8 365281 365273 0.301318 0 - 0.0299513 2026-02-23T15:51:16.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:51:16 trial068 ceph-mon[38805]: pgmap v5510: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:51:16.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:51:16 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3967094091' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:51:17.815 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9471 8 365281 365273 0.301286 0 - 0.0299513 2026-02-23T15:51:18.815 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9472 8 365281 365273 0.301254 0 - 0.0299513 2026-02-23T15:51:18.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:51:18 trial068 ceph-mon[38805]: pgmap v5511: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:51:18.878 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:51:18.996 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:51:19.096 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:51:19.094+0000 7f39cd8bd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:51:19.096 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:51:19.094+0000 7f39cd8bd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:51:19.206 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:51:19.206 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":343,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:50:45.262972+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:46:35.370242+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:48:07.758788+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:50:33.211242+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:51:19.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:51:19 trial068 ceph-mon[38805]: 5.a scrub starts 2026-02-23T15:51:19.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:51:19 trial068 ceph-mon[38805]: 5.a scrub ok 2026-02-23T15:51:19.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:51:19 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3881969949' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:51:19.815 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9473 8 365281 365273 0.301222 0 - 0.0299513 2026-02-23T15:51:20.815 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9474 8 365281 365273 0.301191 0 - 0.0299513 2026-02-23T15:51:20.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:51:20 trial068 ceph-mon[38805]: pgmap v5512: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:51:21.815 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9475 8 365281 365273 0.301159 0 - 0.0299513 2026-02-23T15:51:22.244 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:51:22.364 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:51:22.464 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:51:22.462+0000 7f7cf5b08640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:51:22.464 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:51:22.462+0000 7f7cf5b08640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:51:22.578 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:51:22.579 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":343,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:50:45.262972+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:46:35.370242+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:48:07.758788+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:50:33.211242+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:51:22.579 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:51:22 trial068 ceph-mon[38805]: pgmap v5513: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:51:22.815 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9476 8 365281 365273 0.301127 0 - 0.0299513 2026-02-23T15:51:23.815 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9477 8 365281 365273 0.301095 0 - 0.0299513 2026-02-23T15:51:23.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:51:23 trial068 ceph-mon[38805]: 5.8 scrub starts 2026-02-23T15:51:23.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:51:23 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3630352622' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:51:23.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:51:23 trial068 ceph-mon[38805]: 5.8 scrub ok 2026-02-23T15:51:24.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:51:23 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:51:23] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:51:24.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:51:23 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:51:23.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:51:24.815 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9478 8 365281 365273 0.301063 0 - 0.0299513 2026-02-23T15:51:24.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:51:24 trial068 ceph-mon[38805]: pgmap v5514: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:51:25.620 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:51:25.738 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:51:25.815 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9479 8 365281 365273 0.301032 0 - 0.0299513 2026-02-23T15:51:25.836 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:51:25.835+0000 7f67e9f97640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:51:25.836 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:51:25.835+0000 7f67e9f97640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:51:25.946 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:51:25.946 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":343,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:50:45.262972+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:46:35.370242+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:48:07.758788+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:50:33.211242+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:51:26.816 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T15:51:26.815410+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T15:51:26.816 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T15:51:26.816 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9480 8 365281 365273 0.301 0 - 0.0299513 2026-02-23T15:51:26.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:51:26 trial068 ceph-mon[38805]: pgmap v5515: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:51:26.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:51:26 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2723799262' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:51:27.816 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9481 8 365281 365273 0.300968 0 - 0.0299513 2026-02-23T15:51:28.816 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9482 8 365281 365273 0.300936 0 - 0.0299513 2026-02-23T15:51:28.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:51:28 trial068 ceph-mon[38805]: pgmap v5516: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:51:28.982 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:51:29.098 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:51:29.199 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:51:29.197+0000 7fc35aff3640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:51:29.199 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:51:29.197+0000 7fc35aff3640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:51:29.308 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:51:29.309 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":343,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:50:45.262972+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:46:35.370242+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:48:07.758788+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:50:33.211242+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:51:29.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:51:29 trial068 ceph-mon[38805]: 5.7 scrub starts 2026-02-23T15:51:29.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:51:29 trial068 ceph-mon[38805]: 5.7 scrub ok 2026-02-23T15:51:29.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:51:29 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1323599906' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:51:29.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:51:29 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2026-02-23T15:51:29.816 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9483 8 365281 365273 0.300905 0 - 0.0299513 2026-02-23T15:51:30.816 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9484 8 365281 365273 0.300873 0 - 0.0299513 2026-02-23T15:51:30.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:51:30 trial068 ceph-mon[38805]: pgmap v5517: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:51:30.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:51:30 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-23T15:51:30.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:51:30 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2026-02-23T15:51:30.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:51:30 trial068 ceph-mon[38805]: pgmap v5518: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:51:30.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:51:30 trial068 ceph-mon[38805]: pgmap v5519: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:51:30.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:51:30 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T15:51:30.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:51:30 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2026-02-23T15:51:31.816 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9485 8 365281 365273 0.300841 0 - 0.0299513 2026-02-23T15:51:32.346 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:51:32.463 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:51:32.564 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:51:32.562+0000 7f64e9977640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:51:32.564 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:51:32.562+0000 7f64e9977640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:51:32.672 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:51:32.673 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":343,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:50:45.262972+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:46:35.370242+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:48:07.758788+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:50:33.211242+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:51:32.816 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9486 8 365281 365273 0.30081 0 - 0.0299513 2026-02-23T15:51:33.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:51:32 trial068 ceph-mon[38805]: pgmap v5520: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:51:33.816 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9487 8 365281 365273 0.300778 0 - 0.0299513 2026-02-23T15:51:33.973 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:51:33 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3328856061' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:51:34.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:51:33 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:51:33] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:51:34.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:51:33 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:51:33.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:51:34.816 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9488 8 365281 365273 0.300746 0 - 0.0299513 2026-02-23T15:51:35.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:51:34 trial068 ceph-mon[38805]: pgmap v5521: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:51:35.710 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:51:35.816 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9489 8 365281 365273 0.300714 0 - 0.0299513 2026-02-23T15:51:35.827 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:51:35.929 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:51:35.927+0000 7fdb2ae54640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:51:35.929 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:51:35.927+0000 7fdb2ae54640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:51:36.038 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:51:36.039 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":343,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:50:45.262972+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:46:35.370242+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:48:07.758788+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:50:33.211242+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:51:36.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:51:36 trial068 ceph-mon[38805]: pgmap v5522: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:51:36.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:51:36 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/715555275' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:51:36.817 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9490 8 365281 365273 0.300683 0 - 0.0299513 2026-02-23T15:51:37.817 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9491 8 365281 365273 0.300651 0 - 0.0299513 2026-02-23T15:51:38.817 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9492 8 365281 365273 0.300619 0 - 0.0299513 2026-02-23T15:51:39.076 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:51:39.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:51:38 trial068 ceph-mon[38805]: pgmap v5523: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:51:39.193 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:51:39.297 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:51:39.295+0000 7f2f5584c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:51:39.297 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:51:39.295+0000 7f2f5584c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:51:39.406 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:51:39.407 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":343,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:50:45.262972+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:46:35.370242+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:48:07.758788+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:50:33.211242+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:51:39.817 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9493 8 365281 365273 0.300588 0 - 0.0299513 2026-02-23T15:51:39.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:51:39 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/4226261693' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:51:40.817 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9494 8 365281 365273 0.300556 0 - 0.0299513 2026-02-23T15:51:41.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:51:40 trial068 ceph-mon[38805]: pgmap v5524: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:51:41.817 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9495 8 365281 365273 0.300524 0 - 0.0299513 2026-02-23T15:51:42.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:51:41 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T15:51:42.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:51:41 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T15:51:42.443 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:51:42.561 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:51:42.662 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:51:42.660+0000 7ffab7d90640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:51:42.662 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:51:42.660+0000 7ffab7d90640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:51:42.770 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:51:42.770 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":343,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:50:45.262972+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:46:35.370242+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:48:07.758788+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:50:33.211242+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:51:42.770 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:51:42 trial068 ceph-mon[38805]: pgmap v5525: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:51:42.817 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9496 8 365281 365273 0.300493 0 - 0.0299513 2026-02-23T15:51:43.817 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9497 8 365281 365273 0.300461 0 - 0.0299513 2026-02-23T15:51:43.974 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:51:43 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/4147251222' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:51:43.974 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:51:43 trial068 ceph-mon[38805]: 5.e scrub starts 2026-02-23T15:51:43.974 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:51:43 trial068 ceph-mon[38805]: 5.e scrub ok 2026-02-23T15:51:44.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:51:43 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:51:43] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:51:44.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:51:43 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:51:43.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:51:44.817 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9498 8 365281 365273 0.300429 0 - 0.0299513 2026-02-23T15:51:45.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:51:44 trial068 ceph-mon[38805]: pgmap v5526: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:51:45.807 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:51:45.817 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9499 8 365281 365273 0.300398 0 - 0.0299513 2026-02-23T15:51:45.924 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:51:46.025 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:51:46.023+0000 7f8d5f2cd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:51:46.025 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:51:46.023+0000 7f8d5f2cd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:51:46.137 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:51:46.138 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":343,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:50:45.262972+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:46:35.370242+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:48:07.758788+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:50:33.211242+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:51:46.817 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T15:51:46.817218+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T15:51:46.818 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T15:51:46.818 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9500 8 365281 365273 0.300366 0 - 0.0299513 2026-02-23T15:51:47.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:51:46 trial068 ceph-mon[38805]: pgmap v5527: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:51:47.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:51:46 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2841634214' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:51:47.817 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9501 8 365281 365273 0.300335 0 - 0.0299513 2026-02-23T15:51:48.817 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9502 8 365281 365273 0.300303 0 - 0.0299513 2026-02-23T15:51:49.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:51:48 trial068 ceph-mon[38805]: pgmap v5528: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:51:49.176 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:51:49.295 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:51:49.395 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:51:49.393+0000 7f11887af640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:51:49.395 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:51:49.393+0000 7f11887af640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:51:49.506 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:51:49.507 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":344,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:51:48.647580+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:50:39.080345+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:48:07.758788+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:50:33.211242+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:51:49.818 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9503 8 365281 365273 0.300271 0 - 0.0299513 2026-02-23T15:51:49.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:51:49 trial068 ceph-mon[38805]: osdmap e344: 4 total, 3 up, 4 in 2026-02-23T15:51:49.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:51:49 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/535829936' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:51:50.818 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9504 8 365281 365273 0.30024 0 - 0.0299513 2026-02-23T15:51:51.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:51:50 trial068 ceph-mon[38805]: pgmap v5530: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:51:51.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:51:50 trial068 ceph-mon[38805]: osdmap e345: 4 total, 3 up, 4 in 2026-02-23T15:51:51.818 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9505 8 365281 365273 0.300208 0 - 0.0299513 2026-02-23T15:51:52.545 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:51:52.665 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:51:52.766 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:51:52.764+0000 7f16b51a1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:51:52.766 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:51:52.764+0000 7f16b51a1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:51:52.818 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9506 8 365281 365273 0.300177 0 - 0.0299513 2026-02-23T15:51:52.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:51:52 trial068 ceph-mon[38805]: pgmap v5532: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:51:52.874 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:51:52.875 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":345,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:51:50.270827+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:50:39.080345+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:48:07.758788+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:50:33.211242+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:51:53.818 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9507 8 365281 365273 0.300145 0 - 0.0299513 2026-02-23T15:51:53.974 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:51:53 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2266057321' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:51:54.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:51:53 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:51:53] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:51:54.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:51:53 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:51:53.974Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:51:54.818 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9508 8 365281 365273 0.300114 0 - 0.0299513 2026-02-23T15:51:55.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:51:54 trial068 ceph-mon[38805]: pgmap v5533: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:51:55.818 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9509 8 365281 365273 0.300082 0 - 0.0299513 2026-02-23T15:51:55.928 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:51:56.047 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:51:56.148 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:51:56.146+0000 7fe53d04d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:51:56.149 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:51:56.147+0000 7fe53d04d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:51:56.259 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:51:56.259 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":345,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:51:50.270827+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:50:39.080345+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:48:07.758788+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:50:33.211242+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:51:56.818 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9510 8 365281 365273 0.30005 0 - 0.0299513 2026-02-23T15:51:57.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:51:56 trial068 ceph-mon[38805]: pgmap v5534: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:51:57.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:51:56 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1148902940' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:51:57.818 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9511 8 365281 365273 0.300019 0 - 0.0299513 2026-02-23T15:51:58.818 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9512 8 365281 365273 0.299987 0 - 0.0299513 2026-02-23T15:51:59.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:51:58 trial068 ceph-mon[38805]: pgmap v5535: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:51:59.297 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:51:59.413 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:51:59.511 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:51:59.509+0000 7f8ba8c37640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:51:59.511 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:51:59.509+0000 7f8ba8c37640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:51:59.621 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:51:59.621 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":345,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:51:50.270827+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:50:39.080345+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:48:07.758788+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:50:33.211242+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:51:59.818 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9513 8 365281 365273 0.299956 0 - 0.0299513 2026-02-23T15:52:00.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:51:59 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/716302392' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:52:00.818 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9514 8 365281 365273 0.299924 0 - 0.0299513 2026-02-23T15:52:01.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:52:00 trial068 ceph-mon[38805]: pgmap v5536: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:52:01.818 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9515 8 365281 365273 0.299893 0 - 0.0299513 2026-02-23T15:52:02.658 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:52:02.777 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:52:02.818 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9516 8 365281 365273 0.299861 0 - 0.0299513 2026-02-23T15:52:02.881 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:52:02.879+0000 7ff964234640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:52:02.881 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:52:02.879+0000 7ff964234640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:52:02.992 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:52:02.992 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":345,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:51:50.270827+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:50:39.080345+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:48:07.758788+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:50:33.211242+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:52:02.992 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:52:02 trial068 ceph-mon[38805]: pgmap v5537: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:52:02.992 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:52:02 trial068 ceph-mon[38805]: 5.5 scrub starts 2026-02-23T15:52:03.819 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9517 8 365281 365273 0.29983 0 - 0.0299513 2026-02-23T15:52:03.974 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:52:03 trial068 ceph-mon[38805]: 5.5 scrub ok 2026-02-23T15:52:03.974 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:52:03 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2883990155' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:52:04.265 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:52:03 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:52:03.974Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:52:04.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:52:03 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:52:03] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:52:04.819 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9518 8 365281 365273 0.299798 0 - 0.0299513 2026-02-23T15:52:05.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:52:04 trial068 ceph-mon[38805]: pgmap v5538: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:52:05.819 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9519 8 365281 365273 0.299767 0 - 0.0299513 2026-02-23T15:52:06.029 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:52:06.054 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:52:05 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T15:52:06.054 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:52:05 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T15:52:06.054 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:52:05 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T15:52:06.055 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:52:05 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T15:52:06.147 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:52:06.251 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:52:06.249+0000 7f0687a93640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:52:06.251 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:52:06.249+0000 7f0687a93640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:52:06.361 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:52:06.361 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":345,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:51:50.270827+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:50:39.080345+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:48:07.758788+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:50:33.211242+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:52:06.819 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T15:52:06.818750+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T15:52:06.819 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T15:52:06.819 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9520 8 365281 365273 0.299735 0 - 0.0299513 2026-02-23T15:52:07.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:52:06 trial068 ceph-mon[38805]: pgmap v5539: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:52:07.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:52:06 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1653546487' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:52:07.819 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9521 8 365281 365273 0.299704 0 - 0.0299513 2026-02-23T15:52:08.819 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9522 8 365281 365273 0.299672 0 - 0.0299513 2026-02-23T15:52:09.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:52:08 trial068 ceph-mon[38805]: pgmap v5540: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:52:09.399 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:52:09.517 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:52:09.619 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:52:09.617+0000 7f668ee69640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:52:09.619 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:52:09.617+0000 7f668ee69640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:52:09.729 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:52:09.729 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":345,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:51:50.270827+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:50:39.080345+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:48:07.758788+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:50:33.211242+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:52:09.819 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9523 8 365281 365273 0.299641 0 - 0.0299513 2026-02-23T15:52:10.819 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9524 8 365281 365273 0.299609 0 - 0.0299513 2026-02-23T15:52:11.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:52:10 trial068 ceph-mon[38805]: pgmap v5541: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:52:11.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:52:10 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1240522293' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:52:11.819 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9525 8 365281 365273 0.299578 0 - 0.0299513 2026-02-23T15:52:12.768 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:52:12.819 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9526 8 365281 365273 0.299546 0 - 0.0299513 2026-02-23T15:52:12.886 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:52:12.927 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:52:12 trial068 ceph-mon[38805]: pgmap v5542: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:52:12.984 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:52:12.982+0000 7f4b884a0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:52:12.984 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:52:12.982+0000 7f4b884a0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:52:13.093 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:52:13.094 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":345,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:51:50.270827+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:50:39.080345+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:48:07.758788+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:50:33.211242+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:52:13.819 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9527 8 365281 365273 0.299515 0 - 0.0299513 2026-02-23T15:52:13.973 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:52:13 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2082524462' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:52:14.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:52:13 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:52:13] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:52:14.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:52:13 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:52:13.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:52:14.819 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9528 8 365281 365273 0.299484 0 - 0.0299513 2026-02-23T15:52:15.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:52:14 trial068 ceph-mon[38805]: pgmap v5543: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:52:15.819 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9529 8 365281 365273 0.299452 0 - 0.0299513 2026-02-23T15:52:16.130 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:52:16.247 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:52:16.350 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:52:16.348+0000 7fcb4ce00640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:52:16.350 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:52:16.348+0000 7fcb4ce00640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:52:16.461 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:52:16.461 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":345,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:51:50.270827+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:50:39.080345+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:48:07.758788+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:50:33.211242+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:52:16.820 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9530 8 365281 365273 0.299421 0 - 0.0299513 2026-02-23T15:52:16.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:52:16 trial068 ceph-mon[38805]: pgmap v5544: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:52:16.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:52:16 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/752859736' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:52:17.820 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9531 8 365281 365273 0.299389 0 - 0.0299513 2026-02-23T15:52:18.820 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9532 8 365281 365273 0.299358 0 - 0.0299513 2026-02-23T15:52:19.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:52:18 trial068 ceph-mon[38805]: pgmap v5545: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:52:19.500 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:52:19.619 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:52:19.725 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:52:19.723+0000 7f7143fab640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:52:19.725 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:52:19.723+0000 7f7143fab640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:52:19.820 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9533 8 365281 365273 0.299326 0 - 0.0299513 2026-02-23T15:52:19.839 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:52:19.839 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":345,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:51:50.270827+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:50:39.080345+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:48:07.758788+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:50:33.211242+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:52:20.820 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9534 8 365281 365273 0.299295 0 - 0.0299513 2026-02-23T15:52:21.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:52:20 trial068 ceph-mon[38805]: pgmap v5546: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:52:21.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:52:20 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2463249868' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:52:21.820 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9535 8 365281 365273 0.299264 0 - 0.0299513 2026-02-23T15:52:22.820 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9536 8 365281 365273 0.299232 0 - 0.0299513 2026-02-23T15:52:22.878 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:52:22.995 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:52:23.017 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:52:22 trial068 ceph-mon[38805]: pgmap v5547: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:52:23.097 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:52:23.095+0000 7ffb7920a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:52:23.097 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:52:23.096+0000 7ffb7920a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:52:23.206 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:52:23.206 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":345,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:51:50.270827+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:50:39.080345+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:48:07.758788+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:50:33.211242+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:52:23.820 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9537 8 365281 365273 0.299201 0 - 0.0299513 2026-02-23T15:52:23.973 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:52:23 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3223224854' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:52:24.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:52:23 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:52:23] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:52:24.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:52:23 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:52:23.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:52:24.820 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9538 8 365281 365273 0.29917 0 - 0.0299513 2026-02-23T15:52:25.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:52:24 trial068 ceph-mon[38805]: pgmap v5548: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:52:25.820 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9539 8 365281 365273 0.299138 0 - 0.0299513 2026-02-23T15:52:26.245 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:52:26.363 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:52:26.461 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:52:26.459+0000 7f2d82481640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:52:26.461 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:52:26.460+0000 7f2d82481640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:52:26.572 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:52:26.572 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":345,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:51:50.270827+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:50:39.080345+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:48:07.758788+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:50:33.211242+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:52:26.572 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:52:26 trial068 ceph-mon[38805]: pgmap v5549: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:52:26.820 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T15:52:26.820193+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T15:52:26.821 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T15:52:26.821 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9540 8 365281 365273 0.299107 0 - 0.0299513 2026-02-23T15:52:27.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:52:27 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3180297793' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:52:27.820 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9541 8 365281 365273 0.299075 0 - 0.0299513 2026-02-23T15:52:28.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:52:28 trial068 ceph-mon[38805]: pgmap v5550: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:52:28.820 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9542 8 365281 365273 0.299044 0 - 0.0299513 2026-02-23T15:52:29.609 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:52:29.726 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:52:29.821 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9543 8 365281 365273 0.299013 0 - 0.0299513 2026-02-23T15:52:29.836 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:52:29.834+0000 7ff44d945640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:52:29.836 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:52:29.835+0000 7ff44d945640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:52:29.946 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:52:29.947 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":345,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:51:50.270827+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:50:39.080345+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:48:07.758788+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:50:33.211242+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:52:29.947 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:52:29 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2026-02-23T15:52:30.821 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9544 8 365281 365273 0.298981 0 - 0.0299513 2026-02-23T15:52:31.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:52:30 trial068 ceph-mon[38805]: pgmap v5551: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:52:31.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:52:30 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-23T15:52:31.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:52:30 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2026-02-23T15:52:31.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:52:30 trial068 ceph-mon[38805]: pgmap v5552: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:52:31.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:52:30 trial068 ceph-mon[38805]: pgmap v5553: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:52:31.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:52:30 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T15:52:31.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:52:30 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2026-02-23T15:52:31.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:52:30 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/199701350' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:52:31.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:52:30 trial068 ceph-mon[38805]: 5.8 scrub starts 2026-02-23T15:52:31.821 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9545 8 365281 365273 0.29895 0 - 0.0299513 2026-02-23T15:52:32.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:52:31 trial068 ceph-mon[38805]: 5.8 scrub ok 2026-02-23T15:52:32.821 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9546 8 365281 365273 0.298919 0 - 0.0299513 2026-02-23T15:52:33.006 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:52:33.031 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:52:32 trial068 ceph-mon[38805]: pgmap v5554: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:52:33.032 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:52:32 trial068 ceph-mon[38805]: 5.a scrub starts 2026-02-23T15:52:33.124 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:52:33.223 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:52:33.221+0000 7f03f4f77640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:52:33.223 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:52:33.221+0000 7f03f4f77640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:52:33.330 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:52:33.331 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":345,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:51:50.270827+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:50:39.080345+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:48:07.758788+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:50:33.211242+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:52:33.821 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9547 8 365281 365273 0.298888 0 - 0.0299513 2026-02-23T15:52:33.973 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:52:33 trial068 ceph-mon[38805]: 5.a scrub ok 2026-02-23T15:52:33.973 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:52:33 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/85607971' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:52:34.265 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:52:33 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:52:33.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:52:34.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:52:33 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:52:33] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:52:34.821 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9548 8 365281 365273 0.298856 0 - 0.0299513 2026-02-23T15:52:35.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:52:34 trial068 ceph-mon[38805]: pgmap v5555: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:52:35.821 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9549 8 365281 365273 0.298825 0 - 0.0299513 2026-02-23T15:52:36.368 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:52:36.484 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:52:36.586 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:52:36.583+0000 7ffad2425640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:52:36.586 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:52:36.583+0000 7ffad2425640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:52:36.695 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:52:36.695 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":345,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:51:50.270827+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:50:39.080345+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:48:07.758788+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:50:33.211242+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:52:36.821 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9550 8 365281 365273 0.298794 0 - 0.0299513 2026-02-23T15:52:37.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:52:36 trial068 ceph-mon[38805]: pgmap v5556: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:52:37.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:52:36 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3639544823' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:52:37.821 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9551 8 365281 365273 0.298762 0 - 0.0299513 2026-02-23T15:52:38.821 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9552 8 365281 365273 0.298731 0 - 0.0299513 2026-02-23T15:52:39.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:52:38 trial068 ceph-mon[38805]: pgmap v5557: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:52:39.731 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:52:39.821 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9553 8 365281 365273 0.2987 0 - 0.0299513 2026-02-23T15:52:39.847 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:52:39.955 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:52:39.953+0000 7f657e061640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:52:39.955 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:52:39.953+0000 7f657e061640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:52:40.063 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:52:40.064 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":345,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:51:50.270827+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:50:39.080345+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:48:07.758788+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:50:33.211242+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:52:40.821 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9554 8 365281 365273 0.298669 0 - 0.0299513 2026-02-23T15:52:41.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:52:40 trial068 ceph-mon[38805]: pgmap v5558: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:52:41.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:52:40 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1143799548' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:52:41.821 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9555 8 365281 365273 0.298637 0 - 0.0299513 2026-02-23T15:52:42.821 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9556 8 365281 365273 0.298606 0 - 0.0299513 2026-02-23T15:52:43.100 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:52:43.218 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:52:43.241 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:52:42 trial068 ceph-mon[38805]: pgmap v5559: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:52:43.323 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:52:43.321+0000 7f7904aaa640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:52:43.323 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:52:43.321+0000 7f7904aaa640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:52:43.433 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:52:43.433 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":345,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:51:50.270827+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:50:39.080345+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:48:07.758788+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:50:33.211242+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:52:43.821 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9557 8 365281 365273 0.298575 0 - 0.0299513 2026-02-23T15:52:44.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:52:43 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/858178891' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:52:44.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:52:43 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:52:43] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:52:44.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:52:43 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:52:43.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:52:44.821 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9558 8 365281 365273 0.298544 0 - 0.0299513 2026-02-23T15:52:45.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:52:44 trial068 ceph-mon[38805]: pgmap v5560: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:52:45.821 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9559 8 365281 365273 0.298512 0 - 0.0299513 2026-02-23T15:52:46.470 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:52:46.587 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:52:46.689 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:52:46.687+0000 7fc9864cc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:52:46.689 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:52:46.687+0000 7fc9864cc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:52:46.798 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:52:46.799 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":345,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:51:50.270827+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:50:39.080345+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:48:07.758788+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:50:33.211242+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:52:46.822 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T15:52:46.821714+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T15:52:46.822 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T15:52:46.822 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9560 8 365281 365273 0.298481 0 - 0.0299513 2026-02-23T15:52:47.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:52:46 trial068 ceph-mon[38805]: pgmap v5561: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:52:47.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:52:46 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/209307907' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:52:47.822 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9561 8 365281 365273 0.29845 0 - 0.0299513 2026-02-23T15:52:48.822 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9562 8 365281 365273 0.298419 0 - 0.0299513 2026-02-23T15:52:49.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:52:48 trial068 ceph-mon[38805]: pgmap v5562: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:52:49.822 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9563 8 365281 365273 0.298387 0 - 0.0299513 2026-02-23T15:52:49.836 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:52:49.953 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:52:50.055 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:52:50.053+0000 7f36ee92d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:52:50.055 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:52:50.054+0000 7f36ee92d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:52:50.163 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:52:50.163 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":345,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:51:50.270827+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:50:39.080345+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:48:07.758788+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:50:33.211242+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:52:50.822 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9564 8 365281 365273 0.298356 0 - 0.0299513 2026-02-23T15:52:51.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:52:50 trial068 ceph-mon[38805]: pgmap v5563: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:52:51.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:52:50 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1951766484' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:52:51.822 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9565 8 365281 365273 0.298325 0 - 0.0299513 2026-02-23T15:52:52.822 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9566 8 365281 365273 0.298294 0 - 0.0299513 2026-02-23T15:52:53.215 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:52:53.240 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:52:52 trial068 ceph-mon[38805]: pgmap v5564: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:52:53.332 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:52:53.435 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:52:53.433+0000 7fdc3c0c0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:52:53.435 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:52:53.434+0000 7fdc3c0c0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:52:53.548 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:52:53.548 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":345,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:51:50.270827+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:50:39.080345+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:48:07.758788+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:50:33.211242+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:52:53.822 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9567 8 365281 365273 0.298263 0 - 0.0299513 2026-02-23T15:52:54.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:52:53 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:52:53] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:52:54.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:52:53 trial068 ceph-mon[38805]: 5.7 scrub starts 2026-02-23T15:52:54.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:52:53 trial068 ceph-mon[38805]: 5.7 scrub ok 2026-02-23T15:52:54.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:52:53 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2137828749' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:52:54.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:52:53 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:52:53.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:52:54.822 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9568 8 365281 365273 0.298232 0 - 0.0299513 2026-02-23T15:52:55.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:52:54 trial068 ceph-mon[38805]: pgmap v5565: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:52:55.822 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9569 8 365281 365273 0.2982 0 - 0.0299513 2026-02-23T15:52:56.586 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:52:56.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:52:56 trial068 ceph-mon[38805]: pgmap v5566: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:52:56.706 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:52:56.807 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:52:56.805+0000 7f66a4137640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:52:56.807 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:52:56.805+0000 7f66a4137640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:52:56.822 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9570 8 365281 365273 0.298169 0 - 0.0299513 2026-02-23T15:52:56.915 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:52:56.916 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":345,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:51:50.270827+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:50:39.080345+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:48:07.758788+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:50:33.211242+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:52:57.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:52:57 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/4214071271' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:52:57.822 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9571 8 365281 365273 0.298138 0 - 0.0299513 2026-02-23T15:52:58.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:52:58 trial068 ceph-mon[38805]: pgmap v5567: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:52:58.823 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9572 8 365281 365273 0.298107 0 - 0.0299513 2026-02-23T15:52:59.823 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9573 8 365281 365273 0.298076 0 - 0.0299513 2026-02-23T15:52:59.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:52:59 trial068 ceph-mon[38805]: 5.e scrub starts 2026-02-23T15:52:59.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:52:59 trial068 ceph-mon[38805]: 5.e scrub ok 2026-02-23T15:52:59.952 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:53:00.072 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:53:00.170 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:53:00.169+0000 7f577ff19640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:53:00.170 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:53:00.169+0000 7f577ff19640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:53:00.281 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:53:00.281 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":345,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:51:50.270827+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:50:39.080345+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:48:07.758788+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:50:33.211242+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:53:00.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:53:00 trial068 ceph-mon[38805]: pgmap v5568: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:53:00.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:53:00 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2630252870' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:53:00.823 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9574 8 365281 365273 0.298045 0 - 0.0299513 2026-02-23T15:53:01.823 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9575 8 365281 365273 0.298014 0 - 0.0299513 2026-02-23T15:53:02.823 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9576 8 365281 365273 0.297982 0 - 0.0299513 2026-02-23T15:53:03.319 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:53:03.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:53:02 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T15:53:03.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:53:02 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T15:53:03.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:53:02 trial068 ceph-mon[38805]: pgmap v5569: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:53:03.438 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:53:03.544 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:53:03.542+0000 7f010b748640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:53:03.544 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:53:03.542+0000 7f010b748640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:53:03.656 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:53:03.656 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":345,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:51:50.270827+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:50:39.080345+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:48:07.758788+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:50:33.211242+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:53:03.823 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9577 8 365281 365273 0.297951 0 - 0.0299513 2026-02-23T15:53:03.973 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:53:03 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/4040719626' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:53:04.265 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:53:03 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:53:03.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:53:04.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:53:03 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:53:03] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:53:04.823 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9578 8 365281 365273 0.29792 0 - 0.0299513 2026-02-23T15:53:05.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:53:04 trial068 ceph-mon[38805]: pgmap v5570: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:53:05.823 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9579 8 365281 365273 0.297889 0 - 0.0299513 2026-02-23T15:53:06.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:53:06 trial068 ceph-mon[38805]: pgmap v5571: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:53:06.694 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:53:06.814 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:53:06.823 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T15:53:06.823162+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T15:53:06.823 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T15:53:06.823 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9580 8 365281 365273 0.297858 0 - 0.0299513 2026-02-23T15:53:06.918 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:53:06.916+0000 7f9c3dca9640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:53:06.918 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:53:06.917+0000 7f9c3dca9640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:53:07.031 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:53:07.031 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":345,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:51:50.270827+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:50:39.080345+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:48:07.758788+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:50:33.211242+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:53:07.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:53:07 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1971023029' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:53:07.823 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9581 8 365281 365273 0.297827 0 - 0.0299513 2026-02-23T15:53:08.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:53:08 trial068 ceph-mon[38805]: pgmap v5572: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:53:08.823 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9582 8 365281 365273 0.297796 0 - 0.0299513 2026-02-23T15:53:09.823 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9583 8 365281 365273 0.297765 0 - 0.0299513 2026-02-23T15:53:10.071 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:53:10.190 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:53:10.295 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:53:10.294+0000 7f25c001f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:53:10.296 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:53:10.294+0000 7f25c001f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:53:10.405 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:53:10.405 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":345,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:51:50.270827+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:50:39.080345+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:48:07.758788+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:50:33.211242+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:53:10.824 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9584 8 365281 365273 0.297734 0 - 0.0299513 2026-02-23T15:53:11.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:53:10 trial068 ceph-mon[38805]: pgmap v5573: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:53:11.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:53:10 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/173618504' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:53:11.824 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9585 8 365281 365273 0.297703 0 - 0.0299513 2026-02-23T15:53:12.824 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9586 8 365281 365273 0.297672 0 - 0.0299513 2026-02-23T15:53:13.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:53:12 trial068 ceph-mon[38805]: pgmap v5574: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:53:13.443 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:53:13.561 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:53:13.672 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:53:13.670+0000 7f83d4c2e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:53:13.672 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:53:13.670+0000 7f83d4c2e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:53:13.783 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:53:13.783 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":345,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:51:50.270827+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:50:39.080345+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:48:07.758788+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:50:33.211242+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:53:13.824 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9587 8 365281 365273 0.29764 0 - 0.0299513 2026-02-23T15:53:14.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:53:13 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/225196156' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:53:14.094 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:53:13 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:53:13] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:53:14.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:53:13 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:53:13.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:53:14.824 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9588 8 365281 365273 0.297609 0 - 0.0299513 2026-02-23T15:53:15.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:53:14 trial068 ceph-mon[38805]: pgmap v5575: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:53:15.824 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9589 8 365281 365273 0.297578 0 - 0.0299513 2026-02-23T15:53:16.822 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:53:16.826 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9590 8 365281 365273 0.297547 0 - 0.0299513 2026-02-23T15:53:16.941 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:53:17.041 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:53:17.039+0000 7f87f9a29640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:53:17.042 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:53:17.040+0000 7f87f9a29640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:53:17.153 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:53:17.154 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":345,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:51:50.270827+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:50:39.080345+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:48:07.758788+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:50:33.211242+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:53:17.154 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:53:16 trial068 ceph-mon[38805]: pgmap v5576: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:53:17.154 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:53:16 trial068 ceph-mon[38805]: 5.5 scrub starts 2026-02-23T15:53:17.824 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9591 8 365281 365273 0.297516 0 - 0.0299513 2026-02-23T15:53:18.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:53:17 trial068 ceph-mon[38805]: 5.5 scrub ok 2026-02-23T15:53:18.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:53:17 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3145718412' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:53:18.824 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9592 8 365281 365273 0.297485 0 - 0.0299513 2026-02-23T15:53:19.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:53:18 trial068 ceph-mon[38805]: pgmap v5577: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:53:19.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:53:18 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T15:53:19.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:53:18 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T15:53:19.824 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9593 8 365281 365273 0.297454 0 - 0.0299513 2026-02-23T15:53:20.190 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:53:20.310 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:53:20.415 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:53:20.413+0000 7fd15cfb1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:53:20.415 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:53:20.413+0000 7fd15cfb1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:53:20.524 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:53:20.525 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":345,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:51:50.270827+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:50:39.080345+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:48:07.758788+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:50:33.211242+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:53:20.825 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9594 8 365281 365273 0.297423 0 - 0.0299513 2026-02-23T15:53:21.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:53:20 trial068 ceph-mon[38805]: pgmap v5578: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:53:21.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:53:20 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/833210047' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:53:21.825 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9595 8 365281 365273 0.297392 0 - 0.0299513 2026-02-23T15:53:22.825 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9596 8 365281 365273 0.297361 0 - 0.0299513 2026-02-23T15:53:23.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:53:22 trial068 ceph-mon[38805]: pgmap v5579: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:53:23.564 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:53:23.682 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:53:23.780 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:53:23.779+0000 7f31177c8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:53:23.781 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:53:23.779+0000 7f31177c8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:53:23.825 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9597 8 365281 365273 0.29733 0 - 0.0299513 2026-02-23T15:53:23.892 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:53:23.892 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":345,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:51:50.270827+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:50:39.080345+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:48:07.758788+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:50:33.211242+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:53:24.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:53:23 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2680675771' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:53:24.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:53:23 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:53:23] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:53:24.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:53:23 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:53:23.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:53:24.825 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9598 8 365281 365273 0.297299 0 - 0.0299513 2026-02-23T15:53:25.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:53:24 trial068 ceph-mon[38805]: pgmap v5580: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:53:25.825 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9599 8 365281 365273 0.297268 0 - 0.0299513 2026-02-23T15:53:26.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:53:26 trial068 ceph-mon[38805]: pgmap v5581: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:53:26.825 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T15:53:26.825001+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T15:53:26.825 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T15:53:26.825 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9600 8 365281 365273 0.297237 0 - 0.0299513 2026-02-23T15:53:26.932 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:53:27.054 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:53:27.154 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:53:27.152+0000 7f3f29b93640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:53:27.154 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:53:27.152+0000 7f3f29b93640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:53:27.264 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:53:27.264 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":345,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:51:50.270827+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:50:39.080345+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:48:07.758788+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:50:33.211242+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:53:27.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:53:27 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1655760270' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:53:27.825 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9601 8 365281 365273 0.297206 0 - 0.0299513 2026-02-23T15:53:28.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:53:28 trial068 ceph-mon[38805]: pgmap v5582: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:53:28.825 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9602 8 365281 365273 0.297176 0 - 0.0299513 2026-02-23T15:53:29.825 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9603 8 365281 365273 0.297145 0 - 0.0299513 2026-02-23T15:53:30.303 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:53:30.325 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:53:29 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2026-02-23T15:53:30.418 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:53:30.519 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:53:30.517+0000 7fdcdf9a0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:53:30.519 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:53:30.517+0000 7fdcdf9a0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:53:30.629 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:53:30.629 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":345,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:51:50.270827+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:50:39.080345+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:48:07.758788+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:50:33.211242+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:53:30.825 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9604 8 365281 365273 0.297114 0 - 0.0299513 2026-02-23T15:53:31.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:53:30 trial068 ceph-mon[38805]: pgmap v5583: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:53:31.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:53:30 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-23T15:53:31.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:53:30 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2026-02-23T15:53:31.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:53:30 trial068 ceph-mon[38805]: pgmap v5584: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:53:31.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:53:30 trial068 ceph-mon[38805]: pgmap v5585: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:53:31.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:53:30 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T15:53:31.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:53:30 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2026-02-23T15:53:31.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:53:30 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/551457072' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:53:31.825 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9605 8 365281 365273 0.297083 0 - 0.0299513 2026-02-23T15:53:32.826 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9606 8 365281 365273 0.297052 0 - 0.0299513 2026-02-23T15:53:33.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:53:33 trial068 ceph-mon[38805]: pgmap v5586: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:53:33.666 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:53:33.784 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:53:33.825 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9607 8 365281 365273 0.297021 0 - 0.0299513 2026-02-23T15:53:33.891 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:53:33.889+0000 7f338ed9b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:53:33.891 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:53:33.889+0000 7f338ed9b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:53:34.001 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:53:34.002 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":345,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:51:50.270827+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:50:39.080345+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:48:07.758788+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:50:33.211242+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:53:34.265 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:53:33 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:53:33.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:53:34.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:53:34 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T15:53:34.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:53:34 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T15:53:34.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:53:34 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3432409573' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:53:34.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:53:33 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:53:33] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:53:34.826 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9608 8 365281 365273 0.29699 0 - 0.0299513 2026-02-23T15:53:35.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:53:35 trial068 ceph-mon[38805]: pgmap v5587: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:53:35.826 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9609 8 365281 365273 0.296959 0 - 0.0299513 2026-02-23T15:53:36.826 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9610 8 365281 365273 0.296928 0 - 0.0299513 2026-02-23T15:53:37.042 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:53:37.161 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:53:37.260 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:53:37.258+0000 7fe7a7e91640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:53:37.260 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:53:37.259+0000 7fe7a7e91640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:53:37.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:53:37 trial068 ceph-mon[38805]: pgmap v5588: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:53:37.370 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:53:37.370 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":345,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:51:50.270827+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:50:39.080345+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:48:07.758788+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:50:33.211242+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:53:37.826 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9611 8 365281 365273 0.296897 0 - 0.0299513 2026-02-23T15:53:38.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:53:38 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/687238309' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:53:38.826 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9612 8 365281 365273 0.296866 0 - 0.0299513 2026-02-23T15:53:39.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:53:39 trial068 ceph-mon[38805]: pgmap v5589: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:53:39.826 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9613 8 365281 365273 0.296835 0 - 0.0299513 2026-02-23T15:53:40.424 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:53:40.541 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:53:40.640 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:53:40.639+0000 7f6ed241a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:53:40.641 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:53:40.639+0000 7f6ed241a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:53:40.749 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:53:40.749 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":345,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:51:50.270827+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:50:39.080345+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:48:07.758788+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:50:33.211242+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:53:40.826 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9614 8 365281 365273 0.296805 0 - 0.0299513 2026-02-23T15:53:41.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:53:41 trial068 ceph-mon[38805]: pgmap v5590: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:53:41.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:53:41 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1413468511' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:53:41.827 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9615 8 365281 365273 0.296774 0 - 0.0299513 2026-02-23T15:53:42.827 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9616 8 365281 365273 0.296743 0 - 0.0299513 2026-02-23T15:53:43.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:53:43 trial068 ceph-mon[38805]: pgmap v5591: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:53:43.792 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:53:43.827 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9617 8 365281 365273 0.296712 0 - 0.0299513 2026-02-23T15:53:43.909 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:53:44.007 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:53:44.005+0000 7f395c204640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:53:44.007 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:53:44.006+0000 7f395c204640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:53:44.093 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:53:43 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:53:43] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:53:44.116 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:53:44.116 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":345,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:51:50.270827+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:50:39.080345+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:48:07.758788+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:50:33.211242+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:53:44.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:53:43 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:53:43.972Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:53:44.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:53:44 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/5142765' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:53:44.827 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9618 8 365281 365273 0.296681 0 - 0.0299513 2026-02-23T15:53:45.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:53:45 trial068 ceph-mon[38805]: pgmap v5592: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:53:45.827 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9619 8 365281 365273 0.29665 0 - 0.0299513 2026-02-23T15:53:46.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:53:46 trial068 ceph-mon[38805]: 5.8 scrub starts 2026-02-23T15:53:46.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:53:46 trial068 ceph-mon[38805]: 5.8 scrub ok 2026-02-23T15:53:46.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:53:46 trial068 ceph-mon[38805]: pgmap v5593: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:53:46.827 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T15:53:46.826951+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T15:53:46.827 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T15:53:46.827 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9620 8 365281 365273 0.296619 0 - 0.0299513 2026-02-23T15:53:47.157 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:53:47.277 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:53:47.378 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:53:47.376+0000 7f071b3ca640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:53:47.378 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:53:47.376+0000 7f071b3ca640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:53:47.485 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:53:47.485 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":345,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:51:50.270827+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:50:39.080345+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:48:07.758788+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:50:33.211242+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:53:47.827 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9621 8 365281 365273 0.296589 0 - 0.0299513 2026-02-23T15:53:47.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:53:47 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2994650622' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:53:48.827 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9622 8 365281 365273 0.296558 0 - 0.0299513 2026-02-23T15:53:48.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:53:48 trial068 ceph-mon[38805]: pgmap v5594: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:53:49.827 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9623 8 365281 365273 0.296527 0 - 0.0299513 2026-02-23T15:53:50.539 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:53:50.659 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:53:50.760 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:53:50.758+0000 7f1b1e67d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:53:50.760 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:53:50.759+0000 7f1b1e67d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:53:50.827 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9624 8 365281 365273 0.296496 0 - 0.0299513 2026-02-23T15:53:50.872 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:53:50.872 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":345,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:51:50.270827+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:50:39.080345+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:48:07.758788+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:50:33.211242+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:53:51.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:53:51 trial068 ceph-mon[38805]: pgmap v5595: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:53:51.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:53:51 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2536395970' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:53:51.827 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9625 8 365281 365273 0.296465 0 - 0.0299513 2026-02-23T15:53:52.828 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9626 8 365281 365273 0.296435 0 - 0.0299513 2026-02-23T15:53:53.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:53:53 trial068 ceph-mon[38805]: pgmap v5596: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:53:53.828 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9627 8 365281 365273 0.296404 0 - 0.0299513 2026-02-23T15:53:53.909 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:53:54.025 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:53:54.131 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:53:54.129+0000 7fa228989640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:53:54.131 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:53:54.129+0000 7fa228989640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:53:54.240 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:53:54.241 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":345,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:51:50.270827+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:50:39.080345+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:48:07.758788+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:50:33.211242+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:53:54.241 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:53:53 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:53:53] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:53:54.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:53:53 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:53:53.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:53:54.828 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9628 8 365281 365273 0.296373 0 - 0.0299513 2026-02-23T15:53:55.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:53:55 trial068 ceph-mon[38805]: pgmap v5597: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:53:55.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:53:55 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1321795230' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:53:55.828 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9629 8 365281 365273 0.296342 0 - 0.0299513 2026-02-23T15:53:56.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:53:56 trial068 ceph-mon[38805]: 5.a scrub starts 2026-02-23T15:53:56.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:53:56 trial068 ceph-mon[38805]: 5.a scrub ok 2026-02-23T15:53:56.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:53:56 trial068 ceph-mon[38805]: pgmap v5598: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:53:56.828 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9630 8 365281 365273 0.296311 0 - 0.0299513 2026-02-23T15:53:57.278 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:53:57.397 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:53:57.494 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:53:57.492+0000 7f9ae56e3640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:53:57.494 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:53:57.492+0000 7f9ae56e3640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:53:57.602 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:53:57.602 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":345,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:51:50.270827+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:50:39.080345+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:48:07.758788+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:50:33.211242+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:53:57.828 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9631 8 365281 365273 0.296281 0 - 0.0299513 2026-02-23T15:53:58.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:53:57 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3142573644' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:53:58.828 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9632 8 365281 365273 0.29625 0 - 0.0299513 2026-02-23T15:53:59.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:53:58 trial068 ceph-mon[38805]: pgmap v5599: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:53:59.828 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9633 8 365281 365273 0.296219 0 - 0.0299513 2026-02-23T15:54:00.640 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:54:00.757 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:54:00.829 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9634 8 365281 365273 0.296188 0 - 0.0299513 2026-02-23T15:54:00.857 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:54:00.855+0000 7f5e993e9640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:54:00.857 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:54:00.855+0000 7f5e993e9640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:54:00.971 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:54:00.971 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":345,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:51:50.270827+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:50:39.080345+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:48:07.758788+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:50:33.211242+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:54:01.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:54:01 trial068 ceph-mon[38805]: pgmap v5600: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:54:01.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:54:01 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3479384715' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:54:01.829 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9635 8 365281 365273 0.296158 0 - 0.0299513 2026-02-23T15:54:02.829 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9636 8 365281 365273 0.296127 0 - 0.0299513 2026-02-23T15:54:03.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:54:03 trial068 ceph-mon[38805]: pgmap v5601: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:54:03.829 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9637 8 365281 365273 0.296096 0 - 0.0299513 2026-02-23T15:54:04.009 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:54:04.126 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:54:04.226 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:54:04.224+0000 7fa1c7023640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:54:04.226 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:54:04.224+0000 7fa1c7023640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:54:04.227 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:54:03 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:54:03] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:54:04.265 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:54:03 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:54:03.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:54:04.336 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:54:04.336 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":345,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:51:50.270827+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:50:39.080345+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:48:07.758788+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:50:33.211242+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:54:04.829 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9638 8 365281 365273 0.296065 0 - 0.0299513 2026-02-23T15:54:05.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:54:05 trial068 ceph-mon[38805]: pgmap v5602: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:54:05.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:54:05 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/129248366' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:54:05.829 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9639 8 365281 365273 0.296035 0 - 0.0299513 2026-02-23T15:54:06.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:54:06 trial068 ceph-mon[38805]: pgmap v5603: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:54:06.829 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T15:54:06.828994+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T15:54:06.829 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T15:54:06.829 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9640 8 365281 365273 0.296004 0 - 0.0299513 2026-02-23T15:54:07.389 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:54:07.507 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:54:07.611 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:54:07.609+0000 7f656804f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:54:07.611 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:54:07.609+0000 7f656804f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:54:07.724 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:54:07.724 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":345,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:51:50.270827+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:50:39.080345+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:48:07.758788+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:50:33.211242+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:54:07.829 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9641 8 365281 365273 0.295973 0 - 0.0299513 2026-02-23T15:54:08.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:54:07 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/43877315' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:54:08.830 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9642 8 365281 365273 0.295943 0 - 0.0299513 2026-02-23T15:54:09.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:54:08 trial068 ceph-mon[38805]: pgmap v5604: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:54:09.830 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9643 8 365281 365273 0.295912 0 - 0.0299513 2026-02-23T15:54:10.760 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:54:10.830 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9644 8 365281 365273 0.295881 0 - 0.0299513 2026-02-23T15:54:10.876 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:54:10.978 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:54:10.976+0000 7f40f1868640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:54:10.978 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:54:10.976+0000 7f40f1868640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:54:11.087 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:54:11.087 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":345,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:51:50.270827+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:50:39.080345+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:48:07.758788+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:50:33.211242+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:54:11.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:54:11 trial068 ceph-mon[38805]: pgmap v5605: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:54:11.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:54:11 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3540156102' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:54:11.830 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9645 8 365281 365273 0.295851 0 - 0.0299513 2026-02-23T15:54:12.830 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9646 8 365281 365273 0.29582 0 - 0.0299513 2026-02-23T15:54:13.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:54:13 trial068 ceph-mon[38805]: pgmap v5606: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:54:13.830 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9647 8 365281 365273 0.295789 0 - 0.0299513 2026-02-23T15:54:14.125 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:54:14.245 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:54:14.268 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:54:13 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:54:13] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:54:14.344 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:54:14.342+0000 7fb15e5d7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:54:14.344 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:54:14.342+0000 7fb15e5d7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:54:14.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:54:13 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:54:13.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:54:14.453 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:54:14.454 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":345,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:51:50.270827+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:50:39.080345+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:48:07.758788+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:50:33.211242+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:54:14.830 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9648 8 365281 365273 0.295759 0 - 0.0299513 2026-02-23T15:54:15.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:54:15 trial068 ceph-mon[38805]: pgmap v5607: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:54:15.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:54:15 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/922600867' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:54:15.830 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9649 8 365281 365273 0.295728 0 - 0.0299513 2026-02-23T15:54:16.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:54:16 trial068 ceph-mon[38805]: pgmap v5608: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:54:16.830 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9650 8 365281 365273 0.295697 0 - 0.0299513 2026-02-23T15:54:17.491 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:54:17.610 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:54:17.713 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:54:17.712+0000 7fd1e856d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:54:17.714 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:54:17.712+0000 7fd1e856d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:54:17.825 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:54:17.825 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":345,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:51:50.270827+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:50:39.080345+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:48:07.758788+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:50:33.211242+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:54:17.830 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9651 8 365281 365273 0.295667 0 - 0.0299513 2026-02-23T15:54:18.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:54:17 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2903863159' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:54:18.830 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9652 8 365281 365273 0.295636 0 - 0.0299513 2026-02-23T15:54:19.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:54:18 trial068 ceph-mon[38805]: 5.7 scrub starts 2026-02-23T15:54:19.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:54:18 trial068 ceph-mon[38805]: 5.7 scrub ok 2026-02-23T15:54:19.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:54:18 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T15:54:19.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:54:18 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T15:54:19.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:54:18 trial068 ceph-mon[38805]: pgmap v5609: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:54:19.830 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9653 8 365281 365273 0.295605 0 - 0.0299513 2026-02-23T15:54:20.830 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9654 8 365281 365273 0.295575 0 - 0.0299513 2026-02-23T15:54:20.864 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:54:20.984 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:54:21.086 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:54:21.084+0000 7ff3ab2ce640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:54:21.086 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:54:21.084+0000 7ff3ab2ce640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:54:21.197 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:54:21.197 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":345,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:51:50.270827+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:50:39.080345+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:48:07.758788+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:50:33.211242+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:54:21.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:54:21 trial068 ceph-mon[38805]: pgmap v5610: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:54:21.830 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9655 8 365281 365273 0.295544 0 - 0.0299513 2026-02-23T15:54:22.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:54:22 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3554683608' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:54:22.831 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9656 8 365281 365273 0.295514 0 - 0.0299513 2026-02-23T15:54:23.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:54:23 trial068 ceph-mon[38805]: pgmap v5611: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:54:23.831 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9657 8 365281 365273 0.295483 0 - 0.0299513 2026-02-23T15:54:24.235 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:54:24.259 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:54:23 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:54:23] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:54:24.353 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:54:24.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:54:23 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:54:23.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:54:24.457 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:54:24.455+0000 7f15aeee0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:54:24.457 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:54:24.455+0000 7f15aeee0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:54:24.565 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:54:24.565 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":345,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:51:50.270827+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:50:39.080345+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:48:07.758788+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:50:33.211242+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:54:24.831 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9658 8 365281 365273 0.295452 0 - 0.0299513 2026-02-23T15:54:25.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:54:25 trial068 ceph-mon[38805]: pgmap v5612: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:54:25.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:54:25 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/515918806' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:54:25.831 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9659 8 365281 365273 0.295422 0 - 0.0299513 2026-02-23T15:54:26.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:54:26 trial068 ceph-mon[38805]: 5.e scrub starts 2026-02-23T15:54:26.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:54:26 trial068 ceph-mon[38805]: 5.e scrub ok 2026-02-23T15:54:26.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:54:26 trial068 ceph-mon[38805]: pgmap v5613: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:54:26.831 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T15:54:26.830899+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T15:54:26.831 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T15:54:26.831 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9660 8 365281 365273 0.295391 0 - 0.0299513 2026-02-23T15:54:27.603 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:54:27.722 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:54:27.827 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:54:27.825+0000 7fc569196640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:54:27.827 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:54:27.825+0000 7fc569196640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:54:27.831 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9661 8 365281 365273 0.295361 0 - 0.0299513 2026-02-23T15:54:27.936 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:54:27.936 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":346,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:54:27.291750+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:50:39.080345+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:53:18.149281+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:50:33.211242+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:54:28.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:54:28 trial068 ceph-mon[38805]: osdmap e346: 4 total, 3 up, 4 in 2026-02-23T15:54:28.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:54:28 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1835640536' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:54:28.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:54:28 trial068 ceph-mon[38805]: pgmap v5615: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:54:28.831 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9662 8 365281 365273 0.29533 0 - 0.0299513 2026-02-23T15:54:29.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:54:29 trial068 ceph-mon[38805]: 5.5 scrub starts 2026-02-23T15:54:29.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:54:29 trial068 ceph-mon[38805]: 5.5 scrub ok 2026-02-23T15:54:29.831 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9663 8 365281 365273 0.295299 0 - 0.0299513 2026-02-23T15:54:30.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:54:30 trial068 ceph-mon[38805]: pgmap v5616: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:54:30.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:54:30 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2026-02-23T15:54:30.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:54:30 trial068 ceph-mon[38805]: osdmap e347: 4 total, 3 up, 4 in 2026-02-23T15:54:30.831 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9664 8 365281 365273 0.295269 0 - 0.0299513 2026-02-23T15:54:30.976 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:54:31.093 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:54:31.193 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:54:31.191+0000 7feec8e15640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:54:31.193 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:54:31.191+0000 7feec8e15640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:54:31.303 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:54:31.303 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":347,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:54:30.290030+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:50:39.080345+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:53:18.149281+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:50:33.211242+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:54:31.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:54:31 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-23T15:54:31.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:54:31 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2026-02-23T15:54:31.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:54:31 trial068 ceph-mon[38805]: pgmap v5618: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:54:31.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:54:31 trial068 ceph-mon[38805]: pgmap v5619: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:54:31.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:54:31 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T15:54:31.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:54:31 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2026-02-23T15:54:31.831 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9665 8 365281 365273 0.295238 0 - 0.0299513 2026-02-23T15:54:32.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:54:32 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3395594490' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:54:32.831 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9666 8 365281 365273 0.295208 0 - 0.0299513 2026-02-23T15:54:33.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:54:33 trial068 ceph-mon[38805]: pgmap v5620: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:54:33.831 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9667 8 365281 365273 0.295177 0 - 0.0299513 2026-02-23T15:54:34.265 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:54:33 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:54:33.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:54:34.340 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:54:34.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:54:33 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:54:33] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:54:34.460 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:54:34.558 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:54:34.556+0000 7faff1a1a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:54:34.559 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:54:34.557+0000 7faff1a1a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:54:34.666 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:54:34.667 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":347,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:54:30.290030+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:50:39.080345+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:53:18.149281+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:50:33.211242+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:54:34.832 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9668 8 365281 365273 0.295147 0 - 0.0299513 2026-02-23T15:54:35.832 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9669 8 365281 365273 0.295116 0 - 0.0299513 2026-02-23T15:54:35.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:54:35 trial068 ceph-mon[38805]: pgmap v5621: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:54:35.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:54:35 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3945602920' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:54:36.832 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9670 8 365281 365273 0.295086 0 - 0.0299513 2026-02-23T15:54:37.703 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:54:37.727 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:54:37 trial068 ceph-mon[38805]: pgmap v5622: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:54:37.819 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:54:37.832 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9671 8 365281 365273 0.295055 0 - 0.0299513 2026-02-23T15:54:37.923 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:54:37.922+0000 7f360afc1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:54:37.924 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:54:37.922+0000 7f360afc1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:54:38.032 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:54:38.032 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":347,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:54:30.290030+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:50:39.080345+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:53:18.149281+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:50:33.211242+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:54:38.832 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9672 8 365281 365273 0.295025 0 - 0.0299513 2026-02-23T15:54:38.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:54:38 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/4241522305' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:54:39.832 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9673 8 365281 365273 0.294994 0 - 0.0299513 2026-02-23T15:54:39.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:54:39 trial068 ceph-mon[38805]: pgmap v5623: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:54:40.832 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9674 8 365281 365273 0.294964 0 - 0.0299513 2026-02-23T15:54:41.070 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:54:41.186 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:54:41.286 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:54:41.284+0000 7fba23641640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:54:41.286 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:54:41.284+0000 7fba23641640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:54:41.395 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:54:41.395 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":347,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:54:30.290030+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:50:39.080345+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:53:18.149281+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:50:33.211242+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:54:41.832 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9675 8 365281 365273 0.294933 0 - 0.0299513 2026-02-23T15:54:41.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:54:41 trial068 ceph-mon[38805]: pgmap v5624: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:54:41.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:54:41 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2181309393' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:54:42.832 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9676 8 365281 365273 0.294903 0 - 0.0299513 2026-02-23T15:54:42.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:54:42 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T15:54:42.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:54:42 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T15:54:43.832 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9677 8 365281 365273 0.294872 0 - 0.0299513 2026-02-23T15:54:43.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:54:43 trial068 ceph-mon[38805]: pgmap v5625: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:54:44.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:54:43 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:54:43] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:54:44.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:54:43 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:54:43.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:54:44.432 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:54:44.553 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:54:44.661 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:54:44.659+0000 7f618620e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:54:44.661 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:54:44.659+0000 7f618620e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:54:44.773 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:54:44.773 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":347,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:54:30.290030+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:50:39.080345+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:53:18.149281+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:50:33.211242+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:54:44.832 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9678 8 365281 365273 0.294842 0 - 0.0299513 2026-02-23T15:54:45.832 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9679 8 365281 365273 0.294811 0 - 0.0299513 2026-02-23T15:54:45.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:54:45 trial068 ceph-mon[38805]: pgmap v5626: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:54:45.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:54:45 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3536252391' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:54:46.832 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T15:54:46.832313+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T15:54:46.833 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T15:54:46.833 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9680 8 365281 365273 0.294781 0 - 0.0299513 2026-02-23T15:54:47.812 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:54:47.832 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9681 8 365281 365273 0.29475 0 - 0.0299513 2026-02-23T15:54:47.836 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:54:47 trial068 ceph-mon[38805]: pgmap v5627: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:54:47.931 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:54:48.035 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:54:48.033+0000 7f07de8d2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:54:48.035 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:54:48.033+0000 7f07de8d2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:54:48.146 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:54:48.146 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":347,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:54:30.290030+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:50:39.080345+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:53:18.149281+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:50:33.211242+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:54:48.833 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9682 8 365281 365273 0.29472 0 - 0.0299513 2026-02-23T15:54:48.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:54:48 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3473288877' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:54:49.833 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9683 8 365281 365273 0.29469 0 - 0.0299513 2026-02-23T15:54:49.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:54:49 trial068 ceph-mon[38805]: pgmap v5628: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:54:50.833 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9684 8 365281 365273 0.294659 0 - 0.0299513 2026-02-23T15:54:51.184 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:54:51.300 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:54:51.398 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:54:51.396+0000 7f2cf801e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:54:51.398 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:54:51.396+0000 7f2cf801e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:54:51.507 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:54:51 trial068 ceph-mon[38805]: pgmap v5629: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:54:51.507 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:54:51.508 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":347,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:54:30.290030+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:50:39.080345+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:53:18.149281+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:50:33.211242+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:54:51.833 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9685 8 365281 365273 0.294629 0 - 0.0299513 2026-02-23T15:54:52.833 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9686 8 365281 365273 0.294598 0 - 0.0299513 2026-02-23T15:54:52.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:54:52 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2384604970' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:54:53.833 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9687 8 365281 365273 0.294568 0 - 0.0299513 2026-02-23T15:54:53.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:54:53 trial068 ceph-mon[38805]: pgmap v5630: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:54:54.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:54:53 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:54:53] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:54:54.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:54:53 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:54:53.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:54:54.546 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:54:54.664 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:54:54.763 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:54:54.761+0000 7fcae774f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:54:54.763 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:54:54.761+0000 7fcae774f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:54:54.833 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9688 8 365281 365273 0.294537 0 - 0.0299513 2026-02-23T15:54:54.873 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:54:54.873 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":347,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:54:30.290030+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:50:39.080345+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:53:18.149281+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:50:33.211242+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:54:55.833 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9689 8 365281 365273 0.294507 0 - 0.0299513 2026-02-23T15:54:55.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:54:55 trial068 ceph-mon[38805]: pgmap v5631: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:54:55.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:54:55 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1050704789' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:54:55.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:54:55 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T15:54:55.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:54:55 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T15:54:56.833 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9690 8 365281 365273 0.294477 0 - 0.0299513 2026-02-23T15:54:57.833 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9691 8 365281 365273 0.294446 0 - 0.0299513 2026-02-23T15:54:57.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:54:57 trial068 ceph-mon[38805]: pgmap v5632: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:54:57.911 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:54:58.028 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:54:58.124 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:54:58.122+0000 7fc761ea6640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:54:58.124 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:54:58.122+0000 7fc761ea6640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:54:58.234 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:54:58.234 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":347,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:54:30.290030+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:50:39.080345+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:53:18.149281+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:50:33.211242+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:54:58.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:54:58 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1194113517' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:54:58.833 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9692 8 365281 365273 0.294416 0 - 0.0299513 2026-02-23T15:54:59.833 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9693 8 365281 365273 0.294386 0 - 0.0299513 2026-02-23T15:54:59.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:54:59 trial068 ceph-mon[38805]: pgmap v5633: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:55:00.833 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9694 8 365281 365273 0.294355 0 - 0.0299513 2026-02-23T15:55:01.272 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:55:01.390 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:55:01.487 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:55:01.485+0000 7fa89176a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:55:01.487 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:55:01.485+0000 7fa89176a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:55:01.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:55:01 trial068 ceph-mon[38805]: pgmap v5634: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:55:01.596 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:55:01.596 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":347,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:54:30.290030+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:50:39.080345+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:53:18.149281+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:50:33.211242+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:55:01.834 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9695 8 365281 365273 0.294325 0 - 0.0299513 2026-02-23T15:55:02.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:55:02 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3753494491' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:55:02.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:55:02 trial068 ceph-mon[38805]: 5.a scrub starts 2026-02-23T15:55:02.834 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9696 8 365281 365273 0.294294 0 - 0.0299513 2026-02-23T15:55:03.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:55:03 trial068 ceph-mon[38805]: 5.a scrub ok 2026-02-23T15:55:03.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:55:03 trial068 ceph-mon[38805]: pgmap v5635: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:55:03.834 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9697 8 365281 365273 0.294264 0 - 0.0299513 2026-02-23T15:55:04.265 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:55:03 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:55:03.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:55:04.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:55:03 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:55:03] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:55:04.634 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:55:04.752 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:55:04.834 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9698 8 365281 365273 0.294234 0 - 0.0299513 2026-02-23T15:55:04.851 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:55:04.849+0000 7f0461361640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:55:04.851 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:55:04.850+0000 7f0461361640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:55:04.961 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:55:04.961 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":347,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:54:30.290030+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:50:39.080345+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:53:18.149281+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:50:33.211242+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:55:05.834 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9699 8 365281 365273 0.294203 0 - 0.0299513 2026-02-23T15:55:05.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:55:05 trial068 ceph-mon[38805]: pgmap v5636: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:55:05.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:55:05 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1031516281' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:55:06.834 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T15:55:06.833967+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T15:55:06.834 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T15:55:06.834 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9700 8 365281 365273 0.294173 0 - 0.0299513 2026-02-23T15:55:07.834 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9701 8 365281 365273 0.294143 0 - 0.0299513 2026-02-23T15:55:07.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:55:07 trial068 ceph-mon[38805]: pgmap v5637: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:55:07.998 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:55:08.117 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:55:08.221 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:55:08.219+0000 7f9c394b7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:55:08.221 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:55:08.219+0000 7f9c394b7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:55:08.332 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:55:08.332 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":347,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:54:30.290030+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:50:39.080345+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:53:18.149281+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:50:33.211242+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:55:08.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:55:08 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2594196623' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:55:08.834 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9702 8 365281 365273 0.294112 0 - 0.0299513 2026-02-23T15:55:09.834 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9703 8 365281 365273 0.294082 0 - 0.0299513 2026-02-23T15:55:09.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:55:09 trial068 ceph-mon[38805]: pgmap v5638: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:55:10.834 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9704 8 365281 365273 0.294052 0 - 0.0299513 2026-02-23T15:55:11.370 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:55:11.490 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:55:11.591 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:55:11.589+0000 7ff6c6187640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:55:11.591 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:55:11.589+0000 7ff6c6187640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:55:11.700 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:55:11.701 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":347,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:54:30.290030+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:50:39.080345+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:53:18.149281+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:50:33.211242+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:55:11.701 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:55:11 trial068 ceph-mon[38805]: pgmap v5639: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:55:11.834 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9705 8 365281 365273 0.294022 0 - 0.0299513 2026-02-23T15:55:12.834 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9706 8 365281 365273 0.293991 0 - 0.0299513 2026-02-23T15:55:12.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:55:12 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3056259320' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:55:13.834 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9707 8 365281 365273 0.293961 0 - 0.0299513 2026-02-23T15:55:13.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:55:13 trial068 ceph-mon[38805]: 5.8 scrub starts 2026-02-23T15:55:13.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:55:13 trial068 ceph-mon[38805]: 5.8 scrub ok 2026-02-23T15:55:13.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:55:13 trial068 ceph-mon[38805]: pgmap v5640: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:55:14.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:55:13 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:55:13] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:55:14.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:55:13 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:55:13.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:55:14.739 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:55:14.835 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9708 8 365281 365273 0.293931 0 - 0.0299513 2026-02-23T15:55:14.858 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:55:14.953 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:55:14.952+0000 7fb803658640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:55:14.953 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:55:14.952+0000 7fb803658640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:55:15.063 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:55:15.063 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":347,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:54:30.290030+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:50:39.080345+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:53:18.149281+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:50:33.211242+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:55:15.835 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9709 8 365281 365273 0.2939 0 - 0.0299513 2026-02-23T15:55:15.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:55:15 trial068 ceph-mon[38805]: pgmap v5641: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:55:15.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:55:15 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2233846287' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:55:16.835 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9710 8 365281 365273 0.29387 0 - 0.0299513 2026-02-23T15:55:17.835 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9711 8 365281 365273 0.29384 0 - 0.0299513 2026-02-23T15:55:17.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:55:17 trial068 ceph-mon[38805]: pgmap v5642: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:55:18.100 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:55:18.219 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:55:18.318 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:55:18.316+0000 7f36544a3640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:55:18.319 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:55:18.317+0000 7f36544a3640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:55:18.427 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:55:18.427 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":347,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:54:30.290030+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:50:39.080345+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:53:18.149281+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:50:33.211242+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:55:18.835 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9712 8 365281 365273 0.29381 0 - 0.0299513 2026-02-23T15:55:18.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:55:18 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/928206564' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:55:19.835 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9713 8 365281 365273 0.293779 0 - 0.0299513 2026-02-23T15:55:19.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:55:19 trial068 ceph-mon[38805]: pgmap v5643: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:55:20.835 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9714 8 365281 365273 0.293749 0 - 0.0299513 2026-02-23T15:55:21.465 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:55:21.589 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:55:21.681 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:55:21.680+0000 7fbdc7230640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:55:21.681 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:55:21.680+0000 7fbdc7230640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:55:21.789 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:55:21.789 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":347,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:54:30.290030+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:50:39.080345+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:53:18.149281+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:50:33.211242+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:55:21.789 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:55:21 trial068 ceph-mon[38805]: pgmap v5644: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:55:21.835 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9715 8 365281 365273 0.293719 0 - 0.0299513 2026-02-23T15:55:22.835 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9716 8 365281 365273 0.293689 0 - 0.0299513 2026-02-23T15:55:22.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:55:22 trial068 ceph-mon[38805]: 5.7 scrub starts 2026-02-23T15:55:22.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:55:22 trial068 ceph-mon[38805]: 5.7 scrub ok 2026-02-23T15:55:22.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:55:22 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1461862783' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:55:23.835 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9717 8 365281 365273 0.293658 0 - 0.0299513 2026-02-23T15:55:23.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:55:23 trial068 ceph-mon[38805]: pgmap v5645: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:55:24.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:55:23 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:55:23] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:55:24.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:55:23 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:55:23.974Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:55:24.827 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:55:24.835 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9718 8 365281 365273 0.293628 0 - 0.0299513 2026-02-23T15:55:24.944 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:55:25.043 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:55:25.041+0000 7fa051399640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:55:25.043 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:55:25.041+0000 7fa051399640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:55:25.152 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:55:25.152 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":347,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:54:30.290030+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:50:39.080345+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:53:18.149281+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:50:33.211242+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:55:25.835 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9719 8 365281 365273 0.293598 0 - 0.0299513 2026-02-23T15:55:25.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:55:25 trial068 ceph-mon[38805]: pgmap v5646: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:55:25.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:55:25 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2699787164' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:55:26.835 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T15:55:26.835349+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T15:55:26.836 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T15:55:26.836 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9720 8 365281 365273 0.293568 0 - 0.0299513 2026-02-23T15:55:27.835 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9721 8 365281 365273 0.293538 0 - 0.0299513 2026-02-23T15:55:27.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:55:27 trial068 ceph-mon[38805]: pgmap v5647: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:55:28.190 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:55:28.310 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:55:28.411 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:55:28.409+0000 7f076bcbf640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:55:28.411 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:55:28.409+0000 7f076bcbf640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:55:28.519 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:55:28.520 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":347,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:54:30.290030+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:50:39.080345+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:53:18.149281+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:50:33.211242+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:55:28.836 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9722 8 365281 365273 0.293507 0 - 0.0299513 2026-02-23T15:55:29.836 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9723 8 365281 365273 0.293477 0 - 0.0299513 2026-02-23T15:55:29.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:55:29 trial068 ceph-mon[38805]: pgmap v5648: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:55:29.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:55:29 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2788171310' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:55:30.836 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9724 8 365281 365273 0.293447 0 - 0.0299513 2026-02-23T15:55:31.556 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:55:31.580 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:55:31 trial068 ceph-mon[38805]: pgmap v5649: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:55:31.580 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:55:31 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2026-02-23T15:55:31.580 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:55:31 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-23T15:55:31.580 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:55:31 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2026-02-23T15:55:31.581 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:55:31 trial068 ceph-mon[38805]: pgmap v5650: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:55:31.581 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:55:31 trial068 ceph-mon[38805]: pgmap v5651: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:55:31.581 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:55:31 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T15:55:31.581 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:55:31 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2026-02-23T15:55:31.673 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:55:31.777 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:55:31.775+0000 7ff31dfd6640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:55:31.777 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:55:31.776+0000 7ff31dfd6640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:55:31.836 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9725 8 365281 365273 0.293417 0 - 0.0299513 2026-02-23T15:55:31.886 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:55:31.886 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":347,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:54:30.290030+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:50:39.080345+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:53:18.149281+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:50:33.211242+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:55:32.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:55:32 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3013881974' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:55:32.836 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9726 8 365281 365273 0.293387 0 - 0.0299513 2026-02-23T15:55:33.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:55:33 trial068 ceph-mon[38805]: pgmap v5652: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:55:33.836 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9727 8 365281 365273 0.293357 0 - 0.0299513 2026-02-23T15:55:34.264 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:55:33 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:55:33.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:55:34.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:55:33 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:55:33] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:55:34.836 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9728 8 365281 365273 0.293326 0 - 0.0299513 2026-02-23T15:55:34.924 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:55:35.044 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:55:35.150 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:55:35.148+0000 7f151540a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:55:35.150 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:55:35.148+0000 7f151540a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:55:35.259 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:55:35.259 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":347,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:54:30.290030+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:50:39.080345+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:53:18.149281+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:50:33.211242+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:55:35.836 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9729 8 365281 365273 0.293296 0 - 0.0299513 2026-02-23T15:55:36.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:55:35 trial068 ceph-mon[38805]: pgmap v5653: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:55:36.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:55:35 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1968835709' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:55:36.836 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9730 8 365281 365273 0.293266 0 - 0.0299513 2026-02-23T15:55:37.836 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9731 8 365281 365273 0.293236 0 - 0.0299513 2026-02-23T15:55:38.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:55:37 trial068 ceph-mon[38805]: pgmap v5654: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:55:38.296 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:55:38.411 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:55:38.512 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:55:38.510+0000 7fdb64ad4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:55:38.512 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:55:38.510+0000 7fdb64ad4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:55:38.622 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:55:38.622 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":347,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:54:30.290030+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:50:39.080345+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:53:18.149281+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:50:33.211242+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:55:38.836 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9732 8 365281 365273 0.293206 0 - 0.0299513 2026-02-23T15:55:39.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:55:38 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1851817518' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:55:39.836 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9733 8 365281 365273 0.293176 0 - 0.0299513 2026-02-23T15:55:40.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:55:39 trial068 ceph-mon[38805]: pgmap v5655: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:55:40.836 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9734 8 365281 365273 0.293146 0 - 0.0299513 2026-02-23T15:55:41.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:55:40 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T15:55:41.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:55:40 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T15:55:41.661 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:55:41.777 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:55:41.836 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9735 8 365281 365273 0.293115 0 - 0.0299513 2026-02-23T15:55:41.874 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:55:41.872+0000 7f2353f68640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:55:41.874 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:55:41.873+0000 7f2353f68640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:55:41.985 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:55:41.985 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":347,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:54:30.290030+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:50:39.080345+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:53:18.149281+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:50:33.211242+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:55:41.985 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:55:41 trial068 ceph-mon[38805]: pgmap v5656: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:55:42.836 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9736 8 365281 365273 0.293085 0 - 0.0299513 2026-02-23T15:55:43.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:55:42 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2055985344' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:55:43.837 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9737 8 365281 365273 0.293055 0 - 0.0299513 2026-02-23T15:55:43.973 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:55:43 trial068 ceph-mon[38805]: pgmap v5657: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:55:44.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:55:43 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:55:43] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:55:44.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:55:43 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:55:43.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:55:44.837 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9738 8 365281 365273 0.293025 0 - 0.0299513 2026-02-23T15:55:45.023 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:55:45.139 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:55:45.240 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:55:45.238+0000 7fd68fd24640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:55:45.240 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:55:45.239+0000 7fd68fd24640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:55:45.351 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:55:45.352 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":347,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:54:30.290030+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:50:39.080345+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:53:18.149281+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:50:33.211242+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:55:45.837 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9739 8 365281 365273 0.292995 0 - 0.0299513 2026-02-23T15:55:46.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:55:45 trial068 ceph-mon[38805]: pgmap v5658: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:55:46.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:55:45 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3724215107' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:55:46.837 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T15:55:46.836840+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T15:55:46.837 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T15:55:46.837 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9740 8 365281 365273 0.292965 0 - 0.0299513 2026-02-23T15:55:47.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:55:47 trial068 ceph-mon[38805]: 5.e scrub starts 2026-02-23T15:55:47.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:55:47 trial068 ceph-mon[38805]: 5.e scrub ok 2026-02-23T15:55:47.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:55:47 trial068 ceph-mon[38805]: osdmap e348: 4 total, 3 up, 4 in 2026-02-23T15:55:47.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:55:47 trial068 ceph-mon[38805]: pgmap v5660: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:55:47.837 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9741 8 365281 365273 0.292935 0 - 0.0299513 2026-02-23T15:55:48.391 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:55:48.508 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:55:48.607 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:55:48.605+0000 7faca8dfe640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:55:48.607 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:55:48.606+0000 7faca8dfe640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:55:48.716 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:55:48.716 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":348,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:55:46.303923+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:50:39.080345+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:53:18.149281+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:55:39.674672+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:55:48.837 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9742 8 365281 365273 0.292905 0 - 0.0299513 2026-02-23T15:55:49.837 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9743 8 365281 365273 0.292875 0 - 0.0299513 2026-02-23T15:55:50.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:55:49 trial068 ceph-mon[38805]: pgmap v5661: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:55:50.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:55:49 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1347200059' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:55:50.837 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9744 8 365281 365273 0.292845 0 - 0.0299513 2026-02-23T15:55:51.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:55:50 trial068 ceph-mon[38805]: 5.5 scrub starts 2026-02-23T15:55:51.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:55:50 trial068 ceph-mon[38805]: osdmap e349: 4 total, 3 up, 4 in 2026-02-23T15:55:51.768 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:55:51.837 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9745 8 365281 365273 0.292815 0 - 0.0299513 2026-02-23T15:55:51.884 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:55:51.987 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:55:51.985+0000 7f65d8605640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:55:51.987 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:55:51 trial068 ceph-mon[38805]: 5.5 scrub ok 2026-02-23T15:55:51.987 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:55:51 trial068 ceph-mon[38805]: pgmap v5663: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:55:51.987 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:55:51.985+0000 7f65d8605640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:55:52.096 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:55:52.096 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":349,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:55:50.302547+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:50:39.080345+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:53:18.149281+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:55:39.674672+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:55:52.838 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9746 8 365281 365273 0.292785 0 - 0.0299513 2026-02-23T15:55:53.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:55:52 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1367377853' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:55:53.838 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9747 8 365281 365273 0.292755 0 - 0.0299513 2026-02-23T15:55:53.973 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:55:53 trial068 ceph-mon[38805]: pgmap v5664: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:55:54.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:55:53 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:55:53] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:55:54.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:55:53 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:55:53.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:55:54.838 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9748 8 365281 365273 0.292725 0 - 0.0299513 2026-02-23T15:55:55.133 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:55:55.252 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:55:55.347 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:55:55.345+0000 7f3e05e85640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:55:55.347 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:55:55.346+0000 7f3e05e85640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:55:55.455 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:55:55.455 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":349,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:55:50.302547+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:50:39.080345+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:53:18.149281+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:55:39.674672+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:55:55.838 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9749 8 365281 365273 0.292695 0 - 0.0299513 2026-02-23T15:55:55.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:55:55 trial068 ceph-mon[38805]: pgmap v5665: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:55:55.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:55:55 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/4174157051' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:55:56.838 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9750 8 365281 365273 0.292664 0 - 0.0299513 2026-02-23T15:55:57.838 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9751 8 365281 365273 0.292634 0 - 0.0299513 2026-02-23T15:55:58.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:55:57 trial068 ceph-mon[38805]: pgmap v5666: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:55:58.492 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:55:58.607 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:55:58.709 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:55:58.708+0000 7fb4bb7fd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:55:58.709 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:55:58.708+0000 7fb4bb7fd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:55:58.819 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:55:58.820 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":349,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:55:50.302547+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:50:39.080345+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:53:18.149281+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:55:39.674672+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:55:58.838 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9752 8 365281 365273 0.292604 0 - 0.0299513 2026-02-23T15:55:59.838 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9753 8 365281 365273 0.292574 0 - 0.0299513 2026-02-23T15:56:00.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:55:59 trial068 ceph-mon[38805]: pgmap v5667: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:56:00.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:55:59 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/553811483' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:56:00.838 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9754 8 365281 365273 0.292544 0 - 0.0299513 2026-02-23T15:56:01.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:56:00 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T15:56:01.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:56:00 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T15:56:01.838 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9755 8 365281 365273 0.292514 0 - 0.0299513 2026-02-23T15:56:01.857 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:56:01.976 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:56:01.998 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:56:01 trial068 ceph-mon[38805]: pgmap v5668: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:56:02.074 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:56:02.072+0000 7f8d41d45640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:56:02.074 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:56:02.072+0000 7f8d41d45640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:56:02.182 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:56:02.182 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":349,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:55:50.302547+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:50:39.080345+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:53:18.149281+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:55:39.674672+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:56:02.839 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9756 8 365281 365273 0.292484 0 - 0.0299513 2026-02-23T15:56:03.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:56:02 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1657736903' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:56:03.838 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9757 8 365281 365273 0.292455 0 - 0.0299513 2026-02-23T15:56:03.974 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:56:03 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T15:56:03.974 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:56:03 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T15:56:03.974 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:56:03 trial068 ceph-mon[38805]: pgmap v5669: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:56:04.265 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:56:03 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:56:03.974Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:56:04.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:56:03 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:56:03] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:56:04.838 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9758 8 365281 365273 0.292425 0 - 0.0299513 2026-02-23T15:56:05.220 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:56:05.339 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:56:05.444 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:56:05.442+0000 7fbe77ed7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:56:05.444 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:56:05.442+0000 7fbe77ed7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:56:05.554 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:56:05.554 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":349,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:55:50.302547+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:50:39.080345+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:53:18.149281+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:55:39.674672+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:56:05.839 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9759 8 365281 365273 0.292395 0 - 0.0299513 2026-02-23T15:56:05.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:56:05 trial068 ceph-mon[38805]: pgmap v5670: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:56:05.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:56:05 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/744874557' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:56:06.839 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T15:56:06.838707+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T15:56:06.839 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T15:56:06.839 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9760 8 365281 365273 0.292365 0 - 0.0299513 2026-02-23T15:56:07.839 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9761 8 365281 365273 0.292335 0 - 0.0299513 2026-02-23T15:56:08.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:56:07 trial068 ceph-mon[38805]: pgmap v5671: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:56:08.591 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:56:08.709 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:56:08.804 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:56:08.802+0000 7fdb55ff4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:56:08.804 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:56:08.803+0000 7fdb55ff4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:56:08.839 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9762 8 365281 365273 0.292305 0 - 0.0299513 2026-02-23T15:56:08.914 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:56:08.914 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":349,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:55:50.302547+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:50:39.080345+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:53:18.149281+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:55:39.674672+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:56:09.839 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9763 8 365281 365273 0.292275 0 - 0.0299513 2026-02-23T15:56:10.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:56:09 trial068 ceph-mon[38805]: pgmap v5672: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:56:10.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:56:09 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1761471022' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:56:10.839 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9764 8 365281 365273 0.292245 0 - 0.0299513 2026-02-23T15:56:11.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:56:10 trial068 ceph-mon[38805]: 5.a scrub starts 2026-02-23T15:56:11.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:56:10 trial068 ceph-mon[38805]: 5.a scrub ok 2026-02-23T15:56:11.839 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9765 8 365281 365273 0.292215 0 - 0.0299513 2026-02-23T15:56:11.950 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:56:11.975 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:56:11 trial068 ceph-mon[38805]: pgmap v5673: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:56:12.069 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:56:12.167 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:56:12.165+0000 7fbae12f8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:56:12.167 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:56:12.165+0000 7fbae12f8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:56:12.279 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:56:12.279 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":349,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:55:50.302547+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:50:39.080345+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:53:18.149281+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:55:39.674672+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:56:12.839 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9766 8 365281 365273 0.292185 0 - 0.0299513 2026-02-23T15:56:13.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:56:12 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/314936766' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:56:13.839 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9767 8 365281 365273 0.292155 0 - 0.0299513 2026-02-23T15:56:13.973 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:56:13 trial068 ceph-mon[38805]: pgmap v5674: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:56:14.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:56:13 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:56:13] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:56:14.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:56:13 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:56:13.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:56:14.839 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9768 8 365281 365273 0.292125 0 - 0.0299513 2026-02-23T15:56:15.317 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:56:15.434 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:56:15.531 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:56:15.529+0000 7f444c667640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:56:15.531 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:56:15.529+0000 7f444c667640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:56:15.640 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:56:15.641 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":349,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:55:50.302547+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:50:39.080345+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:53:18.149281+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:55:39.674672+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:56:15.839 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9769 8 365281 365273 0.292095 0 - 0.0299513 2026-02-23T15:56:16.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:56:15 trial068 ceph-mon[38805]: pgmap v5675: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:56:16.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:56:15 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2419335195' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:56:16.839 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9770 8 365281 365273 0.292065 0 - 0.0299513 2026-02-23T15:56:17.839 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9771 8 365281 365273 0.292035 0 - 0.0299513 2026-02-23T15:56:18.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:56:17 trial068 ceph-mon[38805]: pgmap v5676: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:56:18.677 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:56:18.792 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:56:18.839 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9772 8 365281 365273 0.292006 0 - 0.0299513 2026-02-23T15:56:18.887 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:56:18.885+0000 7f5298eb6640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:56:18.887 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:56:18.886+0000 7f5298eb6640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:56:18.995 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:56:18.995 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":349,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:55:50.302547+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:50:39.080345+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:53:18.149281+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:55:39.674672+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:56:19.839 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9773 8 365281 365273 0.291976 0 - 0.0299513 2026-02-23T15:56:20.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:56:19 trial068 ceph-mon[38805]: pgmap v5677: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:56:20.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:56:19 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1059201088' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:56:20.839 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9774 8 365281 365273 0.291946 0 - 0.0299513 2026-02-23T15:56:21.840 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9775 8 365281 365273 0.291916 0 - 0.0299513 2026-02-23T15:56:22.032 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:56:22.056 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:56:21 trial068 ceph-mon[38805]: pgmap v5678: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:56:22.150 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:56:22.256 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:56:22.255+0000 7f211f855640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:56:22.256 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:56:22.255+0000 7f211f855640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:56:22.365 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:56:22.365 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":349,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:55:50.302547+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:50:39.080345+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:53:18.149281+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:55:39.674672+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:56:22.840 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9776 8 365281 365273 0.291886 0 - 0.0299513 2026-02-23T15:56:23.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:56:22 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2418804794' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:56:23.840 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9777 8 365281 365273 0.291856 0 - 0.0299513 2026-02-23T15:56:24.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:56:23 trial068 ceph-mon[38805]: pgmap v5679: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:56:24.094 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:56:23 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:56:23] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:56:24.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:56:23 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:56:23.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:56:24.840 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9778 8 365281 365273 0.291826 0 - 0.0299513 2026-02-23T15:56:25.400 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:56:25.515 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:56:25.611 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:56:25.609+0000 7f639e2f7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:56:25.612 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:56:25.610+0000 7f639e2f7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:56:25.724 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:56:25.724 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":349,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:55:50.302547+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:50:39.080345+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:53:18.149281+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:55:39.674672+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:56:25.840 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9779 8 365281 365273 0.291797 0 - 0.0299513 2026-02-23T15:56:26.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:56:25 trial068 ceph-mon[38805]: pgmap v5680: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:56:26.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:56:25 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1057150878' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:56:26.840 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T15:56:26.840041+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T15:56:26.840 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T15:56:26.840 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9780 8 365281 365273 0.291767 0 - 0.0299513 2026-02-23T15:56:27.840 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9781 8 365281 365273 0.291737 0 - 0.0299513 2026-02-23T15:56:28.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:56:27 trial068 ceph-mon[38805]: pgmap v5681: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:56:28.762 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:56:28.840 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9782 8 365281 365273 0.291707 0 - 0.0299513 2026-02-23T15:56:28.878 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:56:28.977 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:56:28.975+0000 7f4518237640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:56:28.977 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:56:28.976+0000 7f4518237640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:56:29.086 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:56:29.086 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":349,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:55:50.302547+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:50:39.080345+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:53:18.149281+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:55:39.674672+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:56:29.840 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9783 8 365281 365273 0.291677 0 - 0.0299513 2026-02-23T15:56:30.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:56:29 trial068 ceph-mon[38805]: pgmap v5682: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:56:30.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:56:29 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/444335485' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:56:30.840 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9784 8 365281 365273 0.291647 0 - 0.0299513 2026-02-23T15:56:31.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:56:30 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2026-02-23T15:56:31.840 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9785 8 365281 365273 0.291618 0 - 0.0299513 2026-02-23T15:56:32.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:56:31 trial068 ceph-mon[38805]: pgmap v5683: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:56:32.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:56:31 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-23T15:56:32.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:56:31 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2026-02-23T15:56:32.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:56:31 trial068 ceph-mon[38805]: pgmap v5684: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:56:32.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:56:31 trial068 ceph-mon[38805]: pgmap v5685: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:56:32.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:56:31 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T15:56:32.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:56:31 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2026-02-23T15:56:32.123 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:56:32.239 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:56:32.340 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:56:32.338+0000 7faf50ece640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:56:32.340 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:56:32.338+0000 7faf50ece640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:56:32.450 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:56:32.450 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":349,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:55:50.302547+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:50:39.080345+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:53:18.149281+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:55:39.674672+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:56:32.841 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9786 8 365281 365273 0.291588 0 - 0.0299513 2026-02-23T15:56:33.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:56:32 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/985707523' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:56:33.841 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9787 8 365281 365273 0.291558 0 - 0.0299513 2026-02-23T15:56:34.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:56:33 trial068 ceph-mon[38805]: pgmap v5686: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:56:34.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:56:33 trial068 ceph-mon[38805]: 5.8 scrub starts 2026-02-23T15:56:34.094 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:56:33 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:56:33] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:56:34.265 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:56:33 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:56:33.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:56:34.841 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9788 8 365281 365273 0.291528 0 - 0.0299513 2026-02-23T15:56:35.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:56:34 trial068 ceph-mon[38805]: 5.8 scrub ok 2026-02-23T15:56:35.488 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:56:35.608 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:56:35.707 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:56:35.705+0000 7f1e5dc47640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:56:35.707 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:56:35.706+0000 7f1e5dc47640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:56:35.818 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:56:35.818 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":349,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:55:50.302547+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:50:39.080345+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:53:18.149281+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:55:39.674672+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:56:35.818 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:56:35 trial068 ceph-mon[38805]: pgmap v5687: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:56:35.841 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9789 8 365281 365273 0.291498 0 - 0.0299513 2026-02-23T15:56:36.841 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9790 8 365281 365273 0.291469 0 - 0.0299513 2026-02-23T15:56:37.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:56:36 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2658035880' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:56:37.841 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9791 8 365281 365273 0.291439 0 - 0.0299513 2026-02-23T15:56:38.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:56:37 trial068 ceph-mon[38805]: 5.7 scrub starts 2026-02-23T15:56:38.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:56:37 trial068 ceph-mon[38805]: pgmap v5688: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:56:38.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:56:37 trial068 ceph-mon[38805]: 5.7 scrub ok 2026-02-23T15:56:38.841 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9792 8 365281 365273 0.291409 0 - 0.0299513 2026-02-23T15:56:38.870 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:56:38.988 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:56:39.090 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:56:39.088+0000 7efc2008e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:56:39.090 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:56:39.089+0000 7efc2008e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:56:39.200 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:56:39.200 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":349,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:55:50.302547+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:50:39.080345+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:53:18.149281+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:55:39.674672+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:56:39.841 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9793 8 365281 365273 0.291379 0 - 0.0299513 2026-02-23T15:56:40.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:56:39 trial068 ceph-mon[38805]: pgmap v5689: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:56:40.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:56:39 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/276826439' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:56:40.841 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9794 8 365281 365273 0.29135 0 - 0.0299513 2026-02-23T15:56:41.841 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9795 8 365281 365273 0.29132 0 - 0.0299513 2026-02-23T15:56:42.237 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:56:42.260 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:56:41 trial068 ceph-mon[38805]: pgmap v5690: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:56:42.353 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:56:42.455 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:56:42.453+0000 7fac0361c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:56:42.455 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:56:42.453+0000 7fac0361c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:56:42.564 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:56:42.564 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":349,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:55:50.302547+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:50:39.080345+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:53:18.149281+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:55:39.674672+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:56:42.841 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9796 8 365281 365273 0.29129 0 - 0.0299513 2026-02-23T15:56:43.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:56:42 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/649375741' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:56:43.841 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9797 8 365281 365273 0.29126 0 - 0.0299513 2026-02-23T15:56:44.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:56:43 trial068 ceph-mon[38805]: pgmap v5691: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:56:44.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:56:43 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:56:43] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:56:44.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:56:43 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:56:43.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:56:44.841 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9798 8 365281 365273 0.291231 0 - 0.0299513 2026-02-23T15:56:45.603 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:56:45.720 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:56:45.825 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:56:45.823+0000 7f9741fd2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:56:45.825 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:56:45.823+0000 7f9741fd2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:56:45.841 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9799 8 365281 365273 0.291201 0 - 0.0299513 2026-02-23T15:56:45.936 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:56:45.936 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":349,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:55:50.302547+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:50:39.080345+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:53:18.149281+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:55:39.674672+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:56:46.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:56:45 trial068 ceph-mon[38805]: pgmap v5692: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:56:46.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:56:45 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/109259194' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:56:46.841 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T15:56:46.841427+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T15:56:46.842 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T15:56:46.842 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9800 8 365281 365273 0.291171 0 - 0.0299513 2026-02-23T15:56:47.841 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9801 8 365281 365273 0.291142 0 - 0.0299513 2026-02-23T15:56:48.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:56:47 trial068 ceph-mon[38805]: pgmap v5693: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:56:48.842 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9802 8 365281 365273 0.291112 0 - 0.0299513 2026-02-23T15:56:48.973 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:56:49.091 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:56:49.193 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:56:49.190+0000 7fb7a0495640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:56:49.193 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:56:49.191+0000 7fb7a0495640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:56:49.304 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:56:49.304 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":349,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:55:50.302547+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:50:39.080345+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:53:18.149281+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:55:39.674672+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:56:49.842 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9803 8 365281 365273 0.291082 0 - 0.0299513 2026-02-23T15:56:50.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:56:49 trial068 ceph-mon[38805]: pgmap v5694: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:56:50.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:56:49 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3344208088' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:56:50.842 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9804 8 365281 365273 0.291053 0 - 0.0299513 2026-02-23T15:56:51.842 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9805 8 365281 365273 0.291023 0 - 0.0299513 2026-02-23T15:56:52.342 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:56:52.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:56:51 trial068 ceph-mon[38805]: pgmap v5695: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:56:52.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:56:51 trial068 ceph-mon[38805]: osdmap e350: 4 total, 3 up, 4 in 2026-02-23T15:56:52.460 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:56:52.557 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:56:52.555+0000 7fbc12b77640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:56:52.557 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:56:52.555+0000 7fbc12b77640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:56:52.666 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:56:52.666 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":350,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:56:50.965172+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:56:00.217470+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:53:18.149281+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:55:39.674672+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:56:52.842 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9806 8 365281 365273 0.290993 0 - 0.0299513 2026-02-23T15:56:53.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:56:52 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3055984718' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:56:53.842 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9807 8 365281 365273 0.290963 0 - 0.0299513 2026-02-23T15:56:54.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:56:53 trial068 ceph-mon[38805]: pgmap v5697: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:56:54.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:56:53 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:56:53] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:56:54.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:56:53 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:56:53.972Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:56:54.842 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9808 8 365281 365273 0.290934 0 - 0.0299513 2026-02-23T15:56:55.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:56:54 trial068 ceph-mon[38805]: 5.e scrub starts 2026-02-23T15:56:55.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:56:54 trial068 ceph-mon[38805]: 5.e scrub ok 2026-02-23T15:56:55.703 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:56:55.820 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:56:55.842 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9809 8 365281 365273 0.290904 0 - 0.0299513 2026-02-23T15:56:55.924 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:56:55.922+0000 7fbdc8835640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:56:55.924 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:56:55.922+0000 7fbdc8835640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:56:56.033 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:56:56.033 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":351,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:56:55.310042+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:56:00.217470+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:53:18.149281+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:55:39.674672+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:56:56.034 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:56:55 trial068 ceph-mon[38805]: pgmap v5698: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:56:56.034 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:56:55 trial068 ceph-mon[38805]: osdmap e351: 4 total, 3 up, 4 in 2026-02-23T15:56:56.842 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9810 8 365281 365273 0.290875 0 - 0.0299513 2026-02-23T15:56:57.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:56:56 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2167083424' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:56:57.842 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9811 8 365281 365273 0.290845 0 - 0.0299513 2026-02-23T15:56:58.244 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:56:57 trial068 ceph-mon[38805]: pgmap v5700: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:56:58.842 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9812 8 365281 365273 0.290815 0 - 0.0299513 2026-02-23T15:56:59.073 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:56:59.191 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:56:59.296 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:56:59.294+0000 7f2c7ec61640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:56:59.296 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:56:59.294+0000 7f2c7ec61640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:56:59.408 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:56:59.409 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":351,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:56:55.310042+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:56:00.217470+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:53:18.149281+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:55:39.674672+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:56:59.842 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9813 8 365281 365273 0.290786 0 - 0.0299513 2026-02-23T15:57:00.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:56:59 trial068 ceph-mon[38805]: pgmap v5701: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:57:00.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:56:59 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/4235889509' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:57:00.842 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9814 8 365281 365273 0.290756 0 - 0.0299513 2026-02-23T15:57:01.843 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9815 8 365281 365273 0.290726 0 - 0.0299513 2026-02-23T15:57:02.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:57:01 trial068 ceph-mon[38805]: pgmap v5702: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:57:02.448 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:57:02.564 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:57:02.671 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:57:02.669+0000 7f702d417640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:57:02.671 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:57:02.670+0000 7f702d417640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:57:02.780 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:57:02.780 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":351,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:56:55.310042+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:56:00.217470+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:53:18.149281+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:55:39.674672+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:57:02.843 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9816 8 365281 365273 0.290697 0 - 0.0299513 2026-02-23T15:57:03.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:57:02 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/292673178' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:57:03.843 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9817 8 365281 365273 0.290667 0 - 0.0299513 2026-02-23T15:57:04.265 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:57:03 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:57:03.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:57:04.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:57:03 trial068 ceph-mon[38805]: pgmap v5703: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:57:04.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:57:03 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:57:03] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:57:04.843 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9818 8 365281 365273 0.290637 0 - 0.0299513 2026-02-23T15:57:05.818 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:57:05.843 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9819 8 365281 365273 0.290608 0 - 0.0299513 2026-02-23T15:57:05.938 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:57:06.041 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:57:06.039+0000 7f7bf4b0d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:57:06.041 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:57:06.039+0000 7f7bf4b0d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:57:06.149 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:57:06.150 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":351,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:56:55.310042+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:56:00.217470+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:53:18.149281+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:55:39.674672+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:57:06.150 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:57:05 trial068 ceph-mon[38805]: pgmap v5704: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:57:06.843 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T15:57:06.842781+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T15:57:06.843 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T15:57:06.843 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9820 8 365281 365273 0.290578 0 - 0.0299513 2026-02-23T15:57:07.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:57:06 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2815922189' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:57:07.843 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9821 8 365281 365273 0.290549 0 - 0.0299513 2026-02-23T15:57:08.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:57:07 trial068 ceph-mon[38805]: pgmap v5705: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:57:08.843 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9822 8 365281 365273 0.290519 0 - 0.0299513 2026-02-23T15:57:09.189 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:57:09.306 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:57:09.328 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:57:08 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T15:57:09.328 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:57:08 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T15:57:09.406 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:57:09.404+0000 7f53592b4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:57:09.407 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:57:09.405+0000 7f53592b4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:57:09.514 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:57:09.515 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":351,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:56:55.310042+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:56:00.217470+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:53:18.149281+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:55:39.674672+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:57:09.843 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9823 8 365281 365273 0.29049 0 - 0.0299513 2026-02-23T15:57:10.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:57:09 trial068 ceph-mon[38805]: pgmap v5706: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:57:10.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:57:09 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3297563868' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:57:10.843 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9824 8 365281 365273 0.29046 0 - 0.0299513 2026-02-23T15:57:11.843 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9825 8 365281 365273 0.29043 0 - 0.0299513 2026-02-23T15:57:12.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:57:11 trial068 ceph-mon[38805]: pgmap v5707: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:57:12.552 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:57:12.669 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:57:12.769 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:57:12.767+0000 7f3bd7b40640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:57:12.769 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:57:12.767+0000 7f3bd7b40640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:57:12.843 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9826 8 365281 365273 0.290401 0 - 0.0299513 2026-02-23T15:57:12.879 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:57:12.879 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":351,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:56:55.310042+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:56:00.217470+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:53:18.149281+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:55:39.674672+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:57:13.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:57:12 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1591031856' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:57:13.843 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9827 8 365281 365273 0.290371 0 - 0.0299513 2026-02-23T15:57:14.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:57:13 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T15:57:14.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:57:13 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T15:57:14.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:57:13 trial068 ceph-mon[38805]: pgmap v5708: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:57:14.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:57:13 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:57:13] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:57:14.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:57:13 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:57:13.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:57:14.843 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9828 8 365281 365273 0.290342 0 - 0.0299513 2026-02-23T15:57:15.844 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9829 8 365281 365273 0.290312 0 - 0.0299513 2026-02-23T15:57:15.917 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:57:16.036 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:57:16.133 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:57:16.131+0000 7fef4b4e0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:57:16.133 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:57:16.131+0000 7fef4b4e0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:57:16.241 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:57:16.241 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":351,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:56:55.310042+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:56:00.217470+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:53:18.149281+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:55:39.674672+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:57:16.241 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:57:15 trial068 ceph-mon[38805]: pgmap v5709: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:57:16.844 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9830 8 365281 365273 0.290283 0 - 0.0299513 2026-02-23T15:57:17.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:57:16 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T15:57:17.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:57:16 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T15:57:17.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:57:16 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2599117503' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:57:17.844 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9831 8 365281 365273 0.290253 0 - 0.0299513 2026-02-23T15:57:18.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:57:17 trial068 ceph-mon[38805]: pgmap v5710: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:57:18.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:57:17 trial068 ceph-mon[38805]: 5.5 scrub starts 2026-02-23T15:57:18.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:57:17 trial068 ceph-mon[38805]: 5.5 scrub ok 2026-02-23T15:57:18.844 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9832 8 365281 365273 0.290224 0 - 0.0299513 2026-02-23T15:57:19.278 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:57:19.396 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:57:19.491 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:57:19.489+0000 7f38d4bf6640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:57:19.491 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:57:19.490+0000 7f38d4bf6640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:57:19.599 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:57:19.600 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":351,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:56:55.310042+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:56:00.217470+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:53:18.149281+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:55:39.674672+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:57:19.844 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9833 8 365281 365273 0.290194 0 - 0.0299513 2026-02-23T15:57:20.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:57:19 trial068 ceph-mon[38805]: 5.a scrub starts 2026-02-23T15:57:20.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:57:19 trial068 ceph-mon[38805]: pgmap v5711: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:57:20.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:57:19 trial068 ceph-mon[38805]: 5.a scrub ok 2026-02-23T15:57:20.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:57:19 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1899946008' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:57:20.844 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9834 8 365281 365273 0.290165 0 - 0.0299513 2026-02-23T15:57:21.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:57:21 trial068 ceph-mon[38805]: pgmap v5712: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:57:21.844 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9835 8 365281 365273 0.290135 0 - 0.0299513 2026-02-23T15:57:22.636 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:57:22.752 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:57:22.844 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9836 8 365281 365273 0.290106 0 - 0.0299513 2026-02-23T15:57:22.854 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:57:22.852+0000 7fcd6f77a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:57:22.854 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:57:22.852+0000 7fcd6f77a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:57:22.963 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:57:22.963 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":351,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:56:55.310042+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:56:00.217470+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:53:18.149281+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:55:39.674672+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:57:23.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:57:23 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1313334092' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:57:23.844 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9837 8 365281 365273 0.290076 0 - 0.0299513 2026-02-23T15:57:24.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:57:24 trial068 ceph-mon[38805]: pgmap v5713: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:57:24.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:57:23 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:57:23] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:57:24.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:57:23 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:57:23.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:57:24.844 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9838 8 365281 365273 0.290047 0 - 0.0299513 2026-02-23T15:57:25.844 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9839 8 365281 365273 0.290017 0 - 0.0299513 2026-02-23T15:57:26.000 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:57:26.121 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:57:26.220 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:57:26.218+0000 7fc7103e3640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:57:26.220 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:57:26.218+0000 7fc7103e3640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:57:26.331 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:57:26.331 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":351,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:56:55.310042+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:56:00.217470+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:53:18.149281+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:55:39.674672+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:57:26.332 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:57:26 trial068 ceph-mon[38805]: pgmap v5714: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:57:26.844 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T15:57:26.844263+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T15:57:26.845 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T15:57:26.845 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9840 8 365281 365273 0.289988 0 - 0.0299513 2026-02-23T15:57:27.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:57:27 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1037254475' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:57:27.845 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9841 8 365281 365273 0.289958 0 - 0.0299513 2026-02-23T15:57:28.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:57:28 trial068 ceph-mon[38805]: pgmap v5715: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:57:28.845 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9842 8 365281 365273 0.289929 0 - 0.0299513 2026-02-23T15:57:29.368 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:57:29.487 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:57:29.586 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:57:29.584+0000 7f82cb89a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:57:29.586 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:57:29.584+0000 7f82cb89a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:57:29.695 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:57:29.696 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":351,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:56:55.310042+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:56:00.217470+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:53:18.149281+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:55:39.674672+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:57:29.844 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9843 8 365281 365273 0.289899 0 - 0.0299513 2026-02-23T15:57:30.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:57:30 trial068 ceph-mon[38805]: pgmap v5716: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:57:30.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:57:30 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1287902927' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:57:30.845 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9844 8 365281 365273 0.28987 0 - 0.0299513 2026-02-23T15:57:31.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:57:31 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2026-02-23T15:57:31.845 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9845 8 365281 365273 0.28984 0 - 0.0299513 2026-02-23T15:57:32.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:57:32 trial068 ceph-mon[38805]: pgmap v5717: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:57:32.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:57:32 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T15:57:32.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:57:32 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T15:57:32.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:57:32 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T15:57:32.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:57:32 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T15:57:32.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:57:32 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-23T15:57:32.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:57:32 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2026-02-23T15:57:32.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:57:32 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T15:57:32.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:57:32 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2026-02-23T15:57:32.732 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:57:32.845 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9846 8 365281 365273 0.289811 0 - 0.0299513 2026-02-23T15:57:32.850 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:57:32.953 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:57:32.951+0000 7fc4db410640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:57:32.953 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:57:32.952+0000 7fc4db410640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:57:33.062 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:57:33.062 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":351,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:56:55.310042+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:56:00.217470+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:53:18.149281+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:55:39.674672+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:57:33.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:57:33 trial068 ceph-mon[38805]: pgmap v5718: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:57:33.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:57:33 trial068 ceph-mon[38805]: pgmap v5719: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:57:33.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:57:33 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2510005453' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:57:33.845 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9847 8 365281 365273 0.289782 0 - 0.0299513 2026-02-23T15:57:34.265 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:57:33 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:57:33.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:57:34.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:57:33 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:57:33] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:57:34.845 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9848 8 365281 365273 0.289752 0 - 0.0299513 2026-02-23T15:57:35.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:57:35 trial068 ceph-mon[38805]: pgmap v5720: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:57:35.845 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9849 8 365281 365273 0.289723 0 - 0.0299513 2026-02-23T15:57:36.099 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:57:36.216 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:57:36.313 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:57:36.311+0000 7f59d9c59640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:57:36.313 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:57:36.311+0000 7f59d9c59640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:57:36.421 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:57:36.421 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":351,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:56:55.310042+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:56:00.217470+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:53:18.149281+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:55:39.674672+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:57:36.422 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:57:36 trial068 ceph-mon[38805]: pgmap v5721: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:57:36.845 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9850 8 365281 365273 0.289693 0 - 0.0299513 2026-02-23T15:57:37.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:57:37 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/845089986' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:57:37.845 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9851 8 365281 365273 0.289664 0 - 0.0299513 2026-02-23T15:57:38.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:57:38 trial068 ceph-mon[38805]: pgmap v5722: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:57:38.845 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9852 8 365281 365273 0.289634 0 - 0.0299513 2026-02-23T15:57:39.462 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:57:39.580 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:57:39.681 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:57:39.679+0000 7ffb3a2f6640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:57:39.681 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:57:39.680+0000 7ffb3a2f6640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:57:39.790 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:57:39.790 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":351,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:56:55.310042+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:56:00.217470+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:53:18.149281+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:55:39.674672+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:57:39.845 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9853 8 365281 365273 0.289605 0 - 0.0299513 2026-02-23T15:57:40.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:57:40 trial068 ceph-mon[38805]: pgmap v5723: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:57:40.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:57:40 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/73616817' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:57:40.845 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9854 8 365281 365273 0.289576 0 - 0.0299513 2026-02-23T15:57:41.845 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9855 8 365281 365273 0.289546 0 - 0.0299513 2026-02-23T15:57:42.830 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:57:42.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:57:42 trial068 ceph-mon[38805]: pgmap v5724: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:57:42.845 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9856 8 365281 365273 0.289517 0 - 0.0299513 2026-02-23T15:57:42.948 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:57:43.045 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:57:43.044+0000 7f414434b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:57:43.046 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:57:43.044+0000 7f414434b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:57:43.155 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:57:43.155 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":351,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:56:55.310042+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:56:00.217470+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:53:18.149281+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:55:39.674672+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:57:43.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:57:43 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2279010935' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:57:43.845 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9857 8 365281 365273 0.289488 0 - 0.0299513 2026-02-23T15:57:44.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:57:43 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:57:43] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:57:44.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:57:43 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:57:43.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:57:44.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:57:44 trial068 ceph-mon[38805]: pgmap v5725: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:57:44.845 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9858 8 365281 365273 0.289458 0 - 0.0299513 2026-02-23T15:57:45.846 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9859 8 365281 365273 0.289429 0 - 0.0299513 2026-02-23T15:57:46.193 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:57:46.311 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:57:46.411 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:57:46.409+0000 7f00ef111640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:57:46.411 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:57:46.410+0000 7f00ef111640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:57:46.521 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:57:46.521 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":351,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:56:55.310042+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:56:00.217470+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:53:18.149281+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:55:39.674672+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:57:46.522 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:57:46 trial068 ceph-mon[38805]: pgmap v5726: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:57:46.846 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T15:57:46.845624+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T15:57:46.846 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T15:57:46.846 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9860 8 365281 365273 0.289399 0 - 0.0299513 2026-02-23T15:57:47.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:57:47 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/4231313044' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:57:47.846 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9861 8 365281 365273 0.28937 0 - 0.0299513 2026-02-23T15:57:48.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:57:48 trial068 ceph-mon[38805]: pgmap v5727: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:57:48.846 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9862 8 365281 365273 0.289341 0 - 0.0299513 2026-02-23T15:57:49.559 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:57:49.676 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:57:49.776 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:57:49.774+0000 7f44ef46d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:57:49.776 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:57:49.775+0000 7f44ef46d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:57:49.846 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9863 8 365281 365273 0.289311 0 - 0.0299513 2026-02-23T15:57:49.887 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:57:49.888 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":351,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:56:55.310042+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:56:00.217470+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:53:18.149281+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:55:39.674672+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:57:50.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:57:50 trial068 ceph-mon[38805]: pgmap v5728: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:57:50.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:57:50 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2152537619' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:57:50.846 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9864 8 365281 365273 0.289282 0 - 0.0299513 2026-02-23T15:57:51.846 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9865 8 365281 365273 0.289253 0 - 0.0299513 2026-02-23T15:57:52.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:57:52 trial068 ceph-mon[38805]: pgmap v5729: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:57:52.846 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9866 8 365281 365273 0.289223 0 - 0.0299513 2026-02-23T15:57:52.927 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:57:53.044 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:57:53.142 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:57:53.140+0000 7fc45491d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:57:53.142 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:57:53.140+0000 7fc45491d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:57:53.251 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:57:53.251 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":351,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:56:55.310042+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:56:00.217470+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:53:18.149281+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:55:39.674672+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:57:53.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:57:53 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1882890956' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:57:53.846 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9867 8 365281 365273 0.289194 0 - 0.0299513 2026-02-23T15:57:54.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:57:53 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:57:53] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:57:54.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:57:53 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:57:53.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:57:54.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:57:54 trial068 ceph-mon[38805]: pgmap v5730: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:57:54.846 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9868 8 365281 365273 0.289165 0 - 0.0299513 2026-02-23T15:57:55.846 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9869 8 365281 365273 0.289136 0 - 0.0299513 2026-02-23T15:57:56.289 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:57:56.409 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:57:56.515 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:57:56.514+0000 7fe3d934a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:57:56.516 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:57:56.514+0000 7fe3d934a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:57:56.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:57:56 trial068 ceph-mon[38805]: pgmap v5731: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:57:56.624 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:57:56.624 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":351,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:56:55.310042+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:56:00.217470+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:53:18.149281+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:55:39.674672+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:57:56.846 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9870 8 365281 365273 0.289106 0 - 0.0299513 2026-02-23T15:57:57.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:57:57 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2017222192' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:57:57.846 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9871 8 365281 365273 0.289077 0 - 0.0299513 2026-02-23T15:57:58.744 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:57:58 trial068 ceph-mon[38805]: pgmap v5732: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:57:58.846 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9872 8 365281 365273 0.289048 0 - 0.0299513 2026-02-23T15:57:59.661 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:57:59.780 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:57:59.846 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9873 8 365281 365273 0.289018 0 - 0.0299513 2026-02-23T15:57:59.886 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:57:59.884+0000 7f8d2a797640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:57:59.886 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:57:59.884+0000 7f8d2a797640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:57:59.993 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:57:59.994 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":351,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:56:55.310042+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:56:00.217470+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:53:18.149281+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:55:39.674672+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:58:00.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:58:00 trial068 ceph-mon[38805]: pgmap v5733: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:58:00.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:58:00 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3027217898' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:58:00.847 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9874 8 365281 365273 0.288989 0 - 0.0299513 2026-02-23T15:58:01.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:58:01 trial068 ceph-mon[38805]: 5.7 scrub starts 2026-02-23T15:58:01.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:58:01 trial068 ceph-mon[38805]: 5.7 scrub ok 2026-02-23T15:58:01.847 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9875 8 365281 365273 0.28896 0 - 0.0299513 2026-02-23T15:58:02.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:58:02 trial068 ceph-mon[38805]: pgmap v5734: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:58:02.847 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9876 8 365281 365273 0.288931 0 - 0.0299513 2026-02-23T15:58:03.031 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:58:03.147 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:58:03.249 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:58:03.247+0000 7f31bf5ba640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:58:03.249 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:58:03.247+0000 7f31bf5ba640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:58:03.357 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:58:03.357 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":351,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:56:55.310042+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:56:00.217470+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:53:18.149281+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:55:39.674672+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:58:03.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:58:03 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2079550759' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:58:03.847 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9877 8 365281 365273 0.288901 0 - 0.0299513 2026-02-23T15:58:04.265 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:58:03 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:58:03.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:58:04.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:58:03 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:58:03] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:58:04.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:58:04 trial068 ceph-mon[38805]: pgmap v5735: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:58:04.847 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9878 8 365281 365273 0.288872 0 - 0.0299513 2026-02-23T15:58:05.847 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9879 8 365281 365273 0.288843 0 - 0.0299513 2026-02-23T15:58:06.395 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:58:06.514 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:58:06.536 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:58:06 trial068 ceph-mon[38805]: pgmap v5736: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:58:06.612 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:58:06.610+0000 7fbd2be86640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:58:06.612 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:58:06.610+0000 7fbd2be86640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:58:06.721 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:58:06.721 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":351,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:56:55.310042+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:56:00.217470+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:53:18.149281+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:55:39.674672+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:58:06.847 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T15:58:06.847090+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T15:58:06.847 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T15:58:06.847 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9880 8 365281 365273 0.288814 0 - 0.0299513 2026-02-23T15:58:07.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:58:07 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2309315893' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:58:07.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:58:07 trial068 ceph-mon[38805]: 5.8 scrub starts 2026-02-23T15:58:07.847 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9881 8 365281 365273 0.288784 0 - 0.0299513 2026-02-23T15:58:08.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:58:08 trial068 ceph-mon[38805]: 5.8 scrub ok 2026-02-23T15:58:08.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:58:08 trial068 ceph-mon[38805]: pgmap v5737: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:58:08.847 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9882 8 365281 365273 0.288755 0 - 0.0299513 2026-02-23T15:58:09.759 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:58:09.847 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9883 8 365281 365273 0.288726 0 - 0.0299513 2026-02-23T15:58:09.879 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:58:09.975 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:58:09.973+0000 7ff44425d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:58:09.975 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:58:09.973+0000 7ff44425d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:58:10.082 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:58:10.083 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":351,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:56:55.310042+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:56:00.217470+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:53:18.149281+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:55:39.674672+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:58:10.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:58:10 trial068 ceph-mon[38805]: pgmap v5738: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:58:10.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:58:10 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2421034004' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:58:10.847 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9884 8 365281 365273 0.288697 0 - 0.0299513 2026-02-23T15:58:11.847 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9885 8 365281 365273 0.288668 0 - 0.0299513 2026-02-23T15:58:12.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:58:12 trial068 ceph-mon[38805]: pgmap v5739: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:58:12.847 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9886 8 365281 365273 0.288638 0 - 0.0299513 2026-02-23T15:58:13.120 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:58:13.235 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:58:13.333 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:58:13.332+0000 7f3b6cf99640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:58:13.334 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:58:13.332+0000 7f3b6cf99640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:58:13.444 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:58:13.444 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":351,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:56:55.310042+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:56:00.217470+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:53:18.149281+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:55:39.674672+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:58:13.848 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9887 8 365281 365273 0.288609 0 - 0.0299513 2026-02-23T15:58:14.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:58:13 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:58:13] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:58:14.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:58:13 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:58:13.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:58:14.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:58:14 trial068 ceph-mon[38805]: pgmap v5740: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:58:14.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:58:14 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1764415806' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:58:14.848 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9888 8 365281 365273 0.28858 0 - 0.0299513 2026-02-23T15:58:15.848 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9889 8 365281 365273 0.288551 0 - 0.0299513 2026-02-23T15:58:16.482 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:58:16.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:58:16 trial068 ceph-mon[38805]: pgmap v5741: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:58:16.602 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:58:16.703 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:58:16.701+0000 7f97f5157640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:58:16.703 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:58:16.701+0000 7f97f5157640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:58:16.812 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:58:16.812 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":351,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:56:55.310042+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:56:00.217470+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:53:18.149281+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:55:39.674672+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:58:16.848 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9890 8 365281 365273 0.288522 0 - 0.0299513 2026-02-23T15:58:17.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:58:17 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1237034529' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:58:17.848 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9891 8 365281 365273 0.288492 0 - 0.0299513 2026-02-23T15:58:18.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:58:18 trial068 ceph-mon[38805]: pgmap v5742: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:58:18.848 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9892 8 365281 365273 0.288463 0 - 0.0299513 2026-02-23T15:58:19.848 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9893 8 365281 365273 0.288434 0 - 0.0299513 2026-02-23T15:58:19.849 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:58:19.966 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:58:20.064 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:58:20.062+0000 7f288983b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:58:20.064 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:58:20.063+0000 7f288983b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:58:20.174 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:58:20.175 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":351,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:56:55.310042+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:56:00.217470+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:53:18.149281+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:55:39.674672+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:58:20.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:58:20 trial068 ceph-mon[38805]: pgmap v5743: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:58:20.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:58:20 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2112204178' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:58:20.848 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9894 8 365281 365273 0.288405 0 - 0.0299513 2026-02-23T15:58:21.848 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9895 8 365281 365273 0.288376 0 - 0.0299513 2026-02-23T15:58:22.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:58:22 trial068 ceph-mon[38805]: pgmap v5744: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:58:22.848 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9896 8 365281 365273 0.288347 0 - 0.0299513 2026-02-23T15:58:23.212 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:58:23.328 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:58:23.425 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:58:23.423+0000 7f6224cb2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:58:23.426 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:58:23.424+0000 7f6224cb2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:58:23.538 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:58:23.538 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":351,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:56:55.310042+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:56:00.217470+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:53:18.149281+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:55:39.674672+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:58:23.848 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9897 8 365281 365273 0.288318 0 - 0.0299513 2026-02-23T15:58:24.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:58:23 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:58:23] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:58:24.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:58:23 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:58:23.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:58:24.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:58:24 trial068 ceph-mon[38805]: pgmap v5745: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:58:24.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:58:24 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2450638977' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:58:24.848 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9898 8 365281 365273 0.288288 0 - 0.0299513 2026-02-23T15:58:25.849 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9899 8 365281 365273 0.288259 0 - 0.0299513 2026-02-23T15:58:26.577 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:58:26.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:58:26 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T15:58:26.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:58:26 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T15:58:26.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:58:26 trial068 ceph-mon[38805]: pgmap v5746: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:58:26.696 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:58:26.794 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:58:26.793+0000 7f180f93d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:58:26.795 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:58:26.793+0000 7f180f93d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:58:26.848 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T15:58:26.848536+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T15:58:26.849 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T15:58:26.849 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9900 8 365281 365273 0.28823 0 - 0.0299513 2026-02-23T15:58:26.907 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:58:26.907 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":351,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:56:55.310042+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:56:00.217470+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:53:18.149281+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:55:39.674672+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:58:27.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:58:27 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2716859587' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:58:27.849 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9901 8 365281 365273 0.288201 0 - 0.0299513 2026-02-23T15:58:28.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:58:28 trial068 ceph-mon[38805]: pgmap v5747: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:58:28.849 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9902 8 365281 365273 0.288172 0 - 0.0299513 2026-02-23T15:58:29.849 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9903 8 365281 365273 0.288143 0 - 0.0299513 2026-02-23T15:58:29.945 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:58:30.063 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:58:30.162 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:58:30.160+0000 7f8dec27b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:58:30.162 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:58:30.161+0000 7f8dec27b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:58:30.271 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:58:30.272 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":351,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:56:55.310042+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:56:00.217470+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:53:18.149281+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:55:39.674672+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:58:30.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:58:30 trial068 ceph-mon[38805]: pgmap v5748: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:58:30.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:58:30 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3693389804' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:58:30.849 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9904 8 365281 365273 0.288114 0 - 0.0299513 2026-02-23T15:58:31.827 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:58:31 trial068 ceph-mon[38805]: 5.e scrub starts 2026-02-23T15:58:31.827 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:58:31 trial068 ceph-mon[38805]: 5.e scrub ok 2026-02-23T15:58:31.827 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:58:31 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2026-02-23T15:58:31.849 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9905 8 365281 365273 0.288085 0 - 0.0299513 2026-02-23T15:58:32.849 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9906 8 365281 365273 0.288056 0 - 0.0299513 2026-02-23T15:58:33.309 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:58:33.333 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:58:32 trial068 ceph-mon[38805]: pgmap v5749: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:58:33.333 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:58:32 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T15:58:33.333 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:58:32 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T15:58:33.333 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:58:32 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T15:58:33.333 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:58:32 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T15:58:33.333 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:58:32 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-23T15:58:33.334 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:58:32 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2026-02-23T15:58:33.334 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:58:32 trial068 ceph-mon[38805]: pgmap v5750: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:58:33.334 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:58:32 trial068 ceph-mon[38805]: pgmap v5751: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:58:33.334 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:58:32 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T15:58:33.334 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:58:32 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2026-02-23T15:58:33.425 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:58:33.529 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:58:33.527+0000 7fcbcd7e5640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:58:33.529 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:58:33.527+0000 7fcbcd7e5640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:58:33.638 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:58:33.638 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":351,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:56:55.310042+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:56:00.217470+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:53:18.149281+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:55:39.674672+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:58:33.849 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9907 8 365281 365273 0.288027 0 - 0.0299513 2026-02-23T15:58:34.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:58:33 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3556377003' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:58:34.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:58:33 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:58:33] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:58:34.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:58:33 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:58:33.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:58:34.849 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9908 8 365281 365273 0.287997 0 - 0.0299513 2026-02-23T15:58:35.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:58:34 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T15:58:35.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:58:34 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T15:58:35.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:58:34 trial068 ceph-mon[38805]: pgmap v5752: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:58:35.849 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9909 8 365281 365273 0.287968 0 - 0.0299513 2026-02-23T15:58:36.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:58:36 trial068 ceph-mon[38805]: pgmap v5753: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:58:36.675 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:58:36.794 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:58:36.849 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9910 8 365281 365273 0.287939 0 - 0.0299513 2026-02-23T15:58:36.898 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:58:36.896+0000 7f31efcf8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:58:36.898 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:58:36.897+0000 7f31efcf8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:58:37.007 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:58:37.007 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":351,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:56:55.310042+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:56:00.217470+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:53:18.149281+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:55:39.674672+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:58:37.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:58:37 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/126124878' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:58:37.850 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9911 8 365281 365273 0.28791 0 - 0.0299513 2026-02-23T15:58:38.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:58:38 trial068 ceph-mon[38805]: pgmap v5754: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:58:38.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:58:38 trial068 ceph-mon[38805]: 5.a scrub starts 2026-02-23T15:58:38.850 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9912 8 365281 365273 0.287881 0 - 0.0299513 2026-02-23T15:58:39.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:58:39 trial068 ceph-mon[38805]: 5.a scrub ok 2026-02-23T15:58:39.850 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9913 8 365281 365273 0.287852 0 - 0.0299513 2026-02-23T15:58:40.048 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:58:40.165 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:58:40.264 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:58:40.262+0000 7f4aa5814640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:58:40.264 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:58:40.262+0000 7f4aa5814640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:58:40.374 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:58:40.374 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":351,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:56:55.310042+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:56:00.217470+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:53:18.149281+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:55:39.674672+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:58:40.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:58:40 trial068 ceph-mon[38805]: pgmap v5755: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:58:40.850 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9914 8 365281 365273 0.287823 0 - 0.0299513 2026-02-23T15:58:41.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:58:41 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3185334102' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:58:41.850 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9915 8 365281 365273 0.287794 0 - 0.0299513 2026-02-23T15:58:42.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:58:42 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T15:58:42.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:58:42 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T15:58:42.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:58:42 trial068 ceph-mon[38805]: pgmap v5756: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:58:42.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:58:42 trial068 ceph-mon[38805]: 5.5 scrub starts 2026-02-23T15:58:42.850 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9916 8 365281 365273 0.287765 0 - 0.0299513 2026-02-23T15:58:43.412 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:58:43.529 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:58:43.585 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:58:43 trial068 ceph-mon[38805]: 5.5 scrub ok 2026-02-23T15:58:43.628 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:58:43.627+0000 7f22c1c0a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:58:43.628 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:58:43.627+0000 7f22c1c0a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:58:43.737 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:58:43.737 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":351,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:56:55.310042+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:56:00.217470+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:53:18.149281+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:55:39.674672+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:58:43.850 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9917 8 365281 365273 0.287736 0 - 0.0299513 2026-02-23T15:58:44.335 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:58:43 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:58:43] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:58:44.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:58:43 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:58:43.974Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:58:44.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:58:44 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1618253670' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:58:44.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:58:44 trial068 ceph-mon[38805]: pgmap v5757: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:58:44.850 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9918 8 365281 365273 0.287707 0 - 0.0299513 2026-02-23T15:58:45.850 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9919 8 365281 365273 0.287678 0 - 0.0299513 2026-02-23T15:58:46.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:58:46 trial068 ceph-mon[38805]: pgmap v5758: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:58:46.774 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:58:46.850 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T15:58:46.850100+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T15:58:46.850 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T15:58:46.851 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9920 8 365281 365273 0.287649 0 - 0.0299513 2026-02-23T15:58:46.892 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:58:46.993 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:58:46.991+0000 7f98e4f1e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:58:46.993 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:58:46.992+0000 7f98e4f1e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:58:47.102 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:58:47.102 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":351,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:56:55.310042+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:56:00.217470+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:53:18.149281+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:55:39.674672+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:58:47.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:58:47 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2095055747' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:58:47.850 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9921 8 365281 365273 0.28762 0 - 0.0299513 2026-02-23T15:58:48.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:58:48 trial068 ceph-mon[38805]: pgmap v5759: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:58:48.850 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9922 8 365281 365273 0.287591 0 - 0.0299513 2026-02-23T15:58:49.850 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9923 8 365281 365273 0.287562 0 - 0.0299513 2026-02-23T15:58:50.140 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:58:50.258 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:58:50.361 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:58:50.360+0000 7faaeb21a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:58:50.362 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:58:50.360+0000 7faaeb21a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:58:50.470 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:58:50.470 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":351,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:56:55.310042+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:56:00.217470+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:53:18.149281+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:55:39.674672+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:58:50.851 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9924 8 365281 365273 0.287533 0 - 0.0299513 2026-02-23T15:58:51.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:58:51 trial068 ceph-mon[38805]: pgmap v5760: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:58:51.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:58:51 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1542582811' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:58:51.850 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9925 8 365281 365273 0.287504 0 - 0.0299513 2026-02-23T15:58:52.851 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9926 8 365281 365273 0.287475 0 - 0.0299513 2026-02-23T15:58:53.510 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:58:53.534 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:58:53 trial068 ceph-mon[38805]: pgmap v5761: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:58:53.629 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:58:53.734 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:58:53.732+0000 7f1948ea1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:58:53.734 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:58:53.732+0000 7f1948ea1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:58:53.841 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:58:53.841 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":351,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:56:55.310042+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:56:00.217470+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:53:18.149281+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:55:39.674672+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:58:53.851 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9927 8 365281 365273 0.287446 0 - 0.0299513 2026-02-23T15:58:54.094 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:58:53 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:58:53] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:58:54.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:58:53 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:58:53.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:58:54.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:58:54 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2388790117' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:58:54.851 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9928 8 365281 365273 0.287417 0 - 0.0299513 2026-02-23T15:58:55.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:58:55 trial068 ceph-mon[38805]: pgmap v5762: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:58:55.851 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9929 8 365281 365273 0.287388 0 - 0.0299513 2026-02-23T15:58:56.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:58:56 trial068 ceph-mon[38805]: pgmap v5763: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:58:56.851 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9930 8 365281 365273 0.287359 0 - 0.0299513 2026-02-23T15:58:56.880 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:58:56.995 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:58:57.095 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:58:57.093+0000 7f33c0987640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:58:57.096 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:58:57.094+0000 7f33c0987640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:58:57.204 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:58:57.205 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":351,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:56:55.310042+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:56:00.217470+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:53:18.149281+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:55:39.674672+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:58:57.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:58:57 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2704047542' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:58:57.851 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9931 8 365281 365273 0.28733 0 - 0.0299513 2026-02-23T15:58:58.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:58:58 trial068 ceph-mon[38805]: pgmap v5764: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:58:58.851 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9932 8 365281 365273 0.287302 0 - 0.0299513 2026-02-23T15:58:59.851 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9933 8 365281 365273 0.287273 0 - 0.0299513 2026-02-23T15:59:00.242 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:59:00.360 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:59:00.459 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:59:00.457+0000 7fe882658640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:59:00.460 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:59:00.458+0000 7fe882658640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:59:00.570 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:59:00.570 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":351,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:56:55.310042+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:56:00.217470+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:53:18.149281+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:55:39.674672+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:59:00.852 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9934 8 365281 365273 0.287244 0 - 0.0299513 2026-02-23T15:59:01.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:59:01 trial068 ceph-mon[38805]: pgmap v5765: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:59:01.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:59:01 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3315138229' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:59:01.852 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9935 8 365281 365273 0.287215 0 - 0.0299513 2026-02-23T15:59:02.852 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9936 8 365281 365273 0.287186 0 - 0.0299513 2026-02-23T15:59:03.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:59:03 trial068 ceph-mon[38805]: pgmap v5766: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:59:03.607 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:59:03.725 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:59:03.823 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:59:03.821+0000 7fce2e9cc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:59:03.823 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:59:03.821+0000 7fce2e9cc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:59:03.852 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9937 8 365281 365273 0.287157 0 - 0.0299513 2026-02-23T15:59:03.932 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:59:03.932 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":351,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:56:55.310042+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:56:00.217470+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:53:18.149281+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:55:39.674672+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:59:04.265 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:59:03 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:59:03.972Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:59:04.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:59:04 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/944289782' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:59:04.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:59:03 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:59:03] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:59:04.852 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9938 8 365281 365273 0.287128 0 - 0.0299513 2026-02-23T15:59:05.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:59:05 trial068 ceph-mon[38805]: pgmap v5767: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:59:05.852 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9939 8 365281 365273 0.287099 0 - 0.0299513 2026-02-23T15:59:06.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:59:06 trial068 ceph-mon[38805]: pgmap v5768: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:59:06.852 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T15:59:06.851826+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T15:59:06.852 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T15:59:06.852 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9940 8 365281 365273 0.28707 0 - 0.0299513 2026-02-23T15:59:06.970 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:59:07.091 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:59:07.191 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:59:07.189+0000 7fcdaecbf640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:59:07.191 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:59:07.189+0000 7fcdaecbf640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:59:07.299 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:59:07.299 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":351,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:56:55.310042+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:56:00.217470+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:53:18.149281+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:55:39.674672+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:59:07.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:59:07 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3628946527' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:59:07.852 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9941 8 365281 365273 0.287041 0 - 0.0299513 2026-02-23T15:59:08.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:59:08 trial068 ceph-mon[38805]: pgmap v5769: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:59:08.852 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9942 8 365281 365273 0.287013 0 - 0.0299513 2026-02-23T15:59:09.852 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9943 8 365281 365273 0.286984 0 - 0.0299513 2026-02-23T15:59:10.337 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:59:10.453 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:59:10.550 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:59:10.548+0000 7fd9ae08a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:59:10.550 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:59:10.548+0000 7fd9ae08a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:59:10.659 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:59:10.659 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":351,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:56:55.310042+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:56:00.217470+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:53:18.149281+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:55:39.674672+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:59:10.852 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9944 8 365281 365273 0.286955 0 - 0.0299513 2026-02-23T15:59:11.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:59:11 trial068 ceph-mon[38805]: pgmap v5770: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:59:11.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:59:11 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2302644282' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:59:11.852 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9945 8 365281 365273 0.286926 0 - 0.0299513 2026-02-23T15:59:12.852 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9946 8 365281 365273 0.286897 0 - 0.0299513 2026-02-23T15:59:13.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:59:13 trial068 ceph-mon[38805]: pgmap v5771: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:59:13.700 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:59:13.818 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:59:13.853 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9947 8 365281 365273 0.286868 0 - 0.0299513 2026-02-23T15:59:13.920 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:59:13.918+0000 7f38349f7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:59:13.920 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:59:13.918+0000 7f38349f7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:59:14.029 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:59:14.029 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":351,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:56:55.310042+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:56:00.217470+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:53:18.149281+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:55:39.674672+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:59:14.030 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:59:13 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:59:13] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:59:14.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:59:14 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/960769310' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:59:14.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:59:13 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:59:13.972Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:59:14.852 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9948 8 365281 365273 0.286839 0 - 0.0299513 2026-02-23T15:59:15.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:59:15 trial068 ceph-mon[38805]: pgmap v5772: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:59:15.853 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9949 8 365281 365273 0.286811 0 - 0.0299513 2026-02-23T15:59:16.853 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9950 8 365281 365273 0.286782 0 - 0.0299513 2026-02-23T15:59:17.066 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:59:17.186 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:59:17.285 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:59:17.283+0000 7f08eaa23640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:59:17.285 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:59:17.284+0000 7f08eaa23640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:59:17.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:59:17 trial068 ceph-mon[38805]: pgmap v5773: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:59:17.396 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:59:17.396 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":351,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:56:55.310042+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:56:00.217470+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:53:18.149281+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:55:39.674672+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:59:17.853 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9951 8 365281 365273 0.286753 0 - 0.0299513 2026-02-23T15:59:18.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:59:18 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3083219838' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:59:18.853 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9952 8 365281 365273 0.286724 0 - 0.0299513 2026-02-23T15:59:19.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:59:19 trial068 ceph-mon[38805]: pgmap v5774: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:59:19.853 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9953 8 365281 365273 0.286695 0 - 0.0299513 2026-02-23T15:59:20.434 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:59:20.551 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:59:20.650 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:59:20.648+0000 7faa9d114640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:59:20.650 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:59:20.648+0000 7faa9d114640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:59:20.758 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:59:20.758 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":351,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:56:55.310042+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:56:00.217470+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:53:18.149281+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:55:39.674672+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:59:20.853 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9954 8 365281 365273 0.286667 0 - 0.0299513 2026-02-23T15:59:21.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:59:21 trial068 ceph-mon[38805]: pgmap v5775: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:59:21.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:59:21 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/4140599052' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:59:21.853 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9955 8 365281 365273 0.286638 0 - 0.0299513 2026-02-23T15:59:22.853 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9956 8 365281 365273 0.286609 0 - 0.0299513 2026-02-23T15:59:23.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:59:23 trial068 ceph-mon[38805]: pgmap v5776: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:59:23.796 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:59:23.853 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9957 8 365281 365273 0.28658 0 - 0.0299513 2026-02-23T15:59:23.913 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:59:24.015 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:59:24.013+0000 7f679a7f2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:59:24.016 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:59:24.014+0000 7f679a7f2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:59:24.093 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:59:23 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:59:23] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:59:24.125 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:59:24.126 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":351,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:56:55.310042+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:56:00.217470+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:53:18.149281+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:55:39.674672+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:59:24.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:59:23 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:59:23.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:59:24.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:59:24 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3328086835' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:59:24.853 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9958 8 365281 365273 0.286551 0 - 0.0299513 2026-02-23T15:59:25.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:59:25 trial068 ceph-mon[38805]: pgmap v5777: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:59:25.853 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9959 8 365281 365273 0.286523 0 - 0.0299513 2026-02-23T15:59:26.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:59:26 trial068 ceph-mon[38805]: pgmap v5778: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:59:26.853 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T15:59:26.853228+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T15:59:26.854 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T15:59:26.854 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9960 8 365281 365273 0.286494 0 - 0.0299513 2026-02-23T15:59:27.164 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:59:27.283 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:59:27.380 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:59:27.379+0000 7f66df83e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:59:27.381 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:59:27.379+0000 7f66df83e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:59:27.491 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:59:27.491 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":351,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:56:55.310042+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:56:00.217470+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:53:18.149281+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:55:39.674672+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:59:27.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:59:27 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1141848095' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:59:27.853 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9961 8 365281 365273 0.286465 0 - 0.0299513 2026-02-23T15:59:28.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:59:28 trial068 ceph-mon[38805]: pgmap v5779: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:59:28.853 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9962 8 365281 365273 0.286436 0 - 0.0299513 2026-02-23T15:59:29.854 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9963 8 365281 365273 0.286408 0 - 0.0299513 2026-02-23T15:59:30.549 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:59:30.668 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:59:30.768 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:59:30.766+0000 7ff84edc0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:59:30.768 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:59:30.766+0000 7ff84edc0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:59:30.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:59:30 trial068 ceph-mon[38805]: osdmap e352: 4 total, 3 up, 4 in 2026-02-23T15:59:30.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:59:30 trial068 ceph-mon[38805]: pgmap v5781: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:59:30.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:59:30 trial068 ceph-mon[38805]: osdmap e353: 4 total, 3 up, 4 in 2026-02-23T15:59:30.853 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9964 8 365281 365273 0.286379 0 - 0.0299513 2026-02-23T15:59:30.877 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:59:30.877 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":353,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:59:30.329891+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:56:00.217470+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:58:33.486402+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:55:39.674672+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:59:31.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:59:31 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2470384475' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:59:31.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:59:31 trial068 ceph-mon[38805]: 5.7 scrub starts 2026-02-23T15:59:31.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:59:31 trial068 ceph-mon[38805]: 5.7 scrub ok 2026-02-23T15:59:31.854 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9965 8 365281 365273 0.28635 0 - 0.0299513 2026-02-23T15:59:32.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:59:32 trial068 ceph-mon[38805]: pgmap v5783: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:59:32.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:59:32 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2026-02-23T15:59:32.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:59:32 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-23T15:59:32.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:59:32 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2026-02-23T15:59:32.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:59:32 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T15:59:32.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:59:32 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2026-02-23T15:59:32.854 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9966 8 365281 365273 0.286321 0 - 0.0299513 2026-02-23T15:59:33.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:59:33 trial068 ceph-mon[38805]: pgmap v5784: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:59:33.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:59:33 trial068 ceph-mon[38805]: pgmap v5785: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:59:33.854 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9967 8 365281 365273 0.286293 0 - 0.0299513 2026-02-23T15:59:33.913 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:59:34.028 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:59:34.129 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:59:34.127+0000 7fa484a1c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:59:34.129 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:59:34.127+0000 7fa484a1c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:59:34.238 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:59:34.238 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":353,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:59:30.329891+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:56:00.217470+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:58:33.486402+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:55:39.674672+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:59:34.238 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:59:33 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:59:33] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:59:34.265 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:59:33 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:59:33.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:59:34.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:59:34 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/231748085' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:59:34.854 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9968 8 365281 365273 0.286264 0 - 0.0299513 2026-02-23T15:59:35.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:59:35 trial068 ceph-mon[38805]: pgmap v5786: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:59:35.854 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9969 8 365281 365273 0.286235 0 - 0.0299513 2026-02-23T15:59:36.854 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9970 8 365281 365273 0.286207 0 - 0.0299513 2026-02-23T15:59:37.275 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:59:37.392 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:59:37.493 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:59:37.491+0000 7fd9bca2b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:59:37.493 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:59:37.491+0000 7fd9bca2b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:59:37.602 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:59:37.602 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":353,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:59:30.329891+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:56:00.217470+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:58:33.486402+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:55:39.674672+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:59:37.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:59:37 trial068 ceph-mon[38805]: pgmap v5787: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:59:37.854 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9971 8 365281 365273 0.286178 0 - 0.0299513 2026-02-23T15:59:38.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:59:38 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2502205369' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:59:38.854 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9972 8 365281 365273 0.286149 0 - 0.0299513 2026-02-23T15:59:39.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:59:39 trial068 ceph-mon[38805]: pgmap v5788: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:59:39.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:59:39 trial068 ceph-mon[38805]: 5.8 scrub starts 2026-02-23T15:59:39.854 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9973 8 365281 365273 0.28612 0 - 0.0299513 2026-02-23T15:59:40.643 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:59:40.760 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:59:40.811 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:59:40 trial068 ceph-mon[38805]: 5.8 scrub ok 2026-02-23T15:59:40.854 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9974 8 365281 365273 0.286092 0 - 0.0299513 2026-02-23T15:59:40.857 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:59:40.855+0000 7fbbe4ccb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:59:40.857 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:59:40.855+0000 7fbbe4ccb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:59:40.971 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:59:40.971 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":353,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:59:30.329891+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:56:00.217470+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:58:33.486402+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:55:39.674672+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:59:41.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:59:41 trial068 ceph-mon[38805]: pgmap v5789: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:59:41.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:59:41 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2211695674' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:59:41.854 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9975 8 365281 365273 0.286063 0 - 0.0299513 2026-02-23T15:59:42.855 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9976 8 365281 365273 0.286034 0 - 0.0299513 2026-02-23T15:59:43.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:59:43 trial068 ceph-mon[38805]: pgmap v5790: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:59:43.854 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9977 8 365281 365273 0.286006 0 - 0.0299513 2026-02-23T15:59:44.009 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:59:44.142 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:59:44.244 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:59:44.242+0000 7f8fb3bdd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:59:44.244 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:59:44.242+0000 7f8fb3bdd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:59:44.244 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:59:43 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:59:43] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:59:44.353 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:59:44.353 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":353,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:59:30.329891+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:56:00.217470+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:58:33.486402+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:55:39.674672+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:59:44.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:59:43 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:59:43.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:59:44.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:59:44 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T15:59:44.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:59:44 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T15:59:44.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:59:44 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/529804359' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:59:44.855 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9978 8 365281 365273 0.285977 0 - 0.0299513 2026-02-23T15:59:45.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:59:45 trial068 ceph-mon[38805]: pgmap v5791: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:59:45.855 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9979 8 365281 365273 0.285948 0 - 0.0299513 2026-02-23T15:59:46.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:59:46 trial068 ceph-mon[38805]: 5.a scrub starts 2026-02-23T15:59:46.855 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T15:59:46.854668+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T15:59:46.855 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T15:59:46.855 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9980 8 365281 365273 0.28592 0 - 0.0299513 2026-02-23T15:59:47.389 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:59:47.510 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:59:47.613 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:59:47.611+0000 7feb350d3640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:59:47.613 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:59:47.611+0000 7feb350d3640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:59:47.722 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:59:47.723 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":353,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:59:30.329891+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:56:00.217470+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:58:33.486402+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:55:39.674672+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:59:47.723 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:59:47 trial068 ceph-mon[38805]: 5.a scrub ok 2026-02-23T15:59:47.723 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:59:47 trial068 ceph-mon[38805]: pgmap v5792: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:59:47.855 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9981 8 365281 365273 0.285891 0 - 0.0299513 2026-02-23T15:59:48.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:59:48 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/854233251' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:59:48.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:59:48 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T15:59:48.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:59:48 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T15:59:48.855 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9982 8 365281 365273 0.285862 0 - 0.0299513 2026-02-23T15:59:49.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:59:49 trial068 ceph-mon[38805]: pgmap v5793: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:59:49.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:59:49 trial068 ceph-mon[38805]: 5.e scrub starts 2026-02-23T15:59:49.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:59:49 trial068 ceph-mon[38805]: 5.e scrub ok 2026-02-23T15:59:49.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:59:49 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T15:59:49.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:59:49 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T15:59:49.855 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9983 8 365281 365273 0.285834 0 - 0.0299513 2026-02-23T15:59:50.760 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:59:50.855 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9984 8 365281 365273 0.285805 0 - 0.0299513 2026-02-23T15:59:50.879 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:59:50.978 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:59:50.976+0000 7f5f37cbb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:59:50.978 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:59:50.976+0000 7f5f37cbb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:59:51.086 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:59:51.087 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":353,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:59:30.329891+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:56:00.217470+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:58:33.486402+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:55:39.674672+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:59:51.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:59:51 trial068 ceph-mon[38805]: pgmap v5794: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:59:51.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:59:51 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3879420358' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:59:51.855 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9985 8 365281 365273 0.285777 0 - 0.0299513 2026-02-23T15:59:52.855 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9986 8 365281 365273 0.285748 0 - 0.0299513 2026-02-23T15:59:53.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:59:53 trial068 ceph-mon[38805]: pgmap v5795: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:59:53.855 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9987 8 365281 365273 0.285719 0 - 0.0299513 2026-02-23T15:59:54.123 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:59:54.242 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:59:54.264 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 15:59:53 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:15:59:53] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T15:59:54.349 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:59:54.347+0000 7fade5835640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:59:54.349 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:59:54.347+0000 7fade5835640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:59:54.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 15:59:53 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T15:59:53.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T15:59:54.457 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:59:54.458 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":353,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:59:30.329891+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:56:00.217470+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:58:33.486402+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:55:39.674672+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:59:54.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:59:54 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/970066969' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:59:54.855 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9988 8 365281 365273 0.285691 0 - 0.0299513 2026-02-23T15:59:55.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:59:55 trial068 ceph-mon[38805]: pgmap v5796: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:59:55.855 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9989 8 365281 365273 0.285662 0 - 0.0299513 2026-02-23T15:59:56.855 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9990 8 365281 365273 0.285634 0 - 0.0299513 2026-02-23T15:59:57.494 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T15:59:57.613 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T15:59:57.711 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:59:57.709+0000 7f711029a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:59:57.711 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T15:59:57.709+0000 7f711029a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T15:59:57.819 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T15:59:57.820 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":353,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:59:30.329891+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:56:00.217470+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:58:33.486402+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:55:39.674672+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T15:59:57.820 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:59:57 trial068 ceph-mon[38805]: pgmap v5797: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:59:57.856 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9991 8 365281 365273 0.285605 0 - 0.0299513 2026-02-23T15:59:58.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:59:58 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/4090262338' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T15:59:58.856 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9992 8 365281 365273 0.285576 0 - 0.0299513 2026-02-23T15:59:59.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 15:59:59 trial068 ceph-mon[38805]: pgmap v5798: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T15:59:59.856 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9993 8 365281 365273 0.285548 0 - 0.0299513 2026-02-23T16:00:00.856 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9994 8 365281 365273 0.285519 0 - 0.0299513 2026-02-23T16:00:00.858 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:00:00.979 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:00:01.080 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:00:01.078+0000 7fd6d85c1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:00:01.080 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:00:01.078+0000 7fd6d85c1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:00:01.188 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:00:01.188 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":353,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:59:30.329891+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:56:00.217470+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:58:33.486402+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:55:39.674672+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:00:01.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:00:01 trial068 ceph-mon[38805]: pgmap v5799: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:00:01.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:00:01 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/24149589' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:00:01.856 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9995 8 365281 365273 0.285491 0 - 0.0299513 2026-02-23T16:00:02.856 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9996 8 365281 365273 0.285462 0 - 0.0299513 2026-02-23T16:00:03.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:00:03 trial068 ceph-mon[38805]: pgmap v5800: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:00:03.856 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9997 8 365281 365273 0.285434 0 - 0.0299513 2026-02-23T16:00:04.226 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:00:04.252 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 16:00:03 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:16:00:03] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T16:00:04.265 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 16:00:03 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T16:00:03.974Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T16:00:04.345 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:00:04.448 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:00:04.446+0000 7fd5216a1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:00:04.448 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:00:04.446+0000 7fd5216a1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:00:04.556 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:00:04.556 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":353,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:59:30.329891+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:56:00.217470+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:58:33.486402+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:55:39.674672+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:00:04.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:00:04 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1401000759' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:00:04.856 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9998 8 365281 365273 0.285405 0 - 0.0299513 2026-02-23T16:00:05.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:00:05 trial068 ceph-mon[38805]: pgmap v5801: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:00:05.856 INFO:tasks.radosbench.radosbench.0.trial068.stdout: 9999 8 365281 365273 0.285376 0 - 0.0299513 2026-02-23T16:00:06.856 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T16:00:06.856090+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T16:00:06.856 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T16:00:06.857 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10000 8 365281 365273 0.285348 0 - 0.0299513 2026-02-23T16:00:07.617 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:00:07.734 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:00:07.832 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:00:07.830+0000 7f702b503640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:00:07.832 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:00:07.831+0000 7f702b503640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:00:07.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:00:07 trial068 ceph-mon[38805]: pgmap v5802: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:00:07.856 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10001 8 365281 365273 0.285319 0 - 0.0299513 2026-02-23T16:00:07.941 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:00:07.941 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":353,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:59:30.329891+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:56:00.217470+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:58:33.486402+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:55:39.674672+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:00:08.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:00:08 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1804952684' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:00:08.856 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10002 8 365281 365273 0.285291 0 - 0.0299513 2026-02-23T16:00:09.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:00:09 trial068 ceph-mon[38805]: pgmap v5803: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:00:09.856 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10003 8 365281 365273 0.285262 0 - 0.0299513 2026-02-23T16:00:10.857 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10004 8 365281 365273 0.285234 0 - 0.0299513 2026-02-23T16:00:10.979 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:00:11.097 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:00:11.199 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:00:11.197+0000 7f0a0c1f8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:00:11.199 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:00:11.197+0000 7f0a0c1f8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:00:11.308 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:00:11.308 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":353,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:59:30.329891+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:56:00.217470+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:58:33.486402+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:55:39.674672+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:00:11.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:00:11 trial068 ceph-mon[38805]: pgmap v5804: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:00:11.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:00:11 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2640058817' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:00:11.857 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10005 8 365281 365273 0.285205 0 - 0.0299513 2026-02-23T16:00:12.857 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10006 8 365281 365273 0.285177 0 - 0.0299513 2026-02-23T16:00:13.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:00:13 trial068 ceph-mon[38805]: pgmap v5805: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:00:13.857 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10007 8 365281 365273 0.285148 0 - 0.0299513 2026-02-23T16:00:14.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 16:00:13 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:16:00:13] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T16:00:14.346 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:00:14.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 16:00:13 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T16:00:13.974Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T16:00:14.461 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:00:14.559 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:00:14.557+0000 7f290f325640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:00:14.559 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:00:14.557+0000 7f290f325640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:00:14.670 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:00:14.671 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":353,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:59:30.329891+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:56:00.217470+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:58:33.486402+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:55:39.674672+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:00:14.857 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10008 8 365281 365273 0.28512 0 - 0.0299513 2026-02-23T16:00:15.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:00:15 trial068 ceph-mon[38805]: pgmap v5806: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:00:15.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:00:15 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/661944001' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:00:15.857 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10009 8 365281 365273 0.285091 0 - 0.0299513 2026-02-23T16:00:16.857 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10010 8 365281 365273 0.285063 0 - 0.0299513 2026-02-23T16:00:17.707 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:00:17.731 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:00:17 trial068 ceph-mon[38805]: pgmap v5807: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:00:17.826 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:00:17.857 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10011 8 365281 365273 0.285034 0 - 0.0299513 2026-02-23T16:00:17.924 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:00:17.923+0000 7fc1400e5640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:00:17.925 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:00:17.923+0000 7fc1400e5640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:00:18.033 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:00:18.033 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":353,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:59:30.329891+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:56:00.217470+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:58:33.486402+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:55:39.674672+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:00:18.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:00:18 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1995515403' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:00:18.857 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10012 8 365281 365273 0.285006 0 - 0.0299513 2026-02-23T16:00:19.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:00:19 trial068 ceph-mon[38805]: pgmap v5808: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:00:19.857 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10013 8 365281 365273 0.284977 0 - 0.0299513 2026-02-23T16:00:20.857 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10014 8 365281 365273 0.284949 0 - 0.0299513 2026-02-23T16:00:21.070 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:00:21.189 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:00:21.291 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:00:21.290+0000 7feea0966640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:00:21.291 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:00:21.290+0000 7feea0966640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:00:21.403 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:00:21.404 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":353,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:59:30.329891+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:56:00.217470+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:58:33.486402+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:55:39.674672+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:00:21.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:00:21 trial068 ceph-mon[38805]: pgmap v5809: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:00:21.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:00:21 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/4246864954' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:00:21.857 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10015 8 365281 365273 0.284921 0 - 0.0299513 2026-02-23T16:00:22.858 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10016 8 365281 365273 0.284892 0 - 0.0299513 2026-02-23T16:00:23.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:00:23 trial068 ceph-mon[38805]: pgmap v5810: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:00:23.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:00:23 trial068 ceph-mon[38805]: 5.5 scrub starts 2026-02-23T16:00:23.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:00:23 trial068 ceph-mon[38805]: 5.5 scrub ok 2026-02-23T16:00:23.858 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10017 8 365281 365273 0.284864 0 - 0.0299513 2026-02-23T16:00:24.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 16:00:23 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:16:00:23] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T16:00:24.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 16:00:23 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T16:00:23.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T16:00:24.441 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:00:24.561 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:00:24.658 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:00:24.656+0000 7fb4dd2b9640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:00:24.658 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:00:24.656+0000 7fb4dd2b9640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:00:24.767 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:00:24.767 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":353,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:59:30.329891+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:56:00.217470+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:58:33.486402+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:55:39.674672+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:00:24.858 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10018 8 365281 365273 0.284835 0 - 0.0299513 2026-02-23T16:00:25.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:00:25 trial068 ceph-mon[38805]: pgmap v5811: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:00:25.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:00:25 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1466140806' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:00:25.858 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10019 8 365281 365273 0.284807 0 - 0.0299513 2026-02-23T16:00:26.858 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T16:00:26.857707+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T16:00:26.858 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T16:00:26.858 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10020 8 365281 365273 0.284778 0 - 0.0299513 2026-02-23T16:00:27.805 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:00:27.829 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:00:27 trial068 ceph-mon[38805]: pgmap v5812: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:00:27.858 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10021 8 365281 365273 0.28475 0 - 0.0299513 2026-02-23T16:00:27.923 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:00:28.020 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:00:28.018+0000 7fb7aa081640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:00:28.020 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:00:28.018+0000 7fb7aa081640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:00:28.128 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:00:28.128 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":353,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:59:30.329891+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:56:00.217470+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:58:33.486402+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:55:39.674672+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:00:28.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:00:28 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/4156963261' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:00:28.858 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10022 8 365281 365273 0.284722 0 - 0.0299513 2026-02-23T16:00:29.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:00:29 trial068 ceph-mon[38805]: pgmap v5813: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:00:29.858 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10023 8 365281 365273 0.284693 0 - 0.0299513 2026-02-23T16:00:30.858 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10024 8 365281 365273 0.284665 0 - 0.0299513 2026-02-23T16:00:31.167 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:00:31.286 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:00:31.387 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:00:31.385+0000 7f9add8c7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:00:31.387 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:00:31.385+0000 7f9add8c7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:00:31.495 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:00:31.496 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":353,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:59:30.329891+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:56:00.217470+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:58:33.486402+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:55:39.674672+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:00:31.496 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:00:31 trial068 ceph-mon[38805]: pgmap v5814: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:00:31.858 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10025 8 365281 365273 0.284636 0 - 0.0299513 2026-02-23T16:00:32.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:00:32 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3989952985' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:00:32.858 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10026 8 365281 365273 0.284608 0 - 0.0299513 2026-02-23T16:00:33.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:00:33 trial068 ceph-mon[38805]: pgmap v5815: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:00:33.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:00:33 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2026-02-23T16:00:33.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:00:33 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-23T16:00:33.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:00:33 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2026-02-23T16:00:33.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:00:33 trial068 ceph-mon[38805]: pgmap v5816: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:00:33.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:00:33 trial068 ceph-mon[38805]: pgmap v5817: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:00:33.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:00:33 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T16:00:33.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:00:33 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2026-02-23T16:00:33.858 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10027 8 365281 365273 0.28458 0 - 0.0299513 2026-02-23T16:00:34.264 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 16:00:33 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T16:00:33.974Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T16:00:34.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 16:00:33 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:16:00:33] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T16:00:34.533 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:00:34.651 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:00:34.751 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:00:34.749+0000 7fbd0dd98640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:00:34.751 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:00:34.749+0000 7fbd0dd98640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:00:34.858 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10028 8 365281 365273 0.284551 0 - 0.0299513 2026-02-23T16:00:34.860 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:00:34.861 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":353,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:59:30.329891+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:56:00.217470+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:58:33.486402+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:55:39.674672+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:00:35.859 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10029 8 365281 365273 0.284523 0 - 0.0299513 2026-02-23T16:00:36.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:00:35 trial068 ceph-mon[38805]: pgmap v5818: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:00:36.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:00:35 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/182749264' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:00:36.859 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10030 8 365281 365273 0.284494 0 - 0.0299513 2026-02-23T16:00:37.859 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10031 8 365281 365273 0.284466 0 - 0.0299513 2026-02-23T16:00:37.898 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:00:38.015 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:00:38.037 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:00:37 trial068 ceph-mon[38805]: pgmap v5819: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:00:38.117 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:00:38.115+0000 7f815eeef640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:00:38.117 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:00:38.116+0000 7f815eeef640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:00:38.228 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:00:38.229 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":353,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:59:30.329891+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:56:00.217470+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:58:33.486402+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:55:39.674672+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:00:38.859 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10032 8 365281 365273 0.284438 0 - 0.0299513 2026-02-23T16:00:39.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:00:38 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3837755328' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:00:39.859 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10033 8 365281 365273 0.284409 0 - 0.0299513 2026-02-23T16:00:40.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:00:39 trial068 ceph-mon[38805]: pgmap v5820: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:00:40.859 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10034 8 365281 365273 0.284381 0 - 0.0299513 2026-02-23T16:00:41.266 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:00:41.384 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:00:41.482 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:00:41.480+0000 7fd5ae67f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:00:41.482 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:00:41.480+0000 7fd5ae67f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:00:41.590 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:00:41.590 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":353,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:59:30.329891+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:56:00.217470+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:58:33.486402+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:55:39.674672+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:00:41.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:00:41 trial068 ceph-mon[38805]: pgmap v5821: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:00:41.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:00:41 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2244123677' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:00:41.859 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10035 8 365281 365273 0.284353 0 - 0.0299513 2026-02-23T16:00:42.859 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10036 8 365281 365273 0.284324 0 - 0.0299513 2026-02-23T16:00:43.859 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10037 8 365281 365273 0.284296 0 - 0.0299513 2026-02-23T16:00:43.974 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:00:43 trial068 ceph-mon[38805]: pgmap v5822: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:00:44.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 16:00:43 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:16:00:43] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T16:00:44.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 16:00:43 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T16:00:43.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T16:00:44.627 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:00:44.744 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:00:44.844 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:00:44.842+0000 7fd7be1a9640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:00:44.845 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:00:44.843+0000 7fd7be1a9640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:00:44.859 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10038 8 365281 365273 0.284268 0 - 0.0299513 2026-02-23T16:00:44.952 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:00:44.953 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":353,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T15:59:30.329891+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:56:00.217470+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:58:33.486402+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:55:39.674672+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:00:45.860 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10039 8 365281 365273 0.284239 0 - 0.0299513 2026-02-23T16:00:46.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:00:45 trial068 ceph-mon[38805]: pgmap v5823: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:00:46.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:00:45 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/176203765' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:00:46.860 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T16:00:46.859570+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T16:00:46.860 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T16:00:46.860 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10040 8 365281 365273 0.284211 0 - 0.0299513 2026-02-23T16:00:47.860 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10041 8 365281 365273 0.284183 0 - 0.0299513 2026-02-23T16:00:47.991 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:00:48.015 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:00:47 trial068 ceph-mon[38805]: pgmap v5824: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:00:48.016 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:00:47 trial068 ceph-mon[38805]: osdmap e354: 4 total, 3 up, 4 in 2026-02-23T16:00:48.108 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:00:48.212 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:00:48.210+0000 7f7135ffc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:00:48.212 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:00:48.210+0000 7f7135ffc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:00:48.321 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:00:48.321 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":354,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:00:46.703460+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:56:00.217470+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:58:33.486402+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:59:49.303087+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:00:48.860 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10042 8 365281 365273 0.284154 0 - 0.0299513 2026-02-23T16:00:49.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:00:48 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1934296' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:00:49.860 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10043 8 365281 365273 0.284126 0 - 0.0299513 2026-02-23T16:00:50.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:00:49 trial068 ceph-mon[38805]: pgmap v5826: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:00:50.860 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10044 8 365281 365273 0.284098 0 - 0.0299513 2026-02-23T16:00:51.358 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:00:51.474 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:00:51.577 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:00:51.575+0000 7f46754e6640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:00:51.577 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:00:51.575+0000 7f46754e6640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:00:51.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:00:51 trial068 ceph-mon[38805]: osdmap e355: 4 total, 3 up, 4 in 2026-02-23T16:00:51.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:00:51 trial068 ceph-mon[38805]: pgmap v5828: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:00:51.689 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:00:51.689 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":355,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:00:50.340652+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:56:00.217470+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:58:33.486402+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:59:49.303087+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:00:51.860 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10045 8 365281 365273 0.28407 0 - 0.0299513 2026-02-23T16:00:52.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:00:52 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/976530303' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:00:52.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:00:52 trial068 ceph-mon[38805]: 5.8 scrub starts 2026-02-23T16:00:52.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:00:52 trial068 ceph-mon[38805]: 5.8 scrub ok 2026-02-23T16:00:52.860 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10046 8 365281 365273 0.284041 0 - 0.0299513 2026-02-23T16:00:53.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:00:53 trial068 ceph-mon[38805]: pgmap v5829: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:00:53.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:00:53 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T16:00:53.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:00:53 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T16:00:53.860 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10047 8 365281 365273 0.284013 0 - 0.0299513 2026-02-23T16:00:54.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 16:00:53 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:16:00:53] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T16:00:54.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 16:00:53 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T16:00:53.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T16:00:54.726 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:00:54.842 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:00:54.860 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10048 8 365281 365273 0.283985 0 - 0.0299513 2026-02-23T16:00:54.942 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:00:54.940+0000 7f998b16d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:00:54.942 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:00:54.941+0000 7f998b16d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:00:55.052 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:00:55.052 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":355,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:00:50.340652+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:56:00.217470+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:58:33.486402+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:59:49.303087+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:00:55.860 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10049 8 365281 365273 0.283956 0 - 0.0299513 2026-02-23T16:00:56.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:00:55 trial068 ceph-mon[38805]: pgmap v5830: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:00:56.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:00:55 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/472153046' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:00:56.860 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10050 8 365281 365273 0.283928 0 - 0.0299513 2026-02-23T16:00:57.860 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10051 8 365281 365273 0.2839 0 - 0.0299513 2026-02-23T16:00:58.091 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:00:58.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:00:57 trial068 ceph-mon[38805]: pgmap v5831: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:00:58.210 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:00:58.306 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:00:58.305+0000 7fc75f2bd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:00:58.307 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:00:58.305+0000 7fc75f2bd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:00:58.416 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:00:58.416 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":355,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:00:50.340652+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:56:00.217470+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:58:33.486402+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:59:49.303087+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:00:58.861 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10052 8 365281 365273 0.283872 0 - 0.0299513 2026-02-23T16:00:59.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:00:58 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/946961571' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:00:59.861 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10053 8 365281 365273 0.283844 0 - 0.0299513 2026-02-23T16:01:00.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:00:59 trial068 ceph-mon[38805]: pgmap v5832: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:01:00.861 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10054 8 365281 365273 0.283815 0 - 0.0299513 2026-02-23T16:01:01.454 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:01:01.574 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:01:01.671 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:01:01.669+0000 7fc4f9009640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:01:01.671 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:01:01.669+0000 7fc4f9009640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:01:01.780 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:01:01.780 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":355,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:00:50.340652+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:56:00.217470+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:58:33.486402+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:59:49.303087+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:01:01.780 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:01:01 trial068 ceph-mon[38805]: pgmap v5833: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:01:01.780 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:01:01 trial068 ceph-mon[38805]: 5.7 scrub starts 2026-02-23T16:01:01.861 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10055 8 365281 365273 0.283787 0 - 0.0299513 2026-02-23T16:01:02.861 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10056 8 365281 365273 0.283759 0 - 0.0299513 2026-02-23T16:01:03.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:01:02 trial068 ceph-mon[38805]: 5.7 scrub ok 2026-02-23T16:01:03.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:01:02 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1960908394' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:01:03.861 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10057 8 365281 365273 0.283731 0 - 0.0299513 2026-02-23T16:01:03.973 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:01:03 trial068 ceph-mon[38805]: pgmap v5834: 17 pgs: 1 active+clean+scrubbing, 1 active+recovery_unfound+undersized+degraded, 4 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:01:04.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 16:01:03 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:16:01:03] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T16:01:04.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 16:01:03 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T16:01:03.974Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T16:01:04.836 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:01:04.861 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10058 8 365281 365273 0.283702 0 - 0.0299513 2026-02-23T16:01:04.955 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:01:04.976 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:01:04 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T16:01:04.977 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:01:04 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T16:01:05.053 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:01:05.051+0000 7f786f6f1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:01:05.053 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:01:05.052+0000 7f786f6f1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:01:05.163 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:01:05.163 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":355,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:00:50.340652+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:56:00.217470+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:58:33.486402+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:59:49.303087+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:01:05.861 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10059 8 365281 365273 0.283674 0 - 0.0299513 2026-02-23T16:01:06.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:01:05 trial068 ceph-mon[38805]: pgmap v5835: 17 pgs: 1 active+clean+scrubbing, 1 active+recovery_unfound+undersized+degraded, 4 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:01:06.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:01:05 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T16:01:06.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:01:05 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T16:01:06.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:01:05 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1090513946' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:01:06.861 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T16:01:06.860953+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T16:01:06.861 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T16:01:06.861 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10060 8 365281 365273 0.283646 0 - 0.0299513 2026-02-23T16:01:07.861 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10061 8 365281 365273 0.283618 0 - 0.0299513 2026-02-23T16:01:08.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:01:07 trial068 ceph-mon[38805]: pgmap v5836: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:01:08.201 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:01:08.318 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:01:08.422 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:01:08.420+0000 7fae778a0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:01:08.422 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:01:08.421+0000 7fae778a0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:01:08.532 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:01:08.532 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":355,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:00:50.340652+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:56:00.217470+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:58:33.486402+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:59:49.303087+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:01:08.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:01:08 trial068 ceph-mon[38805]: 5.a scrub starts 2026-02-23T16:01:08.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:01:08 trial068 ceph-mon[38805]: 5.a scrub ok 2026-02-23T16:01:08.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:01:08 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1265211674' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:01:08.861 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10062 8 365281 365273 0.28359 0 - 0.0299513 2026-02-23T16:01:09.861 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10063 8 365281 365273 0.283561 0 - 0.0299513 2026-02-23T16:01:10.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:01:09 trial068 ceph-mon[38805]: pgmap v5837: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:01:10.861 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10064 8 365281 365273 0.283533 0 - 0.0299513 2026-02-23T16:01:11.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:01:10 trial068 ceph-mon[38805]: 5.e scrub starts 2026-02-23T16:01:11.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:01:10 trial068 ceph-mon[38805]: 5.e scrub ok 2026-02-23T16:01:11.568 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:01:11.685 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:01:11.791 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:01:11.789+0000 7f2c12c6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:01:11.791 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:01:11.789+0000 7f2c12c6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:01:11.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:01:11 trial068 ceph-mon[38805]: pgmap v5838: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:01:11.861 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10065 8 365281 365273 0.283505 0 - 0.0299513 2026-02-23T16:01:11.904 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:01:11.904 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":355,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:00:50.340652+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:56:00.217470+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:58:33.486402+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:59:49.303087+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:01:12.862 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10066 8 365281 365273 0.283477 0 - 0.0299513 2026-02-23T16:01:13.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:01:12 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1863557912' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:01:13.862 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10067 8 365281 365273 0.283449 0 - 0.0299513 2026-02-23T16:01:13.974 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:01:13 trial068 ceph-mon[38805]: pgmap v5839: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:01:14.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 16:01:13 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:16:01:13] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T16:01:14.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 16:01:13 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T16:01:13.974Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T16:01:14.862 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10068 8 365281 365273 0.283421 0 - 0.0299513 2026-02-23T16:01:14.942 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:01:15.062 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:01:15.160 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:01:15.158+0000 7fb5278e2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:01:15.160 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:01:15.159+0000 7fb5278e2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:01:15.270 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:01:15.271 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":355,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:00:50.340652+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:56:00.217470+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:58:33.486402+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:59:49.303087+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:01:15.862 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10069 8 365281 365273 0.283392 0 - 0.0299513 2026-02-23T16:01:16.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:01:15 trial068 ceph-mon[38805]: pgmap v5840: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:01:16.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:01:15 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/445323016' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:01:16.862 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10070 8 365281 365273 0.283364 0 - 0.0299513 2026-02-23T16:01:17.862 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10071 8 365281 365273 0.283336 0 - 0.0299513 2026-02-23T16:01:18.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:01:17 trial068 ceph-mon[38805]: pgmap v5841: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:01:18.308 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:01:18.425 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:01:18.527 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:01:18.525+0000 7f8dbbb0c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:01:18.527 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:01:18.525+0000 7f8dbbb0c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:01:18.636 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:01:18.636 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":355,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:00:50.340652+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:56:00.217470+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:58:33.486402+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:59:49.303087+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:01:18.862 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10072 8 365281 365273 0.283308 0 - 0.0299513 2026-02-23T16:01:19.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:01:18 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/759173709' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:01:19.862 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10073 8 365281 365273 0.28328 0 - 0.0299513 2026-02-23T16:01:20.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:01:19 trial068 ceph-mon[38805]: pgmap v5842: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:01:20.862 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10074 8 365281 365273 0.283252 0 - 0.0299513 2026-02-23T16:01:21.674 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:01:21.699 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:01:21 trial068 ceph-mon[38805]: pgmap v5843: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:01:21.793 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:01:21.862 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10075 8 365281 365273 0.283224 0 - 0.0299513 2026-02-23T16:01:21.891 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:01:21.889+0000 7eff9eb8c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:01:21.891 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:01:21.890+0000 7eff9eb8c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:01:22.001 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:01:22.001 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":355,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:00:50.340652+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:56:00.217470+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:58:33.486402+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:59:49.303087+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:01:22.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:01:22 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1442124877' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:01:22.863 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10076 8 365281 365273 0.283196 0 - 0.0299513 2026-02-23T16:01:23.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:01:23 trial068 ceph-mon[38805]: pgmap v5844: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:01:23.863 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10077 8 365281 365273 0.283167 0 - 0.0299513 2026-02-23T16:01:24.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 16:01:23 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:16:01:23] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T16:01:24.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 16:01:23 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T16:01:23.974Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T16:01:24.863 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10078 8 365281 365273 0.283139 0 - 0.0299513 2026-02-23T16:01:25.037 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:01:25.157 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:01:25.255 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:01:25.253+0000 7f4101f11640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:01:25.255 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:01:25.253+0000 7f4101f11640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:01:25.364 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:01:25.364 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":355,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:00:50.340652+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:56:00.217470+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:58:33.486402+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:59:49.303087+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:01:25.863 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10079 8 365281 365273 0.283111 0 - 0.0299513 2026-02-23T16:01:26.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:01:25 trial068 ceph-mon[38805]: pgmap v5845: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:01:26.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:01:25 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2988590860' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:01:26.863 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T16:01:26.862803+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T16:01:26.863 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T16:01:26.863 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10080 8 365281 365273 0.283083 0 - 0.0299513 2026-02-23T16:01:27.863 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10081 8 365281 365273 0.283055 0 - 0.0299513 2026-02-23T16:01:28.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:01:27 trial068 ceph-mon[38805]: pgmap v5846: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:01:28.404 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:01:28.520 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:01:28.621 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:01:28.619+0000 7fe923103640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:01:28.621 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:01:28.619+0000 7fe923103640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:01:28.730 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:01:28.730 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":355,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:00:50.340652+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:56:00.217470+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:58:33.486402+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:59:49.303087+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:01:28.863 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10082 8 365281 365273 0.283027 0 - 0.0299513 2026-02-23T16:01:29.863 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10083 8 365281 365273 0.282999 0 - 0.0299513 2026-02-23T16:01:30.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:01:29 trial068 ceph-mon[38805]: pgmap v5847: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:01:30.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:01:29 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1997243282' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:01:30.863 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10084 8 365281 365273 0.282971 0 - 0.0299513 2026-02-23T16:01:31.769 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:01:31.863 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10085 8 365281 365273 0.282943 0 - 0.0299513 2026-02-23T16:01:31.884 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:01:31.984 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:01:31.982+0000 7fa85fdc2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:01:31.984 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:01:31.982+0000 7fa85fdc2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:01:31.985 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:01:31 trial068 ceph-mon[38805]: pgmap v5848: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:01:32.093 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:01:32.093 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":355,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:00:50.340652+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:56:00.217470+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:58:33.486402+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:59:49.303087+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:01:32.863 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10086 8 365281 365273 0.282915 0 - 0.0299513 2026-02-23T16:01:33.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:01:32 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1674459234' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:01:33.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:01:32 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2026-02-23T16:01:33.863 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10087 8 365281 365273 0.282887 0 - 0.0299513 2026-02-23T16:01:33.973 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:01:33 trial068 ceph-mon[38805]: pgmap v5849: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:01:33.973 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:01:33 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-23T16:01:33.973 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:01:33 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2026-02-23T16:01:33.973 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:01:33 trial068 ceph-mon[38805]: pgmap v5850: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:01:33.974 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:01:33 trial068 ceph-mon[38805]: pgmap v5851: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:01:33.974 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:01:33 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T16:01:33.974 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:01:33 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2026-02-23T16:01:34.265 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 16:01:33 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T16:01:33.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T16:01:34.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 16:01:33 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:16:01:33] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T16:01:34.863 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10088 8 365281 365273 0.282859 0 - 0.0299513 2026-02-23T16:01:35.131 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:01:35.249 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:01:35.351 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:01:35.349+0000 7faedb000640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:01:35.351 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:01:35.349+0000 7faedb000640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:01:35.459 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:01:35.459 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":355,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:00:50.340652+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:56:00.217470+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:58:33.486402+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:59:49.303087+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:01:35.864 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10089 8 365281 365273 0.282831 0 - 0.0299513 2026-02-23T16:01:36.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:01:35 trial068 ceph-mon[38805]: pgmap v5852: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:01:36.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:01:35 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1257628731' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:01:36.864 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10090 8 365281 365273 0.282803 0 - 0.0299513 2026-02-23T16:01:37.864 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10091 8 365281 365273 0.282775 0 - 0.0299513 2026-02-23T16:01:38.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:01:37 trial068 ceph-mon[38805]: pgmap v5853: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:01:38.496 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:01:38.617 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:01:38.713 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:01:38.711+0000 7f12af4b6640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:01:38.713 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:01:38.712+0000 7f12af4b6640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:01:38.823 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:01:38.824 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":355,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:00:50.340652+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:56:00.217470+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:58:33.486402+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:59:49.303087+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:01:38.864 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10092 8 365281 365273 0.282747 0 - 0.0299513 2026-02-23T16:01:39.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:01:38 trial068 ceph-mon[38805]: 5.5 scrub starts 2026-02-23T16:01:39.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:01:38 trial068 ceph-mon[38805]: 5.5 scrub ok 2026-02-23T16:01:39.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:01:38 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2466034352' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:01:39.864 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10093 8 365281 365273 0.282719 0 - 0.0299513 2026-02-23T16:01:40.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:01:39 trial068 ceph-mon[38805]: pgmap v5854: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:01:40.864 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10094 8 365281 365273 0.282691 0 - 0.0299513 2026-02-23T16:01:41.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:01:41 trial068 ceph-mon[38805]: pgmap v5855: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:01:41.863 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:01:41.864 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10095 8 365281 365273 0.282663 0 - 0.0299513 2026-02-23T16:01:41.979 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:01:42.086 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:01:42.085+0000 7f83b7ecd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:01:42.086 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:01:42.085+0000 7f83b7ecd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:01:42.196 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:01:42.196 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":355,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:00:50.340652+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:56:00.217470+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:58:33.486402+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:59:49.303087+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:01:42.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:01:42 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3823285747' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:01:42.864 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10096 8 365281 365273 0.282635 0 - 0.0299513 2026-02-23T16:01:43.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:01:43 trial068 ceph-mon[38805]: pgmap v5856: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:01:43.864 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10097 8 365281 365273 0.282607 0 - 0.0299513 2026-02-23T16:01:44.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 16:01:43 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:16:01:43] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T16:01:44.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 16:01:43 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T16:01:43.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T16:01:44.864 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10098 8 365281 365273 0.282579 0 - 0.0299513 2026-02-23T16:01:45.235 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:01:45.352 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:01:45.453 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:01:45.451+0000 7f1cf43bb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:01:45.453 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:01:45.451+0000 7f1cf43bb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:01:45.562 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:01:45.562 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":355,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:00:50.340652+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:56:00.217470+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:58:33.486402+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:59:49.303087+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:01:45.864 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10099 8 365281 365273 0.282551 0 - 0.0299513 2026-02-23T16:01:46.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:01:45 trial068 ceph-mon[38805]: pgmap v5857: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:01:46.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:01:45 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3896577464' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:01:46.864 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T16:01:46.864178+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T16:01:46.864 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T16:01:46.864 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10100 8 365281 365273 0.282523 0 - 0.0299513 2026-02-23T16:01:47.864 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10101 8 365281 365273 0.282495 0 - 0.0299513 2026-02-23T16:01:48.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:01:47 trial068 ceph-mon[38805]: pgmap v5858: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:01:48.600 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:01:48.717 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:01:48.823 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:01:48.821+0000 7fced5f51640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:01:48.823 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:01:48.822+0000 7fced5f51640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:01:48.864 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10102 8 365281 365273 0.282467 0 - 0.0299513 2026-02-23T16:01:48.934 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:01:48.934 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":355,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:00:50.340652+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:56:00.217470+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:58:33.486402+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:59:49.303087+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:01:49.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:01:48 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3818283223' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:01:49.864 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10103 8 365281 365273 0.282439 0 - 0.0299513 2026-02-23T16:01:50.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:01:49 trial068 ceph-mon[38805]: pgmap v5859: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:01:50.864 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10104 8 365281 365273 0.282411 0 - 0.0299513 2026-02-23T16:01:51.865 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10105 8 365281 365273 0.282383 0 - 0.0299513 2026-02-23T16:01:51.974 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:01:52.094 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:01:52.195 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:01:52.193+0000 7f41faf71640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:01:52.195 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:01:52.193+0000 7f41faf71640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:01:52.308 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:01:52.308 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":356,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:01:51.968260+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:00:53.009756+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:58:33.486402+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:59:49.303087+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:01:52.309 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:01:51 trial068 ceph-mon[38805]: pgmap v5860: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:01:52.865 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10106 8 365281 365273 0.282355 0 - 0.0299513 2026-02-23T16:01:53.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:01:52 trial068 ceph-mon[38805]: osdmap e356: 4 total, 3 up, 4 in 2026-02-23T16:01:53.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:01:52 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2128768691' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:01:53.865 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10107 8 365281 365273 0.282327 0 - 0.0299513 2026-02-23T16:01:54.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:01:53 trial068 ceph-mon[38805]: pgmap v5862: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:01:54.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 16:01:53 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:16:01:53] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T16:01:54.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 16:01:53 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T16:01:53.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T16:01:54.865 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10108 8 365281 365273 0.282299 0 - 0.0299513 2026-02-23T16:01:55.347 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:01:55.466 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:01:55.565 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:01:55.563+0000 7fbf5ca62640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:01:55.565 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:01:55.563+0000 7fbf5ca62640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:01:55.672 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:01:55.673 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":357,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:01:55.349329+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:00:53.009756+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:58:33.486402+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:59:49.303087+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:01:55.865 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10109 8 365281 365273 0.282271 0 - 0.0299513 2026-02-23T16:01:56.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:01:55 trial068 ceph-mon[38805]: pgmap v5863: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:01:56.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:01:55 trial068 ceph-mon[38805]: osdmap e357: 4 total, 3 up, 4 in 2026-02-23T16:01:56.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:01:55 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/4174924838' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:01:56.865 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10110 8 365281 365273 0.282243 0 - 0.0299513 2026-02-23T16:01:57.865 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10111 8 365281 365273 0.282215 0 - 0.0299513 2026-02-23T16:01:58.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:01:57 trial068 ceph-mon[38805]: pgmap v5865: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:01:58.711 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:01:58.828 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:01:58.865 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10112 8 365281 365273 0.282187 0 - 0.0299513 2026-02-23T16:01:58.930 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:01:58.928+0000 7fbbffe62640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:01:58.931 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:01:58.929+0000 7fbbffe62640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:01:59.039 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:01:59.039 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":357,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:01:55.349329+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:00:53.009756+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:58:33.486402+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:59:49.303087+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:01:59.865 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10113 8 365281 365273 0.282159 0 - 0.0299513 2026-02-23T16:02:00.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:01:59 trial068 ceph-mon[38805]: pgmap v5866: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:02:00.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:01:59 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/120501031' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:02:00.865 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10114 8 365281 365273 0.282132 0 - 0.0299513 2026-02-23T16:02:01.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:02:00 trial068 ceph-mon[38805]: 5.8 scrub starts 2026-02-23T16:02:01.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:02:00 trial068 ceph-mon[38805]: 5.8 scrub ok 2026-02-23T16:02:01.865 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10115 8 365281 365273 0.282104 0 - 0.0299513 2026-02-23T16:02:02.078 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:02:02.195 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:02:02.235 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:02:01 trial068 ceph-mon[38805]: pgmap v5867: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:02:02.293 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:02:02.291+0000 7f804c5e0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:02:02.293 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:02:02.291+0000 7f804c5e0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:02:02.402 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:02:02.403 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":357,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:01:55.349329+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:00:53.009756+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:58:33.486402+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:59:49.303087+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:02:02.865 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10116 8 365281 365273 0.282076 0 - 0.0299513 2026-02-23T16:02:03.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:02:02 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3280777539' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:02:03.865 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10117 8 365281 365273 0.282048 0 - 0.0299513 2026-02-23T16:02:04.265 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 16:02:03 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T16:02:03.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T16:02:04.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:02:03 trial068 ceph-mon[38805]: pgmap v5868: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:02:04.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 16:02:03 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:16:02:03] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T16:02:04.865 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10118 8 365281 365273 0.28202 0 - 0.0299513 2026-02-23T16:02:05.440 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:02:05.559 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:02:05.663 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:02:05.661+0000 7f0ef2393640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:02:05.663 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:02:05.662+0000 7f0ef2393640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:02:05.776 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:02:05.776 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":357,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:01:55.349329+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:00:53.009756+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:58:33.486402+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:59:49.303087+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:02:05.866 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10119 8 365281 365273 0.281992 0 - 0.0299513 2026-02-23T16:02:06.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:02:05 trial068 ceph-mon[38805]: pgmap v5869: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:02:06.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:02:05 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3679082701' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:02:06.866 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T16:02:06.865525+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T16:02:06.866 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T16:02:06.866 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10120 8 365281 365273 0.281964 0 - 0.0299513 2026-02-23T16:02:07.866 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10121 8 365281 365273 0.281936 0 - 0.0299513 2026-02-23T16:02:08.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:02:07 trial068 ceph-mon[38805]: pgmap v5870: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:02:08.834 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:02:08.866 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10122 8 365281 365273 0.281909 0 - 0.0299513 2026-02-23T16:02:08.951 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:02:09.053 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:02:09.051+0000 7f170e40b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:02:09.053 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:02:09.051+0000 7f170e40b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:02:09.162 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:02:09.162 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":357,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:01:55.349329+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:00:53.009756+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:58:33.486402+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:59:49.303087+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:02:09.866 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10123 8 365281 365273 0.281881 0 - 0.0299513 2026-02-23T16:02:10.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:02:09 trial068 ceph-mon[38805]: pgmap v5871: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:02:10.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:02:09 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2921543872' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:02:10.866 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10124 8 365281 365273 0.281853 0 - 0.0299513 2026-02-23T16:02:11.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:02:11 trial068 ceph-mon[38805]: pgmap v5872: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:02:11.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:02:11 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T16:02:11.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:02:11 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T16:02:11.866 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10125 8 365281 365273 0.281825 0 - 0.0299513 2026-02-23T16:02:12.201 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:02:12.318 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:02:12.423 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:02:12.421+0000 7ff7034d7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:02:12.423 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:02:12.421+0000 7ff7034d7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:02:12.535 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:02:12.536 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":357,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:01:55.349329+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:00:53.009756+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:58:33.486402+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:59:49.303087+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:02:12.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:02:12 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1097785410' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:02:12.866 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10126 8 365281 365273 0.281797 0 - 0.0299513 2026-02-23T16:02:13.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:02:13 trial068 ceph-mon[38805]: pgmap v5873: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:02:13.866 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10127 8 365281 365273 0.281769 0 - 0.0299513 2026-02-23T16:02:14.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 16:02:13 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:16:02:13] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T16:02:14.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 16:02:13 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T16:02:13.974Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T16:02:14.866 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10128 8 365281 365273 0.281742 0 - 0.0299513 2026-02-23T16:02:15.574 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:02:15.691 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:02:15.790 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:02:15.788+0000 7f8ece0b3640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:02:15.791 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:02:15.789+0000 7f8ece0b3640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:02:15.866 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10129 8 365281 365273 0.281714 0 - 0.0299513 2026-02-23T16:02:15.899 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:02:15.900 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":357,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:01:55.349329+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:00:53.009756+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:58:33.486402+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:59:49.303087+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:02:16.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:02:15 trial068 ceph-mon[38805]: pgmap v5874: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:02:16.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:02:15 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1867184525' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:02:16.866 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10130 8 365281 365273 0.281686 0 - 0.0299513 2026-02-23T16:02:17.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:02:16 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T16:02:17.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:02:16 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T16:02:17.866 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10131 8 365281 365273 0.281658 0 - 0.0299513 2026-02-23T16:02:18.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:02:17 trial068 ceph-mon[38805]: pgmap v5875: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:02:18.866 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10132 8 365281 365273 0.28163 0 - 0.0299513 2026-02-23T16:02:18.938 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:02:19.057 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:02:19.153 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:02:19.151+0000 7fdbc4609640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:02:19.153 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:02:19.151+0000 7fdbc4609640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:02:19.264 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:02:19.264 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":357,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:01:55.349329+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:00:53.009756+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:58:33.486402+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:59:49.303087+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:02:19.867 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10133 8 365281 365273 0.281603 0 - 0.0299513 2026-02-23T16:02:20.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:02:19 trial068 ceph-mon[38805]: pgmap v5876: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:02:20.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:02:19 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/620265367' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:02:20.867 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10134 8 365281 365273 0.281575 0 - 0.0299513 2026-02-23T16:02:21.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:02:20 trial068 ceph-mon[38805]: 5.e scrub starts 2026-02-23T16:02:21.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:02:20 trial068 ceph-mon[38805]: 5.e scrub ok 2026-02-23T16:02:21.867 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10135 8 365281 365273 0.281547 0 - 0.0299513 2026-02-23T16:02:22.306 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:02:22.331 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:02:21 trial068 ceph-mon[38805]: pgmap v5877: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:02:22.426 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:02:22.528 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:02:22.526+0000 7f207243e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:02:22.528 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:02:22.526+0000 7f207243e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:02:22.637 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:02:22.637 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":357,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:01:55.349329+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:00:53.009756+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:58:33.486402+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:59:49.303087+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:02:22.867 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10136 8 365281 365273 0.281519 0 - 0.0299513 2026-02-23T16:02:23.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:02:22 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3489791402' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:02:23.867 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10137 8 365281 365273 0.281491 0 - 0.0299513 2026-02-23T16:02:24.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:02:23 trial068 ceph-mon[38805]: pgmap v5878: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:02:24.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 16:02:23 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:16:02:23] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T16:02:24.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 16:02:23 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T16:02:23.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T16:02:24.867 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10138 8 365281 365273 0.281464 0 - 0.0299513 2026-02-23T16:02:25.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:02:24 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T16:02:25.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:02:24 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T16:02:25.674 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:02:25.791 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:02:25.867 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10139 8 365281 365273 0.281436 0 - 0.0299513 2026-02-23T16:02:25.894 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:02:25.893+0000 7f276634f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:02:25.894 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:02:25.893+0000 7f276634f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:02:26.004 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:02:26.004 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":357,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:01:55.349329+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:00:53.009756+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:58:33.486402+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:59:49.303087+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:02:26.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:02:25 trial068 ceph-mon[38805]: pgmap v5879: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:02:26.867 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T16:02:26.866864+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T16:02:26.867 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T16:02:26.867 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10140 8 365281 365273 0.281408 0 - 0.0299513 2026-02-23T16:02:27.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:02:26 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2735119293' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:02:27.867 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10141 8 365281 365273 0.28138 0 - 0.0299513 2026-02-23T16:02:28.245 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:02:27 trial068 ceph-mon[38805]: 5.7 scrub starts 2026-02-23T16:02:28.245 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:02:27 trial068 ceph-mon[38805]: 5.7 scrub ok 2026-02-23T16:02:28.245 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:02:27 trial068 ceph-mon[38805]: pgmap v5880: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:02:28.867 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10142 8 365281 365273 0.281353 0 - 0.0299513 2026-02-23T16:02:29.040 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:02:29.159 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:02:29.256 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:02:29.254+0000 7efcf1e7a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:02:29.256 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:02:29.254+0000 7efcf1e7a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:02:29.365 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:02:29.365 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":357,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:01:55.349329+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:00:53.009756+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:58:33.486402+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:59:49.303087+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:02:29.867 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10143 8 365281 365273 0.281325 0 - 0.0299513 2026-02-23T16:02:30.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:02:29 trial068 ceph-mon[38805]: pgmap v5881: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:02:30.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:02:29 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/4205590370' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:02:30.867 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10144 8 365281 365273 0.281297 0 - 0.0299513 2026-02-23T16:02:31.867 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10145 8 365281 365273 0.281269 0 - 0.0299513 2026-02-23T16:02:32.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:02:31 trial068 ceph-mon[38805]: pgmap v5882: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:02:32.403 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:02:32.520 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:02:32.619 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:02:32.617+0000 7fade6727640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:02:32.619 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:02:32.617+0000 7fade6727640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:02:32.731 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:02:32.731 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":357,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:01:55.349329+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:00:53.009756+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:58:33.486402+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:59:49.303087+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:02:32.868 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10146 8 365281 365273 0.281242 0 - 0.0299513 2026-02-23T16:02:33.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:02:32 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/800467287' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:02:33.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:02:32 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2026-02-23T16:02:33.867 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10147 8 365281 365273 0.281214 0 - 0.0299513 2026-02-23T16:02:34.265 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 16:02:33 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T16:02:33.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T16:02:34.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 16:02:33 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:16:02:33] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T16:02:34.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:02:33 trial068 ceph-mon[38805]: pgmap v5883: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:02:34.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:02:33 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-23T16:02:34.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:02:33 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2026-02-23T16:02:34.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:02:33 trial068 ceph-mon[38805]: pgmap v5884: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:02:34.345 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:02:33 trial068 ceph-mon[38805]: pgmap v5885: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:02:34.345 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:02:33 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T16:02:34.345 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:02:33 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2026-02-23T16:02:34.868 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10148 8 365281 365273 0.281186 0 - 0.0299513 2026-02-23T16:02:35.770 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:02:35.868 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10149 8 365281 365273 0.281159 0 - 0.0299513 2026-02-23T16:02:35.887 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:02:35.986 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:02:35.984+0000 7f140e799640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:02:35.986 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:02:35.985+0000 7f140e799640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:02:36.096 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:02:36.096 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":357,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:01:55.349329+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:00:53.009756+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:58:33.486402+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:59:49.303087+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:02:36.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:02:36 trial068 ceph-mon[38805]: pgmap v5886: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:02:36.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:02:36 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1848786693' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:02:36.868 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10150 8 365281 365273 0.281131 0 - 0.0299513 2026-02-23T16:02:37.868 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10151 8 365281 365273 0.281103 0 - 0.0299513 2026-02-23T16:02:38.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:02:38 trial068 ceph-mon[38805]: pgmap v5887: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:02:38.868 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10152 8 365281 365273 0.281076 0 - 0.0299513 2026-02-23T16:02:39.135 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:02:39.254 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:02:39.350 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:02:39.348+0000 7ff8e9498640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:02:39.350 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:02:39.348+0000 7ff8e9498640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:02:39.458 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:02:39.458 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":357,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:01:55.349329+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:00:53.009756+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:58:33.486402+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:59:49.303087+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:02:39.459 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:02:39 trial068 ceph-mon[38805]: 5.a scrub starts 2026-02-23T16:02:39.459 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:02:39 trial068 ceph-mon[38805]: 5.a scrub ok 2026-02-23T16:02:39.868 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10153 8 365281 365273 0.281048 0 - 0.0299513 2026-02-23T16:02:40.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:02:40 trial068 ceph-mon[38805]: pgmap v5888: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:02:40.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:02:40 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1159970475' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:02:40.868 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10154 8 365281 365273 0.28102 0 - 0.0299513 2026-02-23T16:02:41.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:02:41 trial068 ceph-mon[38805]: pgmap v5889: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:02:41.868 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10155 8 365281 365273 0.280993 0 - 0.0299513 2026-02-23T16:02:42.495 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:02:42.610 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:02:42.718 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:02:42.716+0000 7f9b18f1c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:02:42.718 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:02:42.716+0000 7f9b18f1c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:02:42.829 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:02:42.829 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":357,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:01:55.349329+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:00:53.009756+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:58:33.486402+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:59:49.303087+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:02:42.868 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10156 8 365281 365273 0.280965 0 - 0.0299513 2026-02-23T16:02:43.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:02:42 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/362427133' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:02:43.868 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10157 8 365281 365273 0.280937 0 - 0.0299513 2026-02-23T16:02:44.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:02:43 trial068 ceph-mon[38805]: pgmap v5890: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:02:44.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 16:02:43 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:16:02:43] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T16:02:44.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 16:02:43 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T16:02:43.974Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T16:02:44.868 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10158 8 365281 365273 0.28091 0 - 0.0299513 2026-02-23T16:02:45.865 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:02:45.869 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10159 8 365281 365273 0.280882 0 - 0.0299513 2026-02-23T16:02:45.983 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:02:46.079 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:02:46.077+0000 7f5c5c2cf640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:02:46.079 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:02:46.077+0000 7f5c5c2cf640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:02:46.188 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:02:46.188 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":357,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:01:55.349329+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:00:53.009756+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:58:33.486402+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:59:49.303087+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:02:46.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:02:46 trial068 ceph-mon[38805]: pgmap v5891: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:02:46.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:02:46 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/83798368' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:02:46.869 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T16:02:46.868381+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T16:02:46.869 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T16:02:46.869 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10160 8 365281 365273 0.280854 0 - 0.0299513 2026-02-23T16:02:47.869 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10161 8 365281 365273 0.280827 0 - 0.0299513 2026-02-23T16:02:48.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:02:48 trial068 ceph-mon[38805]: pgmap v5892: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:02:48.869 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10162 8 365281 365273 0.280799 0 - 0.0299513 2026-02-23T16:02:49.227 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:02:49.345 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:02:49.442 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:02:49.440+0000 7f255c17c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:02:49.442 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:02:49.440+0000 7f255c17c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:02:49.552 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:02:49.552 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":357,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:01:55.349329+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:00:53.009756+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:58:33.486402+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:59:49.303087+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:02:49.869 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10163 8 365281 365273 0.280771 0 - 0.0299513 2026-02-23T16:02:50.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:02:50 trial068 ceph-mon[38805]: pgmap v5893: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:02:50.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:02:50 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3131999611' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:02:50.869 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10164 8 365281 365273 0.280744 0 - 0.0299513 2026-02-23T16:02:51.869 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10165 8 365281 365273 0.280716 0 - 0.0299513 2026-02-23T16:02:52.589 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:02:52.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:02:52 trial068 ceph-mon[38805]: pgmap v5894: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:02:52.706 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:02:52.805 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:02:52.803+0000 7fee6fe16640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:02:52.805 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:02:52.803+0000 7fee6fe16640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:02:52.869 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10166 8 365281 365273 0.280688 0 - 0.0299513 2026-02-23T16:02:52.917 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:02:52.917 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":357,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:01:55.349329+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:00:53.009756+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:58:33.486402+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:59:49.303087+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:02:53.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:02:53 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3010672192' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:02:53.869 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10167 8 365281 365273 0.280661 0 - 0.0299513 2026-02-23T16:02:54.235 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 16:02:53 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:16:02:53] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T16:02:54.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 16:02:53 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T16:02:53.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T16:02:54.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:02:54 trial068 ceph-mon[38805]: pgmap v5895: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:02:54.869 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10168 8 365281 365273 0.280633 0 - 0.0299513 2026-02-23T16:02:55.869 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10169 8 365281 365273 0.280606 0 - 0.0299513 2026-02-23T16:02:55.975 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:02:56.091 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:02:56.190 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:02:56.188+0000 7f758fa5b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:02:56.190 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:02:56.189+0000 7f758fa5b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:02:56.300 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:02:56.301 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":357,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:01:55.349329+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:00:53.009756+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:58:33.486402+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:59:49.303087+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:02:56.301 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:02:56 trial068 ceph-mon[38805]: pgmap v5896: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:02:56.869 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10170 8 365281 365273 0.280578 0 - 0.0299513 2026-02-23T16:02:57.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:02:57 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/4289733688' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:02:57.869 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10171 8 365281 365273 0.28055 0 - 0.0299513 2026-02-23T16:02:58.494 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:02:58 trial068 ceph-mon[38805]: pgmap v5897: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:02:58.869 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10172 8 365281 365273 0.280523 0 - 0.0299513 2026-02-23T16:02:59.339 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:02:59.461 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:02:59.572 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:02:59.570+0000 7fdf54a1e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:02:59.572 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:02:59.570+0000 7fdf54a1e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:02:59.683 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:02:59.683 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":357,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:01:55.349329+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:00:53.009756+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:58:33.486402+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:59:49.303087+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:02:59.869 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10173 8 365281 365273 0.280495 0 - 0.0299513 2026-02-23T16:03:00.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:03:00 trial068 ceph-mon[38805]: pgmap v5898: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:03:00.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:03:00 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/221720294' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:03:00.869 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10174 8 365281 365273 0.280468 0 - 0.0299513 2026-02-23T16:03:01.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:03:01 trial068 ceph-mon[38805]: pgmap v5899: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:03:01.869 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10175 8 365281 365273 0.28044 0 - 0.0299513 2026-02-23T16:03:02.721 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:03:02.839 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:03:02.869 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10176 8 365281 365273 0.280413 0 - 0.0299513 2026-02-23T16:03:02.937 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:03:02.936+0000 7fed86441640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:03:02.938 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:03:02.936+0000 7fed86441640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:03:03.047 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:03:03.048 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":357,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:01:55.349329+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:00:53.009756+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:58:33.486402+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:59:49.303087+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:03:03.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:03:03 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/372571261' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:03:03.870 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10177 8 365281 365273 0.280385 0 - 0.0299513 2026-02-23T16:03:04.264 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 16:03:03 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T16:03:03.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T16:03:04.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:03:04 trial068 ceph-mon[38805]: pgmap v5900: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:03:04.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 16:03:03 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:16:03:03] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T16:03:04.870 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10178 8 365281 365273 0.280358 0 - 0.0299513 2026-02-23T16:03:05.870 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10179 8 365281 365273 0.28033 0 - 0.0299513 2026-02-23T16:03:06.085 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:03:06.203 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:03:06.305 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:03:06.303+0000 7f74a68f3640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:03:06.305 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:03:06.303+0000 7f74a68f3640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:03:06.416 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:03:06.416 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":357,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:01:55.349329+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:00:53.009756+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:58:33.486402+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:59:49.303087+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:03:06.416 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:03:06 trial068 ceph-mon[38805]: pgmap v5901: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:03:06.870 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T16:03:06.869721+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T16:03:06.870 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T16:03:06.870 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10180 8 365281 365273 0.280302 0 - 0.0299513 2026-02-23T16:03:07.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:03:07 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/4230080569' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:03:07.870 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10181 8 365281 365273 0.280275 0 - 0.0299513 2026-02-23T16:03:08.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:03:08 trial068 ceph-mon[38805]: 5.5 scrub starts 2026-02-23T16:03:08.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:03:08 trial068 ceph-mon[38805]: 5.5 scrub ok 2026-02-23T16:03:08.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:03:08 trial068 ceph-mon[38805]: pgmap v5902: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:03:08.870 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10182 8 365281 365273 0.280247 0 - 0.0299513 2026-02-23T16:03:09.456 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:03:09.573 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:03:09.672 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:03:09.670+0000 7f2077716640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:03:09.673 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:03:09.671+0000 7f2077716640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:03:09.780 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:03:09.781 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":357,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:01:55.349329+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:00:53.009756+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:58:33.486402+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:59:49.303087+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:03:09.870 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10183 8 365281 365273 0.28022 0 - 0.0299513 2026-02-23T16:03:10.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:03:10 trial068 ceph-mon[38805]: pgmap v5903: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:03:10.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:03:10 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3856492306' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:03:10.870 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10184 8 365281 365273 0.280192 0 - 0.0299513 2026-02-23T16:03:11.870 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10185 8 365281 365273 0.280165 0 - 0.0299513 2026-02-23T16:03:12.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:03:12 trial068 ceph-mon[38805]: pgmap v5904: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:03:12.818 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:03:12.870 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10186 8 365281 365273 0.280137 0 - 0.0299513 2026-02-23T16:03:12.939 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:03:13.044 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:03:13.042+0000 7f0455d30640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:03:13.044 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:03:13.042+0000 7f0455d30640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:03:13.153 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:03:13.153 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":357,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:01:55.349329+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:00:53.009756+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:58:33.486402+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:59:49.303087+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:03:13.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:03:13 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/375216667' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:03:13.870 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10187 8 365281 365273 0.28011 0 - 0.0299513 2026-02-23T16:03:14.238 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 16:03:13 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:16:03:13] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T16:03:14.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 16:03:13 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T16:03:13.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T16:03:14.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:03:14 trial068 ceph-mon[38805]: pgmap v5905: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:03:14.870 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10188 8 365281 365273 0.280082 0 - 0.0299513 2026-02-23T16:03:15.871 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10189 8 365281 365273 0.280055 0 - 0.0299513 2026-02-23T16:03:16.190 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:03:16.309 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:03:16.415 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:03:16.412+0000 7fdc7b256640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:03:16.415 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:03:16.413+0000 7fdc7b256640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:03:16.523 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:03:16.523 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":357,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:01:55.349329+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:00:53.009756+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:58:33.486402+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:59:49.303087+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:03:16.524 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:03:16 trial068 ceph-mon[38805]: pgmap v5906: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:03:16.871 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10190 8 365281 365273 0.280027 0 - 0.0299513 2026-02-23T16:03:17.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:03:17 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1923559675' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:03:17.871 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10191 8 365281 365273 0.28 0 - 0.0299513 2026-02-23T16:03:18.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:03:18 trial068 ceph-mon[38805]: pgmap v5907: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:03:18.871 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10192 8 365281 365273 0.279972 0 - 0.0299513 2026-02-23T16:03:19.561 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:03:19.681 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:03:19.783 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:03:19.781+0000 7ff84733c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:03:19.783 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:03:19.781+0000 7ff84733c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:03:19.871 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10193 8 365281 365273 0.279945 0 - 0.0299513 2026-02-23T16:03:19.894 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:03:19.895 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":357,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:01:55.349329+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:00:53.009756+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:58:33.486402+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:59:49.303087+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:03:20.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:03:20 trial068 ceph-mon[38805]: pgmap v5908: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:03:20.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:03:20 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1232454060' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:03:20.871 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10194 8 365281 365273 0.279917 0 - 0.0299513 2026-02-23T16:03:21.871 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10195 8 365281 365273 0.27989 0 - 0.0299513 2026-02-23T16:03:22.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:03:22 trial068 ceph-mon[38805]: pgmap v5909: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:03:22.871 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10196 8 365281 365273 0.279863 0 - 0.0299513 2026-02-23T16:03:22.933 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:03:23.051 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:03:23.148 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:03:23.146+0000 7f84d115a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:03:23.148 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:03:23.146+0000 7f84d115a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:03:23.258 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:03:23.258 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":357,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:01:55.349329+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:00:53.009756+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:58:33.486402+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:59:49.303087+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:03:23.871 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10197 8 365281 365273 0.279835 0 - 0.0299513 2026-02-23T16:03:24.246 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 16:03:23 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:16:03:23] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T16:03:24.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 16:03:23 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T16:03:23.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T16:03:24.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:03:24 trial068 ceph-mon[38805]: pgmap v5910: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:03:24.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:03:24 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/887416178' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:03:24.871 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10198 8 365281 365273 0.279808 0 - 0.0299513 2026-02-23T16:03:25.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:03:25 trial068 ceph-mon[38805]: pgmap v5911: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:03:25.871 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10199 8 365281 365273 0.27978 0 - 0.0299513 2026-02-23T16:03:26.296 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:03:26.413 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:03:26.512 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:03:26.510+0000 7f5fc9839640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:03:26.512 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:03:26.510+0000 7f5fc9839640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:03:26.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:03:26 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T16:03:26.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:03:26 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T16:03:26.620 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:03:26.620 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":357,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:01:55.349329+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:00:53.009756+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:58:33.486402+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:59:49.303087+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:03:26.871 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T16:03:26.871276+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T16:03:26.872 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T16:03:26.872 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10200 8 365281 365273 0.279753 0 - 0.0299513 2026-02-23T16:03:27.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:03:27 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3237982721' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:03:27.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:03:27 trial068 ceph-mon[38805]: pgmap v5912: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:03:27.871 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10201 8 365281 365273 0.279725 0 - 0.0299513 2026-02-23T16:03:28.745 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:03:28 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T16:03:28.745 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:03:28 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T16:03:28.872 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10202 8 365281 365273 0.279698 0 - 0.0299513 2026-02-23T16:03:29.658 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:03:29.775 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:03:29.796 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:03:29 trial068 ceph-mon[38805]: 5.8 scrub starts 2026-02-23T16:03:29.797 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:03:29 trial068 ceph-mon[38805]: 5.8 scrub ok 2026-02-23T16:03:29.797 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:03:29 trial068 ceph-mon[38805]: pgmap v5913: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:03:29.872 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10203 8 365281 365273 0.279671 0 - 0.0299513 2026-02-23T16:03:29.872 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:03:29.871+0000 7fbe81d10640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:03:29.873 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:03:29.871+0000 7fbe81d10640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:03:29.984 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:03:29.984 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":357,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:01:55.349329+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:00:53.009756+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:58:33.486402+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:59:49.303087+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:03:30.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:03:30 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T16:03:30.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:03:30 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T16:03:30.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:03:30 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3197915841' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:03:30.872 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10204 8 365281 365273 0.279643 0 - 0.0299513 2026-02-23T16:03:31.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:03:31 trial068 ceph-mon[38805]: pgmap v5914: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:03:31.872 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10205 8 365281 365273 0.279616 0 - 0.0299513 2026-02-23T16:03:32.872 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10206 8 365281 365273 0.279588 0 - 0.0299513 2026-02-23T16:03:33.022 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:03:33.139 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:03:33.245 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:03:33.243+0000 7f0e1c87e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:03:33.245 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:03:33.243+0000 7f0e1c87e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:03:33.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:03:33 trial068 ceph-mon[38805]: pgmap v5915: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:03:33.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:03:33 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2026-02-23T16:03:33.359 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:03:33.360 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":357,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:01:55.349329+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:00:53.009756+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:58:33.486402+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:59:49.303087+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:03:33.872 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10207 8 365281 365273 0.279561 0 - 0.0299513 2026-02-23T16:03:34.265 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 16:03:33 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T16:03:33.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T16:03:34.266 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 16:03:33 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:16:03:33] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T16:03:34.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:03:34 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3354157856' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:03:34.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:03:34 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-23T16:03:34.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:03:34 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2026-02-23T16:03:34.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:03:34 trial068 ceph-mon[38805]: pgmap v5916: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:03:34.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:03:34 trial068 ceph-mon[38805]: pgmap v5917: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:03:34.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:03:34 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T16:03:34.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:03:34 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2026-02-23T16:03:34.872 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10208 8 365281 365273 0.279534 0 - 0.0299513 2026-02-23T16:03:35.872 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10209 8 365281 365273 0.279506 0 - 0.0299513 2026-02-23T16:03:36.398 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:03:36.515 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:03:36.615 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:03:36.613+0000 7f8d5eec8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:03:36.615 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:03:36.613+0000 7f8d5eec8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:03:36.725 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:03:36.725 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":357,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:01:55.349329+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:00:53.009756+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:58:33.486402+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:59:49.303087+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:03:36.725 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:03:36 trial068 ceph-mon[38805]: pgmap v5918: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:03:36.872 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10210 8 365281 365273 0.279479 0 - 0.0299513 2026-02-23T16:03:37.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:03:37 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/745377626' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:03:37.872 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10211 8 365281 365273 0.279451 0 - 0.0299513 2026-02-23T16:03:38.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:03:38 trial068 ceph-mon[38805]: pgmap v5919: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:03:38.872 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10212 8 365281 365273 0.279424 0 - 0.0299513 2026-02-23T16:03:39.764 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:03:39.872 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10213 8 365281 365273 0.279397 0 - 0.0299513 2026-02-23T16:03:39.880 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:03:39.988 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:03:39.986+0000 7f9d74646640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:03:39.988 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:03:39.986+0000 7f9d74646640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:03:40.097 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:03:40.097 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":357,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:01:55.349329+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:00:53.009756+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:58:33.486402+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:59:49.303087+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:03:40.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:03:40 trial068 ceph-mon[38805]: pgmap v5920: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:03:40.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:03:40 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/164431319' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:03:40.872 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10214 8 365281 365273 0.279369 0 - 0.0299513 2026-02-23T16:03:41.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:03:41 trial068 ceph-mon[38805]: 5.e scrub starts 2026-02-23T16:03:41.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:03:41 trial068 ceph-mon[38805]: 5.e scrub ok 2026-02-23T16:03:41.872 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10215 8 365281 365273 0.279342 0 - 0.0299513 2026-02-23T16:03:42.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:03:42 trial068 ceph-mon[38805]: pgmap v5921: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:03:42.872 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10216 8 365281 365273 0.279315 0 - 0.0299513 2026-02-23T16:03:43.138 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:03:43.257 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:03:43.357 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:03:43.355+0000 7f02d94b5640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:03:43.357 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:03:43.355+0000 7f02d94b5640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:03:43.467 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:03:43.467 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":357,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:01:55.349329+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:00:53.009756+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:58:33.486402+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:59:49.303087+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:03:43.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:03:43 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1870931282' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:03:43.872 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10217 8 365281 365273 0.279287 0 - 0.0299513 2026-02-23T16:03:44.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 16:03:43 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:16:03:43] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T16:03:44.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 16:03:43 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T16:03:43.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T16:03:44.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:03:44 trial068 ceph-mon[38805]: pgmap v5922: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:03:44.872 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10218 8 365281 365273 0.27926 0 - 0.0299513 2026-02-23T16:03:45.873 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10219 8 365281 365273 0.279233 0 - 0.0299513 2026-02-23T16:03:46.502 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:03:46.621 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:03:46.644 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:03:46 trial068 ceph-mon[38805]: pgmap v5923: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:03:46.720 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:03:46.718+0000 7f7c3409e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:03:46.720 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:03:46.718+0000 7f7c3409e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:03:46.832 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:03:46.832 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":357,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:01:55.349329+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:00:53.009756+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:58:33.486402+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:59:49.303087+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:03:46.873 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T16:03:46.872650+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T16:03:46.873 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T16:03:46.873 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10220 8 365281 365273 0.279205 0 - 0.0299513 2026-02-23T16:03:47.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:03:47 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3551913063' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:03:47.873 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10221 8 365281 365273 0.279178 0 - 0.0299513 2026-02-23T16:03:48.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:03:48 trial068 ceph-mon[38805]: pgmap v5924: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:03:48.873 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10222 8 365281 365273 0.279151 0 - 0.0299513 2026-02-23T16:03:49.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:03:49 trial068 ceph-mon[38805]: 5.7 scrub starts 2026-02-23T16:03:49.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:03:49 trial068 ceph-mon[38805]: 5.7 scrub ok 2026-02-23T16:03:49.869 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:03:49.873 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10223 8 365281 365273 0.279123 0 - 0.0299513 2026-02-23T16:03:49.986 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:03:50.090 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:03:50.088+0000 7f4df2db7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:03:50.090 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:03:50.088+0000 7f4df2db7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:03:50.198 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:03:50.198 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":357,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:01:55.349329+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:00:53.009756+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:58:33.486402+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:59:49.303087+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:03:50.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:03:50 trial068 ceph-mon[38805]: pgmap v5925: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:03:50.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:03:50 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1815296011' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:03:50.873 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10224 8 365281 365273 0.279096 0 - 0.0299513 2026-02-23T16:03:51.873 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10225 8 365281 365273 0.279069 0 - 0.0299513 2026-02-23T16:03:52.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:03:52 trial068 ceph-mon[38805]: pgmap v5926: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:03:52.873 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10226 8 365281 365273 0.279042 0 - 0.0299513 2026-02-23T16:03:53.237 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:03:53.353 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:03:53.454 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:03:53.452+0000 7fd36fec7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:03:53.454 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:03:53.453+0000 7fd36fec7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:03:53.566 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:03:53.566 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":357,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:01:55.349329+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:00:53.009756+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:58:33.486402+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:59:49.303087+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:03:53.873 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10227 8 365281 365273 0.279014 0 - 0.0299513 2026-02-23T16:03:54.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 16:03:53 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:16:03:53] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T16:03:54.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 16:03:53 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T16:03:53.972Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T16:03:54.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:03:54 trial068 ceph-mon[38805]: pgmap v5927: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:03:54.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:03:54 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3738811813' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:03:54.873 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10228 8 365281 365273 0.278987 0 - 0.0299513 2026-02-23T16:03:55.873 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10229 8 365281 365273 0.27896 0 - 0.0299513 2026-02-23T16:03:56.603 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:03:56.721 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:03:56.743 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:03:56 trial068 ceph-mon[38805]: pgmap v5928: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:03:56.820 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:03:56.818+0000 7f7098e0a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:03:56.820 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:03:56.819+0000 7f7098e0a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:03:56.873 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10230 8 365281 365273 0.278932 0 - 0.0299513 2026-02-23T16:03:56.930 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:03:56.930 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":357,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:01:55.349329+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:00:53.009756+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:58:33.486402+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:59:49.303087+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:03:57.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:03:57 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2746901239' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:03:57.873 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10231 8 365281 365273 0.278905 0 - 0.0299513 2026-02-23T16:03:58.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:03:58 trial068 ceph-mon[38805]: pgmap v5929: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:03:58.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:03:58 trial068 ceph-mon[38805]: 5.a scrub starts 2026-02-23T16:03:58.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:03:58 trial068 ceph-mon[38805]: 5.a scrub ok 2026-02-23T16:03:58.874 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10232 8 365281 365273 0.278878 0 - 0.0299513 2026-02-23T16:03:59.874 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10233 8 365281 365273 0.278851 0 - 0.0299513 2026-02-23T16:03:59.969 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:04:00.085 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:04:00.190 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:04:00.188+0000 7fbace363640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:04:00.190 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:04:00.188+0000 7fbace363640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:04:00.300 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:04:00.300 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":357,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:01:55.349329+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:00:53.009756+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:58:33.486402+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:59:49.303087+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:04:00.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:04:00 trial068 ceph-mon[38805]: pgmap v5930: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:04:00.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:04:00 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2036851021' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:04:00.874 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10234 8 365281 365273 0.278823 0 - 0.0299513 2026-02-23T16:04:01.874 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10235 8 365281 365273 0.278796 0 - 0.0299513 2026-02-23T16:04:02.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:04:02 trial068 ceph-mon[38805]: pgmap v5931: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:04:02.874 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10236 8 365281 365273 0.278769 0 - 0.0299513 2026-02-23T16:04:03.338 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:04:03.455 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:04:03.554 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:04:03.552+0000 7faba9399640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:04:03.554 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:04:03.552+0000 7faba9399640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:04:03.663 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:04:03.663 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":357,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:01:55.349329+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:00:53.009756+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:58:33.486402+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:59:49.303087+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:04:03.874 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10237 8 365281 365273 0.278742 0 - 0.0299513 2026-02-23T16:04:04.264 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 16:04:03 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T16:04:03.974Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T16:04:04.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 16:04:03 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:16:04:03] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T16:04:04.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:04:04 trial068 ceph-mon[38805]: pgmap v5932: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:04:04.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:04:04 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2588398429' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:04:04.874 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10238 8 365281 365273 0.278714 0 - 0.0299513 2026-02-23T16:04:05.874 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10239 8 365281 365273 0.278687 0 - 0.0299513 2026-02-23T16:04:06.701 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:04:06.727 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:04:06 trial068 ceph-mon[38805]: pgmap v5933: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:04:06.818 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:04:06.874 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T16:04:06.874124+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T16:04:06.874 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T16:04:06.875 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10240 8 365281 365273 0.27866 0 - 0.0299513 2026-02-23T16:04:06.917 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:04:06.915+0000 7fefeb246640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:04:06.917 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:04:06.915+0000 7fefeb246640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:04:07.026 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:04:07.026 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":357,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:01:55.349329+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:00:53.009756+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:58:33.486402+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:59:49.303087+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:04:07.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:04:07 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/134463671' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:04:07.874 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10241 8 365281 365273 0.278633 0 - 0.0299513 2026-02-23T16:04:08.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:04:08 trial068 ceph-mon[38805]: pgmap v5934: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:04:08.874 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10242 8 365281 365273 0.278606 0 - 0.0299513 2026-02-23T16:04:09.874 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10243 8 365281 365273 0.278578 0 - 0.0299513 2026-02-23T16:04:10.065 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:04:10.182 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:04:10.285 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:04:10.283+0000 7f501342b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:04:10.285 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:04:10.283+0000 7f501342b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:04:10.396 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:04:10.396 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":357,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:01:55.349329+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:00:53.009756+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:58:33.486402+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:59:49.303087+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:04:10.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:04:10 trial068 ceph-mon[38805]: pgmap v5935: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:04:10.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:04:10 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/53898863' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:04:10.875 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10244 8 365281 365273 0.278551 0 - 0.0299513 2026-02-23T16:04:11.875 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10245 8 365281 365273 0.278524 0 - 0.0299513 2026-02-23T16:04:12.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:04:12 trial068 ceph-mon[38805]: pgmap v5936: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:04:12.875 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10246 8 365281 365273 0.278497 0 - 0.0299513 2026-02-23T16:04:13.435 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:04:13.554 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:04:13.659 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:04:13.657+0000 7fb0f8bff640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:04:13.659 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:04:13.657+0000 7fb0f8bff640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:04:13.769 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:04:13.769 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":357,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:01:55.349329+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:00:53.009756+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:58:33.486402+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:59:49.303087+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:04:13.875 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10247 8 365281 365273 0.27847 0 - 0.0299513 2026-02-23T16:04:14.093 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 16:04:13 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:16:04:13] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T16:04:14.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 16:04:13 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T16:04:13.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T16:04:14.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:04:14 trial068 ceph-mon[38805]: pgmap v5937: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:04:14.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:04:14 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1078472147' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:04:14.875 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10248 8 365281 365273 0.278443 0 - 0.0299513 2026-02-23T16:04:15.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:04:15 trial068 ceph-mon[38805]: 5.5 scrub starts 2026-02-23T16:04:15.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:04:15 trial068 ceph-mon[38805]: 5.5 scrub ok 2026-02-23T16:04:15.875 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10249 8 365281 365273 0.278415 0 - 0.0299513 2026-02-23T16:04:16.808 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:04:16.833 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:04:16 trial068 ceph-mon[38805]: pgmap v5938: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:04:16.875 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10250 8 365281 365273 0.278388 0 - 0.0299513 2026-02-23T16:04:16.926 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:04:17.031 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:04:17.029+0000 7f838e4ef640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:04:17.031 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:04:17.029+0000 7f838e4ef640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:04:17.140 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:04:17.140 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":357,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:01:55.349329+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:00:53.009756+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:58:33.486402+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:59:49.303087+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:04:17.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:04:17 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/334079154' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:04:17.875 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10251 8 365281 365273 0.278361 0 - 0.0299513 2026-02-23T16:04:18.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:04:18 trial068 ceph-mon[38805]: pgmap v5939: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:04:18.875 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10252 8 365281 365273 0.278334 0 - 0.0299513 2026-02-23T16:04:19.875 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10253 8 365281 365273 0.278307 0 - 0.0299513 2026-02-23T16:04:20.177 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:04:20.296 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:04:20.397 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:04:20.395+0000 7f1f81122640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:04:20.397 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:04:20.395+0000 7f1f81122640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:04:20.505 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:04:20.505 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":357,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:01:55.349329+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:00:53.009756+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:58:33.486402+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:59:49.303087+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:04:20.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:04:20 trial068 ceph-mon[38805]: pgmap v5940: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:04:20.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:04:20 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3274102076' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:04:20.875 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10254 8 365281 365273 0.27828 0 - 0.0299513 2026-02-23T16:04:21.876 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10255 8 365281 365273 0.278252 0 - 0.0299513 2026-02-23T16:04:22.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:04:22 trial068 ceph-mon[38805]: pgmap v5941: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:04:22.876 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10256 8 365281 365273 0.278225 0 - 0.0299513 2026-02-23T16:04:23.543 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:04:23.662 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:04:23.764 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:04:23.762+0000 7f1086a8d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:04:23.764 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:04:23.762+0000 7f1086a8d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:04:23.876 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:04:23.876 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":357,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:01:55.349329+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:00:53.009756+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:58:33.486402+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:59:49.303087+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:04:23.876 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10257 8 365281 365273 0.278198 0 - 0.0299513 2026-02-23T16:04:24.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 16:04:23 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:16:04:23] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T16:04:24.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 16:04:23 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T16:04:23.972Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T16:04:24.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:04:24 trial068 ceph-mon[38805]: pgmap v5942: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:04:24.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:04:24 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3985007477' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:04:24.876 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10258 8 365281 365273 0.278171 0 - 0.0299513 2026-02-23T16:04:25.876 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10259 8 365281 365273 0.278144 0 - 0.0299513 2026-02-23T16:04:26.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:04:26 trial068 ceph-mon[38805]: pgmap v5943: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:04:26.876 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T16:04:26.875783+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T16:04:26.876 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T16:04:26.876 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10260 8 365281 365273 0.278117 0 - 0.0299513 2026-02-23T16:04:26.914 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:04:27.034 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:04:27.135 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:04:27.133+0000 7f8928a48640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:04:27.135 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:04:27.133+0000 7f8928a48640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:04:27.243 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:04:27.244 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":357,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:01:55.349329+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:00:53.009756+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:58:33.486402+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:59:49.303087+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:04:27.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:04:27 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3258079808' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:04:27.876 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10261 8 365281 365273 0.27809 0 - 0.0299513 2026-02-23T16:04:28.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:04:28 trial068 ceph-mon[38805]: pgmap v5944: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:04:28.876 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10262 8 365281 365273 0.278063 0 - 0.0299513 2026-02-23T16:04:29.876 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10263 8 365281 365273 0.278036 0 - 0.0299513 2026-02-23T16:04:30.283 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:04:30.404 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:04:30.507 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:04:30.505+0000 7fa9e7313640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:04:30.508 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:04:30.506+0000 7fa9e7313640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:04:30.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:04:30 trial068 ceph-mon[38805]: pgmap v5945: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:04:30.617 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:04:30.617 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":357,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:01:55.349329+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:00:53.009756+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:58:33.486402+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:59:49.303087+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:04:30.876 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10264 8 365281 365273 0.278008 0 - 0.0299513 2026-02-23T16:04:31.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:04:31 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2153678237' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:04:31.876 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10265 8 365281 365273 0.277981 0 - 0.0299513 2026-02-23T16:04:32.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:04:32 trial068 ceph-mon[38805]: pgmap v5946: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:04:32.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:04:32 trial068 ceph-mon[38805]: osdmap e358: 4 total, 3 up, 4 in 2026-02-23T16:04:32.876 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10266 8 365281 365273 0.277954 0 - 0.0299513 2026-02-23T16:04:33.654 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:04:33.774 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:04:33.798 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:04:33 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2026-02-23T16:04:33.874 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:04:33.872+0000 7fe56b536640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:04:33.875 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:04:33.873+0000 7fe56b536640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:04:33.876 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10267 8 365281 365273 0.277927 0 - 0.0299513 2026-02-23T16:04:33.984 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:04:33.985 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":358,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:04:31.498367+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:00:53.009756+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:03:29.547807+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:59:49.303087+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:04:34.093 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 16:04:33 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:16:04:33] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T16:04:34.265 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 16:04:33 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T16:04:33.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T16:04:34.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:04:34 trial068 ceph-mon[38805]: pgmap v5948: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:04:34.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:04:34 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-23T16:04:34.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:04:34 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2026-02-23T16:04:34.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:04:34 trial068 ceph-mon[38805]: pgmap v5949: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:04:34.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:04:34 trial068 ceph-mon[38805]: pgmap v5950: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:04:34.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:04:34 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T16:04:34.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:04:34 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2026-02-23T16:04:34.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:04:34 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3169341685' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:04:34.876 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10268 8 365281 365273 0.2779 0 - 0.0299513 2026-02-23T16:04:35.877 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10269 8 365281 365273 0.277873 0 - 0.0299513 2026-02-23T16:04:36.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:04:36 trial068 ceph-mon[38805]: osdmap e359: 4 total, 3 up, 4 in 2026-02-23T16:04:36.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:04:36 trial068 ceph-mon[38805]: pgmap v5952: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:04:36.876 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10270 8 365281 365273 0.277846 0 - 0.0299513 2026-02-23T16:04:37.021 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:04:37.140 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:04:37.242 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:04:37.240+0000 7f1015e2f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:04:37.242 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:04:37.240+0000 7f1015e2f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:04:37.350 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:04:37.350 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":359,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:04:35.369672+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:00:53.009756+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:03:29.547807+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:59:49.303087+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:04:37.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:04:37 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/574053869' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:04:37.877 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10271 8 365281 365273 0.277819 0 - 0.0299513 2026-02-23T16:04:38.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:04:38 trial068 ceph-mon[38805]: pgmap v5953: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:04:38.877 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10272 8 365281 365273 0.277792 0 - 0.0299513 2026-02-23T16:04:39.877 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10273 8 365281 365273 0.277765 0 - 0.0299513 2026-02-23T16:04:40.387 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:04:40.507 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:04:40.609 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:04:40.607+0000 7f1a7e147640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:04:40.609 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:04:40.607+0000 7f1a7e147640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:04:40.717 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:04:40.717 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":359,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:04:35.369672+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:00:53.009756+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:03:29.547807+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:59:49.303087+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:04:40.877 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10274 8 365281 365273 0.277738 0 - 0.0299513 2026-02-23T16:04:41.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:04:40 trial068 ceph-mon[38805]: pgmap v5954: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:04:41.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:04:40 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/4186809514' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:04:41.877 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10275 8 365281 365273 0.277711 0 - 0.0299513 2026-02-23T16:04:42.877 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10276 8 365281 365273 0.277684 0 - 0.0299513 2026-02-23T16:04:43.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:04:42 trial068 ceph-mon[38805]: pgmap v5955: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:04:43.754 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:04:43.871 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:04:43.877 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10277 8 365281 365273 0.277657 0 - 0.0299513 2026-02-23T16:04:43.971 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:04:43.969+0000 7f0e6d508640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:04:43.971 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:04:43.969+0000 7f0e6d508640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:04:44.081 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 16:04:43 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:16:04:43] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T16:04:44.081 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:04:44.081 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":359,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:04:35.369672+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:00:53.009756+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:03:29.547807+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:59:49.303087+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:04:44.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 16:04:43 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T16:04:43.972Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T16:04:44.877 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10278 8 365281 365273 0.27763 0 - 0.0299513 2026-02-23T16:04:45.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:04:44 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T16:04:45.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:04:44 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T16:04:45.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:04:44 trial068 ceph-mon[38805]: pgmap v5956: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:04:45.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:04:44 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3391251455' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:04:45.877 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10279 8 365281 365273 0.277603 0 - 0.0299513 2026-02-23T16:04:46.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:04:46 trial068 ceph-mon[38805]: pgmap v5957: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:04:46.877 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T16:04:46.877195+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T16:04:46.877 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T16:04:46.878 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10280 8 365281 365273 0.277576 0 - 0.0299513 2026-02-23T16:04:47.119 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:04:47.237 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:04:47.338 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:04:47.336+0000 7f04d4435640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:04:47.338 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:04:47.336+0000 7f04d4435640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:04:47.446 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:04:47.447 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":359,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:04:35.369672+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:00:53.009756+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:03:29.547807+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:59:49.303087+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:04:47.447 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:04:47 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T16:04:47.447 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:04:47 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T16:04:47.877 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10281 8 365281 365273 0.277549 0 - 0.0299513 2026-02-23T16:04:48.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:04:48 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2917674284' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:04:48.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:04:48 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T16:04:48.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:04:48 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T16:04:48.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:04:48 trial068 ceph-mon[38805]: pgmap v5958: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:04:48.877 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10282 8 365281 365273 0.277522 0 - 0.0299513 2026-02-23T16:04:49.877 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10283 8 365281 365273 0.277495 0 - 0.0299513 2026-02-23T16:04:50.483 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:04:50.600 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:04:50.700 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:04:50.698+0000 7efc88a4f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:04:50.700 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:04:50.699+0000 7efc88a4f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:04:50.808 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:04:50.809 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":359,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:04:35.369672+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:00:53.009756+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:03:29.547807+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:59:49.303087+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:04:50.809 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:04:50 trial068 ceph-mon[38805]: pgmap v5959: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:04:50.878 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10284 8 365281 365273 0.277468 0 - 0.0299513 2026-02-23T16:04:51.878 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10285 8 365281 365273 0.277441 0 - 0.0299513 2026-02-23T16:04:52.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:04:51 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/464476046' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:04:52.878 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10286 8 365281 365273 0.277414 0 - 0.0299513 2026-02-23T16:04:53.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:04:52 trial068 ceph-mon[38805]: pgmap v5960: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:04:53.848 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:04:53.878 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10287 8 365281 365273 0.277387 0 - 0.0299513 2026-02-23T16:04:53.964 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:04:54.069 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:04:54.067+0000 7efc8665c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:04:54.069 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:04:54.067+0000 7efc8665c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:04:54.180 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:04:54.181 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":359,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:04:35.369672+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:00:53.009756+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:03:29.547807+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:59:49.303087+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:04:54.181 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 16:04:53 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:16:04:53] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T16:04:54.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 16:04:53 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T16:04:53.972Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T16:04:54.878 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10288 8 365281 365273 0.27736 0 - 0.0299513 2026-02-23T16:04:55.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:04:54 trial068 ceph-mon[38805]: pgmap v5961: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:04:55.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:04:54 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1039431766' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:04:55.878 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10289 8 365281 365273 0.277333 0 - 0.0299513 2026-02-23T16:04:56.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:04:56 trial068 ceph-mon[38805]: 5.8 scrub starts 2026-02-23T16:04:56.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:04:56 trial068 ceph-mon[38805]: 5.8 scrub ok 2026-02-23T16:04:56.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:04:56 trial068 ceph-mon[38805]: pgmap v5962: 17 pgs: 1 active+clean+scrubbing, 1 active+recovery_unfound+undersized+degraded, 4 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:04:56.878 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10290 8 365281 365273 0.277306 0 - 0.0299513 2026-02-23T16:04:57.219 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:04:57.337 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:04:57.446 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:04:57.444+0000 7f6a18d5a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:04:57.446 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:04:57.444+0000 7f6a18d5a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:04:57.555 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:04:57.556 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":359,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:04:35.369672+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:00:53.009756+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:03:29.547807+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:59:49.303087+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:04:57.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:04:57 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2811587362' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:04:57.878 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10291 8 365281 365273 0.277279 0 - 0.0299513 2026-02-23T16:04:58.878 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10292 8 365281 365273 0.277252 0 - 0.0299513 2026-02-23T16:04:59.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:04:58 trial068 ceph-mon[38805]: pgmap v5963: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 1 active+clean+scrubbing, 4 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:04:59.878 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10293 8 365281 365273 0.277225 0 - 0.0299513 2026-02-23T16:05:00.592 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:05:00.711 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:05:00.815 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:05:00.813+0000 7f1496bd0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:05:00.816 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:05:00.814+0000 7f1496bd0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:05:00.878 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10294 8 365281 365273 0.277198 0 - 0.0299513 2026-02-23T16:05:00.924 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:05:00.924 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":359,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:04:35.369672+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:00:53.009756+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:03:29.547807+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:59:49.303087+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:05:00.925 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:05:00 trial068 ceph-mon[38805]: pgmap v5964: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 1 active+clean+scrubbing, 4 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:05:01.878 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10295 8 365281 365273 0.277171 0 - 0.0299513 2026-02-23T16:05:02.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:05:01 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3393580518' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:05:02.879 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10296 8 365281 365273 0.277144 0 - 0.0299513 2026-02-23T16:05:03.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:05:02 trial068 ceph-mon[38805]: pgmap v5965: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:05:03.879 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10297 8 365281 365273 0.277117 0 - 0.0299513 2026-02-23T16:05:03.963 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:05:04.081 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:05:04.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:05:03 trial068 ceph-mon[38805]: 5.e scrub starts 2026-02-23T16:05:04.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:05:03 trial068 ceph-mon[38805]: 5.e scrub ok 2026-02-23T16:05:04.094 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 16:05:03 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:16:05:03] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T16:05:04.179 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:05:04.177+0000 7ff795c5b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:05:04.180 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:05:04.178+0000 7ff795c5b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:05:04.265 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 16:05:03 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T16:05:03.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T16:05:04.288 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:05:04.289 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":359,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:04:35.369672+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:00:53.009756+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:03:29.547807+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:59:49.303087+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:05:04.879 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10298 8 365281 365273 0.277091 0 - 0.0299513 2026-02-23T16:05:05.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:05:04 trial068 ceph-mon[38805]: 5.7 scrub starts 2026-02-23T16:05:05.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:05:04 trial068 ceph-mon[38805]: 5.7 scrub ok 2026-02-23T16:05:05.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:05:04 trial068 ceph-mon[38805]: pgmap v5966: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:05:05.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:05:04 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2561532031' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:05:05.879 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10299 8 365281 365273 0.277064 0 - 0.0299513 2026-02-23T16:05:06.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:05:06 trial068 ceph-mon[38805]: pgmap v5967: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:05:06.879 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T16:05:06.879162+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T16:05:06.879 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T16:05:06.880 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10300 8 365281 365273 0.277037 0 - 0.0299513 2026-02-23T16:05:07.327 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:05:07.445 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:05:07.546 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:05:07.544+0000 7fa440353640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:05:07.546 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:05:07.544+0000 7fa440353640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:05:07.655 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:05:07.655 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":359,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:04:35.369672+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:00:53.009756+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:03:29.547807+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:59:49.303087+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:05:07.879 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10301 8 365281 365273 0.27701 0 - 0.0299513 2026-02-23T16:05:08.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:05:07 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/820492846' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:05:08.879 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10302 8 365281 365273 0.276983 0 - 0.0299513 2026-02-23T16:05:09.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:05:08 trial068 ceph-mon[38805]: pgmap v5968: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:05:09.879 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10303 8 365281 365273 0.276956 0 - 0.0299513 2026-02-23T16:05:10.693 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:05:10.810 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:05:10.880 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10304 8 365281 365273 0.276929 0 - 0.0299513 2026-02-23T16:05:10.912 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:05:10.909+0000 7fe423309640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:05:10.912 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:05:10.910+0000 7fe423309640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:05:11.024 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:05:11.024 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":359,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:04:35.369672+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:00:53.009756+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:03:29.547807+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:59:49.303087+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:05:11.024 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:05:10 trial068 ceph-mon[38805]: pgmap v5969: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:05:11.880 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10305 8 365281 365273 0.276902 0 - 0.0299513 2026-02-23T16:05:12.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:05:11 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1001284587' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:05:12.880 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10306 8 365281 365273 0.276875 0 - 0.0299513 2026-02-23T16:05:13.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:05:12 trial068 ceph-mon[38805]: pgmap v5970: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:05:13.880 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10307 8 365281 365273 0.276849 0 - 0.0299513 2026-02-23T16:05:14.064 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:05:14.185 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:05:14.227 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 16:05:13 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:16:05:13] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T16:05:14.286 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:05:14.284+0000 7f26329dc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:05:14.286 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:05:14.284+0000 7f26329dc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:05:14.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 16:05:13 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T16:05:13.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T16:05:14.395 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:05:14.395 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":359,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:04:35.369672+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:00:53.009756+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:03:29.547807+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:59:49.303087+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:05:14.880 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10308 8 365281 365273 0.276822 0 - 0.0299513 2026-02-23T16:05:15.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:05:14 trial068 ceph-mon[38805]: pgmap v5971: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:05:15.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:05:14 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2894970656' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:05:15.880 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10309 8 365281 365273 0.276795 0 - 0.0299513 2026-02-23T16:05:16.880 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10310 8 365281 365273 0.276768 0 - 0.0299513 2026-02-23T16:05:17.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:05:16 trial068 ceph-mon[38805]: pgmap v5972: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:05:17.432 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:05:17.551 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:05:17.650 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:05:17.648+0000 7f05db1ba640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:05:17.650 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:05:17.648+0000 7f05db1ba640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:05:17.760 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:05:17.760 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":359,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:04:35.369672+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:00:53.009756+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:03:29.547807+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:59:49.303087+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:05:17.880 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10311 8 365281 365273 0.276741 0 - 0.0299513 2026-02-23T16:05:18.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:05:17 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1272017751' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:05:18.880 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10312 8 365281 365273 0.276714 0 - 0.0299513 2026-02-23T16:05:19.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:05:18 trial068 ceph-mon[38805]: pgmap v5973: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:05:19.880 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10313 8 365281 365273 0.276688 0 - 0.0299513 2026-02-23T16:05:20.798 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:05:20.880 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10314 8 365281 365273 0.276661 0 - 0.0299513 2026-02-23T16:05:20.919 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:05:21.019 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:05:21.017+0000 7f8528583640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:05:21.019 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:05:21.017+0000 7f8528583640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:05:21.019 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:05:20 trial068 ceph-mon[38805]: pgmap v5974: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:05:21.128 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:05:21.128 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":359,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:04:35.369672+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:00:53.009756+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:03:29.547807+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:59:49.303087+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:05:21.880 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10315 8 365281 365273 0.276634 0 - 0.0299513 2026-02-23T16:05:22.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:05:21 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1605528765' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:05:22.880 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10316 8 365281 365273 0.276607 0 - 0.0299513 2026-02-23T16:05:23.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:05:22 trial068 ceph-mon[38805]: pgmap v5975: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:05:23.880 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10317 8 365281 365273 0.27658 0 - 0.0299513 2026-02-23T16:05:24.166 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:05:24.283 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:05:24.306 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 16:05:23 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:16:05:23] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T16:05:24.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 16:05:23 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T16:05:23.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T16:05:24.387 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:05:24.385+0000 7f13c479f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:05:24.387 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:05:24.385+0000 7f13c479f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:05:24.496 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:05:24.496 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":359,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:04:35.369672+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:00:53.009756+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:03:29.547807+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:59:49.303087+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:05:24.881 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10318 8 365281 365273 0.276553 0 - 0.0299513 2026-02-23T16:05:25.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:05:24 trial068 ceph-mon[38805]: pgmap v5976: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:05:25.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:05:24 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2908494504' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:05:25.881 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10319 8 365281 365273 0.276527 0 - 0.0299513 2026-02-23T16:05:26.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:05:26 trial068 ceph-mon[38805]: pgmap v5977: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:05:26.881 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T16:05:26.880507+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T16:05:26.881 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T16:05:26.881 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10320 8 365281 365273 0.2765 0 - 0.0299513 2026-02-23T16:05:27.533 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:05:27.650 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:05:27.758 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:05:27.756+0000 7fd7a8616640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:05:27.758 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:05:27.756+0000 7fd7a8616640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:05:27.869 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:05:27.869 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":359,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:04:35.369672+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:00:53.009756+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:03:29.547807+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:59:49.303087+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:05:27.881 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10321 8 365281 365273 0.276473 0 - 0.0299513 2026-02-23T16:05:28.881 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10322 8 365281 365273 0.276446 0 - 0.0299513 2026-02-23T16:05:29.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:05:28 trial068 ceph-mon[38805]: pgmap v5978: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:05:29.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:05:28 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1931877486' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:05:29.881 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10323 8 365281 365273 0.27642 0 - 0.0299513 2026-02-23T16:05:30.881 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10324 8 365281 365273 0.276393 0 - 0.0299513 2026-02-23T16:05:30.907 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:05:31.026 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:05:31.048 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:05:30 trial068 ceph-mon[38805]: pgmap v5979: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:05:31.129 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:05:31.127+0000 7f8f9e896640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:05:31.129 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:05:31.127+0000 7f8f9e896640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:05:31.238 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:05:31.239 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":359,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:04:35.369672+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:00:53.009756+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:03:29.547807+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:59:49.303087+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:05:31.881 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10325 8 365281 365273 0.276366 0 - 0.0299513 2026-02-23T16:05:32.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:05:31 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2373266920' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:05:32.881 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10326 8 365281 365273 0.276339 0 - 0.0299513 2026-02-23T16:05:33.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:05:32 trial068 ceph-mon[38805]: 5.5 scrub starts 2026-02-23T16:05:33.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:05:32 trial068 ceph-mon[38805]: pgmap v5980: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:05:33.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:05:32 trial068 ceph-mon[38805]: 5.5 scrub ok 2026-02-23T16:05:33.881 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10327 8 365281 365273 0.276312 0 - 0.0299513 2026-02-23T16:05:34.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:05:33 trial068 ceph-mon[38805]: 5.a scrub starts 2026-02-23T16:05:34.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:05:33 trial068 ceph-mon[38805]: 5.a scrub ok 2026-02-23T16:05:34.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:05:33 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2026-02-23T16:05:34.094 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 16:05:33 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:16:05:33] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T16:05:34.265 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 16:05:33 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T16:05:33.972Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T16:05:34.275 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:05:34.392 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:05:34.490 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:05:34.488+0000 7fbdba133640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:05:34.490 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:05:34.488+0000 7fbdba133640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:05:34.599 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:05:34.599 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":359,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:04:35.369672+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:00:53.009756+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:03:29.547807+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:59:49.303087+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:05:34.881 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10328 8 365281 365273 0.276286 0 - 0.0299513 2026-02-23T16:05:35.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:05:34 trial068 ceph-mon[38805]: pgmap v5981: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:05:35.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:05:34 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-23T16:05:35.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:05:34 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2026-02-23T16:05:35.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:05:34 trial068 ceph-mon[38805]: pgmap v5982: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:05:35.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:05:34 trial068 ceph-mon[38805]: pgmap v5983: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:05:35.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:05:34 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T16:05:35.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:05:34 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2026-02-23T16:05:35.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:05:34 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1213537089' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:05:35.881 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10329 8 365281 365273 0.276259 0 - 0.0299513 2026-02-23T16:05:36.881 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10330 8 365281 365273 0.276232 0 - 0.0299513 2026-02-23T16:05:37.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:05:37 trial068 ceph-mon[38805]: pgmap v5984: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:05:37.636 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:05:37.753 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:05:37.862 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:05:37.860+0000 7f9e560e4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:05:37.862 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:05:37.860+0000 7f9e560e4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:05:37.881 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10331 8 365281 365273 0.276205 0 - 0.0299513 2026-02-23T16:05:37.970 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:05:37.971 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":359,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:04:35.369672+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:00:53.009756+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:03:29.547807+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:59:49.303087+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:05:38.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:05:38 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3615339005' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:05:38.881 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10332 8 365281 365273 0.276179 0 - 0.0299513 2026-02-23T16:05:39.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:05:39 trial068 ceph-mon[38805]: pgmap v5985: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:05:39.881 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10333 8 365281 365273 0.276152 0 - 0.0299513 2026-02-23T16:05:40.882 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10334 8 365281 365273 0.276125 0 - 0.0299513 2026-02-23T16:05:41.010 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:05:41.128 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:05:41.224 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:05:41.222+0000 7fc849c63640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:05:41.224 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:05:41.222+0000 7fc849c63640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:05:41.333 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:05:41.334 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":359,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:04:35.369672+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:00:53.009756+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:03:29.547807+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:59:49.303087+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:05:41.334 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:05:41 trial068 ceph-mon[38805]: pgmap v5986: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:05:41.882 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10335 8 365281 365273 0.276099 0 - 0.0299513 2026-02-23T16:05:42.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:05:42 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/415211490' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:05:42.882 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10336 8 365281 365273 0.276072 0 - 0.0299513 2026-02-23T16:05:43.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:05:43 trial068 ceph-mon[38805]: pgmap v5987: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:05:43.882 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10337 8 365281 365273 0.276045 0 - 0.0299513 2026-02-23T16:05:44.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 16:05:43 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:16:05:43] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T16:05:44.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 16:05:43 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T16:05:43.974Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T16:05:44.370 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:05:44.487 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:05:44.585 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:05:44.583+0000 7ff186ffc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:05:44.585 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:05:44.583+0000 7ff186ffc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:05:44.697 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:05:44.697 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":359,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:04:35.369672+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:00:53.009756+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:03:29.547807+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T15:59:49.303087+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:05:44.882 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10338 8 365281 365273 0.276018 0 - 0.0299513 2026-02-23T16:05:45.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:05:45 trial068 ceph-mon[38805]: pgmap v5988: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:05:45.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:05:45 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1125788021' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:05:45.882 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10339 8 365281 365273 0.275992 0 - 0.0299513 2026-02-23T16:05:46.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:05:46 trial068 ceph-mon[38805]: pgmap v5989: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:05:46.882 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T16:05:46.882304+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T16:05:46.883 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T16:05:46.883 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10340 8 365281 365273 0.275965 0 - 0.0299513 2026-02-23T16:05:47.736 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:05:47.854 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:05:47.882 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10341 8 365281 365273 0.275938 0 - 0.0299513 2026-02-23T16:05:47.958 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:05:47.956+0000 7f3386674640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:05:47.958 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:05:47.957+0000 7f3386674640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:05:48.067 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:05:48.067 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":360,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:05:47.929222+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:00:53.009756+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:03:29.547807+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:04:45.660145+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:05:48.883 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10342 8 365281 365273 0.275912 0 - 0.0299513 2026-02-23T16:05:49.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:05:48 trial068 ceph-mon[38805]: osdmap e360: 4 total, 3 up, 4 in 2026-02-23T16:05:49.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:05:48 trial068 ceph-mon[38805]: pgmap v5991: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:05:49.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:05:48 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1824829729' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:05:49.883 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10343 8 365281 365273 0.275885 0 - 0.0299513 2026-02-23T16:05:50.883 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10344 8 365281 365273 0.275858 0 - 0.0299513 2026-02-23T16:05:51.105 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:05:51.221 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:05:51.276 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:05:51 trial068 ceph-mon[38805]: pgmap v5992: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:05:51.276 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:05:51 trial068 ceph-mon[38805]: osdmap e361: 4 total, 3 up, 4 in 2026-02-23T16:05:51.323 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:05:51.321+0000 7f5d44b85640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:05:51.323 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:05:51.321+0000 7f5d44b85640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:05:51.431 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:05:51.431 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":361,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:05:50.378097+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:00:53.009756+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:03:29.547807+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:04:45.660145+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:05:51.883 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10345 8 365281 365273 0.275832 0 - 0.0299513 2026-02-23T16:05:52.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:05:52 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2439459254' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:05:52.883 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10346 8 365281 365273 0.275805 0 - 0.0299513 2026-02-23T16:05:53.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:05:53 trial068 ceph-mon[38805]: pgmap v5994: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:05:53.883 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10347 8 365281 365273 0.275778 0 - 0.0299513 2026-02-23T16:05:54.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 16:05:53 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:16:05:53] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T16:05:54.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 16:05:53 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T16:05:53.974Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T16:05:54.469 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:05:54.588 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:05:54.695 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:05:54.693+0000 7f35dc8cd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:05:54.696 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:05:54.694+0000 7f35dc8cd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:05:54.803 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:05:54.804 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":361,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:05:50.378097+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:00:53.009756+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:03:29.547807+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:04:45.660145+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:05:54.883 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10348 8 365281 365273 0.275752 0 - 0.0299513 2026-02-23T16:05:55.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:05:55 trial068 ceph-mon[38805]: pgmap v5995: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:05:55.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:05:55 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1085707241' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:05:55.883 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10349 8 365281 365273 0.275725 0 - 0.0299513 2026-02-23T16:05:56.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:05:56 trial068 ceph-mon[38805]: pgmap v5996: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:05:56.883 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10350 8 365281 365273 0.275698 0 - 0.0299513 2026-02-23T16:05:57.841 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:05:57.883 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10351 8 365281 365273 0.275672 0 - 0.0299513 2026-02-23T16:05:57.960 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:05:58.061 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:05:58.059+0000 7ff4ea505640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:05:58.061 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:05:58.059+0000 7ff4ea505640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:05:58.169 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:05:58.169 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":361,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:05:50.378097+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:00:53.009756+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:03:29.547807+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:04:45.660145+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:05:58.883 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10352 8 365281 365273 0.275645 0 - 0.0299513 2026-02-23T16:05:59.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:05:59 trial068 ceph-mon[38805]: pgmap v5997: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:05:59.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:05:59 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/329734915' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:05:59.883 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10353 8 365281 365273 0.275619 0 - 0.0299513 2026-02-23T16:06:00.883 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10354 8 365281 365273 0.275592 0 - 0.0299513 2026-02-23T16:06:01.207 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:06:01.326 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:06:01.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:06:01 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T16:06:01.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:06:01 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T16:06:01.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:06:01 trial068 ceph-mon[38805]: pgmap v5998: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:06:01.428 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:06:01.426+0000 7f91397b5640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:06:01.428 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:06:01.426+0000 7f91397b5640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:06:01.537 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:06:01.537 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":361,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:05:50.378097+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:00:53.009756+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:03:29.547807+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:04:45.660145+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:06:01.884 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10355 8 365281 365273 0.275565 0 - 0.0299513 2026-02-23T16:06:02.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:06:02 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1282491842' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:06:02.884 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10356 8 365281 365273 0.275539 0 - 0.0299513 2026-02-23T16:06:03.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:06:03 trial068 ceph-mon[38805]: pgmap v5999: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:06:03.884 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10357 8 365281 365273 0.275512 0 - 0.0299513 2026-02-23T16:06:04.265 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 16:06:03 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T16:06:03.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T16:06:04.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 16:06:03 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:16:06:03] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T16:06:04.575 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:06:04.691 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:06:04.789 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:06:04.787+0000 7f74acbae640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:06:04.789 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:06:04.788+0000 7f74acbae640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:06:04.884 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10358 8 365281 365273 0.275486 0 - 0.0299513 2026-02-23T16:06:04.900 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:06:04.900 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":361,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:05:50.378097+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:00:53.009756+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:03:29.547807+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:04:45.660145+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:06:05.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:06:05 trial068 ceph-mon[38805]: pgmap v6000: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:06:05.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:06:05 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1016408388' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:06:05.884 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10359 8 365281 365273 0.275459 0 - 0.0299513 2026-02-23T16:06:06.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:06:06 trial068 ceph-mon[38805]: pgmap v6001: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:06:06.884 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T16:06:06.883792+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T16:06:06.884 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T16:06:06.884 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10360 8 365281 365273 0.275432 0 - 0.0299513 2026-02-23T16:06:07.884 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10361 8 365281 365273 0.275406 0 - 0.0299513 2026-02-23T16:06:07.939 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:06:08.059 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:06:08.156 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:06:08.154+0000 7fdc4366c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:06:08.156 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:06:08.154+0000 7fdc4366c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:06:08.266 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:06:08.266 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":361,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:05:50.378097+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:00:53.009756+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:03:29.547807+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:04:45.660145+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:06:08.884 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10362 8 365281 365273 0.275379 0 - 0.0299513 2026-02-23T16:06:09.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:06:09 trial068 ceph-mon[38805]: pgmap v6002: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:06:09.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:06:09 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/259780845' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:06:09.884 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10363 8 365281 365273 0.275353 0 - 0.0299513 2026-02-23T16:06:10.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:06:10 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T16:06:10.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:06:10 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T16:06:10.884 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10364 8 365281 365273 0.275326 0 - 0.0299513 2026-02-23T16:06:11.304 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:06:11.328 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:06:11 trial068 ceph-mon[38805]: pgmap v6003: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:06:11.422 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:06:11.520 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:06:11.518+0000 7fd51bf26640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:06:11.520 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:06:11.518+0000 7fd51bf26640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:06:11.632 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:06:11.632 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":361,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:05:50.378097+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:00:53.009756+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:03:29.547807+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:04:45.660145+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:06:11.884 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10365 8 365281 365273 0.275299 0 - 0.0299513 2026-02-23T16:06:12.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:06:12 trial068 ceph-mon[38805]: 5.8 scrub starts 2026-02-23T16:06:12.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:06:12 trial068 ceph-mon[38805]: 5.8 scrub ok 2026-02-23T16:06:12.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:06:12 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3343216345' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:06:12.884 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10366 8 365281 365273 0.275273 0 - 0.0299513 2026-02-23T16:06:13.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:06:13 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T16:06:13.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:06:13 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T16:06:13.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:06:13 trial068 ceph-mon[38805]: pgmap v6004: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:06:13.884 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10367 8 365281 365273 0.275246 0 - 0.0299513 2026-02-23T16:06:14.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 16:06:13 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:16:06:13] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T16:06:14.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 16:06:13 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T16:06:13.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T16:06:14.668 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:06:14.783 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:06:14.884 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:06:14.882+0000 7fa2c07a2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:06:14.884 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:06:14.883+0000 7fa2c07a2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:06:14.885 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10368 8 365281 365273 0.27522 0 - 0.0299513 2026-02-23T16:06:14.994 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:06:14.994 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":361,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:05:50.378097+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:00:53.009756+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:03:29.547807+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:04:45.660145+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:06:15.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:06:15 trial068 ceph-mon[38805]: pgmap v6005: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:06:15.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:06:15 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3858306269' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:06:15.885 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10369 8 365281 365273 0.275193 0 - 0.0299513 2026-02-23T16:06:16.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:06:16 trial068 ceph-mon[38805]: pgmap v6006: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:06:16.885 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10370 8 365281 365273 0.275167 0 - 0.0299513 2026-02-23T16:06:17.885 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10371 8 365281 365273 0.27514 0 - 0.0299513 2026-02-23T16:06:18.031 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:06:18.148 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:06:18.250 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:06:18.248+0000 7f897132e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:06:18.266 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:06:18.248+0000 7f897132e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:06:18.360 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:06:18.361 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":361,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:05:50.378097+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:00:53.009756+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:03:29.547807+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:04:45.660145+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:06:18.885 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10372 8 365281 365273 0.275114 0 - 0.0299513 2026-02-23T16:06:19.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:06:19 trial068 ceph-mon[38805]: pgmap v6007: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:06:19.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:06:19 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3875370249' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:06:19.885 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10373 8 365281 365273 0.275087 0 - 0.0299513 2026-02-23T16:06:20.885 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10374 8 365281 365273 0.275061 0 - 0.0299513 2026-02-23T16:06:21.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:06:21 trial068 ceph-mon[38805]: pgmap v6008: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:06:21.398 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:06:21.515 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:06:21.614 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:06:21.613+0000 7f70fe6ea640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:06:21.615 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:06:21.613+0000 7f70fe6ea640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:06:21.726 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:06:21.727 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":361,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:05:50.378097+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:00:53.009756+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:03:29.547807+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:04:45.660145+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:06:21.885 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10375 8 365281 365273 0.275034 0 - 0.0299513 2026-02-23T16:06:22.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:06:22 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1480979488' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:06:22.885 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10376 8 365281 365273 0.275008 0 - 0.0299513 2026-02-23T16:06:23.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:06:23 trial068 ceph-mon[38805]: pgmap v6009: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:06:23.885 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10377 8 365281 365273 0.274981 0 - 0.0299513 2026-02-23T16:06:24.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 16:06:23 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:16:06:23] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T16:06:24.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 16:06:23 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T16:06:23.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T16:06:24.764 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:06:24.883 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:06:24.885 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10378 8 365281 365273 0.274955 0 - 0.0299513 2026-02-23T16:06:24.984 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:06:24.982+0000 7fcbdd1cf640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:06:24.985 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:06:24.983+0000 7fcbdd1cf640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:06:25.092 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:06:25.093 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":361,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:05:50.378097+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:00:53.009756+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:03:29.547807+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:04:45.660145+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:06:25.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:06:25 trial068 ceph-mon[38805]: pgmap v6010: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:06:25.885 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10379 8 365281 365273 0.274928 0 - 0.0299513 2026-02-23T16:06:26.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:06:26 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/629147654' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:06:26.885 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T16:06:26.885184+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T16:06:26.886 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T16:06:26.886 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10380 8 365281 365273 0.274902 0 - 0.0299513 2026-02-23T16:06:27.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:06:27 trial068 ceph-mon[38805]: pgmap v6011: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:06:27.885 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10381 8 365281 365273 0.274875 0 - 0.0299513 2026-02-23T16:06:28.130 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:06:28.247 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:06:28.346 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:06:28.344+0000 7f4064d90640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:06:28.347 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:06:28.345+0000 7f4064d90640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:06:28.456 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:06:28.456 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":361,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:05:50.378097+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:00:53.009756+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:03:29.547807+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:04:45.660145+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:06:28.885 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10382 8 365281 365273 0.274849 0 - 0.0299513 2026-02-23T16:06:29.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:06:29 trial068 ceph-mon[38805]: pgmap v6012: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:06:29.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:06:29 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/625671311' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:06:29.886 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10383 8 365281 365273 0.274822 0 - 0.0299513 2026-02-23T16:06:30.886 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10384 8 365281 365273 0.274796 0 - 0.0299513 2026-02-23T16:06:31.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:06:31 trial068 ceph-mon[38805]: 5.e scrub starts 2026-02-23T16:06:31.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:06:31 trial068 ceph-mon[38805]: pgmap v6013: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:06:31.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:06:31 trial068 ceph-mon[38805]: 5.e scrub ok 2026-02-23T16:06:31.494 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:06:31.612 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:06:31.715 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:06:31.713+0000 7f4e6cbc3640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:06:31.715 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:06:31.713+0000 7f4e6cbc3640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:06:31.824 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:06:31.825 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":361,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:05:50.378097+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:00:53.009756+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:03:29.547807+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:04:45.660145+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:06:31.886 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10385 8 365281 365273 0.274769 0 - 0.0299513 2026-02-23T16:06:32.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:06:32 trial068 ceph-mon[38805]: 5.7 scrub starts 2026-02-23T16:06:32.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:06:32 trial068 ceph-mon[38805]: 5.7 scrub ok 2026-02-23T16:06:32.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:06:32 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1175723914' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:06:32.886 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10386 8 365281 365273 0.274743 0 - 0.0299513 2026-02-23T16:06:33.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:06:33 trial068 ceph-mon[38805]: pgmap v6014: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:06:33.886 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10387 8 365281 365273 0.274716 0 - 0.0299513 2026-02-23T16:06:34.264 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 16:06:33 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T16:06:33.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T16:06:34.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:06:34 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2026-02-23T16:06:34.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 16:06:33 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:16:06:33] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T16:06:34.862 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:06:34.886 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10388 8 365281 365273 0.27469 0 - 0.0299513 2026-02-23T16:06:34.978 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:06:35.079 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:06:35.077+0000 7fe441645640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:06:35.079 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:06:35.077+0000 7fe441645640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:06:35.188 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:06:35.189 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":361,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:05:50.378097+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:00:53.009756+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:03:29.547807+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:04:45.660145+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:06:35.189 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:06:35 trial068 ceph-mon[38805]: pgmap v6015: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:06:35.189 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:06:35 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-23T16:06:35.189 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:06:35 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2026-02-23T16:06:35.189 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:06:35 trial068 ceph-mon[38805]: pgmap v6016: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:06:35.189 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:06:35 trial068 ceph-mon[38805]: pgmap v6017: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:06:35.189 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:06:35 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T16:06:35.189 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:06:35 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2026-02-23T16:06:35.886 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10389 8 365281 365273 0.274663 0 - 0.0299513 2026-02-23T16:06:36.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:06:36 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2088269090' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:06:36.886 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10390 8 365281 365273 0.274637 0 - 0.0299513 2026-02-23T16:06:37.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:06:37 trial068 ceph-mon[38805]: pgmap v6018: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:06:37.886 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10391 8 365281 365273 0.274611 0 - 0.0299513 2026-02-23T16:06:38.226 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:06:38.343 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:06:38.451 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:06:38.449+0000 7f154cd70640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:06:38.451 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:06:38.449+0000 7f154cd70640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:06:38.561 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:06:38.561 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":361,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:05:50.378097+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:00:53.009756+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:03:29.547807+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:04:45.660145+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:06:38.561 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:06:38 trial068 ceph-mon[38805]: pgmap v6019: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:06:38.886 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10392 8 365281 365273 0.274584 0 - 0.0299513 2026-02-23T16:06:39.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:06:39 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1978108956' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:06:39.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:06:39 trial068 ceph-mon[38805]: 5.5 scrub starts 2026-02-23T16:06:39.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:06:39 trial068 ceph-mon[38805]: 5.5 scrub ok 2026-02-23T16:06:39.886 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10393 8 365281 365273 0.274558 0 - 0.0299513 2026-02-23T16:06:40.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:06:40 trial068 ceph-mon[38805]: pgmap v6020: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:06:40.886 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10394 8 365281 365273 0.274531 0 - 0.0299513 2026-02-23T16:06:41.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:06:41 trial068 ceph-mon[38805]: 5.a scrub starts 2026-02-23T16:06:41.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:06:41 trial068 ceph-mon[38805]: 5.a scrub ok 2026-02-23T16:06:41.599 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:06:41.716 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:06:41.815 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:06:41.813+0000 7f2b7984e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:06:41.816 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:06:41.814+0000 7f2b7984e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:06:41.886 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10395 8 365281 365273 0.274505 0 - 0.0299513 2026-02-23T16:06:41.925 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:06:41.925 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":361,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:05:50.378097+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:00:53.009756+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:03:29.547807+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:04:45.660145+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:06:42.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:06:42 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1000095627' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:06:42.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:06:42 trial068 ceph-mon[38805]: pgmap v6021: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:06:42.886 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10396 8 365281 365273 0.274479 0 - 0.0299513 2026-02-23T16:06:43.887 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10397 8 365281 365273 0.274452 0 - 0.0299513 2026-02-23T16:06:44.311 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 16:06:43 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:16:06:43] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T16:06:44.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 16:06:43 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T16:06:43.974Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T16:06:44.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:06:44 trial068 ceph-mon[38805]: pgmap v6022: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:06:44.887 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10398 8 365281 365273 0.274426 0 - 0.0299513 2026-02-23T16:06:44.962 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:06:45.081 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:06:45.181 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:06:45.179+0000 7f8c97f17640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:06:45.181 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:06:45.179+0000 7f8c97f17640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:06:45.291 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:06:45.291 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":361,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:05:50.378097+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:00:53.009756+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:03:29.547807+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:04:45.660145+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:06:45.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:06:45 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/4081576505' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:06:45.887 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10399 8 365281 365273 0.274399 0 - 0.0299513 2026-02-23T16:06:46.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:06:46 trial068 ceph-mon[38805]: pgmap v6023: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:06:46.887 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T16:06:46.886620+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T16:06:46.887 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T16:06:46.887 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10400 8 365281 365273 0.274373 0 - 0.0299513 2026-02-23T16:06:47.887 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10401 8 365281 365273 0.274347 0 - 0.0299513 2026-02-23T16:06:48.328 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:06:48.445 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:06:48.547 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:06:48.545+0000 7fb75bdcf640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:06:48.547 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:06:48.546+0000 7fb75bdcf640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:06:48.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:06:48 trial068 ceph-mon[38805]: pgmap v6024: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:06:48.657 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:06:48.657 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":361,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:05:50.378097+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:00:53.009756+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:03:29.547807+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:04:45.660145+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:06:48.887 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10402 8 365281 365273 0.27432 0 - 0.0299513 2026-02-23T16:06:49.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:06:49 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3532824730' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:06:49.887 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10403 8 365281 365273 0.274294 0 - 0.0299513 2026-02-23T16:06:50.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:06:50 trial068 ceph-mon[38805]: pgmap v6025: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:06:50.887 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10404 8 365281 365273 0.274267 0 - 0.0299513 2026-02-23T16:06:51.695 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:06:51.814 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:06:51.887 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10405 8 365281 365273 0.274241 0 - 0.0299513 2026-02-23T16:06:51.915 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:06:51.913+0000 7f67374b4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:06:51.915 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:06:51.913+0000 7f67374b4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:06:52.024 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:06:52.024 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":361,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:05:50.378097+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:00:53.009756+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:03:29.547807+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:04:45.660145+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:06:52.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:06:52 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1413302671' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:06:52.887 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10406 8 365281 365273 0.274215 0 - 0.0299513 2026-02-23T16:06:53.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:06:53 trial068 ceph-mon[38805]: pgmap v6026: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:06:53.887 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10407 8 365281 365273 0.274188 0 - 0.0299513 2026-02-23T16:06:54.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:06:54 trial068 ceph-mon[38805]: osdmap e362: 4 total, 3 up, 4 in 2026-02-23T16:06:54.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 16:06:53 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:16:06:53] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T16:06:54.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 16:06:53 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T16:06:53.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T16:06:54.887 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10408 8 365281 365273 0.274162 0 - 0.0299513 2026-02-23T16:06:55.063 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:06:55.183 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:06:55.280 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:06:55.278+0000 7fbf6f5ed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:06:55.280 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:06:55.278+0000 7fbf6f5ed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:06:55.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:06:55 trial068 ceph-mon[38805]: pgmap v6028: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:06:55.390 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:06:55.390 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":363,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:06:55.386038+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:05:59.814738+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:03:29.547807+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:04:45.660145+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:06:55.887 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10409 8 365281 365273 0.274136 0 - 0.0299513 2026-02-23T16:06:56.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:06:56 trial068 ceph-mon[38805]: osdmap e363: 4 total, 3 up, 4 in 2026-02-23T16:06:56.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:06:56 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2115145365' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:06:56.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:06:56 trial068 ceph-mon[38805]: pgmap v6030: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:06:56.887 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10410 8 365281 365273 0.274109 0 - 0.0299513 2026-02-23T16:06:57.887 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10411 8 365281 365273 0.274083 0 - 0.0299513 2026-02-23T16:06:58.428 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:06:58.544 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:06:58.567 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:06:58 trial068 ceph-mon[38805]: pgmap v6031: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:06:58.651 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:06:58.649+0000 7fdc1bdd1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:06:58.652 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:06:58.650+0000 7fdc1bdd1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:06:58.759 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:06:58.759 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":363,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:06:55.386038+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:05:59.814738+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:03:29.547807+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:04:45.660145+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:06:58.887 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10412 8 365281 365273 0.274057 0 - 0.0299513 2026-02-23T16:06:59.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:06:59 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/720087441' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:06:59.887 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10413 8 365281 365273 0.27403 0 - 0.0299513 2026-02-23T16:07:00.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:07:00 trial068 ceph-mon[38805]: pgmap v6032: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:07:00.887 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10414 8 365281 365273 0.274004 0 - 0.0299513 2026-02-23T16:07:01.797 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:07:01.888 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10415 8 365281 365273 0.273978 0 - 0.0299513 2026-02-23T16:07:01.914 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:07:02.012 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:07:02.010+0000 7ff5ca585640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:07:02.012 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:07:02.010+0000 7ff5ca585640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:07:02.121 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:07:02.121 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":363,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:06:55.386038+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:05:59.814738+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:03:29.547807+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:04:45.660145+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:07:02.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:07:02 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/4152366630' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:07:02.888 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10416 8 365281 365273 0.273952 0 - 0.0299513 2026-02-23T16:07:03.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:07:03 trial068 ceph-mon[38805]: pgmap v6033: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:07:03.888 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10417 8 365281 365273 0.273925 0 - 0.0299513 2026-02-23T16:07:04.265 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 16:07:03 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T16:07:03.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T16:07:04.312 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 16:07:03 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:16:07:03] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T16:07:04.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:07:04 trial068 ceph-mon[38805]: pgmap v6034: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:07:04.888 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10418 8 365281 365273 0.273899 0 - 0.0299513 2026-02-23T16:07:05.158 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:07:05.274 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:07:05.384 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:07:05.383+0000 7f637c3f2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:07:05.385 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:07:05.383+0000 7f637c3f2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:07:05.494 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:07:05 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T16:07:05.495 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:07:05 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T16:07:05.495 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:07:05.495 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":363,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:06:55.386038+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:05:59.814738+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:03:29.547807+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:04:45.660145+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:07:05.888 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10419 8 365281 365273 0.273873 0 - 0.0299513 2026-02-23T16:07:06.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:07:06 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3972668920' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:07:06.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:07:06 trial068 ceph-mon[38805]: pgmap v6035: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:07:06.888 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T16:07:06.887905+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T16:07:06.888 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T16:07:06.888 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10420 8 365281 365273 0.273846 0 - 0.0299513 2026-02-23T16:07:07.888 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10421 8 365281 365273 0.27382 0 - 0.0299513 2026-02-23T16:07:08.533 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:07:08.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:07:08 trial068 ceph-mon[38805]: pgmap v6036: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:07:08.652 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:07:08.756 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:07:08.754+0000 7f76ae2d9640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:07:08.757 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:07:08.755+0000 7f76ae2d9640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:07:08.866 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:07:08.866 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":363,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:06:55.386038+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:05:59.814738+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:03:29.547807+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:04:45.660145+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:07:08.888 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10422 8 365281 365273 0.273794 0 - 0.0299513 2026-02-23T16:07:09.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:07:09 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3399315667' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:07:09.888 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10423 8 365281 365273 0.273768 0 - 0.0299513 2026-02-23T16:07:10.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:07:10 trial068 ceph-mon[38805]: pgmap v6037: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:07:10.888 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10424 8 365281 365273 0.273741 0 - 0.0299513 2026-02-23T16:07:11.888 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10425 8 365281 365273 0.273715 0 - 0.0299513 2026-02-23T16:07:11.903 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:07:12.018 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:07:12.119 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:07:12.117+0000 7f7b7abc7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:07:12.120 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:07:12.118+0000 7f7b7abc7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:07:12.228 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:07:12.228 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":363,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:06:55.386038+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:05:59.814738+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:03:29.547807+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:04:45.660145+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:07:12.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:07:12 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1713320362' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:07:12.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:07:12 trial068 ceph-mon[38805]: pgmap v6038: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:07:12.888 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10426 8 365281 365273 0.273689 0 - 0.0299513 2026-02-23T16:07:13.888 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10427 8 365281 365273 0.273663 0 - 0.0299513 2026-02-23T16:07:14.313 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 16:07:13 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:16:07:13] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T16:07:14.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 16:07:13 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T16:07:13.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T16:07:14.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:07:14 trial068 ceph-mon[38805]: pgmap v6039: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:07:14.888 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10428 8 365281 365273 0.273636 0 - 0.0299513 2026-02-23T16:07:15.265 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:07:15.385 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:07:15.489 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:07:15.487+0000 7f4d058c8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:07:15.490 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:07:15.488+0000 7f4d058c8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:07:15.600 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:07:15.601 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":363,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:06:55.386038+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:05:59.814738+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:03:29.547807+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:04:45.660145+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:07:15.889 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10429 8 365281 365273 0.27361 0 - 0.0299513 2026-02-23T16:07:16.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:07:15 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2102352658' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:07:16.889 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10430 8 365281 365273 0.273584 0 - 0.0299513 2026-02-23T16:07:17.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:07:16 trial068 ceph-mon[38805]: pgmap v6040: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:07:17.889 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10431 8 365281 365273 0.273558 0 - 0.0299513 2026-02-23T16:07:18.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:07:18 trial068 ceph-mon[38805]: pgmap v6041: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:07:18.639 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:07:18.757 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:07:18.863 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:07:18.861+0000 7f30f5eed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:07:18.863 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:07:18.862+0000 7f30f5eed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:07:18.889 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10432 8 365281 365273 0.273531 0 - 0.0299513 2026-02-23T16:07:18.972 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:07:18.972 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":363,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:06:55.386038+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:05:59.814738+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:03:29.547807+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:04:45.660145+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:07:19.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:07:19 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/648952644' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:07:19.889 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10433 8 365281 365273 0.273505 0 - 0.0299513 2026-02-23T16:07:20.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:07:20 trial068 ceph-mon[38805]: pgmap v6042: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:07:20.889 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10434 8 365281 365273 0.273479 0 - 0.0299513 2026-02-23T16:07:21.889 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10435 8 365281 365273 0.273453 0 - 0.0299513 2026-02-23T16:07:22.010 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:07:22.129 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:07:22.233 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:07:22.232+0000 7fad10c4f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:07:22.234 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:07:22.232+0000 7fad10c4f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:07:22.342 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:07:22.342 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":363,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:06:55.386038+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:05:59.814738+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:03:29.547807+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:04:45.660145+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:07:22.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:07:22 trial068 ceph-mon[38805]: pgmap v6043: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:07:22.889 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10436 8 365281 365273 0.273427 0 - 0.0299513 2026-02-23T16:07:23.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:07:23 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/304650496' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:07:23.889 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10437 8 365281 365273 0.2734 0 - 0.0299513 2026-02-23T16:07:24.315 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 16:07:23 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:16:07:23] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T16:07:24.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 16:07:23 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T16:07:23.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T16:07:24.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:07:24 trial068 ceph-mon[38805]: pgmap v6044: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:07:24.889 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10438 8 365281 365273 0.273374 0 - 0.0299513 2026-02-23T16:07:25.382 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:07:25.502 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:07:25.602 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:07:25.600+0000 7f8f86748640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:07:25.602 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:07:25.600+0000 7f8f86748640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:07:25.711 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:07:25.711 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":363,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:06:55.386038+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:05:59.814738+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:03:29.547807+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:04:45.660145+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:07:25.889 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10439 8 365281 365273 0.273348 0 - 0.0299513 2026-02-23T16:07:26.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:07:26 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/233210842' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:07:26.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:07:26 trial068 ceph-mon[38805]: pgmap v6045: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:07:26.889 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T16:07:26.889246+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T16:07:26.889 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T16:07:26.890 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10440 8 365281 365273 0.273322 0 - 0.0299513 2026-02-23T16:07:27.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:07:27 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T16:07:27.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:07:27 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T16:07:27.889 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10441 8 365281 365273 0.273296 0 - 0.0299513 2026-02-23T16:07:28.750 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:07:28.774 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:07:28 trial068 ceph-mon[38805]: pgmap v6046: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:07:28.866 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:07:28.889 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10442 8 365281 365273 0.273269 0 - 0.0299513 2026-02-23T16:07:28.976 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:07:28.974+0000 7fec74f61640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:07:28.976 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:07:28.974+0000 7fec74f61640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:07:29.085 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:07:29.085 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":363,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:06:55.386038+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:05:59.814738+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:03:29.547807+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:04:45.660145+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:07:29.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:07:29 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1422562017' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:07:29.889 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10443 8 365281 365273 0.273243 0 - 0.0299513 2026-02-23T16:07:30.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:07:30 trial068 ceph-mon[38805]: pgmap v6047: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:07:30.890 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10444 8 365281 365273 0.273217 0 - 0.0299513 2026-02-23T16:07:31.890 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10445 8 365281 365273 0.273191 0 - 0.0299513 2026-02-23T16:07:32.123 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:07:32.238 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:07:32.337 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:07:32.335+0000 7ffae1671640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:07:32.337 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:07:32.335+0000 7ffae1671640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:07:32.448 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:07:32.448 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":363,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:06:55.386038+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:05:59.814738+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:03:29.547807+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:04:45.660145+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:07:32.448 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:07:32 trial068 ceph-mon[38805]: pgmap v6048: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:07:32.890 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10446 8 365281 365273 0.273165 0 - 0.0299513 2026-02-23T16:07:33.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:07:33 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/233149914' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:07:33.890 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10447 8 365281 365273 0.273139 0 - 0.0299513 2026-02-23T16:07:34.265 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 16:07:33 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T16:07:33.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T16:07:34.317 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 16:07:33 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:16:07:33] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T16:07:34.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:07:34 trial068 ceph-mon[38805]: pgmap v6049: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:07:34.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:07:34 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2026-02-23T16:07:34.890 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10448 8 365281 365273 0.273112 0 - 0.0299513 2026-02-23T16:07:35.488 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:07:35.603 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:07:35.705 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:07:35.703+0000 7f3fbed92640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:07:35.705 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:07:35.704+0000 7f3fbed92640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:07:35.706 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:07:35 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T16:07:35.706 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:07:35 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T16:07:35.706 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:07:35 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T16:07:35.706 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:07:35 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T16:07:35.706 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:07:35 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T16:07:35.706 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:07:35 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T16:07:35.706 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:07:35 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-23T16:07:35.706 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:07:35 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2026-02-23T16:07:35.707 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:07:35 trial068 ceph-mon[38805]: pgmap v6050: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:07:35.707 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:07:35 trial068 ceph-mon[38805]: pgmap v6051: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:07:35.707 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:07:35 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T16:07:35.707 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:07:35 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2026-02-23T16:07:35.815 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:07:35.815 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":363,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:06:55.386038+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:05:59.814738+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:03:29.547807+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:04:45.660145+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:07:35.890 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10449 8 365281 365273 0.273086 0 - 0.0299513 2026-02-23T16:07:36.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:07:36 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2150625940' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:07:36.890 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10450 8 365281 365273 0.27306 0 - 0.0299513 2026-02-23T16:07:37.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:07:37 trial068 ceph-mon[38805]: pgmap v6052: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:07:37.890 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10451 8 365281 365273 0.273034 0 - 0.0299513 2026-02-23T16:07:38.852 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:07:38.890 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10452 8 365281 365273 0.273008 0 - 0.0299513 2026-02-23T16:07:38.975 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:07:39.076 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:07:39.073+0000 7fa21c6cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:07:39.076 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:07:39.074+0000 7fa21c6cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:07:39.183 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:07:39.183 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":363,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:06:55.386038+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:05:59.814738+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:03:29.547807+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:04:45.660145+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:07:39.890 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10453 8 365281 365273 0.272982 0 - 0.0299513 2026-02-23T16:07:40.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:07:39 trial068 ceph-mon[38805]: 5.8 scrub starts 2026-02-23T16:07:40.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:07:39 trial068 ceph-mon[38805]: pgmap v6053: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:07:40.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:07:39 trial068 ceph-mon[38805]: 5.8 scrub ok 2026-02-23T16:07:40.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:07:39 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2843732345' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:07:40.890 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10454 8 365281 365273 0.272956 0 - 0.0299513 2026-02-23T16:07:41.890 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10455 8 365281 365273 0.27293 0 - 0.0299513 2026-02-23T16:07:42.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:07:41 trial068 ceph-mon[38805]: pgmap v6054: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:07:42.220 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:07:42.335 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:07:42.437 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:07:42.435+0000 7ff7ed1a2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:07:42.437 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:07:42.435+0000 7ff7ed1a2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:07:42.546 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:07:42.546 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":363,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:06:55.386038+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:05:59.814738+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:03:29.547807+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:04:45.660145+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:07:42.605 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 16:07:42 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T16:07:42.155Z level=INFO source=compact.go:598 msg="write block" component=tsdb mint=1771852061111 maxt=1771855200000 ulid=01KJ5M3A9W4SB74PQKJKP3E542 duration=14.735933ms ooo=false 2026-02-23T16:07:42.605 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 16:07:42 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T16:07:42.156Z level=INFO source=head.go:1420 msg="Head GC completed" component=tsdb caller=truncateMemory duration=1.365992ms 2026-02-23T16:07:42.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:07:42 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/425151611' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:07:42.890 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10456 8 365281 365273 0.272904 0 - 0.0299513 2026-02-23T16:07:43.891 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10457 8 365281 365273 0.272877 0 - 0.0299513 2026-02-23T16:07:43.974 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:07:43 trial068 ceph-mon[38805]: pgmap v6055: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:07:44.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 16:07:43 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:16:07:43] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T16:07:44.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 16:07:43 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T16:07:43.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T16:07:44.891 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10458 8 365281 365273 0.272851 0 - 0.0299513 2026-02-23T16:07:45.583 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:07:45.701 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:07:45.794 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:07:45.792+0000 7f9de8183640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:07:45.794 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:07:45.792+0000 7f9de8183640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:07:45.891 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10459 8 365281 365273 0.272825 0 - 0.0299513 2026-02-23T16:07:45.904 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:07:45.905 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":363,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:06:55.386038+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:05:59.814738+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:03:29.547807+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:04:45.660145+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:07:45.905 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:07:45 trial068 ceph-mon[38805]: pgmap v6056: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:07:45.905 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:07:45 trial068 ceph-mon[38805]: 5.7 scrub starts 2026-02-23T16:07:45.905 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:07:45 trial068 ceph-mon[38805]: 5.7 scrub ok 2026-02-23T16:07:46.891 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T16:07:46.890576+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T16:07:46.891 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T16:07:46.891 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10460 8 365281 365273 0.272799 0 - 0.0299513 2026-02-23T16:07:47.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:07:46 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2265112119' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:07:47.891 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10461 8 365281 365273 0.272773 0 - 0.0299513 2026-02-23T16:07:48.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:07:47 trial068 ceph-mon[38805]: pgmap v6057: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:07:48.891 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10462 8 365281 365273 0.272747 0 - 0.0299513 2026-02-23T16:07:48.956 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:07:49.072 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:07:49.171 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:07:49.170+0000 7fb49af3d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:07:49.171 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:07:49.170+0000 7fb49af3d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:07:49.281 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:07:49.282 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":363,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:06:55.386038+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:05:59.814738+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:03:29.547807+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:04:45.660145+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:07:49.891 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10463 8 365281 365273 0.272721 0 - 0.0299513 2026-02-23T16:07:50.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:07:49 trial068 ceph-mon[38805]: pgmap v6058: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:07:50.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:07:49 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3791830851' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:07:50.891 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10464 8 365281 365273 0.272695 0 - 0.0299513 2026-02-23T16:07:51.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:07:51 trial068 ceph-mon[38805]: pgmap v6059: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:07:51.891 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10465 8 365281 365273 0.272669 0 - 0.0299513 2026-02-23T16:07:52.320 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:07:52.437 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:07:52.540 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:07:52.539+0000 7fd98e5bf640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:07:52.541 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:07:52.539+0000 7fd98e5bf640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:07:52.649 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:07:52.649 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":363,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:06:55.386038+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:05:59.814738+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:03:29.547807+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:04:45.660145+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:07:52.891 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10466 8 365281 365273 0.272643 0 - 0.0299513 2026-02-23T16:07:53.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:07:52 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1813213115' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:07:53.891 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10467 8 365281 365273 0.272617 0 - 0.0299513 2026-02-23T16:07:53.973 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:07:53 trial068 ceph-mon[38805]: pgmap v6060: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:07:54.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 16:07:53 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:16:07:53] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T16:07:54.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 16:07:53 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T16:07:53.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T16:07:54.891 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10468 8 365281 365273 0.272591 0 - 0.0299513 2026-02-23T16:07:55.686 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:07:55.802 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:07:55.891 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10469 8 365281 365273 0.272565 0 - 0.0299513 2026-02-23T16:07:55.906 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:07:55.904+0000 7f2cd1e6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:07:55.906 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:07:55.904+0000 7f2cd1e6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:07:56.016 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:07:56.016 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":363,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:06:55.386038+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:05:59.814738+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:03:29.547807+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:04:45.660145+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:07:56.016 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:07:55 trial068 ceph-mon[38805]: pgmap v6061: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:07:56.891 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10470 8 365281 365273 0.272539 0 - 0.0299513 2026-02-23T16:07:57.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:07:56 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2292487398' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:07:57.891 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10471 8 365281 365273 0.272513 0 - 0.0299513 2026-02-23T16:07:58.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:07:57 trial068 ceph-mon[38805]: pgmap v6062: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:07:58.891 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10472 8 365281 365273 0.272487 0 - 0.0299513 2026-02-23T16:07:59.053 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:07:59.171 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:07:59.273 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:07:59.272+0000 7f8d2b571640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:07:59.273 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:07:59.272+0000 7f8d2b571640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:07:59.383 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:07:59.383 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":363,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:06:55.386038+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:05:59.814738+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:03:29.547807+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:04:45.660145+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:07:59.892 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10473 8 365281 365273 0.272461 0 - 0.0299513 2026-02-23T16:08:00.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:07:59 trial068 ceph-mon[38805]: pgmap v6063: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:08:00.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:07:59 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1020815184' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:08:00.892 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10474 8 365281 365273 0.272435 0 - 0.0299513 2026-02-23T16:08:01.892 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10475 8 365281 365273 0.272409 0 - 0.0299513 2026-02-23T16:08:02.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:08:01 trial068 ceph-mon[38805]: pgmap v6064: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:08:02.423 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:08:02.544 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:08:02.644 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:08:02.642+0000 7fb5a2885640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:08:02.644 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:08:02.643+0000 7fb5a2885640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:08:02.753 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:08:02.753 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":363,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:06:55.386038+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:05:59.814738+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:03:29.547807+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:04:45.660145+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:08:02.892 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10476 8 365281 365273 0.272383 0 - 0.0299513 2026-02-23T16:08:03.892 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10477 8 365281 365273 0.272357 0 - 0.0299513 2026-02-23T16:08:04.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:08:03 trial068 ceph-mon[38805]: pgmap v6065: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:08:04.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:08:03 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1639043500' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:08:04.094 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 16:08:03 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:16:08:03] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T16:08:04.265 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 16:08:03 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T16:08:03.974Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T16:08:04.892 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10478 8 365281 365273 0.272331 0 - 0.0299513 2026-02-23T16:08:05.792 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:08:05.892 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10479 8 365281 365273 0.272305 0 - 0.0299513 2026-02-23T16:08:05.909 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:08:06.006 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:08:06.004+0000 7f604f9eb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:08:06.006 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:08:05 trial068 ceph-mon[38805]: 5.a scrub starts 2026-02-23T16:08:06.006 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:08:05 trial068 ceph-mon[38805]: 5.a scrub ok 2026-02-23T16:08:06.006 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:08:05 trial068 ceph-mon[38805]: pgmap v6066: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:08:06.006 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:08:06.004+0000 7f604f9eb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:08:06.116 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:08:06.116 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":363,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:06:55.386038+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:05:59.814738+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:03:29.547807+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:04:45.660145+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:08:06.892 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T16:08:06.892009+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T16:08:06.892 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T16:08:06.892 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10480 8 365281 365273 0.272279 0 - 0.0299513 2026-02-23T16:08:07.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:08:06 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/12073303' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:08:07.892 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10481 8 365281 365273 0.272253 0 - 0.0299513 2026-02-23T16:08:08.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:08:07 trial068 ceph-mon[38805]: pgmap v6067: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:08:08.892 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10482 8 365281 365273 0.272227 0 - 0.0299513 2026-02-23T16:08:09.153 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:08:09.269 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:08:09.364 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:08:09.363+0000 7f3c8c9cd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:08:09.365 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:08:09.363+0000 7f3c8c9cd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:08:09.473 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:08:09.473 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":363,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:06:55.386038+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:05:59.814738+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:03:29.547807+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:04:45.660145+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:08:09.892 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10483 8 365281 365273 0.272201 0 - 0.0299513 2026-02-23T16:08:10.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:08:09 trial068 ceph-mon[38805]: pgmap v6068: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:08:10.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:08:09 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1362961070' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:08:10.892 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10484 8 365281 365273 0.272175 0 - 0.0299513 2026-02-23T16:08:11.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:08:11 trial068 ceph-mon[38805]: pgmap v6069: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:08:11.892 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10485 8 365281 365273 0.272149 0 - 0.0299513 2026-02-23T16:08:12.510 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:08:12.625 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:08:12.732 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:08:12.730+0000 7ffb95a15640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:08:12.732 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:08:12.730+0000 7ffb95a15640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:08:12.841 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:08:12.841 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":363,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:06:55.386038+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:05:59.814738+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:03:29.547807+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:04:45.660145+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:08:12.892 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10486 8 365281 365273 0.272123 0 - 0.0299513 2026-02-23T16:08:13.893 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10487 8 365281 365273 0.272097 0 - 0.0299513 2026-02-23T16:08:13.974 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:08:13 trial068 ceph-mon[38805]: pgmap v6070: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:08:13.974 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:08:13 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1579057095' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:08:14.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 16:08:13 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:16:08:13] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T16:08:14.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 16:08:13 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T16:08:13.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T16:08:14.893 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10488 8 365281 365273 0.272071 0 - 0.0299513 2026-02-23T16:08:15.880 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:08:15.893 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10489 8 365281 365273 0.272045 0 - 0.0299513 2026-02-23T16:08:15.999 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:08:16.022 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:08:15 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T16:08:16.022 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:08:15 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T16:08:16.022 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:08:15 trial068 ceph-mon[38805]: pgmap v6071: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:08:16.103 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:08:16.102+0000 7f229fd24640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:08:16.104 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:08:16.102+0000 7f229fd24640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:08:16.213 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:08:16.213 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":363,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:06:55.386038+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:05:59.814738+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:03:29.547807+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:04:45.660145+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:08:16.893 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10490 8 365281 365273 0.272019 0 - 0.0299513 2026-02-23T16:08:17.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:08:16 trial068 ceph-mon[38805]: 5.e scrub starts 2026-02-23T16:08:17.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:08:16 trial068 ceph-mon[38805]: 5.e scrub ok 2026-02-23T16:08:17.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:08:16 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1742148344' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:08:17.893 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10491 8 365281 365273 0.271993 0 - 0.0299513 2026-02-23T16:08:18.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:08:17 trial068 ceph-mon[38805]: 5.5 scrub starts 2026-02-23T16:08:18.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:08:17 trial068 ceph-mon[38805]: 5.5 scrub ok 2026-02-23T16:08:18.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:08:17 trial068 ceph-mon[38805]: pgmap v6072: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:08:18.893 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10492 8 365281 365273 0.271967 0 - 0.0299513 2026-02-23T16:08:19.251 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:08:19.368 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:08:19.469 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:08:19.467+0000 7f318303f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:08:19.469 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:08:19.467+0000 7f318303f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:08:19.579 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:08:19.579 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":363,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:06:55.386038+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:05:59.814738+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:03:29.547807+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:04:45.660145+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:08:19.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:08:19 trial068 ceph-mon[38805]: pgmap v6073: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:08:19.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:08:19 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1532599340' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:08:19.893 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10493 8 365281 365273 0.271941 0 - 0.0299513 2026-02-23T16:08:20.893 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10494 8 365281 365273 0.271915 0 - 0.0299513 2026-02-23T16:08:21.893 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10495 8 365281 365273 0.271889 0 - 0.0299513 2026-02-23T16:08:22.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:08:21 trial068 ceph-mon[38805]: pgmap v6074: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:08:22.616 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:08:22.732 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:08:22.835 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:08:22.833+0000 7f3c1b059640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:08:22.835 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:08:22.833+0000 7f3c1b059640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:08:22.893 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10496 8 365281 365273 0.271863 0 - 0.0299513 2026-02-23T16:08:22.944 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:08:22.945 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":363,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:06:55.386038+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:05:59.814738+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:03:29.547807+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:04:45.660145+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:08:23.893 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10497 8 365281 365273 0.271838 0 - 0.0299513 2026-02-23T16:08:24.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:08:23 trial068 ceph-mon[38805]: pgmap v6075: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:08:24.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:08:23 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/952494957' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:08:24.094 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 16:08:23 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:16:08:23] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T16:08:24.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 16:08:23 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T16:08:23.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T16:08:24.893 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10498 8 365281 365273 0.271812 0 - 0.0299513 2026-02-23T16:08:25.893 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10499 8 365281 365273 0.271786 0 - 0.0299513 2026-02-23T16:08:25.981 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:08:26.004 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:08:25 trial068 ceph-mon[38805]: pgmap v6076: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:08:26.096 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:08:26.201 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:08:26.199+0000 7ff2f2e43640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:08:26.201 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:08:26.200+0000 7ff2f2e43640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:08:26.312 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:08:26.312 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":363,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:06:55.386038+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:05:59.814738+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:03:29.547807+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:04:45.660145+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:08:26.894 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T16:08:26.893419+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T16:08:26.894 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T16:08:26.894 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10500 8 365281 365273 0.27176 0 - 0.0299513 2026-02-23T16:08:27.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:08:26 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1827577869' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:08:27.894 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10501 8 365281 365273 0.271734 0 - 0.0299513 2026-02-23T16:08:28.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:08:27 trial068 ceph-mon[38805]: pgmap v6077: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:08:28.894 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10502 8 365281 365273 0.271708 0 - 0.0299513 2026-02-23T16:08:29.350 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:08:29.469 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:08:29.567 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:08:29.565+0000 7f7e311a1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:08:29.567 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:08:29.565+0000 7f7e311a1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:08:29.676 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:08:29.676 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":363,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:06:55.386038+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:05:59.814738+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:03:29.547807+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:04:45.660145+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:08:29.894 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10503 8 365281 365273 0.271682 0 - 0.0299513 2026-02-23T16:08:30.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:08:29 trial068 ceph-mon[38805]: pgmap v6078: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:08:30.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:08:29 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3012184474' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:08:30.894 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10504 8 365281 365273 0.271656 0 - 0.0299513 2026-02-23T16:08:31.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:08:31 trial068 ceph-mon[38805]: pgmap v6079: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:08:31.894 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10505 8 365281 365273 0.271631 0 - 0.0299513 2026-02-23T16:08:32.713 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:08:32.832 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:08:32.894 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10506 8 365281 365273 0.271605 0 - 0.0299513 2026-02-23T16:08:32.930 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:08:32.928+0000 7f8bd2d2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:08:32.930 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:08:32.928+0000 7f8bd2d2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:08:33.039 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:08:33.040 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":363,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:06:55.386038+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:05:59.814738+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:03:29.547807+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:04:45.660145+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:08:33.894 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10507 8 365281 365273 0.271579 0 - 0.0299513 2026-02-23T16:08:33.974 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:08:33 trial068 ceph-mon[38805]: pgmap v6080: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:08:33.974 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:08:33 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3606224848' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:08:34.265 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 16:08:33 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T16:08:33.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T16:08:34.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 16:08:33 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:16:08:33] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T16:08:34.894 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10508 8 365281 365273 0.271553 0 - 0.0299513 2026-02-23T16:08:35.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:08:34 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2026-02-23T16:08:35.894 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10509 8 365281 365273 0.271527 0 - 0.0299513 2026-02-23T16:08:36.078 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:08:36.194 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:08:36.302 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:08:36.300+0000 7f4a81257640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:08:36.302 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:08:36.301+0000 7f4a81257640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:08:36.414 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:08:36.414 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":363,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:06:55.386038+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:05:59.814738+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:03:29.547807+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:04:45.660145+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:08:36.414 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:08:36 trial068 ceph-mon[38805]: pgmap v6081: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:08:36.414 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:08:36 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T16:08:36.415 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:08:36 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T16:08:36.415 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:08:36 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T16:08:36.415 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:08:36 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T16:08:36.415 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:08:36 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T16:08:36.415 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:08:36 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T16:08:36.415 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:08:36 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T16:08:36.415 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:08:36 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T16:08:36.894 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10510 8 365281 365273 0.271501 0 - 0.0299513 2026-02-23T16:08:37.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:08:37 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2182298130' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:08:37.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:08:37 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T16:08:37.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:08:37 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T16:08:37.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:08:37 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T16:08:37.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:08:37 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T16:08:37.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:08:37 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-23T16:08:37.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:08:37 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2026-02-23T16:08:37.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:08:37 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T16:08:37.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:08:37 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2026-02-23T16:08:37.894 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10511 8 365281 365273 0.271476 0 - 0.0299513 2026-02-23T16:08:38.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:08:38 trial068 ceph-mon[38805]: pgmap v6082: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:08:38.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:08:38 trial068 ceph-mon[38805]: pgmap v6083: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:08:38.894 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10512 8 365281 365273 0.27145 0 - 0.0299513 2026-02-23T16:08:39.453 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:08:39.569 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:08:39.670 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:08:39.669+0000 7fd5b3851640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:08:39.670 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:08:39.669+0000 7fd5b3851640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:08:39.780 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:08:39.780 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":363,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:06:55.386038+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:05:59.814738+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:03:29.547807+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:04:45.660145+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:08:39.894 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10513 8 365281 365273 0.271424 0 - 0.0299513 2026-02-23T16:08:40.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:08:40 trial068 ceph-mon[38805]: pgmap v6084: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:08:40.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:08:40 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2689518580' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:08:40.895 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10514 8 365281 365273 0.271398 0 - 0.0299513 2026-02-23T16:08:41.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:08:41 trial068 ceph-mon[38805]: pgmap v6085: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:08:41.895 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10515 8 365281 365273 0.271372 0 - 0.0299513 2026-02-23T16:08:42.816 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:08:42.895 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10516 8 365281 365273 0.271346 0 - 0.0299513 2026-02-23T16:08:42.937 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:08:43.034 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:08:43.032+0000 7fe4d11ed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:08:43.035 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:08:43.033+0000 7fe4d11ed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:08:43.143 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:08:43.144 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":363,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:06:55.386038+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:05:59.814738+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:03:29.547807+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:04:45.660145+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:08:43.895 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10517 8 365281 365273 0.271321 0 - 0.0299513 2026-02-23T16:08:43.974 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:08:43 trial068 ceph-mon[38805]: pgmap v6086: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:08:43.974 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:08:43 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1543189687' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:08:44.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 16:08:43 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:16:08:43] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T16:08:44.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 16:08:43 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T16:08:43.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T16:08:44.895 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10518 8 365281 365273 0.271295 0 - 0.0299513 2026-02-23T16:08:45.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:08:44 trial068 ceph-mon[38805]: 5.8 scrub starts 2026-02-23T16:08:45.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:08:44 trial068 ceph-mon[38805]: 5.8 scrub ok 2026-02-23T16:08:45.895 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10519 8 365281 365273 0.271269 0 - 0.0299513 2026-02-23T16:08:46.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:08:45 trial068 ceph-mon[38805]: pgmap v6087: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:08:46.180 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:08:46.299 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:08:46.406 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:08:46.404+0000 7f115763d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:08:46.406 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:08:46.404+0000 7f115763d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:08:46.515 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:08:46.515 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":363,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:06:55.386038+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:05:59.814738+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:03:29.547807+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:04:45.660145+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:08:46.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:08:46 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1064147972' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:08:46.895 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T16:08:46.894941+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T16:08:46.895 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T16:08:46.895 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10520 8 365281 365273 0.271243 0 - 0.0299513 2026-02-23T16:08:47.895 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10521 8 365281 365273 0.271217 0 - 0.0299513 2026-02-23T16:08:48.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:08:47 trial068 ceph-mon[38805]: pgmap v6088: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:08:48.895 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10522 8 365281 365273 0.271192 0 - 0.0299513 2026-02-23T16:08:49.554 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:08:49.671 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:08:49.767 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:08:49.765+0000 7fe9e83ed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:08:49.767 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:08:49.765+0000 7fe9e83ed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:08:49.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:08:49 trial068 ceph-mon[38805]: pgmap v6089: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:08:49.876 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:08:49.876 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":363,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:06:55.386038+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:05:59.814738+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:03:29.547807+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:04:45.660145+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:08:49.895 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10523 8 365281 365273 0.271166 0 - 0.0299513 2026-02-23T16:08:50.895 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10524 8 365281 365273 0.27114 0 - 0.0299513 2026-02-23T16:08:51.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:08:50 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/186004278' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:08:51.895 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10525 8 365281 365273 0.271114 0 - 0.0299513 2026-02-23T16:08:52.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:08:51 trial068 ceph-mon[38805]: pgmap v6090: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:08:52.896 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10526 8 365281 365273 0.271089 0 - 0.0299513 2026-02-23T16:08:52.913 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:08:53.030 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:08:53.125 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:08:53.123+0000 7f0260ded640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:08:53.126 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:08:53.124+0000 7f0260ded640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:08:53.237 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:08:53.238 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":363,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:06:55.386038+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:05:59.814738+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:03:29.547807+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:04:45.660145+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:08:53.896 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10527 8 365281 365273 0.271063 0 - 0.0299513 2026-02-23T16:08:53.974 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:08:53 trial068 ceph-mon[38805]: pgmap v6091: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:08:53.974 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:08:53 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1380671932' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:08:54.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 16:08:53 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:16:08:53] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T16:08:54.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 16:08:53 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T16:08:53.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T16:08:54.896 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10528 8 365281 365273 0.271037 0 - 0.0299513 2026-02-23T16:08:55.896 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10529 8 365281 365273 0.271011 0 - 0.0299513 2026-02-23T16:08:56.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:08:55 trial068 ceph-mon[38805]: pgmap v6092: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:08:56.275 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:08:56.394 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:08:56.496 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:08:56.494+0000 7f567442e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:08:56.496 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:08:56.494+0000 7f567442e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:08:56.606 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:08:56.607 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":363,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:06:55.386038+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:05:59.814738+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:03:29.547807+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:04:45.660145+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:08:56.896 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10530 8 365281 365273 0.270986 0 - 0.0299513 2026-02-23T16:08:57.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:08:56 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T16:08:57.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:08:56 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T16:08:57.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:08:56 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3596761625' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:08:57.896 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10531 8 365281 365273 0.27096 0 - 0.0299513 2026-02-23T16:08:58.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:08:57 trial068 ceph-mon[38805]: pgmap v6093: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:08:58.896 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10532 8 365281 365273 0.270934 0 - 0.0299513 2026-02-23T16:08:59.645 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:08:59.764 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:08:59.862 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:08:59.860+0000 7f4c33e2e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:08:59.862 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:08:59.860+0000 7f4c33e2e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:08:59.878 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:08:59 trial068 ceph-mon[38805]: pgmap v6094: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:08:59.896 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10533 8 365281 365273 0.270908 0 - 0.0299513 2026-02-23T16:08:59.974 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:08:59.974 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":363,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:06:55.386038+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:05:59.814738+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:03:29.547807+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:04:45.660145+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:09:00.896 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10534 8 365281 365273 0.270883 0 - 0.0299513 2026-02-23T16:09:01.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:09:00 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1471278977' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:09:01.896 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10535 8 365281 365273 0.270857 0 - 0.0299513 2026-02-23T16:09:02.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:09:01 trial068 ceph-mon[38805]: pgmap v6095: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:09:02.896 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10536 8 365281 365273 0.270831 0 - 0.0299513 2026-02-23T16:09:03.011 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:09:03.035 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:09:02 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T16:09:03.036 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:09:02 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T16:09:03.129 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:09:03.230 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:09:03.228+0000 7f518f804640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:09:03.230 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:09:03.228+0000 7f518f804640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:09:03.339 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:09:03.339 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":363,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:06:55.386038+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:05:59.814738+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:03:29.547807+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:04:45.660145+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:09:03.896 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10537 8 365281 365273 0.270806 0 - 0.0299513 2026-02-23T16:09:03.974 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:09:03 trial068 ceph-mon[38805]: pgmap v6096: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:09:03.974 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:09:03 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1766922494' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:09:04.265 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 16:09:03 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T16:09:03.974Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T16:09:04.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 16:09:03 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:16:09:03] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T16:09:04.896 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10538 8 365281 365273 0.27078 0 - 0.0299513 2026-02-23T16:09:05.896 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10539 8 365281 365273 0.270754 0 - 0.0299513 2026-02-23T16:09:06.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:09:05 trial068 ceph-mon[38805]: pgmap v6097: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:09:06.376 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:09:06.498 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:09:06.596 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:09:06.594+0000 7f4f43316640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:09:06.596 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:09:06.594+0000 7f4f43316640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:09:06.707 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:09:06.707 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":363,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:06:55.386038+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:05:59.814738+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:03:29.547807+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:04:45.660145+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:09:06.708 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:09:06 trial068 ceph-mon[38805]: 5.7 scrub starts 2026-02-23T16:09:06.708 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:09:06 trial068 ceph-mon[38805]: 5.7 scrub ok 2026-02-23T16:09:06.896 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T16:09:06.896311+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T16:09:06.897 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T16:09:06.897 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10540 8 365281 365273 0.270729 0 - 0.0299513 2026-02-23T16:09:07.897 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10541 8 365281 365273 0.270703 0 - 0.0299513 2026-02-23T16:09:08.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:09:07 trial068 ceph-mon[38805]: pgmap v6098: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:09:08.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:09:07 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2435267187' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:09:08.897 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10542 8 365281 365273 0.270677 0 - 0.0299513 2026-02-23T16:09:09.745 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:09:09.862 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:09:09.883 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:09:09 trial068 ceph-mon[38805]: pgmap v6099: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:09:09.897 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10543 8 365281 365273 0.270652 0 - 0.0299513 2026-02-23T16:09:09.960 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:09:09.958+0000 7ff8b81e8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:09:09.960 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:09:09.958+0000 7ff8b81e8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:09:10.070 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:09:10.070 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":363,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:06:55.386038+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:05:59.814738+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:03:29.547807+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:04:45.660145+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:09:10.897 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10544 8 365281 365273 0.270626 0 - 0.0299513 2026-02-23T16:09:11.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:09:10 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1397829885' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:09:11.897 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10545 8 365281 365273 0.2706 0 - 0.0299513 2026-02-23T16:09:12.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:09:11 trial068 ceph-mon[38805]: pgmap v6100: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:09:12.897 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10546 8 365281 365273 0.270575 0 - 0.0299513 2026-02-23T16:09:13.107 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:09:13.224 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:09:13.329 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:09:13.327+0000 7f988573f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:09:13.330 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:09:13.328+0000 7f988573f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:09:13.439 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:09:13.440 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":363,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:06:55.386038+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:05:59.814738+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:03:29.547807+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:04:45.660145+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:09:13.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:09:13 trial068 ceph-mon[38805]: pgmap v6101: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:09:13.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:09:13 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/4232944493' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:09:13.897 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10547 8 365281 365273 0.270549 0 - 0.0299513 2026-02-23T16:09:14.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 16:09:13 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:16:09:13] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T16:09:14.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 16:09:13 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T16:09:13.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T16:09:14.897 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10548 8 365281 365273 0.270523 0 - 0.0299513 2026-02-23T16:09:15.897 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10549 8 365281 365273 0.270498 0 - 0.0299513 2026-02-23T16:09:16.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:09:15 trial068 ceph-mon[38805]: pgmap v6102: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:09:16.477 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:09:16.595 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:09:16.693 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:09:16.691+0000 7f8996b43640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:09:16.693 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:09:16.691+0000 7f8996b43640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:09:16.802 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:09:16.802 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":363,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:06:55.386038+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:05:59.814738+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:03:29.547807+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:04:45.660145+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:09:16.897 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10550 8 365281 365273 0.270472 0 - 0.0299513 2026-02-23T16:09:17.897 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10551 8 365281 365273 0.270446 0 - 0.0299513 2026-02-23T16:09:18.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:09:17 trial068 ceph-mon[38805]: pgmap v6103: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:09:18.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:09:17 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1882035289' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:09:18.897 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10552 8 365281 365273 0.270421 0 - 0.0299513 2026-02-23T16:09:19.840 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:09:19.897 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10553 8 365281 365273 0.270395 0 - 0.0299513 2026-02-23T16:09:19.956 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:09:19.979 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:09:19 trial068 ceph-mon[38805]: pgmap v6104: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:09:20.059 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:09:20.057+0000 7f08c6e28640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:09:20.059 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:09:20.057+0000 7f08c6e28640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:09:20.168 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:09:20.169 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":363,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:06:55.386038+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:05:59.814738+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:03:29.547807+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:04:45.660145+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:09:20.898 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10554 8 365281 365273 0.270369 0 - 0.0299513 2026-02-23T16:09:21.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:09:20 trial068 ceph-mon[38805]: 5.a scrub starts 2026-02-23T16:09:21.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:09:20 trial068 ceph-mon[38805]: 5.a scrub ok 2026-02-23T16:09:21.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:09:20 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2338666340' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:09:21.898 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10555 8 365281 365273 0.270344 0 - 0.0299513 2026-02-23T16:09:22.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:09:21 trial068 ceph-mon[38805]: pgmap v6105: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:09:22.898 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10556 8 365281 365273 0.270318 0 - 0.0299513 2026-02-23T16:09:23.204 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:09:23.323 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:09:23.425 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:09:23.423+0000 7f4654750640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:09:23.425 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:09:23.423+0000 7f4654750640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:09:23.535 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:09:23.535 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":363,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:06:55.386038+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:05:59.814738+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:03:29.547807+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:04:45.660145+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:09:23.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:09:23 trial068 ceph-mon[38805]: pgmap v6106: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:09:23.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:09:23 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2271475048' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:09:23.898 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10557 8 365281 365273 0.270293 0 - 0.0299513 2026-02-23T16:09:24.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 16:09:23 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:16:09:23] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T16:09:24.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 16:09:23 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T16:09:23.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T16:09:24.898 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10558 8 365281 365273 0.270267 0 - 0.0299513 2026-02-23T16:09:25.898 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10559 8 365281 365273 0.270241 0 - 0.0299513 2026-02-23T16:09:26.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:09:25 trial068 ceph-mon[38805]: pgmap v6107: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:09:26.572 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:09:26.690 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:09:26.792 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:09:26.790+0000 7f36dddb2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:09:26.792 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:09:26.790+0000 7f36dddb2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:09:26.898 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T16:09:26.897814+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T16:09:26.898 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T16:09:26.898 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10560 8 365281 365273 0.270216 0 - 0.0299513 2026-02-23T16:09:26.901 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:09:26.901 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":363,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:06:55.386038+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:05:59.814738+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:03:29.547807+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:04:45.660145+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:09:26.901 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:09:26 trial068 ceph-mon[38805]: 5.e scrub starts 2026-02-23T16:09:26.902 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:09:26 trial068 ceph-mon[38805]: 5.e scrub ok 2026-02-23T16:09:27.898 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10561 8 365281 365273 0.27019 0 - 0.0299513 2026-02-23T16:09:28.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:09:27 trial068 ceph-mon[38805]: pgmap v6108: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:09:28.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:09:27 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3131034739' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:09:28.898 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10562 8 365281 365273 0.270165 0 - 0.0299513 2026-02-23T16:09:29.898 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10563 8 365281 365273 0.270139 0 - 0.0299513 2026-02-23T16:09:29.938 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:09:29.963 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:09:29 trial068 ceph-mon[38805]: pgmap v6109: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:09:29.963 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:09:29 trial068 ceph-mon[38805]: 5.5 scrub starts 2026-02-23T16:09:29.963 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:09:29 trial068 ceph-mon[38805]: 5.5 scrub ok 2026-02-23T16:09:30.055 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:09:30.156 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:09:30.154+0000 7f17c747e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:09:30.156 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:09:30.154+0000 7f17c747e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:09:30.268 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:09:30.268 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":363,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:06:55.386038+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:05:59.814738+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:03:29.547807+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:04:45.660145+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:09:30.898 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10564 8 365281 365273 0.270114 0 - 0.0299513 2026-02-23T16:09:31.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:09:30 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/111654113' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:09:31.898 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10565 8 365281 365273 0.270088 0 - 0.0299513 2026-02-23T16:09:32.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:09:31 trial068 ceph-mon[38805]: pgmap v6110: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:09:32.898 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10566 8 365281 365273 0.270062 0 - 0.0299513 2026-02-23T16:09:33.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:09:32 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T16:09:33.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:09:32 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T16:09:33.307 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:09:33.429 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:09:33.527 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:09:33.526+0000 7f88fb34d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:09:33.528 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:09:33.526+0000 7f88fb34d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:09:33.641 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:09:33.641 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":364,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:09:32.651330+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:05:59.814738+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:08:55.803671+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:04:45.660145+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:09:33.898 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10567 8 365281 365273 0.270037 0 - 0.0299513 2026-02-23T16:09:33.973 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:09:33 trial068 ceph-mon[38805]: pgmap v6111: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:09:33.973 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:09:33 trial068 ceph-mon[38805]: osdmap e364: 4 total, 3 up, 4 in 2026-02-23T16:09:33.974 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:09:33 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3482333093' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:09:34.265 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 16:09:33 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T16:09:33.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T16:09:34.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 16:09:33 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:16:09:33] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T16:09:34.899 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10568 8 365281 365273 0.270011 0 - 0.0299513 2026-02-23T16:09:35.899 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10569 8 365281 365273 0.269986 0 - 0.0299513 2026-02-23T16:09:36.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:09:35 trial068 ceph-mon[38805]: pgmap v6113: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:09:36.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:09:35 trial068 ceph-mon[38805]: osdmap e365: 4 total, 3 up, 4 in 2026-02-23T16:09:36.679 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:09:36.797 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:09:36.899 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10570 8 365281 365273 0.26996 0 - 0.0299513 2026-02-23T16:09:36.904 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:09:36.902+0000 7f567f46b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:09:36.904 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:09:36.903+0000 7f567f46b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:09:36.911 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:09:36 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2026-02-23T16:09:37.014 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:09:37.014 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":365,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:09:35.408622+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:05:59.814738+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:08:55.803671+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:04:45.660145+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:09:37.899 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10571 8 365281 365273 0.269935 0 - 0.0299513 2026-02-23T16:09:38.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:09:37 trial068 ceph-mon[38805]: pgmap v6115: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:09:38.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:09:37 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-23T16:09:38.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:09:37 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2026-02-23T16:09:38.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:09:37 trial068 ceph-mon[38805]: pgmap v6116: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:09:38.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:09:37 trial068 ceph-mon[38805]: pgmap v6117: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:09:38.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:09:37 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T16:09:38.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:09:37 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2026-02-23T16:09:38.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:09:37 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1598291345' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:09:38.899 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10572 8 365281 365273 0.269909 0 - 0.0299513 2026-02-23T16:09:39.899 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10573 8 365281 365273 0.269884 0 - 0.0299513 2026-02-23T16:09:40.067 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:09:40.186 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:09:40.208 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:09:39 trial068 ceph-mon[38805]: pgmap v6118: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:09:40.283 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:09:40.281+0000 7f9b6bdd3640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:09:40.283 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:09:40.282+0000 7f9b6bdd3640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:09:40.393 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:09:40.394 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":365,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:09:35.408622+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:05:59.814738+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:08:55.803671+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:04:45.660145+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:09:40.899 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10574 8 365281 365273 0.269858 0 - 0.0299513 2026-02-23T16:09:41.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:09:40 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2706292326' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:09:41.899 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10575 8 365281 365273 0.269833 0 - 0.0299513 2026-02-23T16:09:42.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:09:41 trial068 ceph-mon[38805]: pgmap v6119: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:09:42.899 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10576 8 365281 365273 0.269807 0 - 0.0299513 2026-02-23T16:09:43.431 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:09:43.549 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:09:43.649 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:09:43.647+0000 7f7791110640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:09:43.650 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:09:43.648+0000 7f7791110640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:09:43.758 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:09:43.758 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":365,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:09:35.408622+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:05:59.814738+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:08:55.803671+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:04:45.660145+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:09:43.899 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10577 8 365281 365273 0.269782 0 - 0.0299513 2026-02-23T16:09:44.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:09:43 trial068 ceph-mon[38805]: pgmap v6120: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:09:44.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:09:43 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/603123133' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:09:44.094 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 16:09:43 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:16:09:43] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T16:09:44.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 16:09:43 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T16:09:43.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T16:09:44.899 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10578 8 365281 365273 0.269756 0 - 0.0299513 2026-02-23T16:09:45.899 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10579 8 365281 365273 0.269731 0 - 0.0299513 2026-02-23T16:09:46.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:09:45 trial068 ceph-mon[38805]: pgmap v6121: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:09:46.795 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:09:46.899 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T16:09:46.899268+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T16:09:46.900 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T16:09:46.900 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10580 8 365281 365273 0.269705 0 - 0.0299513 2026-02-23T16:09:46.915 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:09:47.014 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:09:47.012+0000 7efcb82f9640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:09:47.014 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:09:47.012+0000 7efcb82f9640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:09:47.123 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:09:47.123 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":365,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:09:35.408622+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:05:59.814738+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:08:55.803671+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:04:45.660145+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:09:47.899 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10581 8 365281 365273 0.26968 0 - 0.0299513 2026-02-23T16:09:48.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:09:47 trial068 ceph-mon[38805]: pgmap v6122: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:09:48.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:09:47 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1779103839' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:09:48.899 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10582 8 365281 365273 0.269654 0 - 0.0299513 2026-02-23T16:09:49.900 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10583 8 365281 365273 0.269629 0 - 0.0299513 2026-02-23T16:09:50.161 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:09:50.185 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:09:49 trial068 ceph-mon[38805]: pgmap v6123: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:09:50.277 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:09:50.383 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:09:50.381+0000 7f81e7792640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:09:50.383 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:09:50.381+0000 7f81e7792640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:09:50.492 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:09:50.493 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":365,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:09:35.408622+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:05:59.814738+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:08:55.803671+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:04:45.660145+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:09:50.900 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10584 8 365281 365273 0.269603 0 - 0.0299513 2026-02-23T16:09:51.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:09:50 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2453846619' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:09:51.900 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10585 8 365281 365273 0.269578 0 - 0.0299513 2026-02-23T16:09:52.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:09:51 trial068 ceph-mon[38805]: pgmap v6124: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:09:52.900 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10586 8 365281 365273 0.269552 0 - 0.0299513 2026-02-23T16:09:53.531 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:09:53.649 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:09:53.748 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:09:53.746+0000 7f5fd7909640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:09:53.748 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:09:53.747+0000 7f5fd7909640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:09:53.858 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:09:53.859 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":365,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:09:35.408622+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:05:59.814738+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:08:55.803671+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:04:45.660145+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:09:53.900 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10587 8 365281 365273 0.269527 0 - 0.0299513 2026-02-23T16:09:54.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:09:53 trial068 ceph-mon[38805]: pgmap v6125: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:09:54.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:09:53 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/35260555' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:09:54.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 16:09:53 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:16:09:53] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T16:09:54.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 16:09:53 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T16:09:53.972Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T16:09:54.900 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10588 8 365281 365273 0.269501 0 - 0.0299513 2026-02-23T16:09:55.900 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10589 8 365281 365273 0.269476 0 - 0.0299513 2026-02-23T16:09:56.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:09:55 trial068 ceph-mon[38805]: pgmap v6126: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:09:56.896 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:09:56.900 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10590 8 365281 365273 0.26945 0 - 0.0299513 2026-02-23T16:09:57.014 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:09:57.117 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:09:57.115+0000 7f33908c2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:09:57.117 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:09:57.115+0000 7f33908c2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:09:57.225 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:09:57.225 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":365,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:09:35.408622+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:05:59.814738+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:08:55.803671+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:04:45.660145+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:09:57.900 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10591 8 365281 365273 0.269425 0 - 0.0299513 2026-02-23T16:09:58.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:09:57 trial068 ceph-mon[38805]: pgmap v6127: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:09:58.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:09:57 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2689963086' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:09:58.900 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10592 8 365281 365273 0.269399 0 - 0.0299513 2026-02-23T16:09:59.900 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10593 8 365281 365273 0.269374 0 - 0.0299513 2026-02-23T16:10:00.145 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:09:59 trial068 ceph-mon[38805]: pgmap v6128: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:10:00.263 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:10:00.381 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:10:00.481 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:10:00.479+0000 7fcdaa2e7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:10:00.482 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:10:00.480+0000 7fcdaa2e7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:10:00.591 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:10:00.591 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":365,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:09:35.408622+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:05:59.814738+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:08:55.803671+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:04:45.660145+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:10:00.900 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10594 8 365281 365273 0.269349 0 - 0.0299513 2026-02-23T16:10:01.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:10:00 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3970212371' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:10:01.900 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10595 8 365281 365273 0.269323 0 - 0.0299513 2026-02-23T16:10:02.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:10:01 trial068 ceph-mon[38805]: 5.8 scrub starts 2026-02-23T16:10:02.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:10:01 trial068 ceph-mon[38805]: 5.8 scrub ok 2026-02-23T16:10:02.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:10:01 trial068 ceph-mon[38805]: pgmap v6129: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:10:02.900 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10596 8 365281 365273 0.269298 0 - 0.0299513 2026-02-23T16:10:03.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:10:02 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T16:10:03.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:10:02 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T16:10:03.643 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:10:03.762 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:10:03.865 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:10:03.863+0000 7f4d94124640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:10:03.865 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:10:03.863+0000 7f4d94124640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:10:03.901 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10597 8 365281 365273 0.269272 0 - 0.0299513 2026-02-23T16:10:03.973 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:10:03 trial068 ceph-mon[38805]: pgmap v6130: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:10:03.979 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:10:03.980 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":365,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:09:35.408622+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:05:59.814738+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:08:55.803671+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:04:45.660145+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:10:04.265 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 16:10:03 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T16:10:03.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T16:10:04.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 16:10:03 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:16:10:03] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T16:10:04.901 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10598 8 365281 365273 0.269247 0 - 0.0299513 2026-02-23T16:10:05.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:10:04 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T16:10:05.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:10:04 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T16:10:05.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:10:04 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3384946178' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:10:05.901 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10599 8 365281 365273 0.269222 0 - 0.0299513 2026-02-23T16:10:06.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:10:05 trial068 ceph-mon[38805]: pgmap v6131: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:10:06.901 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T16:10:06.901395+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T16:10:06.902 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T16:10:06.902 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10600 8 365281 365273 0.269196 0 - 0.0299513 2026-02-23T16:10:07.018 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:10:07.137 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:10:07.236 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:10:07.234+0000 7fac851ab640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:10:07.236 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:10:07.234+0000 7fac851ab640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:10:07.348 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:10:07.348 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":365,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:09:35.408622+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:05:59.814738+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:08:55.803671+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:04:45.660145+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:10:07.902 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10601 8 365281 365273 0.269171 0 - 0.0299513 2026-02-23T16:10:08.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:10:07 trial068 ceph-mon[38805]: pgmap v6132: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:10:08.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:10:07 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1133010277' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:10:08.902 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10602 8 365281 365273 0.269145 0 - 0.0299513 2026-02-23T16:10:09.902 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10603 8 365281 365273 0.26912 0 - 0.0299513 2026-02-23T16:10:10.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:10:09 trial068 ceph-mon[38805]: pgmap v6133: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:10:10.400 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:10:10.519 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:10:10.624 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:10:10.622+0000 7f9397fec640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:10:10.624 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:10:10.622+0000 7f9397fec640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:10:10.735 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:10:10.735 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":365,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:09:35.408622+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:05:59.814738+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:08:55.803671+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:04:45.660145+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:10:10.902 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10604 8 365281 365273 0.269095 0 - 0.0299513 2026-02-23T16:10:11.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:10:10 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/632525582' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:10:11.902 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10605 8 365281 365273 0.269069 0 - 0.0299513 2026-02-23T16:10:12.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:10:11 trial068 ceph-mon[38805]: pgmap v6134: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:10:12.902 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10606 8 365281 365273 0.269044 0 - 0.0299513 2026-02-23T16:10:13.773 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:10:13.890 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:10:13.902 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10607 8 365281 365273 0.269018 0 - 0.0299513 2026-02-23T16:10:13.987 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:10:13.986+0000 7fedf4ee3640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:10:13.988 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:10:13.986+0000 7fedf4ee3640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:10:14.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:10:13 trial068 ceph-mon[38805]: pgmap v6135: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:10:14.094 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 16:10:13 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:16:10:13] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T16:10:14.097 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:10:14.097 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":365,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:09:35.408622+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:05:59.814738+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:08:55.803671+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:04:45.660145+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:10:14.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 16:10:13 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T16:10:13.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T16:10:14.902 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10608 8 365281 365273 0.268993 0 - 0.0299513 2026-02-23T16:10:15.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:10:14 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2276340461' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:10:15.902 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10609 8 365281 365273 0.268968 0 - 0.0299513 2026-02-23T16:10:16.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:10:15 trial068 ceph-mon[38805]: pgmap v6136: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:10:16.902 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10610 8 365281 365273 0.268942 0 - 0.0299513 2026-02-23T16:10:17.134 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:10:17.254 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:10:17.350 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:10:17.348+0000 7f4ac2b8b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:10:17.350 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:10:17.348+0000 7f4ac2b8b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:10:17.460 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:10:17.460 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":365,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:09:35.408622+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:05:59.814738+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:08:55.803671+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:04:45.660145+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:10:17.902 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10611 8 365281 365273 0.268917 0 - 0.0299513 2026-02-23T16:10:18.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:10:17 trial068 ceph-mon[38805]: pgmap v6137: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:10:18.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:10:17 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2457081917' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:10:18.902 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10612 8 365281 365273 0.268892 0 - 0.0299513 2026-02-23T16:10:19.902 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10613 8 365281 365273 0.268866 0 - 0.0299513 2026-02-23T16:10:20.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:10:19 trial068 ceph-mon[38805]: pgmap v6138: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:10:20.497 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:10:20.613 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:10:20.711 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:10:20.709+0000 7f3498d3e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:10:20.711 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:10:20.709+0000 7f3498d3e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:10:20.821 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:10:20.821 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":365,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:09:35.408622+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:05:59.814738+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:08:55.803671+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:04:45.660145+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:10:20.903 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10614 8 365281 365273 0.268841 0 - 0.0299513 2026-02-23T16:10:21.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:10:21 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1488191145' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:10:21.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:10:21 trial068 ceph-mon[38805]: pgmap v6139: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:10:21.903 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10615 8 365281 365273 0.268816 0 - 0.0299513 2026-02-23T16:10:22.903 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10616 8 365281 365273 0.26879 0 - 0.0299513 2026-02-23T16:10:23.858 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:10:23.903 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10617 8 365281 365273 0.268765 0 - 0.0299513 2026-02-23T16:10:23.975 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:10:24.072 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:10:24.070+0000 7fc586dbd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:10:24.072 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:10:24.070+0000 7fc586dbd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:10:24.182 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:10:23 trial068 ceph-mon[38805]: pgmap v6140: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:10:24.183 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 16:10:23 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:16:10:23] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T16:10:24.183 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:10:24.183 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":365,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:09:35.408622+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:05:59.814738+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:08:55.803671+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:04:45.660145+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:10:24.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 16:10:23 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T16:10:23.972Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T16:10:24.903 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10618 8 365281 365273 0.26874 0 - 0.0299513 2026-02-23T16:10:25.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:10:24 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1129557671' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:10:25.903 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10619 8 365281 365273 0.268714 0 - 0.0299513 2026-02-23T16:10:26.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:10:25 trial068 ceph-mon[38805]: pgmap v6141: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:10:26.903 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T16:10:26.902856+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T16:10:26.903 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T16:10:26.903 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10620 8 365281 365273 0.268689 0 - 0.0299513 2026-02-23T16:10:27.220 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:10:27.337 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:10:27.443 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:10:27.441+0000 7f9bd0190640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:10:27.443 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:10:27.441+0000 7f9bd0190640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:10:27.552 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:10:27.552 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":365,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:09:35.408622+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:05:59.814738+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:08:55.803671+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:04:45.660145+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:10:27.903 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10621 8 365281 365273 0.268664 0 - 0.0299513 2026-02-23T16:10:28.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:10:27 trial068 ceph-mon[38805]: pgmap v6142: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:10:28.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:10:27 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/4189964376' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:10:28.903 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10622 8 365281 365273 0.268639 0 - 0.0299513 2026-02-23T16:10:29.903 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10623 8 365281 365273 0.268613 0 - 0.0299513 2026-02-23T16:10:30.146 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:10:29 trial068 ceph-mon[38805]: pgmap v6143: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:10:30.588 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:10:30.705 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:10:30.804 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:10:30.802+0000 7f951d945640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:10:30.804 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:10:30.802+0000 7f951d945640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:10:30.903 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10624 8 365281 365273 0.268588 0 - 0.0299513 2026-02-23T16:10:30.913 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:10:30.913 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":365,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:09:35.408622+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:05:59.814738+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:08:55.803671+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:04:45.660145+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:10:31.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:10:30 trial068 ceph-mon[38805]: 5.a scrub starts 2026-02-23T16:10:31.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:10:30 trial068 ceph-mon[38805]: 5.a scrub ok 2026-02-23T16:10:31.903 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10625 8 365281 365273 0.268563 0 - 0.0299513 2026-02-23T16:10:32.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:10:31 trial068 ceph-mon[38805]: 5.5 scrub starts 2026-02-23T16:10:32.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:10:31 trial068 ceph-mon[38805]: 5.5 scrub ok 2026-02-23T16:10:32.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:10:31 trial068 ceph-mon[38805]: pgmap v6144: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:10:32.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:10:31 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3931185553' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:10:32.903 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10626 8 365281 365273 0.268537 0 - 0.0299513 2026-02-23T16:10:33.903 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10627 8 365281 365273 0.268512 0 - 0.0299513 2026-02-23T16:10:33.954 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:10:34.077 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:10:34.179 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:10:34.177+0000 7fc18650a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:10:34.180 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:10:34.178+0000 7fc18650a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:10:34.180 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:10:33 trial068 ceph-mon[38805]: pgmap v6145: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:10:34.180 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 16:10:33 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:16:10:33] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T16:10:34.265 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 16:10:33 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T16:10:33.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T16:10:34.290 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:10:34.290 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":365,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:09:35.408622+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:05:59.814738+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:08:55.803671+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:04:45.660145+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:10:34.904 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10628 8 365281 365273 0.268487 0 - 0.0299513 2026-02-23T16:10:35.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:10:34 trial068 ceph-mon[38805]: 5.e scrub starts 2026-02-23T16:10:35.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:10:34 trial068 ceph-mon[38805]: 5.e scrub ok 2026-02-23T16:10:35.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:10:34 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/4176419857' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:10:35.904 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10629 8 365281 365273 0.268462 0 - 0.0299513 2026-02-23T16:10:36.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:10:35 trial068 ceph-mon[38805]: pgmap v6146: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:10:36.904 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10630 8 365281 365273 0.268436 0 - 0.0299513 2026-02-23T16:10:37.328 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:10:37.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:10:36 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2026-02-23T16:10:37.446 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:10:37.549 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:10:37.547+0000 7fbd267f5640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:10:37.549 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:10:37.547+0000 7fbd267f5640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:10:37.659 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:10:37.659 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":365,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:09:35.408622+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:05:59.814738+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:08:55.803671+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:04:45.660145+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:10:37.904 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10631 8 365281 365273 0.268411 0 - 0.0299513 2026-02-23T16:10:38.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:10:37 trial068 ceph-mon[38805]: pgmap v6147: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:10:38.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:10:37 trial068 ceph-mon[38805]: 5.7 scrub starts 2026-02-23T16:10:38.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:10:37 trial068 ceph-mon[38805]: 5.7 scrub ok 2026-02-23T16:10:38.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:10:37 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-23T16:10:38.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:10:37 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2026-02-23T16:10:38.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:10:37 trial068 ceph-mon[38805]: pgmap v6148: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:10:38.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:10:37 trial068 ceph-mon[38805]: pgmap v6149: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:10:38.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:10:37 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T16:10:38.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:10:37 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2026-02-23T16:10:38.095 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:10:37 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1938918558' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:10:38.904 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10632 8 365281 365273 0.268386 0 - 0.0299513 2026-02-23T16:10:39.904 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10633 8 365281 365273 0.268361 0 - 0.0299513 2026-02-23T16:10:40.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:10:40 trial068 ceph-mon[38805]: pgmap v6150: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:10:40.697 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:10:40.814 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:10:40.904 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10634 8 365281 365273 0.268335 0 - 0.0299513 2026-02-23T16:10:40.913 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:10:40.911+0000 7ff0c4054640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:10:40.913 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:10:40.911+0000 7ff0c4054640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:10:41.022 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:10:41.022 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":365,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:09:35.408622+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:05:59.814738+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:08:55.803671+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:04:45.660145+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:10:41.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:10:41 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/335100261' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:10:41.904 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10635 8 365281 365273 0.26831 0 - 0.0299513 2026-02-23T16:10:42.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:10:42 trial068 ceph-mon[38805]: pgmap v6151: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:10:42.904 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10636 8 365281 365273 0.268285 0 - 0.0299513 2026-02-23T16:10:43.904 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10637 8 365281 365273 0.26826 0 - 0.0299513 2026-02-23T16:10:44.060 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:10:44.180 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:10:44.225 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:10:44 trial068 ceph-mon[38805]: pgmap v6152: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:10:44.225 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 16:10:43 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:16:10:43] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T16:10:44.276 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:10:44.274+0000 7fbe28de9640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:10:44.276 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:10:44.274+0000 7fbe28de9640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:10:44.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 16:10:43 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T16:10:43.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T16:10:44.384 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:10:44.384 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":365,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:09:35.408622+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:05:59.814738+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:08:55.803671+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:04:45.660145+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:10:44.904 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10638 8 365281 365273 0.268235 0 - 0.0299513 2026-02-23T16:10:45.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:10:45 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3486409269' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:10:45.904 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10639 8 365281 365273 0.268209 0 - 0.0299513 2026-02-23T16:10:46.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:10:46 trial068 ceph-mon[38805]: pgmap v6153: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:10:46.904 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T16:10:46.904205+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T16:10:46.904 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T16:10:46.905 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10640 8 365281 365273 0.268184 0 - 0.0299513 2026-02-23T16:10:47.420 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:10:47.540 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:10:47.642 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:10:47.640+0000 7f65381f2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:10:47.642 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:10:47.640+0000 7f65381f2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:10:47.750 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:10:47.750 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":365,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:09:35.408622+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:05:59.814738+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:08:55.803671+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:04:45.660145+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:10:47.904 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10641 8 365281 365273 0.268159 0 - 0.0299513 2026-02-23T16:10:48.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:10:48 trial068 ceph-mon[38805]: pgmap v6154: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:10:48.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:10:48 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2291503969' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:10:48.904 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10642 8 365281 365273 0.268134 0 - 0.0299513 2026-02-23T16:10:49.904 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10643 8 365281 365273 0.268109 0 - 0.0299513 2026-02-23T16:10:50.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:10:50 trial068 ceph-mon[38805]: pgmap v6155: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:10:50.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:10:50 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T16:10:50.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:10:50 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T16:10:50.788 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:10:50.904 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10644 8 365281 365273 0.268083 0 - 0.0299513 2026-02-23T16:10:50.909 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:10:51.009 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:10:51.007+0000 7f73d3076640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:10:51.009 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:10:51.007+0000 7f73d3076640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:10:51.118 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:10:51.119 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":367,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:10:50.417052+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:05:59.814738+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:08:55.803671+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:10:03.916437+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:10:51.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:10:51 trial068 ceph-mon[38805]: osdmap e366: 4 total, 3 up, 4 in 2026-02-23T16:10:51.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:10:51 trial068 ceph-mon[38805]: osdmap e367: 4 total, 3 up, 4 in 2026-02-23T16:10:51.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:10:51 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3315325860' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:10:51.904 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10645 8 365281 365273 0.268058 0 - 0.0299513 2026-02-23T16:10:52.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:10:52 trial068 ceph-mon[38805]: pgmap v6158: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:10:52.905 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10646 8 365281 365273 0.268033 0 - 0.0299513 2026-02-23T16:10:53.905 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10647 8 365281 365273 0.268008 0 - 0.0299513 2026-02-23T16:10:54.158 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:10:54.273 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:10:54.295 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:10:54 trial068 ceph-mon[38805]: pgmap v6159: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:10:54.296 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 16:10:53 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:16:10:53] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T16:10:54.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 16:10:53 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T16:10:53.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T16:10:54.373 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:10:54.371+0000 7f914cf28640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:10:54.373 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:10:54.371+0000 7f914cf28640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:10:54.482 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:10:54.483 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":367,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:10:50.417052+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:05:59.814738+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:08:55.803671+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:10:03.916437+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:10:54.905 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10648 8 365281 365273 0.267983 0 - 0.0299513 2026-02-23T16:10:55.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:10:55 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2959716886' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:10:55.905 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10649 8 365281 365273 0.267957 0 - 0.0299513 2026-02-23T16:10:56.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:10:56 trial068 ceph-mon[38805]: pgmap v6160: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:10:56.905 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10650 8 365281 365273 0.267932 0 - 0.0299513 2026-02-23T16:10:57.521 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:10:57.640 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:10:57.745 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:10:57.743+0000 7f1898d21640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:10:57.745 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:10:57.743+0000 7f1898d21640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:10:57.854 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:10:57.855 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":367,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:10:50.417052+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:05:59.814738+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:08:55.803671+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:10:03.916437+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:10:57.905 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10651 8 365281 365273 0.267907 0 - 0.0299513 2026-02-23T16:10:58.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:10:58 trial068 ceph-mon[38805]: pgmap v6161: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:10:58.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:10:58 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2783166629' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:10:58.905 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10652 8 365281 365273 0.267882 0 - 0.0299513 2026-02-23T16:10:59.905 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10653 8 365281 365273 0.267857 0 - 0.0299513 2026-02-23T16:11:00.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:11:00 trial068 ceph-mon[38805]: pgmap v6162: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:11:00.892 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:11:00.905 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10654 8 365281 365273 0.267832 0 - 0.0299513 2026-02-23T16:11:01.009 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:11:01.116 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:11:01.114+0000 7fe87e982640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:11:01.116 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:11:01.115+0000 7fe87e982640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:11:01.225 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:11:01.225 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":367,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:10:50.417052+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:05:59.814738+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:08:55.803671+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:10:03.916437+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:11:01.905 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10655 8 365281 365273 0.267807 0 - 0.0299513 2026-02-23T16:11:02.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:11:02 trial068 ceph-mon[38805]: pgmap v6163: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:11:02.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:11:02 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3281115924' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:11:02.905 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10656 8 365281 365273 0.267781 0 - 0.0299513 2026-02-23T16:11:03.906 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10657 8 365281 365273 0.267756 0 - 0.0299513 2026-02-23T16:11:04.262 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:11:04.266 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 16:11:03 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T16:11:03.974Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T16:11:04.287 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:11:04 trial068 ceph-mon[38805]: pgmap v6164: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:11:04.288 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 16:11:03 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:16:11:03] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T16:11:04.381 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:11:04.477 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:11:04.475+0000 7f538768d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:11:04.477 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:11:04.475+0000 7f538768d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:11:04.585 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:11:04.585 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":367,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:10:50.417052+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:05:59.814738+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:08:55.803671+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:10:03.916437+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:11:04.906 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10658 8 365281 365273 0.267731 0 - 0.0299513 2026-02-23T16:11:05.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:11:05 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3802872422' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:11:05.906 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10659 8 365281 365273 0.267706 0 - 0.0299513 2026-02-23T16:11:06.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:11:06 trial068 ceph-mon[38805]: pgmap v6165: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:11:06.906 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T16:11:06.905594+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T16:11:06.906 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T16:11:06.906 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10660 8 365281 365273 0.267681 0 - 0.0299513 2026-02-23T16:11:07.623 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:11:07.742 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:11:07.843 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:11:07.841+0000 7fc1be9fe640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:11:07.844 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:11:07.842+0000 7fc1be9fe640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:11:07.906 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10661 8 365281 365273 0.267656 0 - 0.0299513 2026-02-23T16:11:07.954 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:11:07.954 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":367,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:10:50.417052+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:05:59.814738+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:08:55.803671+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:10:03.916437+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:11:08.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:11:08 trial068 ceph-mon[38805]: pgmap v6166: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:11:08.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:11:08 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/841767698' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:11:08.906 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10662 8 365281 365273 0.267631 0 - 0.0299513 2026-02-23T16:11:09.906 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10663 8 365281 365273 0.267606 0 - 0.0299513 2026-02-23T16:11:10.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:11:10 trial068 ceph-mon[38805]: pgmap v6167: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:11:10.906 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10664 8 365281 365273 0.267581 0 - 0.0299513 2026-02-23T16:11:10.993 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:11:11.109 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:11:11.206 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:11:11.204+0000 7f9a76e27640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:11:11.206 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:11:11.204+0000 7f9a76e27640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:11:11.315 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:11:11.316 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":367,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:10:50.417052+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:05:59.814738+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:08:55.803671+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:10:03.916437+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:11:11.316 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:11:11 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T16:11:11.316 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:11:11 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T16:11:11.907 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10665 8 365281 365273 0.267555 0 - 0.0299513 2026-02-23T16:11:12.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:11:12 trial068 ceph-mon[38805]: pgmap v6168: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:11:12.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:11:12 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1787125170' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:11:12.907 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10666 8 365281 365273 0.26753 0 - 0.0299513 2026-02-23T16:11:13.907 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10667 8 365281 365273 0.267505 0 - 0.0299513 2026-02-23T16:11:14.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:11:14 trial068 ceph-mon[38805]: pgmap v6169: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:11:14.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 16:11:13 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:16:11:13] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T16:11:14.353 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:11:14.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 16:11:13 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T16:11:13.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T16:11:14.469 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:11:14.569 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:11:14.567+0000 7f32b66ef640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:11:14.569 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:11:14.567+0000 7f32b66ef640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:11:14.679 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:11:14.679 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":367,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:10:50.417052+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:05:59.814738+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:08:55.803671+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:10:03.916437+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:11:14.907 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10668 8 365281 365273 0.26748 0 - 0.0299513 2026-02-23T16:11:15.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:11:15 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2050493618' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:11:15.907 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10669 8 365281 365273 0.267455 0 - 0.0299513 2026-02-23T16:11:16.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:11:16 trial068 ceph-mon[38805]: pgmap v6170: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:11:16.907 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10670 8 365281 365273 0.26743 0 - 0.0299513 2026-02-23T16:11:17.717 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:11:17.834 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:11:17.907 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10671 8 365281 365273 0.267405 0 - 0.0299513 2026-02-23T16:11:17.938 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:11:17.936+0000 7f7f972a6640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:11:17.939 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:11:17.936+0000 7f7f972a6640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:11:18.048 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:11:18.049 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":367,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:10:50.417052+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:05:59.814738+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:08:55.803671+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:10:03.916437+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:11:18.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:11:18 trial068 ceph-mon[38805]: pgmap v6171: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:11:18.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:11:18 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3512733480' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:11:18.907 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10672 8 365281 365273 0.26738 0 - 0.0299513 2026-02-23T16:11:19.907 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10673 8 365281 365273 0.267355 0 - 0.0299513 2026-02-23T16:11:20.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:11:20 trial068 ceph-mon[38805]: pgmap v6172: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:11:20.907 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10674 8 365281 365273 0.26733 0 - 0.0299513 2026-02-23T16:11:21.087 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:11:21.205 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:11:21.303 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:11:21.301+0000 7f1b96606640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:11:21.303 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:11:21.301+0000 7f1b96606640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:11:21.415 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:11:21.415 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":367,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:10:50.417052+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:05:59.814738+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:08:55.803671+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:10:03.916437+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:11:21.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:11:21 trial068 ceph-mon[38805]: pgmap v6173: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:11:21.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:11:21 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3653467484' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:11:21.907 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10675 8 365281 365273 0.267305 0 - 0.0299513 2026-02-23T16:11:22.907 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10676 8 365281 365273 0.26728 0 - 0.0299513 2026-02-23T16:11:23.907 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10677 8 365281 365273 0.267255 0 - 0.0299513 2026-02-23T16:11:24.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:11:24 trial068 ceph-mon[38805]: pgmap v6174: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:11:24.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 16:11:23 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:16:11:23] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T16:11:24.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 16:11:23 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T16:11:23.974Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T16:11:24.452 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:11:24.571 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:11:24.675 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:11:24.673+0000 7f755456a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:11:24.675 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:11:24.673+0000 7f755456a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:11:24.783 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:11:24.783 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":367,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:10:50.417052+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:05:59.814738+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:08:55.803671+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:10:03.916437+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:11:24.908 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10678 8 365281 365273 0.26723 0 - 0.0299513 2026-02-23T16:11:25.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:11:25 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T16:11:25.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:11:25 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T16:11:25.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:11:25 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/810694596' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:11:25.908 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10679 8 365281 365273 0.267205 0 - 0.0299513 2026-02-23T16:11:26.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:11:26 trial068 ceph-mon[38805]: pgmap v6175: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:11:26.908 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T16:11:26.907518+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T16:11:26.908 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T16:11:26.908 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10680 8 365281 365273 0.26718 0 - 0.0299513 2026-02-23T16:11:27.821 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:11:27.908 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10681 8 365281 365273 0.267155 0 - 0.0299513 2026-02-23T16:11:27.941 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:11:28.050 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:11:28.048+0000 7fd171835640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:11:28.050 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:11:28.049+0000 7fd171835640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:11:28.159 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:11:28.159 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":367,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:10:50.417052+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:05:59.814738+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:08:55.803671+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:10:03.916437+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:11:28.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:11:28 trial068 ceph-mon[38805]: pgmap v6176: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:11:28.908 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10682 8 365281 365273 0.26713 0 - 0.0299513 2026-02-23T16:11:29.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:11:29 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/543828207' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:11:29.908 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10683 8 365281 365273 0.267105 0 - 0.0299513 2026-02-23T16:11:30.495 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:11:30 trial068 ceph-mon[38805]: pgmap v6177: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:11:30.908 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10684 8 365281 365273 0.26708 0 - 0.0299513 2026-02-23T16:11:31.198 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:11:31.315 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:11:31.416 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:11:31.414+0000 7f186d6c9640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:11:31.416 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:11:31.414+0000 7f186d6c9640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:11:31.525 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:11:31.525 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":367,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:10:50.417052+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:05:59.814738+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:08:55.803671+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:10:03.916437+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:11:31.908 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10685 8 365281 365273 0.267055 0 - 0.0299513 2026-02-23T16:11:32.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:11:32 trial068 ceph-mon[38805]: pgmap v6178: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:11:32.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:11:32 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/117417910' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:11:32.908 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10686 8 365281 365273 0.26703 0 - 0.0299513 2026-02-23T16:11:33.908 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10687 8 365281 365273 0.267005 0 - 0.0299513 2026-02-23T16:11:34.265 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 16:11:33 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T16:11:33.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T16:11:34.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:11:34 trial068 ceph-mon[38805]: 5.8 scrub starts 2026-02-23T16:11:34.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:11:34 trial068 ceph-mon[38805]: 5.8 scrub ok 2026-02-23T16:11:34.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:11:34 trial068 ceph-mon[38805]: pgmap v6179: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:11:34.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 16:11:33 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:16:11:33] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T16:11:34.562 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:11:34.682 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:11:34.781 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:11:34.780+0000 7ff276da2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:11:34.782 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:11:34.780+0000 7ff276da2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:11:34.889 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:11:34.890 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":367,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:10:50.417052+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:05:59.814738+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:08:55.803671+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:10:03.916437+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:11:34.908 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10688 8 365281 365273 0.26698 0 - 0.0299513 2026-02-23T16:11:35.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:11:35 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2189886910' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:11:35.908 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10689 8 365281 365273 0.266955 0 - 0.0299513 2026-02-23T16:11:36.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:11:36 trial068 ceph-mon[38805]: pgmap v6180: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:11:36.908 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10690 8 365281 365273 0.26693 0 - 0.0299513 2026-02-23T16:11:37.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:11:37 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2026-02-23T16:11:37.908 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10691 8 365281 365273 0.266905 0 - 0.0299513 2026-02-23T16:11:37.927 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:11:38.042 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:11:38.143 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:11:38.141+0000 7f5b574ab640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:11:38.143 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:11:38.141+0000 7f5b574ab640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:11:38.255 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:11:38.255 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":367,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:10:50.417052+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:05:59.814738+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:08:55.803671+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:10:03.916437+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:11:38.256 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:11:38 trial068 ceph-mon[38805]: pgmap v6181: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:11:38.256 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:11:38 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-23T16:11:38.256 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:11:38 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2026-02-23T16:11:38.256 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:11:38 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T16:11:38.256 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:11:38 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2026-02-23T16:11:38.909 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10692 8 365281 365273 0.26688 0 - 0.0299513 2026-02-23T16:11:39.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:11:39 trial068 ceph-mon[38805]: pgmap v6182: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:11:39.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:11:39 trial068 ceph-mon[38805]: pgmap v6183: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:11:39.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:11:39 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/404880410' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:11:39.909 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10693 8 365281 365273 0.266855 0 - 0.0299513 2026-02-23T16:11:40.909 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10694 8 365281 365273 0.26683 0 - 0.0299513 2026-02-23T16:11:41.294 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:11:41.415 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:11:41.436 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:11:41 trial068 ceph-mon[38805]: pgmap v6184: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:11:41.518 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:11:41.516+0000 7f5a580f6640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:11:41.518 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:11:41.516+0000 7f5a580f6640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:11:41.626 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:11:41.626 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":367,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:10:50.417052+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:05:59.814738+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:08:55.803671+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:10:03.916437+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:11:41.909 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10695 8 365281 365273 0.266805 0 - 0.0299513 2026-02-23T16:11:42.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:11:42 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1200803259' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:11:42.909 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10696 8 365281 365273 0.26678 0 - 0.0299513 2026-02-23T16:11:43.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:11:43 trial068 ceph-mon[38805]: pgmap v6185: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:11:43.909 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10697 8 365281 365273 0.266755 0 - 0.0299513 2026-02-23T16:11:44.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 16:11:43 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:16:11:43] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T16:11:44.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 16:11:43 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T16:11:43.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T16:11:44.664 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:11:44.783 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:11:44.885 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:11:44.883+0000 7fd1d7b34640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:11:44.885 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:11:44.883+0000 7fd1d7b34640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:11:44.909 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10698 8 365281 365273 0.26673 0 - 0.0299513 2026-02-23T16:11:44.995 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:11:44.995 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":367,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:10:50.417052+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:05:59.814738+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:08:55.803671+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:10:03.916437+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:11:45.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:11:45 trial068 ceph-mon[38805]: pgmap v6186: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:11:45.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:11:45 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/431526658' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:11:45.909 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10699 8 365281 365273 0.266705 0 - 0.0299513 2026-02-23T16:11:46.909 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T16:11:46.908981+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T16:11:46.910 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T16:11:46.910 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10700 8 365281 365273 0.26668 0 - 0.0299513 2026-02-23T16:11:47.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:11:47 trial068 ceph-mon[38805]: pgmap v6187: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:11:47.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:11:47 trial068 ceph-mon[38805]: 5.e scrub starts 2026-02-23T16:11:47.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:11:47 trial068 ceph-mon[38805]: 5.e scrub ok 2026-02-23T16:11:47.909 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10701 8 365281 365273 0.266655 0 - 0.0299513 2026-02-23T16:11:48.032 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:11:48.151 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:11:48.254 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:11:48.252+0000 7f75864df640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:11:48.254 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:11:48.252+0000 7f75864df640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:11:48.367 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:11:48.367 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":367,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:10:50.417052+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:05:59.814738+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:08:55.803671+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:10:03.916437+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:11:48.909 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10702 8 365281 365273 0.26663 0 - 0.0299513 2026-02-23T16:11:49.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:11:49 trial068 ceph-mon[38805]: pgmap v6188: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:11:49.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:11:49 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2668191644' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:11:49.909 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10703 8 365281 365273 0.266606 0 - 0.0299513 2026-02-23T16:11:50.909 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10704 8 365281 365273 0.266581 0 - 0.0299513 2026-02-23T16:11:51.404 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:11:51.428 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:11:51 trial068 ceph-mon[38805]: pgmap v6189: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:11:51.522 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:11:51.618 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:11:51.616+0000 7fba648d1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:11:51.618 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:11:51.616+0000 7fba648d1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:11:51.727 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:11:51.728 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":367,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:10:50.417052+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:05:59.814738+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:08:55.803671+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:10:03.916437+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:11:51.910 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10705 8 365281 365273 0.266556 0 - 0.0299513 2026-02-23T16:11:52.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:11:52 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1374196076' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:11:52.910 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10706 8 365281 365273 0.266531 0 - 0.0299513 2026-02-23T16:11:53.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:11:53 trial068 ceph-mon[38805]: pgmap v6190: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:11:53.910 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10707 8 365281 365273 0.266506 0 - 0.0299513 2026-02-23T16:11:54.241 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 16:11:53 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:16:11:53] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T16:11:54.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 16:11:53 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T16:11:53.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T16:11:54.766 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:11:54.887 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:11:54.910 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10708 8 365281 365273 0.266481 0 - 0.0299513 2026-02-23T16:11:54.988 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:11:54.986+0000 7f7c2401b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:11:54.988 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:11:54.986+0000 7f7c2401b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:11:55.096 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:11:55.096 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":367,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:10:50.417052+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:05:59.814738+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:08:55.803671+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:10:03.916437+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:11:55.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:11:55 trial068 ceph-mon[38805]: pgmap v6191: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:11:55.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:11:55 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1130543042' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:11:55.910 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10709 8 365281 365273 0.266456 0 - 0.0299513 2026-02-23T16:11:56.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:11:56 trial068 ceph-mon[38805]: osdmap e368: 4 total, 3 up, 4 in 2026-02-23T16:11:56.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:11:56 trial068 ceph-mon[38805]: osdmap e369: 4 total, 3 up, 4 in 2026-02-23T16:11:56.910 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10710 8 365281 365273 0.266431 0 - 0.0299513 2026-02-23T16:11:57.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:11:57 trial068 ceph-mon[38805]: pgmap v6193: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:11:57.910 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10711 8 365281 365273 0.266406 0 - 0.0299513 2026-02-23T16:11:58.133 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:11:58.254 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:11:58.357 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:11:58.355+0000 7f96e7b5c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:11:58.357 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:11:58.355+0000 7f96e7b5c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:11:58.469 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:11:58.469 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":369,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:11:55.424195+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:10:49.223671+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:08:55.803671+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:10:03.916437+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:11:58.470 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:11:58 trial068 ceph-mon[38805]: 5.5 scrub starts 2026-02-23T16:11:58.470 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:11:58 trial068 ceph-mon[38805]: 5.5 scrub ok 2026-02-23T16:11:58.470 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:11:58 trial068 ceph-mon[38805]: 5.7 scrub starts 2026-02-23T16:11:58.470 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:11:58 trial068 ceph-mon[38805]: 5.7 scrub ok 2026-02-23T16:11:58.910 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10712 8 365281 365273 0.266382 0 - 0.0299513 2026-02-23T16:11:59.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:11:59 trial068 ceph-mon[38805]: pgmap v6195: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:11:59.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:11:59 trial068 ceph-mon[38805]: 5.a scrub starts 2026-02-23T16:11:59.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:11:59 trial068 ceph-mon[38805]: 5.a scrub ok 2026-02-23T16:11:59.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:11:59 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/674369231' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:11:59.910 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10713 8 365281 365273 0.266357 0 - 0.0299513 2026-02-23T16:12:00.910 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10714 8 365281 365273 0.266332 0 - 0.0299513 2026-02-23T16:12:01.509 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:12:01.533 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:12:01 trial068 ceph-mon[38805]: pgmap v6196: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:12:01.626 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:12:01.732 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:12:01.730+0000 7f8292561640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:12:01.732 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:12:01.730+0000 7f8292561640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:12:01.840 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:12:01.840 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":369,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:11:55.424195+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:10:49.223671+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:08:55.803671+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:10:03.916437+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:12:01.910 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10715 8 365281 365273 0.266307 0 - 0.0299513 2026-02-23T16:12:02.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:12:02 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3579547964' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:12:02.910 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10716 8 365281 365273 0.266282 0 - 0.0299513 2026-02-23T16:12:03.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:12:03 trial068 ceph-mon[38805]: pgmap v6197: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:12:03.910 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10717 8 365281 365273 0.266257 0 - 0.0299513 2026-02-23T16:12:04.265 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 16:12:03 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T16:12:03.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T16:12:04.280 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 16:12:03 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:16:12:03] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T16:12:04.877 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:12:04.910 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10718 8 365281 365273 0.266232 0 - 0.0299513 2026-02-23T16:12:04.995 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:12:05.092 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:12:05.090+0000 7f84153da640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:12:05.093 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:12:05.091+0000 7f84153da640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:12:05.203 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:12:05.203 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":369,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:11:55.424195+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:10:49.223671+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:08:55.803671+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:10:03.916437+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:12:05.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:12:05 trial068 ceph-mon[38805]: pgmap v6198: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:12:05.910 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10719 8 365281 365273 0.266208 0 - 0.0299513 2026-02-23T16:12:06.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:12:06 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2602410766' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:12:06.911 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T16:12:06.910279+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T16:12:06.911 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T16:12:06.911 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10720 8 365281 365273 0.266183 0 - 0.0299513 2026-02-23T16:12:07.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:12:07 trial068 ceph-mon[38805]: pgmap v6199: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:12:07.910 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10721 8 365281 365273 0.266158 0 - 0.0299513 2026-02-23T16:12:08.260 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:12:08.376 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:12:08.482 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:12:08.480+0000 7ff39af7f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:12:08.482 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:12:08.480+0000 7ff39af7f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:12:08.594 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:12:08.595 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":369,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:11:55.424195+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:10:49.223671+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:08:55.803671+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:10:03.916437+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:12:08.910 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10722 8 365281 365273 0.266133 0 - 0.0299513 2026-02-23T16:12:09.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:12:09 trial068 ceph-mon[38805]: pgmap v6200: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:12:09.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:12:09 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/32189428' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:12:09.910 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10723 8 365281 365273 0.266108 0 - 0.0299513 2026-02-23T16:12:10.911 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10724 8 365281 365273 0.266083 0 - 0.0299513 2026-02-23T16:12:11.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:12:11 trial068 ceph-mon[38805]: pgmap v6201: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:12:11.632 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:12:11.748 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:12:11.848 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:12:11.846+0000 7fc421ff8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:12:11.848 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:12:11.846+0000 7fc421ff8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:12:11.910 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10725 8 365281 365273 0.266059 0 - 0.0299513 2026-02-23T16:12:11.959 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:12:11.959 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":369,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:11:55.424195+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:10:49.223671+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:08:55.803671+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:10:03.916437+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:12:12.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:12:12 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3113544828' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:12:12.911 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10726 8 365281 365273 0.266034 0 - 0.0299513 2026-02-23T16:12:13.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:12:13 trial068 ceph-mon[38805]: pgmap v6202: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:12:13.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:12:13 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T16:12:13.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:12:13 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T16:12:13.911 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10727 8 365281 365273 0.266009 0 - 0.0299513 2026-02-23T16:12:14.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 16:12:13 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:16:12:13] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T16:12:14.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 16:12:13 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T16:12:13.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T16:12:14.911 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10728 8 365281 365273 0.265984 0 - 0.0299513 2026-02-23T16:12:14.995 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:12:15.116 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:12:15.216 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:12:15.214+0000 7fa30d0a9640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:12:15.216 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:12:15.214+0000 7fa30d0a9640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:12:15.325 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:12:15.325 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":369,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:11:55.424195+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:10:49.223671+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:08:55.803671+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:10:03.916437+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:12:15.325 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:12:15 trial068 ceph-mon[38805]: pgmap v6203: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:12:15.911 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10729 8 365281 365273 0.265959 0 - 0.0299513 2026-02-23T16:12:16.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:12:16 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2626538774' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:12:16.911 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10730 8 365281 365273 0.265935 0 - 0.0299513 2026-02-23T16:12:17.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:12:17 trial068 ceph-mon[38805]: pgmap v6204: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:12:17.911 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10731 8 365281 365273 0.26591 0 - 0.0299513 2026-02-23T16:12:18.376 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:12:18.494 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:12:18.594 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:12:18.591+0000 7f0940a2a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:12:18.594 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:12:18.592+0000 7f0940a2a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:12:18.704 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:12:18.704 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":369,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:11:55.424195+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:10:49.223671+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:08:55.803671+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:10:03.916437+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:12:18.911 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10732 8 365281 365273 0.265885 0 - 0.0299513 2026-02-23T16:12:19.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:12:19 trial068 ceph-mon[38805]: pgmap v6205: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:12:19.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:12:19 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/364470976' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:12:19.911 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10733 8 365281 365273 0.26586 0 - 0.0299513 2026-02-23T16:12:20.911 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10734 8 365281 365273 0.265836 0 - 0.0299513 2026-02-23T16:12:21.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:12:21 trial068 ceph-mon[38805]: pgmap v6206: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:12:21.742 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:12:21.861 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:12:21.911 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10735 8 365281 365273 0.265811 0 - 0.0299513 2026-02-23T16:12:21.957 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:12:21.956+0000 7f2560e74640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:12:21.958 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:12:21.956+0000 7f2560e74640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:12:22.067 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:12:22.067 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":369,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:11:55.424195+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:10:49.223671+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:08:55.803671+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:10:03.916437+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:12:22.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:12:22 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1179740960' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:12:22.911 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10736 8 365281 365273 0.265786 0 - 0.0299513 2026-02-23T16:12:23.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:12:23 trial068 ceph-mon[38805]: pgmap v6207: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:12:23.911 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10737 8 365281 365273 0.265761 0 - 0.0299513 2026-02-23T16:12:24.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 16:12:23 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:16:12:23] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T16:12:24.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 16:12:23 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T16:12:23.974Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T16:12:24.911 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10738 8 365281 365273 0.265737 0 - 0.0299513 2026-02-23T16:12:25.104 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:12:25.222 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:12:25.318 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:12:25.316+0000 7f815c18a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:12:25.318 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:12:25.317+0000 7f815c18a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:12:25.427 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:12:25 trial068 ceph-mon[38805]: pgmap v6208: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:12:25.428 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:12:25.428 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":369,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:11:55.424195+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:10:49.223671+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:08:55.803671+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:10:03.916437+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:12:25.911 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10739 8 365281 365273 0.265712 0 - 0.0299513 2026-02-23T16:12:26.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:12:26 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1826632646' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:12:26.911 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T16:12:26.911276+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T16:12:26.912 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T16:12:26.912 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10740 8 365281 365273 0.265687 0 - 0.0299513 2026-02-23T16:12:27.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:12:27 trial068 ceph-mon[38805]: pgmap v6209: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:12:27.911 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10741 8 365281 365273 0.265662 0 - 0.0299513 2026-02-23T16:12:28.467 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:12:28.584 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:12:28.686 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:12:28.684+0000 7ffa76c94640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:12:28.686 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:12:28.684+0000 7ffa76c94640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:12:28.797 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:12:28.797 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":369,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:11:55.424195+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:10:49.223671+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:08:55.803671+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:10:03.916437+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:12:28.911 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10742 8 365281 365273 0.265638 0 - 0.0299513 2026-02-23T16:12:29.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:12:29 trial068 ceph-mon[38805]: pgmap v6210: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:12:29.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:12:29 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2994057435' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:12:29.912 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10743 8 365281 365273 0.265613 0 - 0.0299513 2026-02-23T16:12:30.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:12:30 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T16:12:30.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:12:30 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T16:12:30.912 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10744 8 365281 365273 0.265588 0 - 0.0299513 2026-02-23T16:12:31.495 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:12:31 trial068 ceph-mon[38805]: pgmap v6211: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:12:31.835 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:12:31.912 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10745 8 365281 365273 0.265563 0 - 0.0299513 2026-02-23T16:12:31.953 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:12:32.061 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:12:32.059+0000 7f70df935640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:12:32.062 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:12:32.060+0000 7f70df935640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:12:32.173 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:12:32.173 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":369,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:11:55.424195+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:10:49.223671+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:08:55.803671+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:10:03.916437+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:12:32.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:12:32 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3016353204' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:12:32.912 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10746 8 365281 365273 0.265539 0 - 0.0299513 2026-02-23T16:12:33.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:12:33 trial068 ceph-mon[38805]: pgmap v6212: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:12:33.912 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10747 8 365281 365273 0.265514 0 - 0.0299513 2026-02-23T16:12:34.265 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 16:12:33 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T16:12:33.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T16:12:34.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 16:12:33 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:16:12:33] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T16:12:34.912 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10748 8 365281 365273 0.265489 0 - 0.0299513 2026-02-23T16:12:35.212 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:12:35.330 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:12:35.429 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:12:35.427+0000 7f78139a6640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:12:35.430 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:12:35.428+0000 7f78139a6640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:12:35.538 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:12:35.538 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":369,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:11:55.424195+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:10:49.223671+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:08:55.803671+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:10:03.916437+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:12:35.539 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:12:35 trial068 ceph-mon[38805]: pgmap v6213: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:12:35.912 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10749 8 365281 365273 0.265465 0 - 0.0299513 2026-02-23T16:12:36.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:12:36 trial068 ceph-mon[38805]: pgmap v6214: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:12:36.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:12:36 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3586506681' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:12:36.912 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10750 8 365281 365273 0.26544 0 - 0.0299513 2026-02-23T16:12:37.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:12:37 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2026-02-23T16:12:37.912 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10751 8 365281 365273 0.265415 0 - 0.0299513 2026-02-23T16:12:38.577 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:12:38.693 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:12:38.714 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:12:38 trial068 ceph-mon[38805]: pgmap v6215: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:12:38.715 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:12:38 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-23T16:12:38.715 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:12:38 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2026-02-23T16:12:38.715 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:12:38 trial068 ceph-mon[38805]: pgmap v6216: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:12:38.715 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:12:38 trial068 ceph-mon[38805]: pgmap v6217: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:12:38.715 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:12:38 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T16:12:38.716 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:12:38 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2026-02-23T16:12:38.790 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:12:38.788+0000 7fa3e637b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:12:38.791 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:12:38.789+0000 7fa3e637b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:12:38.899 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:12:38.899 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":369,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:11:55.424195+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:10:49.223671+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:08:55.803671+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:10:03.916437+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:12:38.912 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10752 8 365281 365273 0.265391 0 - 0.0299513 2026-02-23T16:12:39.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:12:39 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/4060911617' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:12:39.912 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10753 8 365281 365273 0.265366 0 - 0.0299513 2026-02-23T16:12:40.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:12:40 trial068 ceph-mon[38805]: pgmap v6218: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:12:40.912 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10754 8 365281 365273 0.265341 0 - 0.0299513 2026-02-23T16:12:41.912 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10755 8 365281 365273 0.265317 0 - 0.0299513 2026-02-23T16:12:41.936 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:12:42.053 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:12:42.151 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:12:42.149+0000 7f9e0edeb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:12:42.151 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:12:42.149+0000 7f9e0edeb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:12:42.260 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:12:42.260 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":369,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:11:55.424195+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:10:49.223671+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:08:55.803671+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:10:03.916437+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:12:42.913 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10756 8 365281 365273 0.265292 0 - 0.0299513 2026-02-23T16:12:43.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:12:42 trial068 ceph-mon[38805]: pgmap v6219: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:12:43.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:12:42 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/284995096' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:12:43.913 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10757 8 365281 365273 0.265267 0 - 0.0299513 2026-02-23T16:12:44.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 16:12:43 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:16:12:43] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T16:12:44.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 16:12:43 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T16:12:43.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T16:12:44.913 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10758 8 365281 365273 0.265243 0 - 0.0299513 2026-02-23T16:12:45.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:12:44 trial068 ceph-mon[38805]: pgmap v6220: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:12:45.298 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:12:45.417 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:12:45.532 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:12:45.530+0000 7f83b157f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:12:45.532 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:12:45.530+0000 7f83b157f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:12:45.641 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:12:45.641 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":369,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:11:55.424195+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:10:49.223671+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:08:55.803671+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:10:03.916437+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:12:45.913 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10759 8 365281 365273 0.265218 0 - 0.0299513 2026-02-23T16:12:46.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:12:46 trial068 ceph-mon[38805]: pgmap v6221: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:12:46.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:12:46 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/4136364972' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:12:46.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:12:46 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T16:12:46.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:12:46 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T16:12:46.913 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T16:12:46.912686+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T16:12:46.913 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T16:12:46.913 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10760 8 365281 365273 0.265193 0 - 0.0299513 2026-02-23T16:12:47.913 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10761 8 365281 365273 0.265169 0 - 0.0299513 2026-02-23T16:12:48.679 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:12:48.799 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:12:48.903 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:12:48.901+0000 7f415f17b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:12:48.903 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:12:48.901+0000 7f415f17b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:12:48.913 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10762 8 365281 365273 0.265144 0 - 0.0299513 2026-02-23T16:12:49.015 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:12:49.015 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":369,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:11:55.424195+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:10:49.223671+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:08:55.803671+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:10:03.916437+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:12:49.016 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:12:48 trial068 ceph-mon[38805]: pgmap v6222: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:12:49.913 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10763 8 365281 365273 0.265119 0 - 0.0299513 2026-02-23T16:12:50.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:12:49 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/500311807' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:12:50.913 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10764 8 365281 365273 0.265095 0 - 0.0299513 2026-02-23T16:12:51.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:12:50 trial068 ceph-mon[38805]: pgmap v6223: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:12:51.914 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10765 8 365281 365273 0.26507 0 - 0.0299513 2026-02-23T16:12:52.053 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:12:52.170 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:12:52.276 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:12:52.274+0000 7f515245a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:12:52.276 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:12:52.274+0000 7f515245a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:12:52.386 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:12:52.386 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":369,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:11:55.424195+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:10:49.223671+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:08:55.803671+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:10:03.916437+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:12:52.914 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10766 8 365281 365273 0.265045 0 - 0.0299513 2026-02-23T16:12:53.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:12:52 trial068 ceph-mon[38805]: 5.8 scrub starts 2026-02-23T16:12:53.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:12:52 trial068 ceph-mon[38805]: 5.8 scrub ok 2026-02-23T16:12:53.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:12:52 trial068 ceph-mon[38805]: pgmap v6224: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:12:53.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:12:52 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3316536671' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:12:53.914 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10767 8 365281 365273 0.265021 0 - 0.0299513 2026-02-23T16:12:54.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 16:12:53 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:16:12:53] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T16:12:54.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 16:12:53 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T16:12:53.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T16:12:54.914 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10768 8 365281 365273 0.264996 0 - 0.0299513 2026-02-23T16:12:55.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:12:54 trial068 ceph-mon[38805]: pgmap v6225: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:12:55.423 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:12:55.543 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:12:55.639 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:12:55.637+0000 7f0974be3640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:12:55.639 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:12:55.637+0000 7f0974be3640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:12:55.746 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:12:55.747 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":369,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:11:55.424195+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:10:49.223671+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:08:55.803671+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:10:03.916437+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:12:55.914 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10769 8 365281 365273 0.264972 0 - 0.0299513 2026-02-23T16:12:56.914 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10770 8 365281 365273 0.264947 0 - 0.0299513 2026-02-23T16:12:57.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:12:56 trial068 ceph-mon[38805]: pgmap v6226: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:12:57.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:12:56 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2400029671' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:12:57.914 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10771 8 365281 365273 0.264922 0 - 0.0299513 2026-02-23T16:12:58.784 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:12:58.902 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:12:58.914 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10772 8 365281 365273 0.264898 0 - 0.0299513 2026-02-23T16:12:58.949 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:12:58 trial068 ceph-mon[38805]: pgmap v6227: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:12:59.008 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:12:59.006+0000 7f057b628640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:12:59.008 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:12:59.006+0000 7f057b628640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:12:59.116 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:12:59.116 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":369,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:11:55.424195+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:10:49.223671+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:08:55.803671+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:10:03.916437+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:12:59.914 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10773 8 365281 365273 0.264873 0 - 0.0299513 2026-02-23T16:13:00.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:12:59 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2158237801' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:13:00.914 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10774 8 365281 365273 0.264849 0 - 0.0299513 2026-02-23T16:13:01.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:13:00 trial068 ceph-mon[38805]: pgmap v6228: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:13:01.914 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10775 8 365281 365273 0.264824 0 - 0.0299513 2026-02-23T16:13:02.153 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:13:02.270 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:13:02.373 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:13:02.371+0000 7f1797d4a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:13:02.373 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:13:02.371+0000 7f1797d4a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:13:02.481 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:13:02.482 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":369,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:11:55.424195+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:10:49.223671+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:08:55.803671+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:10:03.916437+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:13:02.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:13:02 trial068 ceph-mon[38805]: pgmap v6229: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:13:02.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:13:02 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/436157961' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:13:02.914 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10776 8 365281 365273 0.264799 0 - 0.0299513 2026-02-23T16:13:03.914 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10777 8 365281 365273 0.264775 0 - 0.0299513 2026-02-23T16:13:04.265 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 16:13:03 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T16:13:03.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T16:13:04.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 16:13:03 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:16:13:03] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T16:13:04.914 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10778 8 365281 365273 0.26475 0 - 0.0299513 2026-02-23T16:13:05.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:13:04 trial068 ceph-mon[38805]: pgmap v6230: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:13:05.520 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:13:05.635 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:13:05.743 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:13:05.741+0000 7f2777376640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:13:05.744 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:13:05.742+0000 7f2777376640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:13:05.852 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:13:05.853 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":369,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:11:55.424195+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:10:49.223671+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:08:55.803671+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:10:03.916437+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:13:05.914 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10779 8 365281 365273 0.264726 0 - 0.0299513 2026-02-23T16:13:06.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:13:06 trial068 ceph-mon[38805]: pgmap v6231: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:13:06.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:13:06 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3306874561' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:13:06.915 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T16:13:06.914519+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T16:13:06.915 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T16:13:06.915 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10780 8 365281 365273 0.264701 0 - 0.0299513 2026-02-23T16:13:07.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:13:07 trial068 ceph-mon[38805]: 5.5 scrub starts 2026-02-23T16:13:07.915 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10781 8 365281 365273 0.264677 0 - 0.0299513 2026-02-23T16:13:08.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:13:08 trial068 ceph-mon[38805]: 5.5 scrub ok 2026-02-23T16:13:08.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:13:08 trial068 ceph-mon[38805]: pgmap v6232: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:13:08.890 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:13:08.915 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10782 8 365281 365273 0.264652 0 - 0.0299513 2026-02-23T16:13:09.009 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:13:09.110 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:13:09.108+0000 7f2eccd0a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:13:09.110 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:13:09.108+0000 7f2eccd0a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:13:09.218 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:13:09.218 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":369,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:11:55.424195+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:10:49.223671+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:08:55.803671+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:10:03.916437+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:13:09.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:13:09 trial068 ceph-mon[38805]: 5.e scrub starts 2026-02-23T16:13:09.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:13:09 trial068 ceph-mon[38805]: 5.e scrub ok 2026-02-23T16:13:09.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:13:09 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1668750190' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:13:09.915 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10783 8 365281 365273 0.264628 0 - 0.0299513 2026-02-23T16:13:10.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:13:10 trial068 ceph-mon[38805]: pgmap v6233: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:13:10.915 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10784 8 365281 365273 0.264603 0 - 0.0299513 2026-02-23T16:13:11.915 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10785 8 365281 365273 0.264579 0 - 0.0299513 2026-02-23T16:13:12.256 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:13:12.375 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:13:12.478 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:13:12.476+0000 7fe976616640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:13:12.478 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:13:12.476+0000 7fe976616640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:13:12.587 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:13:12.588 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":369,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:11:55.424195+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:10:49.223671+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:08:55.803671+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:10:03.916437+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:13:12.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:13:12 trial068 ceph-mon[38805]: pgmap v6234: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:13:12.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:13:12 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/37280767' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:13:12.915 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10786 8 365281 365273 0.264554 0 - 0.0299513 2026-02-23T16:13:13.915 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10787 8 365281 365273 0.264529 0 - 0.0299513 2026-02-23T16:13:14.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 16:13:13 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:16:13:13] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T16:13:14.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 16:13:13 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T16:13:13.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T16:13:14.915 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10788 8 365281 365273 0.264505 0 - 0.0299513 2026-02-23T16:13:15.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:13:14 trial068 ceph-mon[38805]: pgmap v6235: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:13:15.624 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:13:15.741 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:13:15.844 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:13:15.842+0000 7f6209288640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:13:15.845 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:13:15.843+0000 7f6209288640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:13:15.915 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10789 8 365281 365273 0.26448 0 - 0.0299513 2026-02-23T16:13:15.957 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:13:15.957 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":369,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:11:55.424195+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:10:49.223671+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:08:55.803671+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:10:03.916437+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:13:16.915 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10790 8 365281 365273 0.264456 0 - 0.0299513 2026-02-23T16:13:17.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:13:16 trial068 ceph-mon[38805]: pgmap v6236: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:13:17.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:13:16 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/4075308227' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:13:17.915 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10791 8 365281 365273 0.264431 0 - 0.0299513 2026-02-23T16:13:18.915 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10792 8 365281 365273 0.264407 0 - 0.0299513 2026-02-23T16:13:18.997 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:13:19.023 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:13:18 trial068 ceph-mon[38805]: pgmap v6237: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:13:19.117 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:13:19.215 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:13:19.213+0000 7f9f2331e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:13:19.215 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:13:19.213+0000 7f9f2331e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:13:19.324 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:13:19.324 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":369,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:11:55.424195+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:10:49.223671+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:08:55.803671+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:10:03.916437+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:13:19.916 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10793 8 365281 365273 0.264382 0 - 0.0299513 2026-02-23T16:13:20.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:13:19 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/197179580' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:13:20.916 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10794 8 365281 365273 0.264358 0 - 0.0299513 2026-02-23T16:13:21.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:13:20 trial068 ceph-mon[38805]: pgmap v6238: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:13:21.916 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10795 8 365281 365273 0.264333 0 - 0.0299513 2026-02-23T16:13:22.361 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:13:22.481 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:13:22.582 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:13:22.580+0000 7fcee5baa640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:13:22.582 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:13:22.580+0000 7fcee5baa640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:13:22.691 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:13:22.691 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":369,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:11:55.424195+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:10:49.223671+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:08:55.803671+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:10:03.916437+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:13:22.916 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10796 8 365281 365273 0.264309 0 - 0.0299513 2026-02-23T16:13:23.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:13:22 trial068 ceph-mon[38805]: pgmap v6239: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:13:23.916 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10797 8 365281 365273 0.264284 0 - 0.0299513 2026-02-23T16:13:23.974 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:13:23 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/335770456' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:13:23.974 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:13:23 trial068 ceph-mon[38805]: 5.a scrub starts 2026-02-23T16:13:24.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 16:13:23 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:16:13:23] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T16:13:24.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 16:13:23 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T16:13:23.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T16:13:24.916 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10798 8 365281 365273 0.26426 0 - 0.0299513 2026-02-23T16:13:25.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:13:24 trial068 ceph-mon[38805]: 5.a scrub ok 2026-02-23T16:13:25.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:13:24 trial068 ceph-mon[38805]: pgmap v6240: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:13:25.729 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:13:25.849 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:13:25.916 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10799 8 365281 365273 0.264236 0 - 0.0299513 2026-02-23T16:13:25.948 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:13:25.946+0000 7f80ad5b0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:13:25.948 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:13:25 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T16:13:25.949 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:13:25 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T16:13:25.949 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:13:25.947+0000 7f80ad5b0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:13:26.065 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:13:26.065 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":369,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:11:55.424195+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:10:49.223671+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:08:55.803671+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:10:03.916437+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:13:26.916 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T16:13:26.915937+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T16:13:26.916 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T16:13:26.916 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10800 8 365281 365273 0.264211 0 - 0.0299513 2026-02-23T16:13:27.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:13:26 trial068 ceph-mon[38805]: pgmap v6241: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:13:27.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:13:26 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2312912887' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:13:27.916 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10801 8 365281 365273 0.264187 0 - 0.0299513 2026-02-23T16:13:28.916 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10802 8 365281 365273 0.264162 0 - 0.0299513 2026-02-23T16:13:29.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:13:28 trial068 ceph-mon[38805]: pgmap v6242: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:13:29.104 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:13:29.221 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:13:29.325 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:13:29.323+0000 7faff998c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:13:29.325 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:13:29.323+0000 7faff998c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:13:29.435 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:13:29.435 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":369,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:11:55.424195+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:10:49.223671+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:08:55.803671+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:10:03.916437+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:13:29.916 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10803 8 365281 365273 0.264138 0 - 0.0299513 2026-02-23T16:13:30.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:13:29 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1515497051' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:13:30.916 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10804 8 365281 365273 0.264113 0 - 0.0299513 2026-02-23T16:13:31.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:13:30 trial068 ceph-mon[38805]: pgmap v6243: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:13:31.916 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10805 8 365281 365273 0.264089 0 - 0.0299513 2026-02-23T16:13:32.472 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:13:32.591 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:13:32.695 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:13:32.693+0000 7fc7b103a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:13:32.695 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:13:32.693+0000 7fc7b103a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:13:32.803 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:13:32.803 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":369,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:11:55.424195+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:10:49.223671+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:08:55.803671+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:10:03.916437+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:13:32.803 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:13:32 trial068 ceph-mon[38805]: pgmap v6244: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:13:32.916 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10806 8 365281 365273 0.264064 0 - 0.0299513 2026-02-23T16:13:33.917 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10807 8 365281 365273 0.26404 0 - 0.0299513 2026-02-23T16:13:33.974 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:13:33 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/299087023' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:13:34.266 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 16:13:33 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T16:13:33.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T16:13:34.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 16:13:33 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:16:13:33] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T16:13:34.917 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10808 8 365281 365273 0.264015 0 - 0.0299513 2026-02-23T16:13:35.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:13:34 trial068 ceph-mon[38805]: pgmap v6245: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:13:35.841 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:13:35.917 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10809 8 365281 365273 0.263991 0 - 0.0299513 2026-02-23T16:13:35.959 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:13:35.980 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:13:35 trial068 ceph-mon[38805]: 5.7 scrub starts 2026-02-23T16:13:35.981 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:13:35 trial068 ceph-mon[38805]: 5.7 scrub ok 2026-02-23T16:13:36.057 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:13:36.055+0000 7f71c2a9f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:13:36.057 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:13:36.055+0000 7f71c2a9f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:13:36.166 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:13:36.166 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":369,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:11:55.424195+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:10:49.223671+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:08:55.803671+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:10:03.916437+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:13:36.917 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10810 8 365281 365273 0.263967 0 - 0.0299513 2026-02-23T16:13:37.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:13:36 trial068 ceph-mon[38805]: pgmap v6246: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:13:37.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:13:36 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2835105977' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:13:37.917 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10811 8 365281 365273 0.263942 0 - 0.0299513 2026-02-23T16:13:38.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:13:37 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2026-02-23T16:13:38.917 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10812 8 365281 365273 0.263918 0 - 0.0299513 2026-02-23T16:13:39.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:13:38 trial068 ceph-mon[38805]: pgmap v6247: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:13:39.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:13:38 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-23T16:13:39.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:13:38 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2026-02-23T16:13:39.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:13:38 trial068 ceph-mon[38805]: pgmap v6248: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:13:39.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:13:38 trial068 ceph-mon[38805]: pgmap v6249: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:13:39.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:13:38 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T16:13:39.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:13:38 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2026-02-23T16:13:39.203 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:13:39.320 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:13:39.421 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:13:39.419+0000 7f28a7a71640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:13:39.421 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:13:39.419+0000 7f28a7a71640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:13:39.530 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:13:39.530 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":369,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:11:55.424195+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:10:49.223671+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:08:55.803671+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:10:03.916437+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:13:39.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:13:39 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/600695123' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:13:39.917 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10813 8 365281 365273 0.263893 0 - 0.0299513 2026-02-23T16:13:40.917 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10814 8 365281 365273 0.263869 0 - 0.0299513 2026-02-23T16:13:41.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:13:40 trial068 ceph-mon[38805]: pgmap v6250: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:13:41.917 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10815 8 365281 365273 0.263845 0 - 0.0299513 2026-02-23T16:13:42.571 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:13:42.690 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:13:42.794 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:13:42.792+0000 7f2cf8064640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:13:42.795 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:13:42.792+0000 7f2cf8064640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:13:42.904 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:13:42.904 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":369,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:11:55.424195+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:10:49.223671+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:08:55.803671+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:10:03.916437+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:13:42.917 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10816 8 365281 365273 0.26382 0 - 0.0299513 2026-02-23T16:13:43.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:13:42 trial068 ceph-mon[38805]: pgmap v6251: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:13:43.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:13:42 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1393023431' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:13:43.917 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10817 8 365281 365273 0.263796 0 - 0.0299513 2026-02-23T16:13:44.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 16:13:43 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:16:13:43] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T16:13:44.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 16:13:43 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T16:13:43.972Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T16:13:44.917 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10818 8 365281 365273 0.263771 0 - 0.0299513 2026-02-23T16:13:45.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:13:44 trial068 ceph-mon[38805]: pgmap v6252: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:13:45.917 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10819 8 365281 365273 0.263747 0 - 0.0299513 2026-02-23T16:13:45.941 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:13:46.057 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:13:46.152 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:13:46.150+0000 7f4aaef68640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:13:46.152 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:13:46.150+0000 7f4aaef68640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:13:46.266 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:13:46.267 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":369,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:11:55.424195+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:10:49.223671+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:08:55.803671+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:10:03.916437+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:13:46.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:13:46 trial068 ceph-mon[38805]: pgmap v6253: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:13:46.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:13:46 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2448480680' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:13:46.917 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T16:13:46.917312+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T16:13:46.918 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T16:13:46.918 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10820 8 365281 365273 0.263723 0 - 0.0299513 2026-02-23T16:13:47.918 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10821 8 365281 365273 0.263698 0 - 0.0299513 2026-02-23T16:13:48.918 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10822 8 365281 365273 0.263674 0 - 0.0299513 2026-02-23T16:13:49.304 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:13:49.330 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:13:48 trial068 ceph-mon[38805]: pgmap v6254: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:13:49.422 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:13:49.524 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:13:49.522+0000 7f80aeff6640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:13:49.524 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:13:49.522+0000 7f80aeff6640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:13:49.632 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:13:49.633 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":369,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:11:55.424195+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:10:49.223671+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:08:55.803671+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:10:03.916437+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:13:49.918 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10823 8 365281 365273 0.26365 0 - 0.0299513 2026-02-23T16:13:50.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:13:49 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/820656992' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:13:50.918 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10824 8 365281 365273 0.263625 0 - 0.0299513 2026-02-23T16:13:51.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:13:50 trial068 ceph-mon[38805]: pgmap v6255: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:13:51.918 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10825 8 365281 365273 0.263601 0 - 0.0299513 2026-02-23T16:13:52.670 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:13:52.789 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:13:52.892 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:13:52.890+0000 7fe148b21640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:13:52.892 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:13:52.890+0000 7fe148b21640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:13:52.918 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10826 8 365281 365273 0.263577 0 - 0.0299513 2026-02-23T16:13:53.000 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:13:53.000 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":369,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:11:55.424195+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:10:49.223671+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:08:55.803671+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:10:03.916437+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:13:53.001 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:13:52 trial068 ceph-mon[38805]: pgmap v6256: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:13:53.918 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10827 8 365281 365273 0.263552 0 - 0.0299513 2026-02-23T16:13:54.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:13:53 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/4049241185' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:13:54.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 16:13:53 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:16:13:53] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T16:13:54.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 16:13:53 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T16:13:53.972Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T16:13:54.918 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10828 8 365281 365273 0.263528 0 - 0.0299513 2026-02-23T16:13:55.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:13:54 trial068 ceph-mon[38805]: pgmap v6257: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:13:55.918 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10829 8 365281 365273 0.263503 0 - 0.0299513 2026-02-23T16:13:56.038 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:13:56.155 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:13:56.255 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:13:56.253+0000 7f3a654a9640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:13:56.255 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:13:56.253+0000 7f3a654a9640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:13:56.374 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:13:56.374 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":369,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:11:55.424195+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:10:49.223671+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:08:55.803671+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:10:03.916437+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:13:56.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:13:56 trial068 ceph-mon[38805]: pgmap v6258: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:13:56.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:13:56 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3458132011' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:13:56.918 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10830 8 365281 365273 0.263479 0 - 0.0299513 2026-02-23T16:13:57.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:13:57 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T16:13:57.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:13:57 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T16:13:57.918 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10831 8 365281 365273 0.263455 0 - 0.0299513 2026-02-23T16:13:58.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:13:58 trial068 ceph-mon[38805]: pgmap v6259: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:13:58.918 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10832 8 365281 365273 0.263431 0 - 0.0299513 2026-02-23T16:13:59.411 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:13:59.530 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:13:59.632 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:13:59.630+0000 7f3ec04f4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:13:59.632 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:13:59.630+0000 7f3ec04f4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:13:59.744 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:13:59.744 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":369,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:11:55.424195+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:10:49.223671+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:08:55.803671+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:10:03.916437+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:13:59.918 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10833 8 365281 365273 0.263406 0 - 0.0299513 2026-02-23T16:14:00.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:13:59 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1886088719' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:14:00.918 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10834 8 365281 365273 0.263382 0 - 0.0299513 2026-02-23T16:14:01.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:14:00 trial068 ceph-mon[38805]: pgmap v6260: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:14:01.918 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10835 8 365281 365273 0.263358 0 - 0.0299513 2026-02-23T16:14:02.783 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:14:02.903 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:14:02.918 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10836 8 365281 365273 0.263333 0 - 0.0299513 2026-02-23T16:14:03.001 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:14:02.999+0000 7fe26ae92640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:14:03.001 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:14:02.999+0000 7fe26ae92640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:14:03.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:14:02 trial068 ceph-mon[38805]: pgmap v6261: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:14:03.110 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:14:03.110 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":369,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:11:55.424195+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:10:49.223671+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:08:55.803671+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:10:03.916437+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:14:03.919 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10837 8 365281 365273 0.263309 0 - 0.0299513 2026-02-23T16:14:04.265 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 16:14:03 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T16:14:03.972Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T16:14:04.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:14:03 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3180151755' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:14:04.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 16:14:03 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:16:14:03] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T16:14:04.919 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10838 8 365281 365273 0.263285 0 - 0.0299513 2026-02-23T16:14:05.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:14:04 trial068 ceph-mon[38805]: pgmap v6262: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:14:05.919 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10839 8 365281 365273 0.26326 0 - 0.0299513 2026-02-23T16:14:06.148 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:14:06.266 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:14:06.365 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:14:06.363+0000 7f6d5684d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:14:06.366 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:14:06.364+0000 7f6d5684d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:14:06.475 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:14:06.476 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":369,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:11:55.424195+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:10:49.223671+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:08:55.803671+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:10:03.916437+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:14:06.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:14:06 trial068 ceph-mon[38805]: pgmap v6263: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:14:06.919 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T16:14:06.918663+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T16:14:06.919 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T16:14:06.919 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10840 8 365281 365273 0.263236 0 - 0.0299513 2026-02-23T16:14:07.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:14:07 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1928790197' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:14:07.919 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10841 8 365281 365273 0.263212 0 - 0.0299513 2026-02-23T16:14:08.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:14:08 trial068 ceph-mon[38805]: pgmap v6264: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:14:08.919 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10842 8 365281 365273 0.263188 0 - 0.0299513 2026-02-23T16:14:09.533 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:14:09.652 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:14:09.751 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:14:09.749+0000 7fae0e57b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:14:09.752 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:14:09.750+0000 7fae0e57b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:14:09.862 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:14:09.862 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":369,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:11:55.424195+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:10:49.223671+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:08:55.803671+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:10:03.916437+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:14:09.919 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10843 8 365281 365273 0.263163 0 - 0.0299513 2026-02-23T16:14:10.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:14:09 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3856299959' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:14:10.919 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10844 8 365281 365273 0.263139 0 - 0.0299513 2026-02-23T16:14:11.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:14:10 trial068 ceph-mon[38805]: pgmap v6265: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:14:11.919 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10845 8 365281 365273 0.263115 0 - 0.0299513 2026-02-23T16:14:12.900 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:14:12.919 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10846 8 365281 365273 0.26309 0 - 0.0299513 2026-02-23T16:14:13.020 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:14:13.120 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:14:13.118+0000 7fa8e4fae640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:14:13.120 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:14:13.118+0000 7fa8e4fae640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:14:13.229 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:14:13.229 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":369,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:11:55.424195+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:10:49.223671+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:08:55.803671+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:10:03.916437+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:14:13.230 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:14:12 trial068 ceph-mon[38805]: pgmap v6266: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:14:13.919 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10847 8 365281 365273 0.263066 0 - 0.0299513 2026-02-23T16:14:14.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:14:13 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/765714090' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:14:14.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:14:13 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T16:14:14.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:14:13 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T16:14:14.345 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 16:14:13 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:16:14:13] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T16:14:14.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 16:14:13 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T16:14:13.972Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T16:14:14.919 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10848 8 365281 365273 0.263042 0 - 0.0299513 2026-02-23T16:14:15.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:14:14 trial068 ceph-mon[38805]: pgmap v6267: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:14:15.919 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10849 8 365281 365273 0.263018 0 - 0.0299513 2026-02-23T16:14:16.266 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:14:16.383 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:14:16.484 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:14:16.482+0000 7f2860e9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:14:16.484 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:14:16.482+0000 7f2860e9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:14:16.593 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:14:16.593 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":369,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:11:55.424195+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:10:49.223671+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:08:55.803671+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:10:03.916437+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:14:16.920 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10850 8 365281 365273 0.262993 0 - 0.0299513 2026-02-23T16:14:17.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:14:16 trial068 ceph-mon[38805]: pgmap v6268: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:14:17.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:14:16 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/435717730' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:14:17.920 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10851 8 365281 365273 0.262969 0 - 0.0299513 2026-02-23T16:14:18.920 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10852 8 365281 365273 0.262945 0 - 0.0299513 2026-02-23T16:14:19.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:14:18 trial068 ceph-mon[38805]: pgmap v6269: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:14:19.631 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:14:19.749 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:14:19.847 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:14:19.845+0000 7f97735cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:14:19.847 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:14:19.845+0000 7f97735cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:14:19.920 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10853 8 365281 365273 0.262921 0 - 0.0299513 2026-02-23T16:14:19.956 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:14:19.957 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":369,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:11:55.424195+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:10:49.223671+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:08:55.803671+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:10:03.916437+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:14:20.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:14:19 trial068 ceph-mon[38805]: 5.8 scrub starts 2026-02-23T16:14:20.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:14:19 trial068 ceph-mon[38805]: 5.8 scrub ok 2026-02-23T16:14:20.920 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10854 8 365281 365273 0.262897 0 - 0.0299513 2026-02-23T16:14:21.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:14:20 trial068 ceph-mon[38805]: pgmap v6270: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:14:21.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:14:20 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1476795017' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:14:21.920 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10855 8 365281 365273 0.262872 0 - 0.0299513 2026-02-23T16:14:22.920 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10856 8 365281 365273 0.262848 0 - 0.0299513 2026-02-23T16:14:22.996 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:14:23.115 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:14:23.218 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:14:23.216+0000 7fc097e3c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:14:23.219 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:14:22 trial068 ceph-mon[38805]: pgmap v6271: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:14:23.219 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:14:23.216+0000 7fc097e3c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:14:23.327 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:14:23.327 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":369,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:11:55.424195+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:10:49.223671+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:08:55.803671+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:10:03.916437+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:14:23.920 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10857 8 365281 365273 0.262824 0 - 0.0299513 2026-02-23T16:14:24.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:14:23 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1981044238' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:14:24.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 16:14:23 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:16:14:23] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T16:14:24.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 16:14:23 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T16:14:23.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T16:14:24.920 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10858 8 365281 365273 0.2628 0 - 0.0299513 2026-02-23T16:14:25.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:14:24 trial068 ceph-mon[38805]: pgmap v6272: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:14:25.920 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10859 8 365281 365273 0.262776 0 - 0.0299513 2026-02-23T16:14:26.364 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:14:26.484 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:14:26.588 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:14:26.586+0000 7f300ca35640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:14:26.588 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:14:26.586+0000 7f300ca35640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:14:26.697 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:14:26.698 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":369,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:11:55.424195+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:10:49.223671+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:08:55.803671+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:10:03.916437+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:14:26.698 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:14:26 trial068 ceph-mon[38805]: pgmap v6273: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:14:26.920 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T16:14:26.920203+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T16:14:26.921 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T16:14:26.921 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10860 8 365281 365273 0.262751 0 - 0.0299513 2026-02-23T16:14:27.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:14:27 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2308800345' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:14:27.920 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10861 8 365281 365273 0.262727 0 - 0.0299513 2026-02-23T16:14:28.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:14:28 trial068 ceph-mon[38805]: pgmap v6274: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:14:28.920 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10862 8 365281 365273 0.262703 0 - 0.0299513 2026-02-23T16:14:29.735 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:14:29.853 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:14:29.921 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10863 8 365281 365273 0.262679 0 - 0.0299513 2026-02-23T16:14:29.954 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:14:29.952+0000 7f3f9ccee640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:14:29.954 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:14:29.952+0000 7f3f9ccee640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:14:30.066 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:14:30.066 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":369,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:11:55.424195+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:10:49.223671+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:08:55.803671+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:10:03.916437+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:14:30.921 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10864 8 365281 365273 0.262655 0 - 0.0299513 2026-02-23T16:14:31.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:14:30 trial068 ceph-mon[38805]: pgmap v6275: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:14:31.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:14:30 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2458923465' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:14:31.921 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10865 8 365281 365273 0.26263 0 - 0.0299513 2026-02-23T16:14:32.921 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10866 8 365281 365273 0.262606 0 - 0.0299513 2026-02-23T16:14:33.105 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:14:33.224 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:14:33.247 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:14:32 trial068 ceph-mon[38805]: pgmap v6276: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:14:33.327 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:14:33.325+0000 7f9c0173e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:14:33.328 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:14:33.326+0000 7f9c0173e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:14:33.437 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:14:33.437 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":369,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:11:55.424195+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:10:49.223671+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:08:55.803671+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:10:03.916437+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:14:33.921 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10867 8 365281 365273 0.262582 0 - 0.0299513 2026-02-23T16:14:34.265 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 16:14:33 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T16:14:33.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T16:14:34.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:14:33 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1125341361' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:14:34.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 16:14:33 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:16:14:33] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T16:14:34.921 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10868 8 365281 365273 0.262558 0 - 0.0299513 2026-02-23T16:14:35.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:14:34 trial068 ceph-mon[38805]: pgmap v6277: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:14:35.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:14:34 trial068 ceph-mon[38805]: osdmap e370: 4 total, 3 up, 4 in 2026-02-23T16:14:35.921 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10869 8 365281 365273 0.262534 0 - 0.0299513 2026-02-23T16:14:36.473 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:14:36.589 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:14:36.690 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:14:36.688+0000 7fd285308640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:14:36.691 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:14:36.689+0000 7fd285308640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:14:36.703 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:14:36 trial068 ceph-mon[38805]: 5.a scrub starts 2026-02-23T16:14:36.703 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:14:36 trial068 ceph-mon[38805]: osdmap e371: 4 total, 3 up, 4 in 2026-02-23T16:14:36.703 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:14:36 trial068 ceph-mon[38805]: 5.a scrub ok 2026-02-23T16:14:36.703 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:14:36 trial068 ceph-mon[38805]: pgmap v6280: 17 pgs: 1 active+clean+scrubbing, 1 active+recovery_unfound+undersized+degraded, 4 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:14:36.803 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:14:36.803 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":371,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:14:35.443613+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:10:49.223671+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:14:13.240512+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:10:03.916437+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:14:36.921 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10870 8 365281 365273 0.26251 0 - 0.0299513 2026-02-23T16:14:37.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:14:37 trial068 ceph-mon[38805]: 5.5 scrub starts 2026-02-23T16:14:37.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:14:37 trial068 ceph-mon[38805]: 5.5 scrub ok 2026-02-23T16:14:37.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:14:37 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1004130803' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:14:37.921 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10871 8 365281 365273 0.262485 0 - 0.0299513 2026-02-23T16:14:38.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:14:38 trial068 ceph-mon[38805]: pgmap v6281: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 1 active+clean+scrubbing, 4 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:14:38.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:14:38 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2026-02-23T16:14:38.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:14:38 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-23T16:14:38.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:14:38 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2026-02-23T16:14:38.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:14:38 trial068 ceph-mon[38805]: pgmap v6282: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 1 active+clean+scrubbing, 4 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:14:38.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:14:38 trial068 ceph-mon[38805]: pgmap v6283: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 1 active+clean+scrubbing, 4 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:14:38.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:14:38 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T16:14:38.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:14:38 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2026-02-23T16:14:38.921 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10872 8 365281 365273 0.262461 0 - 0.0299513 2026-02-23T16:14:39.841 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:14:39.921 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10873 8 365281 365273 0.262437 0 - 0.0299513 2026-02-23T16:14:39.957 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:14:40.063 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:14:40.061+0000 7fa978a3e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:14:40.063 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:14:40.061+0000 7fa978a3e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:14:40.173 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:14:40.173 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":371,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:14:35.443613+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:10:49.223671+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:14:13.240512+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:10:03.916437+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:14:40.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:14:40 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2388376207' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:14:40.921 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10874 8 365281 365273 0.262413 0 - 0.0299513 2026-02-23T16:14:41.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:14:41 trial068 ceph-mon[38805]: pgmap v6284: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 1 active+clean+scrubbing, 4 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:14:41.921 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10875 8 365281 365273 0.262389 0 - 0.0299513 2026-02-23T16:14:42.922 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10876 8 365281 365273 0.262365 0 - 0.0299513 2026-02-23T16:14:43.210 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:14:43.330 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:14:43.439 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:14:43.437+0000 7f8a6a37b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:14:43.439 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:14:43.437+0000 7f8a6a37b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:14:43.550 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:14:43.550 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":371,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:14:35.443613+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:10:49.223671+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:14:13.240512+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:10:03.916437+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:14:43.550 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:14:43 trial068 ceph-mon[38805]: pgmap v6285: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:14:43.922 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10877 8 365281 365273 0.262341 0 - 0.0299513 2026-02-23T16:14:44.225 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 16:14:43 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:16:14:43] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T16:14:44.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 16:14:43 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T16:14:43.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T16:14:44.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:14:44 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/4267516320' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:14:44.922 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10878 8 365281 365273 0.262317 0 - 0.0299513 2026-02-23T16:14:45.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:14:45 trial068 ceph-mon[38805]: pgmap v6286: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:14:45.922 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10879 8 365281 365273 0.262292 0 - 0.0299513 2026-02-23T16:14:46.586 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:14:46.703 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:14:46.724 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:14:46 trial068 ceph-mon[38805]: pgmap v6287: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:14:46.809 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:14:46.807+0000 7f369c12c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:14:46.810 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:14:46.808+0000 7f369c12c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:14:46.918 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:14:46.919 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":371,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:14:35.443613+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:10:49.223671+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:14:13.240512+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:10:03.916437+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:14:46.922 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T16:14:46.921777+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T16:14:46.922 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T16:14:46.923 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10880 8 365281 365273 0.262268 0 - 0.0299513 2026-02-23T16:14:47.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:14:47 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3792833843' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:14:47.922 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10881 8 365281 365273 0.262244 0 - 0.0299513 2026-02-23T16:14:48.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:14:48 trial068 ceph-mon[38805]: pgmap v6288: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:14:48.922 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10882 8 365281 365273 0.26222 0 - 0.0299513 2026-02-23T16:14:49.922 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10883 8 365281 365273 0.262196 0 - 0.0299513 2026-02-23T16:14:49.958 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:14:50.074 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:14:50.187 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:14:50.186+0000 7fc9d14bc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:14:50.188 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:14:50.186+0000 7fc9d14bc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:14:50.297 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:14:50.298 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":371,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:14:35.443613+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:10:49.223671+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:14:13.240512+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:10:03.916437+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:14:50.922 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10884 8 365281 365273 0.262172 0 - 0.0299513 2026-02-23T16:14:51.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:14:51 trial068 ceph-mon[38805]: 5.e scrub starts 2026-02-23T16:14:51.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:14:51 trial068 ceph-mon[38805]: pgmap v6289: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:14:51.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:14:51 trial068 ceph-mon[38805]: 5.e scrub ok 2026-02-23T16:14:51.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:14:51 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/931798332' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:14:51.922 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10885 8 365281 365273 0.262148 0 - 0.0299513 2026-02-23T16:14:52.922 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10886 8 365281 365273 0.262124 0 - 0.0299513 2026-02-23T16:14:53.336 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:14:53.453 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:14:53.475 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:14:53 trial068 ceph-mon[38805]: pgmap v6290: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:14:53.552 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:14:53.550+0000 7f478b04b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:14:53.552 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:14:53.550+0000 7f478b04b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:14:53.661 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:14:53.662 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":371,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:14:35.443613+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:10:49.223671+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:14:13.240512+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:10:03.916437+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:14:53.922 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10887 8 365281 365273 0.2621 0 - 0.0299513 2026-02-23T16:14:54.203 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 16:14:53 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:16:14:53] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T16:14:54.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 16:14:53 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T16:14:53.974Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T16:14:54.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:14:54 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T16:14:54.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:14:54 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T16:14:54.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:14:54 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2059212898' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:14:54.922 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10888 8 365281 365273 0.262076 0 - 0.0299513 2026-02-23T16:14:55.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:14:55 trial068 ceph-mon[38805]: pgmap v6291: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:14:55.922 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10889 8 365281 365273 0.262052 0 - 0.0299513 2026-02-23T16:14:56.699 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:14:56.724 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:14:56 trial068 ceph-mon[38805]: pgmap v6292: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:14:56.817 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:14:56.917 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:14:56.915+0000 7fb9b4e9a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:14:56.917 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:14:56.915+0000 7fb9b4e9a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:14:56.922 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10890 8 365281 365273 0.262027 0 - 0.0299513 2026-02-23T16:14:57.025 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:14:57.025 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":371,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:14:35.443613+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:10:49.223671+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:14:13.240512+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:10:03.916437+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:14:57.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:14:57 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2313926546' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:14:57.923 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10891 8 365281 365273 0.262003 0 - 0.0299513 2026-02-23T16:14:58.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:14:58 trial068 ceph-mon[38805]: pgmap v6293: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:14:58.923 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10892 8 365281 365273 0.261979 0 - 0.0299513 2026-02-23T16:14:59.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:14:59 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T16:14:59.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:14:59 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T16:14:59.923 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10893 8 365281 365273 0.261955 0 - 0.0299513 2026-02-23T16:15:00.063 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:15:00.180 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:15:00.274 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:15:00.272+0000 7fc2c9a3f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:15:00.274 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:15:00.272+0000 7fc2c9a3f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:15:00.383 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:15:00.383 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":371,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:14:35.443613+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:10:49.223671+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:14:13.240512+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:10:03.916437+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:15:00.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:15:00 trial068 ceph-mon[38805]: pgmap v6294: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:15:00.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:15:00 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3177133095' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:15:00.923 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10894 8 365281 365273 0.261931 0 - 0.0299513 2026-02-23T16:15:01.923 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10895 8 365281 365273 0.261907 0 - 0.0299513 2026-02-23T16:15:02.923 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10896 8 365281 365273 0.261883 0 - 0.0299513 2026-02-23T16:15:03.420 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:15:03.539 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:15:03.561 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:15:03 trial068 ceph-mon[38805]: pgmap v6295: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:15:03.638 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:15:03.636+0000 7f4f2b797640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:15:03.638 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:15:03.636+0000 7f4f2b797640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:15:03.747 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:15:03.747 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":371,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:14:35.443613+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:10:49.223671+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:14:13.240512+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:10:03.916437+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:15:03.923 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10897 8 365281 365273 0.261859 0 - 0.0299513 2026-02-23T16:15:04.204 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 16:15:03 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:16:15:03] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T16:15:04.265 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 16:15:03 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T16:15:03.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T16:15:04.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:15:04 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3092413836' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:15:04.923 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10898 8 365281 365273 0.261835 0 - 0.0299513 2026-02-23T16:15:05.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:15:05 trial068 ceph-mon[38805]: pgmap v6296: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:15:05.923 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10899 8 365281 365273 0.261811 0 - 0.0299513 2026-02-23T16:15:06.787 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:15:06.811 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:15:06 trial068 ceph-mon[38805]: pgmap v6297: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:15:06.905 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:15:06.923 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T16:15:06.923216+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T16:15:06.923 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T16:15:06.924 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10900 8 365281 365273 0.261787 0 - 0.0299513 2026-02-23T16:15:07.009 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:15:07.007+0000 7fd1850ab640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:15:07.009 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:15:07.007+0000 7fd1850ab640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:15:07.120 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:15:07.120 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":371,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:14:35.443613+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:10:49.223671+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:14:13.240512+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:10:03.916437+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:15:07.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:15:07 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1570305427' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:15:07.923 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10901 8 365281 365273 0.261763 0 - 0.0299513 2026-02-23T16:15:08.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:15:08 trial068 ceph-mon[38805]: pgmap v6298: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:15:08.923 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10902 8 365281 365273 0.261739 0 - 0.0299513 2026-02-23T16:15:09.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:15:09 trial068 ceph-mon[38805]: 5.7 scrub starts 2026-02-23T16:15:09.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:15:09 trial068 ceph-mon[38805]: 5.7 scrub ok 2026-02-23T16:15:09.923 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10903 8 365281 365273 0.261715 0 - 0.0299513 2026-02-23T16:15:10.158 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:15:10.277 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:15:10.376 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:15:10.374+0000 7f55fdacc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:15:10.377 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:15:10.375+0000 7f55fdacc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:15:10.486 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:15:10.487 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":371,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:14:35.443613+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:10:49.223671+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:14:13.240512+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:10:03.916437+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:15:10.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:15:10 trial068 ceph-mon[38805]: pgmap v6299: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:15:10.924 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10904 8 365281 365273 0.261691 0 - 0.0299513 2026-02-23T16:15:11.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:15:11 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3222991418' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:15:11.924 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10905 8 365281 365273 0.261667 0 - 0.0299513 2026-02-23T16:15:12.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:15:12 trial068 ceph-mon[38805]: pgmap v6300: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:15:12.924 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10906 8 365281 365273 0.261643 0 - 0.0299513 2026-02-23T16:15:13.524 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:15:13.641 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:15:13.741 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:15:13.739+0000 7f388d518640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:15:13.741 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:15:13.740+0000 7f388d518640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:15:13.852 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:15:13.852 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":371,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:14:35.443613+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:10:49.223671+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:14:13.240512+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:10:03.916437+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:15:13.924 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10907 8 365281 365273 0.261619 0 - 0.0299513 2026-02-23T16:15:14.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:15:13 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3071018078' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:15:14.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 16:15:13 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:16:15:13] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T16:15:14.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 16:15:13 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T16:15:13.972Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T16:15:14.924 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10908 8 365281 365273 0.261595 0 - 0.0299513 2026-02-23T16:15:15.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:15:14 trial068 ceph-mon[38805]: pgmap v6301: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:15:15.924 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10909 8 365281 365273 0.261571 0 - 0.0299513 2026-02-23T16:15:16.890 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:15:16.924 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10910 8 365281 365273 0.261547 0 - 0.0299513 2026-02-23T16:15:17.008 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:15:17.106 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:15:17.104+0000 7fe80bd03640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:15:17.106 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:15:17.104+0000 7fe80bd03640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:15:17.215 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:15:17.215 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":371,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:14:35.443613+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:10:49.223671+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:14:13.240512+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:10:03.916437+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:15:17.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:15:17 trial068 ceph-mon[38805]: pgmap v6302: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:15:17.924 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10911 8 365281 365273 0.261523 0 - 0.0299513 2026-02-23T16:15:18.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:15:18 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3132633249' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:15:18.924 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10912 8 365281 365273 0.261499 0 - 0.0299513 2026-02-23T16:15:19.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:15:19 trial068 ceph-mon[38805]: pgmap v6303: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:15:19.924 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10913 8 365281 365273 0.261475 0 - 0.0299513 2026-02-23T16:15:20.253 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:15:20.371 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:15:20.471 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:15:20.469+0000 7f5fd66aa640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:15:20.471 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:15:20.469+0000 7f5fd66aa640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:15:20.579 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:15:20.579 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":371,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:14:35.443613+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:10:49.223671+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:14:13.240512+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:10:03.916437+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:15:20.924 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10914 8 365281 365273 0.261451 0 - 0.0299513 2026-02-23T16:15:21.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:15:21 trial068 ceph-mon[38805]: pgmap v6304: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:15:21.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:15:21 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2971403796' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:15:21.924 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10915 8 365281 365273 0.261427 0 - 0.0299513 2026-02-23T16:15:22.924 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10916 8 365281 365273 0.261403 0 - 0.0299513 2026-02-23T16:15:23.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:15:23 trial068 ceph-mon[38805]: pgmap v6305: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:15:23.616 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:15:23.734 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:15:23.836 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:15:23.835+0000 7f1ab1217640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:15:23.837 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:15:23.835+0000 7f1ab1217640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:15:23.924 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10917 8 365281 365273 0.261379 0 - 0.0299513 2026-02-23T16:15:23.945 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:15:23.945 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":371,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:14:35.443613+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:10:49.223671+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:14:13.240512+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:10:03.916437+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:15:24.214 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 16:15:23 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:16:15:23] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T16:15:24.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 16:15:23 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T16:15:23.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T16:15:24.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:15:24 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2608323773' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:15:24.925 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10918 8 365281 365273 0.261355 0 - 0.0299513 2026-02-23T16:15:25.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:15:25 trial068 ceph-mon[38805]: pgmap v6306: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:15:25.925 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10919 8 365281 365273 0.261332 0 - 0.0299513 2026-02-23T16:15:26.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:15:26 trial068 ceph-mon[38805]: pgmap v6307: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:15:26.925 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T16:15:26.924540+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T16:15:26.925 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T16:15:26.925 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10920 8 365281 365273 0.261308 0 - 0.0299513 2026-02-23T16:15:26.982 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:15:27.101 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:15:27.198 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:15:27.196+0000 7fd02e82a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:15:27.198 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:15:27.196+0000 7fd02e82a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:15:27.308 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:15:27.308 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":371,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:14:35.443613+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:10:49.223671+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:14:13.240512+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:10:03.916437+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:15:27.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:15:27 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1448415791' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:15:27.925 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10921 8 365281 365273 0.261284 0 - 0.0299513 2026-02-23T16:15:28.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:15:28 trial068 ceph-mon[38805]: pgmap v6308: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:15:28.925 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10922 8 365281 365273 0.26126 0 - 0.0299513 2026-02-23T16:15:29.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:15:29 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T16:15:29.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:15:29 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T16:15:29.925 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10923 8 365281 365273 0.261236 0 - 0.0299513 2026-02-23T16:15:30.346 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:15:30.464 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:15:30.566 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:15:30.564+0000 7f9915ddd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:15:30.566 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:15:30.565+0000 7f9915ddd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:15:30.677 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:15:30.678 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":371,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:14:35.443613+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:10:49.223671+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:14:13.240512+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:10:03.916437+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:15:30.678 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:15:30 trial068 ceph-mon[38805]: pgmap v6309: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:15:30.925 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10924 8 365281 365273 0.261212 0 - 0.0299513 2026-02-23T16:15:31.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:15:31 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3511302715' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:15:31.925 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10925 8 365281 365273 0.261188 0 - 0.0299513 2026-02-23T16:15:32.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:15:32 trial068 ceph-mon[38805]: pgmap v6310: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:15:32.925 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10926 8 365281 365273 0.261164 0 - 0.0299513 2026-02-23T16:15:33.714 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:15:33.831 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:15:33.925 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10927 8 365281 365273 0.26114 0 - 0.0299513 2026-02-23T16:15:33.932 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:15:33.930+0000 7ffb6826a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:15:33.932 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:15:33.930+0000 7ffb6826a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:15:34.044 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 16:15:33 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:16:15:33] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T16:15:34.045 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:15:34.045 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":371,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:14:35.443613+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:10:49.223671+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:14:13.240512+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:10:03.916437+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:15:34.266 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 16:15:33 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T16:15:33.972Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T16:15:34.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:15:34 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1442636849' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:15:34.925 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10928 8 365281 365273 0.261116 0 - 0.0299513 2026-02-23T16:15:35.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:15:35 trial068 ceph-mon[38805]: pgmap v6311: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:15:35.925 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10929 8 365281 365273 0.261092 0 - 0.0299513 2026-02-23T16:15:36.925 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10930 8 365281 365273 0.261069 0 - 0.0299513 2026-02-23T16:15:37.083 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:15:37.200 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:15:37.301 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:15:37.299+0000 7f7becda2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:15:37.301 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:15:37.299+0000 7f7becda2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:15:37.410 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:15:37.410 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":371,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:14:35.443613+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:10:49.223671+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:14:13.240512+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:10:03.916437+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:15:37.411 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:15:37 trial068 ceph-mon[38805]: pgmap v6312: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:15:37.925 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10931 8 365281 365273 0.261045 0 - 0.0299513 2026-02-23T16:15:38.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:15:38 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/230852686' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:15:38.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:15:38 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2026-02-23T16:15:38.925 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10932 8 365281 365273 0.261021 0 - 0.0299513 2026-02-23T16:15:39.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:15:39 trial068 ceph-mon[38805]: pgmap v6313: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:15:39.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:15:39 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-23T16:15:39.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:15:39 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2026-02-23T16:15:39.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:15:39 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T16:15:39.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:15:39 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2026-02-23T16:15:39.925 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10933 8 365281 365273 0.260997 0 - 0.0299513 2026-02-23T16:15:40.448 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:15:40.472 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:15:40 trial068 ceph-mon[38805]: pgmap v6314: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:15:40.472 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:15:40 trial068 ceph-mon[38805]: pgmap v6315: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:15:40.566 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:15:40.674 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:15:40.672+0000 7fde12908640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:15:40.674 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:15:40.673+0000 7fde12908640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:15:40.783 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:15:40.784 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":371,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:14:35.443613+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:10:49.223671+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:14:13.240512+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:10:03.916437+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:15:40.925 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10934 8 365281 365273 0.260973 0 - 0.0299513 2026-02-23T16:15:41.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:15:41 trial068 ceph-mon[38805]: pgmap v6316: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:15:41.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:15:41 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1154445318' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:15:41.925 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10935 8 365281 365273 0.260949 0 - 0.0299513 2026-02-23T16:15:42.926 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10936 8 365281 365273 0.260925 0 - 0.0299513 2026-02-23T16:15:43.823 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:15:43.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:15:43 trial068 ceph-mon[38805]: pgmap v6317: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:15:43.926 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10937 8 365281 365273 0.260901 0 - 0.0299513 2026-02-23T16:15:43.939 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:15:44.041 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:15:44.039+0000 7f0413e7a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:15:44.041 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:15:44.039+0000 7f0413e7a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:15:44.153 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:15:44.153 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":371,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:14:35.443613+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:10:49.223671+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:14:13.240512+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:10:03.916437+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:15:44.153 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 16:15:43 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:16:15:43] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T16:15:44.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 16:15:43 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T16:15:43.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T16:15:44.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:15:44 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/153720852' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:15:44.926 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10938 8 365281 365273 0.260878 0 - 0.0299513 2026-02-23T16:15:45.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:15:45 trial068 ceph-mon[38805]: pgmap v6318: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:15:45.926 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10939 8 365281 365273 0.260854 0 - 0.0299513 2026-02-23T16:15:46.926 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T16:15:46.925873+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T16:15:46.926 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T16:15:46.926 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10940 8 365281 365273 0.26083 0 - 0.0299513 2026-02-23T16:15:47.191 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:15:47.309 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:15:47.408 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:15:47.406+0000 7fa55cb3f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:15:47.408 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:15:47.406+0000 7fa55cb3f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:15:47.520 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:15:47.520 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":371,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:14:35.443613+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:10:49.223671+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:14:13.240512+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:10:03.916437+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:15:47.521 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:15:47 trial068 ceph-mon[38805]: pgmap v6319: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:15:47.926 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10941 8 365281 365273 0.260806 0 - 0.0299513 2026-02-23T16:15:48.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:15:48 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1921854625' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:15:48.926 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10942 8 365281 365273 0.260782 0 - 0.0299513 2026-02-23T16:15:49.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:15:49 trial068 ceph-mon[38805]: pgmap v6320: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:15:49.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:15:49 trial068 ceph-mon[38805]: 5.8 scrub starts 2026-02-23T16:15:49.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:15:49 trial068 ceph-mon[38805]: 5.8 scrub ok 2026-02-23T16:15:49.926 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10943 8 365281 365273 0.260758 0 - 0.0299513 2026-02-23T16:15:50.559 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:15:50.679 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:15:50.785 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:15:50.783+0000 7f29c6687640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:15:50.785 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:15:50.783+0000 7f29c6687640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:15:50.894 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:15:50.894 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":371,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:14:35.443613+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:10:49.223671+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:14:13.240512+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:10:03.916437+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:15:50.926 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10944 8 365281 365273 0.260735 0 - 0.0299513 2026-02-23T16:15:51.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:15:51 trial068 ceph-mon[38805]: pgmap v6321: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:15:51.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:15:51 trial068 ceph-mon[38805]: 5.a scrub starts 2026-02-23T16:15:51.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:15:51 trial068 ceph-mon[38805]: 5.a scrub ok 2026-02-23T16:15:51.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:15:51 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2119570675' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:15:51.926 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10945 8 365281 365273 0.260711 0 - 0.0299513 2026-02-23T16:15:52.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:15:52 trial068 ceph-mon[38805]: osdmap e372: 4 total, 3 up, 4 in 2026-02-23T16:15:52.926 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10946 8 365281 365273 0.260687 0 - 0.0299513 2026-02-23T16:15:53.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:15:53 trial068 ceph-mon[38805]: pgmap v6323: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:15:53.926 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10947 8 365281 365273 0.260663 0 - 0.0299513 2026-02-23T16:15:53.931 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:15:54.048 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:15:54.144 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:15:54.142+0000 7f460e586640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:15:54.144 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:15:54.142+0000 7f460e586640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:15:54.256 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:15:54.256 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":372,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:15:51.462804+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:10:49.223671+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:14:13.240512+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:14:58.594223+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:15:54.256 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 16:15:53 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:16:15:53] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T16:15:54.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 16:15:53 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T16:15:53.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T16:15:54.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:15:54 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/185386614' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:15:54.927 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10948 8 365281 365273 0.260639 0 - 0.0299513 2026-02-23T16:15:55.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:15:55 trial068 ceph-mon[38805]: pgmap v6324: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:15:55.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:15:55 trial068 ceph-mon[38805]: osdmap e373: 4 total, 3 up, 4 in 2026-02-23T16:15:55.927 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10949 8 365281 365273 0.260616 0 - 0.0299513 2026-02-23T16:15:56.927 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10950 8 365281 365273 0.260592 0 - 0.0299513 2026-02-23T16:15:57.293 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:15:57.410 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:15:57.511 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:15:57.509+0000 7f85b6c81640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:15:57.511 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:15:57.509+0000 7f85b6c81640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:15:57.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:15:57 trial068 ceph-mon[38805]: pgmap v6326: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:15:57.621 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:15:57.621 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":373,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:15:55.452819+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:10:49.223671+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:14:13.240512+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:14:58.594223+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:15:57.927 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10951 8 365281 365273 0.260568 0 - 0.0299513 2026-02-23T16:15:58.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:15:58 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2054548086' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:15:58.927 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10952 8 365281 365273 0.260544 0 - 0.0299513 2026-02-23T16:15:59.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:15:59 trial068 ceph-mon[38805]: pgmap v6327: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:15:59.927 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10953 8 365281 365273 0.26052 0 - 0.0299513 2026-02-23T16:16:00.659 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:16:00.776 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:16:00.795 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:16:00 trial068 ceph-mon[38805]: 5.5 scrub starts 2026-02-23T16:16:00.795 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:16:00 trial068 ceph-mon[38805]: 5.5 scrub ok 2026-02-23T16:16:00.870 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:16:00.869+0000 7f0a72d23640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:16:00.871 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:16:00.869+0000 7f0a72d23640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:16:00.927 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10954 8 365281 365273 0.260497 0 - 0.0299513 2026-02-23T16:16:00.978 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:16:00.979 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":373,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:15:55.452819+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:10:49.223671+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:14:13.240512+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:14:58.594223+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:16:01.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:16:01 trial068 ceph-mon[38805]: pgmap v6328: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:16:01.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:16:01 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3615954995' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:16:01.927 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10955 8 365281 365273 0.260473 0 - 0.0299513 2026-02-23T16:16:02.927 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10956 8 365281 365273 0.260449 0 - 0.0299513 2026-02-23T16:16:03.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:16:03 trial068 ceph-mon[38805]: pgmap v6329: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:16:03.927 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10957 8 365281 365273 0.260425 0 - 0.0299513 2026-02-23T16:16:04.015 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:16:04.128 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:16:04.225 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:16:04.223+0000 7f9a46b53640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:16:04.225 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 16:16:03 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:16:16:03] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T16:16:04.225 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:16:04.223+0000 7f9a46b53640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:16:04.265 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 16:16:03 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T16:16:03.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T16:16:04.334 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:16:04.334 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":373,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:15:55.452819+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:10:49.223671+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:14:13.240512+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:14:58.594223+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:16:04.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:16:04 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3425692646' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:16:04.927 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10958 8 365281 365273 0.260401 0 - 0.0299513 2026-02-23T16:16:05.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:16:05 trial068 ceph-mon[38805]: pgmap v6330: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:16:05.927 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10959 8 365281 365273 0.260378 0 - 0.0299513 2026-02-23T16:16:06.928 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T16:16:06.927463+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T16:16:06.928 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T16:16:06.928 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10960 8 365281 365273 0.260354 0 - 0.0299513 2026-02-23T16:16:07.371 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:16:07.489 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:16:07.585 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:16:07.584+0000 7f97f025f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:16:07.586 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:16:07.584+0000 7f97f025f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:16:07.695 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:16:07.695 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":373,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:15:55.452819+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:10:49.223671+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:14:13.240512+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:14:58.594223+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:16:07.695 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:16:07 trial068 ceph-mon[38805]: pgmap v6331: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:16:07.928 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10961 8 365281 365273 0.26033 0 - 0.0299513 2026-02-23T16:16:08.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:16:08 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3902787754' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:16:08.928 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10962 8 365281 365273 0.260306 0 - 0.0299513 2026-02-23T16:16:09.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:16:09 trial068 ceph-mon[38805]: pgmap v6332: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:16:09.928 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10963 8 365281 365273 0.260283 0 - 0.0299513 2026-02-23T16:16:10.732 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:16:10.851 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:16:10.928 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10964 8 365281 365273 0.260259 0 - 0.0299513 2026-02-23T16:16:10.947 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:16:10.945+0000 7f06242c0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:16:10.947 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:16:10.946+0000 7f06242c0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:16:11.055 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:16:11.055 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":373,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:15:55.452819+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:10:49.223671+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:14:13.240512+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:14:58.594223+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:16:11.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:16:11 trial068 ceph-mon[38805]: pgmap v6333: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:16:11.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:16:11 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1041140860' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:16:11.928 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10965 8 365281 365273 0.260235 0 - 0.0299513 2026-02-23T16:16:12.928 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10966 8 365281 365273 0.260212 0 - 0.0299513 2026-02-23T16:16:13.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:16:13 trial068 ceph-mon[38805]: pgmap v6334: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:16:13.928 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10967 8 365281 365273 0.260188 0 - 0.0299513 2026-02-23T16:16:14.092 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:16:14.210 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:16:14.233 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 16:16:13 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:16:16:13] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T16:16:14.309 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:16:14.307+0000 7f7e88c61640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:16:14.309 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:16:14.308+0000 7f7e88c61640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:16:14.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 16:16:13 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T16:16:13.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T16:16:14.418 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:16:14.419 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":373,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:15:55.452819+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:10:49.223671+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:14:13.240512+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:14:58.594223+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:16:14.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:16:14 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T16:16:14.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:16:14 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T16:16:14.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:16:14 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3256263285' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:16:14.928 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10968 8 365281 365273 0.260164 0 - 0.0299513 2026-02-23T16:16:15.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:16:15 trial068 ceph-mon[38805]: pgmap v6335: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:16:15.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:16:15 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T16:16:15.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:16:15 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T16:16:15.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:16:15 trial068 ceph-mon[38805]: 5.7 scrub starts 2026-02-23T16:16:15.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:16:15 trial068 ceph-mon[38805]: 5.7 scrub ok 2026-02-23T16:16:15.928 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10969 8 365281 365273 0.26014 0 - 0.0299513 2026-02-23T16:16:16.928 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10970 8 365281 365273 0.260117 0 - 0.0299513 2026-02-23T16:16:17.457 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:16:17.578 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:16:17.681 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:16:17.679+0000 7f9b53df9640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:16:17.681 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:16:17.679+0000 7f9b53df9640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:16:17.793 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:16:17.793 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":373,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:15:55.452819+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:10:49.223671+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:14:13.240512+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:14:58.594223+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:16:17.793 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:16:17 trial068 ceph-mon[38805]: pgmap v6336: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:16:17.928 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10971 8 365281 365273 0.260093 0 - 0.0299513 2026-02-23T16:16:18.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:16:18 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1098299665' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:16:18.928 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10972 8 365281 365273 0.260069 0 - 0.0299513 2026-02-23T16:16:19.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:16:19 trial068 ceph-mon[38805]: pgmap v6337: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:16:19.928 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10973 8 365281 365273 0.260046 0 - 0.0299513 2026-02-23T16:16:20.829 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:16:20.928 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10974 8 365281 365273 0.260022 0 - 0.0299513 2026-02-23T16:16:20.947 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:16:21.043 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:16:21.041+0000 7fef32628640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:16:21.043 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:16:21.041+0000 7fef32628640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:16:21.151 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:16:21.151 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":373,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:15:55.452819+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:10:49.223671+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:14:13.240512+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:14:58.594223+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:16:21.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:16:21 trial068 ceph-mon[38805]: pgmap v6338: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:16:21.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:16:21 trial068 ceph-mon[38805]: 5.e scrub starts 2026-02-23T16:16:21.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:16:21 trial068 ceph-mon[38805]: 5.e scrub ok 2026-02-23T16:16:21.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:16:21 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/4174949506' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:16:21.929 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10975 8 365281 365273 0.259998 0 - 0.0299513 2026-02-23T16:16:22.929 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10976 8 365281 365273 0.259974 0 - 0.0299513 2026-02-23T16:16:23.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:16:23 trial068 ceph-mon[38805]: pgmap v6339: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:16:23.929 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10977 8 365281 365273 0.259951 0 - 0.0299513 2026-02-23T16:16:24.189 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:16:24.306 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:16:24.329 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 16:16:23 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:16:16:23] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T16:16:24.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 16:16:23 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T16:16:23.974Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T16:16:24.412 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:16:24.410+0000 7f1e34c1a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:16:24.412 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:16:24.410+0000 7f1e34c1a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:16:24.522 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:16:24.522 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":373,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:15:55.452819+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:10:49.223671+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:14:13.240512+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:14:58.594223+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:16:24.929 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10978 8 365281 365273 0.259927 0 - 0.0299513 2026-02-23T16:16:25.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:16:25 trial068 ceph-mon[38805]: pgmap v6340: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:16:25.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:16:25 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/188832618' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:16:25.929 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10979 8 365281 365273 0.259903 0 - 0.0299513 2026-02-23T16:16:26.929 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T16:16:26.928918+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T16:16:26.929 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T16:16:26.929 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10980 8 365281 365273 0.25988 0 - 0.0299513 2026-02-23T16:16:27.560 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:16:27.677 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:16:27.777 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:16:27.775+0000 7f714594d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:16:27.778 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:16:27 trial068 ceph-mon[38805]: pgmap v6341: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:16:27.778 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:16:27.776+0000 7f714594d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:16:27.887 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:16:27.887 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":373,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:15:55.452819+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:10:49.223671+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:14:13.240512+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:14:58.594223+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:16:27.929 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10981 8 365281 365273 0.259856 0 - 0.0299513 2026-02-23T16:16:28.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:16:28 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1208379300' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:16:28.929 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10982 8 365281 365273 0.259832 0 - 0.0299513 2026-02-23T16:16:29.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:16:29 trial068 ceph-mon[38805]: pgmap v6342: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:16:29.929 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10983 8 365281 365273 0.259809 0 - 0.0299513 2026-02-23T16:16:30.924 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:16:30.929 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10984 8 365281 365273 0.259785 0 - 0.0299513 2026-02-23T16:16:31.041 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:16:31.137 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:16:31.136+0000 7f89bec13640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:16:31.137 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:16:31.136+0000 7f89bec13640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:16:31.251 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:16:31.251 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":373,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:15:55.452819+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:10:49.223671+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:14:13.240512+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:14:58.594223+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:16:31.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:16:31 trial068 ceph-mon[38805]: pgmap v6343: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:16:31.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:16:31 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1065169942' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:16:31.929 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10985 8 365281 365273 0.259761 0 - 0.0299513 2026-02-23T16:16:32.929 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10986 8 365281 365273 0.259738 0 - 0.0299513 2026-02-23T16:16:33.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:16:33 trial068 ceph-mon[38805]: pgmap v6344: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:16:33.929 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10987 8 365281 365273 0.259714 0 - 0.0299513 2026-02-23T16:16:34.266 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 16:16:33 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T16:16:33.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T16:16:34.288 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:16:34.312 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 16:16:33 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:16:16:33] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T16:16:34.405 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:16:34.504 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:16:34.502+0000 7fc587547640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:16:34.504 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:16:34.502+0000 7fc587547640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:16:34.614 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:16:34.614 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":373,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:15:55.452819+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:10:49.223671+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:14:13.240512+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:14:58.594223+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:16:34.930 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10988 8 365281 365273 0.259691 0 - 0.0299513 2026-02-23T16:16:35.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:16:35 trial068 ceph-mon[38805]: pgmap v6345: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:16:35.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:16:35 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3797137698' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:16:35.930 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10989 8 365281 365273 0.259667 0 - 0.0299513 2026-02-23T16:16:36.930 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10990 8 365281 365273 0.259643 0 - 0.0299513 2026-02-23T16:16:37.667 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:16:37.785 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:16:37.808 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:16:37 trial068 ceph-mon[38805]: pgmap v6346: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:16:37.883 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:16:37.882+0000 7fe272e4e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:16:37.884 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:16:37.882+0000 7fe272e4e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:16:37.930 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10991 8 365281 365273 0.25962 0 - 0.0299513 2026-02-23T16:16:37.992 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:16:37.992 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":373,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:15:55.452819+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:10:49.223671+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:14:13.240512+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:14:58.594223+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:16:38.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:16:38 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2408171905' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:16:38.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:16:38 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2026-02-23T16:16:38.930 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10992 8 365281 365273 0.259596 0 - 0.0299513 2026-02-23T16:16:39.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:16:39 trial068 ceph-mon[38805]: pgmap v6347: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:16:39.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:16:39 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-23T16:16:39.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:16:39 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2026-02-23T16:16:39.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:16:39 trial068 ceph-mon[38805]: pgmap v6348: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:16:39.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:16:39 trial068 ceph-mon[38805]: pgmap v6349: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:16:39.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:16:39 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T16:16:39.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:16:39 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2026-02-23T16:16:39.930 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10993 8 365281 365273 0.259572 0 - 0.0299513 2026-02-23T16:16:40.930 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10994 8 365281 365273 0.259549 0 - 0.0299513 2026-02-23T16:16:41.030 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:16:41.149 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:16:41.250 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:16:41.248+0000 7f6b76a29640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:16:41.250 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:16:41.248+0000 7f6b76a29640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:16:41.360 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:16:41.361 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":373,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:15:55.452819+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:10:49.223671+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:14:13.240512+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:14:58.594223+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:16:41.930 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10995 8 365281 365273 0.259525 0 - 0.0299513 2026-02-23T16:16:42.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:16:41 trial068 ceph-mon[38805]: pgmap v6350: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:16:42.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:16:41 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3342182142' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:16:42.930 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10996 8 365281 365273 0.259502 0 - 0.0299513 2026-02-23T16:16:43.930 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10997 8 365281 365273 0.259478 0 - 0.0299513 2026-02-23T16:16:44.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:16:43 trial068 ceph-mon[38805]: pgmap v6351: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:16:44.094 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 16:16:43 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:16:16:43] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T16:16:44.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 16:16:43 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T16:16:43.974Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T16:16:44.398 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:16:44.518 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:16:44.618 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:16:44.616+0000 7f84f05a1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:16:44.619 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:16:44.617+0000 7f84f05a1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:16:44.726 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:16:44.726 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":373,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:15:55.452819+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:10:49.223671+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:14:13.240512+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:14:58.594223+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:16:44.930 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10998 8 365281 365273 0.259454 0 - 0.0299513 2026-02-23T16:16:45.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:16:44 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T16:16:45.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:16:44 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T16:16:45.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:16:44 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3059230948' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:16:45.931 INFO:tasks.radosbench.radosbench.0.trial068.stdout:10999 8 365281 365273 0.259431 0 - 0.0299513 2026-02-23T16:16:46.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:16:45 trial068 ceph-mon[38805]: pgmap v6352: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:16:46.930 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T16:16:46.930446+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T16:16:46.931 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T16:16:46.931 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11000 8 365281 365273 0.259407 0 - 0.0299513 2026-02-23T16:16:47.764 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:16:47.883 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:16:47.931 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11001 8 365281 365273 0.259384 0 - 0.0299513 2026-02-23T16:16:47.980 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:16:47.978+0000 7f3670b19640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:16:47.980 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:16:47.978+0000 7f3670b19640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:16:47.998 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:16:47 trial068 ceph-mon[38805]: pgmap v6353: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:16:48.088 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:16:48.089 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":373,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:15:55.452819+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:10:49.223671+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:14:13.240512+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:14:58.594223+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:16:48.931 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11002 8 365281 365273 0.25936 0 - 0.0299513 2026-02-23T16:16:49.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:16:48 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2994086143' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:16:49.931 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11003 8 365281 365273 0.259336 0 - 0.0299513 2026-02-23T16:16:50.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:16:49 trial068 ceph-mon[38805]: pgmap v6354: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:16:50.931 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11004 8 365281 365273 0.259313 0 - 0.0299513 2026-02-23T16:16:51.127 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:16:51.244 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:16:51.341 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:16:51.339+0000 7f4a31599640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:16:51.341 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:16:51.339+0000 7f4a31599640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:16:51.450 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:16:51.450 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":373,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:15:55.452819+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:10:49.223671+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:14:13.240512+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:14:58.594223+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:16:51.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:16:51 trial068 ceph-mon[38805]: pgmap v6355: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:16:51.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:16:51 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2171103622' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:16:51.931 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11005 8 365281 365273 0.259289 0 - 0.0299513 2026-02-23T16:16:52.931 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11006 8 365281 365273 0.259266 0 - 0.0299513 2026-02-23T16:16:53.931 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11007 8 365281 365273 0.259242 0 - 0.0299513 2026-02-23T16:16:54.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:16:53 trial068 ceph-mon[38805]: pgmap v6356: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:16:54.094 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 16:16:53 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:16:16:53] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T16:16:54.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 16:16:53 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T16:16:53.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T16:16:54.488 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:16:54.606 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:16:54.713 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:16:54.711+0000 7fa832e24640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:16:54.713 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:16:54.712+0000 7fa832e24640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:16:54.823 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:16:54.823 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":373,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:15:55.452819+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:10:49.223671+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:14:13.240512+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:14:58.594223+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:16:54.932 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11008 8 365281 365273 0.259219 0 - 0.0299513 2026-02-23T16:16:55.932 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11009 8 365281 365273 0.259195 0 - 0.0299513 2026-02-23T16:16:56.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:16:55 trial068 ceph-mon[38805]: pgmap v6357: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:16:56.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:16:55 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1699812819' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:16:56.932 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11010 8 365281 365273 0.259172 0 - 0.0299513 2026-02-23T16:16:57.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:16:56 trial068 ceph-mon[38805]: osdmap e374: 4 total, 3 up, 4 in 2026-02-23T16:16:57.861 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:16:57.932 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11011 8 365281 365273 0.259148 0 - 0.0299513 2026-02-23T16:16:57.978 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:16:58.000 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:16:57 trial068 ceph-mon[38805]: pgmap v6359: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:16:58.080 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:16:58.078+0000 7faddc233640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:16:58.080 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:16:58.078+0000 7faddc233640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:16:58.188 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:16:58.189 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":374,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:16:55.732495+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:16:14.421864+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:14:13.240512+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:14:58.594223+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:16:58.932 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11012 8 365281 365273 0.259125 0 - 0.0299513 2026-02-23T16:16:59.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:16:58 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/699427334' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:16:59.932 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11013 8 365281 365273 0.259101 0 - 0.0299513 2026-02-23T16:17:00.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:16:59 trial068 ceph-mon[38805]: pgmap v6360: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:17:00.932 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11014 8 365281 365273 0.259077 0 - 0.0299513 2026-02-23T16:17:01.226 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:17:01.343 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:17:01.441 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:17:01.439+0000 7fb97074d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:17:01.441 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:17:01.440+0000 7fb97074d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:17:01.550 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:17:01.551 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":375,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:17:00.462137+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:16:14.421864+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:14:13.240512+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:14:58.594223+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:17:01.551 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:17:01 trial068 ceph-mon[38805]: osdmap e375: 4 total, 3 up, 4 in 2026-02-23T16:17:01.551 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:17:01 trial068 ceph-mon[38805]: pgmap v6362: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:17:01.932 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11015 8 365281 365273 0.259054 0 - 0.0299513 2026-02-23T16:17:02.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:17:02 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/578248726' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:17:02.932 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11016 8 365281 365273 0.25903 0 - 0.0299513 2026-02-23T16:17:03.745 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:17:03 trial068 ceph-mon[38805]: 5.a scrub starts 2026-02-23T16:17:03.745 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:17:03 trial068 ceph-mon[38805]: pgmap v6363: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:17:03.745 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:17:03 trial068 ceph-mon[38805]: 5.a scrub ok 2026-02-23T16:17:03.932 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11017 8 365281 365273 0.259007 0 - 0.0299513 2026-02-23T16:17:04.094 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 16:17:03 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:16:17:03] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T16:17:04.265 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 16:17:03 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T16:17:03.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T16:17:04.589 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:17:04.710 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:17:04.733 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:17:04 trial068 ceph-mon[38805]: 5.5 scrub starts 2026-02-23T16:17:04.733 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:17:04 trial068 ceph-mon[38805]: 5.5 scrub ok 2026-02-23T16:17:04.811 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:17:04.809+0000 7f7e7d5a0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:17:04.811 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:17:04.810+0000 7f7e7d5a0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:17:04.919 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:17:04.919 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":375,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:17:00.462137+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:16:14.421864+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:14:13.240512+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:14:58.594223+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:17:04.932 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11018 8 365281 365273 0.258983 0 - 0.0299513 2026-02-23T16:17:05.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:17:05 trial068 ceph-mon[38805]: pgmap v6364: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:17:05.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:17:05 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/4008054190' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:17:05.932 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11019 8 365281 365273 0.25896 0 - 0.0299513 2026-02-23T16:17:06.933 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T16:17:06.932357+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T16:17:06.933 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T16:17:06.933 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11020 8 365281 365273 0.258936 0 - 0.0299513 2026-02-23T16:17:07.933 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11021 8 365281 365273 0.258913 0 - 0.0299513 2026-02-23T16:17:07.956 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:17:08.075 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:17:08.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:17:07 trial068 ceph-mon[38805]: pgmap v6365: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:17:08.179 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:17:08.177+0000 7f92fdd2e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:17:08.179 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:17:08.177+0000 7f92fdd2e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:17:08.288 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:17:08.289 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":375,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:17:00.462137+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:16:14.421864+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:14:13.240512+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:14:58.594223+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:17:08.933 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11022 8 365281 365273 0.258889 0 - 0.0299513 2026-02-23T16:17:09.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:17:08 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1789947333' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:17:09.933 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11023 8 365281 365273 0.258866 0 - 0.0299513 2026-02-23T16:17:10.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:17:09 trial068 ceph-mon[38805]: pgmap v6366: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:17:10.933 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11024 8 365281 365273 0.258842 0 - 0.0299513 2026-02-23T16:17:11.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:17:10 trial068 ceph-mon[38805]: 5.8 scrub starts 2026-02-23T16:17:11.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:17:10 trial068 ceph-mon[38805]: 5.8 scrub ok 2026-02-23T16:17:11.325 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:17:11.441 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:17:11.542 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:17:11.540+0000 7efe7e89b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:17:11.542 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:17:11.540+0000 7efe7e89b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:17:11.654 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:17:11.654 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":375,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:17:00.462137+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:16:14.421864+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:14:13.240512+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:14:58.594223+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:17:11.933 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11025 8 365281 365273 0.258819 0 - 0.0299513 2026-02-23T16:17:12.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:17:11 trial068 ceph-mon[38805]: pgmap v6367: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:17:12.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:17:11 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/876648927' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:17:12.933 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11026 8 365281 365273 0.258796 0 - 0.0299513 2026-02-23T16:17:13.933 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11027 8 365281 365273 0.258772 0 - 0.0299513 2026-02-23T16:17:14.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:17:13 trial068 ceph-mon[38805]: pgmap v6368: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:17:14.094 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 16:17:13 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:16:17:13] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T16:17:14.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 16:17:13 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T16:17:13.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T16:17:14.690 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:17:14.808 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:17:14.907 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:17:14.905+0000 7f5537636640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:17:14.907 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:17:14.905+0000 7f5537636640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:17:14.933 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11028 8 365281 365273 0.258749 0 - 0.0299513 2026-02-23T16:17:15.015 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:17:15.016 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":375,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:17:00.462137+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:16:14.421864+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:14:13.240512+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:14:58.594223+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:17:15.933 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11029 8 365281 365273 0.258725 0 - 0.0299513 2026-02-23T16:17:16.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:17:15 trial068 ceph-mon[38805]: pgmap v6369: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:17:16.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:17:15 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1146034974' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:17:16.933 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11030 8 365281 365273 0.258702 0 - 0.0299513 2026-02-23T16:17:17.933 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11031 8 365281 365273 0.258678 0 - 0.0299513 2026-02-23T16:17:18.053 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:17:18.077 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:17:17 trial068 ceph-mon[38805]: pgmap v6370: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:17:18.170 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:17:18.278 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:17:18.276+0000 7f39c0a1a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:17:18.278 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:17:18.276+0000 7f39c0a1a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:17:18.386 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:17:18.386 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":375,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:17:00.462137+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:16:14.421864+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:14:13.240512+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:14:58.594223+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:17:18.933 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11032 8 365281 365273 0.258655 0 - 0.0299513 2026-02-23T16:17:19.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:17:18 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/757504109' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:17:19.934 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11033 8 365281 365273 0.258631 0 - 0.0299513 2026-02-23T16:17:20.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:17:19 trial068 ceph-mon[38805]: pgmap v6371: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:17:20.934 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11034 8 365281 365273 0.258608 0 - 0.0299513 2026-02-23T16:17:21.424 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:17:21.544 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:17:21.644 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:17:21.642+0000 7efd9a930640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:17:21.644 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:17:21.643+0000 7efd9a930640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:17:21.752 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:17:21.752 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":375,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:17:00.462137+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:16:14.421864+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:14:13.240512+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:14:58.594223+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:17:21.753 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:17:21 trial068 ceph-mon[38805]: 5.7 scrub starts 2026-02-23T16:17:21.753 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:17:21 trial068 ceph-mon[38805]: pgmap v6372: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:17:21.934 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11035 8 365281 365273 0.258584 0 - 0.0299513 2026-02-23T16:17:22.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:17:22 trial068 ceph-mon[38805]: 5.7 scrub ok 2026-02-23T16:17:22.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:17:22 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1262855551' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:17:22.934 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11036 8 365281 365273 0.258561 0 - 0.0299513 2026-02-23T16:17:23.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:17:23 trial068 ceph-mon[38805]: pgmap v6373: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:17:23.934 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11037 8 365281 365273 0.258538 0 - 0.0299513 2026-02-23T16:17:24.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 16:17:23 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:16:17:23] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T16:17:24.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 16:17:23 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T16:17:23.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T16:17:24.789 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:17:24.907 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:17:24.934 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11038 8 365281 365273 0.258514 0 - 0.0299513 2026-02-23T16:17:25.009 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:17:25.007+0000 7f1bf5703640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:17:25.009 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:17:25.007+0000 7f1bf5703640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:17:25.119 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:17:25.119 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":375,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:17:00.462137+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:16:14.421864+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:14:13.240512+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:14:58.594223+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:17:25.934 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11039 8 365281 365273 0.258491 0 - 0.0299513 2026-02-23T16:17:26.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:17:25 trial068 ceph-mon[38805]: pgmap v6374: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:17:26.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:17:25 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3908136743' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:17:26.934 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T16:17:26.933890+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T16:17:26.934 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T16:17:26.934 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11040 8 365281 365273 0.258467 0 - 0.0299513 2026-02-23T16:17:27.934 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11041 8 365281 365273 0.258444 0 - 0.0299513 2026-02-23T16:17:28.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:17:27 trial068 ceph-mon[38805]: pgmap v6375: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:17:28.156 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:17:28.275 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:17:28.383 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:17:28.381+0000 7fc62bdf6640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:17:28.383 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:17:28.381+0000 7fc62bdf6640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:17:28.493 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:17:28.493 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":375,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:17:00.462137+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:16:14.421864+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:14:13.240512+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:14:58.594223+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:17:28.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:17:28 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2074878523' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:17:28.934 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11042 8 365281 365273 0.258421 0 - 0.0299513 2026-02-23T16:17:29.934 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11043 8 365281 365273 0.258397 0 - 0.0299513 2026-02-23T16:17:30.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:17:29 trial068 ceph-mon[38805]: pgmap v6376: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:17:30.934 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11044 8 365281 365273 0.258374 0 - 0.0299513 2026-02-23T16:17:31.551 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:17:31.668 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:17:31.720 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:17:31 trial068 ceph-mon[38805]: pgmap v6377: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:17:31.766 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:17:31.764+0000 7fd069981640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:17:31.766 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:17:31.764+0000 7fd069981640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:17:31.874 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:17:31.874 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":375,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:17:00.462137+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:16:14.421864+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:14:13.240512+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:14:58.594223+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:17:31.934 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11045 8 365281 365273 0.25835 0 - 0.0299513 2026-02-23T16:17:32.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:17:32 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3259646269' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:17:32.934 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11046 8 365281 365273 0.258327 0 - 0.0299513 2026-02-23T16:17:33.744 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:17:33 trial068 ceph-mon[38805]: pgmap v6378: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:17:33.934 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11047 8 365281 365273 0.258304 0 - 0.0299513 2026-02-23T16:17:34.093 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 16:17:33 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:16:17:33] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T16:17:34.266 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 16:17:33 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T16:17:33.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T16:17:34.910 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:17:34.935 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11048 8 365281 365273 0.25828 0 - 0.0299513 2026-02-23T16:17:35.027 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:17:35.131 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:17:35.129+0000 7fb3ec33d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:17:35.132 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:17:35.130+0000 7fb3ec33d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:17:35.239 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:17:35.239 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":375,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:17:00.462137+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:16:14.421864+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:14:13.240512+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:14:58.594223+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:17:35.935 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11049 8 365281 365273 0.258257 0 - 0.0299513 2026-02-23T16:17:36.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:17:35 trial068 ceph-mon[38805]: pgmap v6379: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:17:36.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:17:35 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/840776822' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:17:36.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:17:35 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T16:17:36.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:17:35 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T16:17:36.935 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11050 8 365281 365273 0.258233 0 - 0.0299513 2026-02-23T16:17:37.935 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11051 8 365281 365273 0.25821 0 - 0.0299513 2026-02-23T16:17:38.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:17:37 trial068 ceph-mon[38805]: pgmap v6380: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:17:38.297 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:17:38.414 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:17:38.521 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:17:38.519+0000 7fa4ed2a1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:17:38.521 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:17:38.520+0000 7fa4ed2a1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:17:38.631 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:17:38.631 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":375,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:17:00.462137+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:16:14.421864+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:14:13.240512+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:14:58.594223+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:17:38.935 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11052 8 365281 365273 0.258187 0 - 0.0299513 2026-02-23T16:17:39.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:17:38 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2879549763' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:17:39.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:17:38 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2026-02-23T16:17:39.935 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11053 8 365281 365273 0.258163 0 - 0.0299513 2026-02-23T16:17:40.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:17:39 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T16:17:40.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:17:39 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T16:17:40.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:17:39 trial068 ceph-mon[38805]: pgmap v6381: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:17:40.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:17:39 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T16:17:40.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:17:39 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T16:17:40.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:17:39 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T16:17:40.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:17:39 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T16:17:40.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:17:39 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-23T16:17:40.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:17:39 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2026-02-23T16:17:40.345 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:17:39 trial068 ceph-mon[38805]: pgmap v6382: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:17:40.345 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:17:39 trial068 ceph-mon[38805]: pgmap v6383: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:17:40.345 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:17:39 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T16:17:40.345 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:17:39 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2026-02-23T16:17:40.935 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11054 8 365281 365273 0.25814 0 - 0.0299513 2026-02-23T16:17:41.668 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:17:41.786 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:17:41.808 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:17:41 trial068 ceph-mon[38805]: pgmap v6384: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:17:41.891 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:17:41.889+0000 7f1559c7d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:17:41.891 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:17:41.889+0000 7f1559c7d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:17:41.935 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11055 8 365281 365273 0.258117 0 - 0.0299513 2026-02-23T16:17:41.999 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:17:41.999 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":375,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:17:00.462137+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:16:14.421864+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:14:13.240512+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:14:58.594223+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:17:42.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:17:42 trial068 ceph-mon[38805]: 5.e scrub starts 2026-02-23T16:17:42.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:17:42 trial068 ceph-mon[38805]: 5.e scrub ok 2026-02-23T16:17:42.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:17:42 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1583378338' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:17:42.935 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11056 8 365281 365273 0.258093 0 - 0.0299513 2026-02-23T16:17:43.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:17:43 trial068 ceph-mon[38805]: pgmap v6385: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:17:43.935 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11057 8 365281 365273 0.25807 0 - 0.0299513 2026-02-23T16:17:44.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 16:17:43 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:16:17:43] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T16:17:44.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 16:17:43 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T16:17:43.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T16:17:44.935 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11058 8 365281 365273 0.258047 0 - 0.0299513 2026-02-23T16:17:45.037 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:17:45.154 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:17:45.259 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:17:45.257+0000 7f8f9bfee640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:17:45.259 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:17:45.257+0000 7f8f9bfee640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:17:45.369 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:17:45.369 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":375,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:17:00.462137+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:16:14.421864+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:14:13.240512+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:14:58.594223+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:17:45.935 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11059 8 365281 365273 0.258023 0 - 0.0299513 2026-02-23T16:17:46.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:17:46 trial068 ceph-mon[38805]: pgmap v6386: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:17:46.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:17:46 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1172700435' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:17:46.935 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T16:17:46.935334+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T16:17:46.936 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T16:17:46.936 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11060 8 365281 365273 0.258 0 - 0.0299513 2026-02-23T16:17:47.936 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11061 8 365281 365273 0.257977 0 - 0.0299513 2026-02-23T16:17:48.408 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:17:48.432 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:17:48 trial068 ceph-mon[38805]: pgmap v6387: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:17:48.526 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:17:48.623 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:17:48.621+0000 7f3ec11cf640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:17:48.623 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:17:48.622+0000 7f3ec11cf640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:17:48.732 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:17:48.732 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":375,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:17:00.462137+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:16:14.421864+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:14:13.240512+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:14:58.594223+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:17:48.936 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11062 8 365281 365273 0.257953 0 - 0.0299513 2026-02-23T16:17:49.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:17:49 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/114293788' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:17:49.936 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11063 8 365281 365273 0.25793 0 - 0.0299513 2026-02-23T16:17:50.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:17:50 trial068 ceph-mon[38805]: pgmap v6388: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:17:50.936 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11064 8 365281 365273 0.257907 0 - 0.0299513 2026-02-23T16:17:51.767 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:17:51.792 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:17:51 trial068 ceph-mon[38805]: pgmap v6389: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:17:51.885 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:17:51.936 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11065 8 365281 365273 0.257883 0 - 0.0299513 2026-02-23T16:17:51.988 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:17:51.986+0000 7f7442b2f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:17:51.988 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:17:51.986+0000 7f7442b2f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:17:52.097 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:17:52.097 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":375,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:17:00.462137+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:16:14.421864+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:14:13.240512+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:14:58.594223+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:17:52.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:17:52 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2397878445' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:17:52.936 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11066 8 365281 365273 0.25786 0 - 0.0299513 2026-02-23T16:17:53.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:17:53 trial068 ceph-mon[38805]: pgmap v6390: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:17:53.936 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11067 8 365281 365273 0.257837 0 - 0.0299513 2026-02-23T16:17:54.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 16:17:53 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:16:17:53] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T16:17:54.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 16:17:53 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T16:17:53.972Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T16:17:54.936 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11068 8 365281 365273 0.257813 0 - 0.0299513 2026-02-23T16:17:55.136 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:17:55.253 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:17:55.350 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:17:55.348+0000 7f6470ca3640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:17:55.350 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:17:55.348+0000 7f6470ca3640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:17:55.458 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:17:55.458 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":375,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:17:00.462137+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:16:14.421864+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:14:13.240512+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:14:58.594223+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:17:55.936 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11069 8 365281 365273 0.25779 0 - 0.0299513 2026-02-23T16:17:56.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:17:56 trial068 ceph-mon[38805]: pgmap v6391: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:17:56.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:17:56 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2466313675' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:17:56.936 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11070 8 365281 365273 0.257767 0 - 0.0299513 2026-02-23T16:17:57.936 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11071 8 365281 365273 0.257744 0 - 0.0299513 2026-02-23T16:17:58.496 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:17:58.520 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:17:58 trial068 ceph-mon[38805]: pgmap v6392: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:17:58.613 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:17:58.713 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:17:58.711+0000 7fc29399e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:17:58.714 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:17:58.712+0000 7fc29399e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:17:58.823 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:17:58.824 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":375,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:17:00.462137+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:16:14.421864+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:14:13.240512+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:14:58.594223+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:17:58.936 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11072 8 365281 365273 0.25772 0 - 0.0299513 2026-02-23T16:17:59.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:17:59 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2381208678' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:17:59.936 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11073 8 365281 365273 0.257697 0 - 0.0299513 2026-02-23T16:18:00.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:18:00 trial068 ceph-mon[38805]: pgmap v6393: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:18:00.936 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11074 8 365281 365273 0.257674 0 - 0.0299513 2026-02-23T16:18:01.860 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:18:01.937 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11075 8 365281 365273 0.25765 0 - 0.0299513 2026-02-23T16:18:01.977 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:18:02.074 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:18:02.072+0000 7f38a6d9c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:18:02.074 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:18:02.072+0000 7f38a6d9c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:18:02.185 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:18:02.185 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":375,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:17:00.462137+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:16:14.421864+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:14:13.240512+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:14:58.594223+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:18:02.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:18:02 trial068 ceph-mon[38805]: pgmap v6394: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:18:02.937 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11076 8 365281 365273 0.257627 0 - 0.0299513 2026-02-23T16:18:03.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:18:03 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1624952039' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:18:03.937 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11077 8 365281 365273 0.257604 0 - 0.0299513 2026-02-23T16:18:04.265 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 16:18:03 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T16:18:03.974Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T16:18:04.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:18:04 trial068 ceph-mon[38805]: pgmap v6395: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:18:04.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 16:18:03 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:16:18:03] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T16:18:04.937 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11078 8 365281 365273 0.257581 0 - 0.0299513 2026-02-23T16:18:05.225 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:18:05.342 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:18:05.454 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:18:05.452+0000 7f2ecacfc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:18:05.455 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:18:05.453+0000 7f2ecacfc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:18:05.568 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:18:05.568 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":375,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:17:00.462137+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:16:14.421864+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:14:13.240512+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:14:58.594223+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:18:05.937 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11079 8 365281 365273 0.257557 0 - 0.0299513 2026-02-23T16:18:06.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:18:06 trial068 ceph-mon[38805]: pgmap v6396: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:18:06.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:18:06 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/968193434' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:18:06.937 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T16:18:06.936751+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T16:18:06.937 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T16:18:06.937 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11080 8 365281 365273 0.257534 0 - 0.0299513 2026-02-23T16:18:07.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:18:07 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T16:18:07.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:18:07 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T16:18:07.937 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11081 8 365281 365273 0.257511 0 - 0.0299513 2026-02-23T16:18:08.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:18:08 trial068 ceph-mon[38805]: pgmap v6397: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:18:08.605 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:18:08.722 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:18:08.820 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:18:08.818+0000 7f268818f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:18:08.821 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:18:08.819+0000 7f268818f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:18:08.932 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:18:08.933 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":375,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:17:00.462137+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:16:14.421864+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:14:13.240512+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:14:58.594223+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:18:08.937 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11082 8 365281 365273 0.257488 0 - 0.0299513 2026-02-23T16:18:09.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:18:09 trial068 ceph-mon[38805]: 5.5 scrub starts 2026-02-23T16:18:09.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:18:09 trial068 ceph-mon[38805]: 5.5 scrub ok 2026-02-23T16:18:09.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:18:09 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1691630094' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:18:09.937 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11083 8 365281 365273 0.257465 0 - 0.0299513 2026-02-23T16:18:10.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:18:10 trial068 ceph-mon[38805]: pgmap v6398: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:18:10.937 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11084 8 365281 365273 0.257441 0 - 0.0299513 2026-02-23T16:18:11.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:18:11 trial068 ceph-mon[38805]: pgmap v6399: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:18:11.937 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11085 8 365281 365273 0.257418 0 - 0.0299513 2026-02-23T16:18:11.971 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:18:12.088 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:18:12.186 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:18:12.184+0000 7f69c75c7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:18:12.186 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:18:12.184+0000 7f69c75c7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:18:12.295 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:18:12.295 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":375,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:17:00.462137+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:16:14.421864+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:14:13.240512+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:14:58.594223+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:18:12.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:18:12 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1913227009' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:18:12.937 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11086 8 365281 365273 0.257395 0 - 0.0299513 2026-02-23T16:18:13.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:18:13 trial068 ceph-mon[38805]: pgmap v6400: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:18:13.937 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11087 8 365281 365273 0.257372 0 - 0.0299513 2026-02-23T16:18:14.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 16:18:13 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:16:18:13] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T16:18:14.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 16:18:13 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T16:18:13.974Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T16:18:14.938 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11088 8 365281 365273 0.257348 0 - 0.0299513 2026-02-23T16:18:15.334 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:18:15.450 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:18:15.547 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:18:15.545+0000 7f42e38ca640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:18:15.547 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:18:15.545+0000 7f42e38ca640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:18:15.656 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:18:15.656 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":375,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:17:00.462137+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:16:14.421864+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:14:13.240512+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:14:58.594223+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:18:15.938 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11089 8 365281 365273 0.257325 0 - 0.0299513 2026-02-23T16:18:16.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:18:16 trial068 ceph-mon[38805]: pgmap v6401: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:18:16.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:18:16 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1277482615' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:18:16.938 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11090 8 365281 365273 0.257302 0 - 0.0299513 2026-02-23T16:18:17.938 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11091 8 365281 365273 0.257279 0 - 0.0299513 2026-02-23T16:18:18.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:18:18 trial068 ceph-mon[38805]: pgmap v6402: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:18:18.694 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:18:18.812 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:18:18.911 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:18:18.909+0000 7f03233b7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:18:18.911 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:18:18.909+0000 7f03233b7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:18:18.938 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11092 8 365281 365273 0.257256 0 - 0.0299513 2026-02-23T16:18:19.020 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:18:19.020 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":375,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:17:00.462137+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:16:14.421864+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:14:13.240512+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:14:58.594223+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:18:19.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:18:19 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/647685350' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:18:19.938 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11093 8 365281 365273 0.257232 0 - 0.0299513 2026-02-23T16:18:20.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:18:20 trial068 ceph-mon[38805]: pgmap v6403: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:18:20.938 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11094 8 365281 365273 0.257209 0 - 0.0299513 2026-02-23T16:18:21.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:18:21 trial068 ceph-mon[38805]: pgmap v6404: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:18:21.938 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11095 8 365281 365273 0.257186 0 - 0.0299513 2026-02-23T16:18:22.058 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:18:22.175 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:18:22.279 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:18:22.277+0000 7f6988b7b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:18:22.280 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:18:22.278+0000 7f6988b7b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:18:22.390 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:18:22.390 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":375,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:17:00.462137+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:16:14.421864+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:14:13.240512+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:14:58.594223+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:18:22.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:18:22 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1325840477' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:18:22.938 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11096 8 365281 365273 0.257163 0 - 0.0299513 2026-02-23T16:18:23.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:18:23 trial068 ceph-mon[38805]: pgmap v6405: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:18:23.938 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11097 8 365281 365273 0.25714 0 - 0.0299513 2026-02-23T16:18:24.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 16:18:23 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:16:18:23] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T16:18:24.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 16:18:23 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T16:18:23.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T16:18:24.938 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11098 8 365281 365273 0.257117 0 - 0.0299513 2026-02-23T16:18:25.427 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:18:25.544 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:18:25.640 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:18:25.638+0000 7f6dc3426640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:18:25.640 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:18:25.638+0000 7f6dc3426640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:18:25.750 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:18:25.750 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":375,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:17:00.462137+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:16:14.421864+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:14:13.240512+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:14:58.594223+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:18:25.938 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11099 8 365281 365273 0.257093 0 - 0.0299513 2026-02-23T16:18:26.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:18:26 trial068 ceph-mon[38805]: pgmap v6406: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:18:26.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:18:26 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/160191443' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:18:26.938 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T16:18:26.938332+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T16:18:26.939 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T16:18:26.939 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11100 8 365281 365273 0.25707 0 - 0.0299513 2026-02-23T16:18:27.939 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11101 8 365281 365273 0.257047 0 - 0.0299513 2026-02-23T16:18:28.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:18:28 trial068 ceph-mon[38805]: 5.a scrub starts 2026-02-23T16:18:28.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:18:28 trial068 ceph-mon[38805]: 5.a scrub ok 2026-02-23T16:18:28.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:18:28 trial068 ceph-mon[38805]: pgmap v6407: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:18:28.787 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:18:28.904 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:18:28.939 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11102 8 365281 365273 0.257024 0 - 0.0299513 2026-02-23T16:18:29.004 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:18:29.002+0000 7f65fa3d0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:18:29.004 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:18:29.002+0000 7f65fa3d0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:18:29.113 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:18:29.113 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":375,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:17:00.462137+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:16:14.421864+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:14:13.240512+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:14:58.594223+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:18:29.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:18:29 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2896490761' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:18:29.939 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11103 8 365281 365273 0.257001 0 - 0.0299513 2026-02-23T16:18:30.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:18:30 trial068 ceph-mon[38805]: pgmap v6408: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:18:30.939 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11104 8 365281 365273 0.256978 0 - 0.0299513 2026-02-23T16:18:31.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:18:31 trial068 ceph-mon[38805]: pgmap v6409: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:18:31.939 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11105 8 365281 365273 0.256954 0 - 0.0299513 2026-02-23T16:18:32.151 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:18:32.267 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:18:32.376 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:18:32.374+0000 7fca02b64640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:18:32.376 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:18:32.375+0000 7fca02b64640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:18:32.487 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:18:32.487 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":375,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:17:00.462137+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:16:14.421864+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:14:13.240512+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:14:58.594223+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:18:32.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:18:32 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/475809361' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:18:32.939 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11106 8 365281 365273 0.256931 0 - 0.0299513 2026-02-23T16:18:33.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:18:33 trial068 ceph-mon[38805]: pgmap v6410: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:18:33.939 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11107 8 365281 365273 0.256908 0 - 0.0299513 2026-02-23T16:18:34.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 16:18:33 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:16:18:33] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T16:18:34.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 16:18:33 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T16:18:33.974Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T16:18:34.939 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11108 8 365281 365273 0.256885 0 - 0.0299513 2026-02-23T16:18:35.525 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:18:35.641 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:18:35.740 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:18:35.738+0000 7f1cf7978640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:18:35.740 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:18:35.738+0000 7f1cf7978640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:18:35.848 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:18:35.849 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":375,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:17:00.462137+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:16:14.421864+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:14:13.240512+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:14:58.594223+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:18:35.940 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11109 8 365281 365273 0.256862 0 - 0.0299513 2026-02-23T16:18:36.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:18:36 trial068 ceph-mon[38805]: pgmap v6411: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:18:36.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:18:36 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1541487681' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:18:36.940 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11110 8 365281 365273 0.256839 0 - 0.0299513 2026-02-23T16:18:37.940 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11111 8 365281 365273 0.256816 0 - 0.0299513 2026-02-23T16:18:38.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:18:38 trial068 ceph-mon[38805]: 5.8 scrub starts 2026-02-23T16:18:38.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:18:38 trial068 ceph-mon[38805]: 5.8 scrub ok 2026-02-23T16:18:38.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:18:38 trial068 ceph-mon[38805]: pgmap v6412: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:18:38.886 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:18:38.940 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11112 8 365281 365273 0.256793 0 - 0.0299513 2026-02-23T16:18:39.004 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:18:39.105 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:18:39.103+0000 7f506e3f0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:18:39.105 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:18:39.104+0000 7f506e3f0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:18:39.215 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:18:39.215 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":375,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:17:00.462137+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:16:14.421864+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:14:13.240512+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:14:58.594223+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:18:39.591 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:18:39 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2026-02-23T16:18:39.940 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11113 8 365281 365273 0.256769 0 - 0.0299513 2026-02-23T16:18:40.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:18:40 trial068 ceph-mon[38805]: pgmap v6413: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:18:40.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:18:40 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/922313020' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:18:40.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:18:40 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T16:18:40.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:18:40 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T16:18:40.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:18:40 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T16:18:40.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:18:40 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T16:18:40.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:18:40 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-23T16:18:40.595 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:18:40 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2026-02-23T16:18:40.595 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:18:40 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T16:18:40.595 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:18:40 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2026-02-23T16:18:40.940 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11114 8 365281 365273 0.256746 0 - 0.0299513 2026-02-23T16:18:41.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:18:41 trial068 ceph-mon[38805]: pgmap v6414: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:18:41.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:18:41 trial068 ceph-mon[38805]: pgmap v6415: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:18:41.940 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11115 8 365281 365273 0.256723 0 - 0.0299513 2026-02-23T16:18:42.254 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:18:42.372 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:18:42.474 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:18:42.472+0000 7fa1211d5640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:18:42.474 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:18:42.472+0000 7fa1211d5640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:18:42.582 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:18:42.583 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":375,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:17:00.462137+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:16:14.421864+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:14:13.240512+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:14:58.594223+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:18:42.940 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11116 8 365281 365273 0.2567 0 - 0.0299513 2026-02-23T16:18:43.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:18:43 trial068 ceph-mon[38805]: pgmap v6416: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:18:43.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:18:43 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1640795709' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:18:43.940 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11117 8 365281 365273 0.256677 0 - 0.0299513 2026-02-23T16:18:44.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 16:18:43 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:16:18:43] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T16:18:44.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 16:18:43 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T16:18:43.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T16:18:44.940 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11118 8 365281 365273 0.256654 0 - 0.0299513 2026-02-23T16:18:45.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:18:45 trial068 ceph-mon[38805]: pgmap v6417: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:18:45.619 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:18:45.737 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:18:45.843 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:18:45.841+0000 7f1e4caf9640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:18:45.843 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:18:45.841+0000 7f1e4caf9640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:18:45.940 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11119 8 365281 365273 0.256631 0 - 0.0299513 2026-02-23T16:18:45.953 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:18:45.953 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":375,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:17:00.462137+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:16:14.421864+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:14:13.240512+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:14:58.594223+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:18:46.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:18:46 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2588658921' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:18:46.940 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T16:18:46.940183+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T16:18:46.941 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T16:18:46.941 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11120 8 365281 365273 0.256608 0 - 0.0299513 2026-02-23T16:18:47.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:18:47 trial068 ceph-mon[38805]: pgmap v6418: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:18:47.940 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11121 8 365281 365273 0.256585 0 - 0.0299513 2026-02-23T16:18:48.940 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11122 8 365281 365273 0.256562 0 - 0.0299513 2026-02-23T16:18:48.992 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:18:49.110 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:18:49.211 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:18:49.209+0000 7f8895363640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:18:49.211 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:18:49.209+0000 7f8895363640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:18:49.319 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:18:49.319 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":375,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:17:00.462137+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:16:14.421864+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:14:13.240512+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:14:58.594223+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:18:49.320 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:18:49 trial068 ceph-mon[38805]: pgmap v6419: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:18:49.320 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:18:49 trial068 ceph-mon[38805]: 5.7 scrub starts 2026-02-23T16:18:49.941 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11123 8 365281 365273 0.256539 0 - 0.0299513 2026-02-23T16:18:50.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:18:50 trial068 ceph-mon[38805]: 5.7 scrub ok 2026-02-23T16:18:50.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:18:50 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2217352608' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:18:50.941 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11124 8 365281 365273 0.256516 0 - 0.0299513 2026-02-23T16:18:51.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:18:51 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T16:18:51.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:18:51 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T16:18:51.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:18:51 trial068 ceph-mon[38805]: pgmap v6420: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:18:51.941 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11125 8 365281 365273 0.256493 0 - 0.0299513 2026-02-23T16:18:52.356 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:18:52.476 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:18:52.577 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:18:52.575+0000 7f0cbef43640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:18:52.577 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:18:52.575+0000 7f0cbef43640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:18:52.685 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:18:52.686 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":375,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:17:00.462137+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:16:14.421864+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:14:13.240512+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:14:58.594223+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:18:52.941 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11126 8 365281 365273 0.256469 0 - 0.0299513 2026-02-23T16:18:53.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:18:53 trial068 ceph-mon[38805]: 5.e scrub starts 2026-02-23T16:18:53.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:18:53 trial068 ceph-mon[38805]: 5.e scrub ok 2026-02-23T16:18:53.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:18:53 trial068 ceph-mon[38805]: pgmap v6421: 17 pgs: 1 active+clean+scrubbing, 1 active+recovery_unfound+undersized+degraded, 4 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:18:53.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:18:53 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/921120381' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:18:53.941 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11127 8 365281 365273 0.256446 0 - 0.0299513 2026-02-23T16:18:54.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 16:18:53 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:16:18:53] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T16:18:54.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 16:18:53 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T16:18:53.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T16:18:54.941 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11128 8 365281 365273 0.256423 0 - 0.0299513 2026-02-23T16:18:55.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:18:55 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T16:18:55.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:18:55 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T16:18:55.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:18:55 trial068 ceph-mon[38805]: pgmap v6422: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 1 active+clean+scrubbing, 4 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:18:55.723 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:18:55.839 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:18:55.941 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:18:55.939+0000 7f06702f2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:18:55.941 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:18:55.939+0000 7f06702f2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:18:55.942 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11129 8 365281 365273 0.2564 0 - 0.0299513 2026-02-23T16:18:56.050 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:18:56.050 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":375,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:17:00.462137+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:16:14.421864+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:14:13.240512+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:14:58.594223+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:18:56.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:18:56 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2574641633' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:18:56.941 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11130 8 365281 365273 0.256377 0 - 0.0299513 2026-02-23T16:18:57.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:18:57 trial068 ceph-mon[38805]: pgmap v6423: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 1 active+clean+scrubbing, 4 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:18:57.941 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11131 8 365281 365273 0.256354 0 - 0.0299513 2026-02-23T16:18:58.941 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11132 8 365281 365273 0.256331 0 - 0.0299513 2026-02-23T16:18:59.087 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:18:59.206 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:18:59.314 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:18:59.312+0000 7fbd3e7b9640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:18:59.314 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:18:59.312+0000 7fbd3e7b9640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:18:59.425 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:18:59.425 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":375,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:17:00.462137+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:16:14.421864+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:14:13.240512+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:14:58.594223+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:18:59.425 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:18:59 trial068 ceph-mon[38805]: pgmap v6424: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:18:59.941 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11133 8 365281 365273 0.256308 0 - 0.0299513 2026-02-23T16:19:00.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:19:00 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3118526599' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:19:00.941 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11134 8 365281 365273 0.256285 0 - 0.0299513 2026-02-23T16:19:01.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:19:01 trial068 ceph-mon[38805]: pgmap v6425: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:19:01.941 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11135 8 365281 365273 0.256262 0 - 0.0299513 2026-02-23T16:19:02.463 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:19:02.580 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:19:02.674 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:19:02.672+0000 7fe53738e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:19:02.674 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:19:02.672+0000 7fe53738e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:19:02.785 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:19:02.785 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":375,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:17:00.462137+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:16:14.421864+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:14:13.240512+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:14:58.594223+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:19:02.942 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11136 8 365281 365273 0.256239 0 - 0.0299513 2026-02-23T16:19:03.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:19:03 trial068 ceph-mon[38805]: pgmap v6426: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:19:03.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:19:03 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2923849115' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:19:03.941 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11137 8 365281 365273 0.256216 0 - 0.0299513 2026-02-23T16:19:04.266 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 16:19:03 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T16:19:03.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T16:19:04.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 16:19:03 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:16:19:03] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T16:19:04.942 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11138 8 365281 365273 0.256193 0 - 0.0299513 2026-02-23T16:19:05.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:19:05 trial068 ceph-mon[38805]: pgmap v6427: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:19:05.823 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:19:05.941 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:19:05.942 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11139 8 365281 365273 0.25617 0 - 0.0299513 2026-02-23T16:19:06.041 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:19:06.039+0000 7feff4bac640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:19:06.041 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:19:06.039+0000 7feff4bac640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:19:06.150 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:19:06.150 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":375,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:17:00.462137+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:16:14.421864+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:14:13.240512+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:14:58.594223+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:19:06.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:19:06 trial068 ceph-mon[38805]: pgmap v6428: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:19:06.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:19:06 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2526134748' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:19:06.942 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T16:19:06.941596+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T16:19:06.942 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T16:19:06.942 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11140 8 365281 365273 0.256147 0 - 0.0299513 2026-02-23T16:19:07.942 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11141 8 365281 365273 0.256124 0 - 0.0299513 2026-02-23T16:19:08.942 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11142 8 365281 365273 0.256101 0 - 0.0299513 2026-02-23T16:19:09.189 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:19:09.213 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:19:08 trial068 ceph-mon[38805]: pgmap v6429: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:19:09.307 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:19:09.407 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:19:09.405+0000 7f8b5c423640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:19:09.407 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:19:09.405+0000 7f8b5c423640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:19:09.516 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:19:09.516 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":375,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:17:00.462137+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:16:14.421864+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:14:13.240512+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:14:58.594223+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:19:09.942 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11143 8 365281 365273 0.256078 0 - 0.0299513 2026-02-23T16:19:10.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:19:09 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1226443126' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:19:10.942 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11144 8 365281 365273 0.256055 0 - 0.0299513 2026-02-23T16:19:11.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:19:10 trial068 ceph-mon[38805]: pgmap v6430: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:19:11.942 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11145 8 365281 365273 0.256032 0 - 0.0299513 2026-02-23T16:19:12.552 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:19:12.671 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:19:12.773 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:19:12.771+0000 7feb5611d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:19:12.773 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:19:12.771+0000 7feb5611d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:19:12.882 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:19:12.882 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":375,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:17:00.462137+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:16:14.421864+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:14:13.240512+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:14:58.594223+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:19:12.942 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11146 8 365281 365273 0.256009 0 - 0.0299513 2026-02-23T16:19:13.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:19:12 trial068 ceph-mon[38805]: pgmap v6431: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:19:13.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:19:12 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1044552002' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:19:13.942 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11147 8 365281 365273 0.255986 0 - 0.0299513 2026-02-23T16:19:14.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 16:19:13 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:16:19:13] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T16:19:14.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 16:19:13 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T16:19:13.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T16:19:14.942 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11148 8 365281 365273 0.255963 0 - 0.0299513 2026-02-23T16:19:15.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:19:14 trial068 ceph-mon[38805]: pgmap v6432: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:19:15.921 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:19:15.942 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11149 8 365281 365273 0.25594 0 - 0.0299513 2026-02-23T16:19:16.039 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:19:16.137 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:19:16.135+0000 7fec94d7a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:19:16.137 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:19:16.135+0000 7fec94d7a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:19:16.246 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:19:16.246 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":375,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:17:00.462137+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:16:14.421864+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:14:13.240512+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:14:58.594223+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:19:16.247 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:19:15 trial068 ceph-mon[38805]: 5.5 scrub starts 2026-02-23T16:19:16.247 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:19:15 trial068 ceph-mon[38805]: 5.5 scrub ok 2026-02-23T16:19:16.942 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11150 8 365281 365273 0.255917 0 - 0.0299513 2026-02-23T16:19:17.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:19:16 trial068 ceph-mon[38805]: pgmap v6433: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:19:17.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:19:16 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/615022449' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:19:17.943 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11151 8 365281 365273 0.255894 0 - 0.0299513 2026-02-23T16:19:18.943 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11152 8 365281 365273 0.255872 0 - 0.0299513 2026-02-23T16:19:19.284 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:19:19.307 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:19:18 trial068 ceph-mon[38805]: pgmap v6434: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:19:19.400 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:19:19.496 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:19:19.495+0000 7fb611a0a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:19:19.497 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:19:19.495+0000 7fb611a0a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:19:19.605 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:19:19.605 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":375,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:17:00.462137+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:16:14.421864+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:14:13.240512+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:14:58.594223+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:19:19.943 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11153 8 365281 365273 0.255849 0 - 0.0299513 2026-02-23T16:19:20.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:19:19 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1183966621' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:19:20.943 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11154 8 365281 365273 0.255826 0 - 0.0299513 2026-02-23T16:19:21.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:19:20 trial068 ceph-mon[38805]: pgmap v6435: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:19:21.943 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11155 8 365281 365273 0.255803 0 - 0.0299513 2026-02-23T16:19:22.642 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:19:22.762 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:19:22.859 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:19:22.858+0000 7f635e0b1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:19:22.860 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:19:22.858+0000 7f635e0b1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:19:22.943 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11156 8 365281 365273 0.25578 0 - 0.0299513 2026-02-23T16:19:22.969 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:19:22.969 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":375,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:17:00.462137+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:16:14.421864+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:14:13.240512+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:14:58.594223+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:19:23.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:19:22 trial068 ceph-mon[38805]: pgmap v6436: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:19:23.943 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11157 8 365281 365273 0.255757 0 - 0.0299513 2026-02-23T16:19:24.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:19:23 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/689731966' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:19:24.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 16:19:23 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:16:19:23] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T16:19:24.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 16:19:23 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T16:19:23.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T16:19:24.943 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11158 8 365281 365273 0.255734 0 - 0.0299513 2026-02-23T16:19:25.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:19:24 trial068 ceph-mon[38805]: pgmap v6437: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:19:25.943 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11159 8 365281 365273 0.255711 0 - 0.0299513 2026-02-23T16:19:26.007 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:19:26.123 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:19:26.219 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:19:26.218+0000 7f4af9725640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:19:26.220 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:19:26.218+0000 7f4af9725640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:19:26.329 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:19:26.330 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":375,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:17:00.462137+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:16:14.421864+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:14:13.240512+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:14:58.594223+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:19:26.943 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T16:19:26.943015+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T16:19:26.943 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T16:19:26.944 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11160 8 365281 365273 0.255688 0 - 0.0299513 2026-02-23T16:19:27.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:19:26 trial068 ceph-mon[38805]: pgmap v6438: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:19:27.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:19:26 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2796049289' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:19:27.943 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11161 8 365281 365273 0.255665 0 - 0.0299513 2026-02-23T16:19:28.943 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11162 8 365281 365273 0.255642 0 - 0.0299513 2026-02-23T16:19:29.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:19:28 trial068 ceph-mon[38805]: pgmap v6439: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:19:29.367 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:19:29.484 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:19:29.584 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:19:29.582+0000 7f047c87b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:19:29.584 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:19:29.582+0000 7f047c87b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:19:29.694 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:19:29.694 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":375,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:17:00.462137+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:16:14.421864+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:14:13.240512+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:14:58.594223+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:19:29.943 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11163 8 365281 365273 0.255619 0 - 0.0299513 2026-02-23T16:19:30.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:19:29 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2139868908' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:19:30.944 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11164 8 365281 365273 0.255596 0 - 0.0299513 2026-02-23T16:19:31.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:19:30 trial068 ceph-mon[38805]: pgmap v6440: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:19:31.944 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11165 8 365281 365273 0.255574 0 - 0.0299513 2026-02-23T16:19:32.731 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:19:32.847 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:19:32.944 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11166 8 365281 365273 0.255551 0 - 0.0299513 2026-02-23T16:19:32.949 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:19:32.947+0000 7f3298a80640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:19:32.949 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:19:32.948+0000 7f3298a80640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:19:33.060 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:19:33.060 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":375,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:17:00.462137+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:16:14.421864+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:14:13.240512+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:14:58.594223+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:19:33.060 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:19:32 trial068 ceph-mon[38805]: pgmap v6441: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:19:33.944 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11167 8 365281 365273 0.255528 0 - 0.0299513 2026-02-23T16:19:34.265 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 16:19:33 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T16:19:33.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T16:19:34.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:19:33 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1035269827' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:19:34.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 16:19:33 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:16:19:33] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T16:19:34.944 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11168 8 365281 365273 0.255505 0 - 0.0299513 2026-02-23T16:19:35.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:19:34 trial068 ceph-mon[38805]: pgmap v6442: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:19:35.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:19:34 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T16:19:35.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:19:34 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T16:19:35.944 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11169 8 365281 365273 0.255482 0 - 0.0299513 2026-02-23T16:19:36.097 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:19:36.215 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:19:36.317 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:19:36.315+0000 7f190ec2c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:19:36.317 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:19:36.315+0000 7f190ec2c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:19:36.426 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:19:36.427 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":376,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:19:35.967413+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:16:14.421864+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:19:34.245890+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:14:58.594223+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:19:36.944 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11170 8 365281 365273 0.255459 0 - 0.0299513 2026-02-23T16:19:37.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:19:36 trial068 ceph-mon[38805]: pgmap v6443: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:19:37.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:19:36 trial068 ceph-mon[38805]: osdmap e376: 4 total, 3 up, 4 in 2026-02-23T16:19:37.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:19:36 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2458366432' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:19:37.944 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11171 8 365281 365273 0.255436 0 - 0.0299513 2026-02-23T16:19:38.944 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11172 8 365281 365273 0.255413 0 - 0.0299513 2026-02-23T16:19:39.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:19:38 trial068 ceph-mon[38805]: 5.a scrub starts 2026-02-23T16:19:39.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:19:38 trial068 ceph-mon[38805]: 5.a scrub ok 2026-02-23T16:19:39.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:19:38 trial068 ceph-mon[38805]: pgmap v6445: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:19:39.464 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:19:39.583 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:19:39.687 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:19:39.685+0000 7f6753582640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:19:39.687 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:19:39.685+0000 7f6753582640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:19:39.797 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:19:39.797 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":376,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:19:35.967413+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:16:14.421864+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:19:34.245890+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:14:58.594223+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:19:39.944 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11173 8 365281 365273 0.255391 0 - 0.0299513 2026-02-23T16:19:40.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:19:39 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1799114570' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:19:40.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:19:39 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2026-02-23T16:19:40.944 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11174 8 365281 365273 0.255368 0 - 0.0299513 2026-02-23T16:19:41.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:19:40 trial068 ceph-mon[38805]: pgmap v6446: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:19:41.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:19:40 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-23T16:19:41.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:19:40 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2026-02-23T16:19:41.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:19:40 trial068 ceph-mon[38805]: pgmap v6447: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:19:41.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:19:40 trial068 ceph-mon[38805]: pgmap v6448: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:19:41.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:19:40 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T16:19:41.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:19:40 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2026-02-23T16:19:41.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:19:40 trial068 ceph-mon[38805]: osdmap e377: 4 total, 3 up, 4 in 2026-02-23T16:19:41.944 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11175 8 365281 365273 0.255345 0 - 0.0299513 2026-02-23T16:19:42.835 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:19:42.944 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11176 8 365281 365273 0.255322 0 - 0.0299513 2026-02-23T16:19:42.953 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:19:43.053 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:19:43.051+0000 7ff9b6fc2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:19:43.053 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:19:43.051+0000 7ff9b6fc2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:19:43.163 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:19:43.163 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":377,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:19:40.483485+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:16:14.421864+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:19:34.245890+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:14:58.594223+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:19:43.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:19:43 trial068 ceph-mon[38805]: pgmap v6450: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:19:43.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:19:43 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/4138309806' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:19:43.944 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11177 8 365281 365273 0.255299 0 - 0.0299513 2026-02-23T16:19:44.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 16:19:43 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:16:19:43] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T16:19:44.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 16:19:43 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T16:19:43.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T16:19:44.945 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11178 8 365281 365273 0.255276 0 - 0.0299513 2026-02-23T16:19:45.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:19:45 trial068 ceph-mon[38805]: pgmap v6451: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:19:45.945 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11179 8 365281 365273 0.255254 0 - 0.0299513 2026-02-23T16:19:46.201 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:19:46.318 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:19:46.430 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:19:46.428+0000 7f6b7c468640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:19:46.430 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:19:46.428+0000 7f6b7c468640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:19:46.539 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:19:46.539 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":377,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:19:40.483485+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:16:14.421864+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:19:34.245890+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:14:58.594223+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:19:46.539 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:19:46 trial068 ceph-mon[38805]: 5.8 scrub starts 2026-02-23T16:19:46.540 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:19:46 trial068 ceph-mon[38805]: 5.8 scrub ok 2026-02-23T16:19:46.945 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T16:19:46.944579+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T16:19:46.945 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T16:19:46.945 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11180 8 365281 365273 0.255231 0 - 0.0299513 2026-02-23T16:19:47.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:19:47 trial068 ceph-mon[38805]: pgmap v6452: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:19:47.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:19:47 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/777767993' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:19:47.945 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11181 8 365281 365273 0.255208 0 - 0.0299513 2026-02-23T16:19:48.945 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11182 8 365281 365273 0.255185 0 - 0.0299513 2026-02-23T16:19:49.576 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:19:49.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:19:49 trial068 ceph-mon[38805]: pgmap v6453: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:19:49.695 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:19:49.792 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:19:49.790+0000 7f6dc4a1b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:19:49.792 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:19:49.791+0000 7f6dc4a1b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:19:49.903 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:19:49.904 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":377,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:19:40.483485+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:16:14.421864+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:19:34.245890+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:14:58.594223+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:19:49.945 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11183 8 365281 365273 0.255162 0 - 0.0299513 2026-02-23T16:19:50.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:19:50 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3135043342' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:19:50.945 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11184 8 365281 365273 0.255139 0 - 0.0299513 2026-02-23T16:19:51.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:19:51 trial068 ceph-mon[38805]: pgmap v6454: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:19:51.945 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11185 8 365281 365273 0.255117 0 - 0.0299513 2026-02-23T16:19:52.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:19:52 trial068 ceph-mon[38805]: 5.7 scrub starts 2026-02-23T16:19:52.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:19:52 trial068 ceph-mon[38805]: 5.7 scrub ok 2026-02-23T16:19:52.941 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:19:52.945 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11186 8 365281 365273 0.255094 0 - 0.0299513 2026-02-23T16:19:53.059 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:19:53.158 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:19:53.156+0000 7f514a1bf640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:19:53.158 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:19:53.156+0000 7f514a1bf640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:19:53.268 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:19:53.268 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":377,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:19:40.483485+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:16:14.421864+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:19:34.245890+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:14:58.594223+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:19:53.268 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:19:53 trial068 ceph-mon[38805]: pgmap v6455: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:19:53.945 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11187 8 365281 365273 0.255071 0 - 0.0299513 2026-02-23T16:19:54.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:19:54 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1037100165' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:19:54.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 16:19:53 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:16:19:53] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T16:19:54.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 16:19:53 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T16:19:53.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T16:19:54.945 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11188 8 365281 365273 0.255048 0 - 0.0299513 2026-02-23T16:19:55.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:19:55 trial068 ceph-mon[38805]: pgmap v6456: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:19:55.945 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11189 8 365281 365273 0.255025 0 - 0.0299513 2026-02-23T16:19:56.307 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:19:56.426 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:19:56.470 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:19:56 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T16:19:56.470 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:19:56 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T16:19:56.522 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:19:56.520+0000 7fd004cd1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:19:56.522 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:19:56.520+0000 7fd004cd1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:19:56.633 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:19:56.633 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":377,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:19:40.483485+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:16:14.421864+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:19:34.245890+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:14:58.594223+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:19:56.945 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11190 8 365281 365273 0.255003 0 - 0.0299513 2026-02-23T16:19:57.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:19:57 trial068 ceph-mon[38805]: pgmap v6457: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:19:57.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:19:57 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/163380446' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:19:57.945 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11191 8 365281 365273 0.25498 0 - 0.0299513 2026-02-23T16:19:58.946 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11192 8 365281 365273 0.254957 0 - 0.0299513 2026-02-23T16:19:59.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:19:59 trial068 ceph-mon[38805]: pgmap v6458: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:19:59.670 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:19:59.787 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:19:59.889 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:19:59.887+0000 7f15529b3640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:19:59.889 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:19:59.887+0000 7f15529b3640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:19:59.946 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11193 8 365281 365273 0.254934 0 - 0.0299513 2026-02-23T16:19:59.997 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:19:59.998 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":377,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:19:40.483485+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:16:14.421864+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:19:34.245890+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:14:58.594223+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:20:00.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:20:00 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/59719334' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:20:00.946 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11194 8 365281 365273 0.254911 0 - 0.0299513 2026-02-23T16:20:01.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:20:01 trial068 ceph-mon[38805]: pgmap v6459: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:20:01.946 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11195 8 365281 365273 0.254889 0 - 0.0299513 2026-02-23T16:20:02.946 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11196 8 365281 365273 0.254866 0 - 0.0299513 2026-02-23T16:20:03.046 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:20:03.164 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:20:03.263 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:20:03.261+0000 7fd7ee3dd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:20:03.263 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:20:03.261+0000 7fd7ee3dd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:20:03.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:20:03 trial068 ceph-mon[38805]: pgmap v6460: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:20:03.375 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:20:03.376 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":377,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:19:40.483485+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:16:14.421864+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:19:34.245890+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:14:58.594223+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:20:03.946 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11197 8 365281 365273 0.254843 0 - 0.0299513 2026-02-23T16:20:04.222 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 16:20:03 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:16:20:03] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T16:20:04.265 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 16:20:03 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T16:20:03.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T16:20:04.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:20:04 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2252976747' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:20:04.946 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11198 8 365281 365273 0.25482 0 - 0.0299513 2026-02-23T16:20:05.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:20:05 trial068 ceph-mon[38805]: pgmap v6461: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:20:05.946 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11199 8 365281 365273 0.254798 0 - 0.0299513 2026-02-23T16:20:06.414 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:20:06.532 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:20:06.636 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:20:06.634+0000 7f36d39c9640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:20:06.636 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:20:06.634+0000 7f36d39c9640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:20:06.744 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:20:06.744 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":377,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:19:40.483485+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:16:14.421864+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:19:34.245890+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:14:58.594223+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:20:06.744 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:20:06 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T16:20:06.744 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:20:06 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T16:20:06.744 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:20:06 trial068 ceph-mon[38805]: pgmap v6462: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:20:06.946 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T16:20:06.946024+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T16:20:06.946 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T16:20:06.946 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11200 8 365281 365273 0.254775 0 - 0.0299513 2026-02-23T16:20:07.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:20:07 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/444484042' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:20:07.946 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11201 8 365281 365273 0.254752 0 - 0.0299513 2026-02-23T16:20:08.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:20:08 trial068 ceph-mon[38805]: pgmap v6463: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:20:08.946 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11202 8 365281 365273 0.254729 0 - 0.0299513 2026-02-23T16:20:09.794 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:20:09.914 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:20:09.946 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11203 8 365281 365273 0.254707 0 - 0.0299513 2026-02-23T16:20:10.013 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:20:10.011+0000 7f98d6611640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:20:10.013 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:20:10.011+0000 7f98d6611640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:20:10.121 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:20:10.122 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":377,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:19:40.483485+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:16:14.421864+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:19:34.245890+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:14:58.594223+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:20:10.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:20:10 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/190583875' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:20:10.946 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11204 8 365281 365273 0.254684 0 - 0.0299513 2026-02-23T16:20:11.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:20:11 trial068 ceph-mon[38805]: pgmap v6464: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:20:11.946 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11205 8 365281 365273 0.254661 0 - 0.0299513 2026-02-23T16:20:12.947 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11206 8 365281 365273 0.254639 0 - 0.0299513 2026-02-23T16:20:13.160 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:20:13.278 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:20:13.377 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:20:13.375+0000 7f9d99753640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:20:13.377 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:20:13.375+0000 7f9d99753640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:20:13.486 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:20:13.487 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":377,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:19:40.483485+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:16:14.421864+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:19:34.245890+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:14:58.594223+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:20:13.487 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:20:13 trial068 ceph-mon[38805]: pgmap v6465: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:20:13.947 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11207 8 365281 365273 0.254616 0 - 0.0299513 2026-02-23T16:20:14.204 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 16:20:13 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:16:20:13] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T16:20:14.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 16:20:13 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T16:20:13.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T16:20:14.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:20:14 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2326936204' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:20:14.947 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11208 8 365281 365273 0.254593 0 - 0.0299513 2026-02-23T16:20:15.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:20:15 trial068 ceph-mon[38805]: pgmap v6466: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:20:15.947 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11209 8 365281 365273 0.25457 0 - 0.0299513 2026-02-23T16:20:16.524 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:20:16.642 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:20:16.748 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:20:16.746+0000 7f07ec3bb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:20:16.749 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:20:16.747+0000 7f07ec3bb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:20:16.857 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:20:16.857 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":377,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:19:40.483485+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:16:14.421864+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:19:34.245890+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:14:58.594223+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:20:16.947 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11210 8 365281 365273 0.254548 0 - 0.0299513 2026-02-23T16:20:17.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:20:17 trial068 ceph-mon[38805]: pgmap v6467: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:20:17.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:20:17 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/801206556' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:20:17.947 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11211 8 365281 365273 0.254525 0 - 0.0299513 2026-02-23T16:20:18.947 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11212 8 365281 365273 0.254502 0 - 0.0299513 2026-02-23T16:20:19.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:20:19 trial068 ceph-mon[38805]: pgmap v6468: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:20:19.895 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:20:19.947 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11213 8 365281 365273 0.25448 0 - 0.0299513 2026-02-23T16:20:20.016 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:20:20.115 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:20:20.113+0000 7f34baf8b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:20:20.115 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:20:20.113+0000 7f34baf8b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:20:20.226 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:20:20.226 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":377,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:19:40.483485+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:16:14.421864+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:19:34.245890+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:14:58.594223+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:20:20.947 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11214 8 365281 365273 0.254457 0 - 0.0299513 2026-02-23T16:20:21.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:20:21 trial068 ceph-mon[38805]: pgmap v6469: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:20:21.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:20:21 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2439894640' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:20:21.947 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11215 8 365281 365273 0.254434 0 - 0.0299513 2026-02-23T16:20:22.948 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11216 8 365281 365273 0.254411 0 - 0.0299513 2026-02-23T16:20:23.264 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:20:23.383 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:20:23.480 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:20:23 trial068 ceph-mon[38805]: 5.e scrub starts 2026-02-23T16:20:23.481 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:20:23 trial068 ceph-mon[38805]: 5.e scrub ok 2026-02-23T16:20:23.481 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:20:23 trial068 ceph-mon[38805]: pgmap v6470: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:20:23.481 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:20:23.478+0000 7fb4de57a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:20:23.481 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:20:23.478+0000 7fb4de57a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:20:23.589 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:20:23.589 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":377,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:19:40.483485+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:16:14.421864+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:19:34.245890+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:14:58.594223+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:20:23.948 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11217 8 365281 365273 0.254389 0 - 0.0299513 2026-02-23T16:20:24.212 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 16:20:23 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:16:20:23] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T16:20:24.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 16:20:23 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T16:20:23.974Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T16:20:24.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:20:24 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1532834025' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:20:24.948 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11218 8 365281 365273 0.254366 0 - 0.0299513 2026-02-23T16:20:25.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:20:25 trial068 ceph-mon[38805]: pgmap v6471: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:20:25.948 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11219 8 365281 365273 0.254343 0 - 0.0299513 2026-02-23T16:20:26.627 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:20:26.744 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:20:26.841 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:20:26.839+0000 7f8b7462c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:20:26.841 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:20:26.839+0000 7f8b7462c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:20:26.948 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T16:20:26.947863+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T16:20:26.948 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T16:20:26.948 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11220 8 365281 365273 0.254321 0 - 0.0299513 2026-02-23T16:20:26.950 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:20:26.951 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":377,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:19:40.483485+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:16:14.421864+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:19:34.245890+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:14:58.594223+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:20:27.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:20:27 trial068 ceph-mon[38805]: pgmap v6472: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:20:27.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:20:27 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/503344005' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:20:27.948 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11221 8 365281 365273 0.254298 0 - 0.0299513 2026-02-23T16:20:28.948 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11222 8 365281 365273 0.254275 0 - 0.0299513 2026-02-23T16:20:29.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:20:29 trial068 ceph-mon[38805]: pgmap v6473: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:20:29.948 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11223 8 365281 365273 0.254253 0 - 0.0299513 2026-02-23T16:20:29.989 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:20:30.107 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:20:30.203 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:20:30.201+0000 7f32a7802640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:20:30.203 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:20:30.201+0000 7f32a7802640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:20:30.312 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:20:30.312 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":377,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:19:40.483485+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:16:14.421864+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:19:34.245890+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:14:58.594223+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:20:30.948 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11224 8 365281 365273 0.25423 0 - 0.0299513 2026-02-23T16:20:31.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:20:31 trial068 ceph-mon[38805]: pgmap v6474: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:20:31.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:20:31 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1057514439' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:20:31.949 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11225 8 365281 365273 0.254207 0 - 0.0299513 2026-02-23T16:20:32.949 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11226 8 365281 365273 0.254185 0 - 0.0299513 2026-02-23T16:20:33.350 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:20:33.468 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:20:33.489 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:20:33 trial068 ceph-mon[38805]: pgmap v6475: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:20:33.572 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:20:33.570+0000 7f319b4ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:20:33.572 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:20:33.570+0000 7f319b4ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:20:33.682 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:20:33.682 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":377,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:19:40.483485+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:16:14.421864+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:19:34.245890+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:14:58.594223+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:20:33.949 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11227 8 365281 365273 0.254162 0 - 0.0299513 2026-02-23T16:20:34.219 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 16:20:33 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:16:20:33] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T16:20:34.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 16:20:33 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T16:20:33.972Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T16:20:34.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:20:34 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1994192722' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:20:34.948 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11228 8 365281 365273 0.25414 0 - 0.0299513 2026-02-23T16:20:35.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:20:35 trial068 ceph-mon[38805]: pgmap v6476: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:20:35.949 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11229 8 365281 365273 0.254117 0 - 0.0299513 2026-02-23T16:20:36.718 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:20:36.744 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:20:36 trial068 ceph-mon[38805]: pgmap v6477: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:20:36.838 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:20:36.935 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:20:36.934+0000 7f06edbbd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:20:36.935 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:20:36.934+0000 7f06edbbd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:20:36.949 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11230 8 365281 365273 0.254094 0 - 0.0299513 2026-02-23T16:20:37.043 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:20:37.044 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":377,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:19:40.483485+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:16:14.421864+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:19:34.245890+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:14:58.594223+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:20:37.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:20:37 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3504969406' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:20:37.949 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11231 8 365281 365273 0.254072 0 - 0.0299513 2026-02-23T16:20:38.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:20:38 trial068 ceph-mon[38805]: pgmap v6478: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:20:38.949 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11232 8 365281 365273 0.254049 0 - 0.0299513 2026-02-23T16:20:39.949 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11233 8 365281 365273 0.254026 0 - 0.0299513 2026-02-23T16:20:40.080 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:20:40.199 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:20:40.316 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:20:40.314+0000 7f26637bd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:20:40.317 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:20:40.315+0000 7f26637bd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:20:40.427 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:20:40 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2026-02-23T16:20:40.427 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:20:40.427 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":377,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:19:40.483485+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:16:14.421864+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:19:34.245890+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:14:58.594223+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:20:40.949 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11234 8 365281 365273 0.254004 0 - 0.0299513 2026-02-23T16:20:41.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:20:41 trial068 ceph-mon[38805]: pgmap v6479: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:20:41.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:20:41 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-23T16:20:41.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:20:41 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2026-02-23T16:20:41.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:20:41 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T16:20:41.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:20:41 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2026-02-23T16:20:41.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:20:41 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3705924833' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:20:41.949 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11235 8 365281 365273 0.253981 0 - 0.0299513 2026-02-23T16:20:42.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:20:42 trial068 ceph-mon[38805]: pgmap v6480: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:20:42.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:20:42 trial068 ceph-mon[38805]: pgmap v6481: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:20:42.949 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11236 8 365281 365273 0.253959 0 - 0.0299513 2026-02-23T16:20:43.466 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:20:43.490 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:20:43 trial068 ceph-mon[38805]: 5.5 scrub starts 2026-02-23T16:20:43.490 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:20:43 trial068 ceph-mon[38805]: 5.5 scrub ok 2026-02-23T16:20:43.583 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:20:43.687 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:20:43.686+0000 7f31c126b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:20:43.687 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:20:43.686+0000 7f31c126b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:20:43.795 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:20:43.795 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":377,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:19:40.483485+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:16:14.421864+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:19:34.245890+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:14:58.594223+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:20:43.949 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11237 8 365281 365273 0.253936 0 - 0.0299513 2026-02-23T16:20:44.093 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 16:20:43 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:16:20:43] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T16:20:44.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 16:20:43 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T16:20:43.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T16:20:44.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:20:44 trial068 ceph-mon[38805]: pgmap v6482: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:20:44.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:20:44 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/745038304' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:20:44.949 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11238 8 365281 365273 0.253913 0 - 0.0299513 2026-02-23T16:20:45.949 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11239 8 365281 365273 0.253891 0 - 0.0299513 2026-02-23T16:20:46.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:20:46 trial068 ceph-mon[38805]: pgmap v6483: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:20:46.832 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:20:46.950 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T16:20:46.949408+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T16:20:46.950 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T16:20:46.950 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11240 8 365281 365273 0.253868 0 - 0.0299513 2026-02-23T16:20:46.951 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:20:47.049 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:20:47.047+0000 7f00c1d87640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:20:47.049 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:20:47.047+0000 7f00c1d87640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:20:47.158 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:20:47.158 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":377,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:19:40.483485+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:16:14.421864+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:19:34.245890+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:14:58.594223+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:20:47.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:20:47 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1633364098' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:20:47.950 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11241 8 365281 365273 0.253846 0 - 0.0299513 2026-02-23T16:20:48.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:20:48 trial068 ceph-mon[38805]: pgmap v6484: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:20:48.950 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11242 8 365281 365273 0.253823 0 - 0.0299513 2026-02-23T16:20:49.950 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11243 8 365281 365273 0.253801 0 - 0.0299513 2026-02-23T16:20:50.197 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:20:50.313 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:20:50.416 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:20:50.414+0000 7f524df75640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:20:50.416 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:20:50.414+0000 7f524df75640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:20:50.528 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:20:50.528 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":377,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:19:40.483485+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:16:14.421864+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:19:34.245890+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:14:58.594223+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:20:50.528 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:20:50 trial068 ceph-mon[38805]: pgmap v6485: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:20:50.950 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11244 8 365281 365273 0.253778 0 - 0.0299513 2026-02-23T16:20:51.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:20:51 trial068 ceph-mon[38805]: 5.8 scrub starts 2026-02-23T16:20:51.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:20:51 trial068 ceph-mon[38805]: 5.8 scrub ok 2026-02-23T16:20:51.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:20:51 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/284060921' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:20:51.950 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11245 8 365281 365273 0.253755 0 - 0.0299513 2026-02-23T16:20:52.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:20:52 trial068 ceph-mon[38805]: pgmap v6486: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:20:52.950 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11246 8 365281 365273 0.253733 0 - 0.0299513 2026-02-23T16:20:53.567 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:20:53.683 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:20:53.787 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:20:53.785+0000 7f8cd3ca7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:20:53.787 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:20:53.785+0000 7f8cd3ca7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:20:53.896 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:20:53.896 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":378,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:20:53.248296+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:16:14.421864+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:19:34.245890+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:20:05.790461+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:20:53.950 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11247 8 365281 365273 0.25371 0 - 0.0299513 2026-02-23T16:20:54.251 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 16:20:53 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:16:20:53] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T16:20:54.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 16:20:53 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T16:20:53.972Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T16:20:54.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:20:54 trial068 ceph-mon[38805]: pgmap v6487: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:20:54.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:20:54 trial068 ceph-mon[38805]: osdmap e378: 4 total, 3 up, 4 in 2026-02-23T16:20:54.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:20:54 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3946106239' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:20:54.950 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11248 8 365281 365273 0.253688 0 - 0.0299513 2026-02-23T16:20:55.950 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11249 8 365281 365273 0.253665 0 - 0.0299513 2026-02-23T16:20:56.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:20:56 trial068 ceph-mon[38805]: pgmap v6489: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:20:56.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:20:56 trial068 ceph-mon[38805]: osdmap e379: 4 total, 3 up, 4 in 2026-02-23T16:20:56.933 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:20:56.950 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11250 8 365281 365273 0.253643 0 - 0.0299513 2026-02-23T16:20:57.051 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:20:57.150 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:20:57.148+0000 7f3b48591640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:20:57.150 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:20:57.148+0000 7f3b48591640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:20:57.259 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:20:57.260 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":379,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:20:55.492970+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:16:14.421864+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:19:34.245890+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:20:05.790461+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:20:57.950 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11251 8 365281 365273 0.25362 0 - 0.0299513 2026-02-23T16:20:58.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:20:58 trial068 ceph-mon[38805]: pgmap v6491: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:20:58.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:20:58 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/323394094' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:20:58.951 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11252 8 365281 365273 0.253598 0 - 0.0299513 2026-02-23T16:20:59.951 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11253 8 365281 365273 0.253575 0 - 0.0299513 2026-02-23T16:21:00.297 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:21:00.416 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:21:00.511 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:21:00.509+0000 7fde94a01640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:21:00.511 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:21:00 trial068 ceph-mon[38805]: pgmap v6492: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:21:00.512 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:21:00.510+0000 7fde94a01640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:21:00.622 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:21:00.622 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":379,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:20:55.492970+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:16:14.421864+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:19:34.245890+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:20:05.790461+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:21:00.951 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11254 8 365281 365273 0.253552 0 - 0.0299513 2026-02-23T16:21:01.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:21:01 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1333653760' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:21:01.951 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11255 8 365281 365273 0.25353 0 - 0.0299513 2026-02-23T16:21:02.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:21:02 trial068 ceph-mon[38805]: pgmap v6493: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:21:02.951 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11256 8 365281 365273 0.253507 0 - 0.0299513 2026-02-23T16:21:03.658 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:21:03.778 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:21:03.875 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:21:03.873+0000 7fa4cc84d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:21:03.876 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:21:03.874+0000 7fa4cc84d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:21:03.951 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11257 8 365281 365273 0.253485 0 - 0.0299513 2026-02-23T16:21:03.984 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:21:03.985 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":379,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:20:55.492970+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:16:14.421864+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:19:34.245890+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:20:05.790461+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:21:04.258 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 16:21:03 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:16:21:03] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T16:21:04.266 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 16:21:03 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T16:21:03.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T16:21:04.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:21:04 trial068 ceph-mon[38805]: pgmap v6494: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:21:04.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:21:04 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T16:21:04.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:21:04 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T16:21:04.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:21:04 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1709037243' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:21:04.951 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11258 8 365281 365273 0.253462 0 - 0.0299513 2026-02-23T16:21:05.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:21:05 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T16:21:05.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:21:05 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T16:21:05.951 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11259 8 365281 365273 0.25344 0 - 0.0299513 2026-02-23T16:21:06.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:21:06 trial068 ceph-mon[38805]: pgmap v6495: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:21:06.951 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T16:21:06.950950+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T16:21:06.951 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T16:21:06.952 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11260 8 365281 365273 0.253417 0 - 0.0299513 2026-02-23T16:21:07.038 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:21:07.156 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:21:07.252 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:21:07.250+0000 7f9d57908640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:21:07.252 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:21:07.250+0000 7f9d57908640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:21:07.362 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:21:07.362 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":379,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:20:55.492970+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:16:14.421864+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:19:34.245890+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:20:05.790461+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:21:07.951 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11261 8 365281 365273 0.253395 0 - 0.0299513 2026-02-23T16:21:08.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:21:08 trial068 ceph-mon[38805]: pgmap v6496: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:21:08.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:21:08 trial068 ceph-mon[38805]: 5.a scrub starts 2026-02-23T16:21:08.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:21:08 trial068 ceph-mon[38805]: 5.a scrub ok 2026-02-23T16:21:08.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:21:08 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/549130441' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:21:08.951 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11262 8 365281 365273 0.253372 0 - 0.0299513 2026-02-23T16:21:09.951 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11263 8 365281 365273 0.25335 0 - 0.0299513 2026-02-23T16:21:10.400 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:21:10.520 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:21:10.543 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:21:10 trial068 ceph-mon[38805]: pgmap v6497: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:21:10.617 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:21:10.616+0000 7f1606208640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:21:10.618 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:21:10.616+0000 7f1606208640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:21:10.728 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:21:10.728 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":379,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:20:55.492970+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:16:14.421864+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:19:34.245890+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:20:05.790461+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:21:10.951 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11264 8 365281 365273 0.253327 0 - 0.0299513 2026-02-23T16:21:11.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:21:11 trial068 ceph-mon[38805]: pgmap v6498: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:21:11.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:21:11 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2901873690' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:21:11.952 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11265 8 365281 365273 0.253305 0 - 0.0299513 2026-02-23T16:21:12.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:21:12 trial068 ceph-mon[38805]: 5.7 scrub starts 2026-02-23T16:21:12.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:21:12 trial068 ceph-mon[38805]: 5.7 scrub ok 2026-02-23T16:21:12.952 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11266 8 365281 365273 0.253282 0 - 0.0299513 2026-02-23T16:21:13.764 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:21:13.789 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:21:13 trial068 ceph-mon[38805]: pgmap v6499: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:21:13.882 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:21:13.952 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11267 8 365281 365273 0.25326 0 - 0.0299513 2026-02-23T16:21:13.982 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:21:13.980+0000 7f56b0338640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:21:13.982 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:21:13.980+0000 7f56b0338640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:21:14.090 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:21:14.090 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":379,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:20:55.492970+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:16:14.421864+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:19:34.245890+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:20:05.790461+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:21:14.091 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 16:21:13 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:16:21:13] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T16:21:14.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 16:21:13 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T16:21:13.972Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T16:21:14.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:21:14 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1701388420' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:21:14.952 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11268 8 365281 365273 0.253237 0 - 0.0299513 2026-02-23T16:21:15.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:21:15 trial068 ceph-mon[38805]: pgmap v6500: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:21:15.952 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11269 8 365281 365273 0.253215 0 - 0.0299513 2026-02-23T16:21:16.952 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11270 8 365281 365273 0.253192 0 - 0.0299513 2026-02-23T16:21:17.127 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:21:17.247 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:21:17.354 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:21:17.352+0000 7fe5972f6640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:21:17.354 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:21:17.352+0000 7fe5972f6640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:21:17.463 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:21:17.463 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":379,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:20:55.492970+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:16:14.421864+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:19:34.245890+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:20:05.790461+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:21:17.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:21:17 trial068 ceph-mon[38805]: pgmap v6501: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:21:17.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:21:17 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2149337191' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:21:17.952 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11271 8 365281 365273 0.25317 0 - 0.0299513 2026-02-23T16:21:18.952 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11272 8 365281 365273 0.253148 0 - 0.0299513 2026-02-23T16:21:19.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:21:19 trial068 ceph-mon[38805]: pgmap v6502: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:21:19.952 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11273 8 365281 365273 0.253125 0 - 0.0299513 2026-02-23T16:21:20.501 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:21:20.616 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:21:20.715 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:21:20.713+0000 7f19db04f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:21:20.716 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:21:20.714+0000 7f19db04f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:21:20.824 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:21:20.825 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":379,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:20:55.492970+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:16:14.421864+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:19:34.245890+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:20:05.790461+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:21:20.952 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11274 8 365281 365273 0.253103 0 - 0.0299513 2026-02-23T16:21:21.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:21:21 trial068 ceph-mon[38805]: pgmap v6503: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:21:21.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:21:21 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2703797736' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:21:21.952 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11275 8 365281 365273 0.25308 0 - 0.0299513 2026-02-23T16:21:22.952 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11276 8 365281 365273 0.253058 0 - 0.0299513 2026-02-23T16:21:23.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:21:23 trial068 ceph-mon[38805]: pgmap v6504: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:21:23.861 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:21:23.952 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11277 8 365281 365273 0.253035 0 - 0.0299513 2026-02-23T16:21:23.977 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:21:24.076 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:21:24.074+0000 7fed50a94640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:21:24.077 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:21:24.075+0000 7fed50a94640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:21:24.187 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:21:24.187 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":379,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:20:55.492970+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:16:14.421864+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:19:34.245890+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:20:05.790461+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:21:24.187 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 16:21:23 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:16:21:23] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T16:21:24.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 16:21:23 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T16:21:23.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T16:21:24.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:21:24 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1681784117' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:21:24.952 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11278 8 365281 365273 0.253013 0 - 0.0299513 2026-02-23T16:21:25.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:21:25 trial068 ceph-mon[38805]: pgmap v6505: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:21:25.952 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11279 8 365281 365273 0.25299 0 - 0.0299513 2026-02-23T16:21:26.952 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T16:21:26.952251+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T16:21:26.953 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T16:21:26.953 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11280 8 365281 365273 0.252968 0 - 0.0299513 2026-02-23T16:21:27.225 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:21:27.342 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:21:27.441 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:21:27.439+0000 7f6489c71640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:21:27.442 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:21:27.440+0000 7f6489c71640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:21:27.551 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:21:27.551 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":379,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:20:55.492970+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:16:14.421864+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:19:34.245890+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:20:05.790461+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:21:27.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:21:27 trial068 ceph-mon[38805]: pgmap v6506: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:21:27.953 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11281 8 365281 365273 0.252946 0 - 0.0299513 2026-02-23T16:21:28.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:21:28 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2958932048' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:21:28.953 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11282 8 365281 365273 0.252923 0 - 0.0299513 2026-02-23T16:21:29.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:21:29 trial068 ceph-mon[38805]: pgmap v6507: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:21:29.952 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11283 8 365281 365273 0.252901 0 - 0.0299513 2026-02-23T16:21:30.589 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:21:30.707 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:21:30.774 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:21:30 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T16:21:30.774 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:21:30 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T16:21:30.774 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:21:30 trial068 ceph-mon[38805]: 5.e scrub starts 2026-02-23T16:21:30.774 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:21:30 trial068 ceph-mon[38805]: 5.e scrub ok 2026-02-23T16:21:30.821 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:21:30.819+0000 7f7fa120b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:21:30.821 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:21:30.819+0000 7f7fa120b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:21:30.930 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:21:30.930 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":379,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:20:55.492970+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:16:14.421864+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:19:34.245890+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:20:05.790461+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:21:30.952 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11284 8 365281 365273 0.252878 0 - 0.0299513 2026-02-23T16:21:31.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:21:31 trial068 ceph-mon[38805]: pgmap v6508: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:21:31.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:21:31 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/4189078711' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:21:31.953 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11285 8 365281 365273 0.252856 0 - 0.0299513 2026-02-23T16:21:32.953 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11286 8 365281 365273 0.252834 0 - 0.0299513 2026-02-23T16:21:33.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:21:33 trial068 ceph-mon[38805]: pgmap v6509: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:21:33.953 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11287 8 365281 365273 0.252811 0 - 0.0299513 2026-02-23T16:21:33.966 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:21:34.084 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:21:34.184 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:21:34.182+0000 7f5b8e58f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:21:34.184 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:21:34.182+0000 7f5b8e58f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:21:34.266 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 16:21:33 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T16:21:33.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T16:21:34.296 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:21:34.297 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":379,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:20:55.492970+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:16:14.421864+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:19:34.245890+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:20:05.790461+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:21:34.297 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 16:21:33 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:16:21:33] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T16:21:34.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:21:34 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/4281265375' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:21:34.953 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11288 8 365281 365273 0.252789 0 - 0.0299513 2026-02-23T16:21:35.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:21:35 trial068 ceph-mon[38805]: pgmap v6510: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:21:35.953 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11289 8 365281 365273 0.252766 0 - 0.0299513 2026-02-23T16:21:36.953 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11290 8 365281 365273 0.252744 0 - 0.0299513 2026-02-23T16:21:37.334 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:21:37.450 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:21:37.549 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:21:37.547+0000 7f0d80a42640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:21:37.549 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:21:37.547+0000 7f0d80a42640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:21:37.662 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:21:37 trial068 ceph-mon[38805]: pgmap v6511: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:21:37.662 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:21:37.662 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":379,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:20:55.492970+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:16:14.421864+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:19:34.245890+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:20:05.790461+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:21:37.953 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11291 8 365281 365273 0.252722 0 - 0.0299513 2026-02-23T16:21:38.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:21:38 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/160297074' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:21:38.953 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11292 8 365281 365273 0.252699 0 - 0.0299513 2026-02-23T16:21:39.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:21:39 trial068 ceph-mon[38805]: pgmap v6512: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:21:39.953 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11293 8 365281 365273 0.252677 0 - 0.0299513 2026-02-23T16:21:40.713 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:21:40.828 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:21:40.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:21:40 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2026-02-23T16:21:40.927 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:21:40.925+0000 7f875ed9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:21:40.927 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:21:40.925+0000 7f875ed9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:21:40.953 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11294 8 365281 365273 0.252654 0 - 0.0299513 2026-02-23T16:21:41.038 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:21:41.038 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":379,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:20:55.492970+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:16:14.421864+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:19:34.245890+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:20:05.790461+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:21:41.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:21:41 trial068 ceph-mon[38805]: pgmap v6513: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:21:41.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:21:41 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-23T16:21:41.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:21:41 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2026-02-23T16:21:41.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:21:41 trial068 ceph-mon[38805]: pgmap v6514: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:21:41.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:21:41 trial068 ceph-mon[38805]: pgmap v6515: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:21:41.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:21:41 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T16:21:41.845 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:21:41 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2026-02-23T16:21:41.845 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:21:41 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/245871960' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:21:41.953 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11295 8 365281 365273 0.252632 0 - 0.0299513 2026-02-23T16:21:42.954 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11296 8 365281 365273 0.25261 0 - 0.0299513 2026-02-23T16:21:43.954 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11297 8 365281 365273 0.252587 0 - 0.0299513 2026-02-23T16:21:44.076 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:21:44.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:21:43 trial068 ceph-mon[38805]: pgmap v6516: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:21:44.094 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 16:21:43 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:16:21:43] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T16:21:44.195 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:21:44.299 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:21:44.298+0000 7fdcfe70d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:21:44.300 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:21:44.298+0000 7fdcfe70d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:21:44.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 16:21:43 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T16:21:43.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T16:21:44.408 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:21:44.409 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":379,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:20:55.492970+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:16:14.421864+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:19:34.245890+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:20:05.790461+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:21:44.954 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11298 8 365281 365273 0.252565 0 - 0.0299513 2026-02-23T16:21:45.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:21:44 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/4007759318' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:21:45.954 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11299 8 365281 365273 0.252543 0 - 0.0299513 2026-02-23T16:21:46.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:21:45 trial068 ceph-mon[38805]: pgmap v6517: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:21:46.954 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T16:21:46.953508+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T16:21:46.954 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T16:21:46.954 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11300 8 365281 365273 0.25252 0 - 0.0299513 2026-02-23T16:21:47.447 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:21:47.565 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:21:47.672 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:21:47.670+0000 7f60f1c03640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:21:47.672 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:21:47.670+0000 7f60f1c03640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:21:47.784 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:21:47.784 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":379,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:20:55.492970+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:16:14.421864+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:19:34.245890+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:20:05.790461+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:21:47.954 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11301 8 365281 365273 0.252498 0 - 0.0299513 2026-02-23T16:21:48.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:21:47 trial068 ceph-mon[38805]: pgmap v6518: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:21:48.954 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11302 8 365281 365273 0.252476 0 - 0.0299513 2026-02-23T16:21:49.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:21:48 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1855288019' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:21:49.954 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11303 8 365281 365273 0.252453 0 - 0.0299513 2026-02-23T16:21:50.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:21:49 trial068 ceph-mon[38805]: pgmap v6519: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:21:50.821 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:21:50.938 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:21:50.954 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11304 8 365281 365273 0.252431 0 - 0.0299513 2026-02-23T16:21:51.041 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:21:51.039+0000 7f5eb1f8b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:21:51.041 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:21:51.040+0000 7f5eb1f8b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:21:51.150 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:21:51.151 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":379,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:20:55.492970+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:16:14.421864+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:19:34.245890+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:20:05.790461+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:21:51.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:21:51 trial068 ceph-mon[38805]: pgmap v6520: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:21:51.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:21:51 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/801848887' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:21:51.954 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11305 8 365281 365273 0.252409 0 - 0.0299513 2026-02-23T16:21:52.954 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11306 8 365281 365273 0.252386 0 - 0.0299513 2026-02-23T16:21:53.954 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11307 8 365281 365273 0.252364 0 - 0.0299513 2026-02-23T16:21:54.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:21:53 trial068 ceph-mon[38805]: pgmap v6521: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:21:54.094 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 16:21:53 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:16:21:53] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T16:21:54.187 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:21:54.304 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:21:54.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 16:21:53 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T16:21:53.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T16:21:54.406 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:21:54.404+0000 7fb0e1505640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:21:54.407 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:21:54.405+0000 7fb0e1505640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:21:54.517 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:21:54.518 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":379,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:20:55.492970+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:16:14.421864+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:19:34.245890+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:20:05.790461+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:21:54.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:21:54 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3275947265' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:21:54.954 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11308 8 365281 365273 0.252342 0 - 0.0299513 2026-02-23T16:21:55.954 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11309 8 365281 365273 0.252319 0 - 0.0299513 2026-02-23T16:21:56.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:21:55 trial068 ceph-mon[38805]: pgmap v6522: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:21:56.954 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11310 8 365281 365273 0.252297 0 - 0.0299513 2026-02-23T16:21:57.576 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:21:57.694 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:21:57.796 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:21:57.794+0000 7fe9c63ff640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:21:57.796 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:21:57.795+0000 7fe9c63ff640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:21:57.906 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:21:57.906 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":380,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:21:57.731554+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:21:03.321160+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:19:34.245890+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:20:05.790461+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:21:57.906 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:21:57 trial068 ceph-mon[38805]: pgmap v6523: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:21:57.954 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11311 8 365281 365273 0.252275 0 - 0.0299513 2026-02-23T16:21:58.954 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11312 8 365281 365273 0.252252 0 - 0.0299513 2026-02-23T16:21:59.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:21:58 trial068 ceph-mon[38805]: osdmap e380: 4 total, 3 up, 4 in 2026-02-23T16:21:59.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:21:58 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/269191521' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:21:59.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:21:58 trial068 ceph-mon[38805]: 5.5 scrub starts 2026-02-23T16:21:59.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:21:58 trial068 ceph-mon[38805]: 5.5 scrub ok 2026-02-23T16:21:59.954 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11313 8 365281 365273 0.25223 0 - 0.0299513 2026-02-23T16:22:00.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:21:59 trial068 ceph-mon[38805]: pgmap v6525: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:22:00.942 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:22:00.954 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11314 8 365281 365273 0.252208 0 - 0.0299513 2026-02-23T16:22:01.060 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:22:01.158 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:22:01.156+0000 7f9bfa295640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:22:01.158 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:22:01.156+0000 7f9bfa295640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:22:01.270 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:22:01.270 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":381,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:22:00.500907+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:21:03.321160+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:19:34.245890+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:20:05.790461+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:22:01.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:22:01 trial068 ceph-mon[38805]: osdmap e381: 4 total, 3 up, 4 in 2026-02-23T16:22:01.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:22:01 trial068 ceph-mon[38805]: pgmap v6527: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:22:01.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:22:01 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2269680951' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:22:01.954 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11315 8 365281 365273 0.252186 0 - 0.0299513 2026-02-23T16:22:02.955 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11316 8 365281 365273 0.252163 0 - 0.0299513 2026-02-23T16:22:03.955 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11317 8 365281 365273 0.252141 0 - 0.0299513 2026-02-23T16:22:04.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:22:03 trial068 ceph-mon[38805]: pgmap v6528: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:22:04.094 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 16:22:03 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:16:22:03] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T16:22:04.266 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 16:22:03 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T16:22:03.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T16:22:04.308 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:22:04.427 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:22:04.536 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:22:04.534+0000 7f068317e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:22:04.536 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:22:04.534+0000 7f068317e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:22:04.646 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:22:04.646 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":381,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:22:00.500907+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:21:03.321160+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:19:34.245890+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:20:05.790461+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:22:04.955 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11318 8 365281 365273 0.252119 0 - 0.0299513 2026-02-23T16:22:05.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:22:04 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3876129765' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:22:05.955 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11319 8 365281 365273 0.252096 0 - 0.0299513 2026-02-23T16:22:06.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:22:05 trial068 ceph-mon[38805]: pgmap v6529: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:22:06.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:22:05 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T16:22:06.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:22:05 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T16:22:06.955 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T16:22:06.954756+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T16:22:06.955 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T16:22:06.955 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11320 8 365281 365273 0.252074 0 - 0.0299513 2026-02-23T16:22:07.683 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:22:07.801 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:22:07.898 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:22:07.896+0000 7f80e1a16640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:22:07.898 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:22:07.896+0000 7f80e1a16640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:22:07.955 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11321 8 365281 365273 0.252052 0 - 0.0299513 2026-02-23T16:22:08.007 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:22:08.007 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":381,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:22:00.500907+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:21:03.321160+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:19:34.245890+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:20:05.790461+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:22:08.008 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:22:07 trial068 ceph-mon[38805]: pgmap v6530: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:22:08.955 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11322 8 365281 365273 0.25203 0 - 0.0299513 2026-02-23T16:22:09.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:22:08 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1603450592' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:22:09.955 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11323 8 365281 365273 0.252007 0 - 0.0299513 2026-02-23T16:22:10.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:22:09 trial068 ceph-mon[38805]: pgmap v6531: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:22:10.955 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11324 8 365281 365273 0.251985 0 - 0.0299513 2026-02-23T16:22:11.047 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:22:11.164 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:22:11.265 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:22:11.263+0000 7fb8068b8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:22:11.266 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:22:11.264+0000 7fb8068b8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:22:11.374 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:22:11.374 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":381,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:22:00.500907+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:21:03.321160+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:19:34.245890+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:20:05.790461+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:22:11.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:22:11 trial068 ceph-mon[38805]: pgmap v6532: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:22:11.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:22:11 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2293684648' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:22:11.955 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11325 8 365281 365273 0.251963 0 - 0.0299513 2026-02-23T16:22:12.956 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11326 8 365281 365273 0.251941 0 - 0.0299513 2026-02-23T16:22:13.955 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11327 8 365281 365273 0.251918 0 - 0.0299513 2026-02-23T16:22:14.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:22:13 trial068 ceph-mon[38805]: pgmap v6533: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:22:14.094 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 16:22:13 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:16:22:13] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T16:22:14.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 16:22:13 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T16:22:13.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T16:22:14.427 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:22:14.547 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:22:14.646 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:22:14.644+0000 7fc8180f7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:22:14.646 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:22:14.644+0000 7fc8180f7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:22:14.760 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:22:14.761 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":381,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:22:00.500907+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:21:03.321160+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:19:34.245890+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:20:05.790461+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:22:14.955 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11328 8 365281 365273 0.251896 0 - 0.0299513 2026-02-23T16:22:15.956 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11329 8 365281 365273 0.251874 0 - 0.0299513 2026-02-23T16:22:16.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:22:15 trial068 ceph-mon[38805]: pgmap v6534: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:22:16.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:22:15 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1294322615' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:22:16.956 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11330 8 365281 365273 0.251852 0 - 0.0299513 2026-02-23T16:22:17.797 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:22:17.914 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:22:17.956 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11331 8 365281 365273 0.251829 0 - 0.0299513 2026-02-23T16:22:18.009 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:22:18.007+0000 7f62048aa640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:22:18.009 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:22:18.007+0000 7f62048aa640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:22:18.009 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:22:17 trial068 ceph-mon[38805]: pgmap v6535: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:22:18.117 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:22:18.118 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":381,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:22:00.500907+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:21:03.321160+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:19:34.245890+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:20:05.790461+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:22:18.956 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11332 8 365281 365273 0.251807 0 - 0.0299513 2026-02-23T16:22:19.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:22:18 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1998501166' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:22:19.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:22:18 trial068 ceph-mon[38805]: 5.7 scrub starts 2026-02-23T16:22:19.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:22:18 trial068 ceph-mon[38805]: 5.7 scrub ok 2026-02-23T16:22:19.956 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11333 8 365281 365273 0.251785 0 - 0.0299513 2026-02-23T16:22:20.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:22:19 trial068 ceph-mon[38805]: pgmap v6536: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:22:20.956 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11334 8 365281 365273 0.251763 0 - 0.0299513 2026-02-23T16:22:21.155 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:22:21.273 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:22:21.372 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:22:21.370+0000 7f4303bce640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:22:21.372 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:22:21.370+0000 7f4303bce640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:22:21.481 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:22:21.481 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":381,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:22:00.500907+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:21:03.321160+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:19:34.245890+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:20:05.790461+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:22:21.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:22:21 trial068 ceph-mon[38805]: pgmap v6537: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:22:21.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:22:21 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/4068228500' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:22:21.956 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11335 8 365281 365273 0.251741 0 - 0.0299513 2026-02-23T16:22:22.956 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11336 8 365281 365273 0.251718 0 - 0.0299513 2026-02-23T16:22:23.956 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11337 8 365281 365273 0.251696 0 - 0.0299513 2026-02-23T16:22:24.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:22:23 trial068 ceph-mon[38805]: pgmap v6538: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:22:24.094 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 16:22:23 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:16:22:23] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T16:22:24.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 16:22:23 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T16:22:23.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T16:22:24.518 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:22:24.637 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:22:24.743 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:22:24.741+0000 7fdff3e77640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:22:24.743 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:22:24.742+0000 7fdff3e77640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:22:24.852 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:22:24.853 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":381,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:22:00.500907+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:21:03.321160+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:19:34.245890+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:20:05.790461+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:22:24.956 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11338 8 365281 365273 0.251674 0 - 0.0299513 2026-02-23T16:22:25.956 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11339 8 365281 365273 0.251652 0 - 0.0299513 2026-02-23T16:22:26.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:22:25 trial068 ceph-mon[38805]: pgmap v6539: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:22:26.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:22:25 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3763705682' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:22:26.956 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T16:22:26.956151+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T16:22:26.956 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T16:22:26.956 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11340 8 365281 365273 0.25163 0 - 0.0299513 2026-02-23T16:22:27.891 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:22:27.956 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11341 8 365281 365273 0.251607 0 - 0.0299513 2026-02-23T16:22:28.009 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:22:28.031 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:22:27 trial068 ceph-mon[38805]: pgmap v6540: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:22:28.111 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:22:28.109+0000 7f54bff00640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:22:28.111 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:22:28.109+0000 7f54bff00640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:22:28.220 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:22:28.221 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":381,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:22:00.500907+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:21:03.321160+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:19:34.245890+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:20:05.790461+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:22:28.957 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11342 8 365281 365273 0.251585 0 - 0.0299513 2026-02-23T16:22:29.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:22:28 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1162090625' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:22:29.957 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11343 8 365281 365273 0.251563 0 - 0.0299513 2026-02-23T16:22:30.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:22:29 trial068 ceph-mon[38805]: pgmap v6541: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:22:30.957 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11344 8 365281 365273 0.251541 0 - 0.0299513 2026-02-23T16:22:31.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:22:30 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T16:22:31.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:22:30 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T16:22:31.259 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:22:31.378 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:22:31.483 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:22:31.481+0000 7efd0d9e4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:22:31.483 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:22:31.481+0000 7efd0d9e4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:22:31.592 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:22:31.592 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":381,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:22:00.500907+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:21:03.321160+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:19:34.245890+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:20:05.790461+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:22:31.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:22:31 trial068 ceph-mon[38805]: pgmap v6542: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:22:31.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:22:31 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/934113014' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:22:31.957 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11345 8 365281 365273 0.251519 0 - 0.0299513 2026-02-23T16:22:32.957 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11346 8 365281 365273 0.251497 0 - 0.0299513 2026-02-23T16:22:33.957 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11347 8 365281 365273 0.251474 0 - 0.0299513 2026-02-23T16:22:34.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:22:33 trial068 ceph-mon[38805]: pgmap v6543: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:22:34.094 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 16:22:33 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:16:22:33] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T16:22:34.266 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 16:22:33 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T16:22:33.974Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T16:22:34.629 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:22:34.747 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:22:34.850 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:22:34.848+0000 7f6d8dc34640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:22:34.851 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:22:34.848+0000 7f6d8dc34640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:22:34.957 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11348 8 365281 365273 0.251452 0 - 0.0299513 2026-02-23T16:22:34.961 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:22:34.961 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":381,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:22:00.500907+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:21:03.321160+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:19:34.245890+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:20:05.790461+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:22:35.957 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11349 8 365281 365273 0.25143 0 - 0.0299513 2026-02-23T16:22:36.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:22:35 trial068 ceph-mon[38805]: pgmap v6544: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:22:36.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:22:35 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/4245171199' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:22:36.957 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11350 8 365281 365273 0.251408 0 - 0.0299513 2026-02-23T16:22:37.957 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11351 8 365281 365273 0.251386 0 - 0.0299513 2026-02-23T16:22:37.999 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:22:38.025 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:22:37 trial068 ceph-mon[38805]: pgmap v6545: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:22:38.025 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:22:37 trial068 ceph-mon[38805]: 5.8 scrub starts 2026-02-23T16:22:38.025 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:22:37 trial068 ceph-mon[38805]: 5.8 scrub ok 2026-02-23T16:22:38.118 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:22:38.219 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:22:38.217+0000 7f4047e7e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:22:38.219 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:22:38.217+0000 7f4047e7e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:22:38.328 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:22:38.329 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":381,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:22:00.500907+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:21:03.321160+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:19:34.245890+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:20:05.790461+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:22:38.957 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11352 8 365281 365273 0.251364 0 - 0.0299513 2026-02-23T16:22:39.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:22:38 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3558329497' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:22:39.957 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11353 8 365281 365273 0.251341 0 - 0.0299513 2026-02-23T16:22:40.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:22:39 trial068 ceph-mon[38805]: pgmap v6546: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:22:40.958 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11354 8 365281 365273 0.251319 0 - 0.0299513 2026-02-23T16:22:41.367 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:22:41.483 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:22:41.587 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:22:41.585+0000 7f0d416a0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:22:41.587 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:22:41.585+0000 7f0d416a0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:22:41.699 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:22:41.699 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":381,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:22:00.500907+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:21:03.321160+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:19:34.245890+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:20:05.790461+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:22:41.699 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:22:41 trial068 ceph-mon[38805]: pgmap v6547: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:22:41.699 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:22:41 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2026-02-23T16:22:41.700 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:22:41 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-23T16:22:41.700 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:22:41 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2026-02-23T16:22:41.700 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:22:41 trial068 ceph-mon[38805]: pgmap v6548: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:22:41.700 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:22:41 trial068 ceph-mon[38805]: pgmap v6549: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:22:41.700 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:22:41 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T16:22:41.700 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:22:41 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2026-02-23T16:22:41.958 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11355 8 365281 365273 0.251297 0 - 0.0299513 2026-02-23T16:22:42.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:22:42 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3665246019' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:22:42.958 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11356 8 365281 365273 0.251275 0 - 0.0299513 2026-02-23T16:22:43.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:22:43 trial068 ceph-mon[38805]: pgmap v6550: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:22:43.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:22:43 trial068 ceph-mon[38805]: 5.a scrub starts 2026-02-23T16:22:43.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:22:43 trial068 ceph-mon[38805]: 5.a scrub ok 2026-02-23T16:22:43.958 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11357 8 365281 365273 0.251253 0 - 0.0299513 2026-02-23T16:22:44.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 16:22:43 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:16:22:43] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T16:22:44.355 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 16:22:43 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T16:22:43.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T16:22:44.737 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:22:44.857 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:22:44.958 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:22:44.955+0000 7fb72aa36640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:22:44.958 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:22:44.956+0000 7fb72aa36640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:22:44.958 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11358 8 365281 365273 0.251231 0 - 0.0299513 2026-02-23T16:22:45.067 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:22:45.067 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":381,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:22:00.500907+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:21:03.321160+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:19:34.245890+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:20:05.790461+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:22:45.958 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11359 8 365281 365273 0.251209 0 - 0.0299513 2026-02-23T16:22:46.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:22:45 trial068 ceph-mon[38805]: pgmap v6551: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:22:46.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:22:45 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1541332081' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:22:46.958 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T16:22:46.957813+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T16:22:46.958 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T16:22:46.958 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11360 8 365281 365273 0.251187 0 - 0.0299513 2026-02-23T16:22:47.958 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11361 8 365281 365273 0.251164 0 - 0.0299513 2026-02-23T16:22:48.104 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:22:48.222 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:22:48.243 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:22:47 trial068 ceph-mon[38805]: pgmap v6552: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:22:48.323 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:22:48.321+0000 7f840deb0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:22:48.323 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:22:48.321+0000 7f840deb0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:22:48.432 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:22:48.432 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":381,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:22:00.500907+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:21:03.321160+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:19:34.245890+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:20:05.790461+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:22:48.958 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11362 8 365281 365273 0.251142 0 - 0.0299513 2026-02-23T16:22:49.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:22:48 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/4169775280' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:22:49.958 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11363 8 365281 365273 0.25112 0 - 0.0299513 2026-02-23T16:22:50.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:22:49 trial068 ceph-mon[38805]: pgmap v6553: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:22:50.958 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11364 8 365281 365273 0.251098 0 - 0.0299513 2026-02-23T16:22:51.467 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:22:51.583 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:22:51.683 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:22:51.681+0000 7f0fcbe42640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:22:51.684 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:22:51.682+0000 7f0fcbe42640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:22:51.794 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:22:51.794 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":381,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:22:00.500907+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:21:03.321160+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:19:34.245890+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:20:05.790461+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:22:51.795 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:22:51 trial068 ceph-mon[38805]: pgmap v6554: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:22:51.959 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11365 8 365281 365273 0.251076 0 - 0.0299513 2026-02-23T16:22:52.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:22:52 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2216656837' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:22:52.958 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11366 8 365281 365273 0.251054 0 - 0.0299513 2026-02-23T16:22:53.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:22:53 trial068 ceph-mon[38805]: pgmap v6555: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:22:53.958 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11367 8 365281 365273 0.251032 0 - 0.0299513 2026-02-23T16:22:54.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 16:22:53 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:16:22:53] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T16:22:54.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 16:22:53 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T16:22:53.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T16:22:54.831 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:22:54.950 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:22:54.958 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11368 8 365281 365273 0.25101 0 - 0.0299513 2026-02-23T16:22:55.050 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:22:55.048+0000 7f08db532640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:22:55.050 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:22:55.048+0000 7f08db532640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:22:55.161 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:22:55.161 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":381,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:22:00.500907+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:21:03.321160+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:19:34.245890+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:20:05.790461+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:22:55.959 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11369 8 365281 365273 0.250988 0 - 0.0299513 2026-02-23T16:22:56.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:22:55 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T16:22:56.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:22:55 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T16:22:56.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:22:55 trial068 ceph-mon[38805]: pgmap v6556: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:22:56.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:22:55 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3419009629' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:22:56.959 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11370 8 365281 365273 0.250966 0 - 0.0299513 2026-02-23T16:22:57.959 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11371 8 365281 365273 0.250944 0 - 0.0299513 2026-02-23T16:22:58.199 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:22:58.317 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:22:58.340 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:22:57 trial068 ceph-mon[38805]: pgmap v6557: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:22:58.415 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:22:58.413+0000 7fd9554e5640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:22:58.416 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:22:58.414+0000 7fd9554e5640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:22:58.527 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:22:58.527 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":381,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:22:00.500907+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:21:03.321160+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:19:34.245890+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:20:05.790461+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:22:58.959 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11372 8 365281 365273 0.250922 0 - 0.0299513 2026-02-23T16:22:59.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:22:58 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3444448605' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:22:59.959 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11373 8 365281 365273 0.250899 0 - 0.0299513 2026-02-23T16:23:00.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:22:59 trial068 ceph-mon[38805]: pgmap v6558: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:23:00.959 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11374 8 365281 365273 0.250877 0 - 0.0299513 2026-02-23T16:23:01.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:23:00 trial068 ceph-mon[38805]: 5.e scrub starts 2026-02-23T16:23:01.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:23:00 trial068 ceph-mon[38805]: 5.e scrub ok 2026-02-23T16:23:01.566 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:23:01.683 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:23:01.790 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:23:01.788+0000 7fce86d3c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:23:01.791 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:23:01.789+0000 7fce86d3c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:23:01.902 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:23:01.902 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":381,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:22:00.500907+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:21:03.321160+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:19:34.245890+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:20:05.790461+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:23:01.959 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11375 8 365281 365273 0.250855 0 - 0.0299513 2026-02-23T16:23:02.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:23:01 trial068 ceph-mon[38805]: pgmap v6559: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:23:02.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:23:01 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/901623888' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:23:02.959 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11376 8 365281 365273 0.250833 0 - 0.0299513 2026-02-23T16:23:03.959 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11377 8 365281 365273 0.250811 0 - 0.0299513 2026-02-23T16:23:04.265 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 16:23:03 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T16:23:03.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T16:23:04.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:23:03 trial068 ceph-mon[38805]: pgmap v6560: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:23:04.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 16:23:03 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:16:23:03] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T16:23:04.940 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:23:04.959 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11378 8 365281 365273 0.250789 0 - 0.0299513 2026-02-23T16:23:05.058 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:23:05.162 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:23:05.160+0000 7f830d03a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:23:05.163 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:23:05.161+0000 7f830d03a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:23:05.273 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:23:05.273 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":381,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:22:00.500907+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:21:03.321160+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:19:34.245890+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:20:05.790461+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:23:05.959 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11379 8 365281 365273 0.250767 0 - 0.0299513 2026-02-23T16:23:06.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:23:06 trial068 ceph-mon[38805]: pgmap v6561: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:23:06.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:23:06 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1177467513' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:23:06.960 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T16:23:06.959404+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T16:23:06.960 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T16:23:06.960 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11380 8 365281 365273 0.250745 0 - 0.0299513 2026-02-23T16:23:07.960 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11381 8 365281 365273 0.250723 0 - 0.0299513 2026-02-23T16:23:08.310 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:23:08.336 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:23:08 trial068 ceph-mon[38805]: pgmap v6562: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:23:08.428 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:23:08.528 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:23:08.526+0000 7f18e46bb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:23:08.528 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:23:08.526+0000 7f18e46bb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:23:08.641 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:23:08.642 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":381,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:22:00.500907+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:21:03.321160+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:19:34.245890+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:20:05.790461+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:23:08.960 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11382 8 365281 365273 0.250701 0 - 0.0299513 2026-02-23T16:23:09.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:23:09 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3020579013' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:23:09.960 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11383 8 365281 365273 0.250679 0 - 0.0299513 2026-02-23T16:23:10.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:23:10 trial068 ceph-mon[38805]: pgmap v6563: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:23:10.960 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11384 8 365281 365273 0.250657 0 - 0.0299513 2026-02-23T16:23:11.679 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:23:11.797 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:23:11.820 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:23:11 trial068 ceph-mon[38805]: pgmap v6564: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:23:11.896 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:23:11.895+0000 7fe2ac037640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:23:11.897 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:23:11.895+0000 7fe2ac037640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:23:11.960 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11385 8 365281 365273 0.250635 0 - 0.0299513 2026-02-23T16:23:12.008 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:23:12.008 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":381,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:22:00.500907+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:21:03.321160+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:19:34.245890+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:20:05.790461+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:23:12.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:23:12 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1013447485' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:23:12.960 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11386 8 365281 365273 0.250613 0 - 0.0299513 2026-02-23T16:23:13.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:23:13 trial068 ceph-mon[38805]: pgmap v6565: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:23:13.960 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11387 8 365281 365273 0.250591 0 - 0.0299513 2026-02-23T16:23:14.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 16:23:13 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:16:23:13] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T16:23:14.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 16:23:13 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T16:23:13.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T16:23:14.960 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11388 8 365281 365273 0.250569 0 - 0.0299513 2026-02-23T16:23:15.045 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:23:15.162 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:23:15.259 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:23:15.257+0000 7fd657ebc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:23:15.259 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:23:15.257+0000 7fd657ebc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:23:15.369 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:23:15.369 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":381,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:22:00.500907+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:21:03.321160+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:19:34.245890+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:20:05.790461+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:23:15.960 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11389 8 365281 365273 0.250547 0 - 0.0299513 2026-02-23T16:23:16.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:23:15 trial068 ceph-mon[38805]: pgmap v6566: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:23:16.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:23:15 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3555256633' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:23:16.960 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11390 8 365281 365273 0.250525 0 - 0.0299513 2026-02-23T16:23:17.960 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11391 8 365281 365273 0.250503 0 - 0.0299513 2026-02-23T16:23:18.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:23:17 trial068 ceph-mon[38805]: pgmap v6567: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:23:18.406 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:23:18.523 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:23:18.621 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:23:18.619+0000 7f115c538640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:23:18.621 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:23:18.619+0000 7f115c538640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:23:18.730 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:23:18.730 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":381,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:22:00.500907+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:21:03.321160+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:19:34.245890+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:20:05.790461+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:23:18.960 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11392 8 365281 365273 0.250481 0 - 0.0299513 2026-02-23T16:23:19.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:23:18 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/4174249269' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:23:19.961 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11393 8 365281 365273 0.250459 0 - 0.0299513 2026-02-23T16:23:20.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:23:19 trial068 ceph-mon[38805]: pgmap v6568: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:23:20.961 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11394 8 365281 365273 0.250437 0 - 0.0299513 2026-02-23T16:23:21.766 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:23:21.884 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:23:21.961 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11395 8 365281 365273 0.250415 0 - 0.0299513 2026-02-23T16:23:21.985 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:23:21.983+0000 7f327ca6a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:23:21.985 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:23:21.983+0000 7f327ca6a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:23:22.094 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:23:22 trial068 ceph-mon[38805]: pgmap v6569: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:23:22.095 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:23:22.095 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":381,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:22:00.500907+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:21:03.321160+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:19:34.245890+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:20:05.790461+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:23:22.961 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11396 8 365281 365273 0.250393 0 - 0.0299513 2026-02-23T16:23:23.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:23:23 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3333891439' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:23:23.961 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11397 8 365281 365273 0.250371 0 - 0.0299513 2026-02-23T16:23:24.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:23:24 trial068 ceph-mon[38805]: pgmap v6570: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:23:24.344 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 16:23:23 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:16:23:23] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T16:23:24.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 16:23:23 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T16:23:23.972Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T16:23:24.961 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11398 8 365281 365273 0.250349 0 - 0.0299513 2026-02-23T16:23:25.135 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:23:25.259 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:23:25.283 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:23:25 trial068 ceph-mon[38805]: 5.5 scrub starts 2026-02-23T16:23:25.284 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:23:25 trial068 ceph-mon[38805]: 5.5 scrub ok 2026-02-23T16:23:25.365 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:23:25.363+0000 7fba824cf640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:23:25.365 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:23:25.363+0000 7fba824cf640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:23:25.475 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:23:25.475 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":381,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:22:00.500907+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:21:03.321160+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:19:34.245890+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:20:05.790461+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:23:25.961 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11399 8 365281 365273 0.250327 0 - 0.0299513 2026-02-23T16:23:26.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:23:26 trial068 ceph-mon[38805]: pgmap v6571: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:23:26.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:23:26 trial068 ceph-mon[38805]: 5.7 scrub starts 2026-02-23T16:23:26.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:23:26 trial068 ceph-mon[38805]: 5.7 scrub ok 2026-02-23T16:23:26.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:23:26 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1986764983' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:23:26.961 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T16:23:26.961011+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T16:23:26.961 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T16:23:26.961 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11400 8 365281 365273 0.250305 0 - 0.0299513 2026-02-23T16:23:27.961 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11401 8 365281 365273 0.250283 0 - 0.0299513 2026-02-23T16:23:28.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:23:28 trial068 ceph-mon[38805]: pgmap v6572: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:23:28.513 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:23:28.631 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:23:28.734 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:23:28.732+0000 7f3fac0d0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:23:28.734 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:23:28.733+0000 7f3fac0d0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:23:28.842 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:23:28.842 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":381,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:22:00.500907+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:21:03.321160+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:19:34.245890+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:20:05.790461+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:23:28.961 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11402 8 365281 365273 0.250261 0 - 0.0299513 2026-02-23T16:23:29.093 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:23:29 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1732482487' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:23:29.961 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11403 8 365281 365273 0.250239 0 - 0.0299513 2026-02-23T16:23:30.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:23:30 trial068 ceph-mon[38805]: pgmap v6573: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:23:30.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:23:30 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T16:23:30.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:23:30 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T16:23:30.961 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11404 8 365281 365273 0.250217 0 - 0.0299513 2026-02-23T16:23:31.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:23:31 trial068 ceph-mon[38805]: pgmap v6574: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:23:31.878 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:23:31.961 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11405 8 365281 365273 0.250195 0 - 0.0299513 2026-02-23T16:23:31.995 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:23:32.094 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:23:32.092+0000 7f4dab983640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:23:32.094 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:23:32.092+0000 7f4dab983640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:23:32.205 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:23:32.206 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":381,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:22:00.500907+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:21:03.321160+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:19:34.245890+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:20:05.790461+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:23:32.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:23:32 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/2658868149' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:23:32.962 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11406 8 365281 365273 0.250174 0 - 0.0299513 2026-02-23T16:23:33.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:23:33 trial068 ceph-mon[38805]: pgmap v6575: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:23:33.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:23:33 trial068 ceph-mon[38805]: purged_snaps scrub starts 2026-02-23T16:23:33.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:23:33 trial068 ceph-mon[38805]: purged_snaps scrub ok 2026-02-23T16:23:33.962 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11407 8 365281 365273 0.250152 0 - 0.0299513 2026-02-23T16:23:34.265 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 16:23:33 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T16:23:33.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T16:23:34.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 16:23:33 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:16:23:33] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T16:23:34.962 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11408 8 365281 365273 0.25013 0 - 0.0299513 2026-02-23T16:23:35.244 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:23:35.363 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:23:35.472 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:23:35.471+0000 7f631647b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:23:35.473 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:23:35.471+0000 7f631647b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:23:35.584 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:23:35.585 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":381,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:22:00.500907+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:21:03.321160+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:19:34.245890+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:20:05.790461+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:23:35.962 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11409 8 365281 365273 0.250108 0 - 0.0299513 2026-02-23T16:23:36.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:23:35 trial068 ceph-mon[38805]: pgmap v6576: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:23:36.344 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:23:35 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/879301148' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:23:36.962 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11410 8 365281 365273 0.250086 0 - 0.0299513 2026-02-23T16:23:37.962 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11411 8 365281 365273 0.250064 0 - 0.0299513 2026-02-23T16:23:38.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:23:37 trial068 ceph-mon[38805]: pgmap v6577: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:23:38.622 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:23:38.740 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:23:38.843 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:23:38.841+0000 7f46b67c0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:23:38.843 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:23:38.841+0000 7f46b67c0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:23:38.953 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:23:38.953 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":381,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:22:00.500907+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:21:03.321160+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:19:34.245890+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:20:05.790461+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:23:38.962 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11412 8 365281 365273 0.250042 0 - 0.0299513 2026-02-23T16:23:39.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:23:38 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1565003801' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:23:39.962 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11413 8 365281 365273 0.25002 0 - 0.0299513 2026-02-23T16:23:40.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:23:40 trial068 ceph-mon[38805]: pgmap v6578: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:23:40.962 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11414 8 365281 365273 0.249998 0 - 0.0299513 2026-02-23T16:23:41.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:23:41 trial068 ceph-mon[38805]: pgmap v6579: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:23:41.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:23:41 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2026-02-23T16:23:41.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:23:41 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2026-02-23T16:23:41.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:23:41 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2026-02-23T16:23:41.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:23:41 trial068 ceph-mon[38805]: pgmap v6580: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:23:41.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:23:41 trial068 ceph-mon[38805]: pgmap v6581: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:23:41.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:23:41 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' 2026-02-23T16:23:41.844 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:23:41 trial068 ceph-mon[38805]: from='mgr.14306 10.20.193.68:0/3164286063' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2026-02-23T16:23:41.962 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11415 8 365281 365273 0.249976 0 - 0.0299513 2026-02-23T16:23:41.991 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:23:42.106 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:23:42.208 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:23:42.207+0000 7f85a95d0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:23:42.209 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:23:42.207+0000 7f85a95d0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:23:42.320 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:23:42.320 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":381,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:22:00.500907+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:21:03.321160+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:19:34.245890+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:20:05.790461+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:23:42.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:23:42 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/701594176' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:23:42.962 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11416 8 365281 365273 0.249954 0 - 0.0299513 2026-02-23T16:23:43.843 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:23:43 trial068 ceph-mon[38805]: pgmap v6582: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:23:43.962 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11417 8 365281 365273 0.249932 0 - 0.0299513 2026-02-23T16:23:44.343 INFO:journalctl@ceph.mgr.x.trial068.stdout:Feb 23 16:23:43 trial068 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-mgr-x[39067]: ::ffff:10.20.193.185 - - [23/Feb/2026:16:23:43] "GET /metrics HTTP/1.1" 200 4218 "" "Prometheus/3.6.0" 2026-02-23T16:23:44.356 INFO:journalctl@ceph.prometheus.a.trial185.stdout:Feb 23 16:23:43 trial185 ceph-5f823084-10b8-11f1-be80-d404e6e7d460-prometheus-a[65436]: time=2026-02-23T16:23:43.973Z level=ERROR source=scrape.go:1630 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=ceph target=http://trial068:9283/metrics content_type="text/html;charset=utf-8" fallback_media_type="" err="received unsupported Content-Type \"text/html;charset=utf-8\" and no fallback_scrape_protocol specified for target" 2026-02-23T16:23:44.962 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11418 8 365281 365273 0.249911 0 - 0.0299513 2026-02-23T16:23:45.359 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:23:45.478 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:23:45.583 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:23:45.582+0000 7f727a6e0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:23:45.583 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:23:45.582+0000 7f727a6e0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:23:45.692 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:23:45.692 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":381,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:22:00.500907+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:21:03.321160+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:19:34.245890+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:20:05.790461+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:23:45.963 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11419 8 365281 365273 0.249889 0 - 0.0299513 2026-02-23T16:23:46.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:23:46 trial068 ceph-mon[38805]: pgmap v6583: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:23:46.594 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:23:46 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/1501584528' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:23:46.963 INFO:tasks.radosbench.radosbench.0.trial068.stdout:2026-02-23T16:23:46.962598+0000 min lat: 0.000629733 max lat: 2.42574 avg lat: 0.0299513 2026-02-23T16:23:46.963 INFO:tasks.radosbench.radosbench.0.trial068.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-23T16:23:46.963 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11420 8 365281 365273 0.249867 0 - 0.0299513 2026-02-23T16:23:47.963 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11421 8 365281 365273 0.249845 0 - 0.0299513 2026-02-23T16:23:48.593 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:23:48 trial068 ceph-mon[38805]: pgmap v6584: 17 pgs: 1 active+recovery_unfound+undersized+degraded, 5 active+clean, 11 active+undersized+degraded; 2.8 GiB data, 10 GiB used, 2.7 TiB / 2.7 TiB avail; 31459/136989 objects degraded (22.965%); 1/45663 objects unfound (0.002%) 2026-02-23T16:23:48.730 DEBUG:teuthology.orchestra.run.trial068:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:e371433fcd1390a0b82cfb18effc422c1f5d8c1d shell --fsid 5f823084-10b8-11f1-be80-d404e6e7d460 -- ceph osd dump --format=json 2026-02-23T16:23:48.848 INFO:teuthology.orchestra.run.trial068.stderr:Inferring config /var/lib/ceph/5f823084-10b8-11f1-be80-d404e6e7d460/mon.a/config 2026-02-23T16:23:48.945 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:23:48.944+0000 7fc8d3870640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:23:48.945 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-23T16:23:48.944+0000 7fc8d3870640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T16:23:48.963 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11422 8 365281 365273 0.249823 0 - 0.0299513 2026-02-23T16:23:49.056 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-23T16:23:49.057 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":381,"fsid":"5f823084-10b8-11f1-be80-d404e6e7d460","created":"2026-02-23T13:06:03.098686+0000","modified":"2026-02-23T16:22:00.500907+0000","last_up_change":"2026-02-23T13:13:44.004587+0000","last_in_change":"2026-02-23T13:07:03.096347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T13:07:04.695348+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2026-02-23T13:13:23.465456+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":0.75,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e7d2db8-8070-4095-a077-d0f8420f7d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":99,"up_from":106,"up_thru":183,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6803","nonce":3423688218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6805","nonce":3423688218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6809","nonce":3423688218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":3423688218},{"type":"v1","addr":"10.20.193.68:6807","nonce":3423688218}]},"public_addr":"10.20.193.68:6803/3423688218","cluster_addr":"10.20.193.68:6805/3423688218","heartbeat_back_addr":"10.20.193.68:6809/3423688218","heartbeat_front_addr":"10.20.193.68:6807/3423688218","state":["exists","up"]},{"osd":1,"uuid":"c6854288-3ff8-4295-80a0-8fcaafedb28f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":144,"last_clean_end":171,"up_from":176,"up_thru":181,"down_at":172,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6811","nonce":3818277133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6813","nonce":3818277133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6817","nonce":3818277133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":3818277133},{"type":"v1","addr":"10.20.193.68:6815","nonce":3818277133}]},"public_addr":"10.20.193.68:6811/3818277133","cluster_addr":"10.20.193.68:6813/3818277133","heartbeat_back_addr":"10.20.193.68:6817/3818277133","heartbeat_front_addr":"10.20.193.68:6815/3818277133","state":["exists","up"]},{"osd":2,"uuid":"25faf9eb-f475-49fe-8eeb-0663163402e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":152,"last_clean_end":162,"up_from":166,"up_thru":178,"down_at":180,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6800","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6801","nonce":309948183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6802","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6803","nonce":309948183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6806","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6807","nonce":309948183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6804","nonce":309948183},{"type":"v1","addr":"10.20.193.185:6805","nonce":309948183}]},"public_addr":"10.20.193.185:6801/309948183","cluster_addr":"10.20.193.185:6803/309948183","heartbeat_back_addr":"10.20.193.185:6807/309948183","heartbeat_front_addr":"10.20.193.185:6805/309948183","state":["exists"]},{"osd":3,"uuid":"e606eee6-fc74-4c50-bfdc-a6a2adf9a945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":74,"up_from":80,"up_thru":182,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6808","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6809","nonce":3696769086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6810","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6811","nonce":3696769086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6814","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6815","nonce":3696769086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.185:6812","nonce":3696769086},{"type":"v1","addr":"10.20.193.185:6813","nonce":3696769086}]},"public_addr":"10.20.193.185:6809/3696769086","cluster_addr":"10.20.193.185:6811/3696769086","heartbeat_back_addr":"10.20.193.185:6815/3696769086","heartbeat_front_addr":"10.20.193.185:6813/3696769086","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:21:03.321160+0000","dead_epoch":98},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:19:34.245890+0000","dead_epoch":171},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T13:13:18.585887+0000","dead_epoch":179},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-23T16:20:05.790461+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.68:0/2086895042":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/1209278317":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3495130282":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:6801/2061884286":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/3103836131":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2255147122":"2026-02-24T13:06:12.960687+0000","10.20.193.68:0/2848886808":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6800/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/55681093":"2026-02-24T13:06:22.483019+0000","10.20.193.68:6800/2332248704":"2026-02-24T13:07:24.217938+0000","10.20.193.68:6801/1870073644":"2026-02-24T13:06:22.483019+0000","10.20.193.68:0/1895012493":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2910016328":"2026-02-24T13:07:24.217938+0000","10.20.193.68:0/2415220450":"2026-02-24T13:06:22.483019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T16:23:49.343 INFO:journalctl@ceph.mon.a.trial068.stdout:Feb 23 16:23:49 trial068 ceph-mon[38805]: from='client.? 10.20.193.68:0/3058878536' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2026-02-23T16:23:49.963 INFO:tasks.radosbench.radosbench.0.trial068.stdout:11423 8 365281 365273 0.249801 0 - 0.0299513 2026-02-23T16:23:50.505 DEBUG:teuthology.exit:Got signal 15; running 1 handler... 2026-02-23T16:23:50.505 DEBUG:teuthology.task.console_log:Killing console logger for trial068 2026-02-23T16:23:50.506 DEBUG:teuthology.task.console_log:Killing console logger for trial185 2026-02-23T16:23:50.506 DEBUG:teuthology.exit:Finished running handlers